I need to move some data from one machine to another. Is it a good idea to write a client server app using sockets in Perl to do the transfer? Will I have problems if one side is written in Java?
I mean, should I be aware of any issues I might face when I try to attempt the above?
Short answer: Using a Perl program as the client or server is just fine. Your only problem might be your personal skill and experience level, but after you do it you know how to do it. :) Most of the problem is choosing how you need to do it, not the technology involved. Perl isn't going to be the problem, but it doesn't have an advantage over other languages either.
As some have already noted, the socket portion of the problem is going to be the same in most languages since almost everything uses the BSD stuff. Perl doesn't have any roadblocks or special gotchas for that. To move data around you create one side to listen on a socket and the other to open a connection and send the data. Easy peasy. You might want to check out Lincoln Stein's Network Programming with Perl for that bit. That can get you the low-level bits.
For higher-level networking, POE is very useful and easy to work with once you get started. It's a framework for dealing with event-driven programming and has many plugins to easily communicate between processes. You might spend a little time learning it, but it gives a lot back too.
If you aren't inventing your own protocol, there's most likely already a Perl module that can format and parse the messages.
If you just want to transfer data, there are several things you can do. The easiest in concept might be just to write lines to the socket and read them as lines from the other end. A bit more complicated than that is using something like Data::Dumper, YAML, or JSON to serialize data to text and send that. For more complex things, such as sharing Perl objects, you might want to use Storable. You freeze your objects, send them as data over the network, then thaw them on the other side.
If you want to implement your client and server in different languages you have a bit more work to figure out how they'll talk to each other. The socket stuff is mostly the same, but a Java server won't understand the output of Perl's Storable (it's possible, but you'll have to parse it yourself and that's no good :). If you do everything right, neither side should care what you used on the other side.
I can only think of one gotcha off the top of my head: most text based network protocols use CRLF for line endings, but Perl on UNIX type machines assumes LF endings by default, this means you will need to change the input and output record separators if you want to use readline (aka <>) and print (also beware of printf, since it doesn't use the output record separator). Of course, if you are going to use a pre-existing protocol, there is probably already a Net::<PROTOCOL NAME> module on CPAN, so you won't need to worry about that. If you are designing your own protocol, I would keep the CRLF convention because it makes it easy to debug the server with telnet (which is really the last valid use for that program).
You don't say whether you need to implement your application to support any particular protocol or whether you need to implement a home grown protocol. The networking support in Perl is flexible enough to support either (or most places in between).
At the low level socket end, your code is going to be fairly similar whatever language your are using - BSD socket APIs are pretty well the same everywhere they are supported. The support you need for this is built into Perl but low level socket programming can be frustrating - it's very low level.
However, Perl's standard library contains the Socket module which is rather easier to use (and well documented).
If you need to implement an existing protocol you may well find that it has already been implemented. For example Net::Telnet implements command/response protocols (like Telnet) making a client app trivial.
Searching CPAN may save you a lot of pain. Look at modules in the Net::* hierarchy
I don't think you're gonna have any major issues that you won't have by not using Perl. Even performance will be comparable to other solutions due to network latencies.
You might want to look at POE framework. It makes writing such components a breeze.
It probably depend on a few factors. Does speed or responsiveness matter? Are you moving data between they same type of machines (Unix to Unix, Windows to Windows)? What type of data are you trying to move (Text or Binary)? What is knowledge about sockets and what languages do you have experience?
I have sent and received binary data over PERL sockets from differing applications, but I don't have much experience with the text processing over sockets from differing machines. If you are moving data between machine you need to keep in mind the way the data is marshalled and if it is packed or aligned on some byte boundry. I have not exchanged data with a Java programs, but is should be similiar.
It probably would help to have some experience with PERL, and I would recommend looking at the examples in the "camel" book. I have used the ones in the book as a starting point and made modification for what I needed to achieve. You may have to consult some other areas of the book if you are dealing with binary data, or to help in doing translations for sending data.
Write socket communication in Perl is relatively easy. Do it right and reliable is big pain even CPAN modules are examples of error prone code. It depends of your expectations.
You are basically asking two questions:
Is Perl a proper language for socket communication?
Is Perl a proper language for UI?
Referring to e5's answer, Perl is indeed a string-centric language with a focus on readable strings, less well equipped to handle binary data. Thus the answer probably lies in the questions: Is your communication string based? Is your UI string based?
If doing binary interaction through a socket, well, you probably could be doing better than Perl (not talking about C, but maybe C-ish languages). If you want to do graphical user-interaction you probably reach faster results by choosing one of the higher languages that focus more on gui interaction. (Java-ish might be the thing here.)
Related
I would like to add a REST interface to an existing TCL codebase (so that the programms in other language can use the existing TCL code).
I found a list of Webserver with TCL support but I have no idea which one would be a good solution to quickly map our TCL functions to HTTP/REST calls without tons of boilerplate code.
Has anyone here already done something like this and can tell me which of these servers would be a good (or bad/difficult) solution?
Is there maybe another server/framework that is even better for this use case?
Consider Naviserver. Tcl is its embedded interpreter language. It has a low profile memory overhead, and is regularly maintained and tested for performance and low latency.
For what you’re describing, you might consider Wapp. It’ll do exactly the boilerplate elimination you want, and it’s easy to dive into. You’d probably want to use it as a library, rather than an app, given that you’ve got an existing codebase, but its operation past the initial setup is the same for that use case.
How could one interact with a locally long-running Common Lisp image, possibly daemonized, from the command line?
I know it is possible to run a Common Lisp function from a Terminal command prompt, I am also aware of this.
I would need to do a similar thing, but with a local, already long-running Common Lisp image, being able to poll available functions from the CLI or shell scripts.
Is there a way to do that from a CLI, for example calling a function from a bash script, and receiving back whatever the function returns?
I though one could, for example, create a primitive web service, perhaps using woo or Hunchentoot, calling functions and fetching returned values via curl or wget, but it feels a little convoluted.
Of course, that is one of the many features of Emacs' SLIME, but I would need to call functions just from the CLI, without invoking Emacs.
Is there perhaps a way to reach a swank backend, outside of SLIME?
If possible at all, what would be the lisp idiomatic way of doing that?
I would be grateful for any pointers.
Update
Additional note
Many years ago, I was intrigued by being able to telnet into a long-running LISP image (I believe in this case uppercasing the name should be fine). If I remember correctly, it was available at prompt.franz.com. An article, somehow connected: telnet for remote access to a running application
Telnet is of course quite unsafe, but the usefulness of being able to access the Lisp application(s) in that way, for whatever reason, cannot be overstated, at least to some people.
Some additional pointers, and thanks
I would like to thank Basile Starynkevitch for his elaborate and thorough answer, especially on the theoretical aspect. I was looking for a more practical direction, specially connected to Common Lisp. Still, his answer is very instructive.
I was all ready to start writing a local server, perhaps using one of the fine Common Lisp libraries, like:
usocket: Universal socket library for Common Lisp
iolib: Common Lisp I/O library
cl-aync: Asynchronous IO library for Common Lisp
But, thanks to Stanislav Kondratyev, I didn't have to. He pointed out an already existing solution that nicely answer my question, ScriptL: Shell scripting made Lisp-like
I tested it with success on Linux, FreeBSD and OS X, just make sure to install the thin wrapper over POSIX syscalls first. Among many features (see README), it allows exposition of just selected functions, security is properly handled, and it even supply a custom C client, which builds as part of the ASDF load operation, and supports a number of new features, such as I/O, in place of netcat.
You may find scriptl useful: http://quickdocs.org/scriptl/. However, it depends on iolib, which depends on some nonstandard C or C++ library, so building it is not quite straighforward.
It is indeed possible to communicate with a swank server if you familiarize yourself with the swank protocol, which seems to be underdocumented (see e. g. here: https://github.com/astine/swank-client/blob/master/swank-description.markdown). However, this exposes a TCP socket over a network, which could be unsafe. Once I tried that, too, but I was not satisfied with the IPC speed.
A while ago I wrote a rather naive SBCL-specific server that uses a local domain socket for communication, and a client in C. It's very raw, but you could take a look: https://github.com/quasus/lispserver. In particular, it supports interactive IO and exit codes. The server and the client form the core of a simple framework for deploying Unix style software. Feel free to borrow code and/or contact me for explanations, suggestions, etc.
It certainly is operating system specific, because you want some inter-process communication, and they are provided by the OS.
Let's assume you have a POSIX like OS, e.g. Linux.
Then you could set up a socket(7) or fifo(7) to which you send s-exprs to be evaluated. Of course you need to adapt the Common Lisp program to add such a REPL.
SBCL has some networking facilities, you could build on them.
Of course, you should understand first how IPC work on your OS. If it is Linux, you could read Advanced Linux Programming (it is centered on C programming, which is the low-level way of using OS services on POSIX, but you can adapt what you have learned to SBCL). And indeed, the Common Lisp standard might not have an official POSIX interface, so you need to dive into implementation specific details.
Perhaps you should learn more about BSD sockets. There are tons of tutorials on them. Then you could use TCP sockets (see tcp(7)) or Unix ones (see unix(7)). Advanced users could use the unsafe telnet command. Or you might make your software use SSL, or perhaps some libssh e.g. use ssh as their client.
You could decide and document that the protocol between user apps and your program is : send some-sexpr (on a documented socket) from user-app code to your server which is terminated by a double newline or by a form feed, and have your server eval it and send back the result or some error message. I did similar things in MELT and it is not a big deal. Be careful about buffering.
I guess that you assume that you have responsible and competent users (so don't open such a remote REPL to the wild Internet!). If you care about malicious or stupid use of a remote REPL, it is becoming complex.
Alternatively, make your server a web application (by using some HTTP server library inside it), and ask users to use their browser or some HTTP client program (curl) or library (libcurl) to interact with it.
looking into connecting to a secure ftp site (using perl), and downloading all the .log files, saving in new directories named after the day I downloaded the files. I want to do this without modules, as a learning experience, but before I start I wanted to know if you guys thought it was doing, or is way too much for a relatively new programmer and I should just learn the modules?
If it's production work, no, use the modules. Your implementation will be buggy, missing features and unknown to the next person maintaining that code.
Otherwise, yes. It's good to learn the principles of a network protocol. I do have a reservation about FTP as it is a bit baroque, insecure, inefficient and on its way out. scp, HTTP or rsync would be more useful to put your energy into.
I'd start with reading the RFC and putting together your own FTP module using just network sockets. Document and test it as if you were going to release to CPAN as a full learning exercise in making a network module. Run it against some various FTP server implementations as they often interpret the spec differently (or not at all). Don't be afraid to cheat and look at what the existing modules do. Who knows, you might write something better than what's already there.
Learning the principals, just like we did at school for long multiplication and division, means we know how things work when we use a short hand.
However, when new to the world,just like when you learn to speak, you did "A is for Apple" etc, you didnt get explained about the finesse of grammar and all that, you learnt to express yourself enough to be understood.
Programming is a little like the same. While in an ideal world you can easily argue a prewritten generic library is often way less efficient than a specifically targeted set of routines. If the wheel you are using was already invented, it seems a lot of work to make a new one.
So, use the wheels and cogs afailable, once you really have the hang of it, NOW look at inventing your own more efficient ones.
Ad cpan modules:
Modules are an great learning source. Here is zilion modules and you can really learn much studying some of them.
And when/while you mastering your perl, you will start writing you own modules. When your program will use modules anyway (yours one), you can ask - why don't use modules already developed and debugged?
So, learn perl basics, study some modules (for example Net::SFTP) and if you still want write your own solution - it is up to you. :)
'
I am looking at linking a few applications together (all written in different languages like C#, C++, Python) and I am not sure how to go about it.
What I mean by linking? The system I am working on consists of small programs each responsible for a particular processing task. I need to be able to transfer a data set from one application to another easily (the data set in question is not huge, probably a few megabytes) and I also need some form of way to control the current state of the operation (This is where a client-server model rings a bell)
It seems like sockets or maybe SOAP would be a universal solution but just wanted to get some opinions as to what people think about this subject.
Comments/suggestions will be appreciated, thanks!
I personally take a liking towards ØMQ. It's a library that has a familiar BSD-sockets-like interface for passing messages, but you'll find it implements interesting patterns for distributing tasks.
It sounds like you want to arrange several processes in a pipeline. ØMQ allows you to do that using push and poll sockets. (And afterwards, you'll find it's even possible to scale up across multiple processes and machines with little effort.) Take a look at the guide to get started, and the zmq_socket(3) manpage specifically for how push and pull works.
Bindings are available for all the languages you mention.
As for the contents of the message, ØMQ doesn't concern itself with that, they are just blocks of raw data. You can use any format that suits you, such as JSON, or perhaps Protocol Buffers.
What I'm not sure about is the ‘controlling state’ you mention. Are you interested in, for example, cancelling a job halfway through?
For C# to C# you can use Windows Communication Foundation. You may be able to use it with Python and C++ as well.
You may also want to checkout named pipes.
I would think about moving to a model where you eliminate the issue by having centralized data that all of the applications look at. Keep "one source of the truth" so to speak.
Most outside software has trouble linking against C++ code, due to the name-mangling algorithm it uses for its symbols. For that reason, when interfacing with programs written in other languages, it is often best to declare wrappers to things as extern "C" or inside an extern "C" { block.
I need to be able to transfer a data set from one application to another easily (the data set in question is not huge, probably a few megabytes)
Use the file system.
and I also need some form of way to control the current state of the operation
Again, use the file system. A "current_state.json" file with a JSON serialized object is perfect for multiple languages to work with.
It seems like sockets or maybe SOAP would be a universal solution.
Perhaps. But it's overkill for this kind of thing. Your OS already has all the facilities you need. Just use the file system. It's very simple and very reliable.
There are many ways to do interprocess communication. As you said, sockets may be a universal solution. SOAP, i think, is somewhat an overkill. You may also use mailslots. I wrote C++ application using it a couple of years ago. Named pipes could be also a solution, but if you are coding on Windows, it may be difficult.
In my opinion:
Sockets
Mailslots
Are the best candidates.
I'm designing a game where players are programmed bots competing in a programming contest. The bots can be programmed in any language - Java, Ruby, Python, C#. I'm looking for some way to transmit game data across the network or some way by which the game server can talk to the bots. What would be a better choice for this? Should i use XMPP or some other form of remote method invocation?
What you are descibing is not an RMI problem but a messaging one. I am sure there are several solutions you could use, and based on the limited knowledge of your application, I would say that XMPP is one of them. It is language agnostic and has libraries (and servers) available in most well supported languages.
Whether it is the best solution, I couldn't say, but I would think it is a viable one. It gives you the option for transmitting from point to point, point to many points, and a means for your game server to broadcast to all clients.
A REST based webservice might be easier to use if you need lots of languages to be able to call it.
I always find reinventing the wheel to be tedious. Try and see if you can use OpenTNL.
The issue with many Remoting infrastructures are that they are normally not portable between frameworks.
While XMPP might work for you - the main issue you might find is excessive data crossing the network due to all the header/presence stuff in the data being sent around. Also as XMPP is XML based any binary data would have to be sent around as a Base64 string.
A better bet might be a more low level socket interface - either way having the freedom to do bit-packing to reduce the size of the data will possibly be beneficial.