Interact with a locally long-running Common Lisp image, possibly daemonized, from the command line - lisp

How could one interact with a locally long-running Common Lisp image, possibly daemonized, from the command line?
I know it is possible to run a Common Lisp function from a Terminal command prompt, I am also aware of this.
I would need to do a similar thing, but with a local, already long-running Common Lisp image, being able to poll available functions from the CLI or shell scripts.
Is there a way to do that from a CLI, for example calling a function from a bash script, and receiving back whatever the function returns?
I though one could, for example, create a primitive web service, perhaps using woo or Hunchentoot, calling functions and fetching returned values via curl or wget, but it feels a little convoluted.
Of course, that is one of the many features of Emacs' SLIME, but I would need to call functions just from the CLI, without invoking Emacs.
Is there perhaps a way to reach a swank backend, outside of SLIME?
If possible at all, what would be the lisp idiomatic way of doing that?
I would be grateful for any pointers.
Update
Additional note
Many years ago, I was intrigued by being able to telnet into a long-running LISP image (I believe in this case uppercasing the name should be fine). If I remember correctly, it was available at prompt.franz.com. An article, somehow connected: telnet for remote access to a running application
Telnet is of course quite unsafe, but the usefulness of being able to access the Lisp application(s) in that way, for whatever reason, cannot be overstated, at least to some people.
Some additional pointers, and thanks
I would like to thank Basile Starynkevitch for his elaborate and thorough answer, especially on the theoretical aspect. I was looking for a more practical direction, specially connected to Common Lisp. Still, his answer is very instructive.
I was all ready to start writing a local server, perhaps using one of the fine Common Lisp libraries, like:
usocket: Universal socket library for Common Lisp
iolib: Common Lisp I/O library
cl-aync: Asynchronous IO library for Common Lisp
But, thanks to Stanislav Kondratyev, I didn't have to. He pointed out an already existing solution that nicely answer my question, ScriptL: Shell scripting made Lisp-like
I tested it with success on Linux, FreeBSD and OS X, just make sure to install the thin wrapper over POSIX syscalls first. Among many features (see README), it allows exposition of just selected functions, security is properly handled, and it even supply a custom C client, which builds as part of the ASDF load operation, and supports a number of new features, such as I/O, in place of netcat.

You may find scriptl useful: http://quickdocs.org/scriptl/. However, it depends on iolib, which depends on some nonstandard C or C++ library, so building it is not quite straighforward.
It is indeed possible to communicate with a swank server if you familiarize yourself with the swank protocol, which seems to be underdocumented (see e. g. here: https://github.com/astine/swank-client/blob/master/swank-description.markdown). However, this exposes a TCP socket over a network, which could be unsafe. Once I tried that, too, but I was not satisfied with the IPC speed.
A while ago I wrote a rather naive SBCL-specific server that uses a local domain socket for communication, and a client in C. It's very raw, but you could take a look: https://github.com/quasus/lispserver. In particular, it supports interactive IO and exit codes. The server and the client form the core of a simple framework for deploying Unix style software. Feel free to borrow code and/or contact me for explanations, suggestions, etc.

It certainly is operating system specific, because you want some inter-process communication, and they are provided by the OS.
Let's assume you have a POSIX like OS, e.g. Linux.
Then you could set up a socket(7) or fifo(7) to which you send s-exprs to be evaluated. Of course you need to adapt the Common Lisp program to add such a REPL.
SBCL has some networking facilities, you could build on them.
Of course, you should understand first how IPC work on your OS. If it is Linux, you could read Advanced Linux Programming (it is centered on C programming, which is the low-level way of using OS services on POSIX, but you can adapt what you have learned to SBCL). And indeed, the Common Lisp standard might not have an official POSIX interface, so you need to dive into implementation specific details.
Perhaps you should learn more about BSD sockets. There are tons of tutorials on them. Then you could use TCP sockets (see tcp(7)) or Unix ones (see unix(7)). Advanced users could use the unsafe telnet command. Or you might make your software use SSL, or perhaps some libssh e.g. use ssh as their client.
You could decide and document that the protocol between user apps and your program is : send some-sexpr (on a documented socket) from user-app code to your server which is terminated by a double newline or by a form feed, and have your server eval it and send back the result or some error message. I did similar things in MELT and it is not a big deal. Be careful about buffering.
I guess that you assume that you have responsible and competent users (so don't open such a remote REPL to the wild Internet!). If you care about malicious or stupid use of a remote REPL, it is becoming complex.
Alternatively, make your server a web application (by using some HTTP server library inside it), and ask users to use their browser or some HTTP client program (curl) or library (libcurl) to interact with it.

Related

Webdevelopment in Common Lisp

I am somewhat familiar with Scheme. I am reading OnLisp, and would love to do a real world project in it. Most likely web development. I would love to know what kind of setup people use to develop websites in lisp. I have a couple of macs at home and I have figured I would use clozure cl. I am wary of a few things though. I have read on the internets that cl doesn't have reliable threading facility nor good networking. Is this true? What was your experience?
PS:Don't tell me about clojure :). Lisp is a itch I need to scratch.
Currently I'm using Restas a framework based on Hunchentoot and inspired by the route system of Rails. I also use Postmodern to interact with a PostgreSQL database and I generate HTML with cl-markup though I'm thinking about switching to cl-who which looks more customizable.
When I've started I've also considered using Parenscript to generate the JavaScript but now I'm just happy with Mootools and plain JavaScript.
Everything runs on SBCL and is available with Quicklisp.
Not sure why it wouldn't have "good networking"; you can't rely on threads if you want to write a portable Common Lisp application since certain implementations don't support them (AFAIK, just clisp, though you can compile it with experimental thread support). If you plan on picking an implementation and sticking with it, you won't have that problem.
The "standard" CL web stack is Hunchentoot/cl-who/clsql, though you can find tutorials that run on Araneida or Portable AllegroServe.
The setup I use is Debian/SBCL running quicklisp and the above systems along with MySQL (though I'm seriously considering jumping over to Postgres) for the database and nginx to run in front of Hunchentoot serving any static content.
mck- has been maintaining the heroku common lisp webapp.
https://github.com/mck-/heroku-cl-example/tree/
The problem with Common Lisp's "networking" is, we don't have sockets in CL standard, so each implementation has it's own socket API. We have some attempts to give a common interface though, like usocket. You can find a list of networking related packages here.
If you need a web framework, look into Caveman. I haven't used it yet but it looks like the most complete CL web framework I've ever seen.

Daemon library for FreeBSD

I am contemplating writing a daemon programme for FreeBSD, however, I've never done it, and by the looks of it, it's quite tricky.
Are there any libraries that I could include and then just write the things specific for my daemon?
Sorry for posting this an answer rather than a comment - I don't have enough reputation to do the latter.
You might want to look at daemon(3), a function provided in FreeBSD's libc. It does things like close the standard file descriptors, set things up so SIGHUP will have no effect, and so on. The source code is here:
http://svnweb.freebsd.org/base/head/lib/libc/gen/daemon.c?view=markup
There's also a man page. (Be careful, there's also a wrapper program, daemon(1)). Also note that it's not portable (though it may be available in libbsd, I'm not sure).
FreeBSD also provides libutil (again, not portable I think), which among other things provides pidfile_*() functions for handling PID files, which I suppose is useful for writing a daemon. There are other considerations, but this might help you get started. Can't say much more unless you have a specific question.

Interprocess communication in windows xp

I have least knowledge on network domain.Please bare with my question if it is basic
I would like to know how to communicate between two applications using TCP/ip protocol in
windows XP say for example
Using Perl and TK I am building a GUI with entry widget and i have an other external application which is installed in the system say APPLICATION.exe
Now i need to establish the connection using TCP between my perl program and external application APPLICATION.exe
for testing purpose what ever I type in the GUI Entry this external application has to receive (hope it receives as bytes)and external application will send some data and i need to process that data in my perl program..
I am totally unaware whether this is simple task or complicated.
If there are any modules existing in perl for similar programming please suggest me.
I highly appreciate if anyone can provide me the answer with an example.
Thanks to all
It is simple if you have done network programming before. If not, it is intermediately difficult because you need to get a grip on the underlying concepts and terminology first - read a book about the topic if necessary.
I recommend to use IO::Socket::IP. If you have Perl 5.14 or later, you can also use IO::Socket::INET that is distributed with Perl. See https://stackoverflow.com/a/8582550/46395 for a code example and links to further documentation.

Is a client-server setup a good way to move data between machines?

I need to move some data from one machine to another. Is it a good idea to write a client server app using sockets in Perl to do the transfer? Will I have problems if one side is written in Java?
I mean, should I be aware of any issues I might face when I try to attempt the above?
Short answer: Using a Perl program as the client or server is just fine. Your only problem might be your personal skill and experience level, but after you do it you know how to do it. :) Most of the problem is choosing how you need to do it, not the technology involved. Perl isn't going to be the problem, but it doesn't have an advantage over other languages either.
As some have already noted, the socket portion of the problem is going to be the same in most languages since almost everything uses the BSD stuff. Perl doesn't have any roadblocks or special gotchas for that. To move data around you create one side to listen on a socket and the other to open a connection and send the data. Easy peasy. You might want to check out Lincoln Stein's Network Programming with Perl for that bit. That can get you the low-level bits.
For higher-level networking, POE is very useful and easy to work with once you get started. It's a framework for dealing with event-driven programming and has many plugins to easily communicate between processes. You might spend a little time learning it, but it gives a lot back too.
If you aren't inventing your own protocol, there's most likely already a Perl module that can format and parse the messages.
If you just want to transfer data, there are several things you can do. The easiest in concept might be just to write lines to the socket and read them as lines from the other end. A bit more complicated than that is using something like Data::Dumper, YAML, or JSON to serialize data to text and send that. For more complex things, such as sharing Perl objects, you might want to use Storable. You freeze your objects, send them as data over the network, then thaw them on the other side.
If you want to implement your client and server in different languages you have a bit more work to figure out how they'll talk to each other. The socket stuff is mostly the same, but a Java server won't understand the output of Perl's Storable (it's possible, but you'll have to parse it yourself and that's no good :). If you do everything right, neither side should care what you used on the other side.
I can only think of one gotcha off the top of my head: most text based network protocols use CRLF for line endings, but Perl on UNIX type machines assumes LF endings by default, this means you will need to change the input and output record separators if you want to use readline (aka <>) and print (also beware of printf, since it doesn't use the output record separator). Of course, if you are going to use a pre-existing protocol, there is probably already a Net::<PROTOCOL NAME> module on CPAN, so you won't need to worry about that. If you are designing your own protocol, I would keep the CRLF convention because it makes it easy to debug the server with telnet (which is really the last valid use for that program).
You don't say whether you need to implement your application to support any particular protocol or whether you need to implement a home grown protocol. The networking support in Perl is flexible enough to support either (or most places in between).
At the low level socket end, your code is going to be fairly similar whatever language your are using - BSD socket APIs are pretty well the same everywhere they are supported. The support you need for this is built into Perl but low level socket programming can be frustrating - it's very low level.
However, Perl's standard library contains the Socket module which is rather easier to use (and well documented).
If you need to implement an existing protocol you may well find that it has already been implemented. For example Net::Telnet implements command/response protocols (like Telnet) making a client app trivial.
Searching CPAN may save you a lot of pain. Look at modules in the Net::* hierarchy
I don't think you're gonna have any major issues that you won't have by not using Perl. Even performance will be comparable to other solutions due to network latencies.
You might want to look at POE framework. It makes writing such components a breeze.
It probably depend on a few factors. Does speed or responsiveness matter? Are you moving data between they same type of machines (Unix to Unix, Windows to Windows)? What type of data are you trying to move (Text or Binary)? What is knowledge about sockets and what languages do you have experience?
I have sent and received binary data over PERL sockets from differing applications, but I don't have much experience with the text processing over sockets from differing machines. If you are moving data between machine you need to keep in mind the way the data is marshalled and if it is packed or aligned on some byte boundry. I have not exchanged data with a Java programs, but is should be similiar.
It probably would help to have some experience with PERL, and I would recommend looking at the examples in the "camel" book. I have used the ones in the book as a starting point and made modification for what I needed to achieve. You may have to consult some other areas of the book if you are dealing with binary data, or to help in doing translations for sending data.
Write socket communication in Perl is relatively easy. Do it right and reliable is big pain even CPAN modules are examples of error prone code. It depends of your expectations.
You are basically asking two questions:
Is Perl a proper language for socket communication?
Is Perl a proper language for UI?
Referring to e5's answer, Perl is indeed a string-centric language with a focus on readable strings, less well equipped to handle binary data. Thus the answer probably lies in the questions: Is your communication string based? Is your UI string based?
If doing binary interaction through a socket, well, you probably could be doing better than Perl (not talking about C, but maybe C-ish languages). If you want to do graphical user-interaction you probably reach faster results by choosing one of the higher languages that focus more on gui interaction. (Java-ish might be the thing here.)

Lisp In A Box - Why is it starting a server?

I've decided to get back into LISP (haven't used it since my AI classes) to get more comfortable with functional programming in general, so I downloaded Lisp In A Box (which we actually used in a previous class) which comes with CLISP and Emacs.
When I run it, it says:
Connected on port 1617. Take this REPL, brother, and may it serve you well.
What the? So I looked on the Lisp In A Box webpage more closely and found this:
SLIME is an integrated development environment for Emacs which interfaces with a Common Lisp implementation over a network socket. Lots of information about SLIME can be found at the SLIME node on CLiki. The manual for SLIME is available in PDF format online.
I somewhat understand what SLIME is (some sort of extension to emacs, right?) But why in the world is a text editor starting its own server and connecting to it?
The purpose is so that Lisp will be running in parallel.
Slime connects to the session and then you can have the same environment, definitions, etc from many different windows (or machines even). This means that you can start up your application and debug it on the fly, for instance.
For further information, look at this blog.
Sockets are more flexible than pipes. For one, SLIME lets you connect to Swank servers on the network, which is very useful for doing live fixes on remote machines with long-running processes (such as web servers). Given this, why would you add another layer of complexity by abstracting communication in such a way as to support both pipes and sockets? It's not like pipes are any simpler to program than sockets, anyway.
Well, Slime starts Lisp process to give you integrated development environment. So that you can test and debug your code on the fly and also be able to inspect objects.
I think architecture with sockets was chosen for better portability between different lisps (Btw, Slime also supports Clojure and MIT Scheme ) and OS-es (Slime works on Windows too). Also it allows cross-platform development - you can test your software on target architecture from your Emacs.
So I think, that this decision is great, you just should not put swank (Slime back-end) on production servers.
You get a REPL (read-evaluate-print-loop) running in parallel, so that you can compile and test code snippets on the fly from your editor. "Practical Common Lisp" (freely available on the web) has a good explanation for this, and it is a very good book for learning Lisp.