I read How to create a binary stream (not a file) in Common Lisp?, and it describes how to create a binary stream but not a bidirectional one. I've attempted to do it myself using the library referenced, but failed. My attempt looks like:
(defun make-test-binary-stream ()
(make-two-way-stream (flexi-streams:make-in-memory-input-stream
(vector))
(flexi-streams:make-in-memory-output-stream)))
I use it like:
(let ((str (make-test-binary-stream)))
(lisp-binary:write-float :double 123.45d0 :stream str)
(lisp-binary:read-binary-type 'double-float str))
The result I'd expect there is 123.45d0, but instead it returns 0.0d0.
How could I create a binary stream that behaves as I expect, allowing me to write a value in then read the same value back? I want such a stream for testing the correctness of encoding and decoding functions that take streams as input and output to streams.
A two-way stream S is a couple (I,O) where I is an input stream and O an output stream. Both streams are not necessarily related, it just means that when you read from S, you read from I, and when you write to S, you write to O.
Here, you try to read from an in-memory stream that is backed by an empty sequence. The stream just gives the items in the sequence, but as a stream; here, the stream reaches end-of-file immediately.
Anonymous buffer
Not directly answering the question, but I sometimes use lisp-binary and the way I test is as follows:
(with-input-from-sequence
(in (with-output-to-sequence (out)
(write-float :double 123.45d0 :stream out)))
(read-binary-type 'double-float in))
Let's decompose:
(flex:with-output-to-sequence (out)
(write-float :double 123.45d0 :stream out))
The above locally bind out to a stream that writes into a hidden, in-memory sequence, and eventually return that sequence. The whole expression returns a buffer of bytes:
#(205 204 204 204 204 220 94 64)
This buffer is given to with-input-from-sequence to bind in to a local stream that reads data from that sequence. read-binary-type uses that input stream to decode the value.
Temporary file
(defpackage :so (:use :cl :lisp-binary :flexi-streams :osicat))
(in-package :so)
The osicat system has a macro to open a temporary file in both input and output mode:
(with-temporary-file (io :element-type '(unsigned-byte 8))
(write-float :double pi :stream io)
(file-position io 0)
(read-binary-type 'double-float io))
In memory circular buffer
I couldn't find an existing implementation of an in-memory i/o stream that reads and writes from/to the same vector; you can hack something with two flexi-streams that share an internal vector (but this is dangerous, there are cases where data consistency breaks), or built one from gray streams; see also cl-stream.
Related
I'm trying to communicate from a Lisp script to another program by using TCP/IP sockets (with sbcl and the usocket library in a Linux system). Through some online sources I have managed to put together the following simple code:
(require 'asdf)
(require 'usocket)
(defun start-client (message)
"Connects to server."
(usocket:with-client-socket (socket stream "0.0.0.0" 30000)
(format stream message)
(force-output stream)))
(start-client "Hello!~%")
This code lets me send a message, (I have tested it and it works). My problem is that I need to split this code in two different functions, one for opening the socket connection and another to send different messages at different times. Also I need to add an additional function to receive messages from the other program. However, as I'm quite new with Lisp I have failed to do so.
The best way (I think) would be to have your entire script in the scope of with-client-socket. You might have something like a main function where this would fit. This avoids resource leaks. You might want to use a dynamic variable to avoid passing the socket stream manually through function arguments to wherever it is needed.
Otherwise, you have to manage the closing of the socket yourself. Any call path that might lead to program termination needs to be protected by some unwind-protect that closes the socket using usocket:socket-close. For that, you open the socket using usocket:socket-connect with the same arguments as you used for usocket:with-client-socket. (You can take a look at the source for usocket:with-client-socket and usocket:with-connected-socket to see the interactions taking place.)
In order to be able to write to the socket stream (obtainable through (usocket:socket-stream socket)) and close the socket, you need to remember it somewhere, e. g. by binding a dynamic variable.
Trying to learn lisp (and I guess emacs along with it).
I was wondering how you would go about clearing the output and replacing it.
Could be in a LISP repl, or an emacs buffer.
Something akin to the following in python.
def go(r):
for i in range(r):
sys.stdout.write("\rDoing %i" % i)
sys.stdout.flush()
For common lisp, you are looking for
Functions FINISH-OUTPUT, FORCE-OUTPUT, CLEAR-OUTPUT:
finish-output, force-output, and clear-output exercise control over the internal handling of buffered stream output.
finish-output attempts to ensure that any buffered output sent to output-stream has reached its destination, and then returns.
force-output initiates the emptying of any internal buffers but does not wait for completion or acknowledgment to return.
clear-output attempts to abort any outstanding output operation in progress in order to allow as little output as possible to continue to the destination.
and
Variables *DEBUG-IO*, *ERROR-OUTPUT*, *QUERY-IO*, *STANDARD-INPUT*, *STANDARD-OUTPUT*, *TRACE-OUTPUT*
The value of *debug-io*, called debug I/O, is a stream to be used for interactive debugging purposes.
The value of *error-output*, called error output, is a stream to which warnings and non-interactive error messages should be sent.
The value of *query-io*, called query I/O, is a bidirectional stream to be used when asking questions of the user. The question should be output to this stream, and the answer read from it.
The value of *standard-input*, called standard input, is a stream that is used by many operators as a default source of input when no specific input stream is explicitly supplied.
The value of *standard-output*, called standard output, is a stream that is used by many operators as a default destination for output when no specific output stream is explicitly supplied.
The value of *trace-output*, called trace output, is the stream on which traced functions (see trace) and the time macro print their output.
Emacs Lisp is quite different, you might want to start here: https://www.gnu.org/software/emacs/manual/html_node/elisp/Output-Functions.html
The usocket FAQ suggests that the way I should do this is by reading from a socket-stream and checking for an end-of-file result. That works in the case where I've got one thread active per socket, but it doesn't seem to satisfy for the case where I'm trying to service multiple sockets in the same thread.
Consider something like
(defparameter *socket* (socket-listen "127.0.0.1" 123456))
(defparameter *client-connections*
(list (socket-accept *socket*)
(socket-accept *socket*)
(socket-accept *socket*)
(socket-accept *socket*)))
For this exercise, assume that I've actually got four clients connecting there. It seems like the way to go about serving them from one thread is something like
(wait-for-input *client-connections*)
(loop for sock in *client-connections*
for stream = (socket-stream sock)
when (listen stream)
do (let ((line (read-line stream nil :eof)))
(if (eq line :eof)
(progn (delete sock *client-connections*)
(socket-close sock))
(handle sock line))))
Except that this won't work, because a disconnected socket still returns nil to listen, and an attempt to read from an active socket with no messages will block but wait-for-intput returns immediately when there's a closed socket in the mix, even when no other socket has a message ready (though it seems to fail to specify which sockets caused it to return).
In the situation where no client has spoken in a little while, and third client disconnects, there doesn't seem to be a good way of finding that out and closing that specific socket connection. I'd have to read them in sequence, except that since read blocks on no input, that would cause the thread to wait until the first two clients both sent a message.
The solutions I've got in mind, but haven't found after some determined googling, are (in descending order of preference):
A function otherwise equivalent to listen that returns t if a read on the targets' stream would return an end-of-file marker. (Replacing listen above with this notional function would let the rest of it work as written)
A function otherwise equivalent to wait-for-input that returns a list of closed sockets that cause it to trip. (In this case, I could iterate through the list of closed sockets, check that they're actually closed with the suggested read technique, and close/pop them as needed)
A function otherwise equivalent to wait-for-input that returns the first closed socket that caused it to trip. (As #2, but slower, because it prunes at most one inactive connection per iteration)
Keeping track of how long its been since I've received input from each socket connection, and closing them out regardless after a certain period of inactivity. (Which I'd probably want to do anyway, but doing just this would potentially keep a bunch of dead connections around much longer than necessary)
A function that attempts to read-char from a stream with an instant timeout, returns t if it encounters an :eof, and unread-chars anything else (returning nil after either timing out or unreading). (Which is a last resort since it seems like it would be trivially easy to break in a non-obvious-but-lethal way).
Also, if I'm thinking about this in precisely the wrong way, point that out too.
It turns out that the thing I mention as Option 2 above exists.
wait-for-input defaults to returning the full list of tracked connections for memory management purposes (someone was reportedly very concerned about consing new lists for the result), but it has a &key parameter that tells it to just return the connections that have something to say.
(wait-for-input (list conn1 conn2 conn3 conn4) :ready-only t)
is what I was looking for there. This returns all the ready connections, not just ones that are going to signal end-of-file, so the loop still needs to handle both cases. Something like
(loop for sock in (wait-for-input *client-connections* :ready-only t)
for stream = (socket-stream sock)
do (let ((line (read-line stream nil :eof)))
(if (eq line :eof)
(progn (delete sock *client-connections*)
(socket-close sock))
(handle sock line))))
should do nicely.
There are languages other than Lisp (ruby, scala) that say they use REPL (Read, Eval, Print, Loop), but it is unclear whether what is meant by REPL is the same as in Lisp. How is Lisp REPL different from non-Lisp REPL?
The idea of a REPL comes from the Lisp community. There are other forms of textual interactive interfaces, for example the command line interface. Some textual interfaces also allow a subset of some kind of programming language to be executed.
REPL stands for READ EVAL PRINT LOOP: (loop (print (eval (read)))).
Each of the four above functions are primitive Lisp functions.
In Lisp the REPL is not a command line interpreter (CLI). READ does not read commands and the REPL does not execute commands. READ reads input data in s-expression format and converts it to internal data. Thus the READ function can read all kinds of s-expressions - not just Lisp code.
READ reads a s-expression. This is a data-format that also supports encoding source code. READ returns Lisp data.
EVAL takes Lisp source code in the form of Lisp data and evaluates it. Side effects can happen and EVAL returns one or more values. How EVAL is implemented, with an interpreter or a compiler, is not defined. Implementations use different strategies.
PRINT takes Lisp data and prints it to the output stream as s-expressions.
LOOP just loops around this. In real-life a REPL is more complicated and includes error handling and sub-loops, so-called break loops. In case of an error one gets just another REPL, with added debug commands, in the context of the error. The value produced in one iteration also can be reused as input for the next evaluation.
Since Lisp is both using code-as-data and functional elements, there are slight differences to other programming languages.
Languages that are similar, those will provide also similar interactive interfaces. Smalltalk for example also allows interactive execution, but it does not use a data-format for I/O like Lisp does. Same for any Ruby/Python/... interactive interface.
Question:
So how significant is the original idea of READing EXPRESSIONS, EVALuating them and PRINTing their values? Is that important in relation to what other languages do: reading text, parsing it, executing it, optionally print something and optionally printing a return value? Often the return value is not really used.
So there are two possible answers:
a Lisp REPL is different to most other textual interactive interfaces, because it is based on the idea of data I/O of s-expressions and evaluating these.
a REPL is a general term describing textual interactive interfaces to programming language implementations or subsets of those.
REPLs in Lisp
In real implementations Lisp REPLs have a complex implementation and provide a lot of services, up to clickable presentations (Symbolics, CLIM, SLIME) of input and output objects. Advanced REPL implementations are for example available in SLIME (a popular Emacs-based IDE for Common Lisp), McCLIM, LispWorks and Allegro CL.
Example for a Lisp REPL interaction:
a list of products and prices:
CL-USER 1 > (setf *products* '((shoe (100 euro))
(shirt (20 euro))
(cap (10 euro))))
((SHOE (100 EURO)) (SHIRT (20 EURO)) (CAP (10 EURO)))
an order, a list of product and amount:
CL-USER 2 > '((3 shoe) (4 cap))
((3 SHOE) (4 CAP))
The price for the order, * is a variable containing the last REPL value. It does not contain this value as a string, but the real actual data.
CL-USER 3 > (loop for (n product) in *
sum (* n (first (second (find product *products*
:key 'first)))))
340
But you can also compute Lisp code:
Let's take a function which adds the squares of its two args:
CL-USER 4 > '(defun foo (a b) (+ (* a a) (* b b)))
(DEFUN FOO (A B) (+ (* A A) (* B B)))
The fourth element is just the arithmetic expression. * refers to the last value:
CL-USER 5 > (fourth *)
(+ (* A A) (* B B))
Now we add some code around it to bind the variables a and b to some numbers. We are using the Lisp function LIST to create a new list.
CL-USER 6 > (list 'let '((a 12) (b 10)) *)
(LET ((A 12) (B 10)) (+ (* A A) (* B B)))
Then we evaluate the above expression. Again, * refers to the last value.
CL-USER 7 > (eval *)
244
There are several variables which are updated with each REPL interaction. Examples are *, ** and *** for the previous values. There is also + for the previous input. These variables have as values not strings, but data objects. + will contain the last result of the read operation of the REPL. Example:
What is the value of the variable *print-length*?
CL-USER 8 > *print-length*
NIL
Let's see how a list gets read and printed:
CL-USER 9 > '(1 2 3 4 5)
(1 2 3 4 5)
Now let's set the above symbol *print-length* to 3. ++ refers to the second previous input read, as data. SET sets a symbols value.
CL-USER 10 > (set ++ 3)
3
Then above list prints differently. ** refers to the second previous result - data, not text.
CL-USER 11 > **
(1 2 3 ...)
Seeing as the concept of a REPL is to just Read, Eval, Print & Loop it's not too suprising that there are REPLs for many languages:
C/C++
C#/LINQ
Erlang
Haskell (on windows)
Java
Javascript
Julia
Perl
Python
Ruby
Scala
Smalltalk -- I learned it on a REPL!
I think it is interesting to compare two approaches. A bare bones REPL loop in a Lisp system would look like this:
(loop (print (eval (read))))
Here are two actual Forth implementations of a REPL loop. I'm leaving nothing out here -- this is the full code to these loops.
: DO-QUIT ( -- ) ( R: i*x -- )
EMPTYR
0 >IN CELL+ ! \ set SOURCE-ID to 0
POSTPONE [
BEGIN \ The loop starts here
REFILL \ READ from standard input
WHILE
INTERPRET \ EVALUATE what was read
STATE # 0= IF ." OK" THEN \ PRINT
CR
REPEAT
;
: quit
sp0 # 'tib !
blk off
[compile] [
begin
rp0 # rp!
status
query \ READ
run \ EVALUATE
state # not
if ." ok" then \ PRINT
again \ LOOP
;
Lisp and Forth do completely different things, particularly in the EVAL part, but also in the PRINT part. Yet, they share the fact that a program in both languages is run by feeding its source code to their respective loops, and in both cases code is just data (though in Forth case it is more like data is also code).
I suspect what anyone saying only LISP has a REPL is that the READ loop reads DATA, which is parsed by EVAL, and a program is created because CODE is also DATA. This distinction is interesting in many respects about the difference between Lisp and other languages, but as far as REPL goes, it doesn't matter at all.
Let's consider this from the outside:
READ -- returns input from stdin
EVAL -- process said input as an expression in the language
PRINT -- print EVAL's result
LOOP -- go back to READ
Without going into implementation details, one can't distinguish a Lisp REPL from, for example, a Ruby REPL. As functions, they are the same.
I guess you could say that Scala's "REPL" is an "RCRPL": Read, Compile, Run, Print. But since the compiler is kept "hot" in memory, it's pretty fast for ongoing interactions--it just takes a few seconds to start up.
There are a number of people that consider a REPL to needs to behave exactly like it does in LISP, or it's not a true REPL. Rather, they consider it something different, like a CLI (command line interpreter). Honestly, I tend to think that if it follows the basic flow of:
read input from the user
evaluate that input
print the output
loop back to the read
then it's a REPL. As noted, there are a lot of languages that have the above capability.
See this reddit thread for an example of such a discussion.
How is Lisp REPL different from non-Lisp REPL?
Let's compare Common Lisp's REPL with Python's IPython.
The main two points are:
Lisp is an image-based language. There is no need to restart the process/the REPL/the whole app after a change. We compile our code function by function (with compiler warnings etc).
we don't loose state. Even more, when we update class definitions, our objects in the REPL are also updated, following rules we have control upon. That way we can hot-reload code in a running system.
In Python, typically, you start IPython or you are dropped into ipdb. You define some data until you try out your new function. You edit your source, and you want to try again, so you quit IPython and you start the whole process again. In Lisp (Common Lisp mainly), not at all, it's all more interactive.
There's a nice project called multi-repl which exposes various REPLs via Node.JS:
https://github.com/evilhackerdude/multi-repl
If you look at the list of supported languages, it's quite clear that not only Lisp has the concept of a REPL.
clj (clojure)
ghci (ghc)
ipython
irb (ruby)
js (spidermonkey)
node
python
sbcl
v8
In fact implementing a trivial one in Ruby is fairly easy:
repl = -> prompt { print prompt; puts(" => %s" % eval(gets.chomp!)) }
loop { repl[">> "] }
New version of SBCL 1.0.28 running on debian breaks AllegroServe 1.2.47 on incoming connection with following error:
aserve-accept-6: 05/26/09 - 21:11:01 - accept: error 0 on accept invalid
keyword argument: :AUTO-CLOSE (valid keys are
:INPUT, :OUTPUT, :ELEMENT-TYPE, :EXTERNAL-FORMAT,
:BUFFERING, :TIMEOUT).
Portable AllegroServe page does make a mention of this problem. However, no google searches turn up anything of use for this problem.
Any ideas as to how to move forward with this problem, or alternatively, links pointing to places where this has been dealt with?
After some mucking around, I've come up with the following solution:
In my source files, after I declare my package, compile/load the appropriate modules but before I declare anything in my package, I added the following code:
(defmethod sb-bsd-sockets:socket-make-stream ((socket sb-bsd-sockets:socket)
&key input output
(element-type 'character)
(buffering :full)
(external-format :default)
timeout
(auto-close t))
"Default method for SOCKET objects. An ELEMENT-TYPE of :DEFAULT
will construct a bivalent stream. Acceptable values for BUFFERING
are :FULL, :LINE and :NONE. Streams will have no TIMEOUT
by default.
The stream for SOCKET will be cached, and a second invocation of this
method will return the same stream. This may lead to oddities if this
function is invoked with inconsistent arguments \(e.g., one might request
an input stream and get an output stream in response\)."
(let ((stream
(and (slot-boundp socket 'stream) (slot-value socket 'stream))))
(unless stream
(setf stream (sb-sys:make-fd-stream
(sb-bsd-sockets:socket-file-descriptor socket)
:name "a socket"
:dual-channel-p t
:input input
:output output
:element-type element-type
:buffering buffering
:external-format external-format
:timeout timeout
:auto-close auto-close)))
(setf (slot-value socket 'stream) stream)
(sb-ext:cancel-finalization socket)
stream))
(It's basically a lift from what is in the sb-bsd-sockets/socket.lisp with the auto-close key added to the argument list)
This way I avoid modifying or patching system files, and basically hook into the sb-bsd-sockets package directly.
So far, it seems to be working as it should. Basic testing via successive calls to (room) shows me that there's no obvious memory leaks, and the performance is as expected.
Please feel free to comment on this kludge, and if you think it might affect the stability of my system in unexpected ways.