Related
Prelude
In Raku there's a notion called infinite list AKA lazy list which is defined and used like:
my #inf = (1,2,3 ... Inf);
for #inf { say $_;
exit if $_ == 7 }
# => OUTPUT
1
2
3
4
5
6
7
I'd like to implement this sort of thing in Common Lisp, specifically an infinite list of consecutive integers like:
(defun inf (n)
("the implementation"))
such that
(inf 5)
=> (5 6 7 8 9 10 .... infinity)
;; hypothetical output just for the demo purposes. It won't be used in reality
Then I'll use it for lazy evaluation like this:
(defun try () ;; catch and dolist
(catch 'foo ;; are just for demo purposes
(dolist (n (inf 1) 'done)
(format t "~A~%" n)
(when (= n 7)
(throw 'foo x)))))
CL-USER> (try)
1
2
3
4
5
6
7
; Evaluation aborted.
How can I implement such an infinite list in CL in the most practical way?
A good pedagogical approach to this is to define things which are sometimes called 'streams'. The single best introduction to doing this that I know of is in Structure and Interpretation of Computer Programs. Streams are introduced in section 3.5, but don't just read that: read the book, seriously: it is a book everyone interested in programming should read.
SICP uses Scheme, and this sort of thing is more natural in Scheme. But it can be done in CL reasonably easily. What I've written below is rather 'Schemy' CL: in particular I just assume tail calls are optimised. That's not a safe assumption in CL, but it's good enough to see how you can build these concepts into a language which does not already have them, if your language is competent.
First of all we need a construct which supports lazy evaluation: we need to be able to 'delay' something to create a 'promise' which will be evaluated only when it needs to be. Well, what functions do is evaluate their body only when they are asked to, so we'll use them:
(defmacro delay (form)
(let ((stashn (make-symbol "STASH"))
(forcedn (make-symbol "FORCED")))
`(let ((,stashn nil)
(,forcedn nil))
(lambda ()
(if ,forcedn
,stashn
(setf ,forcedn t
,stashn ,form))))))
(defun force (thing)
(funcall thing))
delay is mildly fiddly, it wants to make sure that a promise is forced only once, and it also wants to make sure that the form being delayed doesn't get infected by the state it uses to do that. You can trace the expansion of delay to see what it makes:
(delay (print 1))
-> (let ((#:stash nil) (#:forced nil))
(lambda ()
(if #:forced #:stash (setf #:forced t #:stash (print 1)))))
This is fine.
So now, we'll invent streams: streams are like conses (they are conses!) but their cdrs are delayed:
(defmacro cons-stream (car cdr)
`(cons ,car (delay ,cdr)))
(defun stream-car (s)
(car s))
(defun stream-cdr (s)
(force (cdr s)))
OK, let's write a function to get the nth element of a stream:
(defun stream-nth (n s)
(cond ((null s)
nil)
((= n 0) (stream-car s))
(t
(stream-nth (1- n) (stream-cdr s)))))
And we can test this:
> (stream-nth 2
(cons-stream 0 (cons-stream 1 (cons-stream 2 nil))))
2
And now we can write a function to enumerate an interval in the naturals, which by default will be an half-infinite interval:
(defun stream-enumerate-interval (low &optional (high nil))
(if (and high (> low high))
nil
(cons-stream
low
(stream-enumerate-interval (1+ low) high))))
And now:
> (stream-nth 1000 (stream-enumerate-interval 0))
1000
And so on.
Well, we'd like some kind of macro which lets us traverse a stream: something like dolist, but for streams. Well we can do this by first writing a function which will call a function for each element in the stream (this is not the way I'd do this in production CL code, but it's fine here):
(defun call/stream-elements (f s)
;; Call f on the elements of s, returning NIL
(if (null s)
nil
(progn
(funcall f (stream-car s))
(call/stream-elements f (stream-cdr s)))))
And now
(defmacro do-stream ((e s &optional (r 'nil)) &body forms)
`(progn
(call/stream-elements (lambda (,e)
,#forms)
,s)
,r))
And now, for instance
(defun look-for (v s)
;; look for an element of S which is EQL to V
(do-stream (e s (values nil nil))
(when (eql e v)
(return-from look-for (values e t)))))
And we can then say
> (look-for 100 (stream-enumerate-interval 0))
100
t
Well, there is a lot more mechanism you need to make streams really useful: you need to be able to combine them, append them and so on. SICP has many of these functions, and they're generally easy to turn into CL, but too long here.
For practical purposes it would be wise to use existing libraries, but since the question is about how to implemented lazy lists, we will do it from scratch.
Closures
Lazy iteration is a matter of producing an object that can generate the new value of a lazy sequence each time it is asked to do so.
A simple approach for this is to return a closure, i.e. a function that closes over variables, which produces values while updating its state by side-effect.
If you evaluate:
(let ((a 0))
(lambda () (incf a)))
You obtain a function object that has a local state, namely here the variable named a.
This is a lexical binding to a location that is exclusive to this function, if you evaluate a second time the same expression, you'll obtain a different anonymous function that has its own local state.
When you call the closure, the value stored in a in incremented and its value is returned.
Let's bind this closure to a variable named counter, call it multiple times and store the successive results in a list:
(let ((counter (let ((a 0))
(lambda () (incf a)))))
(list (funcall counter)
(funcall counter)
(funcall counter)
(funcall counter)))
The resulting list is:
(1 2 3 4)
Simple iterator
In your case, you want to have an iterator that starts counting from 5 when writing:
(inf 5)
This can implemented as follows:
(defun inf (n)
(lambda ()
(shiftf n (1+ n))))
Here is there is no need to add a let, the lexical binding of an argument to n is done when calling the function.
We assign n to a different value within the body over time.
More precisely, SHIFTF assigns n to (1+ n), but returns the previous value of n.
For example:
(let ((it (inf 5)))
(list (funcall it)
(funcall it)
(funcall it)
(funcall it)))
Which gives:
(5 6 7 8)
Generic iterator
The standard dolist expects a proper list as an input, there is no way you can put another kind of data and expect it to work (or maybe in an implementation-specific way).
We need a similar macro to iterate over all the values in an arbitrary iterator.
We also need to specify when iteration stops.
There are multiple possibilities here, let's define a basic iteration protocol as follows:
we can call make-iterator on any object, along with arbitrary arguments, to obtain an iterator
we can call next on an iterator to obtain the next value.
More precisely, if there is a value, next returns the value and T as a secondary value; otherwise, next returns NIL.
Let's define two generic functions:
(defgeneric make-iterator (object &key)
(:documentation "create an iterator for OBJECT and arguments ARGS"))
(defgeneric next (iterator)
(:documentation "returns the next value and T as a secondary value, or NIL"))
Using generic functions allows the user to define custom iterators, as long as they respect the specified behaviour above.
Instead of using dolist, which only works with eager sequences, we define our own macro: for.
It hides calls to make-iterator and next from the user.
In other words, for takes an object and iterates over it.
We can skip iteration with (return v) since for is implemented with loop.
(defmacro for ((value object &rest args) &body body)
(let ((it (gensym)) (exists (gensym)))
`(let ((,it (make-iterator ,object ,#args)))
(loop
(multiple-value-bind (,value ,exists) (next ,it)
(unless ,exists
(return))
,#body)))))
We assume any function object can act as an iterator, so we specialize next for values f of class function, so that the function f gets called:
(defmethod next ((f function))
"A closure is an interator"
(funcall f))
Also, we can also specialize make-iterator to make closures their own iterators (I see no other good default behaviour to provide for closures):
(defmethod make-iterator ((function function) &key)
function)
Vector iterator
For example, we can built an iterator for vectors as follows. We specialize make-iterator for values (here named vec) of class vector.
The returned iterator is a closure, so we will be able to call next on it.
The method accepts a :start argument defaulting to zero:
(defmethod make-iterator ((vec vector) &key (start 0))
"Vector iterator"
(let ((index start))
(lambda ()
(when (array-in-bounds-p vec index)
(values (aref vec (shiftf index (1+ index))) t)))))
You can now write:
(for (v "abcdefg" :start 2)
(print v))
And this prints the following characters:
#\c
#\d
#\e
#\f
#\g
List iterator
Likewise, we can build a list iterator.
Here to demonstrate other kind of iterators, let's have a custom cursor type.
(defstruct list-cursor head)
The cursor is an object which keeps a reference to the current cons-cell in the list being visited, or NIL.
(defmethod make-iterator ((list list) &key)
"List iterator"
(make-list-cursor :head list))
And we define next as follows, specializeing on list-cursor:
(defmethod next ((cursor list-cursor))
(when (list-cursor-head cursor)
(values (pop (list-cursor-head cursor)) t)))
Ranges
Common Lisp also allows methods to be specialized with EQL specializers, which means the object we give to for might be a specific keyword, for example :range.
(defmethod make-iterator ((_ (eql :range)) &key (from 0) (to :infinity) (by 1))
(check-type from number)
(check-type to (or number (eql :infinity)))
(check-type by number)
(let ((counter from))
(case to
(:infinity
(lambda () (values (incf counter by) t)))
(t
(lambda ()
(when (< counter to)
(values (incf counter by) T)))))))
A possible call for make-iterator would be:
(make-iterator :range :from 0 :to 10 :by 2)
This also returns a closure.
Here, for example, you would iterate over a range as follows:
(for (v :range :from 0 :to 10 :by 2)
(print v))
The above expands as:
(let ((#:g1463 (make-iterator :range :from 0 :to 10 :by 2)))
(loop
(multiple-value-bind (v #:g1464)
(next #:g1463)
(unless #:g1464 (return))
(print v))))
Finally, if we add small modification to inf (adding secondary value):
(defun inf (n)
(lambda ()
(values (shiftf n (1+ n)) T)))
We can write:
(for (v (inf 5))
(print v)
(when (= v 7)
(return)))
Which prints:
5
6
7
I'll show it with a library:
How to create and consume an infinite list of integers with the GTWIWTG generators library
This library, called "Generators The Way I Want Them Generated", allows to do three things:
create generators (iterators)
combine them
consume them (once).
It is not unsimilar to the nearly-classic Series.
Install the lib with (ql:quickload "gtwiwtg"). I will work in its package: (in-package :gtwiwtg).
Create a generator for an infinite list of integers, start from 0:
GTWIWTG> (range)
#<RANGE-BACKED-GENERATOR! {10042B4D83}>
We can also specify its :from, :to, :by and :inclusive parameters.
Combine this generator with others: not needed here.
Iterate over it and stop:
GTWIWTG> (for x *
(print x)
(when (= x 7)
(return)))
0
1
2
3
4
5
6
7
T
This solution is very practical :)
Often when I try to write a macro, I run up against the following difficulty: I need one form that is passed to the macro to be evaluated before being processed by a helper function that is invoked while generating the macro's expansion. In the following example, we are only interested in how we could write a macro to emit the code we want, and not in the uselessness of the macro itself:
Imagine (bear with me) a version of Common Lisp's lambda macro, where only the number of arguments is important, and the names and order of the arguments are not. Let's call it jlambda. It would be used like so:
(jlambda 2
...body)
where 2 is the arity of the function returned. In other words, this produces a binary operator.
Now imagine that, given the arity, jlambda produces a dummy lambda-list which it passes to the actual lambda macro, something like this:
(defun build-lambda-list (arity)
(assert (alexandria:non-negative-integer-p arity))
(loop for x below arity collect (gensym)))
(build-lambda-list 2)
==> (#:G15 #:G16)
The expansion of the above call to jlambda will look like this:
(lambda (#:G15 #:16)
(declare (ignore #:G15 #:16))
…body))
Let's say we need the jlambda macro to be able to receive the arity value as a Lisp form that evaluates to a non-negative integer (as opposed to receiving a non-negative integer directly) eg:
(jlambda (+ 1 1)
...body)
The form (+ 1 1) needs to be evaluated, then the result needs to be passed to build-lambda-list and that needs to be evaluated, and the result of that is inserted into the macro expansion.
(+ 1 1)
=> 2
(build-lambda-list 2)
=> (#:G17 #:18)
(jlambda (+ 1 1) ...body)
=> (lambda (#:G19 #:20)
(declare (ignore #:G19 #:20))
…body))
So here's a version of jlambda that works when the arity is provided as a number directly, but not when it's passed as a form to be evaluated:
(defun jlambda-helper (arity)
(let ((dummy-args (build-lambda-list arity)))
`(lambda ,dummy-args
(declare (ignore ,#dummy-args))
body)))
(defmacro jlambda (arity &body body)
(subst (car body) 'body (jlambda-helper arity)))
(jlambda 2 (print “hello”)) ==> #<anonymous-function>
(funcall *
'ignored-but-required-argument-a
'ignored-but-required-argument-b)
==> “hello”
“hello”
(jlambda (+ 1 1) (print “hello”)) ==> failed assertion in build-lambda-list, since it receives (+ 1 1) not 2
I could evaluate the (+ 1 1) using the sharp-dot read macro, like so:
(jlambda #.(+ 1 1) (print “hello”)) ==> #<anonymous-function>
But then the form cannot contain references to lexical variables, since they are not available when evaluating at read-time:
(let ((x 1))
;; Do other stuff with x, then:
(jlambda #.(+ x 1) (print “hello”))) ==> failure – variable x not bound
I could quote all body code that I pass to jlambda, define it as a function instead, and then eval the code that it returns:
(defun jlambda (arity &rest body)
(let ((dummy-args (build-lambda-list arity)))
`(lambda ,dummy-args
(declare (ignore ,#dummy-args))
,#body)))
(eval (jlambda (+ 1 1) `(print “hello”))) ==> #<anonymous-function>
But I can't use eval because, like sharp-dot, it throws out the lexical environment, which is no good.
So jlambda must be a macro, because I don't want the function body code evaluated until the proper context for it has been established by jlambda's expansion; however it must also be a function, because I want the first form (in this example, the arity form) evaluated before passing it to helper functions that generate the macro expansion. How do I overcome this Catch-22 situation?
EDIT
In response to #Sylwester 's question, here's an explanation of the context:
I'm writing something akin to an “esoteric programming language”, implemented as a DSL in Common Lisp. The idea (admittedly silly but potentially fun) is to force the programmer, as far as possible (I'm not sure how far yet!), to write exclusively in point-free style. To do this, I will do several things:
Use curry-compose-reader-macros to provide most of the functionality required to write in point-free style in CL
Enforce functions' arity – i.e. override CL's default behaviour that allows functions to be variadic
Instead of using a type system to determine when a function has been “fully applied” (like in Haskell), just manually specify a function's arity when defining it.
So I'll need a custom version of lambda for defining a function in this silly language, and – if I can't figure that out - a custom version of funcall and/or apply for invoking those functions. Ideally they'll just be skins over the normal CL versions that change the functionality slightly.
A function in this language will somehow have to keep track of its arity. However, for simplicity, I would like the procedure itself to still be a funcallable CL object, but would really like to avoid using the MetaObject Protocol, since it's even more confusing to me than macros.
A potentially simple solution would be to use a closure. Every function could simply close over the binding of a variable that stores its arity. When invoked, the arity value would determine the exact nature of the function application (i.e. full or partial application). If necessary, the closure could be “pandoric” in order to provide external access to the arity value; that could be achieved using plambda and with-pandoric from Let Over Lambda.
In general, functions in my language will behave like so (potentially buggy pseudocode, purely illustrative):
Let n be the number of arguments provided upon invocation of the function f of arity a.
If a = 0 and n != a, throw a “too many arguments” error;
Else if a != 0 and 0 < n < a, partially apply f to create a function g, whose arity is equal to a – n;
Else if n > a, throw a “too many arguments” error;
Else if n = a, fully apply the function to the arguments (or lack thereof).
The fact that the arity of g is equal to a – n is where the problem with jlambda would arise: g would need to be created like so:
(jlambda (- a n)
...body)
Which means that access to the lexical environment is a necessity.
This is a particularly tricky situation because there's no obvious way to create a function of a particular number of arguments at runtime. If there's no way to do that, then it's probably easiest to write a a function that takes an arity and another function, and wraps the function in a new function that requires that is provided the particular number of arguments:
(defun %jlambda (n function)
"Returns a function that accepts only N argument that calls the
provided FUNCTION with 0 arguments."
(lambda (&rest args)
(unless (eql n (length args))
(error "Wrong number of arguments."))
(funcall function)))
Once you have that, it's easy to write the macro around it that you'd like to be able to:
(defmacro jlambda (n &body body)
"Produces a function that takes exactly N arguments and and evalutes
the BODY."
`(%jlambda ,n (lambda () ,#body)))
And it behaves roughly the way you'd want it to, including letting the arity be something that isn't known at compile time.
CL-USER> (let ((a 10) (n 7))
(funcall (jlambda (- a n)
(print 'hello))
1 2 3))
HELLO
HELLO
CL-USER> (let ((a 10) (n 7))
(funcall (jlambda (- a n)
(print 'hello))
1 2))
; Evaluation aborted on #<SIMPLE-ERROR "Wrong number of arguments." {1004B95E63}>.
Now, you might be able to do something that invokes the compiler at runtime, possibly indirectly, using coerce, but that won't let the body of the function be able to refer to variables in the original lexical scope, though you would get the implementation's wrong number of arguments exception:
(defun %jlambda (n function)
(let ((arglist (loop for i below n collect (make-symbol (format nil "$~a" i)))))
(coerce `(lambda ,arglist
(declare (ignore ,#arglist))
(funcall ,function))
'function)))
(defmacro jlambda (n &body body)
`(%jlambda ,n (lambda () ,#body)))
This works in SBCL:
CL-USER> (let ((a 10) (n 7))
(funcall (jlambda (- a n)
(print 'hello))
1 2 3))
HELLO
CL-USER> (let ((a 10) (n 7))
(funcall (jlambda (- a n)
(print 'hello))
1 2))
; Evaluation aborted on #<SB-INT:SIMPLE-PROGRAM-ERROR "invalid number of arguments: ~S" {1005259923}>.
While this works in SBCL, it's not clear to me whether it's actually guaranteed to work. We're using coerce to compile a function that has a literal function object in it. I'm not sure whether that's portable or not.
NB: In your code you use strange quotes so that (print “hello”) doesn't actually print hello but the whatever the variable “hello” evaluates to, while (print "hello") does what one would expect.
My first question is why? Usually you know how many arguments you are taking compile time or at least you just make it multiple arity. Making an n arity function only gives you errors when passwd with wrong number of arguments as added feature with the drawback of using eval and friends.
It cannot be solved as a macro since you are mixing runtime with macro expansion time. Imagine this use:
(defun test (last-index)
(let ((x (1+ last-index)))
(jlambda x (print "hello"))))
The macro is expanded when this form is evaluated and the content replaced before the function is assigned to test. At this time x doesn't have any value whatsoever and sure enough the macro function only gets the symbols so that the result need to use this value. lambda is a special form so it again gets expanded right after the expansion of jlambda, also before any usage of the function.
There is nothing lexical happening since this happens before the program is running. It could happen before loading the file with compile-file and then if you load it will load all forms with the macros already expanded beforehand.
With compile you can make a function from data. It is probably as evil as eval is so you shouldn't be using it for common tasks, but they exist for a reason:
;; Macro just to prevent evaluation of the body
(defmacro jlambda (nexpr &rest body)
`(let ((dummy-args (build-lambda-list ,nexpr)))
(compile nil (list* 'lambda dummy-args ',body))))
So the expansion of the first example turns into this:
(defun test (last-index)
(let ((x (1+ last-index)))
(let ((dummy-args (build-lambda-list x)))
(compile nil (list* 'lambda dummy-args '((print "hello")))))))
This looks like it could work. Lets test it:
(defparameter *test* (test 10))
(disassemble *test*)
;Disassembly of function nil
;(CONST 0) = "hello"
;11 required arguments <!-- this looks right
;0 optional arguments
;No rest parameter
;No keyword parameters
;4 byte-code instructions:
;0 (const&push 0) ; "hello"
;1 (push-unbound 1)
;3 (calls1 142) ; print
;5 (skip&ret 12)
;nil
Possible variations
I've made a macro that takes a literal number and makes bound variables from a ... that can be used in the function.
If you are not using the arguments why not make a macro that does this:
(defmacro jlambda2 (&rest body)
`(lambda (&rest #:rest) ,#body))
The result takes any number of arguments and just ignores it:
(defparameter *test* (jlambda2 (print "hello")))
(disassemble *test*)
;Disassembly of function :lambda
;(CONST 0) = "hello"
;0 required arguments
;0 optional arguments
;Rest parameter <!-- takes any numer of arguments
;No keyword parameters
;4 byte-code instructions:
;0 (const&push 0) ; "hello"
;1 (push-unbound 1)
;3 (calls1 142) ; print
;5 (skip&ret 2)
;nil
(funcall *test* 1 2 3 4 5 6 7)
; ==> "hello" (prints "hello" as side effect)
EDIT
Now that I know what you are up to I have an answer for you. Your initial function does not need to be runtime dependent so all functions indeed have a fixed arity, so what we need to make is currying or partial application.
;; currying
(defmacro fixlam ((&rest args) &body body)
(let ((args (reverse args)))
(loop :for arg :in args
:for r := `(lambda (,arg) ,#body)
:then `(lambda (,arg) ,r)
:finally (return r))))
(fixlam (a b c) (+ a b c))
; ==> #<function :lambda (a) (lambda (b) (lambda (c) (+ a b c)))>
;; can apply multiple and returns partially applied when not enough
(defmacro fixlam ((&rest args) &body body)
`(let ((lam (lambda ,args ,#body)))
(labels ((chk (args)
(cond ((> (length args) ,(length args)) (error "too many args"))
((= (length args) ,(length args)) (apply lam args))
(t (lambda (&rest extra-args)
(chk (append args extra-args)))))))
(lambda (&rest args)
(chk args)))))
(fixlam () "hello") ; ==> #<function :lambda (&rest args) (chk args)>
;;Same but the zero argument functions are applied right away:
(defmacro fixlam ((&rest args) &body body)
`(let ((lam (lambda ,args ,#body)))
(labels ((chk (args)
(cond ((> (length args) ,(length args)) (error "too many args"))
((= (length args) ,(length args)) (apply lam args))
(t (lambda (&rest extra-args)
(chk (append args extra-args)))))))
(chk '()))))
(fixlam () "hello") ; ==> "hello"
If all you want is lambda functions that can be applied either partially or fully, I don't think you need to pass the amount of parameters explicitly. You could just do something like this (uses Alexandria):
(defmacro jlambda (arglist &body body)
(with-gensyms (rest %jlambda)
`(named-lambda ,%jlambda (&rest ,rest)
(cond ((= (length ,rest) ,(length arglist))
(apply (lambda ,arglist ,#body) ,rest))
((> (length ,rest) ,(length arglist))
(error "Too many arguments"))
(t (apply #'curry #',%jlambda ,rest))))))
CL-USER> (jlambda (x y) (format t "X: ~s, Y: ~s~%" x y))
#<FUNCTION (LABELS #:%JLAMBDA1046) {1003839D6B}>
CL-USER> (funcall * 10) ; Apply partially
#<CLOSURE (LAMBDA (&REST ALEXANDRIA.0.DEV::MORE) :IN CURRY) {10038732DB}>
CL-USER> (funcall * 20) ; Apply fully
X: 10, Y: 20
NIL
CL-USER> (funcall ** 100) ; Apply fully again
X: 10, Y: 100
NIL
CL-USER> (funcall *** 100 200) ; Try giving a total of 3 args
; Debugger entered on #<SIMPLE-ERROR "Too many arguments" {100392D7E3}>
Edit: Here's also a version that lets you specify the arity. Frankly, I don't see how this could possibly be useful though. If the user cannot refer to the arguments, and nothing is done with them automatically, then, well, nothing is done with them. They might as well not exist.
(defmacro jlambda (arity &body body)
(with-gensyms (rest %jlambda n)
`(let ((,n ,arity))
(named-lambda ,%jlambda (&rest ,rest)
(cond ((= (length ,rest) ,n)
,#body)
((> (length ,rest) ,n)
(error "Too many arguments"))
(t (apply #'curry #',%jlambda ,rest)))))))
CL-USER> (jlambda (+ 1 1) (print "hello"))
#<CLOSURE (LABELS #:%JLAMBDA1085) {1003B7913B}>
CL-USER> (funcall * 2)
#<CLOSURE (LAMBDA (&REST ALEXANDRIA.0.DEV::MORE) :IN CURRY) {1003B7F7FB}>
CL-USER> (funcall * 5)
"hello"
"hello"
Edit2: If I understood correctly, you might be looking for something like this (?):
(defvar *stack* (list))
(defun jlambda (arity function)
(lambda ()
(push (apply function (loop repeat arity collect (pop *stack*)))
*stack*)))
CL-USER> (push 1 *stack*)
(1)
CL-USER> (push 2 *stack*)
(2 1)
CL-USER> (push 3 *stack*)
(3 2 1)
CL-USER> (push 4 *stack*)
(4 3 2 1)
CL-USER> (funcall (jlambda 4 #'+)) ; take 4 arguments from the stack
(10) ; and apply #'+ to them
CL-USER> (push 10 *stack*)
(10 10)
CL-USER> (push 20 *stack*)
(20 10 10)
CL-USER> (push 30 *stack*)
(30 20 10 10)
CL-USER> (funcall (jlambda 3 [{reduce #'*} #'list])) ; pop 3 args from
(6000 10) ; stack, make a list
; of them and reduce
; it with #'*
In Python you can write this:
def firstn(n):
num = 0
while num < n:
yield num
num += 1
What is the lisp equivalent of this?
Existing package
Download, install and load the GENERATORS system with Quicklisp. Then, use package :generators (or preferably, define your own package first).
(ql:quickload :generators)
(use-package :generators)
Define an infinite generator for random values:
(defun dice (n)
(make-generator ()
;; repeatedly return a random value between 1 and N
(loop (yield (1+ (random n))))))
Use the generator:
(loop
with dice = (dice 6)
repeat 20
collect (next dice))
=> (1 2 6 1 1 4 4 2 4 3 6 2 1 5 6 5 1 5 1 2)
Note however what the author of the library says:
This library is more of an interesting toy, though as far as I know it
does work. I dont think I have ever used this in application code,
though I think that with care, it could be.
See also
The ITERATE package provides a way to define generators for use inside its iteration facility.
The SERIES package provide stream-like data structures and operations on them.
The Snakes library (same approach as GENERATORS as far as I know).
Iterators in generic-cl
Closures
In practice, CL does not rely that much on generators as popularized by Python. What happens instead is that when people need lazy sequences, they use closures:
(defun dice (n)
(lambda ()
(1+ (random n))))
Then, the equivalent of next is simply a call to the thunk generated by dice:
(loop
with dice = (dice 6)
repeat 20
collect (funcall dice))
This is the approach that is preferred, in particular because there is no need to rely on delimited continuations like with generators. Your example involves a state, which the dice example does not require (there is a hidden state that influences random, but that's another story) . Here is how your counter is typically implemented:
(defun first-n (n)
(let ((counter -1))
(lambda ()
(when (< counter n)
(incf counter)))))
Higher-order functions
Alternatively, you design a generator that accepts a callback function which is called by your generator for each value. Any funcallable can be used, which allows the caller to retain control over code execution:
(defun repeatedly-throw-dice (n callback)
(loop (funcall callback (1+ (random n)))))
Then, you can use it as follows:
(prog ((counter 0) stack)
(repeatedly-throw-dice 6
(lambda (value)
(if (<= (incf counter) 20)
(push value stack)
(return (nreverse stack))))))
See documentation for PROG.
do-traversal idiom
Instead of building a function, data sources that provides a custom way of generating values (like matches of a regular expressions in a string) also regularly provide a macro that abstracts their control-flow. You would use it as follows:
(let ((counter 0) stack)
(do-repeatedly-throw-dice (value 6)
(if (<= (incf counter) 20)
(push value stack)
(return (nreverse stack))))))
DO-X macros are expected to define a NIL block around their body, which is why the return above is valid.
A possible implementation for the macro is to wrap the body in a lambda form and use the callback-based version defined above:
(defmacro do-repeatedly-throw-dice ((var n) &body body)
`(block nil (repeatedly-throw-dice ,n (lambda (,var) ,#body))))
Directly expanding into a loop would be possible too:
(defmacro do-repeatedly-throw-dice ((var n) &body body)
(let ((max (gensym)) (label (make-symbol "NEXT")))
`(prog ((,max ,n) ,var)
,label
(setf ,var (1+ (random ,max)))
(progn ,#body)
(go ,label))))
One step of macroexpansion for above form:
(prog ((#:g1078 6) value)
#:next
(setf value (1+ (random #:g1078)))
(progn
(if (<= (incf counter) 20)
(push value stack)
(return (nreverse stack))))
(go #:next))
Bindings
Broadly speaking, building a generator with higher-order functions or directly with a do- macro gives the same result. You can implement one with the other (personally, I prefer to define first the macro and then the function using the macro, but doing the opposite is also interesting, since you can redefine the function without recompiling all usages of the macro).
However, there is still a difference: the macro reuses the same variable across iterations, whereas the closure introduces a fresh binding each time. For example:
(let ((list))
(dotimes (i 10) (push (lambda () i) list))
(mapcar #'funcall list))
.... returns:
(10 10 10 10 10 10 10 10 10 10)
Most (if not all) iterators in Common Lisp tend to work like this1, and it should not come as a surprise for experienced users (the opposite would be surprising, in fact). If dotimes was implemented by repeatedly calling a closure, the result would be different:
(defmacro my-dotimes ((var count-form &optional result-form) &body body)
`(block nil
(alexandria:map-iota (lambda (,var) ,#body) ,count-form)
,result-form))
With the above definition, we can see that:
(let ((list))
(my-dotimes (i 10) (push (lambda () i) list))
(mapcar #'funcall list))
... returns:
(9 8 7 6 5 4 3 2 1 0)
In order to have the same result with the standard dotimes, you only need to create a fresh binding before building the closure:
(let ((list))
(dotimes (i 10)
(let ((j i))
(push (lambda () j) list))))
Here j is a fresh binding whose value is the current value of i at closure creation time; j is never mutated so the closure will constantly return the same value.
If you wanted to, you could always introduce that inner let from the macro, but this is rarely done.
1: Note that the specification for DOTIMES does not require that bindings are fresh at each iteration, or only mutates the same binding at each step: "It is implementation-dependent whether dotimes establishes a new binding of var on each iteration or whether it establishes a binding for var once at the beginning and then assigns it on any subsequent iterations." In order to write portably, it is necessary to assume the worst-case scenario (i.e. mutation, which happens to be what most (all?) implementations do) and manually rebind iteration variables if they are to be captured and reused at a later point.
I have various functions and I want to call each function with the same value. For instance,
I have these functions:
(defun OP1 (arg) ( + 1 arg) )
(defun OP2 (arg) ( + 2 arg) )
(defun OP3 (arg) ( + 3 arg) )
And a list containing the name of each function:
(defconstant *OPERATORS* '(OP1 OP2 OP3))
So far, I'm trying:
(defun TEST (argument) (dolist (n *OPERATORS*) (n argument) ) )
I've tried using eval, mapcar, and apply, but these haven't worked.
This is just a simplified example; the program that I'm writing has eight functions that are needed to expand nodes in a search tree, but for the moment, this example should suffice.
Other answers have provided some idiomatic solutions with mapcar. One pointed out that you might want a list of functions (which *operators* isn't) instead of a list of symbols (which *operators* is), but it's OK in Common Lisp to funcall a symbol. It's probably more common to use some kind of mapping construction (e.g., mapcar) for this, but since you've provided code using dolist, I think it's worth looking at how you can do this iteratively, too. Let's cover the (probably more idiomatic) solution with mapping first, though.
Mapping
You have a fixed argument, argument, and you want to be able to take a function function and call it with that `argument. We can abstract this as a function:
(lambda (function)
(funcall function argument))
Now, we want to call this function with each of the operations that you've defined. This is simple to do with mapcar:
(defun test (argument)
(mapcar (lambda (function)
(funcall function argument))
*operators*))
Instead of operators, you could also write '(op1 op2 op3) or (list 'op1 'op2 'op3), which are lists of symbols, or (list #'op1 #'op2 #'op3) which is a list of functions. All of these work because funcall takes a function designator as its first argument, and a function designator is
an object that denotes a function and that is one of: a symbol (denoting the function named by that symbol in the global environment), or a function (denoting itself).
Iteratively
You can do this using dolist. The [documentation for actually shows that dolist has a few more tricks up its sleeve. The full syntax is from the documentation
dolist (var list-form [result-form]) declaration* {tag | statement}*
We don't need to worry about declarations here, and we won't be using any tags, but notice that optional result-form. You can specify a form to produce the value that dolist returns; you don't have to accept its default nil. The common idiom for collecting values into a list in an iterative loop is to push each value into a new list, and then return the reverse of that list. Since the new list doesn't share structure with anything else, we usually reverse it destructively using nreverse. Your loop would become
(defun test (argument)
(let ((results '()))
(dolist (op *operators* (nreverse results))
(push (funcall op argument) results))))
Stylistically, I don't like that let that just introduces a single value, and would probably use an &aux variable in the function (but this is a matter of taste, not correctness):
(defun test (argument &aux (results '()))
(dolist (op *operators* (nreverse results))
(push (funcall op argument) results)))
You could also conveniently use loop for this:
(defun test2 (argument)
(loop for op in *operators*
collect (funcall op argument)))
You can also do somewhat succinctly, but perhaps less readably, using do:
(defun test3a (argument)
(do ((results '() (list* (funcall (first operators) argument) results))
(operators *operators* (rest operators)))
((endp operators) (nreverse results))))
This says that on the first iteration, results and operators are initialized with '() and *operators*, respectively. The loop terminates when operators is the empty list, and whenever it terminates, the return value is (nreverse results). On successive iterations, results is a assigned new value, (list* (funcall (first operators) argument) results), which is just like pushing the next value onto results, and operators is updated to (rest operators).
FUNCALL works with symbols.
From the department of silly tricks.
(defconstant *operators* '(op1 op2 o3))
(defun test (&rest arg)
(setf (cdr arg) arg)
(mapcar #'funcall *operators* arg))
There's a library, which is almost mandatory in any anywhat complex project: Alexandria. It has many useful functions, and there's also something that would make your code prettier / less verbose and more conscious.
Say, you wanted to call a number of functions with the same value. Here's how you'd do it:
(ql:quickload "alexandria")
(use-package :alexandria)
(defun example-rcurry (value)
"Calls `listp', `string' and `numberp' with VALUE and returns
a list of results"
(let ((predicates '(listp stringp numberp)))
(mapcar (rcurry #'funcall value) predicates)))
(example-rcurry 42) ;; (NIL NIL T)
(example-rcurry "42") ;; (NIL T NIL)
(defun example-compose (value)
"Calls `complexp' with the result of calling `sqrt'
with the result of calling `parse-integer' on VALUE"
(let ((predicates '(complexp sqrt parse-integer)))
(funcall (apply #'compose predicates) value)))
(example-compose "0") ;; NIL
(example-compose "-1") ;; T
Functions rcurry and compose are from Alexandria package.
I am having trouble with Lisp's backquote read macro. Whenever I try to write a macro that seems to require the use of embedded backquotes (e.g., ``(w ,x ,,y) from Paul Graham's ANSI Common Lisp, page 399), I cannot figure out how to write my code in a way that compiles. Typically, my code receives a whole chain of errors preceded with "Comma not inside a backquote." Can someone provide some guidelines for how I can write code that will evaluate properly?
As an example, I currently need a macro which takes a form that describes a rule in the form of '(function-name column-index value) and generates a predicate lambda body to determine whether the element indexed by column-index for a particular row satisfies the rule. If I called this macro with the rule '(< 1 2), I would want a lambda body that looks like the following to be generated:
(lambda (row)
(< (svref row 1) 2))
The best stab I can make at this is as follows:
(defmacro row-satisfies-rule (rule)
(let ((x (gensym)))
`(let ((,x ,rule))
(lambda (row)
(`,(car ,x) (svref row `,(cadr ,x)) `,(caddr ,x))))))
Upon evaluation, SBCL spews the following error report:
; in: ROW-SATISFIES-RULE '(< 1 2)
; ((CAR #:G1121) (SVREF ROW (CADR #:G1121)) (CADDR #:G1121))
;
; caught ERROR:
; illegal function call
; (LAMBDA (ROW) ((CAR #:G1121) (SVREF ROW (CADR #:G1121)) (CADDR #:G1121)))
; ==>
; #'(LAMBDA (ROW) ((CAR #:G1121) (SVREF ROW (CADR #:G1121)) (CADDR #:G1121)))
;
; caught STYLE-WARNING:
; The variable ROW is defined but never used.
; (LET ((#:G1121 '(< 1 2)))
; (LAMBDA (ROW) ((CAR #:G1121) (SVREF ROW (CADR #:G1121)) (CADDR #:G1121))))
;
; caught STYLE-WARNING:
; The variable #:G1121 is defined but never used.
;
; compilation unit finished
; caught 1 ERROR condition
; caught 2 STYLE-WARNING conditions
#<FUNCTION (LAMBDA (ROW)) {2497F245}>
How can I write macros to generate the code I need, and in particular, how do I implement row-satisfies-rule?
Using the ideas from Ivijay and discipulus, I have modified the macro so that it compiles and works, even allowing forms to be passed as the arguments. It runs a bit differently from my originally planned macro since I determined that including row as an argument made for smoother code. However, it is ugly as sin. Does anyone know how to clean it up so it performs the same without the call to eval?
(defmacro row-satisfies-rule-p (row rule)
(let ((x (gensym))
(y (gensym)))
`(let ((,x ,row)
(,y ,rule))
(destructuring-bind (a b c) ,y
(eval `(,a (svref ,,x ,b) ,c))))))
Also, an explanation of clean, Lispy ways to get macros to generate code to properly evaluate the arguments at runtime would be greatly appreciated.
First of all, Lisp macros have "destructuring" argument lists. This is a nice feature that means instead of having an argument list (rule) and then taking it apart with (car rule) (cadr rule) (caddr rule), you can simply make the argument list ((function-name column-index value)). That way the macro expects a list of three elements as an argument, and each element of the list is then bound to the corresponding symbol in the arguemnt list. You can use this or not, but it's usually more convenient.
Next, `, doesn't actually do anything, because the backquote tells Lisp not to evaluate the following expression and the comma tells it to evaluate it after all. I think you meant just ,(car x), which evaluates (car x). This isn't a problem anyway if you use destructuring arguments.
And since you're not introducing any new variables in the macro expansion, I don't think (gensym) is necessary in this case.
So we can rewrite the macro like this:
(defmacro row-satisfies-rule ((function-name column-index value))
`(lambda (row)
(,function-name (svref row ,column-index) ,value)))
Which expands just how you wanted:
(macroexpand-1 '(row-satisfies-rule (< 1 2)))
=> (LAMBDA (ROW) (< (SVREF ROW 1) 2))
Hope this helps!
If you need the argument to be evaluated to get the rule set, then here's a nice way to do it:
(defmacro row-satisfies-rule (rule)
(destructuring-bind (function-name column-index value) (eval rule)
`(lambda (row)
(,function-name (svref row ,column-index) ,value))))
Here's an example:
(let ((rules '((< 1 2) (> 3 4))))
(macroexpand-1 '(row-satisfies-rule (car rules))))
=> (LAMBDA (ROW) (< (SVREF ROW 1) 2))
just like before.
If you want to include row in the macro and have it give you your answer straightaway instead of making a function to do that, try this:
(defmacro row-satisfies-rule-p (row rule)
(destructuring-bind (function-name column-index value) rule
`(,function-name (svref ,row ,column-index) ,value)))
Or if you need to evaluate the rule argument (e.g. passing '(< 1 2) or (car rules) instead of (< 1 2)) then just use (destructuring-bind (function-name column-index value) (eval rule)
Actually, a function seems more appropriate than a macro for what you're trying to do. Simply
(defun row-satisfies-rule-p (row rule)
(destructuring-bind (function-name column-index value) rule
(funcall function-name (svref row column-index) value)))
works the same way as the macro and is much neater, without all the backquoting mess to worry about.
In general, it's bad Lisp style to use macros for things that can be accomplished by functions.
One thing to understand is that the backquote feature is completely unrelated to macros. It can be used for list creation. Since source code usually consists of lists, it may be handy in macros.
CL-USER 4 > `((+ 1 2) ,(+ 2 3))
((+ 1 2) 5)
The backquote introduces a quoted list. The comma does the unquote: the expression after the comma is evaluated and the result inserted. The comma belongs to the backquote: the comma is only valid inside a backquote expression.
Note also that this is strictly a feature of the Lisp reader.
Above is basically similar to:
CL-USER 5 > (list '(+ 1 2) (+ 2 3))
((+ 1 2) 5)
This creates a new list with the first expression (not evaluated, because quoted) and the result of the second expression.
Why does Lisp provide backquote notation?
Because it provides a simple template mechanism when one wants to create lists where most of the elements are not evaluated, but a few are. Additionally the backquoted list looks similar to the result list.
you don't need nested backquotes to solve this problem. Also, when it's a macro, you don't have to quote your arguments. So (row-satisfies-rule (< 1 2)) is lispier than (row-satisfies-rule '(< 1 2)).
(defmacro row-satisfies-rule (rule)
(destructuring-bind (function-name column-index value) rule
`(lambda (row)
(,function-name (svref row ,column-index) ,value))))
will solve the problem for all calls in the first form. Solving the problem when in the second form is left as an exercise.