What does the &rest mean in this: (append &rest sequences)
I have finished the book "elisp_intro"，but I didn't find the answer. I will be very grateful if you can help me.
It indicates that the following argument is the "rest" argument. In the case of your example, sequences is the rest argument.
Unlike a normal argument, the rest argument will consume the "rest" of any parameters provided by the user. It is a way of making a function that accepts a variable number of arguments. Inside the function body, the argument will be a list containing all of the extra parameters.
So if you had a function that looked like (add &rest numbers), then you could call it like (add 1 2 3 4 5), and within the function the numbers argument would have the value (1 2 3 4 5).
Or for example, ((lambda (a &rest r) r) 1 2 3) ;=> (2 3)
Any arguments following the &rest symbol in a function definition will be collected into a single list object. So in the case of (append &rest sequences), if you call the function:
(append 1)
the variable sequences will be the list (1) in the body of the function. If you call it as
(append 1 2 3)
the variable sequences will be the list (1 2 3) in the body of the function. This is explained in the Elisp Reference manual, in the argument list node.
(info "(elisp)Argument List")
elisp/Common Lisp &rest is analogous to the *args parameter in Python
If you put this in your Common Lisp repl and call foo with varying numbers of arguments, you will see what's up quite quickly.
(defun foo (a b &rest trailing)
(format t "~a :: ~a :: ~a~%" a b trailing))
Related
I am going through a lisp book and I am looking at mapcar, my question is why is that this is valid:
> (mapcar #'+ '(1 2) '(3 4))
but this one isn't:
(mapcar + '(1 2) '(3 4))
in other words, is there a reason it was decided in lisp that the first argument to mapcar cannot be the function itself, and has to be its name? what purpose does this serve?
is there a reason it was decided in lisp that the first argument to mapcar cannot be the function itself, and has to be its name? what purpose does this serve?
It's just that in something like Common Lisp, the identifier + has multiple different purposes. It is a variable, a function, plus various other things.
writing + means the variable. It is used by the read eval print loop. The value of + is the last form that was evaluated, the value of ++ is the previous value of +, and the value of +++ is the previous value of ++.
To tell Common Lisp that you want to use the function value of an identifier +, one has to write (function +) or shorter #'+.
Thus
(mapcar (function +) '(1 2) '(3 4))
or shorter
(mapcar #'+ '(1 2) '(3 4))
actually means call mapcar with the function + and the lists (1 2) and (3 4)
There are two other ways to use the function +.
(mapcar '+ '(1 2) '(3 4))
Above will have Lisp retrieve the global function value of the symbol +.
Fourth, we can also have the function object be a part of the source code.
#'+ is a function. Common Lisp is what's called a 'lisp 2', which means that it has two namespaces: during evaluation of a compound form, like (+ a b), the function position is looked up in the function namespace while the other positions are looked up in the value namespace. This means that, for instance (append list list) does not make a list whose two elements are the list function: it makes a list whose two elements are whatever list happens to be bound to as a value.
But sometimes you need to get the function value of a name in a value position: an example is in the first argument of mapcar. To do that there is a special operator, function: (mapcar (function +) x y) adds the elements of two lists. Like quote, function has a read macro, which is #'.
(To make this more fun, mapcar actually expects a function designator so you can use the nsme of a function: (mapcar '+ x y) will work, although it is a bit horrid.)
I am hoping someone can explain why tests 1-5 work but test 6 does not. I thought that quoting a lambda with ' and using #' in front of a lambda both returned pointers to the function with the only difference being that the #' will compile it first.
(defun test-1 (y)
(mapcar (lambda (x) (expt x 2))
'(1 2 3)))
(defun test-2 (y)
(mapcar (lambda (x) (expt x y))
'(1 2 3)))
(defun test-3 (y)
(mapcar #'(lambda (x) (expt x 2))
'(1 2 3)))
(defun test-4 (y)
(mapcar #'(lambda (x) (expt x y))
'(1 2 3)))
(defun test-5 (y)
(mapcar '(lambda (x) (expt x 2))
'(1 2 3)))
(defun test-6 (y)
(mapcar '(lambda (x) (expt x y))
'(1 2 3)))
I am using the free version of Franz Industries Allegro Common Lisp. The following are the outputs:
(test-1 2) ; --> (1 4 9)
(test-2 2) ; --> (1 4 9)
(test-3 2) ; --> (1 4 9)
(test-4 2) ; --> (1 4 9)
(test-5 2) ; --> (1 4 9)
(test-6 2) ; --> Error: Attempt to take the value of the unbound variable `Y'. [condition type: UNBOUND-VARIABLE]
For a start, you should be aware that your tests 1-4 are conforming Common Lisp, while your tests 5 and 6 are not. I believe Allegro is perfectly well allowed to do what it does for 5 and 6, but what it is doing is outside the standard. The bit of the standard that talks about this is the definition of functions like mapcar, which take function designators as argument, and the definition of a function designator:
function designator n. a designator for a function; that is, an object that denotes a function and that is one of: a symbol (denoting the function named by that symbol in the global environment), or a function (denoting itself). The consequences are undefined if a symbol is used as a function designator but it does not have a global definition as a function, or it has a global definition as a macro or a special form. [...]
From this it is clear that a list like (lambda (...) ...) is not a function designator: it's just a list whose car happens to be lambda. What Allegro is doing is noticing that this list is in fact something that can be turned into a function and doing that.
Well, let's just write a version of mapcar which does what Allegro's does:
(defun mapcar/coercing (maybe-f &rest lists)
(apply #'mapcar (coerce maybe-f 'function) lists))
This just uses coerce which is a function which knows how to turn lists like this into functions, among other things. If its argument is already a function, coerce just returns it.
Now we can write the two tests using this function:
(defun test-5/coercing (y)
(mapcar/coercing '(lambda (x) (expt x 2))
'(1 2 3)))
(defun test-6/coercing (y)
(mapcar/coercing '(lambda (x) (expt x y))
'(1 2 3)))
So, after that preamble, why can't test-6/explicit work? Well the answer is that Common Lisp is (except for for special variables) lexically scoped. Lexical scope is just a fancy way of saying that the bindings (variables) that are available are exactly and only the bindings you can see by looking at the source of the program. (Except, in the case of CL for special bindings, which I'll ignore, since there are none here.)
So, given this, think about test-6/coercing, and in particular the call to mapcar/coercing: in that call, coerce has to turn the list (lambda (x) (expt z y)) into a function. So it does that. But the function it returns doesn't bind y and there is no binding for y visible in it: the function uses y 'free'.
The only way that this could work is if the function that coerce constructs for us were to dynamically look for a binding for y. Well, that's what dynamically-scoped languages do, but CL is not dynamically-scoped.
Perhaps a way of making this even clearer is to realise that we can lift the function creation right out of the function:
(defun test-7 (y f)
(mapcar f '(1 2 3)))
> (test-7 1 (coerce '(lambda (x) (expt x y)) 'function))
It's clear that this can't work in a lexically-scoped language.
So, then, how do tests 1-4 work?
Well, firstly there are only actually two tests here. In CL, lambda is a macro and (lambda (...) ...) is entirely equivalent to (function (lambda (...) ...)). And of course #'(lambda (...) ...) is also the same as (function (lambda (...) ...)): it's just a read-macro for it.
And (function ...) is a magic thing (a special form) which says 'this is a function'. The important thing about function is that it's not a function: it's a deeply magic thing which tells the evaluator (or the compiler) that its argument is the description of a function in the current lexical context, so, for instance in
(let ((x 1))
(function (lambda (y) (+ x y))))
The x referred to by the function this creates is the x bound by let. So in your tests 2 and 4 (which are the same):
(defun test-4 (y)
(mapcar (function (lambda (x) (expt x y)))
'(1 2 3)))
The binding of y which the function created refers to is the binding of y which is lexically visible, which is the argument of test-4 itself.
Let's add a y parameter to avoid closing over variables and see what kind of values we are manipulating:
USER> (type-of #'(lambda (x y) (expt x y)))
FUNCTION
USER> (type-of (lambda (x y) (expt x y)))
FUNCTION
USER> (type-of '(lambda (x y) (expt x y)))
CONS
As you can see, the two first lambda-like forms are evaluated as functions, while the third is evaluated as a cons-cell. As far as Lisp is concerned, the third argument is just a tree of symbols with no meaning.
Reader macros
I thought that quoting a lambda with ' and using #' in front of a lambda both returned pointers to the function with the only difference being that the #' will compile it first.
Let's go back to the definitions, ' and #' are reader macros, respectively Single-Quote and Sharpsign Single-Quote. They are found in front of other forms, for example 'f is read as (quote f) and #'f is read as (function f). At read-time, f and the resulting forms are just unevaluated data.
We will see below how both special operators are interpreted, but what matters really is the lexical scope, so let's open a parenthesis.
Lexical environment
Lexical environments are the set of bindings in effect at some point of your code. When you evaluate a let or an flet it enriches the current environment with new bindings. When you call EVAL on an expression, you start evaluating from a null lexical environment, even if the call to eval itself is in a non-null environment.
Here x is just unbound during eval:
(let ((x 3)) (eval '(list x))) ;; ERROR
Here we build a let to be evaluated by eval:
(eval '(let ((x 3)) (list x)))
=> (3)
That's all for the crash course on lexical environments.
Special operators
FUNCTION
Special operator FUNCTION takes an argument that is either the name of a function (symbol or setf), or a lambda expression; in particular:
The value of function is the functional value of name in the current lexical environment.
Here the lambda expression is evaluated in the current lexical environment, which means it can refer to variable outside the lambda expression. That's the definition of closures, they capture the surrounding bindings.
NB. you do not need to prefix lambda with #', because there is a macro named (lambda ...) that expands into (function (lambda ...)). It looks like this could expand recursively forever, but this is not the case: at first the macro is expanded so that (lambda ...) becomes (function (lambda ...)), then the special operator function knows how to evaluate the lambda expression itself.
This means that (lambda ...) and #'(lambda ...) are equivalent. Note in particular that there is nothing about whether one form is compiled or not at this point, the compiler will see the same expression after macroexpansion.
QUOTE
Special operator QUOTE evaluates (quote f) as f, where f itself is unevaluated. In test-5 and test-6, there is no function, just an unevaluated structured expression that can be interpreted as code.
Type coercion
Now, certain functions like MAPCAR are used to apply functions. Notice how the specification says that the function parameter is a function designator:
function --- a designator for a function that must take as many arguments as there are lists.
A designator for a type is not necessarily a value of that type, but can be a value that can be coerced to that type. Sometimes a user wants to specify a pathname, and enters a string, but a string is not a value of type pathname: the system has to converts the string into a pathname.
Common Lisp defines a COERCE function with rules regarding how values can be converted to other values. In you case, mapcar first does (coerce (lambda ...) 'function). This is defined as follows:
If the result-type is function, and object is a lambda expression, then the result is a closure of object in the null lexical environment.
The value is thus evaluated in a null lexical environment, so it does not have access to the surrounding bindings; y is a free variable in your lambda expression, and since it is evaluated in a null environment, it is unbound. That's why test-5 pass but test-6 fails.
Name resolution, compilers and late binding
There is a difference whether you write #'f or 'f when referring to a function f where f is a symbol: in the first case, the expression evaluated to an object of type function, and in the second case, you only evaluate a symbol.
Name resolution for this function can change depending and how the compiler works. With a symbol as a function designator, the function does not even need to be defined, the name is resolved when the symbols has to be coerced as a function.
When you write #'f, some compilers may remove one level of indirection and directly make your code jump to the code associated with the function, without having to resolve the name at runtime.
However, this also means that with such compilers (e.g. SBCL), you need to recompile some call sites on function redefinition, as-if the function was declared inline, otherwise some old code will still reference the previous definition of #'f. This is something that is not necessarily important to consider at the beginning, but it can be a source of confusion to keep in mind when you are live coding.
Is there a way to specify keyword arguments to a function as a dictionary, analogously to the . that allows us to supply a list for positional arguments? I'm looking for something analogous to python's *args and **kwargs.
In more detail:
If we want to handle an arbitrary number of positional arguments in a function, we can define it like so:
(define (my-add . args)
(apply + args))
and use it with a variable number of arguments as:
(define numbers '(1 2 3))
(apply my-add numbers) ;=> 6
(define numbers '(1 2 3 4 5))
(apply my-add numbers) ;=> 15
Even for functions that don't explicitly handle the "rest" of the positional arguments via the . syntax, we can use apply to unpack positional arguments from a list.
(apply expt '(2 3)) ;=> 8
I see that apply does forward on keyword arguments in addition to positional arguments. However, this only works if the invoked function itself identifies the specific keyword arguments it is expecting. For instance, in a dispatch situation where the function doesn't itself use the keyword arguments but merely intends to forward them on to some appropriate handler, using apply in this way doesn't do what we need because the function would need to know about all the keyword arguments accepted by any function it is dispatching to, not to mention do the legwork of repacking them in some way before invoking the handlers.
As a tangible example illustrating the problem, consider a simple "echo args" function that echoes all of the arguments passed to it:
(define (echo-args . args)
(printf "~a" args))
With only positional args:
> (echo-args 1 2 5)
(1 2 5)
With keyword args:
> (echo-args 1 2 5 #:key 'hi)
; application: procedure does not accept keyword arguments
; procedure: echo-args
; arguments...:
; 1
; 2
; 5
; #:key 'hi
; Context:
; /Applications/Racket v7.5/collects/racket/repl.rkt:11:26
Is there a built-in way to accomplish this kind of thing? If not, any ideas on how to make this work?
What you are looking for is make-keyword-procedure. This generates a procedure that accepts all keyword arguments, and can be accessed from a dictionary.Taking the example from the docs:
> (define show
(make-keyword-procedure (lambda (kws kw-args . rest)
(list kws kw-args rest))))
> (show #:blue 3 4)
'((#:blue) (3) (4))
Of course, a dictionary implemented as two lists can be a bit annoying. So Alex Knauth made a small package hash-lambda which gives you a more direct hash-map like API.
I’m trying to get get my macro to do an extra evaluation of its result before returning it. Can this be done without eval?
I'm trying to solve the problem in exercise 4 below:
Define a macro nth-expr that takes an integer n and an arbitrary number of expressions, evaluates the nth expression and returns its value. This exercise is easy to solve, if you assume that the first argument is a literal integer.
4. As exercise 3, but assume that the first argument is an expression to be evaluated.
It's easy to get the macro to pick the right expression:
(defmacro nth-expr% (n &rest es)
`(nth ,n ',es))
CL-USER> (defvar i 1)
I
CL-USER> (nth-expr% (1+ i) (+ 2 3) (- 4 3) (+ 3 1))
(+ 3 1)
The expression (+ 3 1) is the one we want, but we want the macro to evaluate it to 4 before returning it.
It can of course be done with eval:
(defmacro nth-expr%% (n &rest es)
`(eval (nth ,n ',es)))
CL-USER> (nth-expr%% (1+ i) (+ 2 3) (- 4 3) (+ 3 1))
4
But is there another way?
It feels like the solution should be to put the body of nth-expr% in a helper macro and have the top level macro only contain an unquoted call to this helper:
(defmacro helper (n es)
`(nth ,n ',es))
(defmacro nth-expr (n &rest es) ; doesn't work!
(helper n es))
The idea is that the call to helper would return (+ 3 1), and this would then be the expansion of the call to nth-expr, which at run-time would evaluate to 4. It blows up, of course, because N and ES get treated like literals.
That's not that easy.
Using eval is not good, since eval does not evaluate the code in the local lexical environment.
Remember, if we allow an expression to be evaluated to determine the number of another expression to execute, then we don't know this number at macro expansion time - since the expression could be based on a value that needs to be computed - for example based on some variable:
(nth-expression
foo
(bar)
(baz))
So we might want to think about code which does that:
(case foo
(0 (bar))
(1 (baz)))
CASE is evaluating foo and then uses the result to find a clause which has the same value in its head. The consequent forms of that clause then will be evaluated.
Now we need to write code which expands the former into the latter.
This would be a very simple version:
(defmacro nth-expression (n-form &body expressions)
`(case ,n-form
,#(loop for e in expressions
and i from 0
collect `(,i ,e))))
Question: what might be drawbacks of using CASE like that?
Knuto: Rainer Joswig may be asking you to think about how the case statement works. Namely, that after evaluating the keyform (ie, the first argument), it will be compared sequentially to the key in each clause until a match is found. The comparisons could be time consuming if there are many clauses. You can discover this by carefully reading the entry for case in the Hyperspec (as he more than once has insisted I do):
The keyform or keyplace is evaluated to produce the test-key. Each of
the normal-clauses is then considered in turn.
Also note that constructing many case clauses will add to the time to expand and compile the macro at compile time.
Regarding your use of eval in nth-expr%%, you can still achieve the effect of an eval by switching to apply:
(defmacro nth-expr%% (n &rest es)
`(let ((ne (nth ,n ',es)))
(apply (car ne) (cdr ne))))
But see Plugging the Leaks at http://www.gigamonkeys.com/book/macros-defining-your-own.html about a more robust treatment.
In general, a more efficient way to process the expressions is as a simple vector, rather than a list. (The problem statement does not rule out a vector representation.) While nth and case involve searching through the expressions one-by-one, a function like aref or svref can directly index into it. Assuming a vector of expressions is passed to the macro along with an index, perhaps first requiring (coerce expressions 'simple-vector) if a list, then the result can be computed in constant time no matter how many expressions there are:
(defmacro nth-expr%%% (n es)
`(let ((ne (svref ',es ,n)))
(apply (car ne) (cdr ne))))
so that now
(defvar i 1)
(nth-expr%%% (1+ i) #((+ 2 3) (- 4 3) (+ 3 1))) -> 4
If in the REPL I do this:
(dolist (x (1 2 3))
(print x))
then I get an error since in (1 2 3) the digit 1 is not a symbol or a lambda expr.
If I do:
(dolist (x (list 1 2 3))
(print x))
then it works ok.
My question is why the following works:
REPL> (defmacro test (lst)
(dolist (x lst)
(print x)))
=> TEST
REPL> (test (1 2 3))
1
2
3
=>NIL
Why does dolist accept (1 2 3) when it is inside the macro definition but not when directly in the repl?
The assumption:
"Since TEST is a macro ,it does not evaluate its arguments, so (1 2 3) is passed as is to the dolist macro. So dolist must complain like it does when it is passed (1 2 3) in the REPL"
is obviously wrong. But where?
UPDATE: Although the answers help clarify some misunderstandings with macros, my question still stands and i will try to explain why:
We have established that dolist evaluates its list argument(code blocks 1, 2). Well, it doesnt seem to be the case when it is called inside a macro definition and the list argument that is passed to it is one of the defined macro arguments(code block 3). More details:
A macro, when called, does not evaluate its arguments. So my test macro, when it is called, will preserve the list argument and will pass it as it is to the dolist at expansion time. Then at expansion time the dolist will be executed (no backquotes in my test macro def). And it will be executed with (1 2 3) as argument since this is what the test macro call passed to it. So why doesnt it throw an error since dolist tries to evaluate its list argument, and in this case its list argument (1 2 3) is not evaluatable. I hope this clears my confusion a bit.
This form:
(defmacro test (lst)
(dolist (x lst)
(print x)))
defines a macro, which is a "code transformation function" which
gets applied to a form using this macro at macro expansion time. So,
after you have defined this macro, when you evaluate this expression:
(test (1 2 3))
it first gets read to this list:
(test (1 2 3))
Then, since Lisp reads test at the operator position, it gets
macro-expanded by passing the argument, which is the literal list (1
2 3), to the macro expansion function defined above. This means that
the following gets evaluated at macro-expansion time:
(dolist (x '(1 2 3))
(print x))
So, at macro-expansion time, the three values get printed. Finally,
the return value of that form is returned as the code to be compiled
and executed. Dolist returns nil here, so this is the code returned:
nil
Nil evaluates to nil, which is returned.
Generally, such a macro is not very useful. See "Practical Common
Lisp" by Peter Seibel or "On Lisp" by Paul Graham for an introduction
to useful macros.
Update: Perhaps it is useful to recapitulate the order of
reading, expanding, and evaluating Lisp code.
First, the REPL takes in a stream of characters: ( t e s t
( 1 2 3 ) ), which it assembles into
tokens: ( test ( 1 2 3 ) ).
Then, this is translated into a tree of symbols: (test (1 2
3)). There may be so-called reader macros involved in this step.
For example, 'x is translated to (quote x).
Then, from the outside in, each symbol in operator position (i.e., the
first position in a form) is examined. If it names a macro, then the
corresponding macro function is invoked with the code (i.e., the
subtrees of symbols) that is the rest of the form as arguments. The
macro function is supposed to return a new form, i.e. code, which
replaces the macro form. In your case, the macro test gets the code
(1 2 3) as argument, prints each of the symbols contained within
(note that this is even before compile time), and returns nil,
throwing its arguments away (the compiler never even sees your little
list). The returned code is then examined again for possible
macroexpansions.
Finally, the expanded code that does not contain any macro invocations
anymore is evaluated, i.e. compiled and executed. Nil happens to
be a self-evaluating symbol; it evaluates to nil.
This is just a rough sketch, but I hope that it clears some things up.
Your macro test does not return any code. And yes, macro arguments are not evaluated. If you want to see the same error, you have to define your macro as:
(defmacro test (lst)
`(dolist (x ,lst)
(print x)))
Generally when you have questions about macro expansions, referring to MACROEXPAND-1 is a great first step.
* (macroexpand-1 '(test (1 2 3)))
1
2
3
NIL
T
IE, what is happening is that the actual expansion is that sequence of prints.
Nil is what is returned by DOLIST, and is the expanded code.
Macros get their arguments passed unevaluated. They may choose to evaluate them. dolist does that for its list argument. It works with an unquoted list passed in for lst in your macro test:
(defmacro test (lst)
(dolist (x lst)
(print x)))
That's because at macro-expansion time the dolist sees lst as its argument. So when it evaluates it, it gets the list (1 2 3).
lst is a variable, when expand macro test, it also mean eval dolist structure. the first step is to eval the form lst, will get the lisp object (1 2 3).
like follow example:
(defmacro test (a)
(+ a 2))
(test 2) --> 4 ; mean invoke add function, and the first variable a binding a value 2.