I created a function that returns a function in Elisp:
(defun singleton-set (elem)
(defun f (n) (= n elem))
f)
I try to run this in IELM, and it fails:
ELISP> (singleton-set 5)
*** Eval error *** Symbol's value as variable is void: f
ELISP> ((singleton-set 5) 5)
*** Eval error *** Invalid function: (singleton-set 5)
Due to What is the difference between Lisp-1 and Lisp-2? i changed the code to
(defun singleton-set (elem)
(defun f (n) (= n elem))
#'f)
And invocation to (funcall (singleton-set 5) 5), but now the error is
*** Eval error *** Symbol's value as variable is void: elem
I understand from elisp: capturing variable from inner function that this is due to dynamic binding of Emacs Lisp.
How to make functions returning functions possible in Emacs Lisp? What is the reason this mechanism is different from other languages like Python, Scala or Clojure?
Related questions:
elisp functions as parameters and as return value
Elisp interactive function name
How to Create a Temporary Function in Emacs Lisp
In elisp, how do I put a function in a variable?
From the NEWS for Emacs 24:
Lisp changes in Emacs 24.1
Code can now use lexical scoping by default instead of dynamic scoping.
The lexical-binding variable enables lexical scoping for local
variables. It is typically set via a file-local variable in the first
line of the file, in which case it applies to all the code in that
file.
So, in Emacs 24:
(setq lexical-binding t)
(defun singleton-set (elem) (lambda (n) (= n elem)))
(mapcar (singleton-set 1) '(0 1 2 3))
===> (nil t nil nil)
How to make functions returning functions possible in Emacs Lisp?
Using fake closures, and lexical-let.
What is the reason this mechanism is different from other languages like Python, Scala or Clojure?
Richard Stallman answered this question in a paper he wrote a while ago.
(defun singleton-set (elem)
`(lambda (n) (= n ,elem))
See: elisp functions as parameters and as return value
Related
In this post, I ask tangentially why when I declare in SBCL
(defun a (&rest x)
x)
and then check what the function cell holds
(describe 'a)
COMMON-LISP-USER::A
[symbol]
A names a compiled function:
Lambda-list: (&REST X)
Derived type: (FUNCTION * (VALUES LIST &OPTIONAL))
Source form:
(LAMBDA (&REST X) (BLOCK A X))
I see this particular breakdown of the original function. Could someone explain what this output means? I'm especially confused by the last line
Source form:
(LAMBDA (&REST X) (BLOCK A X))
This is mysterious because for some reason not clear to me Lisp has transformed the original function into a lambda expression. It would also be nice to know the details of how a function broken down like this is then called. This example is SBCL. In Elisp
(symbol-function 'a)
gives
(lambda (&rest x) x)
again, bizarre. As I said in the other post, this is easier to understand in Scheme -- but that created confusion in the answers. So once more I ask, Why has Lisp taken a normal function declaration and seemingly stored it as a lambda expression?
I'm still a bit unclear what you are confused about, but here is an attempt to explain it. I will stick to CL (and mostly to ANSI CL), because elisp has a lot of historical oddities which just make things hard to understand (there is an appendix on elisp). Pre-ANSI CL was also a lot less clear on various things.
I'll try to explain things by writing a macro which is a simple version of defun: I'll call this defun/simple, and an example of its use will be
(defun/simple foo (x)
(+ x x))
So what I need to do is to work out what the expansion of this macro should be, so that it does something broadly equivalent (but simpler than) defun.
The function namespace & fdefinition
First of all I assume you are comfortable with the idea that, in CL (and elisp) the namespace of functions is different than the namespace of variable bindings: both languages are lisp-2s. So in a form like (f x), f is looked up in the namespace of function bindings, while x is looked up in the namespace of variable bindings. This means that forms like
(let ((sin 0.0))
(sin sin))
are fine in CL or elisp, while in Scheme they would be an error, as 0.0 is not a function, because Scheme is a lisp-1.
So we need some way of accessing that namespace, and in CL the most general way of doing that is fdefinition: (fdefinition <function name>) gets the function definition of <function name>, where <function name> is something which names a function, which for our purposes will be a symbol.
fdefinition is what CL calls an accessor: this means that the setf macro knows what to do with it, so that we can mutate the function binding of a symbol by (setf (fdefinition ...) ...). (This is not true: what we can access and mutate with fdefinition is the top-level function binding of a symbol, we can't access or mutate lexical function bindings, and CL provides no way to do this, but this does not matter here.)
So this tells us what our macro expansion needs to look like: we want to set the (top-level) definition of the name to some function object. The expansion of the macro should be like this:
(defun/simple foo (x)
x)
should expand to something involving
(setf (fdefinition 'foo) <form which makes a function>)
So we can write this bit of the macro now:
(defmacro defun/simple (name arglist &body forms)
`(progn
(setf (fdefinition ',name)
,(make-function-form name arglist forms))
',name))
This is the complete definition of this macro. It uses progn in its expansion so that the result of expanding it is the name of the function being defined, which is the same as defun: the expansion does all its real work by side-effect.
But defun/simple relies on a helper function, called make-function-form, which I haven't defined yet, so you can't actually use it yet.
Function forms
So now we need to write make-function-form. This function is called at macroexpansion time: it's job is not to make a function: it's to return a bit of source code which will make a function, which I'm calling a 'function form'.
So, what do function forms look like in CL? Well, there's really only one such form in portable CL (this might be wrong, but I think it is true), which is a form constructed using the special operator function. So we're going to need to return some form which looks like (function ...). Well, what can ... be? There are two cases for function.
(function <name>) denotes the function named by <name> in the current lexical environment. So (function car) is the function we call when we say (car x).
(function (lambda ...)) denotes a function specified by (lambda ...): a lambda expression.
The second of these is the only (caveats as above) way we can construct a form which denotes a new function. So make-function-form is going to need to return this second variety of function form.
So we can write an initial version of make-function-form:
(defun make-function-form (name arglist forms)
(declare (ignore name))
`(function (lambda ,arglist ,#forms)))
And this is enough for defun/simple to work:
> (defun/simple plus/2 (a b)
(+ a b))
plus/2
> (plus/2 1 2)
3
But it's not quite right yet: one of the things that functions defined by defun can do is return from themselves: they know their own name and can use return-from to return from it:
> (defun silly (x)
(return-from silly 3)
(explode-the-world x))
silly
> (silly 'yes)
3
defun/simple can't do this, yet. To do this, make-function-form needs to insert a suitable block around the body of the function:
(defun make-function-form (name arglist forms)
`(function (lambda ,arglist
(block ,name
,#forms))))
And now:
> (defun/simple silly (x)
(return-from silly 3)
(explode-the-world x))
silly
> (silly 'yes)
3
And all is well.
This is the final definition of defun/simple and its auxiliary function.
Looking at the expansion of defun/simple
We can do this with macroexpand in the usual way:
> (macroexpand '(defun/simple foo (x) x))
(progn
(setf (fdefinition 'foo)
#'(lambda (x)
(block foo
x)))
'foo)
t
The only thing that's confusing here is that, because (function ...) is common in source code, there's syntactic sugar for it which is #'...: this is the same reason that quote has special syntax.
It's worth looking at the macroexpansion of real defun forms: they usually have a bunch of implementation-specific stuff in them, but you can find the same thing there. Here's an example from LW:
> (macroexpand '(defun foo (x) x))
(compiler-let ((dspec::*location* '(:inside (defun foo) :listener)))
(compiler::top-level-form-name (defun foo)
(dspec:install-defun 'foo
(dspec:location)
#'(lambda (x)
(declare (system::source-level
#<eq Hash Table{0} 42101FCD5B>))
(declare (lambda-name foo))
x))))
t
Well, there's a lot of extra stuff in here, and LW obviously has some trick around this (declare (lambda-name ...)) form which lets return-from work without an explicit block. But you can see that basically the same thing is going on.
Conclusion: how you make functions
In conclusion: a macro like defun, or any other function-defining form, needs to expand to a form which, when evaluated, will construct a function. CL offers exactly one such form: (function (lambda ...)): that's how you make functions in CL. So something like defun necessarily has to expand to something like this. (To be precise: any portable version of defun: implementations are somewhat free to do implementation-magic & may do so. However they are not free to add a new special operator.)
What you are seeing when you call describe is that, after SBCL has compiled your function, it's remembered what the source form was, and the source form was exactly the one you would have got from the defun/simple macro given here.
Notes
lambda as a macro
In ANSI CL, lambda is defined as a macro whose expansion is a suitable (function (lambda ...)) form:
> (macroexpand '(lambda (x) x))
#'(lambda (x) x)
t
> (car (macroexpand '(lambda (x) x)))
function
This means that you don't have to write (function (lambda ...)) yourself: you can rely on the macro definition of lambda doing it for you. Historically, lambda wasn't always a macro in CL: I can't find my copy of CLtL1, but I'm pretty certain it was not defined as one there. I'm reasonably sure that the macro definition of lambda arrived so that it was possible to write ISLisp-compatible programs on top of CL. It has to be in the language because lambda is in the CL package and so users can't portably define macros for it (although quite often they did define such a macro, or at least I did). I have not relied on this macro definition above.
defun/simple does not purport to be a proper clone of defun: its only purpose is to show how such a macro can be written. In particular it doesn't deal with declarations properly, I think: they need to be lifted out of the block & are not.
Elisp
Elisp is much more horrible than CL. In particular, in CL there is a well-defined function type, which is disjoint from lists:
> (typep '(lambda ()) 'function)
nil
> (typep '(lambda ()) 'list)
t
> (typep (function (lambda ())) 'function)
t
> (typep (function (lambda ())) 'list)
nil
(Note in particular that (function (lambda ())) is a function, not a list: function is doing its job of making a function.)
In elisp, however, an interpreted function is just a list whose car is lambda (caveat: if lexical binding is on this is not the case: it's then a list whose car is closure). So in elisp (without lexical binding):
ELISP> (function (lambda (x) x))
(lambda (x)
x)
And
ELISP> (defun foo (x) x)
foo
ELISP> (symbol-function 'foo)
(lambda (x)
x)
The elisp intepreter then just interprets this list, in just the way you could yourself. function in elisp is almost the same thing as quote.
But function isn't quite the same as quote in elisp: the byte-compiler knows that, when it comes across a form like (function (lambda ...)) that this is a function form, and it should byte-compile the body. So, we can look at the expansion of defun in elisp:
ELISP> (macroexpand '(defun foo (x) x))
(defalias 'foo
#'(lambda (x)
x))
(It turns out that defalias is the primitive thing now.)
But if I put this definition in a file, which I byte compile and load, then:
ELISP> (symbol-function 'foo)
#[(x)
"\207"
[x]
1]
And you can explore this a bit further: if you put this in a file:
(fset 'foo '(lambda (x) x))
and then byte compile and load that, then
ELISP> (symbol-function 'foo)
(lambda (x)
x)
So the byte compiler didn't do anything with foo because it didn't get the hint that it should. But foo is still a fine function:
ELISP> (foo 1)
1 (#o1, #x1, ?\C-a)
It just isn't compiled. This is also why, if writing elisp code with anonymous functions in it, you should use function (or equivalently #'). (And finally, of course, (function ...) does the right thing if lexical scoping is on.)
Other ways of making functions in CL
Finally, I've said above that function & specifically (function (lambda ...)) is the only primitive way to make new functions in CL. I'm not completely sure that's true, especially given CLOS (almost any CLOS will have some kind of class instances of which are functions but which can be subclassed). But it does not matter: it is a way and that's sufficient.
DEFUN is a defining macro. Macros transform code.
In Common Lisp:
(defun foo (a)
(+ a 42))
Above is a definition form, but it will be transformed by DEFUN into some other code.
The effect is similar to
(setf (symbol-function 'foo)
(lambda (a)
(block foo
(+ a 42))))
Above sets the function cell of the symbol FOO to a function. The BLOCK construct is added by SBCL, since in Common Lisp named functions defined by DEFUN create a BLOCK with the same name as the function name. This block name can then be used by RETURN-FROM to enable a non-local return from a specific function.
Additionally DEFUN does implementation specific things. Implementations also record development information: the source code, the location of the definition, etc.
Scheme has DEFINE:
(define (foo a)
(+ a 10))
This will set FOO to a function object.
I understand that both are used to temporarily change the value of a function. Beyond the fact that cl-flet is a function and cl-letf is a macro, when do you use them?
The bindings can be recursive
If the function definition calls itself by name, which function will be called? (compare cl-flet vs cl-labels behaviour).
Scoping is lexical ... capturing them in closures ...
Read about lexical binding/scope vs dynamic binding/scope.
cl-letf can be used to set dynamically-bound function values, by using a PLACE of (symbol-function 'FUNC) for some FUNC. This is analogous to the deprecated flet.
Any supported PLACE can be specified, though -- cl-letf isn't only for function bindings.
when do you use them?
When you want to temporarily define (or override) a function. The scoping rules you require for any given use-case will determine which option you would use.
(cl-flet ((FUNC ARGLIST BODY...) ...) FORM...)
FUNC is visible only to the code in FORM.
(cl-labels ((FUNC ARGLIST BODY...) ...) FORM...)
FUNC is visible both to the code in FORM and also to the code in FUNC's own BODY.
(cl-letf (((symbol-function 'FUNC) VALUE) ...) BODY...)
FUNC is visible to absolutely everything until BODY has finished being evaluated.
Some (fairly contrived) examples...
In the first example, the temporary function we have defined is recursive -- it calls itself -- and therefore we use cl-labels:
(n.b. this isn't a robust factorial implementation; it's just for demonstration purposes.)
(defun my-factorial (number)
"Show the factorial of the argument."
(interactive "nFactorial of: ")
(cl-labels ((factorial (n) (if (eq n 1)
1
(* n (factorial (1- n))))))
(message "Factorial of %d is %d" number (factorial number))))
If you change cl-labels to cl-flet you will get an error as soon as the inner (factorial (1- n)) is evaluated, because within our temporary function, no function factorial is known.
If you were to then define a global factorial function which unconditionally returns the value 1:
(defun factorial (n) 1)
Then the factorial function defined by cl-flet will see that when it calls factorial, and my-factorial would calculate (* n 1) as the value for any argument n.
When no recursion is needed, cl-flet is fine to use:
(defun my-square (number)
"Show the square of the argument."
(interactive "nSquare of: ")
(cl-flet ((square (n) (* n n)))
(message "Square of %d is %d" number (square number))))
Both cl-labels and cl-flet provide lexically-scoped functions, visible only to the code written within the bodies of those macro calls; and in particular not to the code of any other functions which we may be calling.
If you are defining a helper function such as in the examples above, lexical scoping is probably what you want, as there's a reasonable chance you'll only be calling your helper within the macro body.
If you are trying to temporarily override an existing function, however, there's a pretty fair chance you'll need the functions you're calling to see the override. In such cases you need the override to have dynamic scope.
In the past flet was the way to provide dynamic scope for temporary functions, but flet is now deprecated in favour of using cl-letf with a 'place' of (symbol-function 'FUNC)
In the following examples, the multiplication function is overridden, and the dynamic scope means that my-square and my-factorial will see and use our temporary definition of multiplication.
(defun my-bad-square ()
"Maths gone wrong."
(interactive)
(cl-letf (((symbol-function '*) '+))
(call-interactively 'my-square)))
(defun my-bad-factorial ()
"More maths gone wrong."
(interactive)
(cl-letf (((symbol-function '*)
(lambda (x y) (- x y))))
(call-interactively 'my-factorial)))
Update 2013 May: As of GNU Emacs 24.3.1, (let .. (defun..)) bytecompiles just fine without warning and the bytecompiled code works the same as not-compiled code. Just don't forget to add the file variable lexical-binding: t to the file to be bytecompiled. Workarounds at the end of this question is now not necessary.
Lexical Binding - Emacs Lisp Manual has this paragraph:
Note that functions like symbol-value, boundp, and set only retrieve or modify a variable's dynamic binding (i.e. the contents of its symbol's value cell). Also, the code in the body of a defun or defmacro cannot refer to surrounding lexical variables.
I am not sure if I am getting the meaning of the second sentence right. In the following code which should be run in lexical binding mode, the code in the body of a defun is successfully referring to the lexical binding value of the name n.
(let ((n 0))
(defun my-counter ()
(incf n)))
(my-counter) ;; 1
(my-counter) ;; 2
Is the sentence simply saying that (let .. (defun ..)) is a bad practice?
Workarounds:
;; -*- lexical-binding: t -*-
;; a way to define the counter function without byte-compile error or warning
(defvar my--counter-func
(let ((n 0))
(lambda ()
(setq n (1+ n)))))
(defun my-counter ()
(funcall my--counter-func))
;; another way to define the counter function, again without byte-compile error or warning
(fset 'my-another-counter
(let ((n 0))
(lambda ()
(setq n (1+ n)))))
And here's the code for testing the above code:
;; run:
;; emacs -q --load path-to-the-el-file-of-this-code.el
(load "path-to-file-defining-my-counter.elc") ;; loading the ELC file to test if byte-compiled code runs as expected.
(print (my-counter)) ;; 1
(print (my-counter)) ;; 2
(print (my-another-counter)) ;; 1
(print (my-another-counter)) ;; 2
The code does not byte-compile well at least in Emacs 24.1.1. I saved the following code in the foo.el file, which uses setq in place of incf in order to avoid any possible effects by the cl library:
;; -*- lexical-binding: t -*-
(let ((n 0))
(defun my-counter ()
(setq n (1+ n))))
When I tried to byte-compile it (M-x byte-compile-filefoo.el), I got the following warning messages:
foo.el:3:1:Warning: Function my-counter will ignore its context (n)
foo.el:3:1:Warning: Unused lexical variable `n'
foo.el:5:11:Warning: reference to free variable `n'
foo.el:5:17:Warning: assignment to free variable `n'
All of the messages are indicating that the code in the body of the defun construct cannot refer to the surrounding lexical variable n as the manual claims.
Actually, when I loaded the byte-compiled code (M-x load-filefoo.elc) and evaluted the (my-counter) form, I got the following erorr:
Debugger entered--Lisp error: (void-variable n)
...
Unfortunately, I'm not sure why the code appears to work when evaluated in the form of source code.
As I replied on gnu.emacs.help, you can use (defalias 'foo (lambda ...)) to work around that limitation.
And that limitation is lifted in Emacs's development code.
It's perfectly fine to refer to variables in lexical scope(*) from within a defun, just as you are doing above, and just as the "my-ticker" example on that manual page does as well.
Either I am missing something the line in the manual that says:
the code in the body of a defun or defmacro cannot refer to
surrounding lexical variables.
should say something more like:
code in the body of a defun can only access lexical variables if they are defined within the same lexical scope.
NOTE: There are comments in the other answers about problems byte-compiling this kind of code. Those should be fixed in the latest emacs. I've verified in v24.2.50.1 that this byte compiles and loads correctly.
As suggested in a macro-related question I recently posted to SO, I coded a macro called "fast" via a call to a function (here is the standalone code in pastebin):
(defun main ()
(progn
(format t "~A~%" (+ 1 2 (* 3 4) (+ 5 (- 8 6))))
(format t "~A~%" (fast (+ 1 2 (* 3 4) (+ 5 (- 8 6)))))))
This works in the REPL, under both SBCL and CMUCL:
$ sbcl
This is SBCL 1.0.52, an implementation of ANSI Common Lisp.
...
* (load "bug.cl")
22
22
$
Unfortunately, however, the code no longer compiles:
$ sbcl
This is SBCL 1.0.52, an implementation of ANSI Common Lisp.
...
* (compile-file "bug.cl")
...
; during macroexpansion of (FAST (+ 1 2 ...)). Use *BREAK-ON-SIGNALS* to
; intercept:
;
; The function COMMON-LISP-USER::CLONE is undefined.
So it seems that by having my macro "fast" call functions ("clone","operation-p") at compile-time, I trigger issues in Lisp compilers (verified in both CMUCL and SBCL).
Any ideas on what I am doing wrong and/or how to fix this?
Some remarks about your code.
multiple tests of an object for equality can be replaced by MEMBER
backquote with a following comma does nothing. You can just remove that.
you can ensure that your functions are available for a macro by a) moving these functions to an additional file and compile/load that before use of the macro, by b) using EVAL-WHENto inform the compiler to evaluate the definition of the functions or by c) adding the functions to the macro as local functions
Example:
(defmacro fast (&rest sexpr)
(labels ((operation-p (x)
(member x '(+ - * /)))
(clone (sexpr)
(if (consp sexpr)
(destructuring-bind (head . tail) sexpr
(if (operation-p head)
`(the fixnum (,head ,#(clone tail)))
(cons (clone head) (clone tail))))
sexpr)))
(car (clone sexpr))))
Note that this and your version of FAST are not complete code walkers. They recognize only simple function calls (and not the other Lisp constructs like LAMBDA, LET, FLET, LABELS, etc.).
Never mind, I figured it out: I had to move the functions invoked by the macro (and therefore required during compilation) in a separate file, "compile-file" it first, "load" it, then "compile-file" the one with the macro.
Macro-expansion tie happens (typically) during compile time.
That means that any functions used during the macro expansion (note, not necessarily in the macro-expansion, the return value as it were) must be defined when the macro is encountered during compilation.
I have an s-expression bound to a variable in Common Lisp:
(defvar x '(+ a 2))
Now I want to create a function that when called, evaluates the expression in the scope in which it was defined. I've tried this:
(let ((a 4))
(lambda () (eval x)))
and
(let ((a 4))
(eval `(lambda () ,x)))
But both of these create a problem: EVAL will evaluate the code at the top level, so I can't capture variables contained in the expression. Note that I cannot put the LET form in the EVAL. Is there any solution?
EDIT: So if there is not solution to the EVAL problem, how else can it be done?
EDIT: There was a question about what exactly I am try to do. I am writing a compiler. I want to accept an s-expression with variables closed in the lexical environment where the expression is defined. It may indeed be better to write it as a macro.
You need to create code that has the necessary bindings. Wrap a LET around your code and bind every variable you want to make available in your code:
(defvar *x* '(+ a 2))
(let ((a 4))
(eval `(let ((a ,a))
,*x*)))
CLISP implements an extension to evaluate a form in the lexical environment. From the fact that it is an extension, I suspect you can't do that in a standard-compliant way.
(ext:eval-env x (ext:the-environment))
See http://clisp.cons.org/impnotes.html#eval-environ.
What is the actual problem that you want to solve? Most likely, you're trying to tackle it the wrong way. Lexical bindings are for things that appear lexically within their scope, not for random stuff you get from outside.
Maybe you want a dynamic closure? Such a thing doesn't exist in Common Lisp, although it does in some Lisp dialects (like Pico Lisp, as far as I understand).
Note that you can do the following, which is similar:
(defvar *a*)
(defvar *x* '(+ *a* 2)) ;'
(let ((a 10))
;; ...
(let ((*a* a))
(eval *x*)))
I advise you to think hard about whether you really want this, though.
In Common Lisp you can define *evalhook* Which allows you to pass an environment to (eval ...). *evalhook* is platform independent.
It is possible to use COMPILE to compile the expression into function and then use PROGV to FUNCALL the compiled function in the environment where variables are dynamically set. Or, better, use COMPILE to compile the expression into function that accepts variables.
Compile accepts the function definition as a list and turns it into function. In case of SBCL, this function is compiled into machine code and will execute efficiently.
First option (using compile and progv):
(defvar *fn* (compile nil '(lambda () (+ a 2)))
(progv '(a) '(4) (funcall *fn*))
=>
6
Second option:
(defvar *fn* (compile nil '(lambda (a) (+ a 2))))
(funcall *fn* 4)
=>
6