Is it possible to expand a macro at run time in a compiled lisp executable? I would expect it to be impossible as macro expansion can only happen pre-compilation yet when i call macroexpand in compiled code i get output.
A macro is a function that's normally called automatically during compilation or evaluation, and whose return value is then compiled or evaluated in place of the original expression.
But since it's just a function, there's nothing preventing it from being called explicitly during run time as well, and that's what MACROEXPAND and MACROEXPAND-1 do.
It's roughly equivalent to:
(defun macroexpand-1 (form &optional env)
(if (and (listp form) (car form)) ;; list expression
(let ((macfun (macro-function (car form)))
(if macfun
(funcall macfun form env)
form))
form))
(Note that this definition doesn't handle symbol macros or use *MACROEXPAND-HOOK*, to keep it simple.)
It's possible to use EVAL-WHEN when defining the macro to make the macro definition only available in the compilation environment. If you do that, trying to expand at run time will fail.
In Lisp, the terms "run time" and "compile time" are situations in which a particular piece of code is being processed, not absolutes like in some static languages. If we evaluate (compile nil '(lambda ())), this is the compile function's run-time, but the lambda form's compile time: both times are happening at the same time.
The entire language is available in all situations. When you build a self-contained executable, that image contains not only support for expanding macros, but for compiling code. Your Lisp application can call compile-file to compile Lisp source to object form, and load to load the resulting object code.
The process of removing garbage, and unused functionality from a Lisp application image to make it smaller is called "tree shaking". If you don't want the compiler or macro expander in your application, find out whether/how they can be removed with your implementation's tree shaking support.
Related
In Lisp, a function's arguments are evaluated first before entering the function body. Macro arguments stay not evaluated.
But sometimes, one wants to inject code pieces stored in variables into a macro. This means evaluating the argument for the macro first, and then apply the macro-of-choice on this evaluated result.
One has to resort to
(eval `(macro ,arg))
To achieve this - but eval does not behave correctly in different environments.
The best thing would be, if one could do:
(apply macro (list arg))
or
(funcall macro arg)
But since the macro is not a function this doesn't work.
Is it possible to achieve something like this? - To circumvent that problem oder to make the macro available in the functions namespace?
Or am I missing some other ways to solve such problems?
I came to this question while trying to answer How to produce HTML from a list. but also in Generate TYPECASE with macro in common lisp, Evaluate arguments passed to a macro that generates functions in lisp, and How to convert a list to code/lambda in scheme?. But I always thought while answering them it would be good to have an apply or funcall-like function which can take macros.
It is not clear what you are trying to do, although it is almost certain that you are confused about something. In particular if you are calling eval inside macroexpansions then in almost all cases you are doing something both seriously wrong and seriously dangerous. I can't ever think of a case where I've wanted macros which expand to things including eval and I have written Lisp for a very very long time.
That being said, here is how you call the function associated with a macro, and why it is very seldom what you want to do.
Macros are simply functions whose domain and range is source code: they are compilers from a language to another language. It is perfectly possible to call the function associated with a macro, but what that function will return is source code, and what you will then need to do with that source code is evaluate it. If you want a function which deals with run-time data which is not source code, then you need that function, and you can't turn a macro into that function by some magic trick which seems to be what you want to do: that magic trick does not, and can not, exist.
So for instance if I have a macro
(defmacro with-x (&body forms)
`(let ((x 1))
,#forms))
Then I can call its macro function on a bit of source code:
> (funcall (macro-function 'with-x)
'(with-x (print "foo")) nil)
(let ((x 1)) (print "foo"))
But the result of this is another bit of source code: I need to compile or evaluate it, and nothing I can do will get around this.
Indeed in (almost?) all cases this is just the same as macroexpand-1):
> (macroexpand-1 '(with-x (print "foo")))
(let ((x 1)) (print "foo"))
t
And you can probably write macroexpand-1 in terms of macro-function:
(defun macroexpand-1/equivalent (form &optional (env nil))
(if (and (consp form)
(symbolp (first form))
(macro-function (first form)))
(values (funcall (macro-function (first form)) form env)
t)
(values form nil)))
So, if the result of calling a macro is source code, what do you do with that source code to get a result which is not source code? Well, you must evaluate it. And then, well, since the evaluator expands macros for you anyway, you might as well just write something like
(defun evaluate-with-x (code)
(funcall (compile nil `(lambda ()
(with-x ,#code)))))
So you didn't need to call the macro's function in any case. And this is not the magic trick which turns macros into functions dealing with data which is not source code: it is a terrible horror which is entirely made of exploding parts.
A concrete example: CL-WHO
It looks like this question might have its origins in this one and the underlying problem there is that that's not what CL-WHO does. In particular it is a confusion to think that something like CL-WHO is a tool for taking some kind of list and turning it into HTML. It's not: it's a tool for taking the source code of a language which is built on CL but includes a way of expressing HTML output mingled with CL code, and compiles it into CL code which will do the same thing. It happens to be the case that CL source code is expressed as lists & symbols, but CL-WHO isn't really about that: it's a compiler from, if you like, 'the CL-WHO language' to CL.
So, let's try the trick we tried above and see why it's a disaster:
(defun form->html/insane (form)
(funcall
(compile nil `(lambda ()
(with-html-output-to-string (,(make-symbol "O"))
,#form)))))
And you might, if you did not look at this too closely, think that this function does in fact do the magic trick:
> (form->html/insane '(:p ((:a :href "foo") "the foo")))
"<p></p><a href='foo'>the foo</a>"
But it doesn't. What happens if we call form->html/insane on this perfectly innocuous list:
(:p (uiop/run-program:run-program "rm -rf $HOME" :output t))
Hint: don't call form->html/insane on this list if you don't have very good backups.
CL-WHO is an implementation of a programming language which is a strict superset of CL: if you try to turn it into a function to turn lists into HTML you end up with something involving the same nuclear weapon you tinker with every time you call eval, except that nuclear weapon is hidden inside a locked cupboard where you can't see it. But it doesn't care about that: if you set it off it will still reduce everything within a few miles to radioactive ash and rubble.
So if you want a tool which will turn lists – lists which aren't source code – into HTML then write that tool. CL-WHO might have the guts of such a tool in its implemenentation, but you can't use it as it is.
And this is the same problem you face whenever you are trying to abuse macros this way: the result of calling a macro's function is Lisp source code, and to evaluate that source code you need eval or an equivalent of eval. And eval is not only not a terrible solution to almost any problem: it's also a nuclear weapon. There are, perhaps problems for which nuclear weapons are good solutions, but they are few and far between.
While reading through Paul Graham's Essays, I've become more and more curious about Lisp.
In this article, he mentions that one of the most powerful features is that you can write programs that write other programs.
I couldn't find an intuitive explanation on his site or elsewhere. Is there some minimal Lisp program that shows an example of how this is done? Or, can you explain in words what this means exactly?
Lisp is homoiconic. Here is a function which build an s-expression representing a sum.
(defun makes(x) (list '+ x 2))
so (makes 5) evaluates to (+ 5 2) which is a valid s-expression. You could pass that to eval
There are more complex examples with Lisp macros. See also this. Read the section on Evaluation and Compilation of Common Lisp HyperSpec (also notice its compile, defmacro, eval forms). Be aware of multi-staged programming.
I strongly recommend reading SICP (it is freely downloadable) then Lisp In Small Pieces. You could also enjoy reading Gödel, Escher, Bach.... and J.Pitrat's blog on Bootstrapping Artificial Intelligence.
BTW, with C on POSIX, you might also code programs generating C code (or use GCCJIT or LLVM), compiling that generated code as a plugin, and dlopen-ing it.
While homoiconicity is the fundamental property that makes this easy, a good example of this in practice is the macro facility present in many lisps. Homoiconicity allows you to write lisp functions that take lisp source (represented as lists of lists) and do list manipulation operations on it to produce other lisp source. A macro is a plain lisp function for doing this which is installed into the compiler/evaluator of your lisp as an extension of the language's syntax. The macro gets called like a normal function, but instead of waiting until runtime the compiler passes the raw code of the macro's arguments to it. The macro is then responsible for returning some alternative code for the compiler to process in its place.
A simple example is the built-in when macro, used like so (assuming some variable x):
(when (evenp x)
(print "It's even!")
(* 5 x))
when is similar to the more fundamental if, but where if takes 3 sub-expressions (test, then-case, else-case) when takes the test and then an arbitrary number of expressions to run in the "then" case (it returns nil in the else case). To write this using if you need an explicit block (a progn in Common Lisp):
(if (evenp x)
(progn
(print "It's even!")
(* 5 x))
nil)
Translating the when version to the if version is some very simple list-manipluation:
(defun when->if (when-expression)
(list 'if
(second when-expression)
(append (list 'progn)
(rest (rest when-expression)))))
Although I'd probably use the list templating syntax and some shorter functions to get this:
(defun when->if (when-expression)
`(if ,(second when-expression) (progn ,#(cddr when-expression)) nil))
This gets called like so: (when->if (list 'when (list 'evenp 'x) ...)).
Now all we need to do is inform the compiler that when it sees an expression like (when ...) (actually I'm writing one for (my-when ...) to avoid clashing with the built-in version) it should use something like our when->if to turn it into code it understands. The actual macro syntax for this actually lets you take apart the expression/list ("destructure" it) as part of the arguments of the macro, so it ends up looking like this:
(defmacro my-when (test &body then-case-expressions)
`(if ,test (progn ,#then-case-expressions) nil))
Looks sorta like a regular function, except it's taking code and outputting other code. Now we can write (my-when (evenp x) ...) and everything works.
The lisp macro facility forms a major component of the expressive power of lisps- they allow you to mold the language to better suit your project and abstract away nearly any boilerplate. Macros can be as simple as when or complex enough to make a third-party OOP library feel like a first-class part of the language (in fact many lisps still implement OOP as a pure lisp library as opposed to a special component of the core compiler, not that you can tell from using them).
A good example are Lisp macros. They aren't evaluated, but instead they transform to the expressions within them. That is what makes them essentially programs that write program. They transform the expressions within them between compile-time and runtime. This means that you can essentially create your own syntax since a macro isn't actually evaluated. A good example would be this invalid common lisp form:
(backwards ("Hello world" nil format))
Clearly the syntax for the format function is backwards. BUT... we are passing it to a macro which isn't evaluated, so we will not get a backtrace error, because the macro isn't actually evaluated. Here is what our macro looks like:
(defmacro backwards (expr)
(reverse expr))
As you can see, we reverse the expression within the macro, which is why it becomes a standard Lisp form between compile-time and runtime. We have essentially altered the syntax of Lisp with a simple example. The call to the macro isn't evaluated, but is translated. A more complex example would be creating a web page in html:
(defmacro standard-page ((&key title href)&body body)
`(with-html-output-to-string (*standard-output* nil :prologue t :indent t)
(:html :lang "en"
(:head
(:meta :charset "utf-8")
(:title ,title)
(:link :rel "stylesheet"
:type "text/css"
:href ,href))
,#body)))
We can essentially create a macro, and the call to that macro will not be evaluated, but it will expand to valid lisp syntax, and that will be evaluated. If we look at the macro expansion we can see that the expansion is what is evaluated:
(pprint (macroexpand-1 '(standard-page (:title "Hello"
:href "my-styles.css")
(:h1 "Hello world"))))
Which expands to:
(WITH-HTML-OUTPUT-TO-STRING (*STANDARD-OUTPUT* NIL :PROLOGUE T :INDENT T)
(:HTML :LANG "en"
(:HEAD (:META :CHARSET "utf-8") (:TITLE "Hello")
(:LINK :REL "stylesheet" :TYPE "text/css" :HREF "my-styles.css"))
(:H1 "Hello world")))
This is why Paul Graham mentions that you can essentially write programs that write programs, and ViaWeb was essentially one big macro. A bunch of macros like this writing code that could write code that could write code...
I have sample code like this:
#!/usr/bin/guile -s
!#
(define (process body)
(list 'list (map (lambda (lst)
(list 'quote (car lst)))
body)))
(defmacro macro (body)
(list 'quote (process body)))
(display (macro ((foo bar) (bar baz))))
(newline)
it run but I've got error from compiler
ERROR: Unbound variable: process
functions inside macros should be allowed, why I got this error?
Functions inside macros are allowed in Guile and in most other Scheme dialects.
However, the crucial question is: Which functions are available for a macro to call out during the expansion process?
Think of it this way: When the compiler is processing your code, it is first focused on turning your source code into something that can be run at some point in the future. But the compiler might not necessarily be able to execute those same functions right now while it is compiling them, at the same time that your macro is running and expanding the source code in general.
Why wouldn't such a function be available? Well, one example would be: What if the function's body used the macro you are in the midst of defining? Then you would have a little chicken/egg problem. The function would need to run the macro to be compiled (since the macro use in the body needs to be expanded, at compile-time) ... But the macro would need the compiled function available in order to even run!
(Furthermore, there might be some functions that you only want to be available at compile-time, as a helper for your macros, but that you do not want to be available at run-time, so that it will not be included in your program executable when you deploy it, as that would waste space in the deployed binary.)
One of my favorite papers describing this problem, and the particular solution adopted by MzScheme (now known as Racket), is the "You Want It When" paper by Matthew Flatt.
So, this is a problem that any Scheme dialect with a procedural macro system has to deal with in some way, and Guile is no exception.
In Guile's case, one fix that is directly documented in the Guile manual is to use the eval-when special form, which allows you to specify at what phases a particular definition is meant to be available.
(The "You Want It When" paper referenced above describes some problems with eval-when, but since it is what the Guile manual documents, I'm going to stick with it for now. I do recommend that after you understand eval-when, that you then look into Racket's solution, and see if Guile offers anything similar.)
So in your case, since you want the process function to be available at compile-time (for use in the macro definition), you could write:
#!/usr/bin/guile -s
!#
(eval-when (expand)
(define (process body)
(list 'list (map (lambda (lst)
(list 'quote (car lst)))
body))))
(defmacro macro (body)
(list 'quote (process body)))
(display (macro ((foo bar) (bar baz))))
(newline)
I'm trying to move from Common Lisp to Chicken Scheme, and having plenty of problems.
My current problem is this: How can I write a macro (presumably using define-syntax?) that calls other macros?
For example, in Common Lisp I could do something like this:
(defmacro append-to (var value)
`(setf ,var (append ,var ,value)))
(defmacro something-else ()
(let ((values (list))
(append-to values '(1)))))
Whereas in Scheme, the equivalent code doesn't work:
(define-syntax append-to
(syntax-rules ()
((_ var value)
(set! var (append var value)))))
(define-syntax something-else
(syntax-rules ()
((_)
(let ((values (list)))
(append-to values '(1))))))
The append-to macro cannot be called from the something-else macro. I get an error saying the append-to "variable" is undefined.
According to all the information I've managed to glean from Google and other sources, macros are evaluated in a closed environment without access to other code. Essentially, nothing else exists - except built-in Scheme functions and macros - when the macro is evaluated. I have tried using er-macro-transformer, syntax-case (which is now deprecated in Chicken anyway) and even the procedural-macros module.
Surely the entire purpose of macros is that they are built upon other macros, to avoid repeating code. If macros must be written in isolation, they're pretty much useless, to my mind.
I have investigated other Scheme implementations, and had no more luck. Seems it simply cannot be done.
Can someone help me with this, please?
It looks like you're confusing expansion-time with run-time. The syntax-rules example you give will expand to the let+set, which means the append will happen at runtime.
syntax-rules simply rewrites input to given output, expanding macros until there's nothing more to expand. If you want to actually perform some computation at expansion time, the only way to do that is with a procedural macro (this is also what happens in your defmacro CL example).
In Scheme, evaluation levels are strictly separated (this makes separate compilation possible), so a procedure can use macros, but the macros themselves can't use the procedures (or macros) defined in the same piece of code. You can load procedures and macros from a module for use in procedural macros by using use-for-syntax. There's limited support for defining things to run at syntax expansion time by wrapping them in begin-for-syntax.
See for example this SO question or this discussion on the ikarus-users mailing list. Matthew Flatt's paper composable and compilable macros explains the theory behind this in more detail.
The "phase separation" thinking is relatively new in the Scheme world (note that the Flatt paper is from 2002), so you'll find quite a few people in the Scheme community who are still a bit confused about it. The reason it's "new" (even though Scheme has had macros for a long long time) is that procedural macros have only become part of the standard since R6RS (and reverted in R7RS because syntax-case is rather controversial), so the need to rigidly specify them hasn't been an issue until now. For more "traditional" Lispy implementations of Scheme, where compile-time and run-time are all mashed together, this was never an issue; you can just run code whenever.
To get back to your example, it works fine if you separate the phases correctly:
(begin-for-syntax
(define-syntax append-to
(ir-macro-transformer
(lambda (e i c)
(let ((var (cadr e))
(val (caddr e)))
`(set! ,var (append ,var ,val)))))) )
(define-syntax something-else
(ir-macro-transformer
(lambda (e i c)
(let ((vals (list 'print)))
(append-to vals '(1))
vals))))
(something-else) ; Expands to (print 1)
If you put the definition of append-to in a module of its own, and you use-for-syntax it, that should work as well. This will also allow you to use the same module both in the macros you define in a body of code as well as in the procedures, by simply requiring it both in a use and a use-for-syntax expression.
What is the best practice for selectively passing evaluated arguments to a macro form?
To elaborate: The usefulness of macros lies in its ability to receives unevaluated parameter, unlike the default evaluation rule for function forms. However, there is a legitimate use cases for evaluating macro arguments.
Consider a contrived example:
(defparameter *func-body* '((print i) (+ i 1)))
Suppose it would be nice that *func-body* could serve as the body of a macro our-defun that is defined as:
(defmacro our-defun (fun args &body body)
`(defun ,fun ,args ,#body))
So after (our-defun foo (i) (+ 1 i)), we could say (foo 1) to get 2. However, if we use (our-defun foo (i) *func-body*), the result of (foo 1) will be ((PRINT I) (+ I 1)) (i.e., the value of *func-body*). It would be nice if we can force the evaluation of *func-body* as an argument to the macro our-defun.
Currently, I can think of a technique of using compile and funcall to do this, as in
(funcall (compile nil `(lambda () (our-defun foo (i) ,#*func-body*))))
after which (our-defun 1) will print out 1 and return 2, as intended. I can think of case of making this work with eval, but I would rather stay away from eval due to its peculiarity in scoping.
This leads to my question at the begining, is there a more straightforward or native way to do this?
P.S.,
A not-so-contrived example is in the function (UPDATE-HOOK), which uses two library macros (ADD-HOOK) and (REMOVE-HOOK) and needs to evaluate its parameters. The (funcall (compile nil `(lambda () ...))) technique above is used here.
(defun update-hook (hook hook-name &optional code)
(funcall (compile nil `(lambda () (remove-hook ,hook ',hook-name))))
(unless (null code)
(compile hook-name `(lambda () ,#code))
(funcall (compile nil `(lambda () (add-hook ,hook ',hook-name))))))
That's slightly confused. A macro does not receive unevaluated parameters.
A macro gets source code and creates source code from that. Remember also that source code in Lisp is actually provided as data. The macro creates code, which evaluates some forms and some not.
Macros need to work in a compiling system. Before runtime. During compile time. All the macro sees is source code and then it creates source code from that. Think of macros as code transformations, not about evaluating arguments or not.
It would be nice if we can force the evaluation of *func-body* as an argument to the macro our-defun
That is not very clean. In a compiled system, you would need to make sure that *func-body* actually has a useful binding and that it can be resolved at COMPILE TIME.
If you have a macro like DEFUN, it makes sense to have the source code static. If you want to insert some source code into a form, then it could make sense to do that at read time:
(defun foo (i) #.`(,#*foo*))
But that's code I usually would want to avoid.
two library macros (ADD-HOOK) and (REMOVE-HOOK) and needs to evaluate its parameters.
Why should ADD-HOOK and REMOVE-HOOK be macros? If you don't have a real reason, they simply should be functions. Already since they make reuse difficult.
If you want to make ADD-HOOK and REMOVE-HOOK macros for some reason, then UPDATE-HOOK usually should be a macro, too.
The list you are giving to your macro has the form
(Quote (...))
So the list you actually want is the CADR of the list you get.