I may be asking for the impossible, but am wondering nonetheless.
Is it possible to obtain an analog of the stack-trace for macros? That is, if one set a break-point inside a certain function, the macro-stack-trace would list all macros (perhaps with their inputs) that were macroexpanded to get to that level in the code.
From what I understand, this is currently impossible, but it may be due to my shallow understanding. Does Allegro or SBCL allow or track this kind of information? It appears that this would be really useful for debugging macros.
Any help or advice is appreciated.
As SBCL is a compiler-only implementation meaning all code is automatically compiled (in contrast to being "interpreted"). Calls to macros are expanded as part of compilation, so the fact that something was a macro call is lost.
(defmacro m (n)
`(/ 10 ,n))
(defun foo (x) (m x))
SBCL:
* (foo 0)
debugger invoked on a DIVISION-BY-ZERO in thread
#<THREAD "main thread" RUNNING {1001E06493}>:
arithmetic error DIVISION-BY-ZERO signalled
Operation was /, operands (10 0).
Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.
restarts (invokable by number or by possibly-abbreviated name):
0: [ABORT] Exit debugger, returning to top level.
(SB-KERNEL::INTEGER-/-INTEGER 10 0)
0] backtrace
Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1001E06493}>
0: (SB-KERNEL::INTEGER-/-INTEGER 10 0)
1: (FOO 0)
2: (SB-INT:SIMPLE-EVAL-IN-LEXENV (FOO 0) #<NULL-LEXENV>)
3: (EVAL (FOO 0))
4: (INTERACTIVE-EVAL (FOO 0) :EVAL NIL)
[...]
Some implementations, e.g. Allegro CL, support both interpreted as well as compiled code, the first being helpful in debugging, the second giving better performance. (I show here the command-line interactions. Allegro also offers a GUI to set breakpoints that I'm not familiar with.)
cl-user(4): (foo 0)
Error: Attempt to divide 10 by zero.
[condition type: division-by-zero]
Restart actions (select using :continue):
0: Return to Top Level (an "abort" restart).
1: Abort entirely from this (lisp) process.
[1] cl-user(5): :zoom
Evaluation stack:
(error division-by-zero :operation ...)
->(/ 10 0)
(foo 0)
(eval (foo 0))
[...]
The zoom command takes many options to be more verbose, this shows the form (block foo (m x)):
[1] cl-user(6): :zoom :all t
Evaluation stack:
... 4 more newer frames ...
((:runsys "lisp_apply"))
[... sys::funcall-tramp ]
(excl::error-from-code 17 nil ...)
(sys::..runtime-operation "integer_divide" :unknown-args)
(excl::/_2op 10 0)
->(/ 10 0)
[... excl::eval-as-progn ]
(block foo (m x))
(foo 0)
(sys::..runtime-operation "comp_to_interp" 0)
[... excl::%eval ]
(eval (foo 0))
When you (compile 'foo) the macro calls will be expanded away (like for SBCL) and not show up in backtraces anymore (but Allegro's source-level debugging could help).
In general when it comes to defining macros, to help debugging try to expand into function calls and not big bodies of code. E.g. instead of:
(defmacro with-foo ((var-x var-y thing) &body body)
`(let ((,var-x (..derive from ,thing ..))
(,var-y (..derive from ,thing ..)))
,#body))
I would write it like:
(defmacro with-foo ((var-x var-y thing) &body body)
`(call-with-foo (lambda (,var-x ,var-y) ,#body) ,thing))
(defun call-with-foo (func thing)
(let ((x (..derive from thing ..)
(y (..derive from thing ..))
(funcall func x y)))
so it ends up in the stack trace and is easy to redefine.
See this great post by Kent Pitman:
Incidentally, too, back to CL, you should know that when I write these
WITH-xxx macros, I almost always accompany them with a CALL-WITH-xxx
so that I can do either kind of call. But I find I almost never use
the CALL-WITH-xxx even when I was the one to provide it as an option.
The main reason I write them is not to use them but to make
redefinition easier, since I can redefine the CALL-WITH-xxx without
redefining the macro, and so I don't have to recompile the callers if
the definition changes.
Yes, AllegroCl supports tracing and in general debugging of macros. Quite an effort for not sure how much benefit, but Franz tends to do good things to make CL more viable. Pro tip: there is a an option to turn off what I think they call source-level debugging of macros, and you will want to do that if your code makes heavy use of macros or compilation times can get crazy. Just turn it back on when you think you need the source debugging.
Related
Is this a conforming Common Lisp program?
(handler-bind ((condition (let ((x 0))
(lambda (c)
(declare (ignore c))
(print (incf x))))))
(signal 'condition)
(signal 'condition))
The output with SBCL (2.0.5.37) is:
1
1
The output with ABCL/CCL/ECL is:
1
2
Which behavior is defined by the Common Lisp standard?
Epilog
This was a bug in SBCL, it is now fixed.
It's not exactly clear. The spec says:
Executes forms in a dynamic environment where the indicated handler bindings are in effect.
and then says
If an appropriate type is found, the associated handler is run in a dynamic environment where none of these handler bindings are visible (to avoid recursive errors).
If you interpret "run" meaning to call the function, that suggests that the handler expressions are evaluted once, when the bindings are made. This is the CCL/ABCL/ECL/LispWorks implementation, so state is maintained in the closure.
But SBCL appears to have intepreted "run" as meaning "evaluated and called". So a new closure is created each time the handler is run, and state is lost.
I suspect the intent was the first interpretation, since CL has no other "lazy" bindings.
If you change the code in the question to this:
(let ((handler
(let ((x 0))
(lambda (c)
(declare (ignore c))
(print (incf x))))))
(handler-bind ((condition handler))
(signal 'condition)
(signal 'condition)))
then SBCL behaves in the same way as the other implementations. I think this makes it fairly clear that the interpretation taken by the other implementations is the intended one, and it also provides a practical workaround for what, if that interpretation is in fact correct, is a bug in SBCL.
My prototype program initially defines a number of global parameters, which subsequently affect the course of analysis. But the program body ended up containing numerous tests of these parameters to determine how to proceed with the detailed analysis. For example:
(defparameter *param1* 0)
(ecase *param1*
(0 (foo))
(1 (bar))
(2 (baz)))
But performing all these various tests at run-time is inefficient. Can these tests be effectively moved to compile-time (or handled otherwise), since all of the parameters are known before analysis begins?
My first idea was to build macros for the tests, so only the relevant code is available at run-time:
(defmacro param1-macro ()
(ecase *param1*
(0 `(foo))
(1 `(bar))
(2 `(baz))))
But having calls like (param1-macro) scattered throughout makes the code hard to read and analyze during debugging. The unique decision process for each macro call is then nonlocal. Is there a more transparent way to increase run-time efficiency?
If these parameters are compile-time constants then make them constants (ie define them with defconstant). This should give the compiler a chance to make assumptions about their values at compile time and turn conditionals into unconditional execution.
How well the compiler can do this depends on the compiler of course: in my limited testing at least some CL compilers do this kind of optimisation.
I'd certainly try to do this before a lot of second-guessing-the-compiler-with-macros.
Another thing, of course, is to raise tests (this must have a name, but I'm not sure what it is: I always call it just 'raising'). Turn code like
(dotimes (i big-number)
(case *parameter*
((1) ...)
...))
into
(case *parameter*
((1) (dotimes (i big-number)
...))
...)
Which reduces the number of tests by a factor of big-number. I suspect this also is an optimisation that good compilers can do themselves.
Finally and perhaps most importantly: measure how slow it is. Are the tests really taking a long time? Almost certainly unless you have measured their cost you don't know it: certainly that's what I find whenever I make assumptions like that!
As an alternative to the above here is a really horrible hack which will allow you to use parameters but will wire in their compile-time values, thus causing the compiler to treat them like constants. Note: when I say 'horrible' I also mean 'mostly untested'.
Here is how you can do this:
(in-package :cl-user)
(eval-when (:compile-toplevel :load-toplevel :execute)
(defvar *wiring* t))
(declaim (inline wiring))
(defun wiring (x)
x)
(define-compiler-macro wiring (&whole form x)
(if (and *wiring*
(symbolp x)
(boundp x))
`(quote ,(symbol-value x))
form))
Now, if *wiring* is false at compile time (wiring ...) is a just an identity function, so you can say (wiring *param*) to mean *param*. It's declared inline so it should have zero cost (and, really, zero purpose).
But if *wiring* is true at compile-time, then the compiler macro does a horrible thing: (wiring *param*) will:
check if *param* is a symbol;
if it is, and if it is bound then it will expand to its compile-time value (quoted for safety).
This means, that if, at compile time, *param* is bound to 2, then
(case (wiring *param*)
((2) (+ x 0))
(otherwise (+ x 1)))
is equivalent, at compile time, to
(case 2
((2) (+ x 0))
(otherwise (+ x 1)))
And the compiler can then turn this (with a note in the case of SBCL) into (+ x 0).
This is horrid because ... well, there are very many ways it is horrid. It certainly violates a lot of assumptions about how code should behave. But it's also kind of lovely: you can do this in Lisp, within the language, which I think is amazing.
AND and OR are macros and since macros aren't first class in scheme/racket they cannot be passed as arguments to other functions. A partial solution is to use and-map or or-map. Is it possible to write a function that would take arbitrary macro and turn it into a function so that it can be passed as an argument to another function? Are there any languages that have first class macros?
In general, no. Consider that let is (or could be) implemented as a macro on top of lambda:
(let ((x 1))
(foo x))
could be a macro that expands to
((lambda (x) (foo x)) 1)
Now, what would it look like to convert let to a function? Clearly it is nonsense. What would its inputs be? Its return value?
Many macros will be like this. In fact, any macro that could be routinely turned into a function without losing any functionality is a bad macro! Such a macro should have been a function to begin with.
I agree with #amalloy. If something is written as a macro, it probably does something that functions can't do (e.g., introduce bindings, change evaluation order). So automatically converting arbitrary macro into a function is a really bad idea even if it is possible.
Is it possible to write a function that would take arbitrary macro and turn it into a function so that it can be passed as an argument to another function?
No, but it is somewhat doable to write a macro that would take some macro and turn it into a function.
#lang racket
(require (for-syntax racket/list))
(define-syntax (->proc stx)
(syntax-case stx ()
[(_ mac #:arity arity)
(with-syntax ([(args ...) (generate-temporaries (range (syntax-e #'arity)))])
#'(λ (args ...) (mac args ...)))]))
((->proc and #:arity 2) 42 12)
(apply (->proc and #:arity 2) '(#f 12))
((->proc and #:arity 2) #f (error 'not-short-circuit))
You might also be interested in identifier macro, which allows us to use an identifier as a macro in some context and function in another context. This could be used to create a first class and/or which short-circuits when it's used as a macro, but could be passed as a function value in non-transformer position.
On the topic of first class macro, take a look at https://en.wikipedia.org/wiki/Fexpr. It's known to be a bad idea.
Not in the way you probably expect
To see why, here is a way of thinking about macros: A macro is a function which takes a bit of source code and turns it into another bit of source code: the expansion of the macro. In other words a macro is a function whose domain and range are source code.
Once the source code is fully expanded, then it's fed to either an evaluator or a compiler. Let's assume it's fed to a compiler because it makes the question easier to answer: a compiler itself is simply a function whose domain is source code and whose range is some sequence of instructions for a machine (which may or may not be a real machine) to execute. Those instructions might include things like 'call this function on these arguments'.
So, what you are asking is: can the 'this function' in 'call this function on these arguments' be some kind of macro? Well, yes, it could be, but whatever source code it is going to transform certainly can not be the source code of the program you are executing, because that is gone: all that's left is the sequence of instructions that was the return value of the compiler.
So you might say: OK, let's say we disallow compilers: can we do it now? Well, leaving aside that 'disallowing compilers' is kind of a serious limitation, this was, in fact, something that very old dialects of Lisp sort-of did, using a construct called a FEXPR, as mentioned in another answer. It's important to realise that FEXPRs existed because people had not yet invented macros. Pretty soon, people did invent macros, and although FEXPRs and macros coexisted for a while – mostly because people had written code which used FEXPRs which they wanted to keep running, and because writing macros was a serious pain before things like backquote existed – FEXPRs died out. And they died out because they were semantically horrible: even by the standards of 1960s Lisps they were semantically horrible.
Here's one small example of why FEXPRs are so horrible: Let's say I write this function in a language with FEXPRs:
(define (foo f g x)
(apply f (g x)))
Now: what happens when I call foo? In particular, what happens if f might be a FEXPR?. Well, the answer is that I can't compile foo at all: I have to wait until run-time and make some on-the-fly decision about what to do.
Of course this isn't what these old Lisps with FEXPRs probably did: they would just silently have assumed that f was a normal function (which they would have called an EXPR) and compiled accordingly (and yes, even very old Lisps had compilers). If you passed something which was a FEXPR you just lost: either the thing detected that, or more likely it fall over horribly or gave you some junk answer.
And this kind of horribleness is why macros were invented: macros provide a semantically sane approach to processing Lisp code which allows (eventually, this took a long time to actually happen) minor details like compilation being possible at all, code having reasonable semantics and compiled code having the same semantics as interpreted code. These are features people like in their languages, it turns out.
Incidentally, in both Racket and Common Lisp, macros are explicitly functions. In Racket they are functions which operate on special 'syntax' objects because that's how you get hygiene, but in Common Lisp, which is much less hygienic, they're just functions which operate on CL source code, where the source code is simply made up of lists, symbols &c.
Here's an example of this in Racket:
> (define foo (syntax-rules ()
[(_ x) x]))
> foo
#<procedure:foo>
OK, foo is now just an ordinary function. But it's a function whose domain & range are Racket source code: it expects a syntax object as an argument and returns another one:
> (foo 1)
; ?: bad syntax
; in: 1
; [,bt for context]
This is because 1 is not a syntax object.
> (foo #'(x 1))
#<syntax:readline-input:5:10 1>
> (syntax-e (foo #'(x 1)))
1
And in CL this is even easier to see: Here's a macro definition:
(defmacro foo (form) form)
And now I can get hold of the macro's function and call it on some CL source code:
> (macro-function 'foo)
#<Function foo 4060000B6C>
> (funcall (macro-function 'foo) '(x 1) nil)
1
In both Racket and CL, macros are, in fact, first-class (or, in the case of Racket: almost first-class, I think): they are functions which operate on source code, which itself is first-class: you can write Racket and CL programs which construct and manipulate source code in arbitrary ways: that's what macros are in these languages.
In the case of Racket I have said 'almost first-class', because I can't see a way, in Racket, to retrieve the function which sits behind a macro defined with define-syntax &c.
I've created something like this in Scheme, it's macro that return lambda that use eval to execute the macro:
(define-macro (macron m)
(let ((x (gensym)))
`(lambda (,x)
(eval `(,',m ,#,x)))))
Example usage:
;; normal eval
(define x (map (lambda (x)
(eval `(lambda ,#x)))
'(((x) (display x)) ((y) (+ y y)))))
;; using macron macro
(define x (map (macron lambda)
'(((x) (display x)) ((y) (+ y y)))))
and x in both cases is list of two functions.
another example:
(define-macro (+++ . args)
`(+ ,#args))
((macron +++) '(1 2 3))
The trace macro is very useful for debugging. But it comes to a halt, when used upon any macro. Like if I try to do the following :
CL-USER> (trace push)
Then, it'll give an error saying:
can't use encapsulation to trace anonymous function #<FUNCTION (MACRO-FUNCTION
PUSH) {100053FB9B}>
[Condition of type SIMPLE-ERROR]
Well, that's obvious because the clhs page of trace, clearly defines it upon functions. So, what is the reason for not having any facility for tracing macros in Common Lisp?
Is there any other (unconventional) way to trace macros in Common Lisp?
The Common Lisp standard only mentions tracing of functions. In compiled implementations, macro expansion usually takes place at compile time and thus tracing of macros is usually not supported.
But some Common Lisp implementations can trace macros via a Lisp interpreter (!):
CLISP can trace macros:
[1]> (defmacro foo (a) a)
FOO
[2]> (trace foo)
;; Tracing macro FOO.
(FOO)
[3]> (loop for i below 4 collect (foo i))
1. Trace: (FOO I)
1. Trace: FOO ==> I
1. Trace: (FOO I)
1. Trace: FOO ==> I
1. Trace: (FOO I)
1. Trace: FOO ==> I
1. Trace: (FOO I)
1. Trace: FOO ==> I
(0 1 2 3)
LispWorks is another implementation which supports tracing of macros.
So, what is the reason for not having any facility for tracing macros in Common Lisp?
As mentioned that's the language standard. Beyond the language standard implementations provide all kinds of language extensions in various ways, including the ability of some Lisp interpreters (!) to trace macros.
If the code is already compiled, tracing won't work anyway. Having a Lisp interpreter helps, but implementations are not required to have an interpreter. Lisp interpreter here means an executing engine which works from Lisp code as data.
Using trace on a macro seems a little odd, but it works in CLISP:
(trace push)
(defparameter *stack* '())
(defun push-xy (x y)
(push x *stack*)
(push y *stack*))
; 1. Trace: (push x *stack*)
; 1. Trace: push ==> (setq *stack* (cons x *stack*))
; 1. Trace: (push y *stack*)
; 1. Trace: push ==> (setq *stack* (cons y *stack*))
; ==> push-xy
The standard does not say when it should expand macros so this might happen when functions and lambdas are defined, compiled and sometimes called. Some implementations run the macro twice so you get double the output.
I never use this. I rather use macroexpand-1:
(macroexpand-1 '(push x *stack)))
; ==> (setq *stack (cons x *stack))
; ==> t
If your form returns a new form that uses macros you might want to try macroexpand instead. Its like calling macroexpand-1 over and over until there are no transformation left.
*macroexpand-hook* is the expected method for tracing macros.
http://www.lispworks.com/documentation/HyperSpec/Body/v_mexp_h.htm#STmacroexpand-hookST
This question already has answers here:
What can you do with Lisp macros that you can't do with first-class functions?
(8 answers)
Closed 5 years ago.
In my quest to fully understand the so powerful lisp macros a question came to my mind. I know that a golden rule about macros is the one saying "Never use a macro when a function will do the work".
However reading Chapter 9 - Practical: Building a Unit Test Framework - from the book Practical Common Lisp I was introduced to the below macro whose purpose was to get rid of the duplication of the test case expression, with its attendant risk of mislabeling of results.
;; Function defintion.
(defun report-result (result form)
(format t "~:[FAIL~;pass~] ... ~a~%" result form))
;; Macro Definition
(defmacro check (form)
`(report-result ,form ',form))
OK, I understand its purpose but I could have done it using a function instead of a macro, for instance:
(setf unevaluated.form '(= 2 (+ 2 3)))
(defun my-func (unevaluated.form)
(report-result (eval unevaluated.form) unevaluated.form))
Is this only possible because the given macro is too simple ?
Furthermore, is Lisp Macro System so powerful relatively its opponents due to the code itself - like control structures, functions, etc - is represented as a LIST ?
But if it were a macro you, could have done:
(check (= 2 (+ 2 3)))
With a function, you have to do:
(check '(= 2 (+ 2 3)))
Also, with the macro the (= 2 (+ 2 3)) is actually compiled by the compiler, whereas with the function it's evaluated by the eval function, not necessarily the same thing.
Addenda:
Yes, it's just evaluating the function. Now what that means is dependent upon the implementation. Some can interpret it, others can compile and execute it. But the simple matter is that you don't know from system to system.
The null lexical environment that others are mentioning is also a big deal.
Consider:
(defun add3f (form)
(eval `(+ 3 ,form)))
(demacro add3m (form)
`(+ 3 ,form))
Then observe:
[28]> (add3m (+ 2 3))
8
[29]> (add3f '(+ 2 3))
8
[30]> (let ((x 2)) (add3m (+ x 3)))
8
[31]> (let ((x 2)) (add3f '(+ x 3)))
*** - EVAL: variable X has no value
The following restarts are available:
USE-VALUE :R1 Input a value to be used instead of X.
STORE-VALUE :R2 Input a new value for X.
ABORT :R3 Abort main loop
Break 1 [32]> :a
That's really quite damning for most use cases. Since the eval has no lexical environment, it can not "see" the x from the enclosing let.
The better substitution would be not with eval, which won't perform as expected for all cases (for example, it doesn't have access to the lexical environment), and is also overkill (see here: https://stackoverflow.com/a/2571549/977052), but something using anonymous functions, like this:
(defun check (fn)
(report-result (funcall fn) (function-body fn)))
CL-USER> (check (lambda () (= 2 (+ 2 3))))
By the way, this is how such things are accomplished in Ruby (anonymous functions are called procs there).
But, as you see, it becomes somewhat less elegant (unless you add syntax sugar) and, there's actually a bigger problem: ther's no function-body function in Lisp (although there may be non-standard ways to get at it). Overall, as you see, for this particular task the alternative solutions are substantially worse, although in some cases such approach could work.
In general, though, if you want to do something with the source code of the expressions passed into the macro (and usually this is the primary reason of using macros), functions would not be sufficient.
The report-result function needs both the source code and the result of the execution.
The macro CHECK provides both from a single source form.
If you put a bunch of check forms into the file, they are easily compiled using the usual process of compiling Lisp files. You'll get a compiled version of the checking code.
Using a function and EVAL (better use COMPILE) you would have deferred the source evaluation to a later time. It would also not be clear if it is interpreted or compiled. In case of compilation, you would then later get the compiler's checks.