I'd like to learn the internals of Lisp, so I want to see how everything is implemented.
For example,
(macroexpand '(loop for i upto 10 collect i))
gives me (in SBCL)
(BLOCK NIL
(LET ((I 0))
(DECLARE (TYPE (AND NUMBER REAL) I))
(SB-LOOP::WITH-LOOP-LIST-COLLECTION-HEAD (#:LOOP-LIST-HEAD-1026
#:LOOP-LIST-TAIL-1027)
(SB-LOOP::LOOP-BODY NIL
(NIL NIL (WHEN (> I '10) (GO SB-LOOP::END-LOOP)) NIL)
((SB-LOOP::LOOP-COLLECT-RPLACD
(#:LOOP-LIST-HEAD-1026 #:LOOP-LIST-TAIL-1027)
(LIST I)))
(NIL (SB-LOOP::LOOP-REALLY-DESETQ I (1+ I))
(WHEN (> I '10) (GO SB-LOOP::END-LOOP)) NIL)
((RETURN-FROM NIL
(SB-LOOP::LOOP-COLLECT-ANSWER
#:LOOP-LIST-HEAD-1026)))))))
But LOOP-BODY, WITH-LOOP-LIST-COLLECTION-HEAD, etc. are still macros. How can I expand a macro form completely?
To see the full expansion one needs to walk the Lisp form on all levels and expand them. For this it is necessary that this so-called code walker understands Lisp syntax (and not just s-expression syntax). For example in (lambda (a b) (setf a b)), the list (a b) is a parameter list and should not be macro expanded.
Various Common Lisp implementations provide such a tool. The answer of 6502 mentions MACROEXPAND-ALL which is provided by SBCL.
If you use a development environment, it is usually provided as a command:
SLIME: M-x slime-macroexpand-all with C-c M-m
LispWorks: menu Expression > Walk or M-x Walk Form, shorter M-Sh-m.
The other answers are excellent for you question but you say you want to see how everything is implemented.
Many macros (as you know already) are implemented using macros and whilst macroexpand-all is very useful but you can lose the context of what macro was responsible for what change.
One nice middle ground (if you are using slime) is to use slime-expand-1 (C-c Enter) which shows the expansion is another buffer. You can then useslime-expand-1 inside this new buffer to expand macros in-place.
This allows you to walk the tree expanding as you read and also to use undo to close the expansions again.
For me this has been a god-send in understanding other people's macros. Hope this helps you too, have fun!
You can try to use MACROEXPAND-ALL but what you may get is not necessarily useful.
In something like LOOP the real meat is the macro itself, not the generated code.
(Note: If you're not interested in portability, SBCL provides macroexpand-all, which will do what you're after. If you're after a portable solution, read on...)
The quick-and-dirty solution would be to macroexpand the form itself, then recursively macroexpand all but the first element of the resulting list. This is an imperfect solution; it will fail completely the moment it attempts to process a let's bindings (the first argument to let, the list of bindings, is not meant to be macroexpanded, but this code will do it anyway).
;;; Quick-and-dirty macroexpand-all
(defun macroexpand* (form)
(let ((form (macroexpand form)))
(cons (car form) (mapcar #'macroexpand (cdr form)))))
A more complete solution would consider special forms specially, not macroexpanding their unevaluated arguments. I could update with such a solution, if desired.
Related
In Lisp, a function's arguments are evaluated first before entering the function body. Macro arguments stay not evaluated.
But sometimes, one wants to inject code pieces stored in variables into a macro. This means evaluating the argument for the macro first, and then apply the macro-of-choice on this evaluated result.
One has to resort to
(eval `(macro ,arg))
To achieve this - but eval does not behave correctly in different environments.
The best thing would be, if one could do:
(apply macro (list arg))
or
(funcall macro arg)
But since the macro is not a function this doesn't work.
Is it possible to achieve something like this? - To circumvent that problem oder to make the macro available in the functions namespace?
Or am I missing some other ways to solve such problems?
I came to this question while trying to answer How to produce HTML from a list. but also in Generate TYPECASE with macro in common lisp, Evaluate arguments passed to a macro that generates functions in lisp, and How to convert a list to code/lambda in scheme?. But I always thought while answering them it would be good to have an apply or funcall-like function which can take macros.
It is not clear what you are trying to do, although it is almost certain that you are confused about something. In particular if you are calling eval inside macroexpansions then in almost all cases you are doing something both seriously wrong and seriously dangerous. I can't ever think of a case where I've wanted macros which expand to things including eval and I have written Lisp for a very very long time.
That being said, here is how you call the function associated with a macro, and why it is very seldom what you want to do.
Macros are simply functions whose domain and range is source code: they are compilers from a language to another language. It is perfectly possible to call the function associated with a macro, but what that function will return is source code, and what you will then need to do with that source code is evaluate it. If you want a function which deals with run-time data which is not source code, then you need that function, and you can't turn a macro into that function by some magic trick which seems to be what you want to do: that magic trick does not, and can not, exist.
So for instance if I have a macro
(defmacro with-x (&body forms)
`(let ((x 1))
,#forms))
Then I can call its macro function on a bit of source code:
> (funcall (macro-function 'with-x)
'(with-x (print "foo")) nil)
(let ((x 1)) (print "foo"))
But the result of this is another bit of source code: I need to compile or evaluate it, and nothing I can do will get around this.
Indeed in (almost?) all cases this is just the same as macroexpand-1):
> (macroexpand-1 '(with-x (print "foo")))
(let ((x 1)) (print "foo"))
t
And you can probably write macroexpand-1 in terms of macro-function:
(defun macroexpand-1/equivalent (form &optional (env nil))
(if (and (consp form)
(symbolp (first form))
(macro-function (first form)))
(values (funcall (macro-function (first form)) form env)
t)
(values form nil)))
So, if the result of calling a macro is source code, what do you do with that source code to get a result which is not source code? Well, you must evaluate it. And then, well, since the evaluator expands macros for you anyway, you might as well just write something like
(defun evaluate-with-x (code)
(funcall (compile nil `(lambda ()
(with-x ,#code)))))
So you didn't need to call the macro's function in any case. And this is not the magic trick which turns macros into functions dealing with data which is not source code: it is a terrible horror which is entirely made of exploding parts.
A concrete example: CL-WHO
It looks like this question might have its origins in this one and the underlying problem there is that that's not what CL-WHO does. In particular it is a confusion to think that something like CL-WHO is a tool for taking some kind of list and turning it into HTML. It's not: it's a tool for taking the source code of a language which is built on CL but includes a way of expressing HTML output mingled with CL code, and compiles it into CL code which will do the same thing. It happens to be the case that CL source code is expressed as lists & symbols, but CL-WHO isn't really about that: it's a compiler from, if you like, 'the CL-WHO language' to CL.
So, let's try the trick we tried above and see why it's a disaster:
(defun form->html/insane (form)
(funcall
(compile nil `(lambda ()
(with-html-output-to-string (,(make-symbol "O"))
,#form)))))
And you might, if you did not look at this too closely, think that this function does in fact do the magic trick:
> (form->html/insane '(:p ((:a :href "foo") "the foo")))
"<p></p><a href='foo'>the foo</a>"
But it doesn't. What happens if we call form->html/insane on this perfectly innocuous list:
(:p (uiop/run-program:run-program "rm -rf $HOME" :output t))
Hint: don't call form->html/insane on this list if you don't have very good backups.
CL-WHO is an implementation of a programming language which is a strict superset of CL: if you try to turn it into a function to turn lists into HTML you end up with something involving the same nuclear weapon you tinker with every time you call eval, except that nuclear weapon is hidden inside a locked cupboard where you can't see it. But it doesn't care about that: if you set it off it will still reduce everything within a few miles to radioactive ash and rubble.
So if you want a tool which will turn lists – lists which aren't source code – into HTML then write that tool. CL-WHO might have the guts of such a tool in its implemenentation, but you can't use it as it is.
And this is the same problem you face whenever you are trying to abuse macros this way: the result of calling a macro's function is Lisp source code, and to evaluate that source code you need eval or an equivalent of eval. And eval is not only not a terrible solution to almost any problem: it's also a nuclear weapon. There are, perhaps problems for which nuclear weapons are good solutions, but they are few and far between.
In my ongoing quest to recreate lodash in lisp as a way of getting familiar with the language I am trying to write a concat-list function that takes an initial list and an arbitrary number of additional lists and concatenates them.
I'm sure that this is just a measure of getting familiar with lisp convention, but right now my loop is just returning the second list in the argument list, which makes sense since it is the first item of other-lists.
Here's my non-working code (edit: refactored):
(defun concat-list (input-list &rest other-lists)
;; takes an arbitrary number of lists and merges them
(loop
for list in other-lists
append list into input-list
return input-list
)
)
Trying to run (concat-list '(this is list one) '(this is list two) '(this is list three)) and have it return (this is list one this is list two this is list three).
How can I spruce this up to return the final, merged list?
The signature of your function is a bit unfortunate, it becomes easier if you don't treat the first list specially.
The easy way:
(defun concat-lists (&rest lists)
(apply #'concatenate 'list lists))
A bit more lower level, using loop:
(defun concat-lists (&rest lists)
(loop :for list :in lists
:append list))
Going lower, using dolist:
(defun concat-lists (&rest lists)
(let ((result ()))
(dolist (list lists (reverse result))
(setf result (revappend list result)))))
Going even lower would maybe entail implementing revappend yourself.
It's actually good style in Lisp not to use LABELS based iteration, since a) it's basically a go-to like low-level iteration style and it's not everywhere supported. For example the ABCL implementation of Common Lisp on the JVM does not support TCO last I looked. Lisp has wonderful iteration facilities, which make the iteration intention clear:
CL-USER 217 > (defun my-append (&rest lists &aux result)
(dolist (list lists (nreverse result))
(dolist (item list)
(push item result))))
MY-APPEND
CL-USER 218 > (my-append '(1 2 3) '(4 5 6) '(7 8 9))
(1 2 3 4 5 6 7 8 9)
Some pedagogical solutions to this problem
If you just want to do this, then use append, or nconc (destructive), which are the functions which do it.
If you want to learn how do to it, then learning about loop is not how to do that, assuming you want to learn Lisp: (loop for list in ... append list) really teaches you nothing but how to write a crappy version of append using arguably the least-lispy part of CL (note I have nothing against loop & use it a lot, but if you want to learn lisp, learning loop is not how to do that).
Instead why not think about how you would write this if you did not have the tools to do it, in a Lispy way.
Well, here's how you might do that:
(defun append-lists (list &rest more-lists)
(labels ((append-loop (this more results)
(if (null this)
(if (null more)
(nreverse results)
(append-loop (first more) (rest more) results))
(append-loop (rest this) more (cons (first this) results)))))
(append-loop list more-lists '())))
There's a dirty trick here: I know that results is completely fresh so I am using nreverse to reverse it, which does so destructively. Can we write nreverse? Well, it's easy to write reverse, the non-destructive variant:
(defun reverse-nondestructively (list)
(labels ((r-loop (tail reversed)
(if (null tail)
reversed
(r-loop (rest tail) (cons (first tail) reversed)))))
(r-loop list '())))
And it turns out that a destructive reversing function is only a little harder:
(defun reverse-destructively (list)
(labels ((rd-loop (tail reversed)
(if (null tail)
reversed
(let ((rtail (rest tail)))
(setf (rest tail) reversed)
(rd-loop rtail tail)))))
(rd-loop list '())))
And you can check it works:
> (let ((l (make-list 1000 :initial-element 1)))
(time (reverse-destructively l))
(values))
Timing the evaluation of (reverse-destructively l)
User time = 0.000
System time = 0.000
Elapsed time = 0.000
Allocation = 0 bytes
0 Page faults
Why I think this is a good approach to learning Lisp
[This is a response to a couple of comments which I thought was worth adding to the answer: it is, of course, my opinion.]
I think that there are at least three different reasons for wanting to solve a particular problem in a particular language, and the approach you might want to take depends very much on what your reason is.
The first reason is because you want to get something done. In that case you want first of all to find out if it has been done already: if you want to do x and the language a built-in mechanism for doing x then use that. If x is more complicated but there is some standard or optional library which does it then use that. If there's another language you could use easily which does x then use that. Writing a program to solve the problem should be something you do only as a last resort.
The second reason is because you've fallen out of the end of the first reason, and you now find yourself needing to write a program. In that case what you want to do is use all of the tools the language provides in the best way to solve the problem, bearing in mind things like maintainability, performance and so on. In the case of CL, then if you have some problem which naturally involves looping, then, well, use loop if you want to. It doesn't matter whether loop is 'not lispy' or 'impure' or 'hacky': just do what you need to do to get the job done and make the code maintainable. If you want to print some list of objects, then by all means write (format t "~&~{~A~^, ~}~%" things).
The third reason is because you want to learn the language. Well, assuming you can program in some other language there are two approaches to doing this.
the first is to say 'I know how to do this thing (write loops, say) in languages I know – how do I do it in Lisp?', and then iterate this for all the thing you already know how to do in some other language;
the second is to say 'what is it that makes Lisp distinctive?' and try and understand those things.
These approaches result in very approaches to learning. In particular I think the first approach is often terrible: if the language you know is, say, Fortran, then you'll end up writing Fortran dressed up as Lisp. And, well, there are perfectly adequate Fortran compilers out there: why not use them? Even worse, you might completely miss important aspects of the language and end up writing horrors like
(defun sum-list (l)
(loop for i below (length l)
summing (nth i l)))
And you will end up thinking that Lisp is slow and pointless and return to the ranks of the heathen where you will spread such vile calumnies until, come the great day, the golden Lisp horde sweeps it all away. This has happened.
The second approach is to ask, well, what are the things that are interesting about Lisp? If you can program already, I think this is a much better approach to the first, because learning the interesting and distinctive features of a language first will help you understand, as quickly as possible, whether its a language you might actually want to know.
Well, there will inevitably be argument about what the interesting & distinctive features of Lisp are, but here's a possible, partial, set.
The language has a recursively-defined data structure (S expressions or sexprs) at its heart, which is used among other things to represent the source code of the language itself. This representation of the source is extremely low-commitment: there's nothing in the syntax of the language which says 'here's a block' or 'this is a conditiona' or 'this is a loop'. This low-commitment can make the language hard to read, but it has huge advantages.
Recursive processes are therefore inherently important and the language is good at expressing them. Some variants of the language take this to the extreme by noticing that iteration is simply a special case of recursion and have no iterative constructs at all (CL does not do this).
There are symbols, which are used as names for things both in the language itself and in programs written in the language (some variants take this more seriously than others: CL takes it very seriously).
There are macros. This really follows from the source code of the language being represented as sexprs and this structure having a very low commitment to what it means. Macros, in particular, are source-to-source transformations, with the source being represented as sexprs, written in the language itself: the macro language of Lisp is Lisp, without restriction. Macros allow the language itself to be seamlessly extended: solving problems in Lisp is done by designing a language in which the problem can be easily expressed and solved.
The end result of this is, I think two things:
recursion, in addition to and sometimes instead of iteration is an unusually important technique in Lisp;
in Lisp, programming means building a programming language.
So, in the answer above I've tried to give you examples of how you might think about solving problems involving a recursive data structure recursively: by defining a local function (append-loop) which then recursively calls itself to process the lists. As Rainer pointed out that's probably not a good way of solving this problem in Common Lisp as it tends to be hard to read and it also relies on the implementation to turn tail calls into iteration which is not garuanteed in CL. But, if your aim is to learn to think the way Lisp wants you to think, I think it is useful: there's a difference between code you might want to write for production use, and code you might want to read and write for pedagogical purposes: this is pedagogical code.
Indeed, it's worth looking at the other half of how Lisp might want you to think to solve problems like this: by extending the language. Let's say that you were programming in 1960, in a flavour of Lisp which has no iterative constructs other than GO TO. And let's say you wanted to process some list iteratively. Well, you might write this (this is in CL, so it is not very like programming in an ancient Lisp would be: in CL tagbody establishes a lexical environment in the body of which you can have tags – symbols – and then go will go to those tags):
(defun print-list-elements (l)
;; print the elements of a list, in order, using GO
(let* ((tail l)
(current (first tail)))
(tagbody
next
(if (null tail)
(go done)
(progn
(print current)
(setf tail (rest tail)
current (first tail))
(go next)))
done)))
And now:
> (print-list-elements '(1 2 3))
1
2
3
nil
Let's program like it's 1956!
So, well, let's say you don't like writing this sort of horror. Instead you'd like to be able to write something like this:
(defun print-list-elements (l)
;; print the elements of a list, in order, using GO
(do-list (e l)
(print e)))
Now if you were using most other languages you need to spend several weeks mucking around with the compiler to do this. But in Lisp you spend a few minutes writing this:
(defmacro do-list ((v l &optional (result-form nil)) &body forms)
;; Iterate over a list. May be buggy.
(let ((tailn (make-symbol "TAIL"))
(nextn (make-symbol "NEXT"))
(donen (make-symbol "DONE")))
`(let* ((,tailn ,l)
(,v (first ,tailn)))
(tagbody
,nextn
(if (null ,tailn)
(go ,donen)
(progn
,#forms
(setf ,tailn (rest ,tailn)
,v (first ,tailn))
(go ,nextn)))
,donen
,result-form))))
And now your language has an iteration construct which it previously did not have. (In real life this macro is called dolist).
And you can go further: given our do-list macro, let's see how we can collect things into a list:
(defun collect (thing)
;; global version: just signal an error
(declare (ignorable thing))
(error "not collecting"))
(defmacro collecting (&body forms)
;; Within the body of this macro, (collect x) will collect x into a
;; list, which is returned from the macro.
(let ((resultn (make-symbol "RESULT"))
(rtailn (make-symbol "RTAIL")))
`(let ((,resultn '())
(,rtailn nil))
(flet ((collect (thing)
(if ,rtailn
(setf (rest ,rtailn) (list thing)
,rtailn (rest ,rtailn))
(setf ,resultn (list thing)
,rtailn ,resultn))
thing))
,#forms)
,resultn)))
And now we can write the original append-lists function entirely in terms of constructs we've invented:
(defun append-lists (list &rest more-lists)
(collecting
(do-list (e list) (collect e))
(do-list (l more-lists)
(do-list (e l)
(collect e)))))
If that's not cool then nothing is.
In fact we can get even more carried away. My original answer above used labels to do iteration As Rainer has pointed out, this is not safe in CL since CL does not mandate TCO. I don't particularly care about that (I am happy to use only CL implementations which mandate TCO), but I do care about the problem that using labels this way is hard to read. Well, you can, of course, hide this in a macro:
(defmacro looping ((&rest bindings) &body forms)
;; A sort-of special-purpose named-let.
(multiple-value-bind (vars inits)
(loop for b in bindings
for var = (typecase b
(symbol b)
(cons (car b))
(t (error "~A is hopeless" b)))
for init = (etypecase b
(symbol nil)
(cons (unless (null (cddr b))
(error "malformed binding ~A" b))
(second b))
(t
(error "~A is hopeless" b)))
collect var into vars
collect init into inits
finally (return (values vars inits)))
`(labels ((next ,vars
,#forms))
(next ,#inits))))
And now:
(defun append-lists (list &rest more-lists)
(collecting
(looping ((tail list) (more more-lists))
(if (null tail)
(unless (null more)
(next (first more) (rest more)))
(progn
(collect (first tail))
(next (rest tail) more))))))
And, well, I just think it is astonishing that I get to use a programming language where you can do things like this.
Note that both collecting and looping are intentionally 'unhygenic': they introduce a binding (for collect and next respectively) which is visible to code in their bodies and which would shadow any other function definition of that name. That's fine, in fact, since that's their purpose.
This kind of iteration-as-recursion is certainly cool to think about, and as I've said I think it really helps you to think about how the language can work, which is my purpose here. Whether it leads to better code is a completely different question. Indeed there is a famous quote by Guy Steele from one of the 'lambda the ultimate ...' papers:
procedure calls may be usefully thought of as GOTO statements which also pass parameters
And that's a lovely quote, except that it cuts both ways: procedure calls, in a language which optimizes tail calls, are pretty much GOTO, and you can do almost all the horrors with them that you can do with GOTO. But GOTO is a problem, right? Well, it turns out so are procedure calls, for most of the same reasons.
So, pragmatically, even in a language (or implementation) where procedure calls do have all these nice characteristics, you end up wanting constructs which can express iteration and not recursion rather than both. So, for instance, Racket which, being a Scheme-family language, does mandate tail-call elimination, has a whole bunch of macros with names like for which do iteration.
And in Common Lisp, which does not mandate tail-call elimination but which does have GOTO, you also need to build macros to do iteration, in the spirit of my do-list above. And, of course, a bunch of people then get hopelessly carried away and the end point is a macro called loop: loop didn't exist (in its current form) in the first version of CL, and it was common at that time to simply obtain a copy of it from somewhere, and make sure it got loaded into the image. In other words, loop, with all its vast complexity, is just a macro which you can define in a CL which does not have it already.
OK, sorry, this is too long.
(loop for list in (cons '(1 2 3)
'((4 5 6) (7 8 9)))
append list)
I'm trying to move from Common Lisp to Chicken Scheme, and having plenty of problems.
My current problem is this: How can I write a macro (presumably using define-syntax?) that calls other macros?
For example, in Common Lisp I could do something like this:
(defmacro append-to (var value)
`(setf ,var (append ,var ,value)))
(defmacro something-else ()
(let ((values (list))
(append-to values '(1)))))
Whereas in Scheme, the equivalent code doesn't work:
(define-syntax append-to
(syntax-rules ()
((_ var value)
(set! var (append var value)))))
(define-syntax something-else
(syntax-rules ()
((_)
(let ((values (list)))
(append-to values '(1))))))
The append-to macro cannot be called from the something-else macro. I get an error saying the append-to "variable" is undefined.
According to all the information I've managed to glean from Google and other sources, macros are evaluated in a closed environment without access to other code. Essentially, nothing else exists - except built-in Scheme functions and macros - when the macro is evaluated. I have tried using er-macro-transformer, syntax-case (which is now deprecated in Chicken anyway) and even the procedural-macros module.
Surely the entire purpose of macros is that they are built upon other macros, to avoid repeating code. If macros must be written in isolation, they're pretty much useless, to my mind.
I have investigated other Scheme implementations, and had no more luck. Seems it simply cannot be done.
Can someone help me with this, please?
It looks like you're confusing expansion-time with run-time. The syntax-rules example you give will expand to the let+set, which means the append will happen at runtime.
syntax-rules simply rewrites input to given output, expanding macros until there's nothing more to expand. If you want to actually perform some computation at expansion time, the only way to do that is with a procedural macro (this is also what happens in your defmacro CL example).
In Scheme, evaluation levels are strictly separated (this makes separate compilation possible), so a procedure can use macros, but the macros themselves can't use the procedures (or macros) defined in the same piece of code. You can load procedures and macros from a module for use in procedural macros by using use-for-syntax. There's limited support for defining things to run at syntax expansion time by wrapping them in begin-for-syntax.
See for example this SO question or this discussion on the ikarus-users mailing list. Matthew Flatt's paper composable and compilable macros explains the theory behind this in more detail.
The "phase separation" thinking is relatively new in the Scheme world (note that the Flatt paper is from 2002), so you'll find quite a few people in the Scheme community who are still a bit confused about it. The reason it's "new" (even though Scheme has had macros for a long long time) is that procedural macros have only become part of the standard since R6RS (and reverted in R7RS because syntax-case is rather controversial), so the need to rigidly specify them hasn't been an issue until now. For more "traditional" Lispy implementations of Scheme, where compile-time and run-time are all mashed together, this was never an issue; you can just run code whenever.
To get back to your example, it works fine if you separate the phases correctly:
(begin-for-syntax
(define-syntax append-to
(ir-macro-transformer
(lambda (e i c)
(let ((var (cadr e))
(val (caddr e)))
`(set! ,var (append ,var ,val)))))) )
(define-syntax something-else
(ir-macro-transformer
(lambda (e i c)
(let ((vals (list 'print)))
(append-to vals '(1))
vals))))
(something-else) ; Expands to (print 1)
If you put the definition of append-to in a module of its own, and you use-for-syntax it, that should work as well. This will also allow you to use the same module both in the macros you define in a body of code as well as in the procedures, by simply requiring it both in a use and a use-for-syntax expression.
I'm trying to write a small system of macros to do iterative tasks in Emacs Lisp. I had taken it for granted that there is nothing beyond while loop. No more primitives or some hidden features, but I decided, I'd better ask.
By "hidden features" I mean something akin to tagbody in Common Lisp, i.e. the very primitive form to model the code in terms of blocks, jumps and labels. Are there any such thing in eLisp? Not even in any "hackish" way, like, for example, through the bytecode? Of course, I know about (catch ... (throw ... )) construct, but it is not quite the same, because it only allows jumping "backwards", but never forward. I also assumed it is a rather complex construct, not suitable for building fast iteration primitives.
Another thing that bugs me is that there doesn't seem to be a way to create an iterator for hash-tables. I.e. a hash-table must be itereated using maphash and once you exit the maphash function, there's no coming back to where you left it. So far I understand, it has to do something like, exporting a vector of keys and a vector of values and iterating over these, but there doesn't seem to be a way to get hold of these vectors / lists / whichever those are. Or am I again wrong?
I've looked into how cl package generates code for loop and dotimes / dolist / do, but they just use while or maphash, whichever is appropriate, and, frankly, I'm not so fond of their code... More than that, if, say, in the loop there are two for-as-hash clauses, they simply ignore the first (you don't even get a warning for that) and generate code for the second :|
Any chance there are some tricks to get hold of these iteration primitives from the user code in eLisp? If not, how feasible it is, and is it really, to write an extension in C?
You can tagbody as a macro:
(defmacro cl-tagbody (&rest tags-or-stmts)
(let ((blocks '()))
(let ((block (list 'cl--preamble)))
(dolist (tag-or-stmt tags-or-stmts)
(if (consp tag-or-stmt) (push tag-or-stmt block)
;; Add a "go to next block" to implement the fallthrough.
(push (nreverse (cons `(go ,tag-or-stmt) block)) blocks)
(setq block (list tag-or-stmt))))
(push (nreverse (cons `(go cl--exit) block)) blocks))
(let ((catch-tag (make-symbol "cl--tagbody-tag")))
(macroexpand-all
`(let ((next-tag 'cl--preamble))
(while
(not (eq (setq next-tag
(catch ',catch-tag
(cl-case next-tag
,#blocks)))
'cl--exit))))
`((go . (lambda (tag) `(throw ',catch-tag ',tag)))
,#macroexpand-all-environment)))))
1. Other looping constructs?
The only general-purpose built-in looping construct in Emacs Lisp is while (see eval.c). The macros dolist and dotimes (in subr.el) are both implemented using while.
There are also built-in functions for mapping over various data structures: mapatoms, mapc, mapcar, map-char-table, mapconcat, maphash, and map-keymap. But these are implemented in such a way that you can't interleave their execution with other Lisp code (see for example maphash in fns.c). If you want to loop over two such data structures, you have to loop over one and then over the other.
So I think you're basically out of luck.
2. Extensions?
Emacs is deliberately designed not to have dynamic C-level extensions, to make it more difficult for someone to mount an "embrace and extend" attack on the freedom of Emacs users (see the emacs-devel thread starting here, for example).
So if you want to add C-level functionality, you have to edit the source code. Good luck!
I don't even know the proper terminology for this lisp syntax, so I don't know if the words I'm using to ask the question, make sense. But the question makes sense, I'm sure.
So let me just show you. cc-mode (cc-fonts.el) has things called "matchers" which are bits of code that run to decide how to fontify a region of code. That sounds simple enough, but the matcher code is in a form I don't completely understand, with backticks and comma-atsign and just comma and so on, and furthermore it is embedded in a c-lang-defcost, which itself is a macro. I don't know what to call all that, but I want to run edebug on that code.
Look:
(c-lang-defconst c-basic-matchers-after
"Font lock matchers for various things that should be fontified after
generic casts and declarations are fontified. Used on level 2 and
higher."
t `(;; Fontify the identifiers inside enum lists. (The enum type
;; name is handled by `c-simple-decl-matchers' or
;; `c-complex-decl-matchers' below.
,#(when (c-lang-const c-brace-id-list-kwds)
`((,(c-make-font-lock-search-function
(concat
"\\<\\("
(c-make-keywords-re nil (c-lang-const c-brace-id-list-kwds))
"\\)\\>"
;; Disallow various common punctuation chars that can't come
;; before the '{' of the enum list, to avoid searching too far.
"[^\]\[{}();,/#=]*"
"{")
'((c-font-lock-declarators limit t nil)
(save-match-data
(goto-char (match-end 0))
(c-put-char-property (1- (point)) 'c-type
'c-decl-id-start)
(c-forward-syntactic-ws))
(goto-char (match-end 0)))))))
I am reading up on lisp syntax to figure out what those things are and what to call them, but aside from that, how can I run edebug on the code that follows the comment that reads ;; Fontify the identifiers inside enum lists. ?
I know how to run edebug on a defun - just invoke edebug-defun within the function's definition, and off I go. Is there a corresponding thing I need to do to edebug the cc-mode matcher code forms?
What does def-edebug-spec do, and would I use it here? If so, how?
According to (elisp)Top > Debugging > Edebug > Edebug and Macros you have to tell Edebug how to debug a macro by defining it with debug statements or by using def-edebug-spec. This tells it what parameters should be evaluated and which shouldn't. So it can be done. In fact it looks as if c-lang-defconst already been fitted for edebug. Here is the definition in case you were interested:
(def-edebug-spec c-lang-defconst
(&define name [&optional stringp] [&rest sexp def-form]))
However, if you just want to see what the body evaluates to, then the way to do that is to use something like macro-expand-last-sexp below to see the result. Position your cursor after the sexp you want expanded (as you would for C-x C-e) and run M-x macro-expand-last-sexp RET. This will show you what it gets expanded to. You may run into troubles if you try to expand something like ,(....) so you may have to copy that sexp somewhere else and delete the , or ,#.
(defun macro-expand-last-sexp (p)
"Macro expand the previous sexp. With a prefix argument
insert the result into the current buffer and pretty print it."
(interactive "P")
(let*
((sexp (preceding-sexp))
(expanded (macroexpand sexp)))
(cond ((eq sexp expanded)
(message "No changes were found when macro expanding"))
(p
(insert (format "%S" expanded))
(save-excursion
(backward-sexp)
(indent-pp-sexp 1)
(indent-pp-sexp)))
(t
(message "%S" expanded)))))
I guess it depends on exactly what you are trying to do.
Use macroexpand or macroexpand-all to turn it into macro-free code and debug as usual?
Backticks &co may be best illustrated by an example:
(let ((a 1)
(b (list 2 3)))
`(a ,a ,b ,#b))
-> (a 1 (2 3) 2 3)
A backtick (or backquote`) is similar to a quote(') in that it prevents evaluation, except its effect can be selectively undone with a comma(,); and ,# is like ,, except that its argument, which must be a list, is spliced into the resulting list.