I'm trying to write a small system of macros to do iterative tasks in Emacs Lisp. I had taken it for granted that there is nothing beyond while loop. No more primitives or some hidden features, but I decided, I'd better ask.
By "hidden features" I mean something akin to tagbody in Common Lisp, i.e. the very primitive form to model the code in terms of blocks, jumps and labels. Are there any such thing in eLisp? Not even in any "hackish" way, like, for example, through the bytecode? Of course, I know about (catch ... (throw ... )) construct, but it is not quite the same, because it only allows jumping "backwards", but never forward. I also assumed it is a rather complex construct, not suitable for building fast iteration primitives.
Another thing that bugs me is that there doesn't seem to be a way to create an iterator for hash-tables. I.e. a hash-table must be itereated using maphash and once you exit the maphash function, there's no coming back to where you left it. So far I understand, it has to do something like, exporting a vector of keys and a vector of values and iterating over these, but there doesn't seem to be a way to get hold of these vectors / lists / whichever those are. Or am I again wrong?
I've looked into how cl package generates code for loop and dotimes / dolist / do, but they just use while or maphash, whichever is appropriate, and, frankly, I'm not so fond of their code... More than that, if, say, in the loop there are two for-as-hash clauses, they simply ignore the first (you don't even get a warning for that) and generate code for the second :|
Any chance there are some tricks to get hold of these iteration primitives from the user code in eLisp? If not, how feasible it is, and is it really, to write an extension in C?
You can tagbody as a macro:
(defmacro cl-tagbody (&rest tags-or-stmts)
(let ((blocks '()))
(let ((block (list 'cl--preamble)))
(dolist (tag-or-stmt tags-or-stmts)
(if (consp tag-or-stmt) (push tag-or-stmt block)
;; Add a "go to next block" to implement the fallthrough.
(push (nreverse (cons `(go ,tag-or-stmt) block)) blocks)
(setq block (list tag-or-stmt))))
(push (nreverse (cons `(go cl--exit) block)) blocks))
(let ((catch-tag (make-symbol "cl--tagbody-tag")))
(macroexpand-all
`(let ((next-tag 'cl--preamble))
(while
(not (eq (setq next-tag
(catch ',catch-tag
(cl-case next-tag
,#blocks)))
'cl--exit))))
`((go . (lambda (tag) `(throw ',catch-tag ',tag)))
,#macroexpand-all-environment)))))
1. Other looping constructs?
The only general-purpose built-in looping construct in Emacs Lisp is while (see eval.c). The macros dolist and dotimes (in subr.el) are both implemented using while.
There are also built-in functions for mapping over various data structures: mapatoms, mapc, mapcar, map-char-table, mapconcat, maphash, and map-keymap. But these are implemented in such a way that you can't interleave their execution with other Lisp code (see for example maphash in fns.c). If you want to loop over two such data structures, you have to loop over one and then over the other.
So I think you're basically out of luck.
2. Extensions?
Emacs is deliberately designed not to have dynamic C-level extensions, to make it more difficult for someone to mount an "embrace and extend" attack on the freedom of Emacs users (see the emacs-devel thread starting here, for example).
So if you want to add C-level functionality, you have to edit the source code. Good luck!
Related
In Lisp, a function's arguments are evaluated first before entering the function body. Macro arguments stay not evaluated.
But sometimes, one wants to inject code pieces stored in variables into a macro. This means evaluating the argument for the macro first, and then apply the macro-of-choice on this evaluated result.
One has to resort to
(eval `(macro ,arg))
To achieve this - but eval does not behave correctly in different environments.
The best thing would be, if one could do:
(apply macro (list arg))
or
(funcall macro arg)
But since the macro is not a function this doesn't work.
Is it possible to achieve something like this? - To circumvent that problem oder to make the macro available in the functions namespace?
Or am I missing some other ways to solve such problems?
I came to this question while trying to answer How to produce HTML from a list. but also in Generate TYPECASE with macro in common lisp, Evaluate arguments passed to a macro that generates functions in lisp, and How to convert a list to code/lambda in scheme?. But I always thought while answering them it would be good to have an apply or funcall-like function which can take macros.
It is not clear what you are trying to do, although it is almost certain that you are confused about something. In particular if you are calling eval inside macroexpansions then in almost all cases you are doing something both seriously wrong and seriously dangerous. I can't ever think of a case where I've wanted macros which expand to things including eval and I have written Lisp for a very very long time.
That being said, here is how you call the function associated with a macro, and why it is very seldom what you want to do.
Macros are simply functions whose domain and range is source code: they are compilers from a language to another language. It is perfectly possible to call the function associated with a macro, but what that function will return is source code, and what you will then need to do with that source code is evaluate it. If you want a function which deals with run-time data which is not source code, then you need that function, and you can't turn a macro into that function by some magic trick which seems to be what you want to do: that magic trick does not, and can not, exist.
So for instance if I have a macro
(defmacro with-x (&body forms)
`(let ((x 1))
,#forms))
Then I can call its macro function on a bit of source code:
> (funcall (macro-function 'with-x)
'(with-x (print "foo")) nil)
(let ((x 1)) (print "foo"))
But the result of this is another bit of source code: I need to compile or evaluate it, and nothing I can do will get around this.
Indeed in (almost?) all cases this is just the same as macroexpand-1):
> (macroexpand-1 '(with-x (print "foo")))
(let ((x 1)) (print "foo"))
t
And you can probably write macroexpand-1 in terms of macro-function:
(defun macroexpand-1/equivalent (form &optional (env nil))
(if (and (consp form)
(symbolp (first form))
(macro-function (first form)))
(values (funcall (macro-function (first form)) form env)
t)
(values form nil)))
So, if the result of calling a macro is source code, what do you do with that source code to get a result which is not source code? Well, you must evaluate it. And then, well, since the evaluator expands macros for you anyway, you might as well just write something like
(defun evaluate-with-x (code)
(funcall (compile nil `(lambda ()
(with-x ,#code)))))
So you didn't need to call the macro's function in any case. And this is not the magic trick which turns macros into functions dealing with data which is not source code: it is a terrible horror which is entirely made of exploding parts.
A concrete example: CL-WHO
It looks like this question might have its origins in this one and the underlying problem there is that that's not what CL-WHO does. In particular it is a confusion to think that something like CL-WHO is a tool for taking some kind of list and turning it into HTML. It's not: it's a tool for taking the source code of a language which is built on CL but includes a way of expressing HTML output mingled with CL code, and compiles it into CL code which will do the same thing. It happens to be the case that CL source code is expressed as lists & symbols, but CL-WHO isn't really about that: it's a compiler from, if you like, 'the CL-WHO language' to CL.
So, let's try the trick we tried above and see why it's a disaster:
(defun form->html/insane (form)
(funcall
(compile nil `(lambda ()
(with-html-output-to-string (,(make-symbol "O"))
,#form)))))
And you might, if you did not look at this too closely, think that this function does in fact do the magic trick:
> (form->html/insane '(:p ((:a :href "foo") "the foo")))
"<p></p><a href='foo'>the foo</a>"
But it doesn't. What happens if we call form->html/insane on this perfectly innocuous list:
(:p (uiop/run-program:run-program "rm -rf $HOME" :output t))
Hint: don't call form->html/insane on this list if you don't have very good backups.
CL-WHO is an implementation of a programming language which is a strict superset of CL: if you try to turn it into a function to turn lists into HTML you end up with something involving the same nuclear weapon you tinker with every time you call eval, except that nuclear weapon is hidden inside a locked cupboard where you can't see it. But it doesn't care about that: if you set it off it will still reduce everything within a few miles to radioactive ash and rubble.
So if you want a tool which will turn lists – lists which aren't source code – into HTML then write that tool. CL-WHO might have the guts of such a tool in its implemenentation, but you can't use it as it is.
And this is the same problem you face whenever you are trying to abuse macros this way: the result of calling a macro's function is Lisp source code, and to evaluate that source code you need eval or an equivalent of eval. And eval is not only not a terrible solution to almost any problem: it's also a nuclear weapon. There are, perhaps problems for which nuclear weapons are good solutions, but they are few and far between.
I am trying to understand all the looping constructs in Emacs Lisp.
In one example I am trying to iterate over a list of symbols and print them to the *message* buffer like so:
(let* ((plist package-activated-list) ;; list of loaded packages
(sorted-plist (sort plist 'string<)))
(--map (message (format "%s" it)) sorted-plist))
--map is a function from the package dash.el.
How to do this in pure Elisp?
Now how do I iterate over a list in Elisp, without using other packages.
I have seen some examples using while and dolist macro, for example here:
https://www.gnu.org/software/emacs/manual/html_node/elisp/Iteration.html
But those are destructive, non-functional ways to express a loop.
Coming from Scheme (having worked with it and with SICP some twenty years ago!), I tend to prefer functional, non destructive (does that always lead to recursive?) ways to express ideas.
So what are idiomatic ways to loop over a list of items in Emacs Lisp?
Also: Are there ways to express loops in a functional fashion in Emacs Lisp?
What I have found so far
Loop Macros (from Common Lisp?) prefixed with "cl-*"
https://www.gnu.org/software/emacs/manual/html_node/cl/Loop-Facility.html
Iteration Clauses
https://www.gnu.org/software/emacs/manual/html_node/cl/Iteration-Clauses.html#Iteration-Clauses
Dash.el
https://github.com/magnars/dash.el
Magnar Sveen's excellent package marketed as "A modern list api for Emacs. No 'cl required."
What else is there? Any recommended reading?
There is a bunch of native ~map~ functions that you can explore for iterating throught lists, with some subbtilities or sugar.
In this case, I choose `mapconcat', it is loaded from C code.
(mapconcat #'message sorted-plist "\n")
dolist is not destructive, and that is probably the most idiomatic way in Emacs Lisp, or Common Lisp for that matter, to loop over a list when you just want to do something with each member in turn:
(setq *properties* '(prop1 prop2 prop3))
(dolist (p *properties*)
(print p))
The seq-doseq function does the same thing as dolist, but accepts a sequence argument (e.g., a list, vector, or string):
(seq-doseq (p *properties*)
(print p))
If a more functional style is desired, the seq-do function applies a function to the elements of a sequence and returns the original sequence. This function is similar to the Scheme procedure for-each, which is also used for its side effects.
(seq-do #'(lambda (p) (print p)) *properties*)
If you're looking for more traditional Lisp map functions, e-lisp has them:
https://www.gnu.org/software/emacs/manual/html_node/elisp/Mapping-Functions.html#Mapping-Functions
In your code snippet, mapc is likely the one you want (throws the values away, just applies the function; mapcar is still there if you want the values):
(mapc #'(lambda (thing) (message (format "%s" thing))) sorted-plist)
Probably the hardest part of learning lisp has been to think in the "lisp way" which is elegant and impressive, but not always easy. I know that recursion is used to solve a lot of problems, and I am working through a book that instead uses apply to solve a lot of problems, which I understand is not as lispy, and also not as portable.
An experienced lisper should be able to help with this logic without knowing specifically what describe-path location and edges refer to. Here is an example in a book I am working through:
(defun describe-paths (location edges)
(apply (function append) (mapcar #'describe-path
(cdr (assoc location edges)))))
I have successfully rewritten this to avoid apply and use recursion instead. It seems to be working:
(defun describe-paths-recursive (location edges)
(labels ((processx-edge (edge)
(if (null edge)
nil
(append (describe-path (first edge))
(processx-edge (rest edge))))))
(processx-edge (cdr (assoc location edges)))))
I would like some more seasoned pairs of eyes on this to advise if there is a more elegant way to translate the apply to recursion, or if I have done something unwise. This code seems decent, but would there been something even more "lispy" ?
(apply (function append) (mapcar #'g ...)) is just mapcan (update: with usual caveats about destructive update and quoted lists, see also this):
(defun describe-paths (location edges)
(mapcan #'describe-path
(cdr (assoc location edges))))
Recursion is good for thinking, for understanding. But actually using it in your code comes with a price.
Your recursive re-write is tail recursive modulo cons; no Lisp has this optimization AFAIK, even though it was first described in 1974, in Lisp.
So what you wrote is good as an executable specification.
But Common Lisp is a practical language. In particular, it has many ways to encode iteration. Remember, iterative processes are our goal; recursive processes are terrible, efficiency-wise. So when we write a code which is syntactically recursive, we still want it to describe an iterative process (such that runs in constant stack space).
Common Lisp, being a practical language, would have us just write the loop out directly. For one,
(defun describe-paths-loop (location edges &aux (res (list 1)) (p res))
(dolist (x (cdr (assoc location edges))
(cdr res)) ; the return form
(setf (cdr p) (describe-path x))
(setf p (last p))))
is guaranteed to work in constant stack space.
update: this destructively concatenates lists returned by describe-path so it should take care not to return lists with the same last cons cell on separate invocations, or this could create circular structure. Alternatively, the call to describe-path could be wrapped in a copy-list call. Of course, if describe-path were to return a list which is already cyclic, last here would go into a loop too.
I saw several opinions about using apply is a bad style. But actually that would be great if somebody will explain me why apply is considered to be bad.
What do you mean using a word "lispy". Common lisp allows to program in any style you want.
If "lispy" means functional programming style, then the first code is written in more functional programming style. A function is passed to a function mapcar and another function is passed to apply and all the job is done by passing the results of one function to another. In you code you don't pass functions as arguments to other functions. But recursion can be considered as functional programming style sign. And code in the book is shorter than yours.
If you don't like apply because of apply determines the argument count in runtime, you can use reduce in this situation (if I understood the data structures correctly):
(Thanks to Joshua Taylor for pointing a huge resource overhead without :from-end t key argument)
(defun describe-paths (location edges)
(reduce #'append (mapcar #'describe-path
(rest (assoc location edges))) :from-end t))
Anyway I'm pretty sure that the purpose of the code in the book is the education reason. It's an example of mapcar and apply that shows how lists are treated as data and code in lisp.
p.s. Actually I figured why apply can be bad (stack is used for function calls).
> (apply #'+ (make-list 500000 :initial-element 1))
*** - Lisp stack overflow. RESET
So as Rainer Joswig told it's lispy to avoid stack overflows. Reduce fix the problem.
> (reduce #'+ (make-list 50000000 :initial-element 1))
50000000
The Lisp way is to use functional, imperative or object-oriented programming (with or without mutable state) to solve a problem, or to invent some other programming as you see fit and express it in macros. Looking for recursion while ignoring other approaches is not the Lisp way; it's the way of the wayward Lisp academic.
The most straightforward way to rewrite the function:
(defun describe-paths (location edges)
(apply (function append) (mapcar #'describe-path
(cdr (assoc location edges)))))
is to use loop. The proper motivation for eliminting apply is that we expect many paths, which could exceed the limit on the number of arguments to a function.
All you are doing with apply is making a big argument list to the append function. We can append any number of lists into a big list with loop like this:
(defun describe-paths (location edges)
(loop for path in (cdr (assoc location edges))
appending (describe-path path))
Presumably, describe-path returns a list, and you want to catenate these together.
The appending clause of loop, which may also be spelled append, gathers appends the value of is argument form into an anonymous list. That list becomes the return value when the loop terminates.
We can use nconcing to improve the performance if we have justification in believing that the lists returned by described-path are freshly allocated on each call.
There's nothing wrong with this question; plenty of questions similar to this are asked in the python category, for example.
But to your question: what you are doing is Good. In fact, it closely resembles, nearly identically, a more general technique Peter Norvig shows in one of his Lisp books, so either you've read that book, or you stumbled upon a good practice on your own. Either way, this is a perfectly acceptable implementation of recursion.
One function and one macro
;; I am sorry for the confusing function name.
;; As one answer suggests, cons-if-not-member is the better name.
(defun cons-if-member (element list)
(if (member element list)
list
(cons element list)))
(defmacro pushnew-no-bells (element list)
"pushnew without place support"
`(setq ,list (cons-if-member ,element ,list)))
(let ((xx (list 1 2)))
(pushnew-no-bells 0 xx)
xx)
I do not know which of the following is correct:
cons-if-member is a non-destructive function and pushnew-no-bells is a destructive macro.
Both are non-destructive.
cons-if-member is a non-destructive function and the adjectives "destructive" and "non-destructive" do not apply to macros.
none of the above
I do not have any idea on whether pushnew is considered destructive or not either, but I wanted to make things simple by dropping place support first.
pushnew changes the value of it's place-form (2nd argument), so it is destructive: it changes something "in place" instead of just creating a new object (which may share structure with an existing one). Your cons-if-member (which would better be called cons-unless-member or cons-if-not-member) does not modify anything "in place", so it's actually non-destructive.
Note, BTW., that you cannot really exclude "general place" support, due to the presence of symbol macros. Observe:
(defclass foo ()
((x :initform nil)))
(let ((instance (make-instance 'foo)))
(with-slots (x) instance
(pushnew-no-bells 1 x))
(format t "~&Now: ~S~%" (slot-value instance 'x)))
cons-if-member is not destructive. It is also roughly equivalent to ADJOIN. pushnew-no-bells modifies a place, but does not modify the structure of the list onto which an element might be pushed. However, it can modify the structure of other lists, because you could use it as (let ((list (list 1 2 3 4))) (pushnew-no-bells '1 (cddr list))). (Also, the form list will be evaluated twice, which is not good (but also not the main point of this question/answer).) This is destructive in the sense that it modifies that place, but it is not destructive in the sense that, e.g., nreverse is (nreverse can change the entire structure of a cons list).
The Hyperspec doesn't make quite the same distinction between destructive and non-destructive. The spec for ADJOIN, for instance, just says what it does
Tests whether item is the same as an existing element of list. If the item is not an existing element, adjoin adds it to list (as if by cons) and returns the resulting list; otherwise, nothing is added and the original list is returned.
and omits any mention of side effects. The documentation for PUSHNEW, on the other hand, mentions in its syntax section that it requires a place
pushnew item place &key key test test-not
=> new-place-value
and the decription mentions that it has side effects:.
the new list is stored in place. [emphasis added] …
Side Effects: The contents of place may be modified.
While destructive and non-destructive capture some general ideas about how things are implemented, actual implementations tend to be a little bit more subtle, because the programmer is concerned with what things might be destructively modified, and what sort of state might be changed.
The approach that you are using (that is, making a functional implementation of some operation and then implementing a modifying macro on top of it), however, is very good, as it will help you document what functions and macros will have side effects. It will help anyone reading that documentation understand what the intended side effects of the macro is (just compute what the function would compute, and then store it back to the place). If you're doing much of this (actually, if you're doing any of this), you should also probably take a good look at DEFINE-MODIFY-MACRO which makes implementing these kinds of function/macro pairs very easy, and will help you avoid common pitfalls (like the double evaluation of list above).
I'd like to learn the internals of Lisp, so I want to see how everything is implemented.
For example,
(macroexpand '(loop for i upto 10 collect i))
gives me (in SBCL)
(BLOCK NIL
(LET ((I 0))
(DECLARE (TYPE (AND NUMBER REAL) I))
(SB-LOOP::WITH-LOOP-LIST-COLLECTION-HEAD (#:LOOP-LIST-HEAD-1026
#:LOOP-LIST-TAIL-1027)
(SB-LOOP::LOOP-BODY NIL
(NIL NIL (WHEN (> I '10) (GO SB-LOOP::END-LOOP)) NIL)
((SB-LOOP::LOOP-COLLECT-RPLACD
(#:LOOP-LIST-HEAD-1026 #:LOOP-LIST-TAIL-1027)
(LIST I)))
(NIL (SB-LOOP::LOOP-REALLY-DESETQ I (1+ I))
(WHEN (> I '10) (GO SB-LOOP::END-LOOP)) NIL)
((RETURN-FROM NIL
(SB-LOOP::LOOP-COLLECT-ANSWER
#:LOOP-LIST-HEAD-1026)))))))
But LOOP-BODY, WITH-LOOP-LIST-COLLECTION-HEAD, etc. are still macros. How can I expand a macro form completely?
To see the full expansion one needs to walk the Lisp form on all levels and expand them. For this it is necessary that this so-called code walker understands Lisp syntax (and not just s-expression syntax). For example in (lambda (a b) (setf a b)), the list (a b) is a parameter list and should not be macro expanded.
Various Common Lisp implementations provide such a tool. The answer of 6502 mentions MACROEXPAND-ALL which is provided by SBCL.
If you use a development environment, it is usually provided as a command:
SLIME: M-x slime-macroexpand-all with C-c M-m
LispWorks: menu Expression > Walk or M-x Walk Form, shorter M-Sh-m.
The other answers are excellent for you question but you say you want to see how everything is implemented.
Many macros (as you know already) are implemented using macros and whilst macroexpand-all is very useful but you can lose the context of what macro was responsible for what change.
One nice middle ground (if you are using slime) is to use slime-expand-1 (C-c Enter) which shows the expansion is another buffer. You can then useslime-expand-1 inside this new buffer to expand macros in-place.
This allows you to walk the tree expanding as you read and also to use undo to close the expansions again.
For me this has been a god-send in understanding other people's macros. Hope this helps you too, have fun!
You can try to use MACROEXPAND-ALL but what you may get is not necessarily useful.
In something like LOOP the real meat is the macro itself, not the generated code.
(Note: If you're not interested in portability, SBCL provides macroexpand-all, which will do what you're after. If you're after a portable solution, read on...)
The quick-and-dirty solution would be to macroexpand the form itself, then recursively macroexpand all but the first element of the resulting list. This is an imperfect solution; it will fail completely the moment it attempts to process a let's bindings (the first argument to let, the list of bindings, is not meant to be macroexpanded, but this code will do it anyway).
;;; Quick-and-dirty macroexpand-all
(defun macroexpand* (form)
(let ((form (macroexpand form)))
(cons (car form) (mapcar #'macroexpand (cdr form)))))
A more complete solution would consider special forms specially, not macroexpanding their unevaluated arguments. I could update with such a solution, if desired.