Can (or should) a macro expansion have side effects? For example, here is a macro which actually goes and grabs the contents of a webpage at compile time:
#lang racket
(require (for-syntax net/url))
(require (for-syntax racket/port))
(define-syntax foo
(lambda (syntx)
(datum->syntax #'lex
(port->string
(get-pure-port
(string->url
(car (cdr (syntax->datum syntx)))))))))
Then, I can do (foo "http://www.pointlesssites.com/") and it will be replaced with "\r\n<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\"\r\n\t <and so on>"
Is this good practice, or not? Am I garunteed that Racket will only run this code once? If I add a (display "running...") line to the macro, it only prints once, but I would hate to generalize from one example ...
PS - the reason I'm asking is because I actually think this could be really useful sometimes. For example this is a library which allows you to load (at compile-time) a discovery document from the Google API Discovery service and automatically create wrappers for it. I think it would be really cool if the library actually fetched the discovery document from the web, instead of from a local file.
Also, to give an example of a macro with a different kind of side effects: I once built a macro which translated a small subset of Racket into (eta-expanded) lambda calculus (which is, of course, still runnable in Racket). Whenever the macro finished translating a function, it would store the result in a dictionary so that later invocations of the macro could make use of that function definition in their own translations.
The short answer
It's fine for macros to have side effects, but you should make sure that your program doesn't change behavior when it's compiled ahead of time.
The longer answer
Macros with side effects are a powerful tool, and can let you do things that make programs much easier to write, or enable things that wouldn't be possible at all. But there are pitfalls to be aware of when you use side effects in macros. Fortunately, Racket provides all the tools to make sure that you can do this correctly.
The simplest kind of macro side effect is where you use some external state to find the code you want to generate. The examples you list in the question (reading Google API description) are of this kind. An even simpler example is the include macro:
#lang racket
(include "my-file.rktl")
This reads the contents of myfile.rktl and drops it in place right where the include form is used.
Now, include isn't a good way to structure your program, but this is a quite benign sort of side effect in the macro. It works the same if you compile the file ahead of time as if you don't, since the result of the include is part of the file.
Another simple example that's not good is something like this:
#lang racket
(define-syntax (show-file stx)
(printf "using file ~a\n" (syntax-source stx))
#'(void))
(show-file)
That's because the printf gets executed only at compile time, so if you compile your program that uses show-file ahead of time (as with raco make) then the printf will happen then, and won't happen when the program is run, which probably isn't the intention.
Fortunately, Racket has a technique for letting you write macros like show-file effectively. The basic idea is to leave residual code that actually performs the side effect. In particular, you can use Racket's begin-for-syntax form for this purpose. Here's how I would write show-file:
#lang racket
(define-syntax (show-file stx)
#`(begin-for-syntax
(printf "using file ~a\n" #,(syntax-source stx))))
(show-file)
Now, instead of happening when the show-file macro is expanded, the printf happens in the code that show-file generates, with the source embedded in the expanded syntax. That way your program keeps working correctly in the presence of ahead-of-time compilation.
There are other uses of macros with side effects, too. One of the most prominent in Racket is for inter-module communication -- because require doesn't produce values that the requiring module can get, to communicate between modules the most effective way is to use side effects. To make this work in the presence of compilation requires almost exactly the same trick with begin-for-syntax.
This is a topic that the Racket community, and I in particular, have thought a lot about, and there are several academic papers talking about how this works:
Composable and Compilable Macros: You want it when?, Matthew Flatt, ICFP 2002
Advanced Macrology and the Implementation of Typed Scheme, Ryan Culpepper, Sam Tobin-Hochstadt, and Matthew Flatt, Scheme Workshop 2007
Languages as Libraries, Sam Tobin-Hochstadt, Ryan Culpepper, Vincent St-Amour, Matthew Flatt, and Matthias Felleisen, PLDI 2011
In common Lisp the function eval-when allows you to decide when the macro will be expanded.
Related
In my ongoing quest to learn lisp, I'm running into a conceptual problem. It's somewhat akin to the question here, but maybe it's thematically appropriate to lisp that my question is a level of abstraction up.
As a rule, when should you create a macro vs. a function? It seems to me, maybe naively, that there would be very few cases where you must create a macro instead of a function, and that in most remainder cases, a function would generally suffice. Of these remainder cases, it seems like the main additional value of a macro would be in clarity of syntax. And if that's the case, then it seems like not just the decision to opt for macro use but also the design of their structures might be fundamentally idiosyncratic to the individual programmer.
Is this wrong? Is there a general case outlining when to use macros over functions? Am I right that the cases where a macro is required by the language are generally few? And lastly, is there a general syntactic form that's expected of macros, or are they generally used as shorthands by programmers?
I found a detailed answer, from Paul Graham's On Lisp, bold emphases added:
Macros can do two things that functions can’t: they can control (or prevent) the evaluation of their arguments, and they are expanded right into the calling context. Any application which requires macros requires, in the end, one or both of these properties.
...
Macros use this control in four major ways:
Transformation. The Common Lisp setf macro is one of a class of macros which pick apart their arguments before evaluation. A built-in access function will often have a converse whose purpose is to set what the access function retrieves. The converse of car is rplaca, of cdr, rplacd, and so on. With setf we can use calls to such access functions as if they were variables to be set, as in (setf (car x) ’a), which could expand into (progn (rplaca x ’a) ’a).
To perform this trick, setf has to look inside its first argument. To know that the case above requires rplaca, setf must be able to see that the first argument is an expression beginning with car. Thus setf, and any other operator which transforms its arguments, must be written as a macro.
Binding. Lexical variables must appear directly in the source code. The first argument to setq is not evaluated, for example, so anything built on setq must be a macro which expands into a setq, rather than a function which calls it. Likewise for operators like let, whose arguments are to appear as parameters in a lambda expression, for macros like do which expand into lets, and so on. Any new operator which is to alter the lexical bindings of its arguments must be written as a macro.
Conditional evaluation. All the arguments to a function are evaluated. In constructs like when, we want some arguments to be evaluated only under certain conditions. Such flexibility is only possible with macros.
Multiple evaluation. Not only are the arguments to a function all evaluated, they are all evaluated exactly once. We need a macro to define a construct like do, where certain arguments are to be evaluated repeatedly.
There are also several ways to take advantage of the inline expansion of macros. It’s important to emphasize that the expansions thus appear in the lexical context of the macro call, since two of the three uses for macros depend on that fact. They are:
Using the calling environment. A macro can generate an expansion containing a variable whose binding comes from the context of the macro call. The behavior of the following macro:
(defmacro foo (x) ‘(+ ,x y))
depends on the binding of y where foo is called.
This kind of lexical intercourse is usually viewed more as a source of contagion than a source of pleasure. Usually it would be bad style to write such a macro. The ideal of functional programming applies as well to macros: the preferred way to communicate with a macro is through its parameters. Indeed, it is so rarely necessary to use the calling environment that most of the time it happens, it happens by mistake...
Wrapping a new environment. A macro can also cause its arguments to be evaluated in a new lexical environment. The classic example is let, which could be implemented as a macro on lambda. Within the body of an expression like (let ((y 2)) (+ x y)), y will refer to a new variable.
Saving function calls. The third consequence of the inline insertion of macro expansions is that in compiled code there is no overhead associated with a macro call. By runtime, the macro call has been replaced by its expansion. (The same is true in principle of functions declared inline.)
...
What about those operators which could be written either way [i.e. as a function or a macro]?... Here are several points to consider when we face such choices:
THE PROS
Computation at compile-time. A macro call involves computation at two times: when the macro is expanded, and when the expansion is evaluated. All the macro expansion in a Lisp program is done when the program is compiled, and every bit of computation which can be done at compile-time is one bit that won’t slow the program down when it’s running. If an operator could be written to do some of its work in the macro expansion stage, it will be more efficient to make it a macro, because whatever work a smart compiler can’t do itself, a function has to do at runtime. Chapter 13 describes macros like avg which do some of their work during the expansion phase.
Integration with Lisp. Sometimes, using macros instead of functions will make a program more closely integrated with Lisp. Instead of writing a program to solve a certain problem, you may be able to use macros to transform the problem into one that Lisp already knows how to solve. This approach, when possible, will usually make programs both smaller and more efficient: smaller because Lisp is doing some of your work for you, and more efficient because production Lisp systems generally have had more of the fat sweated out of them than user programs. This advantage appears mostly in embedded languages, which are described starting in Chapter 19.
Saving function calls. A macro call is expanded right into the code where it appears. So if you write some frequently used piece of code as a macro, you can save a function call every time it’s used. In earlier dialects of Lisp, programmers took advantage of this property of macros to save function calls at runtime. In Common Lisp, this job is supposed to be taken over by functions declared inline.
By declaring a function to be inline, you ask for it to be compiled right into the calling code, just like a macro. However, there is a gap between theory and practice here; CLTL2 (p. 229) says that “a compiler is free to ignore this declaration,” and some Common Lisp compilers do. It may still be reasonable to use macros to save function calls, if you are compelled to use such a compiler...
THE CONS
Functions are data, while macros are more like instructions to the compiler. Functions can be passed as arguments (e.g. to apply), returned by functions, or stored in data structures. None of these things are possible with macros.
In some cases, you can get what you want by enclosing the macro call within a lambda-expression. This works, for example, if you want to apply or funcall certain macros:> (funcall #’(lambda (x y) (avg x y)) 1 3) --> 2. However, this is an inconvenience. It doesn’t always work, either: even if, like avg, the macro has an &rest parameter, there is no way to pass it a varying number of arguments.
Clarity of source code. Macro definitions can be harder to read than the equivalent function definitions. So if writing something as a macro would only make a program marginally better, it might be better to use a function instead.
Clarity at runtime. Macros are sometimes harder to debug than functions. If you get a runtime error in code which contains a lot of macro calls, the code you see in the backtrace could consist of the expansions of all those macro calls, and may bear little resemblance to the code you originally wrote.
And because macros disappear when expanded, they are not accountable at runtime. You can’t usually use trace to see how a macro is being called. If it worked at all, trace would show you the call to the macro’s expander function, not the macro call itself.
Recursion. Using recursion in macros is not so simple as it is in functions. Although the expansion function of a macro may be recursive, the expansion itself may not be. Section 10.4 deals with the subject of recursion in macros...
Having considered what can be done with macros, the next question to ask is: in what sorts of applications can we use them? The closest thing to a general description of macro use would be to say that they are used mainly for syntactic transformations. This is not to suggest that the scope for macros is restricted. Since Lisp programs are made from lists, which are Lisp data structures, “syntactic transformation” can go a long way indeed...
Macro applications form a continuum between small general-purpose macros like while, and the large, special-purpose macros defined in the later chapters. On one end are the utilities, the macros resembling those that every Lisp has built-in. They are usually small, general, and written in isolation. However, you can write utilities for specific classes of programs too, and when you have a collection of macros for use in, say, graphics programs, they begin to look like a programming language for graphics. At the far end of the continuum, macros allow you to write whole programs in a language distinctly different from Lisp. Macros used in this way are said to implement embedded languages.
Yes, the first rule is: don't use a macro where a function will do.
There are a few things you can't do with functions, for example conditional evaluation of code. Others become quite unwieldy.
In general I am aware of three recurring use cases for macros (which doesn't mean that there aren't any others):
Defining forms (e. g. defun, defmacro, define-frobble-twiddle)
These often have to take some code snippet, wrap it (e. g. in a lamdba form), and register it somewhere, maybe even multiple places. The users (programmers) should only concern themselves with the code snippet. This is thus mostly about removing boilerplate. Additionally, the macro can process the body, e. g. registering docstrings, handle declarations etc.
Example: Imagine that you are writing a sort of event mini-framework. Your event handlers are pure functions that take some input and produce an effect declaration (think re-frame from the Clojure world). You want these functions to be normal named functions so that you can just test them with the usual testing frameworks, but also register them in a lookup table for your event loop mechanism. You'd maybe want to have something like a define-handler macro:
(defvar *handlers* (make-hash-table)) ; internal for the framework
(defmacro define-handler (&whole whole name lambda-list &body body)
`(progn (defun ,#(rest whole))
(setf (gethash ,name *handlers*)
(lambda ,lambda-list ,#body)))) ; could also be #',name
Control constructs (e. g. case, cond, switch, some->)
These use conditional evaluation and convenient re-arrangement of the expression.
With- style wrappers
This is an idiom to provide unwind-protect functionality to some arbitrary resource. The difference to a general with construct (as in Clojure) is that the resource type can be anything, you don't have to reify it with something like a Closable interface.
Example:
(defmacro with-foo-bar-0 (&body body)
(let ((foo-bar (gensym "FOO-BAR")))
`(let (,foo-bar))
(shiftf ,foo-bar (aref (gethash :foo *buzz*) 0) 0)
(unwind-protect (progn ,#body)
(setf (aref (gethash :foo *buzz*) 0) ,foo-bar)))))
This sets something inside a nested data structure to 0, and ensures that it is reset to the value it had before on any, even non-local, exit.
[This is a much-reduced version of a longer, incomplete answer which I decided was not appropriate for SE.]
There are no cases where you must use a macro. Indeed, there are no cases where you must use a programming language at all: if you are happy to learn the order code for the machine you are using and competent with a keypunch then you can program that way.
Most of us are not happy doing that: we like to use programming languages. These have two obvious benefits and one less-obvious but far more important one. The two obvious benefits:
programming languages make programming easier;
programming languages make programs portable across machines.
The more important reason is that building languages is an enormously successful approach to problem solving for human beings. It's so successful that we do it all the time, without even thinking we are doing it. Every time we invent some new term for something we are in fact inventing a language; every time a mathematician invents some new bit of notation they are inventing a language. People like to sneer at these languages by calling them 'jargon', 'slang' or 'dialect' but, famously: a shprakh iz a dialekt mit an armey un flot (translated: a language is a dialect with an army and navy).
The same thing is true for programming languages as is true for natural languages, except that programming languages are designed to communicate both with other humans and with a machine, and the machine requires very precise instructions. This means that it can be rather hard to build programming languages, so people tend to stick with the languages they know.
Except that they don't: the approach of building a language to describe some problem is so powerful that people in fact do this anyway. But they don't know that they are doing it and they don't have the tools to do it so what they end up with tends to be a hideous monster stitched together from pieces of other things with the robustness and readability of custard. We've all dealt with such things. A common characteristic is 'language in a string' where one language appears within strings of another language, with constructs of this inner language being put together by string operations in the outer language. If you are really lucky this will go several levels deep (I have seen three).
These things are abominations, but they are still the best way of dealing with large problem areas. Well, they are the best way if you live in a world where constructing a new programming language is so hard that only special clever people can do it
But it's hard only because if your only tool is C then everything looks like a PDP-11. If instead we used a tool which made the incremental construction of programming languages easy by allowing them to be defined in terms of simpler versions of themselves in a lightweight way, then we could just construct whole families of programming languages in which to talk about various problems, each of which would simply be a point in the space of possible languages. And anyone could do this: it would be a little bit harder than just writing functions, because working out grammar rules is a little bit harder than thinking up new words, but it would not be a lot harder.
And that's what macros do: they let you define programming languages to talk about a particular problem area in a way which is extremely lightweight. One such language is Common Lisp, but it's just one starting point in the space of Lisp-family languages: a point from which you can build the language you actually want (and people, of course, will belittle these languages by calling them 'dialects': well, a programming language is only a dialect with a standards committee).
Functions let you add to the vocabulary of the language you are building. Macros let you add to the grammar of the language. Between them they let you define a new language in which to talk about the problem area you are interested in. And doing that is the whole point of programming in Lisp: Lisp is about building languages to talk about problem areas.
An soon as you are little familiar to macros, you will wonder why you ever had this question. :-)
Macros are in no way alternatives to functions and neither vice versa. It just seems to be so, if you are working on the REPL, because macro expansion, compilation and running is happening within the moment you are pressing [enter].
Macros are running at compile time, so any macro-processing is finished, as son as your definition runs. There is no way to "call" a macro at the runtime of the definition that involves this very macro.
Macros just calculate S-exprs, that will be passed to the compiler.
Just think of a macro as something, that is coding for you.
This is easier to understand with little more code in your editor than with small definitions the REPL. Good luck!
In a CppCon talk (https://www.youtube.com/watch?v=80BZxujhY38 at 5:00)
Herb Sutter alluded to defun defun 3 being somehow a problem. After
I googled it, it's still not clear to me why. Can someone elaborate?
In a comment from the same video:
Herb Sutter
See also the paper, P0707 (http://wg21.link/p0707), and search for "defun." Lisp defun (and Scheme define) lets you define a function... but in Lisp and Scheme you can even redefine built-in functions and macros, including defun/define itself which is what "defun defun" / "define define" does. Here's a sample related StackExchange question: https://emacs.stackexchange.com/questions/375/symbols-value-as-a-variable-is-void-defun-when-reloading-emacs .
I'm not interested in doing anything like that in C++, and there's nothing like that in my proposal, you can't change any definitions (including of this class after it is defined), you can't reach out and affect anyone else's type or code, the only thing you can do is participate in generating the one-time-and-then-immutable definition of this class you're writing right now which is nice and localized and bounded... and still very powerful.
The linked paper contains this section:
5.2.1 Problems in other languages
In Lisp and related languages, programmers can redefine other people’s code and even global language facilities (e.g., the notorious (defun defun () 3) in Lisp, or (define define () 3) in Scheme). This is powerful, but undisciplined (causes arbitrary global effects up to and including breaking the language itself), fragile (Lisp makes it notoriously easy to write “write-only” code that is difficult to review, read, and maintain), and causes programs to be tightly coupled among their components and with their developer’s environment (Lisp makes it notoriously easy to write code whose meaning depends on local customizations, is hard to share, and when shared is hard to compose with other code that came from an environment with competing assumptions).4
The footnote says:
4 Various incarnations and offshoots of Lisp attempted to mitigate this problem in various ways without actually taking away the root cause: Common Lisp added the guarantee that all symbols in the package COMMON-LISP are protected and must not be redefined by user code otherwise you get undefined behavior; although this provides some protection for the standard facilities, it does not solve the general problem because it still permits one set of user code to redefine things in another set of user code. Also, implementations like SBCL attempted to further improve the problem by providing ways to “lock” packages so their contents cannot be accidentally redefined; however, even SBCL also provides ways to “unlock” them again.
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it. I have asked this on the Emacs forum with no success. See:
https://emacs.stackexchange.com/questions/35913/program-rewriting-systems-unexpanded-a-defmacro-given-a-list-of-macros-to-undo
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
Ideally, in certain situations such a tool would be able to work directly off the defmacro to do its pattern find and replace on. However even if I have to come up with specific search and replace patterns manually to add to the transformation system, having such a framework to work in would still be useful
Summary of results so far: Although there have been a few answers that explore interesting possibilities, right now there is nothing definitive. So I think best to leave this open. I'll summarize some of the suggestions. (I've upvoted all the answers that were in fact answers instead of commentary on the difficulty.)
First, many people suggest considered the special form of macros that do expansion only,or as Drew puts it:
macro-expansion (i.e., not expansion followed by Lisp evaluation).
Macro-expansion is another way of saying reduction semantics, or
rewriting.
The current front-runner to my mind is in phils post where he uses a pattern-matching facility that seems specific to Emacs: pcase. I will be exploring this and will post results of my findings. If anyone else has thoughts on this please chime in.
Drew wrote a program called FTOC whose purpose was to convert Franz Lisp to Common Lisp; googling turns up a comp.lang.lisp posting
I found a Common Lisp package called optima with fare-quasiquote. Paulo thinks however this might not be powerful enough since it doesn't handle backtracking out of the box, but might be programmed in by hand. Although the generality of backtracking might be nice, I'm not convinced I need that for the most-used situations.)
Side note: Some seem put off by the specific application causing my initial interest. (But note that in research, it is not uncommon for good solutions to get applied in ways not initially envisioned.)
So in that spirit, here are a couple of suggestions for changing the end application. A good solution for these would probably translate to a solution for Emacs Lisp. (And if if helps you to pretend I'm not interested in Emacs Lisp, that's okay with me). Instead of a decompiler for Emacs Lisp, suppose I want to write a decompiler for clojure or some Common Lisp system. Or as suggested by Sylwester's answer, suppose I would like to automatically refactor my code by taking into account the benefit of using more concise macros that exist or that have gotten improved. Recall that at one time Emacs Lisp didn't have "when" or "unless" macros.
30-some years ago I did something similar, using macrolet.
(Actually, I used defmacro because we had only an early implementation of Common Lisp, which did not yet have macrolet. But macrolet is the right thing to use.)
I didn't translate macro-expanded code to what it was expanded from, but the idea is pretty much the same. You will come across some different difficulties, I expect, since your translation is even farther away from one-to-one.
I wrote a translator from (what was then) Franz Lisp to Common Lisp, to help with porting lots of existing code to a Lisp+Prolog-machine project. Franz Lisp back then was only dynamically scoped, while Common Lisp is (in general) lexically scoped.
And yes, obviously there is no general way to automatically translate Lisp code (in particular), especially considering that it can generate and then evaluate other code - but even ignoring that special case. Many functions are quite similar, but there is the lexical/dynamic difference, as well as significant differences in the semantics of some seemingly similar functions.
All of that has to be understood and taken for granted from the outset, by anyone wanting to make use of the results of translation.
Still, much that is useful can be done. And if the resulting code is self-documenting, telling you what it was derived from etc., then when in the resulting context you can decide just what to do with this or that bit that might be tricky (e.g., rewrite it manually, from scratch or just tweak it). In practice, lots of code was easily converted from Franz to Common - it saved much reprogramming effort.
The translator program was written in Common Lisp. It could be used interactively as well as in batch. When used interactively it provided, in effect, a Franz Lisp interpreter on top of Common Lisp.
The program used only macro-expansion (i.e., not expansion followed by Lisp evaluation). Macro-expansion is another way of saying reduction semantics, or rewriting.
Input Franz-Lisp code was macro-expanded via function-definition mapping macros to produce Common-Lisp code. Code that was problematic for translation was flagged (in code) with a description/analysis that described the situation.
The program was called FTOC. I think you can still find it, or at least references to it, by googling (ftoc lisp). (It was the first Lisp program I wrote, and I still have fond memories of the experience. It was a good way to learn both Lisp dialects and to learn Lisp in general.)
Have fun!
In general, I don't think you can do this. The expansion of an lisp macro is Turing complete, so you have to be able to predict the output of a program which could have arbitrary input.
There are some simple things that you could do. defmacros with backquoted forms in appear fairly similar in the output form and might be detected. This sort of heuristic would probably get you a long way.
What I don't understand is your use case. The macro-expanded version of a piece of code is usually only present in the compiled (or in emacs-lisp byte-compiled) form.
Ok so other people have pointed out the fact that this problem is impossible in general. There are two hard parts to this problem: one is that it could be a lot of work to find a preimage of some code fragment through a macro and it is also impossible to determine whether a macro was called or not—there are examples where one may write code which could have come from a macro without using that macro. Imagine for the sake of illustration an sha macro which expands to the SHA hash of the string literal passed to it. Then if you see some sha hash in your expanded code, it would obviously be silly to try to unexpand it. But it may be that the hash was put into the code as a literal, e.g. referencing a specific point in the history of a git repository so it would also be unhelpful to unexpand the macro.
Tractable subproblems
Let me preface this by saying that whilst these may be a little tractable, I still wouldn’t try to solve this problem.
Let’s ignore all the macros that do weird things (like the example above) and all the macros that are just as likely to not have been used in the original (e.g. cond vs if) and all the macros which generate complex code which seems like it would be difficult to unravel (e.g. loop, do, and backquote. Annoyingly these difficult cases are some of those which you would perhaps most want to unexpand). The type this leaves us with (that I’d like to focus on) are macros which basically just reduce boilerplate, e.g. save-excursion or with-XXXX. These are macros whose implementation consists of possibly making some fresh symbols (via gensym) and then having a big simple backquoted block of code. I still think it would be too hard to automatically go from defmacro to a function for unexpansion but I think you could attack some of these on a case-by-case basis. Do this by looking for the forms generated by the macro that delimit (I.e. begin/end) the expanded code. I can’t really offer much beyond that. This is still a hard problem and I don’t think any existing solutions (to other problems) will get you very far on your way.
A further complication I understand is that you do not start at the macroexpanded code but rather at the bytecode. Without knowing anything about the elisp compiler, I worry that more information would be lost in the compilation step and you would have to undo that as well, e.g. perhaps it is hard to determine which code goes inside a let or even when a let begins, or bytecode starts using goto type features even though elisp doesn’t have them.
You suggest that the reason you would like to unexpand macros is so you can decompile bytecode which sometimes comes up in the Emacs debugger and that this would be useful as even though the source code is available in theory, it isn’t always at your fingertips. I put it to you that if you want to make your life debugging elisp easier it would be more worthwhile to figure out how to have the Emacs debugger always take you to the source code for internal functions. This might involve installing extra debugging related packages or downloading the Emacs source code and setting some variable so Emacs knows where to find it or compiling Emacs yourself from source. I don’t really know about that but I bet getting thrown into bytecode instead of source would have been enough of a problem for Emacs developers over the past thirty years that a solution to that problem does exist.
If however what you really want to do is to try to implement a decompiler for elisp then I suppose that’s what you should do. A final observation is that while Lisp provides facilities which make manipulating Lisp code easy, this doesn’t help much with decompiling as all these facilities can be used in compilation so there are infinitely more patterns one might want to detect than in e.g. a C decompiler. Perhaps scheme style macros would be easier to unexpand, although they would still be hard.
If you’re decompiling because you want to give a better idea of which exact subexpression rather than line is being evaluated (normally Lisp debuggers work on expressions not lines anyway) in the debugger then perhaps it would actually be useful to see the code at the expanded level rather than the unexpanded one. Or perhaps it would be best to see both and maybe in between as well. Keeping track of what’s what through forwards macroexpansion is already difficult and fiddly. Doing it in reverse certainly won’t be easier. Good luck!
Edit: seeing as your not currently using Lisp anyway, I wonder if you might have more success using something like prolog for your unexpanding. You’d still have to manually write rules but I think it would be a large amount of work to try to derive rules from macro definitions.
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it.
Macros generate arbitrary expressions, which may contain macros recursively. You have no general way to revert the transformations, because it's not pattern-based.
Even if macros were pattern-based, they could still be infinite.
Even if macros were not infinite, they can certainly contain bugs in expansions of patterns that never matched. Given arbitrary code to try to unwind, it could match an expansion that looks like the code and try to revert to its pattern. Without bugs, you could still abuse this.
Even if you could revert macro expansion, some macros expand to the same code. An approach could be signalling a warning with a restart when all reversions expand equally minus the operator, such that if the restart doesn't handle the signal, it would choose the first expansion; and otherwise signalling an error with a restart, such that if the restart doesn't handle the signal, it errors. Or you could configure it to choose certain macros under certain conditions, such as in which package the code was found.
In practice, there are very few cases where reverting an expansion makes any sense. It could be a useful development tool that suggests macros, but I wouldn't generally rely on it for whole source transformations.
One way you could achieve what you want is through a controlled pattern matching. You could initially create patterns manually, which would already handle cases you care about directly, such as the ones you mention:
(if (not <cond>) <expr>) and (if (not <cond>) (progn <&expr>)) to (unless <cond> <&expr>)
You'd have to decide whether null would be equivalent to not. I personally don't mix the boolean meaning of nil with that of empty list or something else, e.g. no result, nothing found, null object, a designator, etc. But perhaps Lisp code as old as that in Emacs just uses them interchangeably.
(if <cond> <expr>) and (if <cond> (progn <&expr>)) to (when <cond> <&expr>)
If you feel like improving code overall, include cond with a single condition. And be careful with cond clauses with only the condition.
You should have a few dozen more, to see how the pattern matching behaves with more patterns to match in terms of time (CPU) and space (memory).
From the description of fare-quasiquote, optima doesn't support backtracking, which you probably want.
But you can do backtracking with optima by yourself, using recursion on complex inner patterns, and if nothing matches, return a control value to keep searching for matching patterns from the outer input.
Another approach is to treat a pattern as a description of a state machine, and handle each new token to advance the current state machines until one of them reaches the end, discarding the state machines that couldn't advance. This approach may consume more memory, depending on the amount of patterns, the similarity between patterns (if many have the same starting token, many state machines will be generated on a matching token), the length of the patterns and, last but not least, the length of the input (s-expression).
An advantage of this approach is that you can use it interactively to see which patterns have matched the most tokens, and you can give weights to patterns instead of just taking the first that matches.
A disadvantage is that, most probably, you'll have to spend effort to develop it.
EDIT: I just lousily described a kind of trie or radix tree.
Once you got something working, maybe try to obtain patterns automatically. This is really hard, you must probably limit it to simple backquoting and accept the fact you can't generalize for anything that contains more complex code.
I believe the hardest will be code walking, which is hard enough with source code, but much more with macro-expanded code. Perhaps if you could expand the whole picture a bit further to understand the goal, maybe someone could suggest a better approach other than operating on macro-expanded code.
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
There's a huge step from expanding code with defmacro and all that generality. Most Lisp developers will know about hygienic macros, at least in terms of symbols as variables.
But there's still hygienic macros in terms of symbols as operators1, code walking, interaction with a containing macro (usually using macrolet), etc. It's way too complex.
1.
Common Lisp evaluates the operator in a compound form in the lexical environment, and probably everyone makes macros that assume that the global macro or function definition of a symbol will be used.
But it might not be so:
(defmacro my-macro-1 ()
`1)
(defmacro my-macro-2 ()
`(my-function (my-macro-1)))
(defun my-function (n)
(* n 100))
(macrolet ((my-macro-1 ()
`2))
(flet ((my-function (n)
(* n 1000)))
(my-macro-2)))
That last line will expand to (my-function (my-macro-2)), which will be recursively expanded to (my-function 2). When evaluated, it will yield 2000.
For proper operator hygiene, you'd have to do something like this:
(defmacro my-macro-2 ()
;; capture global bindings of my-macro-1 and my-function-1 by name
(flet ((my-macro-1-global (form env)
(funcall (macro-function 'my-macro-1) form env))
(my-function-global (&rest args)
;; hope the compiler can optimize this
(apply 'my-function args)))
;; store them globally in uninterned symbols
;; hopefully, no one will mess with them
(let ((my-macro-1-symbol (gensym (symbol-name 'my-macro-1)))
(my-function-symbol (gensym (symbol-name 'my-function))))
(setf (macro-function my-macro-1-symbol) #'my-macro-1-global)
(setf (symbol-function my-function-symbol) #'my-function-global)
`(,my-function-symbol (,my-macro-1-symbol)))))
With this definition, the example will yield 100.
Common Lisp has some restrictions to avoid this, but it only states the consequences are undefined when (re)defining symbols in the common-lisp package, globally or locally. It doesn't require errors or warnings to be signaled.
I don't think it is possible to do this in general, but you can undo a pattern back into a macro use for every match if you supply code for each unmacroing. Code that mixed cond and if will end up being just if and your code would remove all if into cond making the reverse not the same as the starting point. The more macros you have and the more they expand into each other the more uncertain of the end result will be of the starting point.
You could have rules such that if is not translated into cond unless you used one of the features, like more than one predicate or implicit progn, but you have no idea if the coder actually did use cond everywhere because he liked in consistent regardless. Thus your unmacroing will acyually be more of a simplification.
I don't believe there's a general solution to that, and you certainly
can't guarantee that the structure of the output would match that of
the original code, and I'm not going near the idea of auto-generating
patterns and desired transformations from macro definitions; but you
might achieve a simple version of this with Emacs' own pcase pattern
matching facility.
Here's the simplest example I could think of:
With reference to the definition of when:
(defmacro when (cond &rest body)
(list 'if cond (cons 'progn body)))
We can transform code using a pcase pattern like so:
(let ((form '(if (and foo bar baz) (progn do (all the) things))))
(pcase form
(`(if ,cond (progn . ,body))
`(when ,cond ,#body))
(_ form)))
=> (when (and foo bar baz) do (all the) things)
Obviously if the macro definitions change, then your patterns will
cease to work (but that's a pretty safe kind of failure).
Caveat: This is the first time I've written a pcase form, and I
don't know what I don't know. It seems to work as intended, though.
#lang racket/base
(module x scribble/text
#(display 123))
It seems like #lang statements are not valid in nested sub-modules, and the expanded module version above is missing something:
error: module: no #%module-begin binding in the module's language
update:
looks like this more or less works, but is there a better way? does scribble do something with output ports that isn't being handled?
#lang racket/base
(module x scribble/text/lang
(#%module-begin
#reader scribble/reader #list{
hi
#(+ 1 456)
}))
First, your code has a redundant #%module-begin which can be removed.
#lang does several things -- one is control the semantics of the file
by determining the set of initially imported bindings, and that's
something that the module form has done before #lang came up. With
submodules, it became possible to use module for parts of a file too.
However, #lang can also determine the reader that parses the file, and
that's not possible to do with submodules, so you're stuck with only one
toplevel #lang to set the parser for the whole file.
(Sidenote: There is a technical reason for that. A #lang reader reads
the rest of the file until it reaches an eof value, so a nested
#lang would require getting an eof value before getting the end of
the file, or adding a new kind of eof-like value. That means that it's
a change that should be done carefully -- it's possible to do, of
course, but the need didn't come up often enough. Hopefully it will, in
the future.)
But in your case you don't want a completely new concrete syntax, just
an extension for s-expressions -- and an extension that was chosen to
have a minimal impact on regular code. So in almost all cases it's fine
to just enable the #-form syntax for the whole file, and then use
#-forms where you want it. Since it's just an alternative way for
reading sexprs, you can even use that with module, leading to this
code that doesn't need to use #reader:
#lang at-exp racket/base
#module[x scribble/text/lang]{
hi
#(+ 1 456)
}
(require 'x)
One thing that is a bit strange here is using scribble/text/lang and
not just scribble/text. Usually, #lang foo is exactly the same as
(module x foo ...) after reading the code with foos reader. But in
the case of the scribble/text language there is another difference:
using it as a #lang makes the semantics of the module body be "output
each thing". The idea is that as a language you'll want to spit out
mostly-text files, but as a library you'll want to write code in it
and do the printout yourself.
Since this code uses module, using scribble/text means that you're
not getting the spit-all-out functionality, which is why you need to
explicitly switch to scribble/text/lang. But you could have instead
just do the spitting yourself using the language's output, which would
give you this code:
#lang at-exp racket/base
(module x racket/base
(require scribble/text)
(output #list{
hi
#(+ 1 456)}))
(require 'x)
Note that scribble/text is not used as a language here, since it
doesn't provide enough stuff to be one when used (outside of a #lang).
(Which you've found out, leading to that redundant #%module-begin...)
This version is slightly more verbose, but I'm guessing that it makes
more sense in your case, since using it for some part of the code means
that you want to use it as a library.
Finally, if you really don't want to read the whole file with the #
syntax, only some parts, then the #reader that you've found is
perfectly fine. (And this is made simple with scribble/text that
treats lists as concatenated outputs, so you need just one wrapper for
each chunk of text.)
My problem isn't with the built-in eval procedure but how to create a simplistic version of it. Just for starters I would like to be able to take this in '(+ 1 2) and have it evaluate the expression + where the quote usually takes off the evaluation.
I have been thinking about this and found a couple things that might be useful:
Unquote: ,
(quasiquote)
(apply)
My main problem is regaining the value of + as a procedure and not a symbol. Once I get that I think I should just be able to use it with the other contents of the list.
Any tips or guidance would be much appreciated.
Firstly, if you're doing what you're doing, you can't go wrong reading at least the first chapter of the Metalinguistic Abstraction section of Structure and Interpretation of Computer Programs.
Now for a few suggestions from myself.
The usual thing to do with a symbol for a Scheme (or, indeed, any Lisp) interpreter is to look it up in some sort of "environment". If you're going to write your own eval, you will likely want to provide your own environment structures to go with it. The one thing for which you could fall back to the Scheme system you're building your eval on top of is the initial environment containing bindings for things like +, cons etc.; this can't be achieved in a 100% portable way, as far as I know, due to various Scheme systems providing different means of getting at the initial environment (including the-environment special form in MIT Scheme and interaction-environment in (Petite) Chez Scheme... and don't ask me why this is so), but the basic idea stays the same:
(define (my-eval form env)
(cond ((self-evaluating? form) form)
((symbol? form)
;; note the following calls PCS's built-in eval
(if (my-kind-of-env? env)
(my-lookup form env)
;; apparently we're dealing with an environment
;; from the underlying Scheme system, so fall back to that
;; (note we call the built-in eval here)
(eval form env)))
;; "applicative forms" follow
;; -- special forms, macro / function calls
...))
Note that you will certainly want to check whether the symbol names a special form (lambda and if are necessary -- or you could use cond in place of if -- but you're likely to want more and possibly allow for extentions to the basic set, i.e. macros). With the above skeleton eval, this would have to take place in what I called the "applicative form" handlers, but you could also handle this where you deal with symbols, or maybe put special form handlers first, followed by regular symbol lookup and function application.