Related
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it. I have asked this on the Emacs forum with no success. See:
https://emacs.stackexchange.com/questions/35913/program-rewriting-systems-unexpanded-a-defmacro-given-a-list-of-macros-to-undo
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
Ideally, in certain situations such a tool would be able to work directly off the defmacro to do its pattern find and replace on. However even if I have to come up with specific search and replace patterns manually to add to the transformation system, having such a framework to work in would still be useful
Summary of results so far: Although there have been a few answers that explore interesting possibilities, right now there is nothing definitive. So I think best to leave this open. I'll summarize some of the suggestions. (I've upvoted all the answers that were in fact answers instead of commentary on the difficulty.)
First, many people suggest considered the special form of macros that do expansion only,or as Drew puts it:
macro-expansion (i.e., not expansion followed by Lisp evaluation).
Macro-expansion is another way of saying reduction semantics, or
rewriting.
The current front-runner to my mind is in phils post where he uses a pattern-matching facility that seems specific to Emacs: pcase. I will be exploring this and will post results of my findings. If anyone else has thoughts on this please chime in.
Drew wrote a program called FTOC whose purpose was to convert Franz Lisp to Common Lisp; googling turns up a comp.lang.lisp posting
I found a Common Lisp package called optima with fare-quasiquote. Paulo thinks however this might not be powerful enough since it doesn't handle backtracking out of the box, but might be programmed in by hand. Although the generality of backtracking might be nice, I'm not convinced I need that for the most-used situations.)
Side note: Some seem put off by the specific application causing my initial interest. (But note that in research, it is not uncommon for good solutions to get applied in ways not initially envisioned.)
So in that spirit, here are a couple of suggestions for changing the end application. A good solution for these would probably translate to a solution for Emacs Lisp. (And if if helps you to pretend I'm not interested in Emacs Lisp, that's okay with me). Instead of a decompiler for Emacs Lisp, suppose I want to write a decompiler for clojure or some Common Lisp system. Or as suggested by Sylwester's answer, suppose I would like to automatically refactor my code by taking into account the benefit of using more concise macros that exist or that have gotten improved. Recall that at one time Emacs Lisp didn't have "when" or "unless" macros.
30-some years ago I did something similar, using macrolet.
(Actually, I used defmacro because we had only an early implementation of Common Lisp, which did not yet have macrolet. But macrolet is the right thing to use.)
I didn't translate macro-expanded code to what it was expanded from, but the idea is pretty much the same. You will come across some different difficulties, I expect, since your translation is even farther away from one-to-one.
I wrote a translator from (what was then) Franz Lisp to Common Lisp, to help with porting lots of existing code to a Lisp+Prolog-machine project. Franz Lisp back then was only dynamically scoped, while Common Lisp is (in general) lexically scoped.
And yes, obviously there is no general way to automatically translate Lisp code (in particular), especially considering that it can generate and then evaluate other code - but even ignoring that special case. Many functions are quite similar, but there is the lexical/dynamic difference, as well as significant differences in the semantics of some seemingly similar functions.
All of that has to be understood and taken for granted from the outset, by anyone wanting to make use of the results of translation.
Still, much that is useful can be done. And if the resulting code is self-documenting, telling you what it was derived from etc., then when in the resulting context you can decide just what to do with this or that bit that might be tricky (e.g., rewrite it manually, from scratch or just tweak it). In practice, lots of code was easily converted from Franz to Common - it saved much reprogramming effort.
The translator program was written in Common Lisp. It could be used interactively as well as in batch. When used interactively it provided, in effect, a Franz Lisp interpreter on top of Common Lisp.
The program used only macro-expansion (i.e., not expansion followed by Lisp evaluation). Macro-expansion is another way of saying reduction semantics, or rewriting.
Input Franz-Lisp code was macro-expanded via function-definition mapping macros to produce Common-Lisp code. Code that was problematic for translation was flagged (in code) with a description/analysis that described the situation.
The program was called FTOC. I think you can still find it, or at least references to it, by googling (ftoc lisp). (It was the first Lisp program I wrote, and I still have fond memories of the experience. It was a good way to learn both Lisp dialects and to learn Lisp in general.)
Have fun!
In general, I don't think you can do this. The expansion of an lisp macro is Turing complete, so you have to be able to predict the output of a program which could have arbitrary input.
There are some simple things that you could do. defmacros with backquoted forms in appear fairly similar in the output form and might be detected. This sort of heuristic would probably get you a long way.
What I don't understand is your use case. The macro-expanded version of a piece of code is usually only present in the compiled (or in emacs-lisp byte-compiled) form.
Ok so other people have pointed out the fact that this problem is impossible in general. There are two hard parts to this problem: one is that it could be a lot of work to find a preimage of some code fragment through a macro and it is also impossible to determine whether a macro was called or not—there are examples where one may write code which could have come from a macro without using that macro. Imagine for the sake of illustration an sha macro which expands to the SHA hash of the string literal passed to it. Then if you see some sha hash in your expanded code, it would obviously be silly to try to unexpand it. But it may be that the hash was put into the code as a literal, e.g. referencing a specific point in the history of a git repository so it would also be unhelpful to unexpand the macro.
Tractable subproblems
Let me preface this by saying that whilst these may be a little tractable, I still wouldn’t try to solve this problem.
Let’s ignore all the macros that do weird things (like the example above) and all the macros that are just as likely to not have been used in the original (e.g. cond vs if) and all the macros which generate complex code which seems like it would be difficult to unravel (e.g. loop, do, and backquote. Annoyingly these difficult cases are some of those which you would perhaps most want to unexpand). The type this leaves us with (that I’d like to focus on) are macros which basically just reduce boilerplate, e.g. save-excursion or with-XXXX. These are macros whose implementation consists of possibly making some fresh symbols (via gensym) and then having a big simple backquoted block of code. I still think it would be too hard to automatically go from defmacro to a function for unexpansion but I think you could attack some of these on a case-by-case basis. Do this by looking for the forms generated by the macro that delimit (I.e. begin/end) the expanded code. I can’t really offer much beyond that. This is still a hard problem and I don’t think any existing solutions (to other problems) will get you very far on your way.
A further complication I understand is that you do not start at the macroexpanded code but rather at the bytecode. Without knowing anything about the elisp compiler, I worry that more information would be lost in the compilation step and you would have to undo that as well, e.g. perhaps it is hard to determine which code goes inside a let or even when a let begins, or bytecode starts using goto type features even though elisp doesn’t have them.
You suggest that the reason you would like to unexpand macros is so you can decompile bytecode which sometimes comes up in the Emacs debugger and that this would be useful as even though the source code is available in theory, it isn’t always at your fingertips. I put it to you that if you want to make your life debugging elisp easier it would be more worthwhile to figure out how to have the Emacs debugger always take you to the source code for internal functions. This might involve installing extra debugging related packages or downloading the Emacs source code and setting some variable so Emacs knows where to find it or compiling Emacs yourself from source. I don’t really know about that but I bet getting thrown into bytecode instead of source would have been enough of a problem for Emacs developers over the past thirty years that a solution to that problem does exist.
If however what you really want to do is to try to implement a decompiler for elisp then I suppose that’s what you should do. A final observation is that while Lisp provides facilities which make manipulating Lisp code easy, this doesn’t help much with decompiling as all these facilities can be used in compilation so there are infinitely more patterns one might want to detect than in e.g. a C decompiler. Perhaps scheme style macros would be easier to unexpand, although they would still be hard.
If you’re decompiling because you want to give a better idea of which exact subexpression rather than line is being evaluated (normally Lisp debuggers work on expressions not lines anyway) in the debugger then perhaps it would actually be useful to see the code at the expanded level rather than the unexpanded one. Or perhaps it would be best to see both and maybe in between as well. Keeping track of what’s what through forwards macroexpansion is already difficult and fiddly. Doing it in reverse certainly won’t be easier. Good luck!
Edit: seeing as your not currently using Lisp anyway, I wonder if you might have more success using something like prolog for your unexpanding. You’d still have to manually write rules but I think it would be a large amount of work to try to derive rules from macro definitions.
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it.
Macros generate arbitrary expressions, which may contain macros recursively. You have no general way to revert the transformations, because it's not pattern-based.
Even if macros were pattern-based, they could still be infinite.
Even if macros were not infinite, they can certainly contain bugs in expansions of patterns that never matched. Given arbitrary code to try to unwind, it could match an expansion that looks like the code and try to revert to its pattern. Without bugs, you could still abuse this.
Even if you could revert macro expansion, some macros expand to the same code. An approach could be signalling a warning with a restart when all reversions expand equally minus the operator, such that if the restart doesn't handle the signal, it would choose the first expansion; and otherwise signalling an error with a restart, such that if the restart doesn't handle the signal, it errors. Or you could configure it to choose certain macros under certain conditions, such as in which package the code was found.
In practice, there are very few cases where reverting an expansion makes any sense. It could be a useful development tool that suggests macros, but I wouldn't generally rely on it for whole source transformations.
One way you could achieve what you want is through a controlled pattern matching. You could initially create patterns manually, which would already handle cases you care about directly, such as the ones you mention:
(if (not <cond>) <expr>) and (if (not <cond>) (progn <&expr>)) to (unless <cond> <&expr>)
You'd have to decide whether null would be equivalent to not. I personally don't mix the boolean meaning of nil with that of empty list or something else, e.g. no result, nothing found, null object, a designator, etc. But perhaps Lisp code as old as that in Emacs just uses them interchangeably.
(if <cond> <expr>) and (if <cond> (progn <&expr>)) to (when <cond> <&expr>)
If you feel like improving code overall, include cond with a single condition. And be careful with cond clauses with only the condition.
You should have a few dozen more, to see how the pattern matching behaves with more patterns to match in terms of time (CPU) and space (memory).
From the description of fare-quasiquote, optima doesn't support backtracking, which you probably want.
But you can do backtracking with optima by yourself, using recursion on complex inner patterns, and if nothing matches, return a control value to keep searching for matching patterns from the outer input.
Another approach is to treat a pattern as a description of a state machine, and handle each new token to advance the current state machines until one of them reaches the end, discarding the state machines that couldn't advance. This approach may consume more memory, depending on the amount of patterns, the similarity between patterns (if many have the same starting token, many state machines will be generated on a matching token), the length of the patterns and, last but not least, the length of the input (s-expression).
An advantage of this approach is that you can use it interactively to see which patterns have matched the most tokens, and you can give weights to patterns instead of just taking the first that matches.
A disadvantage is that, most probably, you'll have to spend effort to develop it.
EDIT: I just lousily described a kind of trie or radix tree.
Once you got something working, maybe try to obtain patterns automatically. This is really hard, you must probably limit it to simple backquoting and accept the fact you can't generalize for anything that contains more complex code.
I believe the hardest will be code walking, which is hard enough with source code, but much more with macro-expanded code. Perhaps if you could expand the whole picture a bit further to understand the goal, maybe someone could suggest a better approach other than operating on macro-expanded code.
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
There's a huge step from expanding code with defmacro and all that generality. Most Lisp developers will know about hygienic macros, at least in terms of symbols as variables.
But there's still hygienic macros in terms of symbols as operators1, code walking, interaction with a containing macro (usually using macrolet), etc. It's way too complex.
1.
Common Lisp evaluates the operator in a compound form in the lexical environment, and probably everyone makes macros that assume that the global macro or function definition of a symbol will be used.
But it might not be so:
(defmacro my-macro-1 ()
`1)
(defmacro my-macro-2 ()
`(my-function (my-macro-1)))
(defun my-function (n)
(* n 100))
(macrolet ((my-macro-1 ()
`2))
(flet ((my-function (n)
(* n 1000)))
(my-macro-2)))
That last line will expand to (my-function (my-macro-2)), which will be recursively expanded to (my-function 2). When evaluated, it will yield 2000.
For proper operator hygiene, you'd have to do something like this:
(defmacro my-macro-2 ()
;; capture global bindings of my-macro-1 and my-function-1 by name
(flet ((my-macro-1-global (form env)
(funcall (macro-function 'my-macro-1) form env))
(my-function-global (&rest args)
;; hope the compiler can optimize this
(apply 'my-function args)))
;; store them globally in uninterned symbols
;; hopefully, no one will mess with them
(let ((my-macro-1-symbol (gensym (symbol-name 'my-macro-1)))
(my-function-symbol (gensym (symbol-name 'my-function))))
(setf (macro-function my-macro-1-symbol) #'my-macro-1-global)
(setf (symbol-function my-function-symbol) #'my-function-global)
`(,my-function-symbol (,my-macro-1-symbol)))))
With this definition, the example will yield 100.
Common Lisp has some restrictions to avoid this, but it only states the consequences are undefined when (re)defining symbols in the common-lisp package, globally or locally. It doesn't require errors or warnings to be signaled.
I don't think it is possible to do this in general, but you can undo a pattern back into a macro use for every match if you supply code for each unmacroing. Code that mixed cond and if will end up being just if and your code would remove all if into cond making the reverse not the same as the starting point. The more macros you have and the more they expand into each other the more uncertain of the end result will be of the starting point.
You could have rules such that if is not translated into cond unless you used one of the features, like more than one predicate or implicit progn, but you have no idea if the coder actually did use cond everywhere because he liked in consistent regardless. Thus your unmacroing will acyually be more of a simplification.
I don't believe there's a general solution to that, and you certainly
can't guarantee that the structure of the output would match that of
the original code, and I'm not going near the idea of auto-generating
patterns and desired transformations from macro definitions; but you
might achieve a simple version of this with Emacs' own pcase pattern
matching facility.
Here's the simplest example I could think of:
With reference to the definition of when:
(defmacro when (cond &rest body)
(list 'if cond (cons 'progn body)))
We can transform code using a pcase pattern like so:
(let ((form '(if (and foo bar baz) (progn do (all the) things))))
(pcase form
(`(if ,cond (progn . ,body))
`(when ,cond ,#body))
(_ form)))
=> (when (and foo bar baz) do (all the) things)
Obviously if the macro definitions change, then your patterns will
cease to work (but that's a pretty safe kind of failure).
Caveat: This is the first time I've written a pcase form, and I
don't know what I don't know. It seems to work as intended, though.
Tried to use the persp-mode https://github.com/Bad-ptr/persp-mode.el/blob/master/persp-mode.el to retrieve the emacs windows session after restart. Unable to get it working.
So trying to understand the datastructure used to store the state of the emacs by reading the source code.
The following is the function which is used to store the session state.
(defun* persp-save-state-to-file (&optional (fname persp-auto-save-fname)
(phash *persp-hash*)
respect-persp-file-parameter)
(interactive (list (read-file-name "Save perspectives to file: "
persp-save-dir)))
In the above function two unusual things are observed using edebug (unusual according to my current understanding of elisp).
The optional argument expressions are evaluated.
The expression "(interactive..." is evaluated first and then the optional argument expressions are evaluated.
Any ideas how to debug the code. Also the emacs documentation says "defun*" is related to common-lisp, but no further information is available in emacs docs on how defun* is different from defun. Is there a quick tutorial oh what defun* does without having to learn common-lisp.
Emacs says:
Define NAME as a function. Like normal `defun', except ARGLIST allows
full Common Lisp conventions, and BODY is implicitly surrounded by
(cl-block NAME ...).
Common Lisp arglists provide optional, rest, keyword and aux arguments. Historically this comes from Lisp Machine Lisp and Mumble - two earlier Lisp dialects.
For details see: http://www.gnu.org/software/emacs/manual/html_node/cl/Argument-Lists.html
Have a look at this post for a simplistic snippet explaining
how optional works. The gist is that e.g. persp-auto-save-fname will be the value of fname
if none is given.
And obviously interactive has to be run first, because it provides the arguments.
So if interactive doesn't provide a value for fname it will be persp-auto-save-fname.
I have seen one answer of How does Lisp let you redefine the language itself?
Stack Overflow question (answered by Noah Lavine):
Macros aren't quite a complete redefinition of the language, at least as far as I know (I'm actually a Schemer; I could be wrong), because there is a restriction. A macro can only take a single subtree of your code, and generate a single subtree to replace it. Therefore you can't write whole-program-transforming macros, as cool as that would be.
After reading this I am curious about whether there are "whole-program-transforming macros" in Lisp or Scheme (or some other language).
If not then why?
It is not useful and never required?
Same thing could be achieved by some other ways?
It is not possible to implement it even in Lisp?
It is possible, but not tried or implemented ever?
Update
One kind of use case
e.g.
As in stumpwm code
here are some functions all in different lisp source files
uses a dynamic/global defvar variable *screen-list* that is defined in primitives.lisp , but used in screen.lisp, user.lisp, window.lisp.
(Here each files have functions, class, vars related to one aspect or object)
Now I wanted to define these functions under the closure where
*screen-list* variable available by let form, it should not be
dynamic/global variable, But without moving these all functions into
one place (because I do not want these functions to lose place from their
related file)
So that this variable will be accessible to only these functions.
Above e.g. equally apply to label and flet, so that it will further possible
that we could make it like that only required variable, function will be available,
to those who require it.
Note one way might be
implement and use some macro defun_with_context for defun where first argument is
context where let, flet variables definend.
But apart from it could it be achieved by reader-macro as
Vatine and Gareth Rees answered.
You quoted Noah Lavine as saying:
A macro can only take a single subtree of your code, and generate a single subtree to replace it
This is the case for ordinary macros, but reader macros get access to the input stream and can do whatever they like with it.
See the Hyperspec section 2.2 and the set-macro-character function.
In Racket, you can implement whole-program-transforming macros. See the section in the documentation about defining new languages. There are many examples of this in Racket, for example the lazy language and Typed Racket.
Off the top of my head, a few approaches:
First, you can. Norvig points out that:
We can write a compiler as a set of macros.
so you can transform an entire program, if you want to. I've only seen it done rarely, because typically the intersection between "things you want to do to every part of your program" and "things that you need macro/AST-type transformations for" is a pretty small set. One example is Parenscript, which transforms your Lisp code ("an extended subset of CL") into Javascript. I've used it to compile entire files of Lisp code into Javascript which is served directly to web clients. It's not my favorite environment, but it does what it advertises.
Another related feature is "advice", which Yegge describes as:
Great systems also have advice. There's no universally accepted name for this feature. Sometimes it's called hooks, or filters, or aspect-oriented programming. As far as I know, Lisp had it first, and it's called advice in Lisp. Advice is a mini-framework that provides before, around, and after hooks by which you can programmatically modify the behavior of some action or function call in the system.
Another is special variables. Typically macros (and other constructs) apply to lexical scope. By declaring a variable to be special, you're telling it to apply to dynamic scope (I think of it as "temporal scope"). I can't think of any other language that lets you (the programmer) choose between these two. And, apart from the compiler case, these two really span the space that I'm interested in as a programmer.
A typical approach is to write your own module system. If you just want access to all the code, you can have some sort of pre-processor or reader extension wrap source files with your own module annotation. If you then write your own require or import form, you will ultimately be able to see all the code in scope.
To get started, you could write your own module form that lets you define several functions which you then compile in some clever way before emitting optimized code.
There's always the choice of using compiler macros (they can do whole-function transformation based on a lew of criteria, but shouldn't change the value returned, as that would be confusing).
There's reader macros, they transform the input "as it is read" (or "before it is read", if you prefer). I haven't done much large-scale reader-macro hacking, but I have written some code to allow elisp sourec to be (mostly) read in Common Lisp, with quite a few subtle differences in syntactic sugar between the two.
I believe those sorts of macros are called code-walking macros. I haven't implemented a code walker myself, so I am not familiar with the limits.
In Common LISP, at least, you may wrap top-level forms in PROGN and they still retain their status as top-level forms (see CLTL2, section 5.3). Therefore, the limitation of a macro generating a single subtree is not much of a limitation since it could wrap any number of resulting subtrees within PROGN. This makes whole-program macros quite possible.
E.g.
(my-whole-program-macro ...)
= expands to =>
(progn
(load-system ...)
(defvar ...)
(defconstant ...)
(defmacro ...)
(defclass ...)
(defstruct ...)
(defun ...)
(defun ...)
...
)
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I've read The Nature of Lisp. The only thing I really got out of that was "code is data." But without defining what these terms mean and why they are usually thought to be separate, I gain no insight. My initial reaction to "code is data" is, so what?
The old fashioned view: 'it' is interactive computation with symbolic expressions.
Lisp enables easy representation of all kinds of expressions:
english sentence
(the man saw the moon)
math
(2 * x ** 3 + 4 * x ** 2 - 3 * x + 3)
rules
(<- (likes Kim ?x) (likes ?x Lee) (likes ?x Kim))
and also Lisp itself
(mapcar (function sqr) (quote (1 2 3 4 5)))
and many many many more.
Lisp now allows to write programs that compute with such expressions:
(translate (quote (the man saw the moon)) (quote german))
(solve (quote (2 * x ** 3 + 4 * x ** 2 - 3 * x + 3)) (quote (x . 3)))
(show-all (quote (<- (likes Kim ?x) (likes ?x Lee) (likes ?x Kim))))
(eval (quote (mapcar (function sqr) (quote (1 2 3 4 5)))))
Interactive means that programming is a dialog with Lisp. You enter an expression and Lisp computes the side effects (for example output) and the value.
So your programming session is like 'talking' with the Lisp system. You work with it until you get the right answers.
What are these expressions? They are sentences in some language. They are part descriptions of turbines. They are theorems describing a floating point engine of an AMD processor. They are computer algebra expressions in physics. They are descriptions of circuits. They are rules in a game. They are descriptions of behavior of actors in games. They are rules in a medical diagnosis system.
Lisp allows you to write down facts, rules, formulas as symbolic expressions. It allows you to write programs that work with these expressions. You can compute the value of a formula. But you can equally easy write programs that compute new formulas from formulas (symbolic math: integrate, derive, ...). That was Lisp designed for.
As a side effect Lisp programs are represented as such expressions too. Then there is also a Lisp program that evaluates or compiles other Lisp programs. So the very idea of Lisp, the computation with symbolic expressions, has been applied to Lisp itself. Lisp programs are symbolic expressions and the computation is a Lisp expression.
Alan Kay (of Smalltalk fame) calls the original definition of Lisp evaluation in Lisp the Maxwell's equations of programming.
Write Lisp code. The only way to really 'get' Lisp (or any language, for that matter) is to roll up your sleeves and implement some things in it. Like anything else, you can read all you want, but if you want to really get a firm grasp on what's going on, you've got to step outside the theoretical and start working with the practical.
The way you "get" any language is by trying to write some code in it.
About the "data is code" thing, in most languages there is a clear separation between the code that gets executed, and the data that is processed.
For example, the following simple C-like function:
void foo(int i){
int j;
if (i % 42 == 0){
bar(i-2);
}
for (j = 0; j < i; ++j){
baz();
}
}
the actual control flow is determined once, statically, while writing the code. The function bar isn't going to change, and the if statement at the beginning of the function isn't going to disappear. This code is not data, it can not be manipulated by the program.
All that can be manipulated is the initial value of i. And on the other hand, that value can not be executed the way code can. You can call the function foo, but you can't call the variable i. So i is data, but it is not code.
Lisp does not have this distinction. The program code is data that can be manipulated too. Your code can, at runtime, take the function foo, and perhaps add another if statement, perhaps change the condition in the for-loop, perhaps replace the call to baz with another function call. All your code is data that can be inspected and manipulated as simply as the above function can inspect and manipulate the integer i.
I would highly recommend Structure and Interpretation of Computer Programs, which actually uses scheme, but that is a dialect of lisp. It will help you "get" lisp by having you do many different exercises and goes on to show some of the ways that lisp is so usefull.
I think you have to have more empathy for compiler writers to understand how fundamental the code is data thing is. I'll admit, I've never taken a compilers course, but converting any sufficiently high-level language into machine code is a hard problem, and LISP, in many ways, resembles an intermediate step in this process. In the same way that C is "close to the metal", LISP is close to the compiler.
This worked for me:
Read "The Little Schemer". It's the shortest path to get you thinking in Lisp mode (minus the macros). As a bonus, it's relatively short/fun/inexpensive.
Find a good book/tutorial to get you started with macros. I found chapter 8 of "The Scheme
Programming Language" to be a good starting point for Scheme.
http://www.ccs.neu.edu/home/matthias/BTLS/
http://www.scheme.com/tspl3/syntax.html
By watching legendary Structure and Interpretation of Computer Programs?
In Common Lisp, "code is data" boils down to this. When you write, for example:
(add 1 2)
your Lisp system will parse that text and generate a list with three elements: the symbol ADD, and the numbers 1 and 2. So now they're data. You can do whatever you want with them, replace elements, insert other stuff, etc.
The fun part is that you can pass this data on to the compiler and, because you can manipulate these data structures using Lisp itself, this means you can write programs that write other programs. This is not as complicated as it sounds, and Lispers do it all the time using macros. So, just get a book about Lisp, and try it out.
Okay, I'm going to take a crack at this. I'm new to Lisp myself, just having arrived from the world of python. I haven't experienced that sudden moment of enlightenment that all the old Lispers talk about, but I'll tell you what I am seeing so far.
First, look at this random bit of python code:
def is_palindrome(st):
l = len(st)/2
return True if st[:l] == st[:-l-1:-1] else False
Now look at this:
"""
def is_palindrome(st):
l = len(st)/2
return True if st[:l] == st[:-l-1:-1] else False
"""
What do you, as a programmer, see? The code is identical, FYI.
If you are like me, you'll tend to think of the first as active code. It consists of a number of syntactic elements.
The second, despite its similarity, is a single syntactic item. It's a string. You interact with it as a single entity. To deal with it as code - to handle it comfortably along its syntactic boundaries - you will have to do some parsing. To execute it, you need to invoke an interpreter. It's not the same thing at all as the first.
So when we do code generation in most languages what are we dealing with? Strings. When I generate HTML or SQL with python I use python strings as the interface between the two languages. Even if I generate python with python, strings are the tool.*
Doesn't the thought of that just... make you want to dance with joy? There's always this grotesque mismatch between that which you are working with and that which you are working on. I sensed that the first time that I generated SQL with perl. Differences in escaping. Differences in formatting: think about trying to get a generated html document to look tidy. Stuff isn't easy to reuse. Etc.
To solve the problem we serially create templating libraries. Scads of them. Why so many? My guess is that they're never quite satisfactory. By the time they start getting powerful enough they've turned into monstrosities. Granted, some of them - such as SQLAlchemy and Genshi in the python world - are very beautiful and admirable monstrosities. Let's... um... avoid mention of PHP.
Because strings make an awkward interface between the worked-on language and the worked-with, we create a third language - templates - to avoid them. ** This also tends to be a little awkward.
Now let's look at a block of quoted Lisp code:
'(loop for i from 1 to 8 do (print i))
What do you see? As a new Lisp coder, I've caught myself looking at that as a string. It isn't. It is inactive Lisp code. You are looking at a bunch of lists and symbols. Try to evaluate it after turning one of the parentheses around. The language won't let you do it: syntax is enforced.
Using quasiquote, we can shoehorn our own values into this inactive Lisp code:
`(loop for i from 1 to ,whatever do (print i))
Note the nature of the shoehorning: one item has been replaced with another. We aren't formatting our value into a string. We're sliding it into a slot in the code. It's all neat and tidy.
In fact if you want to directly edit the text of the code, you are in for a hassle. For example if you are inserting a name <varname> into the code, and you also want to use <varname>-tmp in the same code you can't do it directly like you can with a template string: "%s-tmp = %s". You have to extract the name into a string, rewrite the string, then turn it into a symbol again and finally insert.
If you want to grasp the essence of Lisp, I think that you might gain more by ignoring defmacro and gensyms and all that window dressing for the moment. Spend some time exploring the potential of the quasiquote, including the ,# thing. It's pretty accessible. Defmacro itself only provides an easy way to execute the result of quasiquotes. ***
What you should notice is that the hermetic string/template barrier between the worked-on and the worked-with is all but eliminated in Lisp. As you use it, you'll find that your sense of two distinct layers - active and passive - tends to dissipate. Functions call macros which call macros or functions which have functions (or macros!) passed in with their arguments. It's kind of a big soup - a little shocking for the newcomer. That said, I don't find that the distinction between macros and functions is as seamless as some Lisp people say. Mostly it's ok, but every so often as I wander in the soup I find myself bumping up against the ghost of that old barrier - and it really creeps me out!
I'll get over it, I'm sure. No matter. The convenience pays for the scare.
Now that's Lisp working on Lisp. What about working on other languages? I'm not quite there yet, personally, but I think I see the light at the end of the tunnel. You know how Lisp people keep going on about S-expressions being the same thing as a parse tree? I think the idea is to parse the foreign language into S-expressions, work on them in the amazing comfort of the Lisp environment, then send them back to native code. In theory, every language out there could be turned into S-expressions, or even executable lisp code. You're not working in a first language combined with a third language to produce code in a second language. It is all - while you are working on it - Lisp, and you can generate it all with quasiquotes.
Have a look at this (borrowed from PCL):
(define-html-macro :mp3-browser-page ((&key title (header title)) &body body)
`(:html
(:head
(:title ,title)
(:link :rel "stylesheet" :type "text/css" :href "mp3-browser.css"))
(:body
(standard-header)
(when ,header (html (:h1 :class "title" ,header)))
,#body
(standard-footer))))
Looks like an S-expression version of HTML, doesn't it? I have a feeling that Lisp works just fine as its own templating library.
I've started to wonder about an S-expression version of python. Would it qualify as a Lisp? It certainly wouldn't be Common Lisp. Maybe it would be nicer - for python programmers at least. Hey, and what about P-expressions?
* Python now has something called AST, which I haven't explored. Also a person could use python lists to represent other languages. Relative to Lisp, I suspect that both are a bit of a hack.
** SQLAlchemy is kind of an exception. It's done a nice job of turning SQL directly into python. That said, it appears to have involved significant effort.
*** Take it from a newbie. I'm sure I'm glossing over something here. Also, I realize that quasiquote is not the only way to generate code for macros. It's certainly a nice one, though.
Data is code is an interesting paradigm that supports treating a data structure as a command. Treating data in this way allows you to process and manipulate the structure in various ways - e.g. traversal - by evaluating it. Moreover, the 'data is code' paradigm obviates the need in many cases to develop custom parsers for data structures; the language parser itself can be used to parse the structures.
The first step is forgetting everything you have learned with all the C and Pascal-like languages. Empty your mind. This is the hardest step.
Then, take a good introduction to programming that uses Lisp. Don't try to correlate what you see with anything that you know beforehand (when you catch yourself doing that, repeat step 1). I liked Structure and Interpretation of Computer Programs (uses Scheme), Practical Common Lisp, Paradigms of Artificial Intelligence Programming, Casting Spels in Lisp, among others. Be sure to write out the examples. Also try the exercises, but limit yourself to the constructs you have learned in that book. If you find yourself trying to find, for example, some function to set a variable or some statement that resembles a for loop, repeat step 1, then revisit the chapters before to find out how it is done in Lisp.
Read and understand the legendary page 13 of the Lisp 1.5 Programmer's Manual
According to Alan Kay, at least.
One of the reasons that some university computer science programs use Lisp for their intro courses is that it's generally true that a novice can learn functional, procedural, or object-oriented programming more or less equally well. However, it's much harder for someone who already thinks in procedural statements to begin thinking like a functional programmer than to do the inverse.
When I tried to pick up Lisp, I did it "with a C accent." set! amd begin were my friends and constant companions. It is surprisingly easy to write Lisp code without ever writing any functional code, which isn't the point.
You may notice that I'm not answering your question, which is true. I merely wanted to let you know that it's awfully hard to get your mind thinking in a functional style, and it'll be an exciting practice that will make you a stronger programmer in the long run.
Kampai!
P.S. Also, you'll finally understand that "my other car is a cdr" bumper sticker.
To truly grok lisp, you need to write it.
Learn to love car, cdr, and cons. Don't iterate when you can recurse. Start out writing some simple programs (factorial, list reversal, dictionary lookup), and work your way up to more complex ones (sorting sets of items, pattern matching).
On the code is data and data is code thing, I wouldn't worry about it at this point. You'll understand it eventually, and its not critical to learning lisp.
I would suggest checking out some of the newer variants of Lisp like Arc or Clojure. They clean up the syntax a little and are smaller and thus easier to understand than Common Lisp. Clojure would be my choice. It is written on the JVM and so you don't have the issues with various platform implementations and library support that exist with some Lisp implementations like SBCL.
Read On Lisp and Paradigms in Artificial Intelligence Programming. Both of these have excellent coverage of Lisp macros - which really make the code is data concept real.
Also, when writing Lisp, don't iterate when you can recurse or map (learn to love mapcar).
it's important to see that data is code AND code is data. This feeds the eval/apply loop. Recursion is also fun.
(This link is broken:
![Eval/Apply][1]
[1]: http://ely.ath.cx/~piranha/random_images/lolcode-eval_apply-2.jpg
)
I'd suggest that is a horrible introduction to the language. There are better places to start and better people/articles/books than the one you cited.
Are you a programmer? What language(s)?
To help you with your question more background might be helpful.
About the whole "code is data" thing:
Isn't that due to the "von Neumann architecture"? If code and data were located in physically separate memory locations, the bits in the data memory could not be executed whereas the bits in the program memory could not be interpreted as anything but instructions to the CPU.
Do I understand this correctly?
I think to learn anything you have to have a purpose for it, such as a simple project.
For Lisp, a good simple project is a symbolic differentiator, so for example
(diff 'x 'x) -> 1
(diff 'a 'x) -> 0
(diff `(+ ,xx ,yy) 'x) where xx and yy are subexpressions
-> `(+ ,(diff xx 'x),(diff yy 'x))
etc. etc.
and then you need a simplifier, such as
(simp `(+ ,x 0)) -> x
(simp `(* ,x 0)) -> 0
etc. etc.
so if you start with a math expression, you can eval it to get its value, and you can eval its derivative to get its derivative.
I hope this illustrates what can happen when program code manipulates program code.
As Marvin Minsky observed, computer math is always worried about accuracy and roundoff error, right? Well, this is either exactly right or completely wrong!
You can get LISP in many ways, the most common is by using Emacs or working next to somebody who has developed LISP already.
Sadly, once you get LISP, it's hard to get rid of it, antibiotics won't work.
BTW: I also recommend The Adventures of a Pythonista in Schemeland.
This may be helpful: http://www.defmacro.org/ramblings/fp.html (isn't about LISP but about functional programming as a paradigm)
The way I think about it is that the best part of "code is data" is the face that function are, well, functionally no different than another variable. The fact that you can write code that writes code is one of the single most powerful (and often overlooked) features of Lisp. Functions can accept other functions as parameters, and even return functions as a result.
This lets one code at a much higher level of abstraction than, say, Java. It makes many tasks elegant and concise, and therefore, makes the code easier to modify, maintain, and read, or at least in theory.
I would say that the only way to truly "get" Lisp is to spend a lot of time with it -- once you get the hang of it, you'll wish you had some of the features of Lisp in your other programming languages.
Reading Paul Graham's essays on programming languages one would think that Lisp macros are the only way to go. As a busy developer, working on other platforms, I have not had the privilege of using Lisp macros. As someone who wants to understand the buzz, please explain what makes this feature so powerful.
Please also relate this to something I would understand from the worlds of Python, Java, C# or C development.
To give the short answer, macros are used for defining language syntax extensions to Common Lisp or Domain Specific Languages (DSLs). These languages are embedded right into the existing Lisp code. Now, the DSLs can have syntax similar to Lisp (like Peter Norvig's Prolog Interpreter for Common Lisp) or completely different (e.g. Infix Notation Math for Clojure).
Here is a more concrete example:Python has list comprehensions built into the language. This gives a simple syntax for a common case. The line
divisibleByTwo = [x for x in range(10) if x % 2 == 0]
yields a list containing all even numbers between 0 and 9. Back in the Python 1.5 days there was no such syntax; you'd use something more like this:
divisibleByTwo = []
for x in range( 10 ):
if x % 2 == 0:
divisibleByTwo.append( x )
These are both functionally equivalent. Let's invoke our suspension of disbelief and pretend Lisp has a very limited loop macro that just does iteration and no easy way to do the equivalent of list comprehensions.
In Lisp you could write the following. I should note this contrived example is picked to be identical to the Python code not a good example of Lisp code.
;; the following two functions just make equivalent of Python's range function
;; you can safely ignore them unless you are running this code
(defun range-helper (x)
(if (= x 0)
(list x)
(cons x (range-helper (- x 1)))))
(defun range (x)
(reverse (range-helper (- x 1))))
;; equivalent to the python example:
;; define a variable
(defvar divisibleByTwo nil)
;; loop from 0 upto and including 9
(loop for x in (range 10)
;; test for divisibility by two
if (= (mod x 2) 0)
;; append to the list
do (setq divisibleByTwo (append divisibleByTwo (list x))))
Before I go further, I should better explain what a macro is. It is a transformation performed on code by code. That is, a piece of code, read by the interpreter (or compiler), which takes in code as an argument, manipulates and the returns the result, which is then run in-place.
Of course that's a lot of typing and programmers are lazy. So we could define DSL for doing list comprehensions. In fact, we're using one macro already (the loop macro).
Lisp defines a couple of special syntax forms. The quote (') indicates the next token is a literal. The quasiquote or backtick (`) indicates the next token is a literal with escapes. Escapes are indicated by the comma operator. The literal '(1 2 3) is the equivalent of Python's [1, 2, 3]. You can assign it to another variable or use it in place. You can think of `(1 2 ,x) as the equivalent of Python's [1, 2, x] where x is a variable previously defined. This list notation is part of the magic that goes into macros. The second part is the Lisp reader which intelligently substitutes macros for code but that is best illustrated below:
So we can define a macro called lcomp (short for list comprehension). Its syntax will be exactly like the python that we used in the example [x for x in range(10) if x % 2 == 0] - (lcomp x for x in (range 10) if (= (% x 2) 0))
(defmacro lcomp (expression for var in list conditional conditional-test)
;; create a unique variable name for the result
(let ((result (gensym)))
;; the arguments are really code so we can substitute them
;; store nil in the unique variable name generated above
`(let ((,result nil))
;; var is a variable name
;; list is the list literal we are suppose to iterate over
(loop for ,var in ,list
;; conditional is if or unless
;; conditional-test is (= (mod x 2) 0) in our examples
,conditional ,conditional-test
;; and this is the action from the earlier lisp example
;; result = result + [x] in python
do (setq ,result (append ,result (list ,expression))))
;; return the result
,result)))
Now we can execute at the command line:
CL-USER> (lcomp x for x in (range 10) if (= (mod x 2) 0))
(0 2 4 6 8)
Pretty neat, huh? Now it doesn't stop there. You have a mechanism, or a paintbrush, if you like. You can have any syntax you could possibly want. Like Python or C#'s with syntax. Or .NET's LINQ syntax. In end, this is what attracts people to Lisp - ultimate flexibility.
You will find a comprehensive debate around lisp macro here.
An interesting subset of that article:
In most programming languages, syntax is complex. Macros have to take apart program syntax, analyze it, and reassemble it. They do not have access to the program's parser, so they have to depend on heuristics and best-guesses. Sometimes their cut-rate analysis is wrong, and then they break.
But Lisp is different. Lisp macros do have access to the parser, and it is a really simple parser. A Lisp macro is not handed a string, but a preparsed piece of source code in the form of a list, because the source of a Lisp program is not a string; it is a list. And Lisp programs are really good at taking apart lists and putting them back together. They do this reliably, every day.
Here is an extended example. Lisp has a macro, called "setf", that performs assignment. The simplest form of setf is
(setf x whatever)
which sets the value of the symbol "x" to the value of the expression "whatever".
Lisp also has lists; you can use the "car" and "cdr" functions to get the first element of a list or the rest of the list, respectively.
Now what if you want to replace the first element of a list with a new value? There is a standard function for doing that, and incredibly, its name is even worse than "car". It is "rplaca". But you do not have to remember "rplaca", because you can write
(setf (car somelist) whatever)
to set the car of somelist.
What is really happening here is that "setf" is a macro. At compile time, it examines its arguments, and it sees that the first one has the form (car SOMETHING). It says to itself "Oh, the programmer is trying to set the car of somthing. The function to use for that is 'rplaca'." And it quietly rewrites the code in place to:
(rplaca somelist whatever)
Common Lisp macros essentially extend the "syntactic primitives" of your code.
For example, in C, the switch/case construct only works with integral types and if you want to use it for floats or strings, you are left with nested if statements and explicit comparisons. There's also no way you can write a C macro to do the job for you.
But, since a lisp macro is (essentially) a lisp program that takes snippets of code as input and returns code to replace the "invocation" of the macro, you can extend your "primitives" repertoire as far as you want, usually ending up with a more readable program.
To do the same in C, you would have to write a custom pre-processor that eats your initial (not-quite-C) source and spits out something that a C compiler can understand. It's not a wrong way to go about it, but it's not necessarily the easiest.
Lisp macros allow you to decide when (if at all) any part or expression will be evaluated. To put a simple example, think of C's:
expr1 && expr2 && expr3 ...
What this says is: Evaluate expr1, and, should it be true, evaluate expr2, etc.
Now try to make this && into a function... thats right, you can't. Calling something like:
and(expr1, expr2, expr3)
Will evaluate all three exprs before yielding an answer regardless of whether expr1 was false!
With lisp macros you can code something like:
(defmacro && (expr1 &rest exprs)
`(if ,expr1 ;` Warning: I have not tested
(&& ,#exprs) ; this and might be wrong!
nil))
now you have an &&, which you can call just like a function and it won't evaluate any forms you pass to it unless they are all true.
To see how this is useful, contrast:
(&& (very-cheap-operation)
(very-expensive-operation)
(operation-with-serious-side-effects))
and:
and(very_cheap_operation(),
very_expensive_operation(),
operation_with_serious_side_effects());
Other things you can do with macros are creating new keywords and/or mini-languages (check out the (loop ...) macro for an example), integrating other languages into lisp, for example, you could write a macro that lets you say something like:
(setvar *rows* (sql select count(*)
from some-table
where column1 = "Yes"
and column2 like "some%string%")
And thats not even getting into Reader macros.
Hope this helps.
I don't think I've ever seen Lisp macros explained better than by this fellow: http://www.defmacro.org/ramblings/lisp.html
A lisp macro takes a program fragment as input. This program fragment is represented a data structure which can be manipulated and transformed any way you like. In the end the macro outputs another program fragment, and this fragment is what is executed at runtime.
C# does not have a macro facility, however an equivalent would be if the compiler parsed the code into a CodeDOM-tree, and passed that to a method, which transformed this into another CodeDOM, which is then compiled into IL.
This could be used to implement "sugar" syntax like the for each-statement using-clause, linq select-expressions and so on, as macros that transforms into the underlying code.
If Java had macros, you could implement Linq syntax in Java, without needing Sun to change the base language.
Here is pseudo-code for how a lisp-style macro in C# for implementing using could look:
define macro "using":
using ($type $varname = $expression) $block
into:
$type $varname;
try {
$varname = $expression;
$block;
} finally {
$varname.Dispose();
}
Since the existing answers give good concrete examples explaining what macros achieve and how, perhaps it'd help to collect together some of the thoughts on why the macro facility is a significant gain in relation to other languages; first from these answers, then a great one from elsewhere:
... in C, you would have to write a custom pre-processor [which would probably qualify as a sufficiently complicated C program] ...
—Vatine
Talk to anyone that's mastered C++ and ask them how long they spent learning all the template fudgery they need to do template metaprogramming [which is still not as powerful].
—Matt Curtis
... in Java you have to hack your way with bytecode weaving, although some frameworks like AspectJ allows you to do this using a different approach, it's fundamentally a hack.
—Miguel Ping
DOLIST is similar to Perl's foreach or Python's for. Java added a similar kind of loop construct with the "enhanced" for loop in Java 1.5, as part of JSR-201. Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern. A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide "expert group" to hash everything out. That process--according to Sun--takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer's favorite compiler supports the new version of Java, they probably ''still'' can't use the new feature until they're allowed to break source compatibility with older versions of Java. So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.
—Peter Seibel, in "Practical Common Lisp"
Think of what you can do in C or C++ with macros and templates. They're very useful tools for managing repetitive code, but they're limited in quite severe ways.
Limited macro/template syntax restricts their use. For example, you can't write a template which expands to something other than a class or a function. Macros and templates can't easily maintain internal data.
The complex, very irregular syntax of C and C++ makes it difficult to write very general macros.
Lisp and Lisp macros solve these problems.
Lisp macros are written in Lisp. You have the full power of Lisp to write the macro.
Lisp has a very regular syntax.
Talk to anyone that's mastered C++ and ask them how long they spent learning all the template fudgery they need to do template metaprogramming. Or all the crazy tricks in (excellent) books like Modern C++ Design, which are still tough to debug and (in practice) non-portable between real-world compilers even though the language has been standardised for a decade. All of that melts away if the langauge you use for metaprogramming is the same language you use for programming!
I'm not sure I can add some insight to everyone's (excellent) posts, but...
Lisp macros work great because of the Lisp syntax nature.
Lisp is an extremely regular language (think of everything is a list); macros enables you to treat data and code as the same (no string parsing or other hacks are needed to modify lisp expressions). You combine these two features and you have a very clean way to modify code.
Edit: What I was trying to say is that Lisp is homoiconic, which means that the data structure for a lisp program is written in lisp itself.
So, you end up with a way of creating your own code generator on top of the language using the language itself with all its power (eg. in Java you have to hack your way with bytecode weaving, although some frameworks like AspectJ allows you to do this using a different approach, it's fundamentally a hack).
In practice, with macros you end up building your own mini-language on top of lisp, without the need to learn additional languages or tooling, and with using the full power of the language itself.
Lisp macros represents a pattern that occurs in almost any sizeable programming project. Eventually in a large program you have a certain section of code where you realize it would be simpler and less error prone for you to write a program that outputs source code as text which you can then just paste in.
In Python objects have two methods __repr__ and __str__. __str__ is simply the human readable representation. __repr__ returns a representation that is valid Python code, which is to say, something that can be entered into the interpreter as valid Python. This way you can create little snippets of Python that generate valid code that can be pasted into your actually source.
In Lisp this whole process has been formalized by the macro system. Sure it enables you to create extensions to the syntax and do all sorts of fancy things, but it's actual usefulness is summed up by the above. Of course it helps that the Lisp macro system allows you to manipulate these "snippets" with the full power of the entire language.
In short, macros are transformations of code. They allow to introduce many new syntax constructs. E.g., consider LINQ in C#. In lisp, there are similar language extensions that are implemented by macros (e.g., built-in loop construct, iterate). Macros significantly decrease code duplication. Macros allow embedding «little languages» (e.g., where in c#/java one would use xml to configure, in lisp the same thing can be achieved with macros). Macros may hide difficulties of using libraries usage.
E.g., in lisp you can write
(iter (for (id name) in-clsql-query "select id, name from users" on-database *users-database*)
(format t "User with ID of ~A has name ~A.~%" id name))
and this hides all the database stuff (transactions, proper connection closing, fetching data, etc.) whereas in C# this requires creating SqlConnections, SqlCommands, adding SqlParameters to SqlCommands, looping on SqlDataReaders, properly closing them.
While the above all explains what macros are and even have cool examples, I think the key difference between a macro and a normal function is that LISP evaluates all the parameters first before calling the function. With a macro it's the reverse, LISP passes the parameters unevaluated to the macro. For example, if you pass (+ 1 2) to a function, the function will receive the value 3. If you pass this to a macro, it will receive a List( + 1 2). This can be used to do all kinds of incredibly useful stuff.
Adding a new control structure, e.g. loop or the deconstruction of a list
Measure the time it takes to execute a function passed in. With a function the parameter would be evaluated before control is passed to the function. With the macro, you can splice your code between the start and stop of your stopwatch. The below has the exact same code in a macro and a function and the output is very different. Note: This is a contrived example and the implementation was chosen so that it is identical to better highlight the difference.
(defmacro working-timer (b)
(let (
(start (get-universal-time))
(result (eval b))) ;; not splicing here to keep stuff simple
((- (get-universal-time) start))))
(defun my-broken-timer (b)
(let (
(start (get-universal-time))
(result (eval b))) ;; doesn't even need eval
((- (get-universal-time) start))))
(working-timer (sleep 10)) => 10
(broken-timer (sleep 10)) => 0
One-liner answer:
Minimal syntax => Macros over Expressions => Conciseness => Abstraction => Power
Lisp macros do nothing more than writing codes programmatically. That is, after expanding the macros, you got nothing more than Lisp code without macros. So, in principle, they achieve nothing new.
However, they differ from macros in other programming languages in that they write codes on the level of expressions, whereas others' macros write codes on the level of strings. This is unique to lisp thanks to their parenthesis; or put more precisely, their minimal syntax which is possible thanks to their parentheses.
As shown in many examples in this thread, and also Paul Graham's On Lisp, lisp macros can then be a tool to make your code much more concise. When conciseness reaches a point, it offers new levels of abstractions for codes to be much cleaner. Going back to the first point again, in principle they do not offer anything new, but that's like saying since paper and pencils (almost) form a Turing machine, we do not need an actual computer.
If one knows some math, think about why functors and natural transformations are useful ideas. In principle, they do not offer anything new. However by expanding what they are into lower-level math you'll see that a combination of a few simple ideas (in terms of category theory) could take 10 pages to be written down. Which one do you prefer?
I got this from the common lisp cookbook and I think it explained why lisp macros are useful.
"A macro is an ordinary piece of Lisp code that operates on another piece of putative Lisp code, translating it into (a version closer to) executable Lisp. That may sound a bit complicated, so let's give a simple example. Suppose you want a version of setq that sets two variables to the same value. So if you write
(setq2 x y (+ z 3))
when z=8 both x and y are set to 11. (I can't think of any use for this, but it's just an example.)
It should be obvious that we can't define setq2 as a function. If x=50 and y=-5, this function would receive the values 50, -5, and 11; it would have no knowledge of what variables were supposed to be set. What we really want to say is, When you (the Lisp system) see (setq2 v1 v2 e), treat it as equivalent to (progn (setq v1 e) (setq v2 e)). Actually, this isn't quite right, but it will do for now. A macro allows us to do precisely this, by specifying a program for transforming the input pattern (setq2 v1 v2 e)" into the output pattern (progn ...)."
If you thought this was nice you can keep on reading here:
http://cl-cookbook.sourceforge.net/macros.html
In python you have decorators, you basically have a function that takes another function as input. You can do what ever you want: call the function, do something else, wrap the function call in a resource acquire release, etc. but you don't get to peek inside that function. Say we wanted to make it more powerful, say your decorator received the code of the function as a list then you could not only execute the function as is but you can now execute parts of it, reorder lines of the function etc.