Related
This question already has answers here:
What makes Lisp macros so special?
(15 answers)
Closed 3 months ago.
I keep reading that Lisp macros are one of the most powerful features of the language. But reading over the specifications and manuals, they are just functions whose arguments are unevaluated.
Given any macro (defmacro example (arg1 ... argN) (body-forms)) I could just write (defun example (arg1 ... argN) ... (body-forms)) with the last body-form turned into a list and then call it like (eval (example 'arg1 ... 'argN)) to emulate the same behavior of the macro. If this were the case, then macros would just be syntactic sugar, but I doubt that syntactic sugar would be called a powerful language feature. What am I missing? Are there cases where I cannot carry out this procedure to emulate a macro?
I can't talk about powerful because it can be a little bit subjective, but macros are regular Lisp functions that work on Lisp data, so they are as expressive as other functions. This isn't the case with templates or generic functions in other languages that rely more on static types and are more restricted (on purpose).
In some way, yes macros are simple syntactic facilities, but you are focused in your emulation on the dynamic semantics of macros, ie. how you can run code that evaluates macros at runtime. However:
the code using eval is not equivalent to expanded code
the preprocessing/compile-time aspect of macros is not emulated
Lexical scope
Function, like +, do not inherit the lexical scope:
(let ((x 30))
(+ 3 4))
Inside the definition of +, you cannot access x. Being able to do so is what "dynamic scope" is about (more precisely, see dynamic extent, indefinite scope variables). But nowadays it is quite the exception to rely on dynamic scope. Most functions use lexical scope, and this is the case for eval too.
The eval function evaluates a form in the null lexical environment, and it never has access to the surrounding lexical bindings. As such, it behaves like any regular function.
So, in you example, calling eval on the transformed source code will not work, since arg1 to argnN will probably be unbound (it depends on what your macro does).
In order to have an equivalent form, you have to inject bindings in the transformed code, or expand at a higher level:
(defun expand-square (var)
(list '* var var))
;; instead of:
(defun foo (x) (eval (expand-square 'x))) ;; x unbound during eval
;; inject bindings
(defun foo (x) (eval `(let ((z ,x)) (expand-square z))))
;; or expand the top-level form
(eval `(defun foo (x) ,(expand-square 'x)))
Note that macros (in Common Lisp) also have access to the lexical environment through &environment parameters in their lambda-list. The use of this environment is implementation dependent, but can be used to access the declarations associated with a variable, for example.
Notice also how in the last example you evaluate the code when defining the function, and not when running it. This is the second thing about macro.
Expansion time
In order to emulate macros you could locally replace a call to a macro by a form that emulates it at runtime (using let to captures all the bindings you want to see inside the expanded code, which is tedious), but then you would miss the useful aspect of macros that is: generating code ahead of time.
The last example above shows how you can quote defun and wrap it in eval, and basically you would need to do that for all functions if you wanted to emulate the preprocessing work done by macros.
The macro system is a way to integrate this preprocessing step in the language in a way that is simple to use.
Conclusion
Macros themselves are a nice way to abstract things when functions can't. For example you can have a more human-friendly, stable syntax that hides implementation details. That's how you define pattern-matching abilities in Common Lisp that make it look like they are part of the language, without too much runtime penalty or verbosity.
They rely on simple term-rewriting functions that are integrated in the language, but you can emulate their behavior either at compile-time or runtime yourself if you want. They can be used to perform different kinds of abstraction that are usually missing or more cumbersome to do in other languages, but are also limited: they don't "understand" code by themselves, they don't give access to all the facilities of the compiler (type propagation, etc.). If you want more you can use more advanced libraries or compiler tools (see deftransform), but macros at least are portable.
Macros are not just functions whose arguments are unevaluated. Macros are functions between programming languages. In other words a macro is a function whose argument is a fragment of source code of a programming language which includes the macro, and whose value is a fragment of source code of a language which does not include the macro (or which includes it in a simpler way).
In very ancient, very rudimentary, Lisps, before people really understood what macros were, you could simulate macros with things called FEXPRs combined with EVAL. A FEXPR was simply a function which did not evaluate its arguments. This worked in such Lisps only because they were completely dynamically scoped, and the cost of it working was that compilation of such things was not possible at all. Those are two enormous costs.
In any modern Lisp, this won't work at all. You can write a toy version of FEXPRs as a macro (this may be buggy):
(defmacro deffex (fx args &body body)
(assert (every (lambda (arg)
(and (symbolp arg)
(not (member arg lambda-list-keywords))))
args)
(args) "not a simple lambda list")
`(defmacro ,fx ,args
`(let ,(mapcar (lambda (argname argval)
`(,argname ',argval))
',args (list ,#args))
,#',body)))
So now we could try to write a trivial binding construct I'll call with using this thing:
(deffex with (var val form)
(eval `(let ((,var ,val)) ,form)))
And this seems to work:
> (with a 1 a)
1
Of course, we're paying the cost that no code which uses this construct can ever be compiled so all our programs will be extremely slow, but perhaps that is a cost we're willing to accept (it's not, but never mind).
Except, of course, it doesn't work, at all:
> (with a 1
(with b 2
(+ a b)))
Error: The variable a is unbound.
Oh dear.
Why doesn't it work? It doesn't work because Common Lisp is lexically scoped, and eval is a function: it can't see the lexical bindings.
So not only does this kind of approach prevent compilation in a modern Lisp, it doesn't work at all.
People often, at this point, suggest some kind of kludge solution which would allow eval to be able to see lexical bindings. The cost of such a solution is that all the lexical bindings need to exist in compiled code: no variable can ever be compiled away, not even its name. That's essentially saying that no good compilers can ever be used, even for the small part of your programs you can compile at all in a language which makes extensive use of macros like CL. For instance, if you ever use defun you're not going to be able to compile the code in its body. People do use defun occasionally, I think.
So this approach simply won't work: it worked by happenstance in very old Lisps but it can't work, even at the huge cost of preventing compilation, in any modern Lisp.
More to the point this approach obfuscates the understanding of what macros are: as I said at the start, macros are functions between programming languages, and understanding that is critical. When you are designing macros you are implementing a new programming language.
How do you go about solving this problem?
Suppose I want to write a function that does the following: if the user has library X installed, then use function X-function, otherwise - skip?
What I tried:
(when (symbol-function 'X-function)
(X-function))
I'm getting a warning for this code - so what is the right way?
How about this:
(when (fboundp 'X-function)
(X-function))
The docs at http://www.gnu.org/software/emacs/manual/html_node/elisp/Function-Cells.html says about symbol-function
If the symbol's function cell is void, a void-function error is signaled.
I'm guessing that is what you are seeing. On the other hand, fboundp just returns t or nil depending on whether the function exists.
The way to suppress this compiler warning is with something like:
(declare-function X-function "ext:X-library.el")
(when (fboundp 'X-function)
(X-function))
Here X-library is the name of the library that X-function is defined in when the library is there. The byte-compiler will then do the following:
It will look for the library in the load path.
If it finds it, it will check that the function has been defined.
If it does not find the library it will assume that it will be when the library is there and pass on without error.
Thus if there is no X-library it won't complain, but if there is one and it does not define the function then it will. This means that if an updated version of the library does not contain X-function then you will know when you try to re-compile your code.
If you look up the documentation for declare-function you will find that it can also check the argument list of functions.
Incidentally If you get similar warnings about undeclared variables you can suppress these with:
(defvar X-variable)
However it is important not to set the variable even if you know what value the library sets it to as this could change in a later version.
This gives you one version of the program that works whether or not X-library is present. You might prefer to have two versions, one for when X-library is present and one for when it is not. This can be done with a macro:
(defmacro run? (function &rest args)
"Expand to function call if function exists."
(when (fboundp `,function)
`(,function ,#args)))
Now instead of a call like:
(X-function a1 a2 a3)
You write:
(run? X-function a1 a2 a3)
If you compile it with X-library present this expands to the call to X-function. If the library is not present then it expands to nothing at all. You will not need the declare-function in any case. This gives two different versions, but it should be more efficient because the decisions as to whether the library is there or not are taken at compile time not run time.
One small caveat. If you go for this second solution you must either compile the whole program in the X-library environment or outside it. If you try loading the library half way through the program then when interpreted it will work as you might expect with the macro expanding differently before and after the load. But in a compiled program a macro is only expanded once. The test test for the library is in code that does the expanding not in the expansion, so the macro will not work the same before and after the load.
Another case when you can get the warning that a function cannot be found is when you define a function programmatically and use fset to set it. The following example illustrates this and what to do about it:
(eval-and-compile
(fset 'my-function1 (lambda () nil)))
(my-function1)
(fset 'my-function2 (lambda () nil))
(my-function2)
(my-function3)
(eval-and-compile
(fset 'my-function3 (lambda () nil)))
If you compile this you get the warnings:
Warning: the function `my-function2' is not known to be defined.
and:
Warning: the function `my-function3' might not be defined at runtime.
The second warning goes away if you re-compile the code a second time in the same Emacs session, but the first doesn't.
What is happening here is this: When the compiler sees eval-and-compile, it first evaluates the body of the in the current Emacs session and then compiles it. Having evaluated the code, Emacs knows about the programmatically defined function.
In the case of function1, the byte compiler sees the function call after Emacs has evaluated the form and so you don't get any warnings.
In the case of function2 the byte compiler never knows the functions is defined so you always get a warning.
In the case of function3, the first time round, the bite compiler doesn't know the function exists when it sees the function call. By the end of the compilation it knows the function exists but it isn't intelligent enough to work out how it knows so you get a different warning. However, if you re-compile it in the same Emacs session, it does know so the warning goes away.
Note that eval-and-compile, like eval-with-compile, look like a progn to the Emacs interpreter.
Why is the function/macro dichotomy present in Common Lisp?
What are the logical problems in allowing the same name representing both a macro (taking precedence when found in function position in compile/eval) and a function (usable for example with mapcar)?
For example having second defined both as a macro and as a function would allow to use
(setf (second x) 42)
and
(mapcar #'second L)
without having to create any setf trickery.
Of course it's clear that macros can do more than functions and so the analogy cannot be complete (and I don't think of course that every macro shold also be a function) but why forbidding it by making both sharing a single namespace when it could be potentially useful?
I hope I'm not offending anyone, but I don't really find a "Why doing that?" response really pertinent... I'm looking for why this is a bad idea. Imposing an arbitrary limitation because no good use is known is IMO somewhat arrogant (sort of assumes perfect foresight).
Or are there practical problems in allowing it?
Macros and Functions are two very different things:
macros are using source (!!!) code and are generating new source (!!!) code
functions are parameterized blocks of code.
Now we can look at this from several angles, for example:
a) how do we design a language where functions and macros are clearly identifiable and are looking different in our source code, so we (the human) can easily see what is what?
or
b) how do we blend macros and functions in a way that the result is most useful and has the most useful rules controlling its behavior? For the user it should not make a difference to use a macro or a function.
We really need to convince ourselves that b) is the way to go and we would like to use a language where macros and functions usage looks the same and is working according to similar principles. Take ships and cars. They look different, their use case is mostly different, they transport people - should we now make sure that the traffic rules for them are mostly identical, should we make them different or should we design the rules for their special usage?
For functions we have problems like: defining a function, scope of functions, life-time of functions, passing functions around, returning functions, calling functions, shadowing of functions, extension of functions, removing the definition a function, compilation and interpretation of functions, ...
If we would make macros appear mostly similar to functions, we need to address most or all above issues for them.
In your example you mention a SETF form. SETF is a macro that analyses the enclosed form at macro expansion time and generates code for a setter. It has little to do with SECOND being a macro or not. Having SECOND being a macro would not help at all in this situation.
So, what is a problem example?
(defmacro foo (a b)
(if (and (numberp b) (zerop b))
a
`(- ,a ,b)))
(defun bar (x list)
(mapcar #'foo (list x x x x) '(1 2 3 4)))
Now what should that do? Intuitively it looks easy: map FOO over the lists. But it isn't. When Common Lisp was designed, I would guess, it was not clear what that should do and how it should work. If FOO is a function, then it was clear: Common Lisp took the ideas from Scheme behind lexically scoped first-class functions and integrated it into the language.
But first-class macros? After the design of Common Lisp a bunch of research went into this problem and investigated it. But at the time of Common Lisp's design, there was no wide-spread use of first-class macros and no experience with design approaches. Common Lisp is standardizing on what was known at the time and what the language users thought necessary to develop (the object-system CLOS is kind of novel, based on earlier experience with similar object-systems) software with. Common Lisp was not designed to have the theoretically most pleasing Lisp dialect - it was designed to have a powerful Lisp which allows the efficient implementation of software.
We could work around this and say, passing macros is not possible. The developer would have to provide a function under the same name, which we pass around.
But then (funcall #'foo 1 2) and (foo 1 2) would invoke different machineries? In the first case the function fooand in the second case we use the macro foo to generate code for us? Really? Do we (as human programmers) want this? I think not - it looks like it makes programming much more complicated.
From a pragmatic point of view: Macros and the mechanism behind it are already complicated enough that most programmers have difficulties dealing with it in real code. They make debugging and code understanding much harder for a human. On the surface a macro makes code easier to read, but the price is the need to understand the code expansion process and result.
Finding a way to further integrate macros into the language design is not an easy task.
readscheme.org has some pointers to Macro-related research wrt. Scheme: Macros
What about Common Lisp
Common Lisp provides functions which can be first-class (stored, passed around, ...) and lexically scoped naming for them (DEFUN, FLET, LABELS, FUNCTION, LAMBDA).
Common Lisp provides global macros (DEFMACRO) and local macros (MACROLET).
Common Lisp provides global compiler macros (DEFINE-COMPILER-MACRO).
With compiler macros it is possible to have a function or macro for a symbol AND a compiler macro. The Lisp system can decide to prefer the compiler macro over the macro or function. It can also ignore them entirely. This mechanism is mostly used for the user to program specific optimizations. Thus it does not solve any macro related problems, but provides a pragmatic way to program global optimizations.
I think that Common Lisp's two namespaces (functions and values), rather than three (macros, functions, and values), is a historical contingency.
Early Lisps (in the 1960s) represented functions and values in different ways: values as bindings on the runtime stack, and functions as properties attached to symbols in the symbol table. This difference in implementation led to the specification of two namespaces when Common Lisp was standardized in the 1980s. See Richard Gabriel's paper Technical Issues of Separation in Function Cells and Value Cells for an explanation of this decision.
Macros (and their ancestors, FEXPRs, functions which do not evaluate their arguments) were stored in many Lisp implementations in the symbol table, in the same way as functions. It would have been inconvenient for these implementations if a third namespace (for macros) had been specified, and would have caused backwards-compatibility problems for many programs.
See Kent Pitman's paper Special Forms in Lisp for more about the history of FEXPRs, macros and other special forms.
(Note: Kent Pitman's website is not working for me, so I've linked to the papers via archive.org.)
Because then the exact same name would represent two different objects, depending on the context. It makes the programme unnecessarily difficult to understand.
My TXR Lisp dialect allows a symbol to be simultaneously a macro and function. Moreover, certain special operators are also backed by functions.
I put a bit of thought into the design, and haven't run into any problems. It works very well and is conceptually clean.
Common Lisp is the way it is for historic reasons.
Here is a brief rundown of the system:
When a global macro is defined for symbol X with defmacro, the symbol X does not become fboundp. Rather, what becomes fboundp is the compound function name (macro X).
The name (macro X) is then known to symbol-function, trace and in other situations. (symbol-function '(macro X)) retrieves the two-argument expander function which takes the form and an environment.
It's possible to write a macro using (defun (macro X) (form env) ...).
There are no compiler macros; regular macros do the job of compiler macros.
A regular macro can return the unexpanded form to indicate that it's declining to expand. If a lexical macrolet declines to expand, the opportunity goes to a more lexically outer macrolet, and so on up to the global defmacro. If the global defmacro declines to expand, the form is considered expanded, and thus is necessarily either a function call or special form.
If we have both a function and macro called X, we can call the function definition using (call (fun X) ...) or (call 'X ...), or else using the Lisp-1-style dwim evaluator (dwim X ...) that is almost always used through its [] syntactic sugar as [X ...].
For a sort of completeness, the functions mboundp, mmakunbound and symbol-macro are provided, which are macro analogs of fboundp, fmakunbound and symbol-function.
The special operators or, and, if and some others have function definitions also, so that code like [mapcar or '(nil 2 t) '(1 0 3)] -> (1 2 t) is possible.
Example: apply constant folding to sqrt:
1> (sqrt 4.0)
2.0
2> (defmacro sqrt (x :env e :form f)
(if (constantp x e)
(sqrt x)
f))
** warning: (expr-2:1) defmacro: defining sqrt, which is also a built-in defun
sqrt
3> (sqrt 4.0)
2.0
4> (macroexpand '(sqrt 4.0))
2.0
5> (macroexpand '(sqrt x))
(sqrt x)
However, no, (set (second x) 42) is not implemented via a macro definition for second. That would not work very well. The main reason is that it would be too much of a burden. The programmer may want to have, for a given function, a macro definition which has nothing to do with implementing assignment semantics!
Moreover, if (second x) implements place semantics, what happens when it is not embedded in an assignment operation, such that the semantics is not required at all? Basically, to hit all the requirements would require concocting a scheme for writing macros whose complexity would equal or exceed that of existing logic for handling places.
TXR Lisp does, in fact, feature a special kind of macro called a "place macro". A form is only recognized as a place macro invocation when it is used as a place. However, place macros do not implement place semantics themselves; they just do a straightforward rewrite. Place macros must expand down to a form that is recognized as a place.
Example: specify that (foo x), when used as a place, behaves as (car x):
1> (define-place-macro foo (x) ^(car ,x))
foo
2> (macroexpand '(foo a)) ;; not a macro!
(foo a)
3> (macroexpand '(set (foo a) 42)) ;; just a place macro
(sys:rplaca a 42)
If foo expanded to something which is not a place, things would fail:
4> (define-place-macro foo (x) ^(bar ,x))
foo
5> (macroexpand '(foo a))
(foo a)
6> (macroexpand '(set (foo a) 42))
** (bar a) is not an assignable place
I've learned Clojure previously and really like the language. I also love Emacs and have hacked some simple stuff with Emacs Lisp. There is one thing which prevents me mentally from doing anything more substantial with Elisp though. It's the concept of dynamic scoping. I'm just scared of it since it's so alien to me and smells like semi-global variables.
So with variable declarations I don't know which things are safe to do and which are dangerous. From what I've understood, variables set with setq fall under dynamic scoping (is that right?) What about let variables? Somewhere I've read that let allows you to do plain lexical scoping, but somewhere else I read that let vars also are dynamically scoped.
I quess my biggest worry is that my code (using setq or let) accidentally breaks some variables from platform or third-party code that I call or that after such call my local variables are messed up accidentally. How can I avoid this?
Are there a few simple rules of thumb that I can just follow and know exactly what happens with the scope without being bitten in some weird, hard-to-debug way?
It isn't that bad.
The main problems can appear with 'free variables' in functions.
(defun foo (a)
(* a b))
In above function a is a local variable. b is a free variable. In a system with dynamic binding like Emacs Lisp, b will be looked up at runtime. There are now three cases:
b is not defined -> error
b is a local variable bound by some function call in the current dynamic scope -> take that value
b is a global variable -> take that value
The problems can then be:
a bound value (global or local) is shadowed by a function call, possibly unwanted
an undefined variable is NOT shadowed -> error on access
a global variable is NOT shadowed -> picks up the global value, which might be unwanted
In a Lisp with a compiler, compiling the above function might generate a warning that there is a free variable. Typically Common Lisp compilers will do that. An interpreter won't provide that warning, one just will see the effect at runtime.
Advice:
make sure that you don't use free variables accidentally
make sure that global variables have a special name, so that they are easy to spot in source code, usually *foo-var*
Don't write
(defun foo (a b)
...
(setq c (* a b)) ; where c is a free variable
...)
Write:
(defun foo (a b)
...
(let ((c (* a b)))
...)
...)
Bind all variables you want to use and you want to make sure that they are not bound somewhere else.
That's basically it.
Since GNU Emacs version 24 lexical binding is supported in its Emacs Lisp. See: Lexical Binding, GNU Emacs Lisp Reference Manual.
In addition to the last paragraph of Gilles answer, here is how RMS argues in favor of dynamic scoping in an extensible system:
Some language designers believe that
dynamic binding should be avoided, and
explicit argument passing should be
used instead. Imagine that function A
binds the variable FOO, and calls the
function B, which calls the function
C, and C uses the value of FOO.
Supposedly A should pass the value as
an argument to B, which should pass it
as an argument to C.
This cannot be done in an extensible
system, however, because the author of
the system cannot know what all the
parameters will be. Imagine that the
functions A and C are part of a user
extension, while B is part of the
standard system. The variable FOO does
not exist in the standard system; it
is part of the extension. To use
explicit argument passing would
require adding a new argument to B,
which means rewriting B and everything
that calls B. In the most common case,
B is the editor command dispatcher
loop, which is called from an awful
number of places.
What's worse, C must also be passed an
additional argument. B doesn't refer
to C by name (C did not exist when B
was written). It probably finds a
pointer to C in the command dispatch
table. This means that the same call
which sometimes calls C might equally
well call any editor command
definition. So all the editing
commands must be rewritten to accept
and ignore the additional argument. By
now, none of the original system is
left!
Personally, I think that if there is a problem with Emacs-Lisp, it is not dynamic scoping per se, but that it is the default, and that it is not possible to achieve lexical scoping without resorting to extensions. In CL, both dynamic and lexical scoping can be used, and -- except for top-level (which is adressed by several deflex-implementations) and globally declared special variables -- the default is lexical scoping. In Clojure, too, you can use both lexical and dynamic scoping.
To quote RMS again:
It is not necessary for dynamic scope to be the only scope rule provided, just useful
for it to be available.
Are there a few simple rules of thumb that I can just follow and know exactly what happens with the scope without being bitten in some weird, hard-to-debug way?
Read Emacs Lisp Reference, you'll have many details like this one :
Special Form: setq [symbol form]...
This special form is the most common method of changing a
variable's value. Each SYMBOL is given a new value, which is the
result of evaluating the corresponding FORM. The most-local
existing binding of the symbol is changed.
Here is an example :
(defun foo () (setq tata "foo"))
(defun bar (tata) (setq tata "bar"))
(foo)
(message tata)
===> "foo"
(bar tata)
(message tata)
===> "foo"
As Peter Ajtai pointed out:
Since emacs-24.1 you can enable lexical scoping on a per file basis by putting
;; -*- lexical-binding: t -*-
on top of your elisp file.
First, elisp has separate variable and function bindings, so some pitfalls of dynamic scoping are not relevant.
Second, you can still use setq to set variables, but the value set does not survive the exit of the dynamic scope it is done in. This isn't, fundamentally, different from lexical scoping, with the difference that with dynamic scoping a setq in a function you call can affect the value you see after the function call.
There's lexical-let, a macro that (essentially) imitates lexical bindings (I believe it does this by walking the body and changing all occurrences of the lexically let variables to a gensymmed name, eventually uninterning the symbol), if you absolutely need to.
I'd say "write code as normal". There are times when the dynamic nature of elisp will bite you, but I've found that in practice that is surprisingly seldom.
Here's an example of what I was saying about setq and dynamically-bound variables (recently evaluated in a nearby scratch buffer):
(let ((a nil))
(list (let ((a nil))
(setq a 'value)
a)
a))
(value nil)
Everything that has been written here is worthwhile. I would add this: get to know Common Lisp -- if nothing else, read about it. CLTL2 presents lexical and dynamic binding well, as do other books. And Common Lisp integrates them well in a single language.
If you "get it" after some exposure to Common Lisp then things will be clearer for you for Emacs Lisp. Emacs 24 uses lexical scoping to a greater extent by default than older versions, but Common Lisp's approach will still be clearer and cleaner (IMHO). Finally, it is definitely the case that dynamic scope is important for Emacs Lisp, for the reasons that RMS and others have emphasized.
So my suggestion is to get to know how Common Lisp deals with this. Try to forget about Scheme, if that is your main mental model of Lisp -- it will limit you more than help you in understanding scoping, funargs, etc. in Emacs Lisp. Emacs Lisp, like Common Lisp, is "dirty and low-down"; it is not Scheme.
Dynamic and lexical scoping have different behaviors when a piece of code is used in a different scope than the one it was defined in. In practice, there are two patterns that cover most troublesome cases:
A function shadows a global variable, then calls another function that uses that global variable.
(defvar x 3)
(defun foo ()
x)
(defun bar (x)
(+ (foo) x))
(bar 0) ⇒ 0
This doesn't come up often in Emacs because local variables tend to have short names (often single-word) whereas global variables tend to have long names (often prefixed by packagename-). Many standard functions have names that are tempting to use as local variables like list and point, but functions and variables live in separate name spaces are local functions are not used very often.
A function is defined in one lexical context and used outside this lexical context because it's passed to a higher-order function.
(let ((cl-y 10))
(mapcar* (lambda (elt) (* cl-y elt)) '(1 2 3)))
⇒ (10 20 30)
(let ((cl-x 10))
(mapcar* (lambda (elt) (* cl-x elt)) '(1 2 3)))
⇑ (wrong-type-argument number-or-marker-p (1 2 3))
The error is due to the use of cl-x as a variable name in mapcar* (from the cl package). Note that the cl package uses cl- as a prefix even for its local variables in higher-order functions. This works reasonably well in practice, as long as you take care not to use the same variable as a global name and as a local name, and you don't need to write a recursive higher-order function.
P.S. Emacs Lisp's age isn't the only reason why it's dynamically scoped. True, in those days, lisps tended towards dynamic scoping — Scheme and Common Lisp hadn't really taken on yet. But dynamic scoping is also an asset in a language targeted towards extending a system dynamically: it lets you hook into more places without any special effort. With great power comes great rope to hang yourself: you risk accidentally hooking into a place you didn't know about.
The other answers are good at explaining the technical details on how to work with dynamic scoping, so here's my non-technical advice:
Just do it
I've been tinkering with Emacs lisp for 15+ years and don't know that I've ever been bitten by any problems due to the differences between lexical/dynamic scope.
Personally, I've not found the need for closures (I love 'em, just don't need them for Emacs). And, I generally try to avoid global variables in general (whether the scoping was lexical or dynamic).
So I suggest jumping in and writing customizations that suit your needs/desires, chances are you won't have any problems.
I entirely feel your pain. I find the lack of lexical binding in emacs rather annoying - especially not being able to use lexical closures, which seems to be a solution I think of a lot, coming from more modern languages.
While I don't have any more advice on working around the lacking features that the previous answers didn't cover yet, I'd like to point out the existance of an emacs branch called `lexbind', implementing lexical binding in a backward-compatible way. In my experience lexical closures are still a little buggy in some circumstances, but that branch appears to a promising approach.
Just don't.
Emacs-24 lets you use lexical-scope. Just run
(setq lexical-binding t)
or add
;; -*- lexical-binding: t -*-
at the beginning of your file.
I've heard that Lisp's macro system is very powerful. However, I find it difficult to find some practical examples of what they can be used for; things that would be difficult to achieve without them.
Can anyone give some examples?
Source code transformations. All kinds. Examples:
New control flow statements: You need a WHILE statement? Your language doesn't have one? Why wait for the benevolent dictator to maybe add one next year. Write it yourself. In five minutes.
Shorter code: You need twenty class declarations that almost look identical - only a limited amount of places are different. Write a macro form that takes the differences as parameter and generates the source code for you. Want to change it later? Change the macro in one place.
Replacements in the source tree: You want to add code into the source tree? A variable really should be a function call? Wrap a macro around the code that 'walks' the source and changes the places where it finds the variable.
Postfix syntax: You want to write your code in postfix form? Use a macro that rewrites the code to the normal form (prefix in Lisp).
Compile-time effects: You need to run some code in the compiler environment to inform the development environment about definitions? Macros can generate code that runs at compile time.
Code simplifications/optimizations at compile-time: You want to simplify some code at compile time? Use a macro that does the simplification - that way you can shift work from runtime to compile time, based on the source forms.
Code generation from descriptions/configurations: You need to write a complex mix of classes. For example your window has a class, subpanes have classes, there are space constraints between panes, you have a command loop, a menu and a whole bunch of other things. Write a macro that captures the description of your window and its components and creates the classes and the commands that drive the application - from the description.
Syntax improvements: Some language syntax looks not very convenient? Write a macro that makes it more convenient for you, the application writer.
Domain specific languages: You need a language that is nearer to the domain of your application? Create the necessary language forms with a bunch of macros.
Meta-linguistic abstraction
The basic idea: everything that is on the linguistic level (new forms, new syntax, form transformations, simplification, IDE support, ...) can now be programmed by the developer piece by piece - no separate macro processing stage.
Pick any "code generation tool". Read their examples. That's what it can do.
Except you don't need to use a different programming language, put any macro-expansion code where the macro is used, run a separate command to build, or have extra text files sitting on your hard disk that are only of value to your compiler.
For example, I believe reading the Cog example should be enough to make any Lisp programmer cry.
Anything you'd normally want to have done in a pre-processor?
One macro I wrote, is for defining state-machines for driving game objects. It's easier to read the code (using the macro) than it is to read the generated code:
(def-ai ray-ai
(ground
(let* ((o (object))
(r (range o)))
(loop for p in *players*
if (line-of-sight-p o p r)
do (progn
(setf (target o) p)
(transit seek)))))
(seek
(let* ((o (object))
(target (target o))
(r (range o))
(losp (line-of-sight-p o target r)))
(when losp
(let ((dir (find-direction o target)))
(setf (movement o) (object-speed o dir))))
(unless losp
(transit ground)))))
Than it is to read:
(progn
(defclass ray-ai (ai) nil (:default-initargs :current 'ground))
(defmethod gen-act ((ai ray-ai) (state (eql 'ground)))
(macrolet ((transit (state)
(list 'setf (list 'current 'ai) (list 'quote state))))
(flet ((object ()
(object ai)))
(let* ((o (object)) (r (range o)))
(loop for p in *players*
if (line-of-sight-p o p r)
do (progn (setf (target o) p) (transit seek)))))))
(defmethod gen-act ((ai ray-ai) (state (eql 'seek)))
(macrolet ((transit (state)
(list 'setf (list 'current 'ai) (list 'quote state))))
(flet ((object ()
(object ai)))
(let* ((o (object))
(target (target o))
(r (range o))
(losp (line-of-sight-p o target r)))
(when losp
(let ((dir (find-direction o target)))
(setf (movement o) (object-speed o dir))))
(unless losp (transit ground)))))))
By encapsulating the whole state-machine generation in a macro, I can also ensure that I only refer to defined states and warn if that is not the case.
With macros you can define your own syntax, thus you extend Lisp and make it
suited for the programs you write.
Check out the, very good, online book Practical Common Lisp, for practical examples.
7. Macros: Standard Control Constructs
8. Macros: Defining Your Own
Besides extending the language's syntax to allow you to express yourself more clearly, it also gives you control over evaluation. Try writing your own if in your language of choice so that you can actually write my_if something my_then print "success" my_else print "failure" and not have both print statements get evaluated. In any strict language without a sufficiently powerful macro system, this is impossible. No Common Lisp programmers would find the task too challenging, though. Ditto for for-loops, foreach loops, etc. You can't express these things in C because they require special evaluation semantics (people actually tried to introduce foreach into Objective-C, but it didn't work well), but they are almost trivial in Common Lisp because of its macros.
R, the standard statistics programming language, has macros (R manual, chapter 6). You can use this to implement the function lm(), which analyzes data based on a model that you specify as code.
Here's how it works: lm(Y ~ aX + b, data) will try to find a and b parameters that best fit your data. The cool part is, you can substitute any linear equation for aX + b and it will still work. It's a brilliant feature to make statistics computation easier, and it only works so elegantly because lm() can analyze the equation it's given, which is exactly what Lisp macros do.
Just a guess -- Domain Specific Languages.
Macros are essential in providing access to language features. For instance, in TXR Lisp, I have a single function called sys:capture-cont for capturing a delimited continuation. But this is awkward to use by itself. So there are macros wrapped around it, such as suspend, or obtain and yield which provide alternative models for resumable, suspended execution. They are implemented here.
Another example is the complex macro defstruct which provides syntax for defining a structure type. It compiles its arguments into lambda-s and other material which is passed to the function make-struct-type. If programs used make-struct-type directly for defining OOP structures, they would be ugly:
1> (macroexpand '(defstruct foo bar x y (z 9) (:init (self) (setf self.x 42))))
(sys:make-struct-type 'foo 'bar '()
'(x y z) ()
(lambda (#:g0101)
(let ((#:g0102 (struct-type #:g0101)))
(unless (static-slot-p #:g0102 'z)
(slotset #:g0101 'z
9)))
(let ((self #:g0101))
(setf (qref self x)
42)))
())
Yikes! There is a lot going on that has to be right. For instance, we don't just stick a 9 into slot z because (due to inheritance) we could actually be the base structure of a derived structure, and in the derived structure, z could be a static slot (shared by instances). We would be clobbering the value set up for z in the derived class.
In ANSI Common Lisp, a nice example of a macro is loop, which provides an entire sub-language for parallel iteration. A single loop invocation can express an entire complicated algorithm.
Macros let us think independently about the syntax we would like in a language feature, and the underlying functions or special operators required to implement it. Whatever choices we make in these two, macros will bridge them for us. I don't have to worry that make-struct is ugly to use, so I can focus on the technical aspects; I know that the macro can look the same regardless of how I make various trade-offs. I made the design decision that all struct initialization is going to be done by some functions registered to the type. Okay, that means that my macro has to take all the initializations in the slot-defining syntax, and compile the anonymous functions, where the slot initialization is done by code generated in the bodies.
Macros are compilers for bits of syntax, for which functions and special operators are the target language.
Sometimes people (non-Lisp people, usually) criticize macros in this way: macros don't add any capabilities, only syntactic sugar.
Firstly, syntactic sugar is a capability.
Secondly, you also have to consider macros from a "total hacker perspective": combining macros with implementation-level work. If I'm adding features to a Lisp dialect, such as structures or continuations, I am actually extending the power. The involvement of macros in that enterprise is essential. Even though macros aren't the source of the new power (it doesn't emanate from the macros themselves), they help tame and harness it, giving it expression.
If you don't have sys:capture-cont, you can't just hack up its behavior with a suspend macro. But if you don't have macros, then you have to do something awfully inconvenient to provide access to a new feature that isn't a library function, namely hard-coding some new phrase structure rules into a parser.