Just came across a quote from Peter Norvig in Paradigms in AI Programing:
In the past, eval was seen as the key to Lisp's flexibility. In
modern Lisps with lexical scoping, such as Common Lisp, eval is used
less often (in fact, in Scheme there is no eval at all). Instead,
programmers are expected to use lambda to create a new function, and
then apply or funcall the function.
There are considerations on why eval is not a good idea, such as this, however I am more interested in the interaction between eval and dynamic (special?) variables and how introduction of lexicals contributed to eval losing favor. What were some common eval idioms before the introduction of lexical variables?
Unfortunately, I don't have a citation handy, but from what I remember hearing, the pattern that that text might be referring to is a way to emulate lexical scoping. That is, in lexical Common Lisp, we can simply write:
(defun constant-adder (a)
(lambda (b) (+ a b)))
In a language which used dynamic scope for all variables (but with the same syntax otherwise), lexical scope could be emulated by constructing a lambda expression:
(defun constant-adder (a)
(eval `(lambda (b) (+ (quote ,a) b))))
That is, we are constructing at runtime an expression which has the value of a substituted in, rather than having the language capture a automatically.
(However, your quote suggests that there was also no lambda in use. I have no plausible explanations for what would be happening in that case.)
eval
Evaluates form in the current dynamic environment and the null lexical environment.
This means that
(defun f (a) (eval '(1+ a)))
; warning: The variable A is defined but never used.
The compiler notices that, contrary to a naive expectation, a is not passed to eval.
Indeed:
(f 10)
; error: UNBOUND-VARIABLE: The variable A is unbound.
Related
This question already has answers here:
What makes Lisp macros so special?
(15 answers)
Closed 3 months ago.
I keep reading that Lisp macros are one of the most powerful features of the language. But reading over the specifications and manuals, they are just functions whose arguments are unevaluated.
Given any macro (defmacro example (arg1 ... argN) (body-forms)) I could just write (defun example (arg1 ... argN) ... (body-forms)) with the last body-form turned into a list and then call it like (eval (example 'arg1 ... 'argN)) to emulate the same behavior of the macro. If this were the case, then macros would just be syntactic sugar, but I doubt that syntactic sugar would be called a powerful language feature. What am I missing? Are there cases where I cannot carry out this procedure to emulate a macro?
I can't talk about powerful because it can be a little bit subjective, but macros are regular Lisp functions that work on Lisp data, so they are as expressive as other functions. This isn't the case with templates or generic functions in other languages that rely more on static types and are more restricted (on purpose).
In some way, yes macros are simple syntactic facilities, but you are focused in your emulation on the dynamic semantics of macros, ie. how you can run code that evaluates macros at runtime. However:
the code using eval is not equivalent to expanded code
the preprocessing/compile-time aspect of macros is not emulated
Lexical scope
Function, like +, do not inherit the lexical scope:
(let ((x 30))
(+ 3 4))
Inside the definition of +, you cannot access x. Being able to do so is what "dynamic scope" is about (more precisely, see dynamic extent, indefinite scope variables). But nowadays it is quite the exception to rely on dynamic scope. Most functions use lexical scope, and this is the case for eval too.
The eval function evaluates a form in the null lexical environment, and it never has access to the surrounding lexical bindings. As such, it behaves like any regular function.
So, in you example, calling eval on the transformed source code will not work, since arg1 to argnN will probably be unbound (it depends on what your macro does).
In order to have an equivalent form, you have to inject bindings in the transformed code, or expand at a higher level:
(defun expand-square (var)
(list '* var var))
;; instead of:
(defun foo (x) (eval (expand-square 'x))) ;; x unbound during eval
;; inject bindings
(defun foo (x) (eval `(let ((z ,x)) (expand-square z))))
;; or expand the top-level form
(eval `(defun foo (x) ,(expand-square 'x)))
Note that macros (in Common Lisp) also have access to the lexical environment through &environment parameters in their lambda-list. The use of this environment is implementation dependent, but can be used to access the declarations associated with a variable, for example.
Notice also how in the last example you evaluate the code when defining the function, and not when running it. This is the second thing about macro.
Expansion time
In order to emulate macros you could locally replace a call to a macro by a form that emulates it at runtime (using let to captures all the bindings you want to see inside the expanded code, which is tedious), but then you would miss the useful aspect of macros that is: generating code ahead of time.
The last example above shows how you can quote defun and wrap it in eval, and basically you would need to do that for all functions if you wanted to emulate the preprocessing work done by macros.
The macro system is a way to integrate this preprocessing step in the language in a way that is simple to use.
Conclusion
Macros themselves are a nice way to abstract things when functions can't. For example you can have a more human-friendly, stable syntax that hides implementation details. That's how you define pattern-matching abilities in Common Lisp that make it look like they are part of the language, without too much runtime penalty or verbosity.
They rely on simple term-rewriting functions that are integrated in the language, but you can emulate their behavior either at compile-time or runtime yourself if you want. They can be used to perform different kinds of abstraction that are usually missing or more cumbersome to do in other languages, but are also limited: they don't "understand" code by themselves, they don't give access to all the facilities of the compiler (type propagation, etc.). If you want more you can use more advanced libraries or compiler tools (see deftransform), but macros at least are portable.
Macros are not just functions whose arguments are unevaluated. Macros are functions between programming languages. In other words a macro is a function whose argument is a fragment of source code of a programming language which includes the macro, and whose value is a fragment of source code of a language which does not include the macro (or which includes it in a simpler way).
In very ancient, very rudimentary, Lisps, before people really understood what macros were, you could simulate macros with things called FEXPRs combined with EVAL. A FEXPR was simply a function which did not evaluate its arguments. This worked in such Lisps only because they were completely dynamically scoped, and the cost of it working was that compilation of such things was not possible at all. Those are two enormous costs.
In any modern Lisp, this won't work at all. You can write a toy version of FEXPRs as a macro (this may be buggy):
(defmacro deffex (fx args &body body)
(assert (every (lambda (arg)
(and (symbolp arg)
(not (member arg lambda-list-keywords))))
args)
(args) "not a simple lambda list")
`(defmacro ,fx ,args
`(let ,(mapcar (lambda (argname argval)
`(,argname ',argval))
',args (list ,#args))
,#',body)))
So now we could try to write a trivial binding construct I'll call with using this thing:
(deffex with (var val form)
(eval `(let ((,var ,val)) ,form)))
And this seems to work:
> (with a 1 a)
1
Of course, we're paying the cost that no code which uses this construct can ever be compiled so all our programs will be extremely slow, but perhaps that is a cost we're willing to accept (it's not, but never mind).
Except, of course, it doesn't work, at all:
> (with a 1
(with b 2
(+ a b)))
Error: The variable a is unbound.
Oh dear.
Why doesn't it work? It doesn't work because Common Lisp is lexically scoped, and eval is a function: it can't see the lexical bindings.
So not only does this kind of approach prevent compilation in a modern Lisp, it doesn't work at all.
People often, at this point, suggest some kind of kludge solution which would allow eval to be able to see lexical bindings. The cost of such a solution is that all the lexical bindings need to exist in compiled code: no variable can ever be compiled away, not even its name. That's essentially saying that no good compilers can ever be used, even for the small part of your programs you can compile at all in a language which makes extensive use of macros like CL. For instance, if you ever use defun you're not going to be able to compile the code in its body. People do use defun occasionally, I think.
So this approach simply won't work: it worked by happenstance in very old Lisps but it can't work, even at the huge cost of preventing compilation, in any modern Lisp.
More to the point this approach obfuscates the understanding of what macros are: as I said at the start, macros are functions between programming languages, and understanding that is critical. When you are designing macros you are implementing a new programming language.
Say I have a hash table at runtime that has strings as keys. Can a macro have access to this information and build a let expression from it?
(define env (hash 'a 123 'b 321))
(magic-let env (+ a b)) ; 444
I know I can hack around with with identifier-binding by replacing non-defined identifiers with a lookup in the hash table but then shadowing will not work as in a normal let.
Tagging scheme too as I assume its macro system is similar.
No, you can’t do that. At least not the way you describe.
The general reason why you cannot access runtime values within macros is simple: macros are fully expanded at compile time. When your program is compiled, the runtime values simply do not exist. A program can be compiled, and the bytecode can be placed on another computer, which will run it weeks later. Macro-expansion has already happened. No matter what happens at runtime, the program isn’t going to change.
This guarantee turns out to be incredibly important for a multitude of reasons, but that’s too general a discussion for this question. It would be relevant to discuss a particular question, which is why bindings themselves need to be static.
In Racket, as long as you are within a module (i.e. not at the top-level/REPL), all bindings can be resolved statically, at compile-time. This is a very useful property in other programming languages, mostly because the compiler can generate much more efficiently optimized code, but it is especially important in Racket or Scheme. This is because of how the macro system operates: in a language with hygienic macros, scope is complicated.
This is actually a very good thing—it is robust enough to support very complex systems that would be much harder to manage without hygiene—but it introduces some constraints:
Since every binding can be a macro or a runtime value, the binding needs to be known ahead of time in order to perform program expansion. The compiler needs to know if it needs to perform macro expansion or simply emit a variable reference.
Additionally, scoping rules are much more intricate because macro-introduced bindings live in their own scope. Because of this, binding scopes do not need to be strictly lexical.
Your magic-let could not work quite as you describe because the compiler could not possibly deduce the bindings for a and b statically. However, all is not lost: you could hook into #%top, a magical identifier introduced by the expander when encountering an unbound identifier. You could use this to replace unbound values with a hash lookup, and you could use syntax parameters to adjust #%top hygienically within each magic-let. Here’s an example:
#lang racket
(require (rename-in racket/base [#%top base-#%top])
racket/stxparam)
(define-syntax-parameter #%top (make-rename-transformer #'base-#%top))
(define-syntax-rule (magic-let env-expr expr ...)
(let ([env env-expr])
(syntax-parameterize ([#%top (syntax-rules ()
[(_ . id) (hash-ref env 'id)])])
(let () expr ...))))
(magic-let (hash 'a 123 'b 321) (+ a b)) ; => 444
Of course, keep in mind that this would replace all unbound identifiers with hash lookups. The effects of this are twofold. First of all, it will not shadow identifiers that are already bound:
(let ([a 1])
(magic-let (hash 'a 2)
a)) ; => 1
This is probably for the best, just to keep things semi-sane. It also means that the following would raise a runtime exception, not a compile-time error:
(magic-let (hash 'a 123) (+ a b))
; hash-ref: no value found for key
; key: 'b
I wouldn’t recommend doing this, as it goes against a lot of the Racket philosophy, and it would likely cause some hard-to-find bugs. There’s probably a better way to solve your problem without abusing things like #%top. Still, it is possible, if you really want it.
The reason I don't understand why Emacs 24's new lexical scoping features are that great is that I can't think of any new functionality that couldn't have been implemented without them. For example, the following closure:
(setq lexical-binding t)
(defun f1 (num1)
(lambda (num2)
(setq num1 (* num1 num2))))
(fset 'f2 (f1 5))
==> (closure ((num1 . 5) t) (num2) (setq num1 (* num1 num2)))
(f2 5)
==> 25
(f2 2)
==> 50
Can be implemented with regular dynamic scoping like so:
(defun f1 (num)
(let ((tmpvar (make-symbol "num")))
(set tmpvar num)
`(lambda (num2)
(set ',tmpvar (* (eval ,tmpvar) num2)))))
(fset 'f2 (f1 5))
==> (lambda (num2) (set (quote num) (+ (eval num) num2)))
(f2 5)
==> 25
(f2 2)
==> 50
(fset 'f3 (f1 9))
==> (lambda (num2) (set (quote num) (+ (eval num) num2)))
(f3 3)
==> 27
(f3 2)
==> 54
(f2 10)
==> 500
Okay, so not all languages have something analogous to elisp's uninterned symbols, so I understand why lexical scoping is so great in their case. But what about elisp? Can you think of anything that I can do now (as of Emacs 24) that I couldn't do before, thanks to lexical scoping?
You don't need uninterned symbols, use cons instead of make-symbol, car instead of eval, and setcar instead of set and your example will work just as well (and be more efficient).
Note also that the progression from machine-language to ever higher-level languages has been mostly based on making more and more things impossible (or at least much harder). Of course those facilities taken away from the programmers were rarely used and/or considered too dangerous. Think of using uninitialized variables (possible in C, but impossible in Java and many other languages), or jumping into the middle of an instruction.
As for some downside of your example code: not only it's less readable, but the compiler basically won't be able to know that you're constructing code, so it won't be allowed to look inside that "`(lambda ...)" to compile it, expand its macro calls, give you warnings about suspicious elements, ...
There have always been workarounds to simulate lexical binding in Emacs, so it's not so much about being able to do new things.
The manual says:
Lexical binding opens up a lot more opportunities for optimization, so
Emacs Lisp code that makes use of lexical binding is likely to run
faster in future Emacs versions. Such code is also much more friendly
to concurrency, which we want to add to Emacs in the near future.
I think that is the primary benefit.
The flip side to this is that dynamic binding was purposely chosen when Emacs was written for good reasons which still hold true today, and so lexical binding should certainly not be considered the New Way of doing things.
Global variables are generally considered a bad idea in programming, but Emacs is an unusual case where this doesn't really apply, because much of its immense flexibility -- one of the key things that makes Emacs great -- derives directly from dynamic binding. Without dynamic binding, it would not be possible to bend the application to the requirements of the individual user to anything like the extent that Emacs allows.
In my opinion, lexical binding should be used cautiously and only for variables for which another user could not conceivably find a reason to override. By default variables should be defvar'd so that the ability to customize the behaviour (even in ways that the author did not anticipate) is preserved.
I think implementations of OO programming tend to fit well with lexical scope. An object with state maps rather directly to a lexical closure.
I'm sure that the CLOS implementation in common lisp leverages lexical scope heavily. It's hard for me to imagine how that spec could be implemented with dynamic scope only, but I'm sure it's possible.
I have a question concerning evaluation of lists in lisp.
Why is (a) and (+ a 1) not evaluated,
(defun test (a) (+ a 1))
just like (print 4) is not evaluated here
(if (< 1 2) (print 3) (print 4))
but (print (+ 2 3)) is evaluated here
(test (print (+ 2 3)))
Does it have something to do with them being standard library functions? Is it possible for me to define functions like that in my lisp program?
As you probably know, Lisp compound forms are generally processed from the outside in. You must look at the symbol in the first position of the outermost nesting to understand a form. That symbol completely determines the meaning of the form. The following expressions all contain (b c) with completely different meaning; therefore, we cannot understand them by analyzing the (b c) part first:
;; Common Lisp: define a class A derived from B and C
(defclass a (b c) ())
;; Common Lisp: define a function of two arguments
(defun a (b c) ())
;; add A to the result of calling function B on variable C:
(+ a (b c))
Traditionally, Lisp dialects have divided forms into operator forms and function call forms. An operator form has a completely arbitrary meaning, determined by the piece of code which compiles or interprets that functions (e.g. the evaluation simply recurses over all of the function call's argument forms, and the resulting values are passed to the function).
From the early history, Lisp has allowed users to write their own operators. There existed two approaches to this: interpretive operators (historically known as fexprs) and compiling operators known as macros. Both hinge around the idea of a function which receives the unevaluated form as an argument, so that it can implement a custom strategy, thereby extending the evaluation model with new behaviors.
A fexpr type operator is simply handed the form at run-time, along with an environment object with which it can look up the values of variables and such. That operator then walks the form and implements the behavior.
A macro operator is handed the form at macro-expansion time (which usually happens when top-level forms are read, just before they are evaluated or compiled). Its job is not to interpret the form's behavior, but instead to translate it by generating code. I.e. a macro is a mini compiler. (Generated code can contain more macro calls; the macro expander will take care of that, ensuring that all macro calls are decimated.)
The fexpr approach fell out of favor, most likely because it is inefficient. It basically makes compilation impossible, whereas Lisp hackers valued compilation. (Lisp was already a compiled language from as early as circa 1960.) The fexpr approach is also hostile toward lexical environments; it requires the fexpr, which is a function, to be able to peer into the variable binding environment of the form in which its invoked, which is a kind of encapsulation violation that is not allowed by lexical scopes.
Macro writing is slightly more difficult, and in some ways less flexible than fexprs, but support for macro writing improved in Lisp through the 1960's into the 70's to make it close to as easy as possible. Macro originally had receive the whole form and then have to parse it themselves. The macro-defining system developed into something that provides macro functions with arguments that receive the broken-down syntax in easily digestible pieces, including some nested aspects of the syntax. The backquote syntax for writing code templates was also developed, making it much easier to express code generation.
So to answer your question, how can I write forms like that myself? For instance if:
;; Imitation of old-fashioned technique: receive the whole form,
;; extract parts from it and return the translation.
;; Common Lisp defmacro supports this via the &whole keyword
;; in macro lambda lists which lets us have access to the whole form.
;;
;; (Because we are using defmacro, we need to declare arguments "an co &optional al",
;; to make this a three argument macro with an optional third argument, but
;; we don't use those arguments. In ancient lisps, they would not appear:
;; a macro would be a one-argument function, and would have to check the number
;; of arguments itself, to flag bad syntax like (my-if 42) or (my-if).)
;;
(defmacro my-if (&whole if-form an co &optional al)
(let ((antecedent (second if-form)) ;; extract pieces ourselves
(consequent (third if-form)) ;; from whole (my-if ...) form
(alternative (fourth if-form)))
(list 'cond (list antecedent consequent) (list t alternative))))
;; "Modern" version. Use the parsed arguments, and also take advantage of
;; backquote syntax to write the COND with a syntax that looks like the code.
(defmacro my-if (antecedent consequent &optional alternative)
`(cond (,antecedent ,consequent) (t ,alternative))))
This is a fitting example because originally Lisp only had cond. There was no if in McCarthy's Lisp. That "syntactic sugar" was invented later, probably as a macro expanding to cond, just like my-if above.
if and defun are macros. Macros expand a form into a longer piece of code. At expansion time, none of the macro's arguments are evaluated.
When you try to write a function, but struggle because you need to implement a custom evaluation strategy, its a strong signal that you should be writing a macro instead.
Disclaimer: Depending on what kind of lisp you are using, if and defun might technically be called "special forms" and not macros, but the concept of delayed evaluation still applies.
Lisp consists of a model of evaluation of forms. Different Lisp dialects have different rules for those.
Let's look at Common Lisp.
data evaluates to itself
a function form is evaluated by calling the function on the evaluated arguments
special forms are evaluated according to rules defined for each special operator. The Common Lisp standard lists all of those, defines what they do in an informal way and there is no way to define new special operators by the user.
macros forms are transformed, the result is evaluated
How IF, DEFUN etc. works and what they evaluated, when it is doen and what is not evaluated is defined in the Common Lisp standard.
I've learned Clojure previously and really like the language. I also love Emacs and have hacked some simple stuff with Emacs Lisp. There is one thing which prevents me mentally from doing anything more substantial with Elisp though. It's the concept of dynamic scoping. I'm just scared of it since it's so alien to me and smells like semi-global variables.
So with variable declarations I don't know which things are safe to do and which are dangerous. From what I've understood, variables set with setq fall under dynamic scoping (is that right?) What about let variables? Somewhere I've read that let allows you to do plain lexical scoping, but somewhere else I read that let vars also are dynamically scoped.
I quess my biggest worry is that my code (using setq or let) accidentally breaks some variables from platform or third-party code that I call or that after such call my local variables are messed up accidentally. How can I avoid this?
Are there a few simple rules of thumb that I can just follow and know exactly what happens with the scope without being bitten in some weird, hard-to-debug way?
It isn't that bad.
The main problems can appear with 'free variables' in functions.
(defun foo (a)
(* a b))
In above function a is a local variable. b is a free variable. In a system with dynamic binding like Emacs Lisp, b will be looked up at runtime. There are now three cases:
b is not defined -> error
b is a local variable bound by some function call in the current dynamic scope -> take that value
b is a global variable -> take that value
The problems can then be:
a bound value (global or local) is shadowed by a function call, possibly unwanted
an undefined variable is NOT shadowed -> error on access
a global variable is NOT shadowed -> picks up the global value, which might be unwanted
In a Lisp with a compiler, compiling the above function might generate a warning that there is a free variable. Typically Common Lisp compilers will do that. An interpreter won't provide that warning, one just will see the effect at runtime.
Advice:
make sure that you don't use free variables accidentally
make sure that global variables have a special name, so that they are easy to spot in source code, usually *foo-var*
Don't write
(defun foo (a b)
...
(setq c (* a b)) ; where c is a free variable
...)
Write:
(defun foo (a b)
...
(let ((c (* a b)))
...)
...)
Bind all variables you want to use and you want to make sure that they are not bound somewhere else.
That's basically it.
Since GNU Emacs version 24 lexical binding is supported in its Emacs Lisp. See: Lexical Binding, GNU Emacs Lisp Reference Manual.
In addition to the last paragraph of Gilles answer, here is how RMS argues in favor of dynamic scoping in an extensible system:
Some language designers believe that
dynamic binding should be avoided, and
explicit argument passing should be
used instead. Imagine that function A
binds the variable FOO, and calls the
function B, which calls the function
C, and C uses the value of FOO.
Supposedly A should pass the value as
an argument to B, which should pass it
as an argument to C.
This cannot be done in an extensible
system, however, because the author of
the system cannot know what all the
parameters will be. Imagine that the
functions A and C are part of a user
extension, while B is part of the
standard system. The variable FOO does
not exist in the standard system; it
is part of the extension. To use
explicit argument passing would
require adding a new argument to B,
which means rewriting B and everything
that calls B. In the most common case,
B is the editor command dispatcher
loop, which is called from an awful
number of places.
What's worse, C must also be passed an
additional argument. B doesn't refer
to C by name (C did not exist when B
was written). It probably finds a
pointer to C in the command dispatch
table. This means that the same call
which sometimes calls C might equally
well call any editor command
definition. So all the editing
commands must be rewritten to accept
and ignore the additional argument. By
now, none of the original system is
left!
Personally, I think that if there is a problem with Emacs-Lisp, it is not dynamic scoping per se, but that it is the default, and that it is not possible to achieve lexical scoping without resorting to extensions. In CL, both dynamic and lexical scoping can be used, and -- except for top-level (which is adressed by several deflex-implementations) and globally declared special variables -- the default is lexical scoping. In Clojure, too, you can use both lexical and dynamic scoping.
To quote RMS again:
It is not necessary for dynamic scope to be the only scope rule provided, just useful
for it to be available.
Are there a few simple rules of thumb that I can just follow and know exactly what happens with the scope without being bitten in some weird, hard-to-debug way?
Read Emacs Lisp Reference, you'll have many details like this one :
Special Form: setq [symbol form]...
This special form is the most common method of changing a
variable's value. Each SYMBOL is given a new value, which is the
result of evaluating the corresponding FORM. The most-local
existing binding of the symbol is changed.
Here is an example :
(defun foo () (setq tata "foo"))
(defun bar (tata) (setq tata "bar"))
(foo)
(message tata)
===> "foo"
(bar tata)
(message tata)
===> "foo"
As Peter Ajtai pointed out:
Since emacs-24.1 you can enable lexical scoping on a per file basis by putting
;; -*- lexical-binding: t -*-
on top of your elisp file.
First, elisp has separate variable and function bindings, so some pitfalls of dynamic scoping are not relevant.
Second, you can still use setq to set variables, but the value set does not survive the exit of the dynamic scope it is done in. This isn't, fundamentally, different from lexical scoping, with the difference that with dynamic scoping a setq in a function you call can affect the value you see after the function call.
There's lexical-let, a macro that (essentially) imitates lexical bindings (I believe it does this by walking the body and changing all occurrences of the lexically let variables to a gensymmed name, eventually uninterning the symbol), if you absolutely need to.
I'd say "write code as normal". There are times when the dynamic nature of elisp will bite you, but I've found that in practice that is surprisingly seldom.
Here's an example of what I was saying about setq and dynamically-bound variables (recently evaluated in a nearby scratch buffer):
(let ((a nil))
(list (let ((a nil))
(setq a 'value)
a)
a))
(value nil)
Everything that has been written here is worthwhile. I would add this: get to know Common Lisp -- if nothing else, read about it. CLTL2 presents lexical and dynamic binding well, as do other books. And Common Lisp integrates them well in a single language.
If you "get it" after some exposure to Common Lisp then things will be clearer for you for Emacs Lisp. Emacs 24 uses lexical scoping to a greater extent by default than older versions, but Common Lisp's approach will still be clearer and cleaner (IMHO). Finally, it is definitely the case that dynamic scope is important for Emacs Lisp, for the reasons that RMS and others have emphasized.
So my suggestion is to get to know how Common Lisp deals with this. Try to forget about Scheme, if that is your main mental model of Lisp -- it will limit you more than help you in understanding scoping, funargs, etc. in Emacs Lisp. Emacs Lisp, like Common Lisp, is "dirty and low-down"; it is not Scheme.
Dynamic and lexical scoping have different behaviors when a piece of code is used in a different scope than the one it was defined in. In practice, there are two patterns that cover most troublesome cases:
A function shadows a global variable, then calls another function that uses that global variable.
(defvar x 3)
(defun foo ()
x)
(defun bar (x)
(+ (foo) x))
(bar 0) ⇒ 0
This doesn't come up often in Emacs because local variables tend to have short names (often single-word) whereas global variables tend to have long names (often prefixed by packagename-). Many standard functions have names that are tempting to use as local variables like list and point, but functions and variables live in separate name spaces are local functions are not used very often.
A function is defined in one lexical context and used outside this lexical context because it's passed to a higher-order function.
(let ((cl-y 10))
(mapcar* (lambda (elt) (* cl-y elt)) '(1 2 3)))
⇒ (10 20 30)
(let ((cl-x 10))
(mapcar* (lambda (elt) (* cl-x elt)) '(1 2 3)))
⇑ (wrong-type-argument number-or-marker-p (1 2 3))
The error is due to the use of cl-x as a variable name in mapcar* (from the cl package). Note that the cl package uses cl- as a prefix even for its local variables in higher-order functions. This works reasonably well in practice, as long as you take care not to use the same variable as a global name and as a local name, and you don't need to write a recursive higher-order function.
P.S. Emacs Lisp's age isn't the only reason why it's dynamically scoped. True, in those days, lisps tended towards dynamic scoping — Scheme and Common Lisp hadn't really taken on yet. But dynamic scoping is also an asset in a language targeted towards extending a system dynamically: it lets you hook into more places without any special effort. With great power comes great rope to hang yourself: you risk accidentally hooking into a place you didn't know about.
The other answers are good at explaining the technical details on how to work with dynamic scoping, so here's my non-technical advice:
Just do it
I've been tinkering with Emacs lisp for 15+ years and don't know that I've ever been bitten by any problems due to the differences between lexical/dynamic scope.
Personally, I've not found the need for closures (I love 'em, just don't need them for Emacs). And, I generally try to avoid global variables in general (whether the scoping was lexical or dynamic).
So I suggest jumping in and writing customizations that suit your needs/desires, chances are you won't have any problems.
I entirely feel your pain. I find the lack of lexical binding in emacs rather annoying - especially not being able to use lexical closures, which seems to be a solution I think of a lot, coming from more modern languages.
While I don't have any more advice on working around the lacking features that the previous answers didn't cover yet, I'd like to point out the existance of an emacs branch called `lexbind', implementing lexical binding in a backward-compatible way. In my experience lexical closures are still a little buggy in some circumstances, but that branch appears to a promising approach.
Just don't.
Emacs-24 lets you use lexical-scope. Just run
(setq lexical-binding t)
or add
;; -*- lexical-binding: t -*-
at the beginning of your file.