I'm writing an emacs major mode for a programming environment which supports two slightly different programming languages. Both are lisps (one is Scheme), so both use s-expressions. The two languages are differentiated by their function definition keyword: scheme uses define, while the other language (which is called xtlang) uses bind-func
So, in the same buffer, I have
(define foo ; this is scheme
(lambda (a)
(+ scheme-fn a))
(bind-func bar ; this is xtlang
(lambda (b)
(* xtlang-fn b))
I've got the font-locking working, using something like
(font-lock-add-keywords nil
'(("(\\(define\\)\\>"
(1 font-lock-keyword-face))
("(\\(bind-func\\)\\>"
(1 font-lock-keyword-face))
("\\<scheme-fn\\>"
(0 font-lock-function-name-face))
("\\<xtlang-fn\\>"
(0 font-lock-function-name-face))
))
What I'd like to be able to do is to be able to colour the parens differently based on the language (scheme/xtlang).
So, colour all the parens in the form red if the top-level defun is a define, and blue if it's a bind-func, while still highlighting all the keywords/functions within the forms as normal.
This may require multi-line font locking, since the define/bind-func will probably be on a previous line to the keywords to be highlighted. This thread suggests that font-lock-multiline in conjunction with match-anchored in font-lock-keywords may be the answer, but then goes on to suggest that font-lock-multiline should only be used in situations where the multi-line aspect is the exception rather than the rule.
My other option seems to be to use syntax-propertize, but I'm a bit confused about how that works - the documentation is a bit sparse.
The easiest way to handle this is utilizing the following fact:
MATCHER can be either the regexp to search for, or the function name
to call to make the search (called with one argument, the limit of the
search;
In other words, you could replace regexps like "\\<scheme-fn\\>" with a function that repeatedly search for the regexp using re-search-forward, and return if a match is found in the correct context.
For a concrete example of a package that use this technique, look at cwarn-mode, which is part of the standard Emacs distribution.
Related
Say I have macros foo and bar. If I write (foo (bar)) my understanding is that in most (all?) lisps foo is going to be given '(bar), not whatever bar would have expanded to had it been expanded first. Some lisps have something like local-expand where the implementation of foo can explicitly request the expansion of its argument before continuing, but why isn't that the default? It seems more natural to me. Is this an accident of history or is there a strong reason to do it the way most lisps do it?
I've been noticing in Rust that I want the macros to work this way. I'd like to be able to wrap a bunch of declarations inside a macro call so that the macro can then crawl the declarations and generate reflection information. But if I use a macro to generate the definitions that I want crawled, my macro wrapping the declarations sees the macro invocations that would generate the declarations rather than the actual declarations.
If I write (foo (bar)) my understanding is that in most (all?) lisps foo is going to be given '(bar), not whatever bar would have expanded to had it been expanded first.
That would restrict Lisp such that (bar) would need to be something that can be expanded -> probably something which is written in the Lisp language.
Lisp developers would like to see macros where the inner stuff can be a completely new language with different syntax rules. For example something, where FOO does not expand it's subforms, but transpiles/compiles a fully/partially different language to Lisp. Something which does not have the usual prefix expression syntax:
Examples
(postfix (a b +) sin)
-> (sin (+ a b))
Here the + in the macro form is not the infix +.
or
(query-all (person name)
where (person = "foo") in database DB)
Lisp macros don't work on language parse trees, but arbitrary, possibly nested, s-expressions. Those don't need to be valid Lisp code outside of that macro -> don't need to follow the usual syntax / semantics.
Common Lisp has the function MACROEXPAND and MACROEXPAND-1, such that the outer macro can expand inner code if it wants to do it during its own macro expansion:
CL-USER 26 > (defmacro bar (a) `(* ,a ,a))
BAR
CL-USER 27 > (bar 10)
100
CL-USER 28 > (defmacro foo (a &environment e)
(let ((f (macroexpand a e)))
(print (list a '-> f))
`(+ ,(second f) ,(third f))))
FOO
CL-USER 29 > (foo (bar 10))
((bar 10) -> (* 10 10))
20
In above macro, if FOO would see only an expanded form, it could not print both the source and the expansion.
This works also with scoped macros. Here the BAR macro gets locally redefined and the MACROEXPAND generates different code inside FOO for the same form:
CL-USER 30 > (macrolet ((bar (a)
`(expt ,a ,a)))
(foo (bar 10)))
((bar 10) -> (EXPT 10 10))
20
If foo is a macro then (foo (bar)) must pass the raw syntax (bar) to the foo macro expander. This is absolutely essential.
This is because foo can give any meaning whatsoever to bar.
Consider the defmacro macro itself:
(defmacro foo (bar) body)
Here, the argument (bar) is a parameter list ("macro lambda list") and not a form (Common Lisp jargon for to-be-evaluated expression). It says that the macro shall have a single parameter called bar. Therefore it is nonsensically wrong to try to expand (bar) before handing it to defmacro's expander.
Only if we know that an expression is going to be evaluated is it legitimate to expand it as a macro. But we don't know that about an expression which is the argument to a macro.
Other counterexamples are easy to come up with. (defstruct point (x 0) (y 0)): (x 0) isn't a call to operator x, but a slot x whose default value is 0. (dolist (x list) ...): x is a variable to be stepped over list.
That said, there are implementation choices regarding the timing of macro expansion.
A Lisp implementation can macro-expand an entire top-level form before evaluating or compiling any of it. Or it can expand incrementally, so that for instance when (+ x y) is being processed, x is macro-expanded and evaluated or compiled into some intermediate form already before y is even looked at.
A pure syntax tree interpreter for Lisp which always keeps the code in the original form and always expands (and re-expands) the code as it is evaluating has certain interactivity advantages. Any macro that you rewrite goes instantly "live" in all the existing code that you have input into the REPL, like existing function definitions. It is obviously quite inefficient in terms of execution speed, but any code that you call uses the latest definition of your macros without any hassle of telling the system to reload that code to have it expanded again. That also eliminates the risk that you're testing something that is still based on the old, buggy version of some macro that you fixed. If you're ever writing a Lisp, the range of timing choices for expansion is good to keep in mind, so that you consciously reject the choices you don't go with.
In turn, that said, there are some constraints on the timing of macro expansion. Conceivably, a Lisp interpreter or compiler, when processing an entire file, could go through all the top level forms and expand all of them at once before processing any of them. The implementor will quickly learn that this is bad, because some of the later forms depend on the side effects of the earlier forms. Such as, oh, macros being defined! If the first form defines a macro, which the second one uses, then we cannot expand the second form without evaluating the effect of the first.
It makes sense, in a Lisp, to split up physical top-level forms into logical ones. Suppose that someone writes (or uses a macro to generate) codde like (progn (defmacro foo ...) (foo)). This entire progn cannot be macro expanded up-front before evaluation; it won't work! There has to be a rule such as "whenever a top-level form is based on the progn operator, then the children of the progn operator are considered top-level forms by all the processing which treats top-level forms specially, and this rule is recursively applied." The top-level entry point into the macro-expanding code walker then has to contain special case hacks to do this recognition of logical top-level forms, breaking them up and recursing into a lower level expander which doesn't do those checks any more.
I've been noticing in Rust that I want the macros to work this way.
I'd like to be able to wrap a bunch of declarations inside a macro
call so that the macro can then crawl the declarations and generate
reflection information.
It does sound like local-expand is the right tool for that job.
However, an alternative approach would be something like this:
Suppose that wrapper is our outer macro, and that the intended
syntax is:
(wrapper decl1 decl2 ...)
where decl is a declaration that potenteally uses some standard form declare.
We can let
(wrapper decl1 decl2 ...)
expand to
(let-syntax ([declare our-declare])
decl1 decl2 ...
(post-process-reflection-information))
where our-declare is a helper macro that expands both to the standard declaration as well as some form that stores the reflection information,
also post-process-reflection-information is another macro that
does any needed post processing.
I think you are trying to use macros for something they are not designed to solve. Macros are primarily a text/code substitution mechanism, and in the case of Lisp this looks a lot like a simplified term-rewriting system (see also How does term-rewriting based evaluation work?). There are different strategies possible for how to substitute a pattern of code, and in which order, but in C/C++ preprocessor macros, in LaTeX, and in Lisp, the process is typically done by computing the expansion until the form is no longer expandable, starting from the topmost terms. This order is quite natural and because it is distinct from normal evaluation rules, it can be used to implement things the normal evaluation rules cannot.
In your case, you are interested in getting access to all the declarations of some object/type, something which falls under the introspection/reflection category (as you said yourself). But implementing reflection/introspection with macros doesn't look totally doable, since macros work on abstract syntax trees and this might be a poor way to access the metadata you want.
Typically the compiler is going to parse/analyze the struct definitions and build the definitive, canonical representation of the struct, even if there are different way to express that syntactically; it may even use prior information not available directly as source code to compute more interesting metadata (e.g. if you had inheritance, there could be a set of properties inherited from a type defined in another module (I don't think this applies to Rust)).
I think currently Rust does not offer compile-time or runtime introspection facilities, which explains why are you going with the macro route. In Common Lisp macros are definitely not used for introspection, the actual values obtained after evaluation (at different times) is used to gain information about an object. For example, defclass expands as a set of instructions that register a class in the language, but in order to get all the slots of a class, you ask the language to give it to you, e.g:
(defclass foo () (x)) ;; define class foo with slot X
(defclass bar () (y)) ;; define class bar with slot Y
(defclass zot (foo bar) ()) ;; define class zot with foo and bar as superclasses
USER> (c2mop:class-slots (find-class 'zot))
(#<SB-MOP:STANDARD-EFFECTIVE-SLOT-DEFINITION X>
#<SB-MOP:STANDARD-EFFECTIVE-SLOT-DEFINITION Y>)
I don't know what the solution for your problem is, but in addition to the other answers, I think it is not specifically a fault of the macro system. If a macro is defined as done usually as only a term rewriting system, it will always have difficulties to perform some tasks on the semantic level. But Rust is still evolving so there might be better ways to do things in the future.
I love Racket's #;. I want to see it in every language that I ever use again. Can it be added to other Lisps via their macro systems? Or does the commenting character break the macro system's ability to read the code?
A sufficient answer will demonstrate a macro being built in any Lisp other than Racket that allows for a change in the commenting system. You need not actually implement Racket's #;, but I would like it if you do. Lisps with the least similarity to Racket, e.g. Clojure or any non-Scheme will be particularity nice to see.
#; isn't a macro, it's what Common lisp would call a readmacro: what it does is defined at read time, not later than that. Read macros which aim to completely suppress input are mildly perilous because there needs to be a way of saying 'read the following thing, but ignore it', and that's only possible if any other readmacros behave well: there's nothing to stop someone defining a readmacro which produces some side-effect even if reading is suppressed.
However, well-behaved readmacros (which includes all of the standard ones and the rest of the standard reader) in CL won't do that: they'll listen to whether reading is being suppressed, and behave accordingly.
CL allows you to do this as standard by using its conditionalisation on features, and in particular #+(or) <expr> will always skip <expr>.
But you can define your own: #; is not predefined so you can define it:
(set-dispatch-macro-character
#\# #\;
(lambda (stream char n)
(declare (ignore char))
(let ((*read-suppress* t))
(dotimes (i (or n 1) (values))
(read stream)))))
After this, or at least with a better tested version of this, then #; <expr> (or obviously #;<expr>) will read as whitespace, and #2; ... ... will skip two following expressions:
> (let ((x #;1 #2; 2 3 4)) x)
4
What you are looking for is #+(or) reader macro.
Since (or) evaluates to nil, the condition is always false the following form is never evaluated.
I have some s-expression (let ((whatever (foo (bar 4) (bar 5)))) ...) (as a result of (read "(whatever (foo (bar 4) (bar 5)))"). Now I would like to transform it with a custom macro that expands for example foo into the macro expression.
What is the mechanism to apply macros to given S-Expressions? Note that I do not want this substitution to happen at compiletime of the .scm file but but after reading in the s expression.
Furthermore, I don't want to use eval, since these S-Expressions are not Scheme code.
I would not want to expand a let statement in that given s-expression, only foo.
Basically I thought that Scheme would contain a sufficiently flexible Tree-Transformation Framework that would enable me to use specific macros for manipulating any kind of S-expressions at any time. In the meantime however, I learned that Scheme's macros are not first class objects and cannot be used this way, as also implied by Sylwester.
I presume you want to expand data and end up with data. If that is not the case, please update your question with more information.
You'll need to run your own macro-implementation. Let's say you create macros with make-macro and expand them with expand-macros such that only expands you user defined macros.
Then the solution would be:
(let ((whatever (expand-macros (read))))
...)
There are lots of macro-implementation you can look at. Alexpander is one based on on hygienic syntax-rules while a classical unygenic defmacro + expander would be very easy. However, it would impossible to make if completely safe. Imagine your foo:
(let ((foo +))
(foo 5 6))
This should override you definition of foo with + in the let. To fix this your expander needs to know all language constructs of Scheme that you are going to use and be able to understand that it's bindings can become shadowed. All in all you'll almost need it to be an interpreter for Scheme to be able to see what to expand or not. It's not so easy, but it's doable.
Are there any practical differences between special forms and macros? In what do they differ?
The terms aren't quite synonymous, but they aren't exclusive either (this answer assumes Scheme):
A special form (also known as a syntax in the Scheme Reports) is an expression that's not evaluated according to the default rule for function application. (The default rule, just to be explicit, is to eval all of the subexpressions, and then apply the result of the first one to the list of the results of the others.)
The macro system is a language feature that allows definition of new special forms within the language itself. A macro is a special form defined using the macro system.
So you could say that "special form" is a term that pertains to interface or semantics, whereas "macro" is a term that pertains to implementation. "Special form" means "these expressions are evaluated with a special rule," while "macro" means "here's an implementation of a special rule for evaluating some expressions."
Now one important thing is that most Scheme special forms can be defined as macros from a really small core of primitives: lambda, if and macros. A minimal Scheme implementation that provides only these can still implement the rest as macros; recent Scheme Reports have made that distinction by referring to such special forms as "library syntax" that can be defined in terms of macros. In practice, however, practical Scheme systems often implement a richer set of forms as primitives.
Semantically speaking, the only thing that matters about an expression is what rule is used to evaluate it, not how that rule is implemented. So in that sense, it's not important whether a special form is implemented as a macro or a primitive. But on the other hand, the implementation details of a Scheme system often "leak," so you may find yourself caring about it...
Lisp has certain language primitives, which make up Lisp forms:
literal data: numbers, strings, structures, ...
function calls, like (sin 2.1) or like ((lambda (a b) (+ a b 2)) 3 4)
special operators used in special forms. These are the primitive built-in language elements. See Special Operators in Common Lisp. These need to be implemented in the interpreter and compiler. Common Lisp provides no way for the developer to introduce new special operators or to provide your own version of these. A code parsing tool will need to understand these special operators; these tools are usually called 'code walkers' in the Lisp community. During the definition of the Common Lisp standard, it was made sure that the number is very small and that all extensions otherwise are done via new functions and new macros.
macros: macros are functions which are transforming source code. The transformation will happen recursively until no macro is left in the source code. Common Lisp has built-in macros and allows the user to write new ones.
So the most important practical difference between special forms and macros is this: special operators are built-in syntax and semantics. They can't be written by the developer. Macros can be written by the developer.
In contrast to special forms, macro forms can be macroexpanded:
CL-USER(1): (macroexpand '(with-slots (x y z)
foo
(format t "~&X = ~A" x)))
(LET ((#:G925 FOO))
(DECLARE (IGNORABLE #:G925))
(DECLARE (SB-PCL::%VARIABLE-REBINDING #:G925 FOO))
#:G925
(SYMBOL-MACROLET ((X (SLOT-VALUE #:G925 'X))
(Y (SLOT-VALUE #:G925 'Y))
(Z (SLOT-VALUE #:G925 'Z)))
(FORMAT T "~&X = ~A" X)))
T
For me the most practical difference has been in the debugger: Macros don't show up in the debugger; instead, the (typically) obscure code from the macro's expansion shows up in the debugger. It is a real pain to debug such code and a good reason to ensure your macros are rock solid before you start relying upon them.
the super short answer for the lazy
You can write your own macroes any time you want, though you can't add special forms without recompiling clojure.
Why is the function/macro dichotomy present in Common Lisp?
What are the logical problems in allowing the same name representing both a macro (taking precedence when found in function position in compile/eval) and a function (usable for example with mapcar)?
For example having second defined both as a macro and as a function would allow to use
(setf (second x) 42)
and
(mapcar #'second L)
without having to create any setf trickery.
Of course it's clear that macros can do more than functions and so the analogy cannot be complete (and I don't think of course that every macro shold also be a function) but why forbidding it by making both sharing a single namespace when it could be potentially useful?
I hope I'm not offending anyone, but I don't really find a "Why doing that?" response really pertinent... I'm looking for why this is a bad idea. Imposing an arbitrary limitation because no good use is known is IMO somewhat arrogant (sort of assumes perfect foresight).
Or are there practical problems in allowing it?
Macros and Functions are two very different things:
macros are using source (!!!) code and are generating new source (!!!) code
functions are parameterized blocks of code.
Now we can look at this from several angles, for example:
a) how do we design a language where functions and macros are clearly identifiable and are looking different in our source code, so we (the human) can easily see what is what?
or
b) how do we blend macros and functions in a way that the result is most useful and has the most useful rules controlling its behavior? For the user it should not make a difference to use a macro or a function.
We really need to convince ourselves that b) is the way to go and we would like to use a language where macros and functions usage looks the same and is working according to similar principles. Take ships and cars. They look different, their use case is mostly different, they transport people - should we now make sure that the traffic rules for them are mostly identical, should we make them different or should we design the rules for their special usage?
For functions we have problems like: defining a function, scope of functions, life-time of functions, passing functions around, returning functions, calling functions, shadowing of functions, extension of functions, removing the definition a function, compilation and interpretation of functions, ...
If we would make macros appear mostly similar to functions, we need to address most or all above issues for them.
In your example you mention a SETF form. SETF is a macro that analyses the enclosed form at macro expansion time and generates code for a setter. It has little to do with SECOND being a macro or not. Having SECOND being a macro would not help at all in this situation.
So, what is a problem example?
(defmacro foo (a b)
(if (and (numberp b) (zerop b))
a
`(- ,a ,b)))
(defun bar (x list)
(mapcar #'foo (list x x x x) '(1 2 3 4)))
Now what should that do? Intuitively it looks easy: map FOO over the lists. But it isn't. When Common Lisp was designed, I would guess, it was not clear what that should do and how it should work. If FOO is a function, then it was clear: Common Lisp took the ideas from Scheme behind lexically scoped first-class functions and integrated it into the language.
But first-class macros? After the design of Common Lisp a bunch of research went into this problem and investigated it. But at the time of Common Lisp's design, there was no wide-spread use of first-class macros and no experience with design approaches. Common Lisp is standardizing on what was known at the time and what the language users thought necessary to develop (the object-system CLOS is kind of novel, based on earlier experience with similar object-systems) software with. Common Lisp was not designed to have the theoretically most pleasing Lisp dialect - it was designed to have a powerful Lisp which allows the efficient implementation of software.
We could work around this and say, passing macros is not possible. The developer would have to provide a function under the same name, which we pass around.
But then (funcall #'foo 1 2) and (foo 1 2) would invoke different machineries? In the first case the function fooand in the second case we use the macro foo to generate code for us? Really? Do we (as human programmers) want this? I think not - it looks like it makes programming much more complicated.
From a pragmatic point of view: Macros and the mechanism behind it are already complicated enough that most programmers have difficulties dealing with it in real code. They make debugging and code understanding much harder for a human. On the surface a macro makes code easier to read, but the price is the need to understand the code expansion process and result.
Finding a way to further integrate macros into the language design is not an easy task.
readscheme.org has some pointers to Macro-related research wrt. Scheme: Macros
What about Common Lisp
Common Lisp provides functions which can be first-class (stored, passed around, ...) and lexically scoped naming for them (DEFUN, FLET, LABELS, FUNCTION, LAMBDA).
Common Lisp provides global macros (DEFMACRO) and local macros (MACROLET).
Common Lisp provides global compiler macros (DEFINE-COMPILER-MACRO).
With compiler macros it is possible to have a function or macro for a symbol AND a compiler macro. The Lisp system can decide to prefer the compiler macro over the macro or function. It can also ignore them entirely. This mechanism is mostly used for the user to program specific optimizations. Thus it does not solve any macro related problems, but provides a pragmatic way to program global optimizations.
I think that Common Lisp's two namespaces (functions and values), rather than three (macros, functions, and values), is a historical contingency.
Early Lisps (in the 1960s) represented functions and values in different ways: values as bindings on the runtime stack, and functions as properties attached to symbols in the symbol table. This difference in implementation led to the specification of two namespaces when Common Lisp was standardized in the 1980s. See Richard Gabriel's paper Technical Issues of Separation in Function Cells and Value Cells for an explanation of this decision.
Macros (and their ancestors, FEXPRs, functions which do not evaluate their arguments) were stored in many Lisp implementations in the symbol table, in the same way as functions. It would have been inconvenient for these implementations if a third namespace (for macros) had been specified, and would have caused backwards-compatibility problems for many programs.
See Kent Pitman's paper Special Forms in Lisp for more about the history of FEXPRs, macros and other special forms.
(Note: Kent Pitman's website is not working for me, so I've linked to the papers via archive.org.)
Because then the exact same name would represent two different objects, depending on the context. It makes the programme unnecessarily difficult to understand.
My TXR Lisp dialect allows a symbol to be simultaneously a macro and function. Moreover, certain special operators are also backed by functions.
I put a bit of thought into the design, and haven't run into any problems. It works very well and is conceptually clean.
Common Lisp is the way it is for historic reasons.
Here is a brief rundown of the system:
When a global macro is defined for symbol X with defmacro, the symbol X does not become fboundp. Rather, what becomes fboundp is the compound function name (macro X).
The name (macro X) is then known to symbol-function, trace and in other situations. (symbol-function '(macro X)) retrieves the two-argument expander function which takes the form and an environment.
It's possible to write a macro using (defun (macro X) (form env) ...).
There are no compiler macros; regular macros do the job of compiler macros.
A regular macro can return the unexpanded form to indicate that it's declining to expand. If a lexical macrolet declines to expand, the opportunity goes to a more lexically outer macrolet, and so on up to the global defmacro. If the global defmacro declines to expand, the form is considered expanded, and thus is necessarily either a function call or special form.
If we have both a function and macro called X, we can call the function definition using (call (fun X) ...) or (call 'X ...), or else using the Lisp-1-style dwim evaluator (dwim X ...) that is almost always used through its [] syntactic sugar as [X ...].
For a sort of completeness, the functions mboundp, mmakunbound and symbol-macro are provided, which are macro analogs of fboundp, fmakunbound and symbol-function.
The special operators or, and, if and some others have function definitions also, so that code like [mapcar or '(nil 2 t) '(1 0 3)] -> (1 2 t) is possible.
Example: apply constant folding to sqrt:
1> (sqrt 4.0)
2.0
2> (defmacro sqrt (x :env e :form f)
(if (constantp x e)
(sqrt x)
f))
** warning: (expr-2:1) defmacro: defining sqrt, which is also a built-in defun
sqrt
3> (sqrt 4.0)
2.0
4> (macroexpand '(sqrt 4.0))
2.0
5> (macroexpand '(sqrt x))
(sqrt x)
However, no, (set (second x) 42) is not implemented via a macro definition for second. That would not work very well. The main reason is that it would be too much of a burden. The programmer may want to have, for a given function, a macro definition which has nothing to do with implementing assignment semantics!
Moreover, if (second x) implements place semantics, what happens when it is not embedded in an assignment operation, such that the semantics is not required at all? Basically, to hit all the requirements would require concocting a scheme for writing macros whose complexity would equal or exceed that of existing logic for handling places.
TXR Lisp does, in fact, feature a special kind of macro called a "place macro". A form is only recognized as a place macro invocation when it is used as a place. However, place macros do not implement place semantics themselves; they just do a straightforward rewrite. Place macros must expand down to a form that is recognized as a place.
Example: specify that (foo x), when used as a place, behaves as (car x):
1> (define-place-macro foo (x) ^(car ,x))
foo
2> (macroexpand '(foo a)) ;; not a macro!
(foo a)
3> (macroexpand '(set (foo a) 42)) ;; just a place macro
(sys:rplaca a 42)
If foo expanded to something which is not a place, things would fail:
4> (define-place-macro foo (x) ^(bar ,x))
foo
5> (macroexpand '(foo a))
(foo a)
6> (macroexpand '(set (foo a) 42))
** (bar a) is not an assignable place