I want to keep a list of required modules of a particular module (let's say the current-module).
I feel like there are some other options (such as parsing the module?) that could be tried, but I started playing with the idea of shadowing (require) and adding the required items to a hash-table with the module-name. The problem is I cannot figure how to write a syntax definition for it.
Although not working, a function definition equivalent would be like below:
(define require-list (make-hash))
(define require
(lambda vals
; add vals to hash-table with key (current-namespace)
(let ([cn (current-namespace)])
(hash-set! require-list cn
(append vals (hash-ref require-list cn))))
(require vals)))
.. it seems the last line call should be modified to call the original (require) as well?
A correct version or a pointer to how to do it, or any other way of achieving the original goal highly appreciated.
If you just want to get a list of imports for a particular module, there is a convenient built-in called module->imports that will do what you want. It will return a mapping between phase levels and module imports—phase levels higher than 0 indicate imports used at compile-time for use in macro expansion.
> (require racket/async-channel)
> (module->imports 'racket/async-channel)
'((0
#<module-path-index:(racket/base)>
#<module-path-index:(racket/contract/base)>
#<module-path-index:(racket/contract/combinator)>
#<module-path-index:(racket/generic)>))
Note that the module in question must be included into the current namespace in order for module->imports to work, which require or dynamic-require will both do.
This will inspect the information known by the compiler, so it will find all static imports for a particular module. However, the caveats about dynamic requires mentioned by John Clements still apply: those can be dynamically performed at runtime and therefore will not be detected by module->imports.
Short short version:
Have you tried turning on the module browser?
Short version:
You're going to need a macro for this, and
It's not going to be a complete solution
The existing require is not a function; it's a language form, implemented as a macro. This is because the compiler needs to collect the same information you do, and therefore the required modules must be known at compile time.
The right way to do this--as you suggest--is definitely to leverage the existing parsing. If you expand the module and then walk the resulting tree, you should be able to find
everything you need. The tree will be extremely large, but will contain (many instances of) a relatively small number of primitives, so writing this traversal shouldn't be too hard. There will however be a lot of fiddling involved in setting up namespace anchors etc. in order to get the expansion to happen in the first place.
Regarding your original idea: you can definitely create a macro that shadows require. You're going to want to define it in another file and rename it on the way out so that your macro can refer to the original require. Also, the require form has a bunch of interesting subforms, and coming up with a macro that tries to handle all of these subforms will be tricky. If you're looking at writing a macro, though, you're already thinking about an 80% solution, so maybe this won't bother you.
Finally: there are forms that perform dynamic module evaluation, and so you can't ever know for sure all of the modules that might be required, though you could potentially annotate these forms (or maybe shadow the dynamic module-loading function) to see these as they happen.
Also, it's worth mentioning that you may get more precise answers on the Racket mailing list.
Related
I am implementing an interpreter that codegen to another language using Racket. As a novice I'm trying to avoid macros to the extent that I can ;) Hence I came up with the following "interpreter":
(define op (open-output-bytes))
(define (interpret arg)
(define r
(syntax-case arg (if)
[(if a b) #'(fprintf op "if (~a) {~a}" a b)]))
; other cases here
(eval r))
This looks a bit clumsy to me. Is there a "best practice" for doing this? Am I doing a totally crazy thing here?
Short answer: yes, this is a reasonable thing to do. The way in which you do it is going to depend a lot on the specifics of your situation, though.
You're absolutely right to observe that generating programs as strings is an error-prone and fragile way to do it. Avoiding this, though, requires being able to express the target language at a higher level, in order to circumvent that language's parser.
Again, it really has a lot to do with the language that you're targeting, and how good a job you want to do. I've hacked together things like this for generating Python myself, in a situation where I knew I didn't have time to do things right.
EDIT: oh, you're doing Python too? Bleah! :)
You have a number of different choices. Your cleanest choice is to generate a representation of Python AST nodes, so you can either inject them directly or use existing serialization. You're going to ask me whether there are libraries for this, and ... I fergits. I do believe that the current Python architecture includes ... okay, yes, I went and looked, and you're in good shape. Python's "Parser" module generates ASTs, and it looks like the AST module can be constructed directly.
https://docs.python.org/3/library/ast.html#module-ast
I'm guessing your cleanest path would be to generate JSON that represents these AST modules, then write a Python stub that translates these to Python ASTs.
All of this assumes that you want to take the high road; there's a broad spectrum of in-between approaches involving simple generalizations of python syntax (e.g.: oh, it looks like this kind of statement has a colon followed by an indented block of code, etc.).
If your source language shares syntax with Racket, then use read-syntax to produce a syntax-object representing the input program. Then use recursive descent using syntax-case or syntax-parse to discern between the various constructs.
Instead of printing directly to an output port, I recommend building a tree of elements (strings, numbers, symbols etc). The last step is then to print all the elements of the tree. Representing the output using a tree is very flexible and allows you to handle sub expressions out of order. It also allows you to efficiently concatenate output from different sources.
Macros are not needed.
The Racket docs indicate that Racket has separate forms for: require, load, include, and import. Many other languages only contain one of these and are generally used synonymously (although obviously with language specific differences such as #include in C, and import in Java).
Since Racket has all four of these, what is the difference between each one and which should I be using in general? Also if each has a specific use, when should I use an alternative type? Also, this question seems to indicate that require (paired with provide) is preferred, why?
1. Require
You are correct, the default one you want is almost always require (paired with a provide). These two forms go hand in hand with Racket's modules and allows you to more easily determine what variables should be scoped in which files. For example, the following file defines three variables, but only exports 2.
#lang racket ; a.rkt
(provide b c)
(define a 1)
(define b 2)
(define c 3)
As per the Racket style guide, the provide should ideally be the first form in your file after the #lang so that you can easily tell what a module provides. There are a few cases where this isn't possible, but you probably won't encounter those until you start making your own Racket libraries you intend for public distribution. Personally, I still put a file's require before its provide, but I do sometimes get flack for it.
In a repl, or another module, you can now require this file and see the variables it provides.
Welcome to Racket v6.12.
> (require "a.rkt")
> c
3
> b
2
> a
; a: undefined;
; cannot reference undefined identifier
; [,bt for context]
There are ways to get around this, but this serves as a way for a module to communicate what its explicit exports are.
2. Load
This is a more dynamic variant of require. In general you should not use it, and instead use dynamic-require when you need to load a module dynamically. In this case, load is effectively a primitive that require uses behind the scenes. If you are explicitly looking to emulate the top level however (which, to be clear, you almost never do), then load is a fine option. Although in those rare cases, I would still direct you to the racket/load language. Which interacts exactly as if each form was entered into the repl directly.
#lang racket/load
(define x 5)
(displayln x) ; => prints 5
(define x 6)
(displayln x) ; => prints 6
3. Include
Include is similar to #include in C. There are even fewer cases where you should use it. The include form grabs the s-expression syntax of the given path, and puts it directly in the file where the include form was. At first, this can appear as a nice solution to allow you to split up a single module into multiple files, or to have a module 'piece' you want to put in multiple files. There are however better ways to do both of those things without using include, that also don't come with the confusing side effects you get with include.1 One thing to remember if you still insist on using import, is that the file you are importing probably shouldn't have a #lang line unless you are explicitly wanting to embed a submodule. (In which case you will also need to have a require form in addition to the include).
4. Import
Finally, import is not actually a core part of Racket, but is part of its unit system. Units are similar in some ways to modules, but allow for circular dependencies (unit A can depend on Unit B while Unit B depends on Unit A). They have fallen out of favor in recent years due to the syntactic overhead they have.
Also unlike the other forms import (and additionally export), take signatures, and relies on an external linker to decide which actual units should get linked together. Units are themselves a complex topic and deserve their own question on how to create and link them.
Conclusion (tldr)
TLDR; Use require and provide. They are the best supported and easiest to understand. The other forms do have their uses, but should be thought of 'advanced uses' only.
1These side effects are the same as you would expect for #include in C. Such as order being important, and also with expressions getting intermixed in very unpredictable ways.
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it. I have asked this on the Emacs forum with no success. See:
https://emacs.stackexchange.com/questions/35913/program-rewriting-systems-unexpanded-a-defmacro-given-a-list-of-macros-to-undo
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
Ideally, in certain situations such a tool would be able to work directly off the defmacro to do its pattern find and replace on. However even if I have to come up with specific search and replace patterns manually to add to the transformation system, having such a framework to work in would still be useful
Summary of results so far: Although there have been a few answers that explore interesting possibilities, right now there is nothing definitive. So I think best to leave this open. I'll summarize some of the suggestions. (I've upvoted all the answers that were in fact answers instead of commentary on the difficulty.)
First, many people suggest considered the special form of macros that do expansion only,or as Drew puts it:
macro-expansion (i.e., not expansion followed by Lisp evaluation).
Macro-expansion is another way of saying reduction semantics, or
rewriting.
The current front-runner to my mind is in phils post where he uses a pattern-matching facility that seems specific to Emacs: pcase. I will be exploring this and will post results of my findings. If anyone else has thoughts on this please chime in.
Drew wrote a program called FTOC whose purpose was to convert Franz Lisp to Common Lisp; googling turns up a comp.lang.lisp posting
I found a Common Lisp package called optima with fare-quasiquote. Paulo thinks however this might not be powerful enough since it doesn't handle backtracking out of the box, but might be programmed in by hand. Although the generality of backtracking might be nice, I'm not convinced I need that for the most-used situations.)
Side note: Some seem put off by the specific application causing my initial interest. (But note that in research, it is not uncommon for good solutions to get applied in ways not initially envisioned.)
So in that spirit, here are a couple of suggestions for changing the end application. A good solution for these would probably translate to a solution for Emacs Lisp. (And if if helps you to pretend I'm not interested in Emacs Lisp, that's okay with me). Instead of a decompiler for Emacs Lisp, suppose I want to write a decompiler for clojure or some Common Lisp system. Or as suggested by Sylwester's answer, suppose I would like to automatically refactor my code by taking into account the benefit of using more concise macros that exist or that have gotten improved. Recall that at one time Emacs Lisp didn't have "when" or "unless" macros.
30-some years ago I did something similar, using macrolet.
(Actually, I used defmacro because we had only an early implementation of Common Lisp, which did not yet have macrolet. But macrolet is the right thing to use.)
I didn't translate macro-expanded code to what it was expanded from, but the idea is pretty much the same. You will come across some different difficulties, I expect, since your translation is even farther away from one-to-one.
I wrote a translator from (what was then) Franz Lisp to Common Lisp, to help with porting lots of existing code to a Lisp+Prolog-machine project. Franz Lisp back then was only dynamically scoped, while Common Lisp is (in general) lexically scoped.
And yes, obviously there is no general way to automatically translate Lisp code (in particular), especially considering that it can generate and then evaluate other code - but even ignoring that special case. Many functions are quite similar, but there is the lexical/dynamic difference, as well as significant differences in the semantics of some seemingly similar functions.
All of that has to be understood and taken for granted from the outset, by anyone wanting to make use of the results of translation.
Still, much that is useful can be done. And if the resulting code is self-documenting, telling you what it was derived from etc., then when in the resulting context you can decide just what to do with this or that bit that might be tricky (e.g., rewrite it manually, from scratch or just tweak it). In practice, lots of code was easily converted from Franz to Common - it saved much reprogramming effort.
The translator program was written in Common Lisp. It could be used interactively as well as in batch. When used interactively it provided, in effect, a Franz Lisp interpreter on top of Common Lisp.
The program used only macro-expansion (i.e., not expansion followed by Lisp evaluation). Macro-expansion is another way of saying reduction semantics, or rewriting.
Input Franz-Lisp code was macro-expanded via function-definition mapping macros to produce Common-Lisp code. Code that was problematic for translation was flagged (in code) with a description/analysis that described the situation.
The program was called FTOC. I think you can still find it, or at least references to it, by googling (ftoc lisp). (It was the first Lisp program I wrote, and I still have fond memories of the experience. It was a good way to learn both Lisp dialects and to learn Lisp in general.)
Have fun!
In general, I don't think you can do this. The expansion of an lisp macro is Turing complete, so you have to be able to predict the output of a program which could have arbitrary input.
There are some simple things that you could do. defmacros with backquoted forms in appear fairly similar in the output form and might be detected. This sort of heuristic would probably get you a long way.
What I don't understand is your use case. The macro-expanded version of a piece of code is usually only present in the compiled (or in emacs-lisp byte-compiled) form.
Ok so other people have pointed out the fact that this problem is impossible in general. There are two hard parts to this problem: one is that it could be a lot of work to find a preimage of some code fragment through a macro and it is also impossible to determine whether a macro was called or not—there are examples where one may write code which could have come from a macro without using that macro. Imagine for the sake of illustration an sha macro which expands to the SHA hash of the string literal passed to it. Then if you see some sha hash in your expanded code, it would obviously be silly to try to unexpand it. But it may be that the hash was put into the code as a literal, e.g. referencing a specific point in the history of a git repository so it would also be unhelpful to unexpand the macro.
Tractable subproblems
Let me preface this by saying that whilst these may be a little tractable, I still wouldn’t try to solve this problem.
Let’s ignore all the macros that do weird things (like the example above) and all the macros that are just as likely to not have been used in the original (e.g. cond vs if) and all the macros which generate complex code which seems like it would be difficult to unravel (e.g. loop, do, and backquote. Annoyingly these difficult cases are some of those which you would perhaps most want to unexpand). The type this leaves us with (that I’d like to focus on) are macros which basically just reduce boilerplate, e.g. save-excursion or with-XXXX. These are macros whose implementation consists of possibly making some fresh symbols (via gensym) and then having a big simple backquoted block of code. I still think it would be too hard to automatically go from defmacro to a function for unexpansion but I think you could attack some of these on a case-by-case basis. Do this by looking for the forms generated by the macro that delimit (I.e. begin/end) the expanded code. I can’t really offer much beyond that. This is still a hard problem and I don’t think any existing solutions (to other problems) will get you very far on your way.
A further complication I understand is that you do not start at the macroexpanded code but rather at the bytecode. Without knowing anything about the elisp compiler, I worry that more information would be lost in the compilation step and you would have to undo that as well, e.g. perhaps it is hard to determine which code goes inside a let or even when a let begins, or bytecode starts using goto type features even though elisp doesn’t have them.
You suggest that the reason you would like to unexpand macros is so you can decompile bytecode which sometimes comes up in the Emacs debugger and that this would be useful as even though the source code is available in theory, it isn’t always at your fingertips. I put it to you that if you want to make your life debugging elisp easier it would be more worthwhile to figure out how to have the Emacs debugger always take you to the source code for internal functions. This might involve installing extra debugging related packages or downloading the Emacs source code and setting some variable so Emacs knows where to find it or compiling Emacs yourself from source. I don’t really know about that but I bet getting thrown into bytecode instead of source would have been enough of a problem for Emacs developers over the past thirty years that a solution to that problem does exist.
If however what you really want to do is to try to implement a decompiler for elisp then I suppose that’s what you should do. A final observation is that while Lisp provides facilities which make manipulating Lisp code easy, this doesn’t help much with decompiling as all these facilities can be used in compilation so there are infinitely more patterns one might want to detect than in e.g. a C decompiler. Perhaps scheme style macros would be easier to unexpand, although they would still be hard.
If you’re decompiling because you want to give a better idea of which exact subexpression rather than line is being evaluated (normally Lisp debuggers work on expressions not lines anyway) in the debugger then perhaps it would actually be useful to see the code at the expanded level rather than the unexpanded one. Or perhaps it would be best to see both and maybe in between as well. Keeping track of what’s what through forwards macroexpansion is already difficult and fiddly. Doing it in reverse certainly won’t be easier. Good luck!
Edit: seeing as your not currently using Lisp anyway, I wonder if you might have more success using something like prolog for your unexpanding. You’d still have to manually write rules but I think it would be a large amount of work to try to derive rules from macro definitions.
I would like to take Emacs Lisp code that has been macro expanded and unmacro expand it.
Macros generate arbitrary expressions, which may contain macros recursively. You have no general way to revert the transformations, because it's not pattern-based.
Even if macros were pattern-based, they could still be infinite.
Even if macros were not infinite, they can certainly contain bugs in expansions of patterns that never matched. Given arbitrary code to try to unwind, it could match an expansion that looks like the code and try to revert to its pattern. Without bugs, you could still abuse this.
Even if you could revert macro expansion, some macros expand to the same code. An approach could be signalling a warning with a restart when all reversions expand equally minus the operator, such that if the restart doesn't handle the signal, it would choose the first expansion; and otherwise signalling an error with a restart, such that if the restart doesn't handle the signal, it errors. Or you could configure it to choose certain macros under certain conditions, such as in which package the code was found.
In practice, there are very few cases where reverting an expansion makes any sense. It could be a useful development tool that suggests macros, but I wouldn't generally rely on it for whole source transformations.
One way you could achieve what you want is through a controlled pattern matching. You could initially create patterns manually, which would already handle cases you care about directly, such as the ones you mention:
(if (not <cond>) <expr>) and (if (not <cond>) (progn <&expr>)) to (unless <cond> <&expr>)
You'd have to decide whether null would be equivalent to not. I personally don't mix the boolean meaning of nil with that of empty list or something else, e.g. no result, nothing found, null object, a designator, etc. But perhaps Lisp code as old as that in Emacs just uses them interchangeably.
(if <cond> <expr>) and (if <cond> (progn <&expr>)) to (when <cond> <&expr>)
If you feel like improving code overall, include cond with a single condition. And be careful with cond clauses with only the condition.
You should have a few dozen more, to see how the pattern matching behaves with more patterns to match in terms of time (CPU) and space (memory).
From the description of fare-quasiquote, optima doesn't support backtracking, which you probably want.
But you can do backtracking with optima by yourself, using recursion on complex inner patterns, and if nothing matches, return a control value to keep searching for matching patterns from the outer input.
Another approach is to treat a pattern as a description of a state machine, and handle each new token to advance the current state machines until one of them reaches the end, discarding the state machines that couldn't advance. This approach may consume more memory, depending on the amount of patterns, the similarity between patterns (if many have the same starting token, many state machines will be generated on a matching token), the length of the patterns and, last but not least, the length of the input (s-expression).
An advantage of this approach is that you can use it interactively to see which patterns have matched the most tokens, and you can give weights to patterns instead of just taking the first that matches.
A disadvantage is that, most probably, you'll have to spend effort to develop it.
EDIT: I just lousily described a kind of trie or radix tree.
Once you got something working, maybe try to obtain patterns automatically. This is really hard, you must probably limit it to simple backquoting and accept the fact you can't generalize for anything that contains more complex code.
I believe the hardest will be code walking, which is hard enough with source code, but much more with macro-expanded code. Perhaps if you could expand the whole picture a bit further to understand the goal, maybe someone could suggest a better approach other than operating on macro-expanded code.
However one would think that this kind of thing, S-expression transformation, is right up Lisp's alley. And defmacro is I believe available in Lisp as it is in Emacs Lisp.
So surely there are program transformation systems, or term-rewriting systems that can be adapted here.
There's a huge step from expanding code with defmacro and all that generality. Most Lisp developers will know about hygienic macros, at least in terms of symbols as variables.
But there's still hygienic macros in terms of symbols as operators1, code walking, interaction with a containing macro (usually using macrolet), etc. It's way too complex.
1.
Common Lisp evaluates the operator in a compound form in the lexical environment, and probably everyone makes macros that assume that the global macro or function definition of a symbol will be used.
But it might not be so:
(defmacro my-macro-1 ()
`1)
(defmacro my-macro-2 ()
`(my-function (my-macro-1)))
(defun my-function (n)
(* n 100))
(macrolet ((my-macro-1 ()
`2))
(flet ((my-function (n)
(* n 1000)))
(my-macro-2)))
That last line will expand to (my-function (my-macro-2)), which will be recursively expanded to (my-function 2). When evaluated, it will yield 2000.
For proper operator hygiene, you'd have to do something like this:
(defmacro my-macro-2 ()
;; capture global bindings of my-macro-1 and my-function-1 by name
(flet ((my-macro-1-global (form env)
(funcall (macro-function 'my-macro-1) form env))
(my-function-global (&rest args)
;; hope the compiler can optimize this
(apply 'my-function args)))
;; store them globally in uninterned symbols
;; hopefully, no one will mess with them
(let ((my-macro-1-symbol (gensym (symbol-name 'my-macro-1)))
(my-function-symbol (gensym (symbol-name 'my-function))))
(setf (macro-function my-macro-1-symbol) #'my-macro-1-global)
(setf (symbol-function my-function-symbol) #'my-function-global)
`(,my-function-symbol (,my-macro-1-symbol)))))
With this definition, the example will yield 100.
Common Lisp has some restrictions to avoid this, but it only states the consequences are undefined when (re)defining symbols in the common-lisp package, globally or locally. It doesn't require errors or warnings to be signaled.
I don't think it is possible to do this in general, but you can undo a pattern back into a macro use for every match if you supply code for each unmacroing. Code that mixed cond and if will end up being just if and your code would remove all if into cond making the reverse not the same as the starting point. The more macros you have and the more they expand into each other the more uncertain of the end result will be of the starting point.
You could have rules such that if is not translated into cond unless you used one of the features, like more than one predicate or implicit progn, but you have no idea if the coder actually did use cond everywhere because he liked in consistent regardless. Thus your unmacroing will acyually be more of a simplification.
I don't believe there's a general solution to that, and you certainly
can't guarantee that the structure of the output would match that of
the original code, and I'm not going near the idea of auto-generating
patterns and desired transformations from macro definitions; but you
might achieve a simple version of this with Emacs' own pcase pattern
matching facility.
Here's the simplest example I could think of:
With reference to the definition of when:
(defmacro when (cond &rest body)
(list 'if cond (cons 'progn body)))
We can transform code using a pcase pattern like so:
(let ((form '(if (and foo bar baz) (progn do (all the) things))))
(pcase form
(`(if ,cond (progn . ,body))
`(when ,cond ,#body))
(_ form)))
=> (when (and foo bar baz) do (all the) things)
Obviously if the macro definitions change, then your patterns will
cease to work (but that's a pretty safe kind of failure).
Caveat: This is the first time I've written a pcase form, and I
don't know what I don't know. It seems to work as intended, though.
I have seen one answer of How does Lisp let you redefine the language itself?
Stack Overflow question (answered by Noah Lavine):
Macros aren't quite a complete redefinition of the language, at least as far as I know (I'm actually a Schemer; I could be wrong), because there is a restriction. A macro can only take a single subtree of your code, and generate a single subtree to replace it. Therefore you can't write whole-program-transforming macros, as cool as that would be.
After reading this I am curious about whether there are "whole-program-transforming macros" in Lisp or Scheme (or some other language).
If not then why?
It is not useful and never required?
Same thing could be achieved by some other ways?
It is not possible to implement it even in Lisp?
It is possible, but not tried or implemented ever?
Update
One kind of use case
e.g.
As in stumpwm code
here are some functions all in different lisp source files
uses a dynamic/global defvar variable *screen-list* that is defined in primitives.lisp , but used in screen.lisp, user.lisp, window.lisp.
(Here each files have functions, class, vars related to one aspect or object)
Now I wanted to define these functions under the closure where
*screen-list* variable available by let form, it should not be
dynamic/global variable, But without moving these all functions into
one place (because I do not want these functions to lose place from their
related file)
So that this variable will be accessible to only these functions.
Above e.g. equally apply to label and flet, so that it will further possible
that we could make it like that only required variable, function will be available,
to those who require it.
Note one way might be
implement and use some macro defun_with_context for defun where first argument is
context where let, flet variables definend.
But apart from it could it be achieved by reader-macro as
Vatine and Gareth Rees answered.
You quoted Noah Lavine as saying:
A macro can only take a single subtree of your code, and generate a single subtree to replace it
This is the case for ordinary macros, but reader macros get access to the input stream and can do whatever they like with it.
See the Hyperspec section 2.2 and the set-macro-character function.
In Racket, you can implement whole-program-transforming macros. See the section in the documentation about defining new languages. There are many examples of this in Racket, for example the lazy language and Typed Racket.
Off the top of my head, a few approaches:
First, you can. Norvig points out that:
We can write a compiler as a set of macros.
so you can transform an entire program, if you want to. I've only seen it done rarely, because typically the intersection between "things you want to do to every part of your program" and "things that you need macro/AST-type transformations for" is a pretty small set. One example is Parenscript, which transforms your Lisp code ("an extended subset of CL") into Javascript. I've used it to compile entire files of Lisp code into Javascript which is served directly to web clients. It's not my favorite environment, but it does what it advertises.
Another related feature is "advice", which Yegge describes as:
Great systems also have advice. There's no universally accepted name for this feature. Sometimes it's called hooks, or filters, or aspect-oriented programming. As far as I know, Lisp had it first, and it's called advice in Lisp. Advice is a mini-framework that provides before, around, and after hooks by which you can programmatically modify the behavior of some action or function call in the system.
Another is special variables. Typically macros (and other constructs) apply to lexical scope. By declaring a variable to be special, you're telling it to apply to dynamic scope (I think of it as "temporal scope"). I can't think of any other language that lets you (the programmer) choose between these two. And, apart from the compiler case, these two really span the space that I'm interested in as a programmer.
A typical approach is to write your own module system. If you just want access to all the code, you can have some sort of pre-processor or reader extension wrap source files with your own module annotation. If you then write your own require or import form, you will ultimately be able to see all the code in scope.
To get started, you could write your own module form that lets you define several functions which you then compile in some clever way before emitting optimized code.
There's always the choice of using compiler macros (they can do whole-function transformation based on a lew of criteria, but shouldn't change the value returned, as that would be confusing).
There's reader macros, they transform the input "as it is read" (or "before it is read", if you prefer). I haven't done much large-scale reader-macro hacking, but I have written some code to allow elisp sourec to be (mostly) read in Common Lisp, with quite a few subtle differences in syntactic sugar between the two.
I believe those sorts of macros are called code-walking macros. I haven't implemented a code walker myself, so I am not familiar with the limits.
In Common LISP, at least, you may wrap top-level forms in PROGN and they still retain their status as top-level forms (see CLTL2, section 5.3). Therefore, the limitation of a macro generating a single subtree is not much of a limitation since it could wrap any number of resulting subtrees within PROGN. This makes whole-program macros quite possible.
E.g.
(my-whole-program-macro ...)
= expands to =>
(progn
(load-system ...)
(defvar ...)
(defconstant ...)
(defmacro ...)
(defclass ...)
(defstruct ...)
(defun ...)
(defun ...)
...
)
I can read the documentation, so I'm not asking for a cut-and-paste of that.
I'm trying to understand the motivation for this function.
When would I want to use it?
The documentation in the Emacs lisp manual does have some example situations that seem to answer your question (as opposed to the doc string).
From looking at the Emacs source code, eval-and-compile is used to quiet the compiler, to make macros/functions available during compilation (and evaluation), or to make feature/version specific variants of macros/functions available during compilation.
One usage I found helpful to see was in ezimage.el. In there, an if statement was put inside the eval-and-compile to conditionally define macros depending on whether the package was compiled/eval'ed in Emacs or XEmacs, and additionally whether a particular feature was present. By wrapping that conditional inside the eval-and-compile you enable the appropriate macro usage during compilation. A similar situation can be found in mwheel.el.
Similarly, if you want to define a function via fset and have it available during compilation, you need to have the call to fset wrapped with eval-and-compile because otherwise the symbol -> function association isn't available until the file is evaluated (because compilation of a call to fset just optimizes the assignment, it doesn't actually do the assignment). Why would you want this assignment during compilation? To quiet the compiler. Note: this is just my re-wording of what is in the elisp documentation.
I did notice a lot of uses in Emacs code which just wrapped calls to require, which sounds redundant when you read the documentation. I'm at a loss as to how to explain those.