Supposed I have a class:
(class object%
(init val)
(define/public (get-val) ... )
(define/public (get-norm) ... ))
Why if I have two instances foo and bar i can do so:
(eval '(send foo get-val))
but can't do so:
(for/list ([who '(foo bar)])
(for/list ([what '(get-val get-norm)])
(eval (cons 'send (list who what)))))
In the latter case I get the error:
send: unbound identifier;
also, no #%app syntax transformer is bound in: send
What is the good way to get the result I want?
UPDATE: I have discovered that the nested loop works fine in the interactive mode, but produces the error if written in the definition area. Could anyone please explain what's the trick?
You can use this:
(for*/list ([who (list foo bar)]
[what '(get-val get-norm)])
(dynamic-send who what))
The reason for that is that '(foo bar) is a list containing two symbols, foo and bar, and not a list containing the values of the variables foo and bar; you have to use (list foo bar) for that.
Also, in order to be able to send to a method name that's in a variable, as opposed to naming the method directly, you have to use dynamic-send instead.
Notice that my code does not use eval, and neither should you. Most of the time, you never need to use eval, and it's best avoided whenever possible.
The problem is the use of eval. Using eval in the REPL works just fine because the REPL provides a namespace to eval with all the exports of racket. The REPL essentially works by first running the module then creating a new namespace with all the resulting definitions and calling eval on anything you enter using that namespace, so send is defined when you call eval. However, in the definitions window, you need to provide a namespace to eval yourself for this to work properly because the initial current namespace is empty. So what you could do is add this to the beginning of your definitions:
(define ns (make-base-namespace))
Now whenever you use eval, pass in ns as the namespace argument as in (eval '(+ 1 2) ns). There's a number of caveats however.
This is a seperate namespace. Anything you've defined in your module won't be visible in anything evaluated with this namespace. If you (define foo 3), (eval '(+ 1 foo) ns) won't work - foo isn't defined in the ns namespace, it's defined in your current module namespace.
This namespace only imports racket/base by default. You'll have to do some eval-shenanigans or some namespace-shenanigans to require racket/class.
You could pass eval the value of the (current-namespace) parameter, but this comes with it's own special blend of caveats and complexity.
Communicating between namespaces is not fun. That's usually what modules are for.
The central problem is that eval is very, very, very rarely an appropriate tool for a job, let alone the best tool. There are a lot of corner cases and gotchas that can creep up on you without you having any idea what happened, not to mention the performance problems that can fester. Avoid it like the plague unless you really know what you're doing and are confident there's no better alternative.
May I ask why exactly foo and bar need to be processed using eval in your problem?
EDIT
More information on eval can be found in the Racket documentation, which is extremely thorough and a useful resource.
Related
Say I have macros foo and bar. If I write (foo (bar)) my understanding is that in most (all?) lisps foo is going to be given '(bar), not whatever bar would have expanded to had it been expanded first. Some lisps have something like local-expand where the implementation of foo can explicitly request the expansion of its argument before continuing, but why isn't that the default? It seems more natural to me. Is this an accident of history or is there a strong reason to do it the way most lisps do it?
I've been noticing in Rust that I want the macros to work this way. I'd like to be able to wrap a bunch of declarations inside a macro call so that the macro can then crawl the declarations and generate reflection information. But if I use a macro to generate the definitions that I want crawled, my macro wrapping the declarations sees the macro invocations that would generate the declarations rather than the actual declarations.
If I write (foo (bar)) my understanding is that in most (all?) lisps foo is going to be given '(bar), not whatever bar would have expanded to had it been expanded first.
That would restrict Lisp such that (bar) would need to be something that can be expanded -> probably something which is written in the Lisp language.
Lisp developers would like to see macros where the inner stuff can be a completely new language with different syntax rules. For example something, where FOO does not expand it's subforms, but transpiles/compiles a fully/partially different language to Lisp. Something which does not have the usual prefix expression syntax:
Examples
(postfix (a b +) sin)
-> (sin (+ a b))
Here the + in the macro form is not the infix +.
or
(query-all (person name)
where (person = "foo") in database DB)
Lisp macros don't work on language parse trees, but arbitrary, possibly nested, s-expressions. Those don't need to be valid Lisp code outside of that macro -> don't need to follow the usual syntax / semantics.
Common Lisp has the function MACROEXPAND and MACROEXPAND-1, such that the outer macro can expand inner code if it wants to do it during its own macro expansion:
CL-USER 26 > (defmacro bar (a) `(* ,a ,a))
BAR
CL-USER 27 > (bar 10)
100
CL-USER 28 > (defmacro foo (a &environment e)
(let ((f (macroexpand a e)))
(print (list a '-> f))
`(+ ,(second f) ,(third f))))
FOO
CL-USER 29 > (foo (bar 10))
((bar 10) -> (* 10 10))
20
In above macro, if FOO would see only an expanded form, it could not print both the source and the expansion.
This works also with scoped macros. Here the BAR macro gets locally redefined and the MACROEXPAND generates different code inside FOO for the same form:
CL-USER 30 > (macrolet ((bar (a)
`(expt ,a ,a)))
(foo (bar 10)))
((bar 10) -> (EXPT 10 10))
20
If foo is a macro then (foo (bar)) must pass the raw syntax (bar) to the foo macro expander. This is absolutely essential.
This is because foo can give any meaning whatsoever to bar.
Consider the defmacro macro itself:
(defmacro foo (bar) body)
Here, the argument (bar) is a parameter list ("macro lambda list") and not a form (Common Lisp jargon for to-be-evaluated expression). It says that the macro shall have a single parameter called bar. Therefore it is nonsensically wrong to try to expand (bar) before handing it to defmacro's expander.
Only if we know that an expression is going to be evaluated is it legitimate to expand it as a macro. But we don't know that about an expression which is the argument to a macro.
Other counterexamples are easy to come up with. (defstruct point (x 0) (y 0)): (x 0) isn't a call to operator x, but a slot x whose default value is 0. (dolist (x list) ...): x is a variable to be stepped over list.
That said, there are implementation choices regarding the timing of macro expansion.
A Lisp implementation can macro-expand an entire top-level form before evaluating or compiling any of it. Or it can expand incrementally, so that for instance when (+ x y) is being processed, x is macro-expanded and evaluated or compiled into some intermediate form already before y is even looked at.
A pure syntax tree interpreter for Lisp which always keeps the code in the original form and always expands (and re-expands) the code as it is evaluating has certain interactivity advantages. Any macro that you rewrite goes instantly "live" in all the existing code that you have input into the REPL, like existing function definitions. It is obviously quite inefficient in terms of execution speed, but any code that you call uses the latest definition of your macros without any hassle of telling the system to reload that code to have it expanded again. That also eliminates the risk that you're testing something that is still based on the old, buggy version of some macro that you fixed. If you're ever writing a Lisp, the range of timing choices for expansion is good to keep in mind, so that you consciously reject the choices you don't go with.
In turn, that said, there are some constraints on the timing of macro expansion. Conceivably, a Lisp interpreter or compiler, when processing an entire file, could go through all the top level forms and expand all of them at once before processing any of them. The implementor will quickly learn that this is bad, because some of the later forms depend on the side effects of the earlier forms. Such as, oh, macros being defined! If the first form defines a macro, which the second one uses, then we cannot expand the second form without evaluating the effect of the first.
It makes sense, in a Lisp, to split up physical top-level forms into logical ones. Suppose that someone writes (or uses a macro to generate) codde like (progn (defmacro foo ...) (foo)). This entire progn cannot be macro expanded up-front before evaluation; it won't work! There has to be a rule such as "whenever a top-level form is based on the progn operator, then the children of the progn operator are considered top-level forms by all the processing which treats top-level forms specially, and this rule is recursively applied." The top-level entry point into the macro-expanding code walker then has to contain special case hacks to do this recognition of logical top-level forms, breaking them up and recursing into a lower level expander which doesn't do those checks any more.
I've been noticing in Rust that I want the macros to work this way.
I'd like to be able to wrap a bunch of declarations inside a macro
call so that the macro can then crawl the declarations and generate
reflection information.
It does sound like local-expand is the right tool for that job.
However, an alternative approach would be something like this:
Suppose that wrapper is our outer macro, and that the intended
syntax is:
(wrapper decl1 decl2 ...)
where decl is a declaration that potenteally uses some standard form declare.
We can let
(wrapper decl1 decl2 ...)
expand to
(let-syntax ([declare our-declare])
decl1 decl2 ...
(post-process-reflection-information))
where our-declare is a helper macro that expands both to the standard declaration as well as some form that stores the reflection information,
also post-process-reflection-information is another macro that
does any needed post processing.
I think you are trying to use macros for something they are not designed to solve. Macros are primarily a text/code substitution mechanism, and in the case of Lisp this looks a lot like a simplified term-rewriting system (see also How does term-rewriting based evaluation work?). There are different strategies possible for how to substitute a pattern of code, and in which order, but in C/C++ preprocessor macros, in LaTeX, and in Lisp, the process is typically done by computing the expansion until the form is no longer expandable, starting from the topmost terms. This order is quite natural and because it is distinct from normal evaluation rules, it can be used to implement things the normal evaluation rules cannot.
In your case, you are interested in getting access to all the declarations of some object/type, something which falls under the introspection/reflection category (as you said yourself). But implementing reflection/introspection with macros doesn't look totally doable, since macros work on abstract syntax trees and this might be a poor way to access the metadata you want.
Typically the compiler is going to parse/analyze the struct definitions and build the definitive, canonical representation of the struct, even if there are different way to express that syntactically; it may even use prior information not available directly as source code to compute more interesting metadata (e.g. if you had inheritance, there could be a set of properties inherited from a type defined in another module (I don't think this applies to Rust)).
I think currently Rust does not offer compile-time or runtime introspection facilities, which explains why are you going with the macro route. In Common Lisp macros are definitely not used for introspection, the actual values obtained after evaluation (at different times) is used to gain information about an object. For example, defclass expands as a set of instructions that register a class in the language, but in order to get all the slots of a class, you ask the language to give it to you, e.g:
(defclass foo () (x)) ;; define class foo with slot X
(defclass bar () (y)) ;; define class bar with slot Y
(defclass zot (foo bar) ()) ;; define class zot with foo and bar as superclasses
USER> (c2mop:class-slots (find-class 'zot))
(#<SB-MOP:STANDARD-EFFECTIVE-SLOT-DEFINITION X>
#<SB-MOP:STANDARD-EFFECTIVE-SLOT-DEFINITION Y>)
I don't know what the solution for your problem is, but in addition to the other answers, I think it is not specifically a fault of the macro system. If a macro is defined as done usually as only a term rewriting system, it will always have difficulties to perform some tasks on the semantic level. But Rust is still evolving so there might be better ways to do things in the future.
Lets say i have fn begin: (begin "hello"). The result of `(begin "hello") will be (my-ns/begin "hello"). It's good. But now I do this: (def x '(begin "hello")). How can I expand x with backquote to get (my-ns/begin "hello"), not just (begin "hello")?
In the first example you used ` which is a reader macro named "syntax-quote", In the second example you used ' Which is a reader macro named "quote". Syntax-quote offers a couple of features beyond what quote offers:
unquoting with ~, ~#, etc
namespace expansion of symbols for use in writing hygenic macros.
Plain old quote does neither of these. So if you want namespace expansion in both examples use ` in both places. If you want to get an un-namespaced symbol in a syntax-quooted list you can use both of them together `(println ~'symbol-name) which will evaluate to simply (println symbol-name). (in this case symbol-name needs to be defined in the namespace where this is called (this process in known as "symbol capture")). The syntax-quote evaluates first to a call to quote, when in turn evaluates to the symbol.
If you are looking to expand a symbol that already exists for some reason, or you want to expand it in several different namespaces that both define it, you can use the ns-resolve function:
(ns-resolve *ns* 'begin)
so in your example you can map this over the list to print it with the NS qualified symbols:
user> (map #(if (symbol? %) (ns-resolve *ns* %) %) x)
(#'user/begin "hello")
Though this is not exactly the desired output because it refers to the var in that namespace rather than the symbol that resolves to this var. Things get a little dicey here because ` is a reader macro and we can't build calls to it in other macros. For instance I can think of no way to write:
(defmacro expand-x [thing]
`(syntax-quote-here ~thing)
Because as far as I know syntax-quote doesn't have a name (again because it is a reader macro) so there is nothing to use in place of syntax-quote-here in the above example. Though never fear, in a language with eval nothing is impossible, things just get arbitrarily ugly:
user> (eval (read-string (str "`" x)))
(user/begin "hello")
PS: don't actually use this last example or gremlins will inhabit your code for all time
I have been programming in common lisp for a little while now, and throughout my experience using lisp, I have yet to see any function/macro that acts anything similar to function prototypes in C or C++.
Currently I have to very careful about the ordering of my functions, otherwise, when I try to call a function from another, Lisp says the function "does not exist" because it is defined later in the file. Is there a way to get around this? Can I declare all my function prototypes at the top of the file, and the full definitions below?
Declaim and Proclaim
You can use declaim to globally declare that a certain thing has a certain function type. For instance, look at what happens first if you define foo1 that calls undefined baz (in SBCL):
CL-USER> (defun foo1 ()
(baz))
; in: DEFUN FOO1
; (BAZ)
;
; caught STYLE-WARNING:
; undefined function: BAZ
;
; compilation unit finished
; Undefined function:
; BAZ
; caught 1 STYLE-WARNING condition
FOO1
Now, let's add a declamation that says that baz is a function of no arguments, and returns something. You could obviously add more type information if you wanted to, but this will at least provide the arity and the knowledge that baz is a function.
CL-USER> (declaim (ftype (function () t) baz))
; No value
Now when you define foo2 that also calls baz, you'll get no warning:
CL-USER> (defun foo2 ()
(baz))
FOO2
Declaim is a macro, but if you need to be able to generate some of these things at runtime, you can use proclaim, which is a function. E.g.,
CL-USER> (dolist (f '(square cube))
(proclaim `(ftype (function (number) number) ,f)))
NIL
CL-USER> (defun add-square-and-cube (x y)
(+ (square x) (cube y)))
ADD-SQUARE-AND-CUBE
That said, this is not very idiomatic Common Lisp. It's much more common to put the code you need into a file and then to compile that file and load it. If that's not possible for some reason, this will work, but it's worth considering other options of loading your code if they're available.
Muffling warnings
It's also worth noting that while SBCL will take the hint from proclaim or declaim and silence the undefined function warning, the function is still actually undefined. Other implementations (e.g., CLISP) will still issue a warning about the undefined function.
I don't really recommend the following approach, because warnings are around for a reason, but you can choose to muffle warnings when you evaluate code. E.g., in CLISP, we get a warning when we compile with undefined functions:
CL-USER> (compile nil (lambda () (baz)))
WARNING: Function BAZ is not defined
#<COMPILED-FUNCTION NIL>
1
1
We can bind a handler that will muffle any warnings that occur when the form is evaluated, though:
CL-USER> (handler-bind ((warning
(lambda (x)
(muffle-warning x))))
(compile nil (lambda () (baz))))
#<COMPILED-FUNCTION NIL>
1
1
This has its ow caveats, too, since the type of warning that you might get for compiling a reference to an undefined function might vary, and what muffling the warning does may vary.
As Rainer pointed out you can ignore this issue if the forward reference is within a single compilation unit. But that won't help if the forward reference crosses compilation units. Usually that's a sign that your code is poorly layered. Low level code makes calls to higher level code? Well, people will say the low level code is providing a hook for the high level code.
That said I certainly have seen and written code that had this problem. Spaghetti code, yum! It can arise when you start breaking huge source files into smaller ones. A compilation unit is usually a single file. But take a look at with-compilation-unit. I don't recall asdf doesn't provide easy access that though.
I don't know if Joshua's solution of using providing a declaration will work across all CL implementations. My memory is that when I had to solve this problem many many years ago we had to implement something more crude, we would give the function a stand in definition, and then hack together a way to suppress the definition warnings.
No doubt cl-launch could be used to see if Joshua's solution works across the spectrum of implementations.
You can define something to be a generic function before you define any of its methods. So, if you write
(defgeneric foo (x y z))
then any function calling FOO will see a defined function (although one without any methods). The body of the function can then be added later as a method
(defmethod foo (x y z)
;; The body of the function
...)
I have been programming in common lisp for a little while now, and throughout my experience using lisp, I have yet to see any function/macro that acts anything similar to function prototypes in C or C++.
Currently I have to very careful about the ordering of my functions, otherwise, when I try to call a function from another, Lisp says the function "does not exist" because it is defined later in the file. Is there a way to get around this? Can I declare all my function prototypes at the top of the file, and the full definitions below?
Declaim and Proclaim
You can use declaim to globally declare that a certain thing has a certain function type. For instance, look at what happens first if you define foo1 that calls undefined baz (in SBCL):
CL-USER> (defun foo1 ()
(baz))
; in: DEFUN FOO1
; (BAZ)
;
; caught STYLE-WARNING:
; undefined function: BAZ
;
; compilation unit finished
; Undefined function:
; BAZ
; caught 1 STYLE-WARNING condition
FOO1
Now, let's add a declamation that says that baz is a function of no arguments, and returns something. You could obviously add more type information if you wanted to, but this will at least provide the arity and the knowledge that baz is a function.
CL-USER> (declaim (ftype (function () t) baz))
; No value
Now when you define foo2 that also calls baz, you'll get no warning:
CL-USER> (defun foo2 ()
(baz))
FOO2
Declaim is a macro, but if you need to be able to generate some of these things at runtime, you can use proclaim, which is a function. E.g.,
CL-USER> (dolist (f '(square cube))
(proclaim `(ftype (function (number) number) ,f)))
NIL
CL-USER> (defun add-square-and-cube (x y)
(+ (square x) (cube y)))
ADD-SQUARE-AND-CUBE
That said, this is not very idiomatic Common Lisp. It's much more common to put the code you need into a file and then to compile that file and load it. If that's not possible for some reason, this will work, but it's worth considering other options of loading your code if they're available.
Muffling warnings
It's also worth noting that while SBCL will take the hint from proclaim or declaim and silence the undefined function warning, the function is still actually undefined. Other implementations (e.g., CLISP) will still issue a warning about the undefined function.
I don't really recommend the following approach, because warnings are around for a reason, but you can choose to muffle warnings when you evaluate code. E.g., in CLISP, we get a warning when we compile with undefined functions:
CL-USER> (compile nil (lambda () (baz)))
WARNING: Function BAZ is not defined
#<COMPILED-FUNCTION NIL>
1
1
We can bind a handler that will muffle any warnings that occur when the form is evaluated, though:
CL-USER> (handler-bind ((warning
(lambda (x)
(muffle-warning x))))
(compile nil (lambda () (baz))))
#<COMPILED-FUNCTION NIL>
1
1
This has its ow caveats, too, since the type of warning that you might get for compiling a reference to an undefined function might vary, and what muffling the warning does may vary.
As Rainer pointed out you can ignore this issue if the forward reference is within a single compilation unit. But that won't help if the forward reference crosses compilation units. Usually that's a sign that your code is poorly layered. Low level code makes calls to higher level code? Well, people will say the low level code is providing a hook for the high level code.
That said I certainly have seen and written code that had this problem. Spaghetti code, yum! It can arise when you start breaking huge source files into smaller ones. A compilation unit is usually a single file. But take a look at with-compilation-unit. I don't recall asdf doesn't provide easy access that though.
I don't know if Joshua's solution of using providing a declaration will work across all CL implementations. My memory is that when I had to solve this problem many many years ago we had to implement something more crude, we would give the function a stand in definition, and then hack together a way to suppress the definition warnings.
No doubt cl-launch could be used to see if Joshua's solution works across the spectrum of implementations.
You can define something to be a generic function before you define any of its methods. So, if you write
(defgeneric foo (x y z))
then any function calling FOO will see a defined function (although one without any methods). The body of the function can then be added later as a method
(defmethod foo (x y z)
;; The body of the function
...)
I'm a bit confused as to exactly when symbol capture will occur with clojure macros. Suppose that I have a macro which defines a function from keywords. In this trivial example,
(defmacro foo [keywd1 keywd2] `(defn ~(symbol (name keywd1))
[~(symbol (name keywd2))] (* 2 ~(symbol (name keywd2)))))
I call (foo :bar :baz), and this gets expanded into (defn bar [baz] (* 2 baz)).
So now the question -- can this lead to symbol capture? If so, under what circumstances?
I know that it's preferred to use gensym (e.g. bar#) to prevent symbol capture, but in some cases (not many, but still) I'd like to have a pretty macro-expansion, without the auto-generated symbols.
Bonus question: does the answer change if we are considering a macro that creates macros?
In your example symbol capture does not happen, because provide the variable parts as parameters. So the developer can choose the names himself.
Symbol capture happens when your macro introduces new locals which are not specified by the user. Consider the following (really silly and nonsensical just to show the point) example:
(defmacro foo
[name & body]
`(defn ~name
[~'bar]
(println ~'bar)
~#body))
In this case bar is captured. Now suppose the user has some code like this.
(def bar 5)
(foo baz (* 2 bar))
(baz 7)
This would not give what the would expect. Because the global bar, the user refering to gets shadowed by the local bar introduced by the macro. As you already said: in this case one should use bar# to introduce the local.
So capture is always indicated by ~'. Macro writing macros don't really change that. Just add one more level: ~~'.