In clojure, macros give a tremendous power to the programmer. eval also is very powerful. There exist some subtle differences between the two. I hope that this riddle will shine some light on this topic.
(ns hello)
(defmacro my-eval [x] `~(read-string x))
(defn hello[] "Hello")
(defn run-stuff []
(println (hello))
(println (my-eval "(hello)"))
(println (eval (read-string "(hello)"))))
(ns main)
(try (hello/run-stuff)
(catch Exception e (println e)))
Over the 3 statements inside run-stuff body, which one causes an exception and why the other ones don't?
I have formulated the following riddle following the investigation of this beautiful question Clojure - (read-string String calling function. Thanks to #Matthias Benkard for the clarifications
(println (hello)) and (println (my-eval "(hello)")) are completely identical statements -- the only difference is that one will confuse your editor more. my-eval is NOT comparable to the real eval. The difference is that the argument to my-eval needs to be a string at compile time -- the following errors out because the symbol x can't be cast to a string.
(def x "(hello)")
(my-eval x)
This makes my-eval utterly pointless - you can "eval" a literal string or you can remove the quotes and the my-eval and have equally functional code (that your editor will understand).
Real eval, on the other hand, tries to compile code at run time. Here, it fails because it is being run from the main namespace, not the hello namespace, as #Matthias Benkard pointed out.
Related
I recently discovered that all of my implementations of Scheme throw an error when I try to use (cadaddr (list 1 3 (list 5 7) 9)). Apparently, by default Scheme does not allow any car and cdr combinations in the single-function form that use more than four abbreviated car and cdrcalls. I originally blamed this on Scheme's minimalism, but then I discovered that Common Lisp also shares this defect.
Can this be solved with a macro? Can we write a macro that allows an arbitrary amount of a and d in its c[...]r calls and returns the expected result, while also having the Common Lisp-like compatibility with macros like setf? If not, why not? And if so, has a reason ever been given for this is not a default feature in any lisp that I've seen?
Such a macro is described in Let Over Lambda for common lisp. You must wrap your code with (with-cxrs ...) to bring them all into scope, but it walks your code to see which combinators you need. I wrote a Clojure port of it years ago for fun, though of course nobody (including me) has ever wanted to use it for real. You could port it to Scheme if you liked.
(defn cxr-impl [name]
(when-let [op (second (re-matches #"c([ad]+)r" name))]
`(comp ~#(map {\a `first \d `rest} op))))
(defmacro with-cxrs [& body]
(let [symbols (remove coll? (tree-seq coll? seq body))]
`(let [~#(for [sym symbols
:let [impl (cxr-impl (name sym))]
:when impl
thing [sym impl]]
thing)]
~#body)))
user> (macroexpand-1 '(with-cxrs (inc (caadaaddadr x))))
(let [caadaaddadr (comp first first rest first first rest rest first rest)]
(inc (caadaaddadr x)))
https://groups.google.com/g/clojure/c/CanBrJPJ4aI/m/S7wMNqmj_Q0J
As noted in the mailing list thread, there are some bugs you'd have to work out if you wanted to use this for real.
AND and OR are macros and since macros aren't first class in scheme/racket they cannot be passed as arguments to other functions. A partial solution is to use and-map or or-map. Is it possible to write a function that would take arbitrary macro and turn it into a function so that it can be passed as an argument to another function? Are there any languages that have first class macros?
In general, no. Consider that let is (or could be) implemented as a macro on top of lambda:
(let ((x 1))
(foo x))
could be a macro that expands to
((lambda (x) (foo x)) 1)
Now, what would it look like to convert let to a function? Clearly it is nonsense. What would its inputs be? Its return value?
Many macros will be like this. In fact, any macro that could be routinely turned into a function without losing any functionality is a bad macro! Such a macro should have been a function to begin with.
I agree with #amalloy. If something is written as a macro, it probably does something that functions can't do (e.g., introduce bindings, change evaluation order). So automatically converting arbitrary macro into a function is a really bad idea even if it is possible.
Is it possible to write a function that would take arbitrary macro and turn it into a function so that it can be passed as an argument to another function?
No, but it is somewhat doable to write a macro that would take some macro and turn it into a function.
#lang racket
(require (for-syntax racket/list))
(define-syntax (->proc stx)
(syntax-case stx ()
[(_ mac #:arity arity)
(with-syntax ([(args ...) (generate-temporaries (range (syntax-e #'arity)))])
#'(λ (args ...) (mac args ...)))]))
((->proc and #:arity 2) 42 12)
(apply (->proc and #:arity 2) '(#f 12))
((->proc and #:arity 2) #f (error 'not-short-circuit))
You might also be interested in identifier macro, which allows us to use an identifier as a macro in some context and function in another context. This could be used to create a first class and/or which short-circuits when it's used as a macro, but could be passed as a function value in non-transformer position.
On the topic of first class macro, take a look at https://en.wikipedia.org/wiki/Fexpr. It's known to be a bad idea.
Not in the way you probably expect
To see why, here is a way of thinking about macros: A macro is a function which takes a bit of source code and turns it into another bit of source code: the expansion of the macro. In other words a macro is a function whose domain and range are source code.
Once the source code is fully expanded, then it's fed to either an evaluator or a compiler. Let's assume it's fed to a compiler because it makes the question easier to answer: a compiler itself is simply a function whose domain is source code and whose range is some sequence of instructions for a machine (which may or may not be a real machine) to execute. Those instructions might include things like 'call this function on these arguments'.
So, what you are asking is: can the 'this function' in 'call this function on these arguments' be some kind of macro? Well, yes, it could be, but whatever source code it is going to transform certainly can not be the source code of the program you are executing, because that is gone: all that's left is the sequence of instructions that was the return value of the compiler.
So you might say: OK, let's say we disallow compilers: can we do it now? Well, leaving aside that 'disallowing compilers' is kind of a serious limitation, this was, in fact, something that very old dialects of Lisp sort-of did, using a construct called a FEXPR, as mentioned in another answer. It's important to realise that FEXPRs existed because people had not yet invented macros. Pretty soon, people did invent macros, and although FEXPRs and macros coexisted for a while – mostly because people had written code which used FEXPRs which they wanted to keep running, and because writing macros was a serious pain before things like backquote existed – FEXPRs died out. And they died out because they were semantically horrible: even by the standards of 1960s Lisps they were semantically horrible.
Here's one small example of why FEXPRs are so horrible: Let's say I write this function in a language with FEXPRs:
(define (foo f g x)
(apply f (g x)))
Now: what happens when I call foo? In particular, what happens if f might be a FEXPR?. Well, the answer is that I can't compile foo at all: I have to wait until run-time and make some on-the-fly decision about what to do.
Of course this isn't what these old Lisps with FEXPRs probably did: they would just silently have assumed that f was a normal function (which they would have called an EXPR) and compiled accordingly (and yes, even very old Lisps had compilers). If you passed something which was a FEXPR you just lost: either the thing detected that, or more likely it fall over horribly or gave you some junk answer.
And this kind of horribleness is why macros were invented: macros provide a semantically sane approach to processing Lisp code which allows (eventually, this took a long time to actually happen) minor details like compilation being possible at all, code having reasonable semantics and compiled code having the same semantics as interpreted code. These are features people like in their languages, it turns out.
Incidentally, in both Racket and Common Lisp, macros are explicitly functions. In Racket they are functions which operate on special 'syntax' objects because that's how you get hygiene, but in Common Lisp, which is much less hygienic, they're just functions which operate on CL source code, where the source code is simply made up of lists, symbols &c.
Here's an example of this in Racket:
> (define foo (syntax-rules ()
[(_ x) x]))
> foo
#<procedure:foo>
OK, foo is now just an ordinary function. But it's a function whose domain & range are Racket source code: it expects a syntax object as an argument and returns another one:
> (foo 1)
; ?: bad syntax
; in: 1
; [,bt for context]
This is because 1 is not a syntax object.
> (foo #'(x 1))
#<syntax:readline-input:5:10 1>
> (syntax-e (foo #'(x 1)))
1
And in CL this is even easier to see: Here's a macro definition:
(defmacro foo (form) form)
And now I can get hold of the macro's function and call it on some CL source code:
> (macro-function 'foo)
#<Function foo 4060000B6C>
> (funcall (macro-function 'foo) '(x 1) nil)
1
In both Racket and CL, macros are, in fact, first-class (or, in the case of Racket: almost first-class, I think): they are functions which operate on source code, which itself is first-class: you can write Racket and CL programs which construct and manipulate source code in arbitrary ways: that's what macros are in these languages.
In the case of Racket I have said 'almost first-class', because I can't see a way, in Racket, to retrieve the function which sits behind a macro defined with define-syntax &c.
I've created something like this in Scheme, it's macro that return lambda that use eval to execute the macro:
(define-macro (macron m)
(let ((x (gensym)))
`(lambda (,x)
(eval `(,',m ,#,x)))))
Example usage:
;; normal eval
(define x (map (lambda (x)
(eval `(lambda ,#x)))
'(((x) (display x)) ((y) (+ y y)))))
;; using macron macro
(define x (map (macron lambda)
'(((x) (display x)) ((y) (+ y y)))))
and x in both cases is list of two functions.
another example:
(define-macro (+++ . args)
`(+ ,#args))
((macron +++) '(1 2 3))
While reading through Paul Graham's Essays, I've become more and more curious about Lisp.
In this article, he mentions that one of the most powerful features is that you can write programs that write other programs.
I couldn't find an intuitive explanation on his site or elsewhere. Is there some minimal Lisp program that shows an example of how this is done? Or, can you explain in words what this means exactly?
Lisp is homoiconic. Here is a function which build an s-expression representing a sum.
(defun makes(x) (list '+ x 2))
so (makes 5) evaluates to (+ 5 2) which is a valid s-expression. You could pass that to eval
There are more complex examples with Lisp macros. See also this. Read the section on Evaluation and Compilation of Common Lisp HyperSpec (also notice its compile, defmacro, eval forms). Be aware of multi-staged programming.
I strongly recommend reading SICP (it is freely downloadable) then Lisp In Small Pieces. You could also enjoy reading Gödel, Escher, Bach.... and J.Pitrat's blog on Bootstrapping Artificial Intelligence.
BTW, with C on POSIX, you might also code programs generating C code (or use GCCJIT or LLVM), compiling that generated code as a plugin, and dlopen-ing it.
While homoiconicity is the fundamental property that makes this easy, a good example of this in practice is the macro facility present in many lisps. Homoiconicity allows you to write lisp functions that take lisp source (represented as lists of lists) and do list manipulation operations on it to produce other lisp source. A macro is a plain lisp function for doing this which is installed into the compiler/evaluator of your lisp as an extension of the language's syntax. The macro gets called like a normal function, but instead of waiting until runtime the compiler passes the raw code of the macro's arguments to it. The macro is then responsible for returning some alternative code for the compiler to process in its place.
A simple example is the built-in when macro, used like so (assuming some variable x):
(when (evenp x)
(print "It's even!")
(* 5 x))
when is similar to the more fundamental if, but where if takes 3 sub-expressions (test, then-case, else-case) when takes the test and then an arbitrary number of expressions to run in the "then" case (it returns nil in the else case). To write this using if you need an explicit block (a progn in Common Lisp):
(if (evenp x)
(progn
(print "It's even!")
(* 5 x))
nil)
Translating the when version to the if version is some very simple list-manipluation:
(defun when->if (when-expression)
(list 'if
(second when-expression)
(append (list 'progn)
(rest (rest when-expression)))))
Although I'd probably use the list templating syntax and some shorter functions to get this:
(defun when->if (when-expression)
`(if ,(second when-expression) (progn ,#(cddr when-expression)) nil))
This gets called like so: (when->if (list 'when (list 'evenp 'x) ...)).
Now all we need to do is inform the compiler that when it sees an expression like (when ...) (actually I'm writing one for (my-when ...) to avoid clashing with the built-in version) it should use something like our when->if to turn it into code it understands. The actual macro syntax for this actually lets you take apart the expression/list ("destructure" it) as part of the arguments of the macro, so it ends up looking like this:
(defmacro my-when (test &body then-case-expressions)
`(if ,test (progn ,#then-case-expressions) nil))
Looks sorta like a regular function, except it's taking code and outputting other code. Now we can write (my-when (evenp x) ...) and everything works.
The lisp macro facility forms a major component of the expressive power of lisps- they allow you to mold the language to better suit your project and abstract away nearly any boilerplate. Macros can be as simple as when or complex enough to make a third-party OOP library feel like a first-class part of the language (in fact many lisps still implement OOP as a pure lisp library as opposed to a special component of the core compiler, not that you can tell from using them).
A good example are Lisp macros. They aren't evaluated, but instead they transform to the expressions within them. That is what makes them essentially programs that write program. They transform the expressions within them between compile-time and runtime. This means that you can essentially create your own syntax since a macro isn't actually evaluated. A good example would be this invalid common lisp form:
(backwards ("Hello world" nil format))
Clearly the syntax for the format function is backwards. BUT... we are passing it to a macro which isn't evaluated, so we will not get a backtrace error, because the macro isn't actually evaluated. Here is what our macro looks like:
(defmacro backwards (expr)
(reverse expr))
As you can see, we reverse the expression within the macro, which is why it becomes a standard Lisp form between compile-time and runtime. We have essentially altered the syntax of Lisp with a simple example. The call to the macro isn't evaluated, but is translated. A more complex example would be creating a web page in html:
(defmacro standard-page ((&key title href)&body body)
`(with-html-output-to-string (*standard-output* nil :prologue t :indent t)
(:html :lang "en"
(:head
(:meta :charset "utf-8")
(:title ,title)
(:link :rel "stylesheet"
:type "text/css"
:href ,href))
,#body)))
We can essentially create a macro, and the call to that macro will not be evaluated, but it will expand to valid lisp syntax, and that will be evaluated. If we look at the macro expansion we can see that the expansion is what is evaluated:
(pprint (macroexpand-1 '(standard-page (:title "Hello"
:href "my-styles.css")
(:h1 "Hello world"))))
Which expands to:
(WITH-HTML-OUTPUT-TO-STRING (*STANDARD-OUTPUT* NIL :PROLOGUE T :INDENT T)
(:HTML :LANG "en"
(:HEAD (:META :CHARSET "utf-8") (:TITLE "Hello")
(:LINK :REL "stylesheet" :TYPE "text/css" :HREF "my-styles.css"))
(:H1 "Hello world")))
This is why Paul Graham mentions that you can essentially write programs that write programs, and ViaWeb was essentially one big macro. A bunch of macros like this writing code that could write code that could write code...
I have sample code like this:
#!/usr/bin/guile -s
!#
(define (process body)
(list 'list (map (lambda (lst)
(list 'quote (car lst)))
body)))
(defmacro macro (body)
(list 'quote (process body)))
(display (macro ((foo bar) (bar baz))))
(newline)
it run but I've got error from compiler
ERROR: Unbound variable: process
functions inside macros should be allowed, why I got this error?
Functions inside macros are allowed in Guile and in most other Scheme dialects.
However, the crucial question is: Which functions are available for a macro to call out during the expansion process?
Think of it this way: When the compiler is processing your code, it is first focused on turning your source code into something that can be run at some point in the future. But the compiler might not necessarily be able to execute those same functions right now while it is compiling them, at the same time that your macro is running and expanding the source code in general.
Why wouldn't such a function be available? Well, one example would be: What if the function's body used the macro you are in the midst of defining? Then you would have a little chicken/egg problem. The function would need to run the macro to be compiled (since the macro use in the body needs to be expanded, at compile-time) ... But the macro would need the compiled function available in order to even run!
(Furthermore, there might be some functions that you only want to be available at compile-time, as a helper for your macros, but that you do not want to be available at run-time, so that it will not be included in your program executable when you deploy it, as that would waste space in the deployed binary.)
One of my favorite papers describing this problem, and the particular solution adopted by MzScheme (now known as Racket), is the "You Want It When" paper by Matthew Flatt.
So, this is a problem that any Scheme dialect with a procedural macro system has to deal with in some way, and Guile is no exception.
In Guile's case, one fix that is directly documented in the Guile manual is to use the eval-when special form, which allows you to specify at what phases a particular definition is meant to be available.
(The "You Want It When" paper referenced above describes some problems with eval-when, but since it is what the Guile manual documents, I'm going to stick with it for now. I do recommend that after you understand eval-when, that you then look into Racket's solution, and see if Guile offers anything similar.)
So in your case, since you want the process function to be available at compile-time (for use in the macro definition), you could write:
#!/usr/bin/guile -s
!#
(eval-when (expand)
(define (process body)
(list 'list (map (lambda (lst)
(list 'quote (car lst)))
body))))
(defmacro macro (body)
(list 'quote (process body)))
(display (macro ((foo bar) (bar baz))))
(newline)
I'm trying to move from Common Lisp to Chicken Scheme, and having plenty of problems.
My current problem is this: How can I write a macro (presumably using define-syntax?) that calls other macros?
For example, in Common Lisp I could do something like this:
(defmacro append-to (var value)
`(setf ,var (append ,var ,value)))
(defmacro something-else ()
(let ((values (list))
(append-to values '(1)))))
Whereas in Scheme, the equivalent code doesn't work:
(define-syntax append-to
(syntax-rules ()
((_ var value)
(set! var (append var value)))))
(define-syntax something-else
(syntax-rules ()
((_)
(let ((values (list)))
(append-to values '(1))))))
The append-to macro cannot be called from the something-else macro. I get an error saying the append-to "variable" is undefined.
According to all the information I've managed to glean from Google and other sources, macros are evaluated in a closed environment without access to other code. Essentially, nothing else exists - except built-in Scheme functions and macros - when the macro is evaluated. I have tried using er-macro-transformer, syntax-case (which is now deprecated in Chicken anyway) and even the procedural-macros module.
Surely the entire purpose of macros is that they are built upon other macros, to avoid repeating code. If macros must be written in isolation, they're pretty much useless, to my mind.
I have investigated other Scheme implementations, and had no more luck. Seems it simply cannot be done.
Can someone help me with this, please?
It looks like you're confusing expansion-time with run-time. The syntax-rules example you give will expand to the let+set, which means the append will happen at runtime.
syntax-rules simply rewrites input to given output, expanding macros until there's nothing more to expand. If you want to actually perform some computation at expansion time, the only way to do that is with a procedural macro (this is also what happens in your defmacro CL example).
In Scheme, evaluation levels are strictly separated (this makes separate compilation possible), so a procedure can use macros, but the macros themselves can't use the procedures (or macros) defined in the same piece of code. You can load procedures and macros from a module for use in procedural macros by using use-for-syntax. There's limited support for defining things to run at syntax expansion time by wrapping them in begin-for-syntax.
See for example this SO question or this discussion on the ikarus-users mailing list. Matthew Flatt's paper composable and compilable macros explains the theory behind this in more detail.
The "phase separation" thinking is relatively new in the Scheme world (note that the Flatt paper is from 2002), so you'll find quite a few people in the Scheme community who are still a bit confused about it. The reason it's "new" (even though Scheme has had macros for a long long time) is that procedural macros have only become part of the standard since R6RS (and reverted in R7RS because syntax-case is rather controversial), so the need to rigidly specify them hasn't been an issue until now. For more "traditional" Lispy implementations of Scheme, where compile-time and run-time are all mashed together, this was never an issue; you can just run code whenever.
To get back to your example, it works fine if you separate the phases correctly:
(begin-for-syntax
(define-syntax append-to
(ir-macro-transformer
(lambda (e i c)
(let ((var (cadr e))
(val (caddr e)))
`(set! ,var (append ,var ,val)))))) )
(define-syntax something-else
(ir-macro-transformer
(lambda (e i c)
(let ((vals (list 'print)))
(append-to vals '(1))
vals))))
(something-else) ; Expands to (print 1)
If you put the definition of append-to in a module of its own, and you use-for-syntax it, that should work as well. This will also allow you to use the same module both in the macros you define in a body of code as well as in the procedures, by simply requiring it both in a use and a use-for-syntax expression.