I would like to create a new language that has the same syntax as typed racket, but when executed will do two things:
run the given program as typed racket
if type checks, then translates the input into another language (say python). I am planning to write a translator from typed racket forms to python syntax.
Any suggestions on how to start or pointers to some skeleton code? I have read this tutorial but it mostly talks about creating new syntax which I don't need in my case.
I understand how I can write racket code to translate from one language to another, what I don't get is how I can do both of the above, i.e., first run it as another language, and then do something else with the same input.
It sounds like you want to make a language with a new #%module-begin form. This form is either inserted by the reader (when you do a #lang .... line at the top of your file, or by the language if you wrote the module out by hand. Either way, its generally associated with your language definition. This macro has full access to the entire module's unexpanded syntax. So for example, a macro like this:
(provide (rename-out [-module-begin #%module-begin]))
(define-simple-macro (-module-begin body ...)
(#%module-begin
(writeln '(body ...))
body ...))
Would create a language that does two things:
Print out the body of the code (as an s-expression), and
Run the body with the #%module-begin that the language definition was written in.
You can see how you can use this technique to grab the body of your program twice, and do two different things with it. So let's try running this example. First, lets take the sample from above, and put it in a file "mylang.rkt":
#lang racket
(provide (rename-out [-module-begin #%module-begin])
(except-out (all-from-out racket) #%module-begin))
(require syntax/parse/define)
(define-simple-macro (-module-begin body ...)
(#%module-begin
(writeln '(body ...))
body ...))
And now we can write a program in mylang like this:
#lang s-exp "mylang.rkt"
(+ 1 2)
And when you run it, you'll get something like this:
((+ 1 2))
3
First its printing out the program text, then running it.
You can read more about this process in a paper I wrote discussing this aspect of the Video language. You might also find the book Beautiful Racket to have some examples you might find useful.
Related
I am working through the exercises in Matthew Flatt's Creating Languages in Racket. I am currently at step 3, where #lang s-exp "txtadv.rkt top line for a language "client" (world.rkt) is used. In this case, I am assuming that the reader is the s-exp racket reader and that the expander is "txtadv.rkt". Per the text:
"The exports of txtadv.rkt completely determine the bindings that are available in world.rkt—not
only the functions, but also syntactic forms such as require or lambda. For example, txtadv.rkt could
supply a lambda binding to world.rkt that implements a different kind of function than the usual
lambda, such as functions with lazy evaluation."
I see then that txtadv.rkt has this:
(except-out (all-from-out racket) #%module-begin)
I understand by this: re-export (provide) all the imports I got from racket, except for #%module-begin.
What I want: I would like to figure out a proccess to further trim-down this provide list as I do not want to export all of racket.
Things I have tried so far:
I changed at the top of txtadv.rkt to require racket/base and then put this:
(except-out (all-from-out racket/base) #%module-begin)
That worked, but it's still too coarse and fails at the next step with no clue other than revert to require racket.
I started trying to figure out (guess) what s-exps were in "world.rkt" the client, that would require extra syntax:
I commented out the (except-out (all-from-out racket)..)
started "guessing" exports:
that were missing, i.e. define-one-verb
racket functions/forms that are used in macros like symbol->string, list
For all I have tried, the expander txtadv.rkt passes Ctrl-R / syntax-chek, but in the client world.rkt I keep getting:
txtadv.rkt:84:4: define-one-verb: bad syntax in: (define-one-verb north (= n) "go north")
In Lisp, a function's arguments are evaluated first before entering the function body. Macro arguments stay not evaluated.
But sometimes, one wants to inject code pieces stored in variables into a macro. This means evaluating the argument for the macro first, and then apply the macro-of-choice on this evaluated result.
One has to resort to
(eval `(macro ,arg))
To achieve this - but eval does not behave correctly in different environments.
The best thing would be, if one could do:
(apply macro (list arg))
or
(funcall macro arg)
But since the macro is not a function this doesn't work.
Is it possible to achieve something like this? - To circumvent that problem oder to make the macro available in the functions namespace?
Or am I missing some other ways to solve such problems?
I came to this question while trying to answer How to produce HTML from a list. but also in Generate TYPECASE with macro in common lisp, Evaluate arguments passed to a macro that generates functions in lisp, and How to convert a list to code/lambda in scheme?. But I always thought while answering them it would be good to have an apply or funcall-like function which can take macros.
It is not clear what you are trying to do, although it is almost certain that you are confused about something. In particular if you are calling eval inside macroexpansions then in almost all cases you are doing something both seriously wrong and seriously dangerous. I can't ever think of a case where I've wanted macros which expand to things including eval and I have written Lisp for a very very long time.
That being said, here is how you call the function associated with a macro, and why it is very seldom what you want to do.
Macros are simply functions whose domain and range is source code: they are compilers from a language to another language. It is perfectly possible to call the function associated with a macro, but what that function will return is source code, and what you will then need to do with that source code is evaluate it. If you want a function which deals with run-time data which is not source code, then you need that function, and you can't turn a macro into that function by some magic trick which seems to be what you want to do: that magic trick does not, and can not, exist.
So for instance if I have a macro
(defmacro with-x (&body forms)
`(let ((x 1))
,#forms))
Then I can call its macro function on a bit of source code:
> (funcall (macro-function 'with-x)
'(with-x (print "foo")) nil)
(let ((x 1)) (print "foo"))
But the result of this is another bit of source code: I need to compile or evaluate it, and nothing I can do will get around this.
Indeed in (almost?) all cases this is just the same as macroexpand-1):
> (macroexpand-1 '(with-x (print "foo")))
(let ((x 1)) (print "foo"))
t
And you can probably write macroexpand-1 in terms of macro-function:
(defun macroexpand-1/equivalent (form &optional (env nil))
(if (and (consp form)
(symbolp (first form))
(macro-function (first form)))
(values (funcall (macro-function (first form)) form env)
t)
(values form nil)))
So, if the result of calling a macro is source code, what do you do with that source code to get a result which is not source code? Well, you must evaluate it. And then, well, since the evaluator expands macros for you anyway, you might as well just write something like
(defun evaluate-with-x (code)
(funcall (compile nil `(lambda ()
(with-x ,#code)))))
So you didn't need to call the macro's function in any case. And this is not the magic trick which turns macros into functions dealing with data which is not source code: it is a terrible horror which is entirely made of exploding parts.
A concrete example: CL-WHO
It looks like this question might have its origins in this one and the underlying problem there is that that's not what CL-WHO does. In particular it is a confusion to think that something like CL-WHO is a tool for taking some kind of list and turning it into HTML. It's not: it's a tool for taking the source code of a language which is built on CL but includes a way of expressing HTML output mingled with CL code, and compiles it into CL code which will do the same thing. It happens to be the case that CL source code is expressed as lists & symbols, but CL-WHO isn't really about that: it's a compiler from, if you like, 'the CL-WHO language' to CL.
So, let's try the trick we tried above and see why it's a disaster:
(defun form->html/insane (form)
(funcall
(compile nil `(lambda ()
(with-html-output-to-string (,(make-symbol "O"))
,#form)))))
And you might, if you did not look at this too closely, think that this function does in fact do the magic trick:
> (form->html/insane '(:p ((:a :href "foo") "the foo")))
"<p></p><a href='foo'>the foo</a>"
But it doesn't. What happens if we call form->html/insane on this perfectly innocuous list:
(:p (uiop/run-program:run-program "rm -rf $HOME" :output t))
Hint: don't call form->html/insane on this list if you don't have very good backups.
CL-WHO is an implementation of a programming language which is a strict superset of CL: if you try to turn it into a function to turn lists into HTML you end up with something involving the same nuclear weapon you tinker with every time you call eval, except that nuclear weapon is hidden inside a locked cupboard where you can't see it. But it doesn't care about that: if you set it off it will still reduce everything within a few miles to radioactive ash and rubble.
So if you want a tool which will turn lists – lists which aren't source code – into HTML then write that tool. CL-WHO might have the guts of such a tool in its implemenentation, but you can't use it as it is.
And this is the same problem you face whenever you are trying to abuse macros this way: the result of calling a macro's function is Lisp source code, and to evaluate that source code you need eval or an equivalent of eval. And eval is not only not a terrible solution to almost any problem: it's also a nuclear weapon. There are, perhaps problems for which nuclear weapons are good solutions, but they are few and far between.
While reading through Paul Graham's Essays, I've become more and more curious about Lisp.
In this article, he mentions that one of the most powerful features is that you can write programs that write other programs.
I couldn't find an intuitive explanation on his site or elsewhere. Is there some minimal Lisp program that shows an example of how this is done? Or, can you explain in words what this means exactly?
Lisp is homoiconic. Here is a function which build an s-expression representing a sum.
(defun makes(x) (list '+ x 2))
so (makes 5) evaluates to (+ 5 2) which is a valid s-expression. You could pass that to eval
There are more complex examples with Lisp macros. See also this. Read the section on Evaluation and Compilation of Common Lisp HyperSpec (also notice its compile, defmacro, eval forms). Be aware of multi-staged programming.
I strongly recommend reading SICP (it is freely downloadable) then Lisp In Small Pieces. You could also enjoy reading Gödel, Escher, Bach.... and J.Pitrat's blog on Bootstrapping Artificial Intelligence.
BTW, with C on POSIX, you might also code programs generating C code (or use GCCJIT or LLVM), compiling that generated code as a plugin, and dlopen-ing it.
While homoiconicity is the fundamental property that makes this easy, a good example of this in practice is the macro facility present in many lisps. Homoiconicity allows you to write lisp functions that take lisp source (represented as lists of lists) and do list manipulation operations on it to produce other lisp source. A macro is a plain lisp function for doing this which is installed into the compiler/evaluator of your lisp as an extension of the language's syntax. The macro gets called like a normal function, but instead of waiting until runtime the compiler passes the raw code of the macro's arguments to it. The macro is then responsible for returning some alternative code for the compiler to process in its place.
A simple example is the built-in when macro, used like so (assuming some variable x):
(when (evenp x)
(print "It's even!")
(* 5 x))
when is similar to the more fundamental if, but where if takes 3 sub-expressions (test, then-case, else-case) when takes the test and then an arbitrary number of expressions to run in the "then" case (it returns nil in the else case). To write this using if you need an explicit block (a progn in Common Lisp):
(if (evenp x)
(progn
(print "It's even!")
(* 5 x))
nil)
Translating the when version to the if version is some very simple list-manipluation:
(defun when->if (when-expression)
(list 'if
(second when-expression)
(append (list 'progn)
(rest (rest when-expression)))))
Although I'd probably use the list templating syntax and some shorter functions to get this:
(defun when->if (when-expression)
`(if ,(second when-expression) (progn ,#(cddr when-expression)) nil))
This gets called like so: (when->if (list 'when (list 'evenp 'x) ...)).
Now all we need to do is inform the compiler that when it sees an expression like (when ...) (actually I'm writing one for (my-when ...) to avoid clashing with the built-in version) it should use something like our when->if to turn it into code it understands. The actual macro syntax for this actually lets you take apart the expression/list ("destructure" it) as part of the arguments of the macro, so it ends up looking like this:
(defmacro my-when (test &body then-case-expressions)
`(if ,test (progn ,#then-case-expressions) nil))
Looks sorta like a regular function, except it's taking code and outputting other code. Now we can write (my-when (evenp x) ...) and everything works.
The lisp macro facility forms a major component of the expressive power of lisps- they allow you to mold the language to better suit your project and abstract away nearly any boilerplate. Macros can be as simple as when or complex enough to make a third-party OOP library feel like a first-class part of the language (in fact many lisps still implement OOP as a pure lisp library as opposed to a special component of the core compiler, not that you can tell from using them).
A good example are Lisp macros. They aren't evaluated, but instead they transform to the expressions within them. That is what makes them essentially programs that write program. They transform the expressions within them between compile-time and runtime. This means that you can essentially create your own syntax since a macro isn't actually evaluated. A good example would be this invalid common lisp form:
(backwards ("Hello world" nil format))
Clearly the syntax for the format function is backwards. BUT... we are passing it to a macro which isn't evaluated, so we will not get a backtrace error, because the macro isn't actually evaluated. Here is what our macro looks like:
(defmacro backwards (expr)
(reverse expr))
As you can see, we reverse the expression within the macro, which is why it becomes a standard Lisp form between compile-time and runtime. We have essentially altered the syntax of Lisp with a simple example. The call to the macro isn't evaluated, but is translated. A more complex example would be creating a web page in html:
(defmacro standard-page ((&key title href)&body body)
`(with-html-output-to-string (*standard-output* nil :prologue t :indent t)
(:html :lang "en"
(:head
(:meta :charset "utf-8")
(:title ,title)
(:link :rel "stylesheet"
:type "text/css"
:href ,href))
,#body)))
We can essentially create a macro, and the call to that macro will not be evaluated, but it will expand to valid lisp syntax, and that will be evaluated. If we look at the macro expansion we can see that the expansion is what is evaluated:
(pprint (macroexpand-1 '(standard-page (:title "Hello"
:href "my-styles.css")
(:h1 "Hello world"))))
Which expands to:
(WITH-HTML-OUTPUT-TO-STRING (*STANDARD-OUTPUT* NIL :PROLOGUE T :INDENT T)
(:HTML :LANG "en"
(:HEAD (:META :CHARSET "utf-8") (:TITLE "Hello")
(:LINK :REL "stylesheet" :TYPE "text/css" :HREF "my-styles.css"))
(:H1 "Hello world")))
This is why Paul Graham mentions that you can essentially write programs that write programs, and ViaWeb was essentially one big macro. A bunch of macros like this writing code that could write code that could write code...
Lately I'm learning Racket, and I'm having some difficulties with understanding the "define-type" syntax. I've tried the following code:
#lang racket
(define-type Num Number)
but it outputs the following error message:
define-type: unbound identifier in module in: define-type
May anyone help me dealing this error? I have read all possible documentation and it seems like it should be working.
There is a reason every Racket program starts with a line beginning with #lang: Racket is not just a programing language, but an ecosystem of programming languages. Every file (and more specifically, every module) can be in its own programming language, and they can all talk to each other with ease.
By default, Dr. Racket creates new files with a #lang racket line at the top. This is the “Racket language”, but it is not the only language provided by the “Racket system”, which actually includes dozens of languages, some not too different from #lang racket, others almost entirely unrelated.
When you want to use Typed Racket, you need to opt in to using that language instead of ordinary #lang racket, which is dynamically typed. You can do this by writing #lang typed/racket at the top of your program.
#lang typed/racket
(define-type Num Number)
Now all the features of Typed Racket will be available to you within that module.
I'm trying to create a new language in Racket where statements are on separate lines. A newline defines the end of a statement and the start of a new one.
I read through the Create Languages chapter of the guide which was very useful but the examples were focused on extending s-exp-like languages. The only option I see is manually writing my own parser for read and read-syntax.
I was hoping to use readtables but I don't know if I can. I tried:
(make-readtable #f #f 'non-terminating-macro my-read-line-fn)
but I don't know if this is much help. I guess I could create a sub-readtable that does things like read-word, read-string which I dispatch to based on what character my-read-line-fn gets.
Is that the best strategy or is there a predefined way of reading until the end of line?
I don't think you need to do anything with the readtable. Your lang/reader.rkt can provide your own read-syntax that can read/parse however it wants, and presumably stop when it encounters EOL.
One interesting example is Brainfudge. Its concept of a "statement" is a single character, but IIUC also [ brackets ].
See its lang/reader.rkt and parser.rkt for the low-level bits, and then try to understand how that is ultimately evaluated as Racket expressions.
You do indeed need to write version of read and read-syntax that parse your language. The readtable is only meant to modify the builtin read, so I suggest that you take a look at Parser Tools (http://docs.racket-lang.org/parser-tools/index.html), which is tool for writing parsers in the lex/yacc style.
An alternative is to use ragg:
http://www.hashcollision.org/ragg/
Install Ragg using the package manager in DrRacket. Search for ragg in the list of available packages.
Make your own reader.rkt:
#lang s-exp syntax/module-reader
(test test)
#:read-syntax my-read-syntax
#:read my-read
;; override default read (won't be used but is required)
(define (my-read in) (read-line in))
;; override read-syntax by reading in one string at a time and
;; pass it to statement-string->code to get code as dara and
;; make it syntax with datum->syntax
(define (my-read-syntax in)
(datum->syntax #f (statement-string->code (read-line in))))
;; This is actually how you want your code
;; to become s-expressions. I imagine that my
;; module has a primitive called code that
;; interprets the string as is
(define (statement-string->code str)
(list 'code str))
Racket doesn't have "statements", so the concept of newlines ending "statements" is nonsensical.
If your motivation is to reduce or do away with parentheses, I encourage you to use a "standard alternative" reader like sweet-expressions, rather than making something home-grown.