I've got a custom language I'm designing in Racket, let's call it waffle.
Let's say I have
(define (printstr . input)
(if (string? (car input))
(write (string-join input ""))
(write input)))
; ... a whole bunch of other definitions
(command-line
#:multi
[("-v" "--verbose") "more verbose" (set! loglevel (add1 loglevel))]
[("-q" "--quiet") "be quiet" (set! loglevel 0)]
#:once-any
[("-i" "--in-place") "edit in-place" (set! mode 'in-place)]
[("-c" "--create-new") "create a new file" (set! mode 'new)]
[("-n" "--dry-run") "do nothing" (set! mode #f)]
#:once-each
[("-d" "--directory") dir "work in a given directory" (set! root dir)]
#:help-labels "operations to perform:"
#:multi
[("+l" "++line") "add a line" (set! ops `(,#ops "add"))]
[("-l" "--line") "delete a line" (set! ops `(,#ops "delete"))]
[("-e" "--edit") "edit a line" (set! ops `(,#ops "edit"))]
#:args (file)
(define in (open-input-file file))
; This is probably where I'm going wrong with how my language REPL evaluates files passed to it.
(eval (file->list file) ns))
Then I create an executable from DrRacket using 'Racket [menu] -> Create Executable ... -> [Type] Launcher'. Name is e.g. waffle-test.
I've got a file written in my waffle language, hello.waffle:
(printstr "Hello!")
I expect this to print 'Hello!' on the command line and then exit without errors. But I get a strange error I don't understand, and I get my prompt back without a newline.
$ ./waffle-test hello.waffle
application: not a procedure;
expected a procedure that can be applied to arguments
given: #<void>
arguments...: [none]
context...:
eval-one-top12
"/home/connie/Desktop/racket-ffgardfghf/waffle": [running body]
temp37_0
for-loop
run-module-instance!125
perform-require!78
"Hello!" $
I know you're not supposed to use eval but I can't find out how to make my language executable read and run files I pass to it. What is the best approach to doing this?
Just with this simple test I figured out a couple of things:
#!racket
(file->list "test.waffle")
With test.waffle as:
(waffle me)
(waffle it)
The repl printed:
((waffle me)
(waffle it))
However that is not valid code, even if waffle is a valid procedure. You need it to look like this:
(begin
(waffle me)
(waffle it))
Now you can do this by require your language to have it, but you can also just cons a begin to the resulting structure and eval will evaluate each top level form one by one in order.
You will run into problems with using eval. You'll know that soon enough. The correct way to make an interpreter is by creating your own eval that implements your languages syntax and takes en environment with primitives. It has an interface to use the host, but not regardless. With eval waffle programs has access to all internals and you're not really creating an interpreter since you just expose Racket.
I remember someone did the same with Ruby and had a web page and someone just tried to type in a command that deleted all the files on the system and the web service went away.
By fixing a few bugs in my code I managed to solve my problem!
I changed
(eval (file->list file) ns))
to
(define program (cons 'begin (file->list file)))
(eval program ns))
I also changed the 'printstr' function from
(define (printstr . input)
(if (string? (car input))
(write (string-join input ""))
(write input)))
to
(define (printstr input)
(displayln input))
Related
I have sample code like this:
#!/usr/bin/guile -s
!#
(define (process body)
(list 'list (map (lambda (lst)
(list 'quote (car lst)))
body)))
(defmacro macro (body)
(list 'quote (process body)))
(display (macro ((foo bar) (bar baz))))
(newline)
it run but I've got error from compiler
ERROR: Unbound variable: process
functions inside macros should be allowed, why I got this error?
Functions inside macros are allowed in Guile and in most other Scheme dialects.
However, the crucial question is: Which functions are available for a macro to call out during the expansion process?
Think of it this way: When the compiler is processing your code, it is first focused on turning your source code into something that can be run at some point in the future. But the compiler might not necessarily be able to execute those same functions right now while it is compiling them, at the same time that your macro is running and expanding the source code in general.
Why wouldn't such a function be available? Well, one example would be: What if the function's body used the macro you are in the midst of defining? Then you would have a little chicken/egg problem. The function would need to run the macro to be compiled (since the macro use in the body needs to be expanded, at compile-time) ... But the macro would need the compiled function available in order to even run!
(Furthermore, there might be some functions that you only want to be available at compile-time, as a helper for your macros, but that you do not want to be available at run-time, so that it will not be included in your program executable when you deploy it, as that would waste space in the deployed binary.)
One of my favorite papers describing this problem, and the particular solution adopted by MzScheme (now known as Racket), is the "You Want It When" paper by Matthew Flatt.
So, this is a problem that any Scheme dialect with a procedural macro system has to deal with in some way, and Guile is no exception.
In Guile's case, one fix that is directly documented in the Guile manual is to use the eval-when special form, which allows you to specify at what phases a particular definition is meant to be available.
(The "You Want It When" paper referenced above describes some problems with eval-when, but since it is what the Guile manual documents, I'm going to stick with it for now. I do recommend that after you understand eval-when, that you then look into Racket's solution, and see if Guile offers anything similar.)
So in your case, since you want the process function to be available at compile-time (for use in the macro definition), you could write:
#!/usr/bin/guile -s
!#
(eval-when (expand)
(define (process body)
(list 'list (map (lambda (lst)
(list 'quote (car lst)))
body))))
(defmacro macro (body)
(list 'quote (process body)))
(display (macro ((foo bar) (bar baz))))
(newline)
After discovering a very simple way to watch YouTube videos from the command line using my new Raspberry Pi 2 (running Raspbian) using only easily obtainable packages, namely:
omxplayer -o local $(youtube-dl -g {videoURL})
I immediately wanted a way to watch entire YouTube playlists that way. So I saw this as a perfect excuse to hack together a solution in Common Lisp :)
My solution (imaginatively dubbed RpiTube) is a script that, when given the URL of a YouTube playlist, searches the page's HTML source and extracts the URLs for the videos contained within it. I can then
pass these URLs to a Bash script that ultimately calls the above command for each video individually, one after the other. The Common Lisp script itself is complete and works, however I'm having difficulty invoking it with the URL as a command-line argument. This is mainly because I'm still quite new to Quicklisp, Lisp packages and creating executables from Common Lisp code.
I'm running Clozure Common Lisp (CCL) with Quicklisp (installed as per Rainer Joswig's instructions). I've included the complete code below. It may be a little inefficient, but to my amazement it runs reasonably quickly even on the Raspberry Pi. (Suggested improvements are appreciated.)
;rpitube.lisp
;Given the URL of a YouTube playlist's overview page, return a list of the URLs of videos in said playlist.
(load "/home/pi/quicklisp/setup.lisp")
(ql:quickload :drakma)
(ql:quickload "cl-html-parse")
(ql:quickload "split-sequence")
(defun flatten (x)
"Paul Graham's utility function from On Lisp."
(labels ((rec (x acc)
(cond ((null x) acc)
((atom x) (cons x acc))
(t (rec (car x) (rec (cdr x) acc))))))
(rec x nil)))
(defun parse-page-source (url)
"Generate lisp list of a page's html source."
(cl-html-parse:parse-html (drakma:http-request url)))
(defun occurences (e l)
"Returns the number of occurences of an element in a list. Note: not fully tail recursive."
(cond
((null l) 0)
((equal e (car l)) (1+ (occurences e (cdr l))))
(t (occurences e (cdr l)))))
(defun extract-url-stubs (flatlist unique-atom url-retrieval-fn)
"In a playlist's overview page the title of each video is represented in HTML as a link,
whose href entry is part of the video's actual URL (referred to here as a stub).
Within the link's tag there is also an entry that doesn't occur anywhere else in the
page source. This is the unique-atom (a string) that we will use to locate the link's tag
within the flattened list of the page source, from which we can then extract the video's URL
stub using a simple url-retrieval-fn (see comments below this function). This function is iterative, not
recursive, because the latter approach was too confusing."
(let* ((tail (member unique-atom flatlist :test #'equal))
(n (occurences unique-atom tail))
(urls nil))
(loop for x in tail with i = 0
while (< (length urls) n) do
(if (string= x unique-atom)
(setf urls (cons (funcall url-retrieval-fn tail i) urls)))
(incf i))
(reverse urls)))
;Example HTML tag:
;<a class="pl-video-title-link yt-uix-tile-link yt-uix-sessionlink spf-link " data-sessionlink="verylongirrelevantinfo" href="/watch?v=uniquevideocode&index=numberofvideoinplaylist&list=uniqueplaylistcode" dir="ltr"></a>
;Example tag when parsed and flattened:
;(:A :CLASS "pl-video-title-link yt-uix-tile-link yt-uix-sessionlink spf-link " :DATA-SESSIONLINK "verylongirrelevantinfo" :HREF "/watch?v=uniquevideocode&index=numberofvideoinplaylist&list=uniqueplaylistcode" :DIR "ltr")
;The URL stub is the fourth list element after unique-atom ("pl-video-title..."), so the url-retreival-fn is:
;(lambda (l i) (elt l (+ i 4))), where i is the index of unique-atom.
(defun get-vid-urls (url)
"Extracts the URL stubs, turns them into full URLs, and returns them in a list."
(mapcar (lambda (s)
(concatenate 'string
"https://www.youtube.com"
(car (split-sequence:split-sequence #\& s))))
(extract-url-stubs (flatten (parse-page-source url))
"pl-video-title-link yt-uix-tile-link yt-uix-sessionlink spf-link "
(lambda (l i) (elt l (+ i 4))))))
(let ((args #+clozure *unprocessed-command-line-arguments*))
(if (and (= (length args) 1)
(stringp (car args)))
(loop for url in (get-vid-urls (car args)) do
(format t "~a " url))
(error "Usage: rpitube <URL of youtube playlist>
where URL is of the form:
'https://www.youtube.com/playlist?list=uniqueplaylistcode'")))
First I tried adding the following line to the script
#!/home/pi/ccl/armcl
and then running
$ chmod +x rpitube.lisp
$ ./rpitube.lisp {playlistURL}
which gives:
Unrecognized non-option arguments: (./rpitube.lisp {playlistURL})
when I would at least have expected that ./rpitube.lisp be absent from this list of unrecognized arguments. I know that in Clozure CL, in order to pass a command line argument to an REPL session untouched, I have to separate them from the other arguments with a double hyphen, like this:
~/ccl/armcl -l rpitube.lisp -- {playlistURL}
But invoking the script like this clearly lands me in a REPL after the script has run, which I don't want. Additionally the Quicklisp loading information and progress bars are printed to the terminal, which I also don't want. (Incidentally, as Rainer suggested, I haven't added Quicklisp to my CCL init file, since I generally don't want the additional overhead i.e. few second's loading time on the Raspberry Pi. I'm not sure if that's relevant).
I then decided to try creating a standalone executable by running (once the above code is loaded):
(ccl:save-application "rpitube" :prepend-kernel t)
And calling it from a shell like this:
$ ./rpitube {playlistURL}
which gives:
Unrecognized non-option arguments: ({playlistURL})
which seems to be an improvement, but I'm still doing something wrong. Do I need to replace the Quicklisp-related code by creating my own asdf package that requires drakma, cl-html-extract and split-sequence, and loading that with in-package, etc.? I have created my own package before in another project - specifically because I wanted to split up my code into multiple files - and it seems to work, but I still loaded my package via ql:quickload as opposed to in-package, since the latter never seemed to work (perhaps I should ask about that as a separate question). Here, the rpitube.lisp code is so short that it seems unecessary to create a whole quickproject and package for it, especially since I want it to be a standalone executable anyway.
So: how do I change the script (or its invocation) so that it can accept the URL as a command-line argument, can be run non-interactively (i.e. doesn't open a REPL), and ONLY prints the desired output to the terminal - a space-delimited list of URLs - without any Quicklisp loading information?
Ok, I've managed to adapt a solution from the suggestion linked by user #m-n above. RpiTube now seems to work for most playlists that I have tried except some music playlists, which are unreliable since I live in Germany and many music videos are blocked in this country for legal reasons. Huge playlists, very high quality (or very long) videos might be unreliable.
The BASH script:
#! /bin/bash
#Calls rpitube.lisp to retrieve the URLs of the videos in the provided
#playlist, and then plays them in order using omxplayer, optionally
#starting from the nth video instead of the first.
CCL_PATH='/home/pi/ccl/armcl'
RPITUBE_PATH='/home/pi/lisp/rpitube.lisp'
N=0
USAGE='
Usage: ./rpitube [-h help] [-n start at nth video] <playlist URL>
where URL is of the form: https://www.youtube.com/playlist?list=uniqueplaylistcode
******** Be sure to surround the URL with single quotes! *********'
play()
{
if `omxplayer -o local $(youtube-dl -g "$1") > /dev/null`; then
return 0
else
echo "An error occured while playing $1."
exit 1
fi
}
while getopts ":n:h" opt; do
case $opt in
n ) N=$((OPTARG - 1)) ;;
h ) echo "$USAGE"
exit 1 ;;
\? ) echo "Invalid option."
echo "$USAGE"
exit 1 ;;
esac
done
shift $(($OPTIND - 1))
if [[ "$#" -ne 1 ]]; then
echo "Invalid number of arguments."
echo "$USAGE"
exit 1
elif [[ "$1" != *'https://www.youtube.com/playlist?list='* ]]; then
echo "URL is of the wrong form."
echo "$USAGE"
exit 1
else
echo 'Welcome to RpiTube!'
echo 'Fetching video URLs... (may take a moment, especially for large playlists)'
urls="$(exec $CCL_PATH -b -e '(progn (load "'$RPITUBE_PATH'") (main "'$1'") (ccl::quit))')"
echo 'Starting video... press Q to skip to next video, left/right arrow keys to rewind/fast-forward, Ctrl-C to quit.'
count=0
for u in $urls; do #do NOT quote $urls here
[[ $count -lt $N ]] && count=$((count + 1)) && continue
play "$u"
echo 'Loading next video...'
done
echo 'Reached end of playlist. Hope you enjoyed it! :)'
fi
I made the following changes to the CL script: added the :silent option to the ql:quickload calls; replace my own ocurrences function with the built-in count (:test #'equal); and most importantly several things to the code at the end of the script that actually calls the URL-fetching functions. First I wrapped it in a main function that takes one argument, namely the playlist URL, and removed the references to *command-line-argument-list* etc. The important part: instead of invoking the entire rpitube.lisp script with the URL as a command line argument to CCL, I invoke it without arguments, and instead pass the URL to the main function directly (in the call to exec). See below:
(defun main (url)
(if (stringp url)
(loop for u in (get-vid-urls url) do
(format t "~a " u))
(error "Usage: rpitube <URL of youtube playlist>
where URL is of the form:
'https://www.youtube.com/playlist?list=uniqueplaylistcode'")))
This method could be applied widely and it works fine, but I'd be amazed if there isn't a better way to do it. If I can make any progress with the "toplevel" function + executable idea, I'll edit this answer.
An example working invocation, run on a small playlist of short videos, with playback beginning at the 3rd video:
$ ./rpitube -n 3 'https://www.youtube.com/playlist?list=PLVPJ1jbg0CaE9eZCTWS4KxOWi3NWv_oXL'
Many thanks.
I looked at this some and would like to share what I found. There are also several Lisp libraries which aim to facilitate scripting, executable building, or command-line argument handling.
For your executable building approach, save-application lets you specify a :toplevel-function, a function of zero arguments. In this case you will need to get the command line arguments through ccl:*command-line-argument-list*, and skip the first element (the name of the program). This is probably the minimal change to get your program running (I haven't run this; so it may have typos):
(defun toplevel ()
(let ((args #+clozure *command-line-argument-list*))
(if (and (= (length args) 2)
(stringp (second args)))
(loop for url in (get-vid-urls (second args)) do
(format t "~a " url))
(error "Usage: rpitube <URL of youtube playlist>
where URL is of the form:
'https://www.youtube.com/playlist?list=uniqueplaylistcode'"))))
(save-application "rpitube" :prepend-kernal t :toplevel-function #'toplevel)
Alternatively, some Lisp implementations have a --scpript command-line parameter which allows something similar to your #!/home/pi/ccl/armcl script to work. CCL doesn't seem to have an equivalent option, but a previous answer -- https://stackoverflow.com/a/3445196/2626993 -- suggests writing a short Bash script which would essentially behave like you hoped CCL would with this attempt.
quickload calls can be silenced with an argument:
(ql:quickload :drakma :silent t)
I wrote a procedure (do-test).
Since the test might have some effects on the variable env,
I defined env inside do-test hoping that env would not be carried
with the procedure, so everytime I run it, I will get a fresh environment to work with.
To my surprise, my test procedures are actually carrying the previous env.
Please find the follwing code:
(define (do-test)
(define env '(1))
;(define env (list 1))
(display env)
(if (not (equal? (car env) 1))
(error "assertion failed.")
'ok)
(set-car! env 2)
'ok)
(do-test)
(do-test)
I've tried to run this code using mit-scheme / guile / codepad.org, and all of which
told me runing (do-test) twice yielded different results.
But if I change the line (define env '(1)) to (define env (list 1)), I will get the expected result.
(you can find my code before and after the change in codepad.org)
To my knowledge, '(1) and (list 1) should be the same except that the second will the call procedure list.
I'm wondering why this happens and how can I prevent the program from reusing previous values?
Please take a look at R5RS, which says that
it is an error to alter a constant (i.e. the value of a literal
expression) using a mutation procedure like set-car! or
string-set!.
So '(1) and (list 1) are not exactly the same thing: when you are intended to build some data that would be mutated in the future, don't use quotations.
If you try to do this:
(define (f) (list 1))
(define (g) '(1))
(display (eq? (f) (f)))(newline)
(display (eq? (g) (g)))(newline)
You will get:
#f
#t
This is suggesting that f will make a list whenever called. But g will treat its list '(1) as a constant and that list is allocated only once, no matter how many times it is called.
I'm trying to make a loop which asks for a command, executes the command, and then loops again. It only exits if the command entered is "exit". I have tried a couple different things, but when I run them they work once before (read-line) reads an # and then does this infinitely without waiting for input. My initial attempt looked like this:
(define (inputLoop)
(define command "init")
(do()((equal? command "exit") (display "exited successfully..."))
(display "What would you like to do?(start,stop,exit)")
(set! command (read-line))
(cond [(equal? command "start") (start)]
[(equal? command "stop") (stop)]
[else (void)])))
My next attempt was similar, but instead of using a do-loop, it just recursively called the function called inputLoop if the command was not "exit".
Here is a sample of the output:
What would you like to do?(start,stop,exit)start
What would you like to do?(start,stop,exit)
What would you like to do?(start,stop,exit)
...<thousands-of-lines-here>
What would you like to do?(start,stop,exit)
What would you like to do?(start,stop,exit)
What would you like to do?(start,stop,exit)exit
exited successfully...
This is a simple version of what you describe:
(define (input-loop)
(display "What would you like to do? (start,stop,exit)")
(define command (read-line))
(cond [(string=? command "start") (start) (input-loop)]
[(string=? command "stop") (stop) (input-loop)]
[(string=? command "exit") (displayln "exited successfully...")]
[else (displayln "unknown command") (input-loop)]))
You could also have an escape-continuation and break out of the loop; this comes closest to a classical while True: [...] break approach:
(define (input-loop)
(let/ec break
(let loop ()
(display "What would you like to do? (start,stop,exit)")
(define command (read-line))
(cond [(string=? command "start") (start)]
[(string=? command "stop") (stop) ]
[(string=? command "exit") (break)]
[else (displayln "unknown command")])
(loop)))
(displayln "exited successfully..."))
Note that in the second case, you can conveniently put the closing logic (display "exited successfully") at the end of the procedure, whereas in the first case you need to put it into the loop, otherwise the code may be executed several times.
Example execution on command-line (OS X):
pu#pumbair: ~/Projects/L-Racket racket input-loop.rkt
What would you like to do? (start,stop,exit)a
unknown command
What would you like to do? (start,stop,exit)starr
unknown command
What would you like to do? (start,stop,exit)start
start!
What would you like to do? (start,stop,exit)stop
stop!
What would you like to do? (start,stop,exit)exit
exited successfully...
using this (mock-up) version:
#lang racket
(define (start) (displayln "start!"))
(define (stop) (displayln "stop!"))
(define (input-loop)
(let/ec break
(let loop ()
(display "What would you like to do? (start,stop,exit)")
(define command (read-line))
(cond [(string=? command "start") (start)]
[(string=? command "stop") (stop) ]
[(string=? command "exit") (break)]
[else (displayln "unknown command")])
(loop)))
(displayln "exited successfully..."))
(input-loop)
After following #uselpa 's advice about checking my start and stop methods, I've come to the conclusion that my problem was in my start method. It created several places to run concurrently using the (place id body ...+) form documented in the racket documentation. The documentation states,
The bodys close only over id plus the top-level bindings of the enclosing module
Since my code had the call to the loop in the top-level, each created place executed the loop upon creation. My solution was instead to use the (dynamic-place module-path start-name) form documented here. I just had to move my code to an external file. There is no access to top-level bindings from the calling code for the place code using this method.
After making these changes, I was able to successfully run both of the loop techniques provided by #uselpa in their answer, without any infinite loops or errors.
I have the following function (I am a very beginner at Lisp):
(defun my-fun (a b)
(my-commandsend-and-reply-macro (cmd)
(:reply (ok result)
(do-something a b result)))
)
where my-commandsend-and-reply-macro is a macro written by another programmer. I am unable to modify it.
my-commandsend-and-reply-macro sends a command (in this example cmd) to a server process (it is written in another programming language) and then waits for its answer.
The answer is processed then in the macro using the user-given ":reply part of the code". The list (ok result) is a kind of pattern, in the macro a destructuring-bind destructures and binds the proper parts of the answer to ok and result (ok is just a flag). After this the other user-given lines of the ":reply part" are excuted. (for result processing)
I would like to do the following:
1, send a command like to the other process (this is ok)
2, call a function (like do-something) using the result AND using some other parameters which are the actual parameters of my-fun (this part fails...)
How can I do this? I think the problem is that a and b are not evaluated before the macro expansion and when the macro is expanded Lisp searches for a local a and b but there is no a or b. Is there any way to evaluate a and b? (so the macro could treat them like concrete values)
This is the macro def: (written by another programmer)
(defmacro* my-commandsend-and-reply-macro ((cmd &rest args) &body body)
`(progn
(with-request-id ()
(setf (gethash *request-id* *my-callbacks*)
(lambda (status &rest status-args)
(case status
,#(loop for (kind . clause) in body when (eql kind :reply)
collect
(destructuring-bind
((status-flag &rest lambda-form-pattern)
&body action-given-by-user) clause
`(,status-flag
(destructuring-bind ,lambda-form-pattern status-args
,#action-given-by-user))))
((error)
(message "Error: %s" (elt (elt status-args 0) 1))))))
(apply #'send-command-to-process *request-id* cmd args)))))
Def of with-request-id:
(defmacro* with-request-id ((&rest vars) &body body)
"Send `getid' to the server, and call `body' once the response
with the new ID has arrived. By then, global variable `*request-id*'
is bound to the latest request ID."
`(progn
(when (not (server-is-running))
(error "Server isn't running!"))
(when *reqid-queue*
(error "Some internal error occured. Please, restart the program!"))
(lexical-let (,#(loop for var in vars
collect `(,var ,var)))
(setf *reqid-queue* (lambda ()
(unwind-protect
(progn ,#body)
(setf *reqid-queue* nil)))))
(get-id)))
And getting id from the other process:
(defun get-id ()
(send-command-to-process 'getid))
Without looking into your code at all (apologies -- no time) ---
a and b are evaluated by the function my-fun. All functions evaluate their arguments to begin with -- only macros and special forms do not necessarily evaluate all of their arguments.
But those a and b values are not passed to the macro -- the only thing passed to it is the unevaluated sexp that is bound to cmd. And you do not even define cmd in your function!
What you need to do is substitute the values of a and b into the cmd sexp. You have not shown how cmd is defined/constructed, at all. Construct it using the values of a and b, and you should be OK.
To construct the cmd sexp, remember that you can use backquote syntax to simplify things, using comma syntax to pass the values of a and b. E.g.
(let ((cmd `(some funny (expression) that ((uses)) ,a AND ,b)))
code-that-uses-CMD)
This assumes that the code you pass to the macro does not need the variables a and b, and it needs only their values.
When the function my-fun is called the arguments have already been evaluated so it's not clear to me what is the problem you are facing.
The only strange thing I see is that the macro is un-hygienic and so if your arguments are named instead of a and b for example status or status-args you're going to be in trouble because the expression
(do-something <a> <b> results)
will be compiled in a context where those names have been reused by the macro.