I have a system written in Lisp that runs state machines. I'd like to dynamically load the definition of the state machine and any required assets (images, etc) from a directory, given just the name of the directory. There will be multiple different state machines. This is similar, but not identical, to Apache loading and running a WAR file.
My concern is that simply compiling and loading a file could run literally anything. Ideally I'd like to get just the state machine definition, configure it with the path to the assets, and have it available to execute. Right now I'm playing around with loading a class that implements a particular base class, but that's not straightforward. Is there a standard technique for this?
Thanks.
Are you saying you're worried about the possibilities of arbitrary code execution from reading in a file? I so you should look into redefining the read-table to exclude unwanted symbols.
For an example checkout this, look for 'SAFE-READ-FROM-STRING'.
It's not complete but then you can use #'read to get the datastructure, do some sanity check and compile if you need to.
If this isn't what you were looking for then my apologies, would you be able to explain further what you are looking for?
Given that you want to read definition of a state machine without executing arbitrary code, you may consider the following macro:
(defmacro def-state-machine (name (&rest assets) &rest states)
`(defparameter ,name
(list
:assets ',(remove-if-not #'legal-asset? assets)
:states ',(remove-if-not #'legal-state? states))))
which will create list of valid assets and states (since I don't know, how your machine looks like, I'm putting some abstract predicates here - they may check for legal syntax, or if argument is of the certain type, or e throw an error if asset or state are illegal).
Let's assume, you also need to define some function to run machine:
(defmacro def-transition (name args &body body)
`(defun ,name (,#args)
,#body))
Separate macro for defining function allows additional sanity checks. Finally, you may define reader function:
(defun load-toy-state-machine (directory)
(let ((path (cl-fad:merge-pathnames-as-file directory #P"machine.lisp"))
;(*readtable* (copy-readtable nil))
)
; (make-dispatch-macro-character #\#)
(with-open-file (stream path :direction :input)
(do ((form (read stream nil 'done)
(read stream nil 'done)))
((eql form 'done) T)
(if (member (car form) '(def-state-machine def-transition))
(eval form)
(error "malformed state machine definition file"))))))
Which will eval only allowed macros (def-state-machine and def-transition), which have fixed syntax, and may contain additional sanity checks. Neither of these executes code.
Related
The function find-data-from-command works fine when I run it without first doing the 'quickload' of the package. If I load the package it gives the error that split-sequence is undefined.
I have tried to reload split-sequence after loading the custom package. Doesn't work
(ql:quickload :<a-custom-package>)
(defun find-data-from-command (x desired-command)
(setq desired-data ())
(loop for line = (read-line x nil)
while (and line (not desired-data)) do
(progn
(setq commands (first (split-sequence ":" line)))
(setq data (split-sequence "," (first (rest (split-sequence ":" line)))))
(print (cons "command:" commands))
(cond
((equal commands desired-command) (return-from find-data-from-command data))))))
FIND-DATA-FROM-COMMAND
SIGMA 24 >
(setq obj-type (find-data-from-command (open "log.txt") "types"))
Error: Undefined operator SPLIT-SEQUENCE in form (SPLIT-SEQUENCE ":" LINE).
The problem is nothing to do with Quicklisp, it's to do with a package you've defined somewhere called SIGMA. In particular somewhere in your code is a form which looks like:
(defpackage "SIGMA" ;or :sigma or :SIGMA or #:sigma or ...
...
(:use ...)
...)
And then later
(in-package "SIGMA")
And the problem with this is that your package definition has an explicit (:use ...) clause.
defpackage, and the underlying function make-package has slightly interesting behaviour for the :use clause (or keyword argument in the case of make-package):
if none is given then there is an implementation-defined default;
if one is given then it overrides the default.
The idea, I think, is that implementations may want to provide a bunch of additional functionality which is available by default, and this functionality can't be in the CL package since the contents of that package is defined in the standard. So if you just say
(defpackage "FOO")
Then the implementation is allowed (and, perhaps, encouraged), to make the use-list of FOO have some useful packages in it. These packages might be the same ones that are in the default use-list of CL-USER, but I'm not sure that's required: the whole thing is somewhat under-specified.
The end result of this is that if you want to define packages which both make use of implementation-defined functionality and have explicit use-lists you have to resort to some slight trickery. How you do this is slightly up to you, but since you are by definition writing implementation-dependent code where you are defining packages like this, you probably want to make it clear that what you are doing is implementation-dependent, by some form like
(defpackage :foo
(:use ...)
#+LispWorks
(:use :lispworks :harlequin-common-lisp :cl)
...)
Or, if you just want some particular set of symbols
(defpackage :foo
(:use ...)
#+LispWorks
(:import-from :lispworks #:split-sequence))
Note that this is not quite the same thing as using a package containing the symbol.
In all these cases if your code has pretensions to be portable then there should be appropriate clauses for other implementations and a way of knowing when you're trying to run on an implemention you haven't yet seen: how to do this is outwith the scope of this answer I think.
Solved it.
The quickloading of the custom package was taking me into the package. I did not realize that. So I had to specify split-sequence is from outside. So, I replaced all occurrences of split-sequence in the function definition with
LISPWORKS:split-sequence
Then it worked.
If any one has a better solution, please do let me know. Thanks
Is it possible to expand a macro at run time in a compiled lisp executable? I would expect it to be impossible as macro expansion can only happen pre-compilation yet when i call macroexpand in compiled code i get output.
A macro is a function that's normally called automatically during compilation or evaluation, and whose return value is then compiled or evaluated in place of the original expression.
But since it's just a function, there's nothing preventing it from being called explicitly during run time as well, and that's what MACROEXPAND and MACROEXPAND-1 do.
It's roughly equivalent to:
(defun macroexpand-1 (form &optional env)
(if (and (listp form) (car form)) ;; list expression
(let ((macfun (macro-function (car form)))
(if macfun
(funcall macfun form env)
form))
form))
(Note that this definition doesn't handle symbol macros or use *MACROEXPAND-HOOK*, to keep it simple.)
It's possible to use EVAL-WHEN when defining the macro to make the macro definition only available in the compilation environment. If you do that, trying to expand at run time will fail.
In Lisp, the terms "run time" and "compile time" are situations in which a particular piece of code is being processed, not absolutes like in some static languages. If we evaluate (compile nil '(lambda ())), this is the compile function's run-time, but the lambda form's compile time: both times are happening at the same time.
The entire language is available in all situations. When you build a self-contained executable, that image contains not only support for expanding macros, but for compiling code. Your Lisp application can call compile-file to compile Lisp source to object form, and load to load the resulting object code.
The process of removing garbage, and unused functionality from a Lisp application image to make it smaller is called "tree shaking". If you don't want the compiler or macro expander in your application, find out whether/how they can be removed with your implementation's tree shaking support.
(solved, see the comments)
Recently I've been working on an API that has to interface with an already existing service. Everything seems to be working quite well, and my project is just starting to get large enough that I would see some benefit from throwing things into a package. As this is my first "real" project in CL, I think I'm not fully understanding the packaging/loading mechanisms going on here.
My basic problem is that I have a bunch of code that uses macros to generate functions/classes, interns them into my package, and then exports certain functions/accessors that people will eventually use to interact with the API. If I load the files individually like so:
(load "~/src/lisp/cl-bitcoin/bitcoin.lisp")
(load "~/src/lisp/cl-bitcoin/classes.lisp")
(load "~/src/lisp/cl-bitcoin/functions.lisp")
Everything works well. The functions declared by my macros are interned and exported correctly and I'm able to call them to interact with the API. However, if I attempt to do the following:
(ql:quickload :btc)
Quicklisp tells me that everything was loaded properly - and it seems that most of the loading process happened as I expected since all of my dependencies are loaded and available for use. The problem is that everything related to my package is not available. This includes functions that are exported directly from my package.lisp file. For reference, here are my .asd and package files:
package.lisp
(defpackage #:btc
(:use #:cl)
(:export #:set-connection-parameters
#:reset-rpc-id
#:with-connection-parameters
#:btc-base-err
#:btc-base-id))
btc.asd
(asdf:defsystem #:btc
:serial t
:depends-on (#:drakma
#:flexi-streams
#:cl-json)
:components ((:file "package")
(:file "bitcoin")
(:file "classes")
(:file "functions")))
I feel like I'm missing something fairly obvious here - I've been looking into eval-when and other related loading functions but haven't been able to figure this out. Can someone explain to me what's going on here?
Thanks for your help.
Edit: Here's what my REPL looks like:
; SLIME 2011-02-04
CL-USER> (ql:quickload :btc)
To load "btc":
Load 1 ASDF system:
btc
; Loading "btc"
..
(:BTC)
CL-USER> (btc:help "getinfo")
Invoking restart: Return to SLIME's top level.
; Evaluation aborted on #<SIMPLE-ERROR #x30200175DF0D>.
CL-USER> ; Reader error: No external symbol named "HELP" in package #<Package "BTC"> .
; No value
CL-USER> (load "/Users/jordan/src/lisp/cl-bitcoin/bitcoin.lisp")
#P"/Users/jordan/src/lisp/cl-bitcoin/bitcoin.lisp"
CL-USER> (load "/Users/jordan/src/lisp/cl-bitcoin/classes.lisp")
#P"/Users/jordan/src/lisp/cl-bitcoin/classes.lisp"
CL-USER> (load "/Users/jordan/src/lisp/cl-bitcoin/functions.lisp")
#P"/Users/jordan/src/lisp/cl-bitcoin/functions.lisp"
CL-USER> (btc:help "getinfo")
#<BTC::BTC-SINGLE #x3020017DBDDD>
CL-USER>
And the code to generate functions:
;;;; package information
(in-package #:btc)
;;; externally visible functions - this class contains the public api for
;;; cl-bitcoin as well as the function building framework that we need to
;;; easily handle the multiple return types that are possible from the
;;; bitcoind server methods
;; function building framework - resolving function return types into
;; specific btc objects (as defined in classes.lisp)
(defun create-btc-obj (fn result err id)
(case fn
((:getbalance :help) (make-btc-single result err id))
(otherwise (error "Unable to parse function ~S to a btc object" fn))))
(defmacro defbtcfun (name &rest args)
(let ((g (gensym)) (result (gensym)) (err (gensym)) (id (gensym)))
`(progn
(defun ,name ,args
(let ((,g (intern (string ',name) :keyword)))
(multiple-value-bind (,result ,err ,id) (get-bitcoind-result ,g ,#args)
(create-btc-obj ,g ,result ,err ,id))))
(export ',name 'btc))))
;; function definitions (each of these functions should have a corresponding case in
;; create-btc-obj above, otherwise a condition will be signaled
(defbtcfun help method)
(defbtcfun getbalance account minconf)
Jordan Kaye posted this as a comment, but never turned it into an answer:
It turns out that this was just an incorrect configuration issue in
asdf by me.. I accidentally set up the project in an incorrect
directory. When loading the project, quickload was actually loading an
entirely separate .asd file than the one that I was expecting. So,
when I loaded the (correct) files manually, the code worked as
expected - quickload was loading the wrong files. Thanks a lot for
your help, it led me to figure out the cause!
I want to customize environment while the specific package is installed properly. How to check whether some package is installed in elisp?
Something like this?:
(if (require 'ecb)
(progn (setq ....))
(message "ECB not installed!"))
tripleee's answer is a handy example of error handling, but unnecessary in this instance.
(when (require 'some-library nil 'noerror)
do-things)
That 'noerror can be any non-nil value, but of course it's more descriptive this way. I often see :noerror used as well, but I've no idea if there's any particular advantage to using a keyword argument over a symbol (comments, anyone? I'm quite interested to know).
require is a built-in function in C source code.
(require FEATURE &optional FILENAME NOERROR)
If feature FEATURE is not loaded, load it from FILENAME.
If FEATURE is not a member of the list features, then the feature
is not loaded; so load the file FILENAME.
If FILENAME is omitted, the printname of FEATURE is used as the file name,
and load will try to load this name appended with the suffix .elc or
.el, in that order. The name without appended suffix will not be used.
See get-load-suffixes for the complete list of suffixes.
If the optional third argument NOERROR is non-nil,
then return nil if the file is not found instead of signaling an error.
Normally the return value is FEATURE.
The normal messages at start and end of loading FILENAME are suppressed.
The (require) will throw an error if it fails. That should really be all you need.
If you want to configure ECB's behavior only when it is available, look primarily into adding stuff to ecb-hook -- this is the normal way to configure an Emacs package conditionally.
If no hook is available, or you want to roll it by hand for some reason, try something like
(eval-after-load 'ecb '(setq ecb-be-more-like-better-yes-p t))
If you really really want to roll this by hand, you can trap the error from a failed require like this:
(condition-case nil
(progn
(require 'ecb)
(setq ecb-be-more-like-better-yes-p t) )
(file-error (message "ECB not available; not configuring") ))
Note that the condition-case will catch any file-error from inside the progn so you want to make sure you don't do any other file operations inside. Ultimately you may want to put just the require inside a condition-case and use that as the condition for your original if form, but this is already getting out of hand ...
(if (condition-case nil (require 'ecb) (error nil))
(setq ecb-be-more-like-better-yes-p t)
(message "ECB not available; not configuring") )
Why not using "featurep"?
(featurep FEATURE &optional SUBFEATURE)
Return t if FEATURE is present in this Emacs.
(if (featurep 'ecb)
(message "ECB is there!"))
For people wondering how to check if a package.el package is installed, use package-installed-p.
Four years late on this question, but... here's a simple macro that will do this for you.
(defmacro when-feature-loaded (feature &rest body)
"Executes BODY if and only if FEATURE is loaded."
(declare (indent defun))
`(when (featurep ,module)
#,body))
For example:
(when-feature-loaded 'foo
(message "foo is loaded!"))
Here's another version with an "else" case, in case you need to handle that as well.
(defmacro if-feature-loaded (module then-form else-form)
"Executes THEN-FORM if and only if MODULE is already loaded, otherwise executes ELSE-FORM."
(declare (indent 2))
`(if (featurep ,module)
,then-form
,else-form))
The simplest answer is to use require and eval-after-load as stated in other answers.
However that is not always convenient, such as in a function called by a mode hook that wants to activate another minor mode but only if the package for it is installed. In this case featurep is relevant.
Emacs packages increasingly use autoload to improve startup time. If you test for the presence of a package by using require then you are wearing the cost of loading the file(s). If using ELPA/MELPA/Marmalade packages (available by default since version 24) then many packages may be available in an as-yet unloaded state, but a package foo with autoloads will provide a feature foo-autoloads. I wrote a function that is useful for testing whether a package is available in terms of already being loaded or set to autoload.
(defun autofeaturep (feature)
"For a feature symbol 'foo, return a result equivalent to:
(or (featurep 'foo-autoloads) (featurep 'foo))
Does not support subfeatures."
(catch 'result
(let ((feature-name (symbol-name feature)))
(unless (string-match "-autoloads$" feature-name)
(let ((feature-autoloads (intern-soft (concat feature-name "-autoloads"))))
(when (and feature-autoloads (featurep feature-autoloads))
(throw 'result t))))
(featurep feature))))
N.B.: Using (require ..) will load the package, which defeats the benefit of autoloads (deferred loading).
I use this in my code to configure functions which are autoloaded:
(when (fboundp 'some-function)
...)
Example
I have this in my init.el:
(when (fboundp 'ace-select-window)
(keymap-global-set "M-s-u" 'ace-select-window))
If you have ace-window installed, you can test this yourself by copying it into your own init file, restarting Emacs, and running:
(autoloadp (symbol-function 'ace-select-window))
It should return T. Then run the command, and execute that snippet again; it should return nil.
(newbie tip: Hit altshift:, then enter that, and press enter)
This avoids loading your package just to configure it (which is the point of autoloading).
(newbie tip nr 2: using fbound is for functions; use boundp for values)
This is mostly a follow-up to this question. I decided to just keep YAGNI in mind and created a global variable (libpython). I set it to #f initially, then set! it when init is called. I added a function that should handle checking if that value has been initialized:
(define (get-cpyfunc name type)
(lambda args
(if libpython
(apply (get-ffi-obj name libpython type) args)
(error "Call init before using any Python C functions"))))
So now here's what I want to do. I want to define a macro that will take the following:
(define-cpyfunc Py_Initialize (_fun -> _void))
And convert it into this:
(define Py_Initialize (get-cpyfunc "Py_Initialize" (_fun -> _void)))
I've been reading through the macro documentation to try figuring this out, but I can't seem to figure out a way to make it work. Can anyone help me with this (or at least give me a general idea of what the macro would look like)? Or is there a way to do this without macros?
I've answered most of this question in the other one (I didn't see this one). It's fine to use a function that pulls out the bindings like this, but one possible problem here is that since you generate the binding only when the resulting function is called, this binding is re-created on each and every call. An easy way to solve this quickly is using promises, something like this:
(require scheme/promise)
(define (get-cpyfunc name type)
(define the-function
(delay (if libpython
(get-ffi-obj name libpython type)
(error "Call init before using any Python C functions"))))
(lambda args (apply (force the-function) args)))
But this is essentially almost the same as the code I posted in your previous question.
More random notes:
get-ffi-obj will accept a symbol as the name to bind to -- this is intentional, to make such macros (as in the last question) easy.
Using (symbol->string 'name) in a macro is fine. As I noted above in my comment reply to Nathan's comment, this means that it gets called at runtime, but mzscheme should be able to optimize that anyway, so there's no need to try and write some sophisticated macro that does the job at compile time.
Look inside the PLT directory -- you will find a collection called ffi. This is a collection of examples of bindings with various styles. Macros that create the bindings are very common in these examples.
Why don't you change the generated code to
(define Py_Initialize (get-cpyfunc 'Py_Initialize (_fun -> _void)))
and then have get-cpyfunc run (symbol->string name)?
Granted, there is probably a way to do this with syntax-case (I can never remember its syntax though), and definitely if you're using a Scheme with CL-esque define-macro.
It's not the complete answer, but I came up with a macro that meets both requirements (defines a variable and a string with the name of that variable):
> (define-syntax (my-syntax stx)
(syntax-case stx ()
[(_ id)
#'(define-values (id) (values (symbol->string (quote id))))]))
> (my-syntax y)
> y
"y"