I am currently trying to wrap my head around packages, systems & co.
I now have read Packages, systems, modules, libraries - WTF? a few times, and I think I'm still having difficulties to get it right.
If I simply want to split a Lisp source file into two files, where one shall "use" the other - how do I do that? Do I need to build a system for this? Should I use a module? …? I'm coming from a Node.js background, and there you can simply say
var foo = require('./foo');
to get a reference to what's exported in file foo.js. What is the closest equivalent to this in Lisp?
I understand that ASDF is for systems, and that it is bundled as part of Quicklisp, at least according to its documentation:
ASDF comes bundled with all recent releases of active Common Lisp implementations as well as with quicklisp […]
Okay, Quicklisp is for libraries, but what is their relationship? Is Quicklisp something such as a "package manager" in other languages? And if so, then what exactly does ASDF provide?
Sorry for these many questions, but I think it just shows the trouble I have to understand how to structure Lisp applications. Any help would be greatly appreciated :-)
System
For structuring large system use a system management tool. A 'free' one is ASDF.
You would need a system declaration, which lists the parts of you library or application. Typically it goes into an own file. Then you load a system or compile a system. There should be tutorials how to do that.
A simple Lisp system might have the following files:
a system file describing the system, its parts and whatever other stuff is needed (other systems)
a package file which describes the namespaces used
a basic tools file (for examples functions used by the macro)
a macro file which lists the macros (used so that they get compiled/loaded before the rest of the software)
one or more other files with functionality.
Quicklisp is independent of that. It's a software distribution tool.
Quick hack to compile and load files
But you can also compile and load files the old fashioned way without a system tool:
(defparameter *files*
'("/yourdir/foo.lisp" "/yourdir/bar.lisp"))
(defun compile-foobar ()
(mapc #'compile-file *files*))
(defun load-foobar ()
(mapc #'load *files*))
(defun compile-and-load ()
(mapc (lambda (file)
(load (compile-file file)))
*files*))
In reality there might be more to it, but often it's enough. It should be easy to write your own building tool. A typical system tool will provide many more features for building more complex software in a structured way. Many of the ideas for these tools reach back at least 35 years. See for example the Lisp Machine manual, here the edition from 1984, chapter Maintaining Large Systems.
The role of files
Note that in plain Common Lisp the role of files and its semantics are not very complex.
A file is not a namespace, it is not associated with a class/subclass or an object, it is not a module. You mix Lisp constructs in a file like you want. Files can be arbitrary large (for example one complex library has a version where it is delivered as single source file with 30000 lines). The only real place in the standard semantics where a file plays a role is when compiling a file. What side effects has compiling a file? What optimizations can a compiler do?
Other than that it is assumed that the development environment provides services like load and compiling groups of files aka systems, provide overviews of compilation errors, record source locations of definitions, can locate definitions and more. A tool like ASDF handles the system part.
There is a require function in Common Lisp, but it deprecated. If you simply want to split your code in one or more pieces, to use it interactively in the REPL, you can put the code in different files and then load each of them. If instead you want to write a full lisp project, I have found very useful the quickproject package, that provides a simple starting point for the creation of new packages.
Related
Following this clisp basic website tutorial it asks me to define a package to persist the code:
(defpackage :retro-games
(:use :cl :cl-who :hunchentoot :parenscript))
However, I cannot work out where my code is being persisted. Where are the files?
NB: I have only one day's experience with lisp!
Where are the files? They are where you want them.
Create a new file anywhere you want.
Put the definitions into the file.
Done
Lisp can LOAD files and you can use COMPILE-FILE.
In Common Lisp there no connections between files, the location of a file and a package. The package definition just defines a namespace to symbols. But this is not necessarily connected to a file. Some tools help maintain something called a system, which is a collection of files, which can be compiled and loaded together.
To get a basic understanding how to use Common Lisp I recommend reading Practical Common Lisp by Peter Seibel.
This is I suspect, a matter of style and/or personal taste but I thought I'd ask anyway.
I have been in the habit of defining packages thus:
(defpackage :wibble
(:use :cl :drakma)
(:export :main))
Once I have executed IN-PACKAGE (:wibble, in this case), I can then use the symbols in DRAKMA unadorned:
(http-request ...
Then I recently read that seasoned Lisp hackers would rather not :use but:
(drakma:http-request ...
Just wondered what the consensus of opinion was on here and whether there were any pros or cons (not that type of CONS :) ) either way?
Cheers,
Peter
When you use a package, there are a couple subtle ways things might go wrong if the used package changes.
First, the package might export more symbols in the future. If, for example, the package exports a new symbol library:rhombus and you're already using that myapp::rhombus to name something, you are suddenly using the inherited symbol, with all possible attachments (e.g. classes, defuns, macros, etc), with sometimes strange results. If you use qualified symbol names, you will not get any more or any less than the symbols you want.
Second, the package might stop exporting symbols in the future. So if, for example, library:with-rhombus disappears, your call to (with-rhombus (42 42 42) ...) will suddenly get an error for an invalid function call (42 ...) rather than something that points directly to the source of the problem, the "missing" symbol. If you use qualified symbol names, you will get an error along the lines of Symbol WITH-RHOMBUS is not exported from the LIBRARY package which is clearer.
Importing symbols (with :import-from or :shadowing-import-from or import) is not without its own trouble. Importing works on any symbol, regardless of whether it's external or not. So it could be the case that the symbol is now library::rhombus, i.e. not intended for public consumption any more, but importing will still work with no errors.
Which option you use depends on your comfort level with the source package. Do you control it, and you will not make any conflicting changes without thorough testing? Go ahead and import or use to your heart's content. Otherwise, be careful about checking for unintended side-effects as library package interfaces change.
This is more a style issue, so it's impossible to categorize it in black and white, but here are the pros and cons:
Using package-qualified symbols.
Avoids symbol conflicts.
Allows to clearly distinguish foreign symbols.
Allows to easily search, replace, copy,... uses of a certain symbol from the external library (for refactoring, extracting the code to some other place etc.)
Makes code uglier, but only when library names are too long. (For example, I add a nickname re to cl-pprce, and now the code using it is even better, than w/o qualification: think re:scan)
Importing the whole package
Basically, the opposite of the previous case. But I tend to use it with utility libraries, because using qualified names often beats their whole purpose of making code more concise and clear :)
:import-from package symbol
This is one option you've forgotten to mention. I think it may be useful, when you use one or too very distinct symbols from a certain package very frequently. It also allows to import unexported symbols.
Good answers so far.
Another view is that a package and its symbols make up a language. If you think a symbol should be a part of this language, then you should make it available without the need to qualify it with another package - when programming in this language.
For example in the CLIM implementation there is a CLIM-LISP package which sets up the implementation language. It is a variant of the COMMON-LISP package. Then there are packages like CLIM-SYS (resources, processes, locks, ...), CLIM-UTILS (various utilities and extensions of Common Lisp) and CLIM itself. Now in a new package SILICA (an abstract window system) these four packages are used. The implementation of Silica thus is implemented in a language which is built as a union of two languages (the Common Lisp variant CLIM-LISP and the UI commands of CLIM) plus two utility packages which extend CLIM-LISP with some facilities.
In above example it makes sense to use the packages, since they are extending each other to form a new language and the implementation in that new package makes heavy use of those.
If you had a package which needs conflicting packages, then it would not make sense to use them. For example a package could use drawing commands tailored to a GUI and for Postscript output. They would have similar names. Using them both would lead to conflicts. You also want to make clear in the source code for the human reader from where these symbols are coming. Is it a line-drawing command from a postscript or a GTK+ library? Would be great if you can find it out easily - even though the function names are the same.
As a rule of thumb, I :use packages that extend the general language, but use qualified symbols for packages that have some special application. For example, I'd always :use alexandria, but refer fully qualified to symbols from Hunchentoot. When in doubt, I use qualified names.
Christian Queinnec has written a masterpiece called LISP In Small Pieces, which features eleven Lisp Interpreters and two Lisp compilers.
When you go to download the code from the website here - it has the comment:
The programs of this book are available on the net.
These programs used to run with some Scheme systems around 1994.
Any idea:
(a) What Scheme systems these ran on at the time, and more importantly;
(b) What Scheme systems these would run on today?
There's a lot of programs in there. I did a few tests to see how well I could answer this without having to try them individually. There are 131 files in the tarball with extension ".scm". However there appear to be Scheme programs with other extensions such as .bgl. So I did a search for files containing 'L i S P' in the first five lines. That yields 173 files. I tried running all of these on my preferred Scheme implementation. 31 of these run without error. Almost all of these are in the "src" directory. So the language-specific programs really do seem language-specific. Let's look at one of the src/ files that failed, "chap9z.scm". It's choking on define-abbreviation. I don't know the origin of this symbol, but it's not defined anywhere in guile. But all of its uses could be performed by guile's syntax-rules.
Some Scheme implementations that existed in 1994 still are still around and maintained: Scheme 48, Chez Scheme, Gambit, Bigloo, MIT Scheme and SCM.
Probably the code from LiSP will run in other modern Scheme systems such as Guile or Larceny.
Personally, I would recommend using Racket. Most likely, much of the code will run in #lang racket with no changes, and there's no requirement to use [] (but your code may be easier to read :). Things that don't work are probably easy to fix, and you can also use the R5RS language implementation provided by Racket which will likely work for all of the code.
(a) What Scheme systems these ran on at the time
The Makefile in the source tarball from the author's website has targets for running the code under bigloo, elk, gambit, mit-scheme, scheme2c, and scm.
The Makefile mentions SCM 4e1 and Bigloo 1.9d as known working versions, though I haven't tested them myself. I didn't find any mention of specific versions for the other schemes.
(b) What Scheme systems these would run on today?
The code in this github repo has been updated so that almost all of the tests in the included test suite pass with current (as of 06/2014) versions of bigloo, gambit, and mit-scheme.
If you just want to be able to run the code and follow along with the book, one of those schemes should work for you.
[full disclosure: I'm the owner of the repo and I'm a Scheme noob. The code in the repo is WOMM certified, but your mileage may vary.]
If, on the other hand, you're not content to use bigloo / gambit / mit-scheme, it shouldn't be too hard to add support for guile / racket / insert-favorite-scheme-here. Use one of the book.* files as a starting point, e.g. gambit/book.scm or mitscheme/book.mit. If you can get a version of book.scm to load in your favorite scheme, then have a look at the test.interpreters make target, and finally the grand.test target to verify things are working as expected.
The included README file states:
These files were tested with a Scheme interpreter augmented with
a test-suite driver (tester.scm),
the define-syntax and define-abbreviation macros (using
Dybvig's syntax-case package),
and an object system: Meroonet (meroonet.scm).
Bigloo, Scheme->C, Gambit, Elk or SCM can be used. The first three are
better since a specialized interpreter may be built that contains a
compiled Meroonet and compiled hygienic macros.
Apparently Appleby has posted an updated version of the source code. Racket is missing though )=
See https://github.com/appleby/Lisp-In-Small-Pieces
The example below is given in Paul Grahams ANSI Common Lisp as an example of doing encapsulation:
(defpackage "CTR"
(:use "COMMON-LISP")
(:export "COUNTER" "INCREMENT" "CLEAR"))
(in-package ctr)
;function definitions here
However in Peter Seibels Practical Common Lisp, link here, he says:
Because packages are used by the reader, a package must be defined
before you can LOAD or COMPILE-FILE a file that contains an IN-PACKAGE
expression switching to that package. Packages also must be defined
before other DEFPACKAGE forms can refer to them...
The best first step toward making sure packages exist when they need
to is to put all your DEFPACKAGEs in files separate from the code that
needs to be read in those packages
So he recommends creating two files for every package, one for the defpackage and one for the code. The files containing defpackages should start with (in-package "COMMON-LISP-USER").
To me it seems like putting the defpackage in the same file, before the in-package and code, is a good way to ensure that the package is defined before used. So the first method, collecting everything into one file seems easier. Are there any problems with using this method for package creation?
I think that using a separate file for defpackage is a good habit
because:
You don't « pollute » your files with defpackage.
It makes it easier to find the exported/shadowed/... symbols, you
know you just have to look at package.lisp.
You don't have to worry about the order when you use ASDF.
(defsystem :your-system
:components ((:file "package")
... the rest ...))`
Peter Seibel says so ;)
EDIT:
I forgot to mention quickproject which facilitates the creation of
new CL projects.
REPL> (quickproject:make-project "~/src/lisp/my-wonderful-project/"
:depends-on '(drakma cl-ppcre local-time))`
This command will create a directory "~/src/lisp/my-wonderful-project/"
and the following files:
package.lisp
my-wonderful-project.asd (filled)
my-wonderful-project.lisp
README.txt
And thus, I think it's good to use the same convention.
I tend to use multiple source code files, a single "packages.lisp" file and a singe "project.asd" system definition file for most of my projects. If the project requires multiple packages, they're all defined in "packages.lisp", with the relevant exports in place exported.
There is this reason for putting DEFPACKAGE in its own file: if you have a large package, then you might have several groups of related functions, and you might want to have separate source files per function group. Then all the source files would have their own IN-PACKAGE at the top, but they would all "share" the external DEFPACKAGE. Then as long as you get the DEFPACKAGE loaded first, it doesn't matter the order you load the other source files.
An example I'm currently working on has multiple classes in the package, and the source files are broken up to be per class, each having a class definition and the related generic function and method definitions.
I'm learning Lisp (SBCL) and I'm confused about how lisp code is structured. I believe that Lisp Packages are like Python's Modules?
I want to do something like this.
utilities.py:
def foo():
print "And there is silence!"
main.py:
import utilities as u
u.foo()
I've looked up packages, loading and asdf systems. I still don't understand how it all fits together.
Regarding the comparison to Python:
Packages are the most similar thing in CL to Python modules since both are namespaces. Systems and system tools (like ASDF) are more similar to stuff like Distutils and Setuptools in Python.
The most important difference between Python modules and CL packages: packages are independent from files and directories. Instead you use defpackage to create packages and use-package to place the following code into a specific package. There are other package-related operators, but they are mostly for interactive use, these two are the most important in source files.
See the relevant chapter in Practical Common Lisp to get more detailed information.
A package is a namespace for symbols. One can import and export symbols from and to symbols. A symbol maybe interned in one package. Packages can be used by other packages.
A program is structured into systems. A system is a collection of files, dependencies, build rules, default information and more - depebding on the system tool used. ASDF is one of those. Loading libraries is then done by loading systems. Systems can also be compiled, printed, ...
Packages and systems are independent of each other and not related.
It makes sense for each larger piece of software, library or program, to use one or more custom packages. This avoids name clashes with symbols from other loaded software.
As has been remarked, compared to other languages:
Packages correspond to namespaces.
Systems correspond to modules or libraries (i.e., collections of code).
About packages: In order to grok those (which is necessary to avoid symbol conflicts and the like), I strongly recommend reading Ron Garret's Complete Idiot's Guide to Common Lisp Packages.