Over the weekend, I had a name clash that was very hard to track down, but I managed to boil it down to a short example - thing is, I thought the package system was supposed to protect me from this, so I'm wondering how it can in future.
If I do this:
(ql:quickload "cl-irc")
(defpackage #:clash-demo
(:use #:cl
#:cl-irc))
(in-package #:clash-demo)
;; This is the name that clashes - I get a warning about this if I compile
;; this interactively (i.e. from slime) but not if I quickload the whole project.
(defun server-name (server)
(format nil "server-name ~a" server))
;; This needs an IRC server to work - if you have docker
;; then this will do the trick:
;;
;; docker run -d --rm --name ircd -p 6667:6667 inspircd/inspircd-docker
(defparameter *connection*
(cl-irc:connect :nickname "clash-demo"
:server "localhost"
:port 6667
:connection-security :none
:username "username"))
After the above, I get the following warning when defining server-name:
WARNING: redefining CL-IRC:SERVER-NAME in DEFUN
And the following error if I try and print *connection* (in my more full-fledged project I got a missing slot in a class that I'd defined - I think the root cause of both the problem I found and the minimal example above is the same though):
Control stack guard page temporarily disabled: proceed with caution
While I get the warning if I define things interactively, in practice I moved a bunch of code into a quickproject:make-project'd and ql:quickload-ed it, which I think silenced the warning as it always loaded cleanly, hence why it took me so long to track down the name clash.
My questions are:
Isn't the package system supposed to protect me from this? I think I can sort of see why the above happens - the reader's already seen the symbol server-name so it thinks I'm referring to the already defined cl-irc:server-name, and re-uses that - but surely the package system should somehow allow me to work around this?
I'm assuming the warning when I quickload-ed the project was silence because quickload assumes I don't want to see warnings from projects, is there a way I can make this more forceful when I load projects I'm making so that it raises an error, or at least warns me of these name clashes? For all I know there are a bunch more that just haven't caused me a problem yet.
I was expecting either (i) the names not to clash (i.e. my file would define the symbol clash-demo:server-name, not re-use cl-irc:server-name and cause it to be redefined) or (ii) this to be an error, or at least a warning when I quickload the project.
Thanks very much in advance for any advice!
Suppose you really like packages A and B and want to :use both of them in your package MINE. They have a hundred external symbols each and you don't want to have to type any package prefixes. However, they both export a symbol with the same name, a:frob and b:frob. If you simply :use them, you will get a symbol conflict.
To resolve the conflict, there are three options to decide what to do when you're in package MINE and you refer to the unqualified symbol frob:
Prefer A so it refers to the the symbol a:frob: (defpackage mine (:use a b) (:shadowing-import-from a frob))
Prefer B so it refers to the symbol b:frob: (defpackage mine (:use a b) (:shadowing-import-from b frob))
Prefer neither so it refers to mine::frob: (defpackage mine (:use a b) (:shadow frob)) - then, to use one from A or B you must write a:frob or b:frob explicitly
Any of these three cases may be preferable depending on your situation. Common Lisp will not automatically choose one for you. I think this is a reasonable design choice.
Briefly, no: the package system is not meant to protect you against this. If you use a package CL-IRC then you're saying that you want, for instance, to get the symbol CL-IRC:SERVER-NAME when you type server-name. What you are not saying is whether you should be allowed to modify the values associated with that symbol in any way, which is an orthogonal question. The package system is just about names, not values.
In the case of functions, then it's very often (but not always! consider loading patches) a mistake to define a function with a given name in multiple contexts. In the case of variables that's slightly less clear: it's probably a mistake if there are multiple def* forms for a given variable in different contexts, but simply assigning to the variable might well be fine.
So what is needed is a way for defining macros (defun etc) to be able to detect this and complain about it.
CL does not provide such a mechanism. Many implementations do however, either by detecting 'different contexts' (which I have been vague about) or by providing a way of saying that certain packages are sacred and redefinitions should not be allowed, or both.
In this case, the implementation has warned you about the redefinition, but Quicklisp may have suppressed that. I am not sure how to desupress warnings like this in Quicklisp.
In summary the answer is that the problem of controlling and limiting redefinition is orthogonal to what the package system does, and unfortunately CL does not provide a standard solution to this second problem.
If you are interested I have a little shim which uses the condition system to make very sure warnings are treated as errors in contexts like this. I could append it to this answer, but not until after Christmas probably.
Related
In a CppCon talk (https://www.youtube.com/watch?v=80BZxujhY38 at 5:00)
Herb Sutter alluded to defun defun 3 being somehow a problem. After
I googled it, it's still not clear to me why. Can someone elaborate?
In a comment from the same video:
Herb Sutter
See also the paper, P0707 (http://wg21.link/p0707), and search for "defun." Lisp defun (and Scheme define) lets you define a function... but in Lisp and Scheme you can even redefine built-in functions and macros, including defun/define itself which is what "defun defun" / "define define" does. Here's a sample related StackExchange question: https://emacs.stackexchange.com/questions/375/symbols-value-as-a-variable-is-void-defun-when-reloading-emacs .
I'm not interested in doing anything like that in C++, and there's nothing like that in my proposal, you can't change any definitions (including of this class after it is defined), you can't reach out and affect anyone else's type or code, the only thing you can do is participate in generating the one-time-and-then-immutable definition of this class you're writing right now which is nice and localized and bounded... and still very powerful.
The linked paper contains this section:
5.2.1 Problems in other languages
In Lisp and related languages, programmers can redefine other people’s code and even global language facilities (e.g., the notorious (defun defun () 3) in Lisp, or (define define () 3) in Scheme). This is powerful, but undisciplined (causes arbitrary global effects up to and including breaking the language itself), fragile (Lisp makes it notoriously easy to write “write-only” code that is difficult to review, read, and maintain), and causes programs to be tightly coupled among their components and with their developer’s environment (Lisp makes it notoriously easy to write code whose meaning depends on local customizations, is hard to share, and when shared is hard to compose with other code that came from an environment with competing assumptions).4
The footnote says:
4 Various incarnations and offshoots of Lisp attempted to mitigate this problem in various ways without actually taking away the root cause: Common Lisp added the guarantee that all symbols in the package COMMON-LISP are protected and must not be redefined by user code otherwise you get undefined behavior; although this provides some protection for the standard facilities, it does not solve the general problem because it still permits one set of user code to redefine things in another set of user code. Also, implementations like SBCL attempted to further improve the problem by providing ways to “lock” packages so their contents cannot be accidentally redefined; however, even SBCL also provides ways to “unlock” them again.
I want to keep a list of required modules of a particular module (let's say the current-module).
I feel like there are some other options (such as parsing the module?) that could be tried, but I started playing with the idea of shadowing (require) and adding the required items to a hash-table with the module-name. The problem is I cannot figure how to write a syntax definition for it.
Although not working, a function definition equivalent would be like below:
(define require-list (make-hash))
(define require
(lambda vals
; add vals to hash-table with key (current-namespace)
(let ([cn (current-namespace)])
(hash-set! require-list cn
(append vals (hash-ref require-list cn))))
(require vals)))
.. it seems the last line call should be modified to call the original (require) as well?
A correct version or a pointer to how to do it, or any other way of achieving the original goal highly appreciated.
If you just want to get a list of imports for a particular module, there is a convenient built-in called module->imports that will do what you want. It will return a mapping between phase levels and module imports—phase levels higher than 0 indicate imports used at compile-time for use in macro expansion.
> (require racket/async-channel)
> (module->imports 'racket/async-channel)
'((0
#<module-path-index:(racket/base)>
#<module-path-index:(racket/contract/base)>
#<module-path-index:(racket/contract/combinator)>
#<module-path-index:(racket/generic)>))
Note that the module in question must be included into the current namespace in order for module->imports to work, which require or dynamic-require will both do.
This will inspect the information known by the compiler, so it will find all static imports for a particular module. However, the caveats about dynamic requires mentioned by John Clements still apply: those can be dynamically performed at runtime and therefore will not be detected by module->imports.
Short short version:
Have you tried turning on the module browser?
Short version:
You're going to need a macro for this, and
It's not going to be a complete solution
The existing require is not a function; it's a language form, implemented as a macro. This is because the compiler needs to collect the same information you do, and therefore the required modules must be known at compile time.
The right way to do this--as you suggest--is definitely to leverage the existing parsing. If you expand the module and then walk the resulting tree, you should be able to find
everything you need. The tree will be extremely large, but will contain (many instances of) a relatively small number of primitives, so writing this traversal shouldn't be too hard. There will however be a lot of fiddling involved in setting up namespace anchors etc. in order to get the expansion to happen in the first place.
Regarding your original idea: you can definitely create a macro that shadows require. You're going to want to define it in another file and rename it on the way out so that your macro can refer to the original require. Also, the require form has a bunch of interesting subforms, and coming up with a macro that tries to handle all of these subforms will be tricky. If you're looking at writing a macro, though, you're already thinking about an 80% solution, so maybe this won't bother you.
Finally: there are forms that perform dynamic module evaluation, and so you can't ever know for sure all of the modules that might be required, though you could potentially annotate these forms (or maybe shadow the dynamic module-loading function) to see these as they happen.
Also, it's worth mentioning that you may get more precise answers on the Racket mailing list.
In Python we have the del statement for deleting variables.
E.g:
a = 1
del a
What the equivalent of this in Lisp?
(setq foo 1)
;; (del foo) ?
In Common Lisp.
For symbols as variables:
CL-USER 7 > (setf foo 42)
42
CL-USER 8 > foo
42
CL-USER 9 > (makunbound 'foo)
FOO
CL-USER 10 > foo
Error: The variable FOO is unbound.
See:
MAKUNBOUND (defined)
SLOT-MAKUNBOUND (defined)
FMAKUNBOUND (defined)
Python names reside in namespaces, del removes a name from a namespace. Common Lisp has a different design, one much more sympathetic to compiling efficient code.
In common lisp we have two kinds of "variables." Lexical variables account for the majority of these. Lexical variables are analogous to a C local. At runtime a lexical variable usually is implemented as a bit of storage (on the stack say) and the association with it's name is retained only for debugging purposes. It doesn't really make any sense to talk about deleting lexical variables in the sense python uses because the closest analogy to python's namespace that exists for lexical variables is the lexical scope, and that purely an abstraction used by the spec and the compiler/evaluator.
The second variable kind of "variable" in CL are "global" symbols. Symbols are very rich data structures, much richer than a label in python. They can have lots of information associated with them, a value, a printed name, their "home" package, a function, and other arbitrary information stored in a list of properties. Most of these are optional. When you use a name in your source code, e.g. (+ X 3), that name X will usually denote a lexical symbol. But failing that the compiler/evaluator will assume you want the value of the "global" symbol. I.e. you effectively wrote (symbol-value 'X) rather than X. Because of typos, programming conventions, and other things some decades ago the compilers started complaining about references to "global" symbols in the absence of a declaration that signaled that the symbols was intended to be a "global." This declaration is known as "special." Yes it's a stupid bit of nomenclature. And worse, special variables aren't just global they also have a very useful feature known as dynamic binding - but that's another topic.
Symbols that are special are almost always declared using defvar, defparameter, or defconstant. There is a nearly mandatory coding convention that they are spelled uniquely, i.e. *X* rather than X. Some compilers, and most developers, will complain if you deviate from that convention.
Ok. So now we can get back to del. Special variables are denoted with a symbol; and this is analogous to how in python variables are denoted with a name. In python the names are looked up in the current namespace. In Common Lisp they are looked up in the current package. But when the lookup happens differs. In python it's done at runtime, since names can by dynamically added and removed as the program is running. In Common Lisp the names are looked up as the program is read from a file prior to compiling it. (There are exceptions but let's avoid thinking about those.)
You can remove a symbol from a package (see unintern). But this is an rare thing and is likely to just make your brain hurt. It is a simple operation but it get's confusing around the edges because the package system has a small dose of clever features which while very helpful take a bit of effort to become comfortable with. So, in a sense, the short answer to your question is that for global symbols the unintern is the analogous operation. But your probably doing something quite exceptional (and likely wrong) if your using that.
While what #Ben writes is true, my guess is that what you are looking for is makunbound, not unintern. The former does not remove the symbol from the obarray (for Emacs Lisp) or package (for Common Lisp). It just removes its symbol-value, that is, its value as a variable. If you want the behavior that trying to get the variable value results in a not-bound (aka void) error, then try makunbound.
I am not contradicting the previous answers but adding.
Consider that Lisp has garbage collection. As you know, you get a symbol defined in many ways: setf, defvar, defparameter, make-symbol, and a lot of other ways.
But how do you get a clean system? How do you make sure that you don't use a variable by mistake? For example, you defined abc, then later decided you don't want to use abc. Instead you want an abc-1 and an abc-2. And you want the Lisp system to signal error if you try to use abc. If you cannot somehow erase abc, then the system won't stop you by signalling "undefined" error.
The other answers basically tell you that Lisp does not provide a way to get rid of abc. You can use makunbound so that the symbol is fresh with no value assigned to it. And you can use unintern so that the symbol in not in any package. Yet boundp will tell you the symbol still exists.
So, I think that as long as no other symbol refers to abc, garbage collection will eventually get rid of abc. For example, (setf pt-to-abc abc). As long as pt-to-abc is still bound to the symbol abc, abc will continue to exist even with garbage collection.
The other way to really get rid of abc is to close Lisp and restart it. Doesn't seem so desirable. But I think that closing Lisp and starting fresh is what actually gets rid of all the symbols. Then you define the symbols you want.
You probably usually want makunbound, because then, the system will signal error if you try to add the value of abc to something else because abc won't have any value. And if you try to append abc to a string, the system will signal error because abc has no string. Etc.
This is I suspect, a matter of style and/or personal taste but I thought I'd ask anyway.
I have been in the habit of defining packages thus:
(defpackage :wibble
(:use :cl :drakma)
(:export :main))
Once I have executed IN-PACKAGE (:wibble, in this case), I can then use the symbols in DRAKMA unadorned:
(http-request ...
Then I recently read that seasoned Lisp hackers would rather not :use but:
(drakma:http-request ...
Just wondered what the consensus of opinion was on here and whether there were any pros or cons (not that type of CONS :) ) either way?
Cheers,
Peter
When you use a package, there are a couple subtle ways things might go wrong if the used package changes.
First, the package might export more symbols in the future. If, for example, the package exports a new symbol library:rhombus and you're already using that myapp::rhombus to name something, you are suddenly using the inherited symbol, with all possible attachments (e.g. classes, defuns, macros, etc), with sometimes strange results. If you use qualified symbol names, you will not get any more or any less than the symbols you want.
Second, the package might stop exporting symbols in the future. So if, for example, library:with-rhombus disappears, your call to (with-rhombus (42 42 42) ...) will suddenly get an error for an invalid function call (42 ...) rather than something that points directly to the source of the problem, the "missing" symbol. If you use qualified symbol names, you will get an error along the lines of Symbol WITH-RHOMBUS is not exported from the LIBRARY package which is clearer.
Importing symbols (with :import-from or :shadowing-import-from or import) is not without its own trouble. Importing works on any symbol, regardless of whether it's external or not. So it could be the case that the symbol is now library::rhombus, i.e. not intended for public consumption any more, but importing will still work with no errors.
Which option you use depends on your comfort level with the source package. Do you control it, and you will not make any conflicting changes without thorough testing? Go ahead and import or use to your heart's content. Otherwise, be careful about checking for unintended side-effects as library package interfaces change.
This is more a style issue, so it's impossible to categorize it in black and white, but here are the pros and cons:
Using package-qualified symbols.
Avoids symbol conflicts.
Allows to clearly distinguish foreign symbols.
Allows to easily search, replace, copy,... uses of a certain symbol from the external library (for refactoring, extracting the code to some other place etc.)
Makes code uglier, but only when library names are too long. (For example, I add a nickname re to cl-pprce, and now the code using it is even better, than w/o qualification: think re:scan)
Importing the whole package
Basically, the opposite of the previous case. But I tend to use it with utility libraries, because using qualified names often beats their whole purpose of making code more concise and clear :)
:import-from package symbol
This is one option you've forgotten to mention. I think it may be useful, when you use one or too very distinct symbols from a certain package very frequently. It also allows to import unexported symbols.
Good answers so far.
Another view is that a package and its symbols make up a language. If you think a symbol should be a part of this language, then you should make it available without the need to qualify it with another package - when programming in this language.
For example in the CLIM implementation there is a CLIM-LISP package which sets up the implementation language. It is a variant of the COMMON-LISP package. Then there are packages like CLIM-SYS (resources, processes, locks, ...), CLIM-UTILS (various utilities and extensions of Common Lisp) and CLIM itself. Now in a new package SILICA (an abstract window system) these four packages are used. The implementation of Silica thus is implemented in a language which is built as a union of two languages (the Common Lisp variant CLIM-LISP and the UI commands of CLIM) plus two utility packages which extend CLIM-LISP with some facilities.
In above example it makes sense to use the packages, since they are extending each other to form a new language and the implementation in that new package makes heavy use of those.
If you had a package which needs conflicting packages, then it would not make sense to use them. For example a package could use drawing commands tailored to a GUI and for Postscript output. They would have similar names. Using them both would lead to conflicts. You also want to make clear in the source code for the human reader from where these symbols are coming. Is it a line-drawing command from a postscript or a GTK+ library? Would be great if you can find it out easily - even though the function names are the same.
As a rule of thumb, I :use packages that extend the general language, but use qualified symbols for packages that have some special application. For example, I'd always :use alexandria, but refer fully qualified to symbols from Hunchentoot. When in doubt, I use qualified names.
I am byte-compiling a module. It gives me this warning:
Warning: cl package required at runtime
Why is this a warning? I am well aware that I am using the cl package. In fact there is a (require 'cl) statement in the module.
Is there something wrong with using the cl stuff?
If so, is there a list of published workarounds? The main things I use are mapcan and delete-duplicates.
The reason of this warning is a GNU policy which does not want a package cl to be used in Elisp. But it would be foolish as well to prohibit it completely. So they decided to show a warning.
You can find more information here
Just in case someone reads this on his quest for proper use of cl: The methods described here are now deprecated.
As least as of emacs 24, instead of cl you should use cl-lib or, if the macros suffice, cl-macs. These are new versions of cl that work with a clean namespace. E.g. instead of defun* you have cl-defun.
The old cl-package now is only for backward-compatibility and shouldn't be used in new code.
There are namespace clashes between Elisp and Common Lisp but the cl package gets round them by appending an asterisk to the repeated names. For instance it implements the Common Lisp version of defun but calls it defun*. The upshot is that there are no namespaces clashes between cl and Elisp and it is quite safe to (require 'cl).
If you want to get rid of the silly warning, customize the variable byte-compiler-warnings.[1] This will turn off the warning when you compile the code. If you distribute the code the warning will probably came back when someone else compiles it. If you don't want this to happen use the code:
(with-no-warnings
(require 'cl))
You can stop the byte compiler warning about any Lisp form in a similar way.[2] It's probably a not good idea in general, but you may be able to justify it in this case.
The code:
(eval-when-compile
(require 'cl))
will get rid of the warning, but you will only be able to use the macros from the package if you do this. Macros are evaluated at compile time and Elisp does not need to know about them at run time. If you only use the macros from any package, not just cl, then it is a good idea to use eval-when-compile as it will stop unnecessary packages loading at run time, both saving memory and making the code faster. But it seems to me that it's a misuse of the function to use it just to avoid a warning. And, of course, if you do want to use any of the functions from cl, you can't use eval-when-compile anyway.
[1] You may need to add (require 'bytecomp) to your .emacs file to get access to this variable.
[2] In theory, anyway, but there's a bug in with-no-warnings that means it doesn't supress some warnings about lexical variables.
Common Lisp has lots of namespace clashes with elisp, often the functions seem to do the same thing, but differ in some subtle detail. Mixing the two is a risk that is best not done behind the user's back. For this reason, most of the more useful functions in cl.el are defined as macros, so that cl.el can be required at compile time only, and the macros will then only affect the code that uses them in future sessions of Emacs.
I wasn't able to suppress this message after reading the comments before mine.
However, I received the following instruction from a kind person on the GNU emacs mailing list:
Require cl-lib, and then change the call to use cl-remove-if-not,
instead of remove-if-not.
Which proved to be the remedy.
In sum: by 'requiring cl-lib, one must also change the name of the function/macro call.
HTH....