What is SBCL's pathname wildcard conventions on macOS, Linux, FreeBSD? - lisp

Question

Trying out Lisp today on my Mac, I found the following a bit disorienting:
$ sbcl
This is SBCL 1.4.14, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
* (directory "*")
NIL
* (directory "*.*")
(#P"/private/tmp/sbcl/quicklisp.lisp" #P"/private/tmp/sbcl/test1.lisp"
#P"/private/tmp/sbcl/test2.lisp")
On macOS, Linux, FreeBSD, probably most other UNIX-likes, * represents all files. I've only ever seen *.* used on DOS and Windows.
Per the directory description, wildcards are implementation dependent. So, then, what exactly is the convention used by SBCL on the above operating systems? How do I find this out? SBCL's manual doesn't describe this. Could it be different even between the three OS' above?

Answer 1

The SBCL sources have this comment:
;;; FIXME: the below shouldn't really be here, but in documentation
;;; (chapter 19 makes a lot of requirements for documenting
;;; implementation-dependent decisions), but anyway it's probably not
;;; what we currently do.
;;;
;;; Unix namestrings have the following format:
;;;
;;; namestring := [ directory ] [ file [ type [ version ]]]
;;; directory := [ "/" ] { file "/" }*
;;; file := [^/]*
;;; type := "." [^/.]*
;;; version := "." ([0-9]+ | "*")
;;;
;;; Note: this grammar is ambiguous. The string foo.bar.5 can be
;;; parsed as either just the file specified or as specifying the
;;; file, type, and version. Therefore, we use the following rules
;;; when confronted with an ambiguous file.type.version string:
;;;
;;; - If the first character is a dot, it's part of the file. It is not
;;; considered a dot in the following rules.
;;;
;;; - Otherwise, the last dot separates the file and the type.
;;;
;;; Wildcard characters:
;;;
;;; If the directory, file, type components contain any of the
;;; following characters, it is considered part of a wildcard pattern
;;; and has the following meaning.
;;;
;;; ? - matches any one character
;;; * - matches any zero or more characters.
;;; [abc] - matches any of a, b, or c.
;;; {str1,str2,...,strn} - matches any of str1, str2, ..., or strn.
;;; (FIXME: no it doesn't)
;;;
;;; Any of these special characters can be preceded by an escape
;;; character to cause it to be treated as a regular character.
It says "but anyway it's probably not what we currently do", but I'm not sure if that is meant to mean that this comment may be inaccurate for current versions, or something else. I'm assuming at least the wildcard part is correct enough.
The important part here being that the directory, file and type components are all considered separately, so you must have a separate wildcard for both the filename and the extension (aka. the type component) with a dot to separate them.
The comment calls these "Unix namestrings", but I assume this applies to all platforms.

Answer 2

The following works, and I think should work in most implementations (I have tried this on LW on MacOS, SBCL on Linux and Clojure on MacOS).
(defun directory-pathname (p)
;; Everything but the file part: this should be called
;; non-file-pathname or something
(make-pathname :name nil :type nil :version nil
:defaults (pathname p)))
(defconstant +wild-file+
;; a completely wildcarded filename with no other components
(make-pathname :name ':wild
:type ':wild
:version ':wild))
(defun wild-directory (d)
;; Take the directory parts of D and return a pathname which matches
;; all files in tat directory.
;;
;; Actually we could just use D for the defaults since it's only
;; used for defaults and that's less consy, but never mind
(merge-pathnames +wild-file+ (directory-pathname d)))
Then, for instance (directory (wild-directory "/tmp/")) will return everything in the /tmp/ directory, and (directory (wild-directory (user-homedir-pathname))) will return everything in your home directory.
However note that (this is clearly very implementation-dependent but it looks like all the implementations I use agree on this and I agree with their interpretation): (directory (wild-directory "/tmp")) will return everything in /, because "/tmp" as a pathname means 'the file called tmp in the directory called /'. As I said I think this is a reasonable interpretation: pathnames which refer to directories end in /.
Also note that +wild-file+ really is wild: it doesn't treat files whose names begin with . in any special way. I think there is fairly obvious ambiguity about how filenames like /foo/bar/.x should be parsed, although all three implementations I've tried take what I think is the right interpretation: the name is ".x" and the type is nil (/foo/bar/.x.y seems to parse with a type of "y" which I think is also reasonable).
In any case where you need to do any serious pathname manipulation the trick is to define a bunch of wrapper functions and constants which you then use, and behind which you can hide implementation-dependence (in this case the implementation-dependence is how to get all the files in a directory, everything else seems to be common between the implementations).
I suspect such an abstraction layer exists, but I don't know what it is unfortunately.
After writing this I tried CLISP, and it does something interestingly different: it looks like directory doesn't return subdirectory names, just filenames. That's a perfectly reasonable implementation I think, but it would require some more abstraction.