Is it common practice to simply use the load function at the top of lisp code files to bring in functions defined elsewhere, similar to the #include directive in C/C++?
i.e.
(load "math_functions.lisp")
(load "string_processing.lisp")
Or are there other more common ways to share functions between files?
Of course, you can just load files into your "master" file, but the more common and actually convenient way to go is to use the ASDF facility.
They way I do this is together with Quicklisp to create a directory for your project in the ~/quicklisp/local-projects directory and there put the .asd file together with sources.
You might also find this guide to Quickproject useful.
Usually libraries use some tools that allows you to compile and load the library correctly. If the library is quite big, then it's hard to load it without knowledge of the internal dependencies. One of such tools is ASDF(asdf article on wikipedia), that is the standard for Common Lisp.
Also if you develop some package with several files you can use asdf to describe dependencies and relations between files.
Related
I've been diving into some of the more advanced features of powershell modules and manifests recently, with a view to handling scenarios more advanced than just a basic export of a few functions. It sounds like it should be obvious, but I'm struggling to find a nice solution for sharing common 'helper' type functions across several large non trivial modules. In particular, I'm looking for a solution that:
Allows sharing of 'helper' type functions without necessarily being exported by anyone
Allow installation via PsGet from a local repo path
Let me go into some of the challenges I see.
First of all, as far as I can tell, PsGet does not handle module dependencies well. This implies sharing between modules is going to be a struggle. Maybe a solution to this is to avoid PsGet, and use a custom script to 'install' modules to the local module path, which might be more tolerant of dependencies and load order.
My point about not using module exports to share helper functions also seems to be an issue. The reason I can see for this is desiring aliases, helpers etc for common internal actions (needed inside useful functions), that are either useless or unsafe to expose. For example, a nice brief alias for getting the local script path (commonly used, noisier than it should be). Or I recently made a nice simple wrapper around PromptForChoice with fewer options. Maybe this whole thing isn't a real issue. But I can't help but feel that shipping a 'utils' module that exports low level functions that are useful inside real modules, but not to an end user, seems like the wrong way to go.
What I've been playing with is a small build structure that tests and then packs modules, and I want to get some code sharing possible. I've been looking for an alternative using ScriptsToProcess in the manifest, but these seem to be absolute paths, not relative.
Imagine a folder structure:
modules
utils
console_helpers.ps1
moduleA
moduleA.psm1
moduleA.psd1
moduleB
moduleB.psm1
moduleB.psd1
packed_modules
moduleA.zip
moduleB.zip
What I was considering was that you could list relative paths in each ScriptsToProcess, and then my pack phase will go and drag those relative paths in to each zip.
Is this a horrible crazy idea? Am I right that ps modules and PsGet really don't have decent dependency support? I would love to hear feedback from anyone who has looked into this space. I think the answer I'm hoping to get in rough priority might be:
Here's an example of sharing code without exposing it (probably a build/pack level solution)
Here's how to make module dependencies work nicely, using PsGet
Here's how to make module dependencies work nicely, but you can't use PsGet
Just expose everything from modules
This is a terrible idea and you're terrible
Thanks!
UPDATE as suggested by CalebB
Here's another example to illustrate what I'm trying to resolve. I find it useful to wrap up '&' style execution of commands with a wrapper function, to deal with stuff like checking exit codes etc. If i'm building half a dozen modules, many of them will want to make use of that helper (obviously).
My options today seem to be put it in a module and export it, but maybe I don't want it exported, I want more of a . source style access. And if I've got a family of modules all trying to use this stuff, the options for module dependency management are limited (PsGet limitation etc).
If I'm 'building' all the modules at once (with some decent psake and pester infrastructure), maybe I can use a hack at this point to embed scripts into my zipped modules to 'solve' all these problems?
Allows sharing of 'helper' type functions without necessarily being exported by anyone
Mhm... what is wrong with dot sourcing the scripts you need within particular module ? You could :
Keep your folder structure and symlink the desired functions into module folder.
Try to use AbsolutePath with ScriptProcess that has "relative part" in it, for example %PSScriptRoot%\..\utils (not tried in that context but generally works). If not, u can always add preprocessor to fix paths for you if it doesn't work
Delete undesired imported elements manually via function:, alias:, and var: provider.
Import extra utilities only when u use them then remove them at the end ? If the desire is that user can't see them you can encrypt them.
Here's how to make module dependencies work nicely, but you can't use PsGet
Chocolatey uses NuGet so it handles dependencies and can load from the local store. As a benefit, OneGet supports it which is something everybody will use eventually.
I've posted the solution I've come up with on github. I've rolled in a few other features I want when building modules, but the key solution for this question here uses reading and updating the psd1 of each module.
You include scripts that you want to embed in the NestedModules property of your manifest. My build phase will find each script and copy it into the module folder for packing and zipping. The manifest that ships in the package has the script paths converted to the now local file name.
I'm still not sure of this is ideal, but it seems to be a nice compromise to deal with the issues here.
A key issue I encountered along the way was that the ScriptsToProcess list is executed literally at module import time, so it is only useful for bootstrapping the import of your functionality. The NestedModules property is actually the list of additional scripts you want to be . sourced and available when your module is used.
I have been a vim user for years now and recently started to try out emacs. There are a lot of components available for this editor, like mail clients, IRC clients, etc. All these components need a package manager to update them and to make handling of them easier in general.
On vim there is a plethora of plugin managers (like Vundle or pathogen) and different people use different things.
On the other side, emacs includes package.el with recent versions (which is great to begin with), but there is also el-get and the package.el package list is pretty short.
Should I stick to package.el (and maybe extend the package list with marmalade)? Or do I need el-get for sane package management?
If you use a recent Emacs version then go with package.el. And add MELPA to the repository list.
Also, it is not the case that all "packages" (or libraries) need a package manager to update them and to make handling of them easier in general. There is a world of useful Emacs Lisp code out there that is not "packaged" and uploaded to a package.el repository.
More importantly, regardless of whether you use package.el to retrieve and "install" packages, you should learn the basics of loading, byte-compiling, and using Emacs Lisp code. Don't just use package.el blindly, without understanding something about Lisp code.
The basics of "installing" Lisp code manually include the following:
In general, you will want to byte-compile libraries that you use, for better performance. Keep in mind this caveat, if you do so:
1a. Load any source files (*.el) that define Lisp macros that are needed by any of the other Lisp libraries you use.
1b. Then byte-compile (the macro-defining libraries and) the libraries that use the macro-defining libraries. This is important. It is likely that the latest versions of the macro-using libraries need the latest versions of the macros, and if you compile them without having first loaded the latest macro definitions then you are asking for trouble.
Put any libraries you use in a directory that is in your load-path. Said differently: set variable load-path in your init file (~/.emacs) to a value that includes all directories containing Lisp code that you use.
Generally, you want to use (require 'foo) in your own code (e.g. in your init file), to load library foo.
However, #3 works only if library foo (file foo.el) actually provides feature foo: (provide 'foo). If it does not, you can use (load-library "foo") instead.
(Note the difference here between using 'foo and "foo": the first refers to feature foo; the second refers to files foo.el and foo.elc.
Finally, before you try to use a library, do yourself a favor and take a look at the Commentary section of the file header. Often it tells you important things about using the library. And even when it does not say something terribly important it might say something useful (e.g. a tip or two).
I mention 1-5 because I see too many (no, not many, but too many) new users just "installing" stuff using a package manager and never taking a look at what the code they are installing is all about: what it is for, how to use it, etc. If there is something in the Commentary then read it; the library author put it there to help you.
Beyond that, have fun!
I have developed some Common Lisp functions in a couple of Lisp source files that I'd like easily available to other functions I write, or make available on github if I think they'd be useful for someone else. For now, I've just been putting them in some pre-defined folder and using (require "/path/to/my/modules/module.lisp").
I'm wanting to understand what is the correct (canonical Lisp) way of defining Lisp library of modules. And the second part of the question is how to use them (whether I've defined them, or whether I've obtained one from someone else).
I've been reading a lot about defpackage and defsystem and asdf. But everything I've read seems to focus on some specific corner of the universe of this task. I'm having a lot of trouble seeing the big picture of custom module creation, deployment, and use. So assuming I have the Lisp environment in front of me (CLISP or SBCL) and one or more .lisp files I'd like to make a package or library out of, is there a document somewhere that explains what steps are required to do that? It's probably something I've already read, but didn't track due to not understanding the context. What I've read about ASDF seems functionally to be what I'm after, but I'm left not understanding whether ASDF is my only option, or whether it just happens to be a de facto standard and what most other people use, or whatever. I played with it a bit in SBCL and wasn't sure I was using it right, and didn't see info on how to set it up in CLISP. So I'm wanting to understand what is the up-the-middle, vanilla approach to this task.
I know this is a big, sloppy set of sub-questions. Again, if there are some good references to look at, I can read. I'm just having some trouble getting a big picture view of how this is supposed to work, and whether there is any "best" approach, or whether, in Lisp, it's a bit of a "Wild West" choose-the-library-manager-you like approach. I did the Google thing and read anything that looked relevant, but my brain is spinning from all of it.
Thanks.
A system is a collection of files and sub-systems. One can compile or load such a system. There are also other operations possible. It keeps track of dependencies and tries to do a minimal amount of work.
If you are using SBCL and CLISP, then ASDF is the tool to choose. See http://www.cliki.net/asdf
ASDF provides, amongst other things, a DEFSYSTEM macro to describe such systems.
Don't use PROVIDE/ REQUIRE- unless you know what you are doing. ASDF is the way to go.
To publish your code and make it easily loadable by others then use QUICKLISP. See: http://www.quicklisp.org/beta/
A package in Common Lisp is not like a package in most other senses. It's not an archive of list files, but more like what most other languages would call a module or namespace that lets you select which symbols (names) from your code you want to show to the outside world of code.
If you just have one file for your little library, you can just distribute that. If you have multiple files, that where a tool like ASDF comes in to make sure, for example, that files defining macros are loaded before the files that use those macros.
Here are some good resources for you to look at, both chapters from Practical Common Lisp:
21. Programming in the Large: Packages and Symbols: This will give you a much better sense of what packages are and how and why to use them.
32. Conclusion: What's Next?: The "Delivering Applications" section has some good resources on this stuff in general, and has some mention of ASDF.
ASDF and Quicklisp are useful tools, that is an established fact. However, I would like to give an alternative point of view on the concept of "library" as it is discussed on previous answers
ASDF is designed to automate the compilation and loading of a set of source files. It is to CL what make is to Unix. It is perfectly valid to write and distribute a program without make just as it is fine to write a program in CL that does not use ASDF.
If your project is simple enough, it is sufficient to provide a file (e.g. load.lisp) that contains the statements to load the dependent files of your project in the right order. Therefore ASDF is not involved in the concept of a library.
In a provocative way, I would say that the canonical way of defining a module in CL is to use the defpackage declaration because it is the language unit that allows a programmer to isolate his/her declarations from those of someone else.
Then is the question of how to make it available to others. If you write portable CL code, then ASDF is the most popular system definition facility and you should use it. If you want to make it easier for others to obtain, then Quicklisp is the tool that changed the face of CL in the last few years.
Finally I would like to add that neither ASDF nor Quicklisp are standards, they are tools (which does not remove to their usefulness). ANSI Common Lisp is a standard and I would love to see a system definition standardized in CL.
Yes, ASDF is a de facto standard, and Quicklisp is another standard.
From Lisp's viewpoint what you want is to define one or several namespaces (packages). This is regulated by the ANSI standard. From your code's viewpoint you want to arrange a bunch of files so that they became a whole and somehow provide that packages. This is where ASDF plugs in. And Quicklisp allows you to manage ASDF systems in the easiest way concievable. You can both download a lot of libraries from Quicklisp's repository and manage your local systems creating symlinks in the quicklisp/local-project folder.
If you have Quicklisp installed, you can type
(ql:quickload :cl-fad)
at the REPL and thus load the CL-FAD library (possibly downloading it); then the packages CL-FAD and PATH become available, you forget about ASDF-systems and stick with the logic of packages.
A good idea would be to take a look at asd-files of several projects downloaded with Quicklisp.
I setup my own libraries, that I use often, using ASDF system definitions and linking them into the local-projects (or similar named) folder of Quicklisp. That allows me to load them using quicklisp like any published package.
If you want to learn how to setup such an ASDF system, I would suggest installing Quicklisp (which for once is really easy, when it comes to installing 'unfinished' software from the internet), quickload a well known package or two and look at its .asd file, consulting the documentation when in doubt.
This way, you have your libraries already setup to be published like most of the well known CommonLisp packages out there.
I'd like to use emacs to work on my project that is built using CMake, while this generally works fine, I'd like to implement better project management commands. Is there a simple way to generate some sort of file that acts as a listing of the project files.
It seems that the best way may just be some set of CMake macros that do a custom write to a file, is there perhaps any better solutions?
I have no direct experience with CMake. But there are a couple of approaches to solving this.
The canonical way is to generate a TAGS table as a part of your buid process. You will get symbol completion/navigation on top of easy access to file-list. And ctags is hyper fast. I'll leave you to google how to do that specifically, hint: wiki.
Alternatively, you can get a Emacs project management package like EDE, eproject, mk-project that defines the concept of a project. See wiki.
You can look onto CEDET mailing list - CMake support was discussed not so long time ago, and at least one person is actively working on CMake support in EDE (CEDET's project management)
Here is the back story skip to the bottom if you do not care and only want to see the question.
So I have been playing around in LISP for a little while. Some basic functions, some classes ,and file IO. When I run across this article:
http://www.adampetersen.se/articles/lispweb.htm
And I am excited to try and use lisp for a web application. I go and download the packages, but for the life of me do not know how to load them into my Allegro IDE.
Hmm... ok, well the hunchentoot site says a lot of the basic packages are in LispWorks. So I download that. Still not sure how to get the source for the packages that I downloaded into these IDEs. They seem to have binaries of the packages?
Oh well maybe ill switch to my ubuntu server and apt-get all the packages and setup slime (i have not used it before because I just wanted to learn lisp. Learning emacs and lisp at the same time seemed real daunting). I apt get all the packages needed and load up slime and again same problem there aren't available.
I dig around some more and see this program called ASDF. It looks like ASDF is some kind of package builder for lisp? I don't know it seems confusing. I'm about to give up at this point.
If you are still reading this here is my question.
1. How do I load the source for these packages into my lisp environment. trying to learn lisp hasn't been too hard but the information about the environments has been sparse. Do I need to build the packages I download with ASDF.
2. Is there a simple way for someone to just get up and running in lisp without having to speed a large amount of time upfront learning all the tools?
Hmm... ok, well the hunchentoot site
says a lot of the basic packages are
in LispWorks. So I download that.
This just means that the author has written a lot of Lispworks-specific code in Hunchentoot. It does not mean that Hunchentoot only works on Lispworks.
Still not sure how to get the source for the packages that I downloaded into these IDEs.
You need to use ASDF.
They seem to have binaries of the packages?
That's unlikely.
Oh well maybe ill switch to my ubuntu server and apt-get all the packages and setup slime > (i have not used it before because I just wanted to learn lisp. Learning emacs and lisp
at the same time seemed real daunting).
Don't do it then. You don't need to use Emacs or Slime.
I apt get all the packages needed and load up slime and again same problem there aren't
available.
For quick results try clbuild: http://common-lisp.net/project/clbuild/
I dig around some more and see this program called ASDF. It looks like ASDF is some
kind of package builder for lisp? I don't know it seems confusing.
ASDF is a bit like a Makefile for Common Lisp applications.
I'm about to give up at this point.
That's about the worst thing you could so (at this or any other point). I'm glad you have decided to post your problems here instead.
How do I load the source for these packages into my lisp environment.
trying to learn lisp hasn't been too hard but the information about the
environments has been sparse. Do I need to build the packages I download with ASDF.
clbuild should give you all you need, but here are some hints if you don't want to use it:
CLISP, SBCL: ASDF is part of your Lisp. Run (require :asdf). Lispworks, Allegro: you need to download and load ASDF. Save asdf.lisp somewhere then run (load "/path/to/asdf.lisp").
For every library/application ("system" in ASDF speak) you need to download und unpack it to some place. Repeat until all dependencies are satisfied. Note down these places (directories).
For every place from step #2 add the place to the ASDF registry: (push "/path/to/dir/" asdf:*central-registry*). Don't forget the trailing slash.
Load the system using (asdf:oos 'asdf:load-op :system-name).
Is there a simple way for someone to just get up and running in lisp without having to speed a large amount of time upfront learning all the tools?
See above -- use clbuild.
The quickest way in Ubuntu is to use the packages included in that distribution. It is "ok" if you just want to try some things, but these versions are often comparatively old. I would recommend the packages sbcl and slime. If you don't know emacs yet, you can get into that quite fast through its built-in tutorial (C-h t (press Control-h, release, then press t)).
You can then start emacs, start slime (through M-x slime), open a lisp file (C-x C-f ~/lisp/first-try.lisp), and you're ready to go. As a tutorial for Lisp, I think that Practical Common Lisp is a very nice book, and it's freely available.
Now, when you have come to like Lisp, you might want more up to date packages. I would recommend to use clbuild for that (see the link for further information, including FAQ). You can then also build a new sbcl (bootstrapped by the distribution's version).
ASDF, by the way, is only a system definition facility. It doesn't know how to download packages, it only knows how to load systems into a running Lisp image. In other words, it just solves the problem of automatically loading the multiple files that some "system" (library) consists of in the right order. The newest versions allow loading a package (after it is installed, e.g. through clbuild) with a simple
(asdf:load-sys 'foo)
Older versions show ASDF's internal concept of operations:
(asdf:operate 'asdf:load-op 'foo)
The above load-sys is a shorthand for this common use case. Further information (one could say, all you need to know about it) is at the ASDF Getting Started guide. ASDF is also included in SBCL.
When you load a source file, it is automatically compiled (producing .fasl files (fast-load)) so that loading is much faster next time.
Probably one of the fastest ways to get started is to use Lisp in a Box (or a spinoff like LispBox). These are full sets of everything you need.
You could also try the Lisp Resource Kit, which is a bootable CDROM with Lisp tools and documentation, all already set up for you. Just put it into your CDROM drive and boot!
All of these answers are good, however they've become a little outdated with the new popularity of Quicklisp. Very loosely speaking, quicklisp is the package manager to asdf's make. Once Quicklisp is installed on a system, you can use (ql:quickload "name of lisp library") to load that library into your lisp environment, including downloading it and any of its dependencies if required. For example, to download, install, and load Hunchentoot and all of its dependencies, use (ql:quickload "hunchentoot"). In later lisp sessions, calling (ql:quickload "hunchentoot") again will simply load the version already downloaded and installed, making ql:quickload a simple way to load any library available locally or remotely. To install Quicklisp, I refer you to quicklisp.org.
More detailed explanation
Lisp works a little differently from other languages when it comes to libraries. The first thing to know is that the language itself provides almost no library functionality- it has load (which goes through and runs each line in a file as if you'd typed them at the REPL) and compile-file (which creates a "fast load" file, a precompiled version of the file which loads and can perform much faster). Using just what the core language provides, in order to load a library you have to go to each of its files and load it (or (load (compile-file "filename")) it, for better load speed/performance). This got tedious, so a variety of libraries were created for managing the loading of libraries, and at the moment asdf is king (so much so that many lisp implementations actually bundle it). In asdf terminology, libraries are called "systems" and .asdf files describe all of the metadata involved in loading them- the systems they depend on and what order to load files in, mostly, but they still can get quite complex. Quicklisp, then, sits on top of asdf. Basically, when asdf discovers that it cannot locate a system, Quicklisp steps in and checks to see if that system is available from one of Quicklisp's online repositories, and if so downloads it and has asdf continue on its way. ql:quickload is basically just a wrapper around the asdf machinery for loading a system that enables quicklisp to help out as needed.