Lisp allows one to recompile segments of code during debugging producing live stack frames. I know that lisp is dynamic and that it's easy to rebind a symbol associated to a method. However Lisp also easily allows to redefine single forms inside a function. This is even more powerful and flexible than what C# and Java debuggers can do. Why and how does it work? Why modern languages do not provide the same level of sophisitication? Where I can find some papers or books so to learn more?
It's not magic. I did the same in the old days in assembly. I created a DOS TSR program that pointed the interrupts I could handle to my own location. When you did a division by zero or something like that it jumped into my program and the stack had all the previous programs stack and where it had gone wrong. I just provided a menu with ways to mend it so that it didn't exit with DOS default behaviour.
In CL you have like a default exception handler that calls a function that you can see your options on restarts. Restarts are like code within the scope of where the error originates with ways to mend the situation. It's possible to make something like this in any language that has exceptions and closures. The debugger and restarts become a part of the running program and thus it becomes more safe on the cost of less performance. Here is a JavaScript implementation.
There is a way to turn it off and it might make your programs faster in CL. The standard does not require to do anything so it's optional, but implementations do make unsafe fast code that might end up as a segmentation fault instead of the debugger.
Related
I work on computational music. I have found the ps13 pitch spelling algorithm implemented in Lisp in 2003, precisely "Digitool MCL 4.3". I would like to run this code, preferably on a Linux x86 machine, to compare its results with other similar codes.
I am new to Lisp, but so far my research led me to think that Digitool MCL is no longer available. I thought of two ways which may help me:
a virtual environment (Docker or else) which would emulate a machine from 2003…
a code translation tool which would transform the 2003 source code into something executable today
I have not succeeded in finding one of these two options, nor running it directly with sbcl (but, as a newbie, I may have missed a small modification to make it run easily).
May someone help me?
Summary
This code is very close to being portable CL: you won't need something emulating an antique Mac to run it. I ran it on three implementations (SBCL, LispWorks, CCL) within a few minutes. However if you're not a Lisp person (and don't want to become one) it will be somewhat more fiddly to do that.
However I can't just send you a fixed version, both because this isn't the right forum for that, and also because we'd need to get the author's permission to do so. I have asked him if he would be interested in a portabalised version, and if he is, I will send him one in due course. You could also get in touch and ask to be notified.
(Meta-summary: while I think the question is fine, any reasonable answer probably doesn't fit on SO.)
Details
One initial problem with this code is that the file uses old Mac line end conventions (I think: not Unix anyway): unless whatever Lisp you're using is smart enough to spot this (some are, SBCL seems not to be although I am sure there are options to tell it) you'll need to convert it.
Given that, the code that implements this algorithm is very, very close to being portable Common Lisp. It has four dependencies on non-standard things:
two global variables, *save-local-symbols* and *verbose-eval-selection*;
two functions: choose-file-dialog and choose-directory-dialog.
The global variables can probably be safely commented out as I think they are just controls for the compiler, probably. The functions have fairly obvious specifications: they're obviously meant to pop up file / directory choosers.
However you can just not use the bits of the code that use these functions, so you can compile it, get a few compiler warnings about undefined functions, and then it's fine.
But it gets better than that in fact: the latter-day descendant of MCL is Clozure CL: CCL is free, and open source. CCL has both choose-file-dialog and choose-directory-dialog already and both of the globals exist although one is no longer exported.
Unfortunately there are then some hidden portability problems to do with assumptions about what pathnames look like as strings: it's making some assumption about what things looked like on pre-OSX Macs I think. This kind of problem is easy but often a bit fiddly to fix (I think in this case it would be easy). So, again, the answer to that is just not call the things that are doing a lot of pathname munging:
> (ps13-test-from-file-list (directory "~/Downloads/d/*.opnd"))
[... much output ...]
Total number of errors = 81.
Total number of notes = 41544.
Percentage correct = 99.81%
nil
Note that the above output came from LispWorks, not CCL: CCL works just as well though, as will any CL probably.
SBCL has one additional problem: the CL-USER package in SBCL already uses a package which exports int which is defined in this code. So you need to compile it in some other package. But given that, it's fine in SBCL as well.
Short version:
Is there a way to allow other programs to call Lisp functions of a Lisp program?
Long version:
I'm contemplating a graph database project :) Not to be started immediately, I'm only probing the ground so far. I've tried couple of graph databases, and my biggest gripe about them is they all are written in Java, (some are in C++, which isn't going to cut it either...). Java has no good way of communicating outwards. It may only be embedded inside another Java program. C++ is just hard to embed / I'm dubious that embedding was even planned.
So, I would obviously want to write it in CL, but I'm considering other options too. So, if you believe that CL simply won't do it, but you have reasons to believe that some other language will, then that's an interesting answer! My requirements to the "other language" would be that it must support parallel computing in some way. Obviously, high performance. And, as mentioned, extensibility.
I see multiple ways to call Lisp from other languages:
The simplest way that should work with all implementations would be to just maintain a bidirectional stream to the REPL. So you could send commands to the REPL and receive the REPL's response. One drawback of this would of course be that everything would be converted to strings.
You could mirror the way SLIME communicates with SWANK. In that case, you either use SWANK directly on the Lisp side and communicate through the same protocol SLIME uses, or write your own version of such a library.
Finally, there are Lisp implementations designed with embeddability in mind. I'm thinking particularly of Embeddabble Common Lisp (ECL) here which has a C API. For example, this section in the manual explains how to call functions, by getting hold of the function's symbol with ecl_make_symbol and then calling it with cl_funcall or cl_apply.
As alternatives to Common Lisp, other Lisp languages might be worthwhile to consider. Various Scheme implementations are designed to be embeddable, this is for example the documentation of Racket's C API. It seems you prefer the native code side of the runtime world over the JVM, but otherwise, Clojure is also interesting for being embeddable within Java.
For the host language there are few limits because most languages should support "pipes" (i.e. streams to other processes) or have a C FFI to call some Lisp's C API.
I wish that John McCarthy was still alive, but...
From LISP 1.5 Programmer's Manual :
LISP can interpret and execute programs written in the form of S-
expressions. Thus, like machine language, and unlike most other higher
level languages, it can be used to generate programs for further
execution.
I need more clarification about how machine language can used to generate programs and how Lisp can do it?
All that is saying is that machine code can directly write machine instructions to memory and jump to those instructions to execute them; this is the basis of many attack vectors to break into software, in fact.
The point is, when you're writing machine code, it's easy to generate machine code. But when you're writing in a compiled language like C, you can't just generate C code at run time and then execute it - unless your program includes a C compiler.
Lisp - and, these days, many other languages, especially "scripting languages" like Perl, Python, Ruby, Tcl, Javascript, and command shells - have the ability to execute code that is generated at runtime. In Lisp, since code and data have the same structure, this is usually less work than it is in the other languages, where the code to be evaluated is generally a string that has to be parsed. (Though Perl has the ability to eval a block instead of a string, which lets the compiler do the parsing ahead of time for literal code.)
A machine language can alter itself while running. The last assembly programming i did was for MS DOS and resident program that i used to run before testing other programs. When my program misbehaved, a keystroke switched to the resident program and could peek into the running program and alter it directly before resuming. It was quite handy since I didn't have a debugger.
LISP had this from the very beginning since it was originally interpreted. You could change the definition of a function while you were running and the whole langugage was always available at runtime, even eval and define. When it started getting compiled it wasn't compiled like Algol, but partially, allowing for interpreted and compiled code to intermix at the same time. The fact that its code structure was list structure and that symbols are a data type contributed to this.
Last interview I saw with McCarthy he was asked about what he thought of modern programming languages (Not LISP family but the Algol family language Ruby, that is said to be influenced by LISP), and before answering he asked if they could represent code as data (like list structure). Since it didn't, Ruby is still behind what LISP was in the 60s in his opinion.
Many new programming languages are emerging in the Algol family and some of the most promising ones, like Perl6 and Nemerle, are getting closer to the features LISP had in the 60s.
Machine language programs can fill memory regions with arbitrary bytes. Then they can just jump to the start of such region which will thus get executed right away.
Lisp language programs can easily create arbitrary S-expressions in memory, using cons. Then they can just call eval on these S-expressions to evaluate (interpret) them.
High level languages programs can easily fill memory regions with characters representing new code in the language's syntax. But they can not run such a code.
I'm an experienced C++/.NET/Java Windows/web programmer trying to learn (Common) Lisp. I'm reading Practical Common Lisp and using SLIME.
I'm getting the Lisp language easily enough, but I'm having trouble groking the mechanics of development. One of my issues is dealing with Emacs. I have no experience with it and find it generally confusing with hard to find/figure out commands.
Specific questions:
I get the REPL, but I don't quite get how I can use it effectively. When I need to change a function I have to retype the defun and make changes (tedious and error prone). How can I do this better?
How do I get from entering code at the REPL to actually having a program? I'm used to the C model where you have code files that you can review, edit and recompile. I know Lisp has something similar (via the load function), but how does one get a function I just defined and tested into a file to be saved? I'm hoping there's a better method than my current select+copy+paste.
How do you debug errors? Is there any ability to step into code like I would with other languages?
As long as the S-expression is valid, I don't get any errors when entering a (defun ...). It's not until I try to use it that I find out it's not valid. Is there any way to get errors sooner (i.e. compile the code)?
Is there some IDE that would be more familiar to me or allow me to play with the language easier?
Should I switch to learning Scheme instead?
Any tips would be greatly appreciated!
-I get the REPL, but don't quite get how I can use it effectively. When I
need to change a function I have to
retype the defun and make changes
(tedious and error prone). How can I
do this better?
-How do I get from entering code at the REPL to actually having a program?
I'm used to the C model where you have
code files that you can review, edit
and recompile. I know lisp has
something similar (via the load
function), but how does one get a
function I just defined and tested
into a file to be saved? I'm hoping
there's a better method than my
current select+copy+paste.
Load SLIME. Enter code in your .lisp file, and then run slime-eval-buffer to load all your code into Lisp. Then, for a specific function you are hacking on C-e, C-j to redefine it in your running Lisp.
-How do you debug errors? Is there any ability to step into code like I would with other languages?
Yes. (trace 'my-function) - or some variant on that - and when my-function is called, it will break into the debugger.
-As long as the S-expression is valid, I don't get any errors when entering a
(defun ...). It's not until I try to
use it that I find out it's not valid.
Is there any way to get errors sooner
(i.e. compile the code)?
To some degree, that is a characteristic of dynamic languages (Python, Perl, Ruby, Lisp, etc.). Try SBCL for better error-checking.
-Is there some IDE that would be more familiar to me or allow me to play with the language easier?
Emacs is the free IDE for Lisp. Allegro has a free edition I believe; I've never tried it though..
-Should I switch to learning Scheme instead?
Nah, Scheme is not as pragmatic a language.
I'm an experienced C++/.NET/Java Windows/Web programmer trying to learn (Common) Lisp. I'm reading Practical Common Lisp and using SLIME.
One can also use the LispWorks Personal Edition for learning Lisp. It has some limitations and the full product is commercial, but it is quite a bit easier to use.
I get the REPL, but don't quite get how I can use it effectively. When I need to change a function I have to retype the defun and make changes (tedious and error prone). How can I do this better?
The REPL has a history. With keyboard commands you can get back prior input and change it. Other than that just edit a Lisp file and compile code from there. In Lisp you can compile/eval individual expressions and definitions. Typical IDEs like SLIME, LispWorks or Allegro CL allow you to run code also from normal Lisp text windows - additionally to executing expressions in the REPLA (aka Listener).
How do I get from entering code at the REPL to actually having a program? I'm used to the C model where you have code files that you can review, edit and recompile. I know Lisp has something similar (via the load function), but how does one get a function I just defined and tested into a file to be saved? I'm hoping there's a better method than my current select+copy+paste.
Copy and paste in one thing. But the correct way is to work from a text file in an editor window. One can compile/load expressions, the editor buffer or the associated file.
How do you debug errors? Is there any ability to step into code like I would with other languages?
See STEP, TRACE and related. SLIME, LispWorks and Allegro CL have lots of additional features.
As long as the S-expression is valid, I don't get any errors when entering a (defun ...). It's not until I try to use it that I find out it's not valid. Is there any way to get errors sooner (i.e. compile the code)?
For many cases one uses a compiler. The compiler will find a range of errors and also will note when something is unusual (for example a function does not exist or a variable has not been defined).
-Is there some IDE that would be more familiar to me or allow me to play with the language easier?
LispWorks, Allegro CL are the best under Windows. There are some alternatives like Corman Lisp (I don't know it is maintained right now) or even Ufasoft Lisp.
Should I switch to learning Scheme instead?
Not really.
It doesn't sound like you're really using SLIME, or at least not in the way it was intended to be used. ("have to retype the defun", "the C model where you have code files")
I recommend watching some SLIME screencasts (or, even better, watching a Lisp programmer use SLIME for a few minutes, if you have one handy). The SLIME webpage has a couple.
It sounds like you'd really enjoy the DrRacket IDE. Racket is closer to Scheme than to Common Lisp, but you could dip your toes into the Lisp family without the speed bump of the Emacs style of development.
I need to find out, from within a Common Lisp program, how much memory is currently being used.
I'm given to understand there is no portable method (the standard function room prints the information to standard output in text form instead of returning it as a value), but sb-kernel:dynamic-usage works in SBCL.
What are the equivalents in other Common Lisp implementations? Or is there another way to solve this problem I should be looking at?
It may not help you much, but anyway:
You can capture the output of (room) and parse it.
(with-output-to-string (*standard-output*)
(room))
Above returns a string with the output of ROOM.
Additionally it may help to request the memory size of the process via an external call to a standard unix command (if you are on Unix).
For things which virtually every implementation supports, but not in the same way (because it's not in CL), one common approach is to make a library called trivial-whatever.
If you started a package like trivial-memory, and supplied the first implementation, I'm sure we could get everybody to contribute the function for their own favorite Lisp compiler in short order. :-)