In lisp how can I measure and capture the time spent evaluating an expression? - lisp

I want to capture the results of a call to the time macro in order to gather multiple measurements and process them. I tried to locally setf the standard output and redirect it to a string but it didn't work with the time macro. Maybe it is wrong, but what I tried is:
(with-output-to-string (str)
(let ((*standard-output* str))
(time (test-with-size 40))))
The questions:
Is a way to capture the output of time?
If not, can I capture the slime-profile-report command's output?
If none of the above works, how can I measure the time spent evaluating an arbitrary expression?
What I want to accomplish is to measure the run-time of an algorithm as the size of the input increases so for each input size (ranging from 1 to 100) I will measure a lot of times and keep the average. Then I want to plot the results. Plotting is easy, and I have found many ways in Cliki, but how can I gather the results?
I am using CLISP and CCL.
EDIT: Paul Nathan pointed that the time macro outputs to *trace-output* which is a solution. I would like a nicer, simpler, solution though, because with this one, I have to parse an implementation specific trace.

If you want to capture the text, you need to use the right stream. The ANSI CL standard says that TIME prints to trace output.
So this would give you the text as a string.
(with-output-to-string (*trace-output*)
(time (sin 40.0)))
You could also write your own macro using the time primitives. See 25.1.4.3 Internal Time to get numeric data.

Related

Emacs: nesting exceeds `max-lisp-eval-depth'

From time to time I get a "nesting exceeds `max-lisp-eval-depth'" error.
What does it mean?
When I get one, is there something I can do, other than "killall emacs"?
Edit:
You can get the error if you evaluate:
(defun func ()
(func))
(func)
However, in this case emacs remains responsive.
An immediate remedy can be to simply increase the maximum. Its default value is 500, but you could set it to, say, 10000 like this:
(setq max-lisp-eval-depth 10000)
But that's generally not a great idea, because the fact that you run into a nesting exceeds `max-lisp-eval-depth' error in the first place is a sign that some part of your code is taking up too much stack space. But at least increasing the maximum temporarily can help you analyze the problem without getting the same error message over and over again.
Basically, it means that some Lisp code used up more stack than Emacs was compiled to permit.
In practice, it's a sign of a bug in Lisp code. Correctly written code should avoid nesting this deeply, even if the algorithm and the input data were "correct"; but more frequently, it happens because of an unhandled corner case or unexpected input.
In other words, you have probably created an endless loop via recusion, or perhaps e.g. a regular expression with exponential backtracking.
If you are lucky, repeated control-G keypresses could get you out of the conundrum without killing Emacs.
If you are developing Emacs Lisp code, you might want to tweak down the value of max-lisp-eval-depth artificially to help find spots where your code might need hardening or bug fixing. And of course, having debug-on-error set to t should help by showing you a backtrace of the stack.

Refining key-chord.el triggering

I do really like key-chord.el but I'd need it to only trigger when I start pressing keystrokes after a short delay during which I didn't do anything elapsed. I'd like to know if it's easy to modify key-chord.el to do that.
Why I want that is simple and I'll try to explain it as easily as I can:
I do want keychords to be assigned to keys that are on my "strong" fingers (no pinky) and that are on my home row (I do touch-type). Hence I'm potentially creating quite a few keychords which may clash with common words / code when I'm typing.
I realized that everytime there's a clash (i.e. keychords kicking in while I didn't want to) it's because I'm in the process of frenziedly modifying the buffer.
Here's an example...
I'm a fast typist so if I'm entering, say, "held", there is never going to be a long delay between when I add the 'e' of "held" and when I then type the 'l'. So in this case it should be "obvious" that I do not want key-chord to kick in when I type 'ld'. However if there's a pause and if 'ld' is an assigned key-chord and then I hit 'ld', then it is very unlikely that I'm writing code / text that would be starting with 'ld'. So in this later case (and because I have assigned 'ld' to a key-chord), I do want it to kick in.
Now don't get me wrong: I'm not sayin this would eliminate every single possible clash in every single case. All I'm saying is that, in my case, this would probably eliminate 99.9% of the (already rare) clashes. The "time lost" should one clash still occur after this change to key-chord would be negligible compared to the amount of time key-chord is making me gain.
The code to key-chord.el is very small so maybe such an addition wouldn't be too hard?
Does anyone have an idea as to how if it would be easy to modify key-chord.el to provide such a feature? (I hope I explained correctly what I want to do)
Here's the code to key-chord.el and half of it is comments:
http://emacswiki.org/emacs/key-chord.el
The Idle Timer mechanism should be able to facilitate this.
C-hig (elisp) Idle Timers RET
I'll leave it to you (or someone else) to figure out the details. Offhand thoughts are:
enable the Key Chord functionality via an idle timer
use post-command-hook to disable it again
That might be all that is needed?
I see two timings at play here
time before the chord (between the letter before the chord and the chord. If short enough, no chord)
time after the chord (if new letter quickly typed, no chord)
The second type of timing is probably more natural, and easy to get used to. If one type the chord, a short timeout before it executes can be accepted. (This is not what you asked for and would not solve the "ld" example. But it could potentially solve the problem with letter combinations that may appear in the beginning of words.)
The first type of timing is probably harder to get used to. When one have completed typing a word and the next thing is to type a chord, I suspect it is not a natural instinct to make a short pause. If one uses the chord often, one is likely to type it quickly, and get annoyed if it is not recognized (and the two chars are inserted into the text instead). What I want to say is, I am not sure a timing before the chord will solve the problem to a satisfactory degree.
As for the implementation of such a timing. The way the state machine in the key-chord-input-method function is currently structured, it would grow exponentially with the number of timers to consider (if I recall correctly). I.e. there would be at least about a dozen new lines of code.
To experiment with the functionality, try phils suggestion in the other answer: a post command that disables chords, and an idle timer that, after a fraction of a second, enables them again. A quick and dirty way to disable and enable chords, without actually changing minor mode, would be to set the input-method-function variable.
Code for test purpose only. It assumes, but does not check, that key-chord-mode version 0.5 is loaded and enabled:
;; Helper functions to be used in timers and hooks
(defun my-enable-chords () (setq input-method-function 'key-chord-input-method))
(defun my-disable-chords () (setq input-method-function nil))
;; Start test
(add-hook 'post-command-hook 'my-disable-chords)
(setq my-timer (run-with-idle-timer 0.3 'repeat 'my-enable-chords))
;; Finish test
(remove-hook 'post-command-hook 'my-disable-chords)
(cancel-timer my-timer)

Current memory usage in Lisp

I need to find out, from within a Common Lisp program, how much memory is currently being used.
I'm given to understand there is no portable method (the standard function room prints the information to standard output in text form instead of returning it as a value), but sb-kernel:dynamic-usage works in SBCL.
What are the equivalents in other Common Lisp implementations? Or is there another way to solve this problem I should be looking at?
It may not help you much, but anyway:
You can capture the output of (room) and parse it.
(with-output-to-string (*standard-output*)
(room))
Above returns a string with the output of ROOM.
Additionally it may help to request the memory size of the process via an external call to a standard unix command (if you are on Unix).
For things which virtually every implementation supports, but not in the same way (because it's not in CL), one common approach is to make a library called trivial-whatever.
If you started a package like trivial-memory, and supplied the first implementation, I'm sure we could get everybody to contribute the function for their own favorite Lisp compiler in short order. :-)

Turning off the result printing in common lisp

I am working with a reasonably large dataset in GNU clisp. It would be really nice if I could turn off the P of the REPL. Having thousands of results spew across my screen really isn't very useful.
I rummaged through the docs and couldn't find out how to turn it off. I assume it's one of the variables.
You might try changing the value of *print-length*.

How do I 'get' Lisp? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I've read The Nature of Lisp. The only thing I really got out of that was "code is data." But without defining what these terms mean and why they are usually thought to be separate, I gain no insight. My initial reaction to "code is data" is, so what?
The old fashioned view: 'it' is interactive computation with symbolic expressions.
Lisp enables easy representation of all kinds of expressions:
english sentence
(the man saw the moon)
math
(2 * x ** 3 + 4 * x ** 2 - 3 * x + 3)
rules
(<- (likes Kim ?x) (likes ?x Lee) (likes ?x Kim))
and also Lisp itself
(mapcar (function sqr) (quote (1 2 3 4 5)))
and many many many more.
Lisp now allows to write programs that compute with such expressions:
(translate (quote (the man saw the moon)) (quote german))
(solve (quote (2 * x ** 3 + 4 * x ** 2 - 3 * x + 3)) (quote (x . 3)))
(show-all (quote (<- (likes Kim ?x) (likes ?x Lee) (likes ?x Kim))))
(eval (quote (mapcar (function sqr) (quote (1 2 3 4 5)))))
Interactive means that programming is a dialog with Lisp. You enter an expression and Lisp computes the side effects (for example output) and the value.
So your programming session is like 'talking' with the Lisp system. You work with it until you get the right answers.
What are these expressions? They are sentences in some language. They are part descriptions of turbines. They are theorems describing a floating point engine of an AMD processor. They are computer algebra expressions in physics. They are descriptions of circuits. They are rules in a game. They are descriptions of behavior of actors in games. They are rules in a medical diagnosis system.
Lisp allows you to write down facts, rules, formulas as symbolic expressions. It allows you to write programs that work with these expressions. You can compute the value of a formula. But you can equally easy write programs that compute new formulas from formulas (symbolic math: integrate, derive, ...). That was Lisp designed for.
As a side effect Lisp programs are represented as such expressions too. Then there is also a Lisp program that evaluates or compiles other Lisp programs. So the very idea of Lisp, the computation with symbolic expressions, has been applied to Lisp itself. Lisp programs are symbolic expressions and the computation is a Lisp expression.
Alan Kay (of Smalltalk fame) calls the original definition of Lisp evaluation in Lisp the Maxwell's equations of programming.
Write Lisp code. The only way to really 'get' Lisp (or any language, for that matter) is to roll up your sleeves and implement some things in it. Like anything else, you can read all you want, but if you want to really get a firm grasp on what's going on, you've got to step outside the theoretical and start working with the practical.
The way you "get" any language is by trying to write some code in it.
About the "data is code" thing, in most languages there is a clear separation between the code that gets executed, and the data that is processed.
For example, the following simple C-like function:
void foo(int i){
int j;
if (i % 42 == 0){
bar(i-2);
}
for (j = 0; j < i; ++j){
baz();
}
}
the actual control flow is determined once, statically, while writing the code. The function bar isn't going to change, and the if statement at the beginning of the function isn't going to disappear. This code is not data, it can not be manipulated by the program.
All that can be manipulated is the initial value of i. And on the other hand, that value can not be executed the way code can. You can call the function foo, but you can't call the variable i. So i is data, but it is not code.
Lisp does not have this distinction. The program code is data that can be manipulated too. Your code can, at runtime, take the function foo, and perhaps add another if statement, perhaps change the condition in the for-loop, perhaps replace the call to baz with another function call. All your code is data that can be inspected and manipulated as simply as the above function can inspect and manipulate the integer i.
I would highly recommend Structure and Interpretation of Computer Programs, which actually uses scheme, but that is a dialect of lisp. It will help you "get" lisp by having you do many different exercises and goes on to show some of the ways that lisp is so usefull.
I think you have to have more empathy for compiler writers to understand how fundamental the code is data thing is. I'll admit, I've never taken a compilers course, but converting any sufficiently high-level language into machine code is a hard problem, and LISP, in many ways, resembles an intermediate step in this process. In the same way that C is "close to the metal", LISP is close to the compiler.
This worked for me:
Read "The Little Schemer". It's the shortest path to get you thinking in Lisp mode (minus the macros). As a bonus, it's relatively short/fun/inexpensive.
Find a good book/tutorial to get you started with macros. I found chapter 8 of "The Scheme
Programming Language" to be a good starting point for Scheme.
http://www.ccs.neu.edu/home/matthias/BTLS/
http://www.scheme.com/tspl3/syntax.html
By watching legendary Structure and Interpretation of Computer Programs?
In Common Lisp, "code is data" boils down to this. When you write, for example:
(add 1 2)
your Lisp system will parse that text and generate a list with three elements: the symbol ADD, and the numbers 1 and 2. So now they're data. You can do whatever you want with them, replace elements, insert other stuff, etc.
The fun part is that you can pass this data on to the compiler and, because you can manipulate these data structures using Lisp itself, this means you can write programs that write other programs. This is not as complicated as it sounds, and Lispers do it all the time using macros. So, just get a book about Lisp, and try it out.
Okay, I'm going to take a crack at this. I'm new to Lisp myself, just having arrived from the world of python. I haven't experienced that sudden moment of enlightenment that all the old Lispers talk about, but I'll tell you what I am seeing so far.
First, look at this random bit of python code:
def is_palindrome(st):
l = len(st)/2
return True if st[:l] == st[:-l-1:-1] else False
Now look at this:
"""
def is_palindrome(st):
l = len(st)/2
return True if st[:l] == st[:-l-1:-1] else False
"""
What do you, as a programmer, see? The code is identical, FYI.
If you are like me, you'll tend to think of the first as active code. It consists of a number of syntactic elements.
The second, despite its similarity, is a single syntactic item. It's a string. You interact with it as a single entity. To deal with it as code - to handle it comfortably along its syntactic boundaries - you will have to do some parsing. To execute it, you need to invoke an interpreter. It's not the same thing at all as the first.
So when we do code generation in most languages what are we dealing with? Strings. When I generate HTML or SQL with python I use python strings as the interface between the two languages. Even if I generate python with python, strings are the tool.*
Doesn't the thought of that just... make you want to dance with joy? There's always this grotesque mismatch between that which you are working with and that which you are working on. I sensed that the first time that I generated SQL with perl. Differences in escaping. Differences in formatting: think about trying to get a generated html document to look tidy. Stuff isn't easy to reuse. Etc.
To solve the problem we serially create templating libraries. Scads of them. Why so many? My guess is that they're never quite satisfactory. By the time they start getting powerful enough they've turned into monstrosities. Granted, some of them - such as SQLAlchemy and Genshi in the python world - are very beautiful and admirable monstrosities. Let's... um... avoid mention of PHP.
Because strings make an awkward interface between the worked-on language and the worked-with, we create a third language - templates - to avoid them. ** This also tends to be a little awkward.
Now let's look at a block of quoted Lisp code:
'(loop for i from 1 to 8 do (print i))
What do you see? As a new Lisp coder, I've caught myself looking at that as a string. It isn't. It is inactive Lisp code. You are looking at a bunch of lists and symbols. Try to evaluate it after turning one of the parentheses around. The language won't let you do it: syntax is enforced.
Using quasiquote, we can shoehorn our own values into this inactive Lisp code:
`(loop for i from 1 to ,whatever do (print i))
Note the nature of the shoehorning: one item has been replaced with another. We aren't formatting our value into a string. We're sliding it into a slot in the code. It's all neat and tidy.
In fact if you want to directly edit the text of the code, you are in for a hassle. For example if you are inserting a name <varname> into the code, and you also want to use <varname>-tmp in the same code you can't do it directly like you can with a template string: "%s-tmp = %s". You have to extract the name into a string, rewrite the string, then turn it into a symbol again and finally insert.
If you want to grasp the essence of Lisp, I think that you might gain more by ignoring defmacro and gensyms and all that window dressing for the moment. Spend some time exploring the potential of the quasiquote, including the ,# thing. It's pretty accessible. Defmacro itself only provides an easy way to execute the result of quasiquotes. ***
What you should notice is that the hermetic string/template barrier between the worked-on and the worked-with is all but eliminated in Lisp. As you use it, you'll find that your sense of two distinct layers - active and passive - tends to dissipate. Functions call macros which call macros or functions which have functions (or macros!) passed in with their arguments. It's kind of a big soup - a little shocking for the newcomer. That said, I don't find that the distinction between macros and functions is as seamless as some Lisp people say. Mostly it's ok, but every so often as I wander in the soup I find myself bumping up against the ghost of that old barrier - and it really creeps me out!
I'll get over it, I'm sure. No matter. The convenience pays for the scare.
Now that's Lisp working on Lisp. What about working on other languages? I'm not quite there yet, personally, but I think I see the light at the end of the tunnel. You know how Lisp people keep going on about S-expressions being the same thing as a parse tree? I think the idea is to parse the foreign language into S-expressions, work on them in the amazing comfort of the Lisp environment, then send them back to native code. In theory, every language out there could be turned into S-expressions, or even executable lisp code. You're not working in a first language combined with a third language to produce code in a second language. It is all - while you are working on it - Lisp, and you can generate it all with quasiquotes.
Have a look at this (borrowed from PCL):
(define-html-macro :mp3-browser-page ((&key title (header title)) &body body)
`(:html
(:head
(:title ,title)
(:link :rel "stylesheet" :type "text/css" :href "mp3-browser.css"))
(:body
(standard-header)
(when ,header (html (:h1 :class "title" ,header)))
,#body
(standard-footer))))
Looks like an S-expression version of HTML, doesn't it? I have a feeling that Lisp works just fine as its own templating library.
I've started to wonder about an S-expression version of python. Would it qualify as a Lisp? It certainly wouldn't be Common Lisp. Maybe it would be nicer - for python programmers at least. Hey, and what about P-expressions?
* Python now has something called AST, which I haven't explored. Also a person could use python lists to represent other languages. Relative to Lisp, I suspect that both are a bit of a hack.
** SQLAlchemy is kind of an exception. It's done a nice job of turning SQL directly into python. That said, it appears to have involved significant effort.
*** Take it from a newbie. I'm sure I'm glossing over something here. Also, I realize that quasiquote is not the only way to generate code for macros. It's certainly a nice one, though.
Data is code is an interesting paradigm that supports treating a data structure as a command. Treating data in this way allows you to process and manipulate the structure in various ways - e.g. traversal - by evaluating it. Moreover, the 'data is code' paradigm obviates the need in many cases to develop custom parsers for data structures; the language parser itself can be used to parse the structures.
The first step is forgetting everything you have learned with all the C and Pascal-like languages. Empty your mind. This is the hardest step.
Then, take a good introduction to programming that uses Lisp. Don't try to correlate what you see with anything that you know beforehand (when you catch yourself doing that, repeat step 1). I liked Structure and Interpretation of Computer Programs (uses Scheme), Practical Common Lisp, Paradigms of Artificial Intelligence Programming, Casting Spels in Lisp, among others. Be sure to write out the examples. Also try the exercises, but limit yourself to the constructs you have learned in that book. If you find yourself trying to find, for example, some function to set a variable or some statement that resembles a for loop, repeat step 1, then revisit the chapters before to find out how it is done in Lisp.
Read and understand the legendary page 13 of the Lisp 1.5 Programmer's Manual
According to Alan Kay, at least.
One of the reasons that some university computer science programs use Lisp for their intro courses is that it's generally true that a novice can learn functional, procedural, or object-oriented programming more or less equally well. However, it's much harder for someone who already thinks in procedural statements to begin thinking like a functional programmer than to do the inverse.
When I tried to pick up Lisp, I did it "with a C accent." set! amd begin were my friends and constant companions. It is surprisingly easy to write Lisp code without ever writing any functional code, which isn't the point.
You may notice that I'm not answering your question, which is true. I merely wanted to let you know that it's awfully hard to get your mind thinking in a functional style, and it'll be an exciting practice that will make you a stronger programmer in the long run.
Kampai!
P.S. Also, you'll finally understand that "my other car is a cdr" bumper sticker.
To truly grok lisp, you need to write it.
Learn to love car, cdr, and cons. Don't iterate when you can recurse. Start out writing some simple programs (factorial, list reversal, dictionary lookup), and work your way up to more complex ones (sorting sets of items, pattern matching).
On the code is data and data is code thing, I wouldn't worry about it at this point. You'll understand it eventually, and its not critical to learning lisp.
I would suggest checking out some of the newer variants of Lisp like Arc or Clojure. They clean up the syntax a little and are smaller and thus easier to understand than Common Lisp. Clojure would be my choice. It is written on the JVM and so you don't have the issues with various platform implementations and library support that exist with some Lisp implementations like SBCL.
Read On Lisp and Paradigms in Artificial Intelligence Programming. Both of these have excellent coverage of Lisp macros - which really make the code is data concept real.
Also, when writing Lisp, don't iterate when you can recurse or map (learn to love mapcar).
it's important to see that data is code AND code is data. This feeds the eval/apply loop. Recursion is also fun.
(This link is broken:
![Eval/Apply][1]
[1]: http://ely.ath.cx/~piranha/random_images/lolcode-eval_apply-2.jpg
)
I'd suggest that is a horrible introduction to the language. There are better places to start and better people/articles/books than the one you cited.
Are you a programmer? What language(s)?
To help you with your question more background might be helpful.
About the whole "code is data" thing:
Isn't that due to the "von Neumann architecture"? If code and data were located in physically separate memory locations, the bits in the data memory could not be executed whereas the bits in the program memory could not be interpreted as anything but instructions to the CPU.
Do I understand this correctly?
I think to learn anything you have to have a purpose for it, such as a simple project.
For Lisp, a good simple project is a symbolic differentiator, so for example
(diff 'x 'x) -> 1
(diff 'a 'x) -> 0
(diff `(+ ,xx ,yy) 'x) where xx and yy are subexpressions
-> `(+ ,(diff xx 'x),(diff yy 'x))
etc. etc.
and then you need a simplifier, such as
(simp `(+ ,x 0)) -> x
(simp `(* ,x 0)) -> 0
etc. etc.
so if you start with a math expression, you can eval it to get its value, and you can eval its derivative to get its derivative.
I hope this illustrates what can happen when program code manipulates program code.
As Marvin Minsky observed, computer math is always worried about accuracy and roundoff error, right? Well, this is either exactly right or completely wrong!
You can get LISP in many ways, the most common is by using Emacs or working next to somebody who has developed LISP already.
Sadly, once you get LISP, it's hard to get rid of it, antibiotics won't work.
BTW: I also recommend The Adventures of a Pythonista in Schemeland.
This may be helpful: http://www.defmacro.org/ramblings/fp.html (isn't about LISP but about functional programming as a paradigm)
The way I think about it is that the best part of "code is data" is the face that function are, well, functionally no different than another variable. The fact that you can write code that writes code is one of the single most powerful (and often overlooked) features of Lisp. Functions can accept other functions as parameters, and even return functions as a result.
This lets one code at a much higher level of abstraction than, say, Java. It makes many tasks elegant and concise, and therefore, makes the code easier to modify, maintain, and read, or at least in theory.
I would say that the only way to truly "get" Lisp is to spend a lot of time with it -- once you get the hang of it, you'll wish you had some of the features of Lisp in your other programming languages.