I was poking around in the Macro Stepper for the first time and noticed that a for/list expanded into code involving something called alt-reverse. Does for/list cons each item onto the front of an empty list and then reverse it? That seems very inefficient.
I wrote a little test:
(define (test n)
(time
(for/list ([x (in-range n)])
(list x x)))
(time
(for/fold ([result '()])
([x (in-range n)])
(cons (list x x) result)))
(void))
Indeed the for/list version runs in about 150% of the time of for/fold without reverse, the difference apparently due entirely to additional garbage collection:
> (test 500000)
cpu time: 1059 real time: 2079 gc time: 940
cpu time: 614 real time: 1231 gc time: 550
> (test 500000)
cpu time: 1060 real time: 3889 gc time: 907
cpu time: 770 real time: 1363 gc time: 699
> (test 500000)
cpu time: 1035 real time: 2479 gc time: 917
cpu time: 736 real time: 2535 gc time: 651
It sounds like I shouldn't get in the habit of calling for/list. Is there a more efficient way to make a list in "forward" order (i.e. where the last item evaluated is the last item in the list)?
Looking through the Git history, I see that a change to make for/list avoid reverse was committed, but it didn't work because of a subtle interaction with continuations that could tigger quadtradic-time behavior. (I do wonder if the upcoming move to a Chez Scheme backend might mean that "when Racket one day gets a better implementation of continuations" could actually be coming soon.)
You can build a list in "forward" order by, as the first commit message put it, "consing onto a recursive call." The Racket Guide sections on Tail Recursion and Recursion versus Iteration actually discuss the trade-offs between the map-style and for/fold-style approaches in some detail.
Also, for future reference, the Racket community tends to live more on the very active and friendly racket-users mailing list and to some extent the Slack channel than here on Stack Overflow.
Related
My prototype program initially defines a number of global parameters, which subsequently affect the course of analysis. But the program body ended up containing numerous tests of these parameters to determine how to proceed with the detailed analysis. For example:
(defparameter *param1* 0)
(ecase *param1*
(0 (foo))
(1 (bar))
(2 (baz)))
But performing all these various tests at run-time is inefficient. Can these tests be effectively moved to compile-time (or handled otherwise), since all of the parameters are known before analysis begins?
My first idea was to build macros for the tests, so only the relevant code is available at run-time:
(defmacro param1-macro ()
(ecase *param1*
(0 `(foo))
(1 `(bar))
(2 `(baz))))
But having calls like (param1-macro) scattered throughout makes the code hard to read and analyze during debugging. The unique decision process for each macro call is then nonlocal. Is there a more transparent way to increase run-time efficiency?
If these parameters are compile-time constants then make them constants (ie define them with defconstant). This should give the compiler a chance to make assumptions about their values at compile time and turn conditionals into unconditional execution.
How well the compiler can do this depends on the compiler of course: in my limited testing at least some CL compilers do this kind of optimisation.
I'd certainly try to do this before a lot of second-guessing-the-compiler-with-macros.
Another thing, of course, is to raise tests (this must have a name, but I'm not sure what it is: I always call it just 'raising'). Turn code like
(dotimes (i big-number)
(case *parameter*
((1) ...)
...))
into
(case *parameter*
((1) (dotimes (i big-number)
...))
...)
Which reduces the number of tests by a factor of big-number. I suspect this also is an optimisation that good compilers can do themselves.
Finally and perhaps most importantly: measure how slow it is. Are the tests really taking a long time? Almost certainly unless you have measured their cost you don't know it: certainly that's what I find whenever I make assumptions like that!
As an alternative to the above here is a really horrible hack which will allow you to use parameters but will wire in their compile-time values, thus causing the compiler to treat them like constants. Note: when I say 'horrible' I also mean 'mostly untested'.
Here is how you can do this:
(in-package :cl-user)
(eval-when (:compile-toplevel :load-toplevel :execute)
(defvar *wiring* t))
(declaim (inline wiring))
(defun wiring (x)
x)
(define-compiler-macro wiring (&whole form x)
(if (and *wiring*
(symbolp x)
(boundp x))
`(quote ,(symbol-value x))
form))
Now, if *wiring* is false at compile time (wiring ...) is a just an identity function, so you can say (wiring *param*) to mean *param*. It's declared inline so it should have zero cost (and, really, zero purpose).
But if *wiring* is true at compile-time, then the compiler macro does a horrible thing: (wiring *param*) will:
check if *param* is a symbol;
if it is, and if it is bound then it will expand to its compile-time value (quoted for safety).
This means, that if, at compile time, *param* is bound to 2, then
(case (wiring *param*)
((2) (+ x 0))
(otherwise (+ x 1)))
is equivalent, at compile time, to
(case 2
((2) (+ x 0))
(otherwise (+ x 1)))
And the compiler can then turn this (with a note in the case of SBCL) into (+ x 0).
This is horrid because ... well, there are very many ways it is horrid. It certainly violates a lot of assumptions about how code should behave. But it's also kind of lovely: you can do this in Lisp, within the language, which I think is amazing.
I'm wondering how to store a single variable and to have specific functions on that variable. I'm wondering if there are alternatives to creating a class.
Specifically, I am creating an application where I store a value for time that represents the number of seconds elapsed from a base time (e.g., Jan 1, 2000, 00:00:00). I want to perform operations on this value such as convert it from seconds from to a specific time or date, or from a date to the specific seconds.
I have done this using a class, however it seems wasteful. Specifically, every time I access the stored value for seconds elapsed it would look similar to (time-time time) where time-time is the accessor for the time instance time.
Is there a better way to design this, perhaps without classes?
Accessor names
You can name accessors in CLOS any way you like. The accessor function could be called seconds:
CL-USER 23 > (defclass mytime ()
((seconds :accessor seconds :initarg :seconds)))
#<STANDARD-CLASS MYTIME 422015CDD3>
CL-USER 24 > (let ((mt (make-instance 'mytime :seconds 100)))
(values (seconds mt)
(truncate (seconds mt) 60)))
100
1
Shorter access to slots via accessor functions
Common Lisp also has a form WITH-ACCESSORS. It allows us to use a symbol instead of an accessor form in the code - for a certain CLOS object. In the following example we can use secs and it looks like a variable in the code, but Common Lisp will make sure that it actually calls the accessor seconds. We can write secs instead of (seconds mt). Thus it helps to make the enclosed code shorter. Compare the next example with the code above.
CL-USER 25 > (let ((mt (make-instance 'mytime :seconds 200)))
(with-accessors ((secs seconds))
mt
(values secs (truncate secs 60))))
200
3
Shorter access to slots via SLOT-VALUE
CLOS also has WITH-SLOTS for slot access via slot names, here the slot named seconds of the mytime instance can be accessed via the name secs:
CL-USER 26 > (let ((mt (make-instance 'mytime :seconds 200)))
(with-slots ((secs seconds))
mt
(values secs (truncate secs 60))))
200
3
If you have a class which is just wrapping a single object, and that object has a known type, then you can always just write methods for the class of that object instead:
(defmethod time-as-unix-time ((tm integer))
(- tm (load-time-value (encode-universal-time 0 0 0 1 1 1970 0))))
For instance.
Of course object-oriented zealots will throw you into a pit full of spikes if they catch you doing this sort of thing: it no doubt violates encapsulation or some other rule of the cult.
You could try a closure over a lexical scope like this:
(let ((time (get-universal-time)))
(defun set-time(tm)
(setf time tm))
(defun get-time()
time)
(defun get-time-in-some-other-format()
;; calculate return value based on 'time'
)
)
I've been searching a way to stop a function after X seconds on lisp and I haven't found anything and have no ideia how to do it.
It's for finding heuristic weights on a function for a tetris game and if the weights are bad the program will run for more than 30 seconds and I don't want that. Any ideias, please?
Common Lisp has a library for that: Trivial Timeout.
The API is simple: Trivial Timeout API.
One possible way is to pass down an "expiry timer" and check if the current time is later than the expiry time in every iteration and, if it has expired, return the best solution so far. The standard function get-universal-time may be useful, but gives you a minimal granularity of seconds. A rough skeleton below. If you use recursive functions, simply pass the expiry timer down with whatever else you pass down and use that as your first recursion termination criterion.
(defun do-tetris-stuff (tetris-weights expiry-time)
(let ((best-solution nil))
(loop while (and (<= expiry-time (get-universal-time))
(not (good-enough-p best-solution)))
do (let ((next-solution ...))
(when (better-than-p next-solution best-solution)
(setf best-solution next-solution))))
best-solution))
Why is this code overflowing the heap with CMUCL? Even if i give it 400MB memory (setq extensions:*bytes-consed-between-gcs* 400000000) CMUCL will still choke on it.
; [GC threshold exceeded with 12,012,544 bytes in use. Commencing GC.]
; [GC completed with 188,064 bytes retained and 11,824,480 bytes freed.]
; [GC will next occur when at least 400,188,064 bytes are in use.]
; [GC threshold exceeded with 400,202,280 bytes in use. Commencing GC.]
; [GC completed with 207,120 bytes retained and 399,995,160 bytes freed.]
; [GC will next occur when at least 400,207,120 bytes are in use.]
This code runs ok with CCL and SBCL, though i havent looked at their memory usage.
Is this a bug in CMUCL?? I believe all these func's are tail recursive.
(defun sqrt-iter (guess x)
(if (good-enough? guess x)
guess
(sqrt-iter (improve guess x)
x)))
(defun improve (guess x)
(average guess (/ x guess)))
(defun average (x y)
(/ (+ x y) 2))
(defun good-enough? (guess x)
(< (abs (- (* guess guess) x)) 0.001))
(defun mysqrt (x)
(sqrt-iter 1.0 x))
(defun zint (x acc step)
(setq num-iter (+ 1 num-iter))
(if (>= x 10000.0)
acc
(zint (+ x step)
(+ acc (* step (mysqrt x)))
step)))
(setq num-iter 0)
(format t "result=~A; iterations=~A~%" (zint 0.0 0.0 .001) num-iter)
(quit)
EDIT:
Yes, CMUCL definitely conses a lot and unnecessarily. Try this simple example:
$ ~/cmucl/bin/lisp
...
* (defun foo () (bar))
FOO
* (defun bar () (foo))
BAR
* (foo)
; [GC threshold exceeded with 12,009,008 bytes in use. Commencing GC.]
; [GC completed with 111,816 bytes retained and 11,897,192 bytes freed.]
; [GC will next occur when at least 12,111,816 bytes are in use.]
; [GC threshold exceeded with 12,120,912 bytes in use. Commencing GC.]
; [GC completed with 120,016 bytes retained and 12,000,896 bytes freed.]
; [GC will next occur when at least 12,120,016 bytes are in use.]
; [GC threshold exceeded with 12,133,576 bytes in use. Commencing GC.]
There are two functions which are 'tail recursive': sqr-iter and zint.
General advice:
To let the compiler optimize these, the DEBUG optimization level needs to be 2 or less.
You can see if the compiler generates tail recursive code by disassembling them. Use the function disassemble.
The GC invocation itself is not a problem. Most implementations don't print anything on GC. CMUCL prints it by default (IIRC). CMUCL may allocate huge amounts of floats for un-optimized code - which then might cause a lot of GCs.
Only if CMUCL has a stack overflow you would see that the tail call optimization is not working. GCs themselves only show large amounts of consing.
So to debug your problem you need first to see if the code is compiled with tail call optimization on. You can disassemble the code. Another option is to put the code while running into the debugger and then by looking at the stack backtrace. On the stack there should not be a large number of recursive calls - they would be replaced by jumps.
If the code is running with constant stack space, then you need to look at the allocation of floats. You would need to check then that the code does not allocate too much floats.
I think it depends on the settings
... This is SBCL which is related.
* (declaim (optimize (speed 0) (compilation-speed 0) (safety 3) (debug 3)))
* (defun foo () (bar))
* (defun bar () (foo))
* (foo)
INFO: Control stack guard page unprotected
Control stack guard page temporarily disabled: proceed with caution
INFO: Control stack guard page reprotected
#<sb-kernel::control-stack-exhausted {1002DBD4D3}>
but if you tell it you want it to go fast and not worry about debugging... it does better
(declaim (optimize (speed 3) (compilation-speed 0) (safety 1) (debug 0)))
(defun foo () (bar))
(defun bar () (foo))
(foo) ;;; CPU topped out, but no stack issues or garbage collecting
Should I use
(apply + (filter prime? (range 1 20)))
or
(reduce + (filter prime? (range 1 20)))
Edit: This is the source for prime in clojure from optimizing toolkit.
(defn prime? [n]
(cond
(or (= n 2) (= n 3)) true
(or (divisible? n 2) (< n 2)) false
:else
(let [sqrt-n (Math/sqrt n)]
(loop [i 3]
(cond
(divisible? n i) false
(< sqrt-n i) true
:else (recur (+ i 2)))))))
If you are asking in terms of performance, the reduce is better by a little:
(time (dotimes [_ 1e6] (apply + (filter even? (range 1 20)))))
"Elapsed time: 9059.251 msecs"
nil
(time (dotimes [_ 1e6] (reduce + (filter even? (range 1 20)))))
"Elapsed time: 8420.323 msecs"
nil
About 7% difference in this case, but YMMV depending on the machine.
You haven't provided your source for the prime? function, so I have substituted even? as the predicate. Keep in mind that your runtime may be dominated by prime?, in which case the choice between reduce and apply matters even less.
If you are asking which is more "lispy" then I would say that the reduce implementation is preferrable, as what you are doing is a reduce/fold in the functional programming sense.
I would think that reduce would be preferable when it is available, because apply uses the list as arguments to the function, but when you have a large number -- say, a million -- elements in the list, you will construct a function call with a million arguments! That might cause some problems with some implementations of Lisp.
(reduce op ...) is the norm and (apply op ...) the exception (notably for str and concat).
I would expect apply to realize a lazy list which could be ugly, and you never want to assume your list is not lazy, 'cause you could suddenly find yourself getting smacked with massive memory useage.
Reduce is going to grab them 1 by one and roll the results together into a single whole, never taking the whole list in at once.
i am going to play devil's advocate and argue for apply.
reduce is clojure's take on fold (more exactly foldl), the left-fold, and is usually defined with an initial element, because a fold operation has two parts:
an initial (or "zero") value
an operation for combining two values
so to find the sum of a set of numbers the natural way to use + is as (fold + 0 values) or, in clojure, (reduce + 0 values).
this explicitly shows the result for an empty list, which is important because it is not obvious to me that + returns 0 in this case - after all, + is a binary operator (all that fold needs or assumes).
now, in practice, it turns out that clojure's + is defined as more than a binary operator. it will take many or, even, zero values. cool. but if we're using this "extra" information it's friendly to signal that to the reader. (apply + values) does this - it says "i am using + in a strange way, as more than a binary operator". and that helps people (me, at least) understand the code.
[it's interesting to ask why apply feels clearer. and i think it's partly that you are saying to the reader: "look, + was designed to accept multiple values (that's what apply is used for), and so the language implementation will include the case of zero values." that implicit argument is not present with reduce applied to a single list.]
alternatively, (reduce + 0 values) is also fine. but (reduce + values) triggers an instinctive reaction in me: "huh, does + provide a zero?".
and if you disagree, then please, before you downvote or post a reply, are you sure about what (reduce * values) will return for an empty list?
I agree with Andrew Cooke. But I would like to add that since it already knows how to work and handle multiple arguments, we are essentially re-telling the '+' operator how to work by using it in the 'reduce', in my opinion.
Also, if I told you do to add a bunch of numbers together, you would probably use the '+' operator by itself, and not use 'reduce' at all.
(+ 1 2 3 etc...)
'apply' is just saying, I want to do the same thing, but use what's inside this list for the arguments. There is no extra reasoning required and you are using it as if the numbers didn't exist in a list to begin with, like you would if you were writing it out yourself.
(Reduce is so heavily used that using/reasoning about it in this case isn't a big deal at all, I think either way is fine, but personally prefer apply)