I'd like to build a list of operations to execute in a Redis pipeline. I was using accession since it's much simpler than carmine, but now I need connection pooling (missing from accession) and thus I'm looking at carmine again (it seems the go-to and most complete clojure library for redis)
I managed to get this working:
; require [taoensso.carmine :as redis]
(defn execute1 [request] (redis/wcar {} (eval request)))
(defmacro execute [body] `(redis/wcar {} ~#(eval body)))
(def get1 `(redis/get 1))
(execute1 get1)
(execute [get1])
but given that I'll build vectors of thousands of elements, I'm a bit worried about the possible performance hit of eval (besides the fact that I've been taught to always avoid eval if possible). I reckon that defmacro instead is/can be evaluated at macro-expansion time, which might be earlier (during AOT compilation?) and without using eval.
Is there anything that I can do? should I move to a different library?
(I had a look at carmine's source code: my pain is only due to a small convenience for the author: a *context* used to avoid passing around a couple extra argument: getting rid of it should be simple enough, but I'm not invested enough in it... I might as well decide to move to another data store in the future)
edit: I've been asked to write an example of what I think is the boilerplate that I'd prefer to avoid writing in my actual code, so: (the following is untested, it's just a POC)
(defn hset [id key val]
#(redis/hset id key val))
(defn hsetnx [id key val]
#(redis/hsetnx id key val))
(defn hincrby [id key increment]
#(redis/hincrby id key increment))
(defn hgetall [id key]
#(redis/hgetall id key))
(defn sadd [id el]
#(redis/sadd id el))
(defn scard [id]
#(redis/scard id))
(defn smembers [id]
#(redis/smembers id))
(defmacro execute [forms]
`(redis/wcar {} ~#(map apply forms)))
; end boilerplate
(defn munge-element [[a b c]]
(conj
(mapcat #(hincrby a :whatever %) b)
(sadd c b)
(hsetnx a c))
(defn flush-queue! [queue_]
(execute queue_)
[])
(defn receive [item]
(if (< (count #queue) 2000)
(swap! queue conj (munge-element item))
(swap! queue flush-queue!)))
Obviously, I could write something like this, but if this was truly the inteneded way to use carmine, these curried functions would be provided along (or instead of) the normal ones. Also a lot of lines could be shaved off by constructing up the defs with syntax quoting, but this is accidental complexity, not inherent the original problem.
The following piece of code does not differ very much from yours:
(defmacro execute [& forms]
`(redis/wcar {} ~#forms))
(defn get1 []
(redis/get 1))
(execute (get1) (get1) ...)
This is basically how carmine should be used (following this suggestion in the README). If it doesn't suit your needs, could you clarify why?
After the question was edited to present more clearly what should be accomplished I think I have a solution for you. You're trying to create a list of statements to be executed at some specific time in the future. To achieve this you're wrapping each statement inside a function and I agree, that's a lot of boilerplate.
But unnecessary. You can get the same result by having a macro that automatically creates thunks for you:
(defmacro statements [& forms]
`(vector
~#(for [f forms]
`(fn [] ~f))))
Now, whatever you pass to this macro will result in a vector of zero-parameter functions that can be evaluated e.g. using:
(defn execute-statements [fns]
(redis/wcar {} (mapv #(%) fns))
And your example turns into something along the lines of:
(defn munge-element [[a b c]]
(statements
(mapcat #(redis/hincrby a :whatever %) b)
(redis/sadd c b)
(redis/hsetnx a c))
(defn flush-queue! [queue_]
(mapv execute-statements queue_)
[])
Edit: This executes every batch in its own pipeline. If you want to do it in a single one, use concat instead of conj to build up your queue (in receive) and (execute-statements queue_) instead of (mapv execute-statements queue_).
Note: IIRC correctly, this:
(redis/wcar {} a [b c]])
returns the same result as this:
(redis/wcar {} a b c)
I.e., carmine collects the results somewhere and always returns a vector for all of them. Even if not, I think you can still avoid your dreaded boilerplate by just tweaking the stuff presented here a little bit.
Related
I recently discovered that all of my implementations of Scheme throw an error when I try to use (cadaddr (list 1 3 (list 5 7) 9)). Apparently, by default Scheme does not allow any car and cdr combinations in the single-function form that use more than four abbreviated car and cdrcalls. I originally blamed this on Scheme's minimalism, but then I discovered that Common Lisp also shares this defect.
Can this be solved with a macro? Can we write a macro that allows an arbitrary amount of a and d in its c[...]r calls and returns the expected result, while also having the Common Lisp-like compatibility with macros like setf? If not, why not? And if so, has a reason ever been given for this is not a default feature in any lisp that I've seen?
Such a macro is described in Let Over Lambda for common lisp. You must wrap your code with (with-cxrs ...) to bring them all into scope, but it walks your code to see which combinators you need. I wrote a Clojure port of it years ago for fun, though of course nobody (including me) has ever wanted to use it for real. You could port it to Scheme if you liked.
(defn cxr-impl [name]
(when-let [op (second (re-matches #"c([ad]+)r" name))]
`(comp ~#(map {\a `first \d `rest} op))))
(defmacro with-cxrs [& body]
(let [symbols (remove coll? (tree-seq coll? seq body))]
`(let [~#(for [sym symbols
:let [impl (cxr-impl (name sym))]
:when impl
thing [sym impl]]
thing)]
~#body)))
user> (macroexpand-1 '(with-cxrs (inc (caadaaddadr x))))
(let [caadaaddadr (comp first first rest first first rest rest first rest)]
(inc (caadaaddadr x)))
https://groups.google.com/g/clojure/c/CanBrJPJ4aI/m/S7wMNqmj_Q0J
As noted in the mailing list thread, there are some bugs you'd have to work out if you wanted to use this for real.
Imagine the following code to dynamically create a macro:
(def a (list '+ 1 2))
(def b (list '- 10 5))
(def c (list '/ 22 2))
(defmacro gg [h]
(let [k# `~h]
k#))
The intent is to pass a vector of symbols to a macro, do some evaluation on each element of the vector such that it returns a nice macro-esque form, then have the macro combine them into a nice macro and evaluate it. The above example works all except the actual evaluation.
When I run it I get:
(gg [a b c])
=> [(+ 1 2) (- 10 5) (/ 22 2)]
What is the secret to passing a symbol that is a list of symbols and getting a macro to evaluate them? I have tried lots of combinations of quoting and have yet to hit the right one.
The real purpose of this question is to build an Archimedes Ogre query based on a definition of a path through the graph. If someone has an example of that, I would be grateful.
EDIT:
(defmacro gg2 [h]
`(do ~#(map identity h)))
(macroexpand '(gg2 [a b c]))
=> (do a b c)
(gg2 [a b c])
=> (/ 22 2)
I was hoping to get 11 rather than the form.
You don't need a macro. Macros don't do what you're looking for here. What you are looking for is eval.
(def a '/)
(def b 22)
(def c 2)
(eval (list* [a b c]))
=> 11
Of course, you can write a macro which expands into (eval (list* ...)) if you want. It could just as well be a function though.
This is a very common mistake when starting out with macros; trying to write a macro which depends on the run-time value of its arguments. Macros run at compile-time, and generally the values of the symbols which you pass to a macro are not yet available when the macro is expanded.
About the use of eval, some cautions are in order. No one said it better than Paul Graham:
Generally it is not a good idea to call eval at runtime, for two reasons:
It’s inefficient: eval is handed a raw list, and either has to compile it on the spot, or evaluate it in an interpreter. Either way is slower than compiling the code beforehand, and just calling it.
It’s less powerful, because the expression is evaluated with no lexical context. Among other things, this means that you can’t refer to ordinary variables visible outside the expression being evaluated.
Usually, calling eval explicitly is like buying something in an airport gift-shop. Having waited till the last moment, you have to pay high prices for a limited selection of second-rate goods.
Apologies for any mistaken terminology--I'm quite new to computer science, and I pretty much only know Clojure (but I guess I'd say I know it pretty well).
So, I haven't done a ton of research on this, but I've sometimes found it useful when writing Clojure code to be able to refer to some "intermediate version of whatever data structure I'm in" from within that data structure (a lot like in a let). Quick examples:
=> (self-ish {:a 10
:b (inc (this :a))
:c (count (vals this))})
=> {:a 10, :b 11, :c 3}
=> (self-ish ["a" "b" (reduce str this)])
=> ["a" "b" "ab"]
//Works in any nested bits too
=> (self-ish [1 2 3 [4 5 (first this)] 6 [7 [8 (cons (second this) (nth this 3))]]])
=> [1 2 3 [4 5 1] 6 [7 [8 (2 4 5 1)]]]
The idea is that the structure builds itself up incrementally, and at any stage has the ability to refer to the current intermediate structure as this. Here's the code for my current implementation:
//Random straightforward but helpful definitions
(defn map-entry? [obj]
(instance? clojure.lang.AMapEntry obj))
(def Map clojure.lang.IPersistentMap)
(def Vector clojure.lang.IPersistentVector)
(def List clojure.lang.IPersistentList)
(def Set clojure.lang.IPersistentSet)
(defn append
[x coll]
(if-not coll x
(condp instance? coll
Map (if (empty? x) coll
(assoc coll (first x) (second x)))
Vector (conj coll x)
Set (conj coll x)
List (apply list (concat coll [x]))
(concat coll [x]))))
(defn build-this
[acc-stack acc]
(->> (cons acc acc-stack)
(drop-while list?)
(drop-while (every-pred empty? identity))
(reduce append)))
(defn self-indulge
[acc-stack acc form]
;//Un-comment the following to see it print intermediate stages of processing
#_(println "this:" (build-this acc-stack acc) "\n at:" form)
(append (cond
(coll? form) (reduce (partial self-indulge (cons acc acc-stack))
(if (map-entry? form) []
(empty form))
form)
(= (quote this) form) (build-this acc-stack acc)
:else form)
acc))
(defmacro self-ish
[form]
(self-indulge () nil form))
The append function appends an item onto a collection and returns the same type of collection. The self-indulge function has a standard reduce-like accumulator, which just builds up elements of form. It also has a accumulator stack, which gets longer every time self-indulge recurs upon itself. The point of this is to keep track of other "higher up" accumulators, so that this will be the entire structure, not just a local piece. The self-ish macro just nicely wraps up self-indulge (which calls itself using partial, so it can't wear the macro pants).
Edit: example use case
To me, this macro is about trying to increase code readability, not truly extending functionality. Where I have found this useful is in cases where I have hand-written structures with partially redundant data--or maybe "dependent" is a better word. It can be easier to read the code and see what different parts of the data structure hold, and it can also be useful if I modify data values in one part of a structure and want that change to be reflected in other parts. For example:
=> (self-ish {:favorite-books (list "Crime and Punishment" "Mrs. Dalloway")
:favorite-things (list* "Ice Cream" "Hammocks" (this :favorite-books)})
=> {:favorite-things ("Ice Cream" "Hammocks" "Crime and Punishment" "Mrs. Dalloway"),
:favorite-books ("Crime and Punishment" "Mrs. Dalloway")}
It could also be useful in times where one might really like to include something baked into the data, as opposed to derived on the fly using some function. These cases are probably much rarer, and I think it would be a bad idea to tangle the data unnecessarily when you could just have nice clean functions manipulating it.
My main questions:
Is this actually useful, or would the ambiguity/complexity incurred be too much? I imagine I'm not alone in wanting/using this type of macro. What are others' experiences here? Do you use something like this? Have you found better workarounds? Are there reasons something like this isn't in any Clojure libraries? Or is there something that I haven't yet seen?
Are there better naming conventions I might use--as opposed to self-ish and this? For example, maybe this is too loaded with OOP meaning, I'm not sure, I am basically only familiar with Clojure.
I am pretty new to computer science, are there accessible and informative resources related to this type of thing--I guess I would call it anonymous self referential (maybe reflexive is a better word?) data structures? I haven't found anything both approachable and informative yet.
Is there a better way to write the self-ish macro? Above, I've included my current version of it, but I can't shake the feeling there may be a simpler way.
I have various questions about what might be the "wisest" implementation details.
Traversal: Should it be breadth first or depth first? If depth first, preorder, postorder, or inorder? Right now, I believe it's depth first preorder, which makes sense to me, but maybe it has some drawbacks I haven't noticed.
Order problems: (See here for a related previous question of mine) Within {} (i.e. maps written by hand) it's impossible to maintain order properly (above 8 map entries) without using array-map or sorted-map--in other words, above 8 map entries, {} usage is unsafe. Maybe instead of hand-written order, the macro could do some fancy magic to process items in some "ideal" order? Or perhaps it would be better to wrap all maps within (array-map ...) instead of the eye-pleasing {}?
//Showing maps with 9 entries failing
=> (self-ish {:a 1
:b (inc (this :a))
:c (inc (this :b))
:d (inc (this :c))
:e (inc (this :d))
:f (inc (this :e))
:g (inc (this :f))
:h (inc (this :g))
:i (inc (this :h))})
=> NullPointerException clojure.lang.Numbers.ops (Numbers.java:942)
//8 works fine
=> (self-ish {:a 1
:b (inc (this :a))
:c (inc (this :b))
:d (inc (this :c))
:e (inc (this :d))
:f (inc (this :e))
:g (inc (this :f))
:h (inc (this :g))})
=> {:h 8, :g 7, :f 6, :e 5, :d 4, :c 3, :b 2, :a 1}
Serial: As I've written it, the macro avoids infinite recursion by dealing with its elements serially, similar to let, but that does produce potentially odd behavior. For example, in my above quick example, (reduce str this) returns "ab" because this is ["a" "b"] at that step. Maybe it would be useful sometimes to create some sort of infinite lazy sequence instead? If so, how might that be implemented?
Map entries: Right now, map entries are treated like vectors, but because of how this could be invoked at any intermediate step, it is totally possible to get a nil value from a key that has "not yet" been assigned a value. That is why in my first quick example, :c ended up mapped to 3--because intermediately there was a nil corresponding to :c, and that got counted as well. Do you think this warrants fixing?
Non-macro utility: It would be trivial to use just self-indulge outside of the macro context, but could this ever be useful?
Thanks for reading, any help is appreciated :)
This approach "feels" a bit wrong to me, though I'm not quite sure why. Maybe I don't like the idea of map construction being dependent on order....
Having said that it's a pretty easy macro to write, you effectively want something that expands to:
(let [this {}
this (assoc this :a 1)
this (assoc this :b (+ (this :a) 3))]
this)
Hence an appropriate macro would be something like (for the map case):
(defmacro self-ish [bindings]
`(let [~'this {}
~#(mapcat
#(do `(~'this (assoc ~'this ~#%)) )
(partition 2 bindings) )]
~'this))
(self-ish [:a 1
:b (+ (this :a) 3)])
=> {:b 4, :a 1}
Note that I'm made the binding form a vector as a map binding form is unordered.
Still not sure how much I like this idiom. My preferred way is usually to define a structure with a let form and give meaningful names to interim calculations e.g.:
(let [meaningful-foo (something)
meaningful-bar (something-else)]
{:foo meaningful-foo
:bar meaningful-bar
:baz (some-calculation meaningful-foo meaningful-bar)})
in scheme this is done with (letrec ...) which lets you refer to the name of the data structure inside the structure itself. so if you want to define your own way of doing this it might make more sense to implement that. you could do it using the tricks with references described in the answers to Is it possible to create circular references in Clojure?
one advantage of letrec is that it has a user-specified name (if your macro is nested then this is shadowed).
[edited to remove comment on types as i don't understand your macro.]
[update] also, you may be interested in the discussion of anaphora in joy of clojure section 8.5.1
I am trying to implement a huge Java interface with numerous (~50) getter and setter methods (some with irregular names). I thought it would be nice to use a macro to reduce the amount of code. So instead of
(def data (atom {:x nil}))
(reify HugeInterface
(getX [this] (:x #data))
(setX [this v] (swap! data assoc :x v)))
I want to be able to write
(def data (atom {:x nil}))
(reify HugeInterface
(set-and-get getX setX :x))
Is this set-and-get macro (or something similar) possible? I haven't been able to make it work.
(Updated with a second approach -- see below the second horizontal rule -- as well as some explanatory remarks re: the first one.)
I wonder if this might be a step in the right direction:
(defmacro reify-from-maps [iface implicits-map emit-map & ms]
`(reify ~iface
~#(apply concat
(for [[mname & args :as m] ms]
(if-let [emit ((keyword mname) emit-map)]
(apply emit implicits-map args)
[m])))))
(def emit-atom-g&ss
{:set-and-get (fn [implicits-map gname sname k]
[`(~gname [~'this] (~k #~(:atom-name implicits-map)))
`(~sname [~'this ~'v]
(swap! ~(:atom-name implicits-map) assoc ~k ~'v))])})
(defmacro atom-bean [iface a & ms]
`(reify-from-maps ~iface {:atom-name ~a} ~emit-atom-g&ss ~#ms))
NB. that the atom-bean macro passes the actual compile-time value of emit-atom-g&ss on to reify-from-maps. Once a particular atom-bean form is compiled, any subsequent changes to emit-atom-g&ss have no effect on the behaviour of the created object.
An example macroexpansion from the REPL (with some line breaks and indentation added for clarity):
user> (-> '(atom-bean HugeInterface data
(set-and-get setX getX :x))
macroexpand-1
macroexpand-1)
(clojure.core/reify HugeInterface
(setX [this] (:x (clojure.core/deref data)))
(getX [this v] (clojure.core/swap! data clojure.core/assoc :x v)))
Two macroexpand-1s are necessary, because atom-bean is a macro which expands to a further macro call. macroexpand would not be particularly useful, as it would expand this all the way to a call to reify*, the implementation detail behind reify.
The idea here is that you can supply an emit-map like emit-atom-g&ss above, keyed by keywords whose names (in symbolic form) will trigger magic method generation in reify-from-maps calls. The magic is performed by the functions stored as functions in the given emit-map; the arguments to the functions are a map of "implicits" (basically any and all information which should be accessible to all method definitions in a reify-from-maps form, like the name of the atom in this particular case) followed by whichever arguments were given to the "magic method specifier" in the reify-from-maps form. As mentioned above, reify-from-maps needs to see an actual keyword -> function map, not its symbolic name; so, it's only really usable with literal maps, inside other macros or with help of eval.
Normal method definitions can still be included and will be treated as in a regular reify form, provided keys matching their names do not occur in the emit-map. The emit functions must return seqables (e.g. vectors) of method definitions in the format expected by reify: in this way, the case with multiple method definitions returned for one "magic method specifier" is relatively simple. If the iface argument were replaced with ifaces and ~iface with ~#ifaces in reify-from-maps' body, multiple interfaces could be specified for implementation.
Here's another approach, possibly easier to reason about:
(defn compile-atom-bean-converter [ifaces get-set-map]
(eval
(let [asym (gensym)]
`(fn [~asym]
(reify ~#ifaces
~#(apply concat
(for [[k [g s]] get-set-map]
[`(~g [~'this] (~k #~asym))
`(~s [~'this ~'v]
(swap! ~asym assoc ~k ~'v))])))))))
This calls on the compiler at runtime, which is somewhat expensive, but only needs to be done once per set of interfaces to be implemented. The result is a function which takes an atom as an argument and reifies a wrapper around the atom implementing the given interfaces with getters and setters as specified in the get-set-map argument. (Written this way, this is less flexible than the previous approach, but most of the code above could be reused here.)
Here's a sample interface and a getter/setter map:
(definterface IFunky
(getFoo [])
(^void setFoo [v])
(getFunkyBar [])
(^void setWeirdBar [v]))
(def gsm
'{:foo [getFoo setFoo]
:bar [getFunkyBar setWeirdBar]})
And some REPL interactions:
user> (def data {:foo 1 :bar 2})
#'user/data
user> (def atom-bean-converter (compile-atom-bean-converter '[IFunky] gsm))
#'user/atom-bean-converter
user> (def atom-bean (atom-bean-converter data))
#'user/atom-bean
user> (.setFoo data-bean 3)
nil
user> (.getFoo atom-bean)
3
user> (.getFunkyBar data-bean)
2
user> (.setWeirdBar data-bean 5)
nil
user> (.getFunkyBar data-bean)
5
The point is reify being a macro itself which is expanded before your own set-and-get macro - so the set-and-get approach doesn't work. So, instead of an inner macro inside reify, you need a macro on the "outside" that generates the reify, too.
Since the trick is to expand the body before reify sees it, a more general solution could be something along these lines:
(defmacro reify+ [& body]
`(reify ~#(map macroexpand-1 body)))
You can also try to force your macro to expand first:
(ns qqq (:use clojure.walk))
(defmacro expand-first [the-set & code] `(do ~#(prewalk #(if (and (list? %) (contains? the-set (first %))) (macroexpand-all %) %) code)))
(defmacro setter [setterf kw] `(~setterf [~'this ~'v] (swap! ~'data assoc ~kw ~'v)))
(defmacro getter [getterf kw] `(~getterf [~'this] (~kw #~'data)))
(expand-first #{setter getter}
(reify HugeInterface
(getter getX :x)
(setter setX :x)))
I've got a Clojure file with a lot of string constants. I want to collect these strings in a collection by wrapping them in a macro. After several tries I succeeded, but my solution looks fairly hideous.
(ns Memorable)
(def MEMORY (atom []))
(defmacro memorize [s] (swap! MEMORY conj s) s)
(prn (str (memorize "hello") " brave new " (memorize "world"))) ; test
(defmacro make-memories-constant [] `(def MEMORIES ~(deref MEMORY)))
(make-memories-constant)
Are there more elegant solutions for this problem?
As far as code cleanup is concerned, I'd remove the make-memories-constant macro -- you can just do
(def MEMORIES #MEMORY)
or even
(def MEMORY #MEMORY)
so as not to clutter your namespace with another Var. As long as you put this after all calls to memorize, this will save a snapshot of MEMORY with all memorized strings inside.
I'd say this is actually pretty clean, with the actual code executing at runtime no different from what it would have been without any memorize-ing...
Another approach would be to prepare a sort of "memorisation framework" as a separate namespace exporting a macros called setup-memorization, say, which could look something like this (just a rough sketch which won't work without some polishing... updated with a (not so thoroughly) tested version -- this actually works!... still, please tweak it to your needs):
(ns memorization-framework)
(defmacro setup-memorization []
(let [MEMORY (gensym "MEMORY")
MEMORIES (gensym "MEMORIES")]
`(do (def ~MEMORY (atom []))
(defmacro ~'memorize [s#] (swap! ~MEMORY conj s#) s#)
(defmacro ~'get-memory [] #~MEMORY)
(defmacro ~'defmemories [~MEMORIES]
`(do (def ~~MEMORIES #~~MEMORY)
(ns-unmap ~~*ns* '~'~'get-memory)
(ns-unmap ~~*ns* '~'~'memorize)
(ns-unmap ~~*ns* '~'~MEMORY)
~~MEMORIES)))))
Then you'd use the memorization-framework namespace, do (setup-memorization) towards the top of your namespace to set things up, call memorize like you do in your example code to memorise things and finally use end-memorization defmemories to store the string collection somewhere in a Var and remove the temporary Var used to store the atom used for construction-time storage along with the ability to make further calls to memorize from this namespace. (The end-memorization macro is meant to just return the collection to wherever you call it from when called with no arguments or define a new Var to store it if given a single argument, which must be a symbol to be used to name the Var. Update: I've only tested the defmemories version, so I'm leaving it here and removing the "in-place return" variant.)
An example interaction at the REPL while in namespace foo (note I defined setup-memorization in the user namespace):
foo> (user/setup-memorization)
#'foo/defmemories
foo> (get-memory)
[]
foo> (memorize "foo")
"foo"
foo> (memorize "bar")
"bar"
foo> (get-memory)
["foo" "bar"]
foo> (defmemories quux)
["foo" "bar"]
foo> quux
["foo" "bar"]
foo> (get-memory)
; Evaluation aborted.
foo> (memorize)
; Evaluation aborted.
Those "Evaluation aborted." messages indicate that calls to get-memory and memorize throw exceptions after defmemories has been called. This is by design, as mentioned above. Also, get-memory is here mostly to ease testing / debugging.
If you're only going to use all this once, the setup-memorization approach might be overkill, but I guess it'll remove some boilerplate for you if you use it more.