I came across that Clojure has clojure.core.reducers/fold function.
Also Scala has built-in fold function but could not understand that they are working differently or not?
I assume that you are talking about clojure.core.reducers/fold.
Scala's default fold implementation on sequences is very simple:
collection.fold(identityElem)(binOp)
simply starts with the identityElem and then traverses the collection sequentially, and applies the binary operation binOp to the already accumulated result and the current sequence value, e.g.
(1 to 3).fold(42000)(_ + _)
will result in 42000 + 1 + 2 + 3 = 42006.
Clojure's fold with the full signature
(r/fold n combinef reducef coll)
from the above mentioned package works in parallel in two stages. First, it splits the input into smaller groups of size n (approximately), then reduces each group using reducef, and finally combines the results of each group using combinef.
The main difference is that combinef is expected to be both a zeroary and binary at the same time (Clojure has multi-ary functions), and (combinef) (without arguments) will be invoked to produce identity elements for each partition (thus, this documentation is correct, and this documentation lies).
That is, in order to simulate Scala's fold from the above example, one would have to write something like this:
(require '[clojure.core.reducers :as r])
(r/fold 3 (fn ([] 42000) ([x y] y)) + [1 2 3])
And in general, Scala's fold
collection.fold(identityElement)(binOp)
can be emulated by reducers/fold as follows:
(r/fold collectionSize (fn ([] identityElem) ([x y] y)) binOp collection)
(note the ([x y] y) contraption that throws away the first argument, it's intentional).
I guess the interface wasn't intended to be used with any zero-binary operations that are not monoids, that's the reason why Scala's fold is so awkward to simulate using Clojure's fold. If you want something that behaves like Scala's fold, use reduce in Clojure.
EDIT
Oh, wait. The documentation actually states that
combinef must be associative, and, when called with no
arguments, (combinef) must produce its identity element
that is, we are actually forced to use a monoid as the combinef, so the above 42000, ([x y] y)-example is actually invalid, and the behavior is actually undefined. The fact that I somehow got the 42006 out was a hack in the strictly technical sense that it relied on undefined behavior of a library function to obtain the desired result 42006.
Taking this extra information into account, I'm not sure whether Scala's fold can be simulated by Clojure's core.reducers/fold at all. Clojure's fold seems to be constrained to reductions with a monoid, whereas Scala's fold is closer to the general List catamorphism, at the expense of parallelism.
The clojure.core.reducers namespace is a specialized implementation designed for parallel processing of large datasets. You can find full docs here:
https://clojure.org/reference/reducers.
(r/fold reducef coll)
(r/fold combinef reducef coll)
(r/fold n combinef reducef coll)
r/fold takes a reducible collection and partitions it into groups of
approximately n (default 512) elements. Each group is reduced using
the reducef function. The reducef function will be called with no
arguments to produce an identity value in each partition. The results
of those reductions are then reduced with the combinef (defaults to
reducef) function. When called with no arguments, (combinef) must
produce its identity element - this will be called multiple times.
Operations may be performed in parallel. Results will preserve order.
Until you are maxing out your machine, you should just stick to the basic reduce function:
https://clojuredocs.org/clojure.core/reduce
This is essentially the same as Scala's fold function:
(reduce + 0 [1 2 3 4 5]) => 15
where the function signature is:
(reduce <op> <init-val> <collection-to-be-reduced> )
Related
I'm using vectors to build a table for implement a dynamic program, it involves updating each element of the vector sequentially. But why is there no vector-set for immutable vectors? There is only vector-set! for mutable vectors, but we can see that there is dict-set and dict-set! for immutable and mutable dictionaries, also there is hash-set and hash-set! for immutable and mutable hash tables.
The reason why vector-set is missing, is to prevent people inadvertently using it without realizing the operation is O(n) and not O(1). Since vector-set! is O(1) it is not unlikely for someone to make this mistake.
Furthermore it is simple to write a vector-set when it is really needed:
#lang racket
(define (vector-set v i o)
(vector->immutable-vector
(for/vector ([j (in-range (vector-length v))])
(if (= i j)
o
(vector-ref v j)))))
(vector-set (vector-immutable 10 11 12 13) 2 'a)
Output:
'#(10 11 a 13)
As far as I understand aggregate is a generalisation of fold which in turn is a generalisation of reduce.
Similarily combineByKey is a generalisation of aggregateByKey which in turn is a generalisation of foldByKey which in turn is a generalisation of reduceByKey.
However I have trouble finding simple examples for each of those seven methods which can in turn only be expressed by them and not their less general versions. For example I found http://blog.madhukaraphatak.com/spark-rdd-fold/ giving an example for fold, but I have been able to use reduce in the same situation as well.
What I found out so far:
I read that the more generalised methods can be more efficient, but that would be a non-functional requirement and I would like to get examples which can not be implemented with the more specific method.
I also read that e.g. the function passed to fold only has to be associative, while the one for reduce has to be commutative additionally: https://stackoverflow.com/a/25158790/4533188 (However, I still don't know any good simple example.) whereas in https://stackoverflow.com/a/26635928/4533188 I read that fold needs both properties to hold...
We could think of the zero value as a feature (e.g. for fold over reduce) as in "add all elements and add 3" and using 3 as the zero value, but that would be misleading, because 3 would be added for each partition, not just once. Also this is simply not the purpose of fold as far as I understood - it wasn't meant as a feature, but as a necessity to implement it to be able to take non-commutative functions.
What would simple examples for those seven methods be?
Let's work through what is actually needed logically.
First, note that if your collection is unordered, any set of (binary) operations on it need to be both commutative and associative, or you'll get different answers depending on which (arbitrary) order you choose each time. Since reduce, fold, and aggregate all use binary operations, if you use these things on a collection that is unordered (or is viewed as unordered), everything must be commutative and associative.
reduce is an implementation of the idea that if you can take two things and turn them into one thing, you can collapse an arbitrarily long collection into a single element. Associativity is exactly the property that it doesn't matter how you pair things up as long as you eventually pair them all and keep the left-to-right order unchanged, so that's exactly what you need.
a b c d a b c d a b c d
a # b c d a # b c d a b # c d
(a#b) c # d (a#b) # c d a (b#c) d
(a#b) # (c#d) ((a#b)#c) # d a # ((b#c)#d)
All of the above are the same as long as the operation (here called #) is associative. There is no reason to swap around which things go on the left and which go on the right, so the operation does not need to be commutative (addition is: a+b == b+a; concat is not: ab != ba).
reduce is mathematically simple and requires only an associative operation
Reduce is limited, though, in that it doesn't work on empty collections, and in that you can't change the type. If you're working sequentially, you can a function that takes a new type and the old type, and produces something with the new type. This is a sequential fold (left-fold if the new type goes on the left, right-fold if it goes on the right). There is no choice about the order of operations here, so commutativity and associativity and everything are irrelevant. There's exactly one way to work through your list sequentially. (If you want your left-fold and right-fold to always be the same, then the operation must be associative and commutative, but since left- and right-folds don't generally get accidentally swapped, this isn't very important to ensure.)
The problem comes when you want to work in parallel. You can't sequentially go through your collection; that's not parallel by definition! So you have to insert the new type at multiple places! Let's call our fold operation #, and we'll say that the new type goes on the left. Furthermore, we'll say that we always start with the same element, Z. Now we could do any of the following (and more):
a b c d a b c d a b c d
Z#a b c d Z#a b Z#c d Z#a Z#b Z#c Z#d
(Z#a) # b c d (Z#a) # b (Z#c) # d
((Z#a)#b) # c d
(((Z#a)#b)#c) # d
Now we have a collection of one or more things of the new type. (If the original collection was empty, we just take Z.) We know what to do with that! Reduce! So we make a reduce operation for our new type (let's call it $, and remember it has to be associative), and then we have aggregate:
a b c d a b c d a b c d
Z#a b c d Z#a b Z#c d Z#a Z#b Z#c Z#d
(Z#a) # b c d (Z#a) # b (Z#c) # d Z#a $ Z#b Z#c $ Z#d
((Z#a)#b) # c d ((Z#a)#b) $ ((Z#c)#d) ((Z#a)$(Z#b)) $ ((Z#c)$(Z#d))
(((Z#a)#b)#c) # d
Now, these things all look really different. How can we make sure that they end up to be the same? There is no single concept that describes this, but the Z# operation has to be zero-like and $ and # have to be homomorphic, in that we need (Z#a)#b == (Z#a)$(Z#b). That's the actual relationship that you need (and it is technically very similar to a semigroup homomorphism). There are all sorts of ways to pick badly even if everything is associative and commutative. For example, if Z is the double value 0.0 and # is actually +, then Z is zero-like and # is associative and commutative. But if $ is actually *, which is also associative and commutative, everything goes wrong:
(0.0+2) * (0.0+3) == 2.0 * 3.0 == 6.0
((0.0+2) + 3) == 2.0 + 3 == 5.0
One example of a non-trival aggregate is building a collection, where # is the "append an element" operator and $ is the "concat two collections" operation.
aggregate is tricky and requires an associative reduce operation, plus a zero-like value and a fold-like operation that is homomorphic to the reduce
The bottom line is that aggregate is not simply a generalization of reduce.
But there is a simplification (less general form) if you're not actually changing the type. If Z is actually z and is an actual zero, we can just stick it in wherever we want and use reduce. Again, we don't need commutativity conceptually; we just stick in one or more z's and reduce, and our # and $ operations can be the same thing, namely the original # we used on the reduce
a b c d () <- empty
z#a z#b z
z#a (z#b)#c
z#a ((z#b)#c)#d
(z#a)#((z#b)#c)#d
If we just delete the z's from here, it works perfectly well, and in fact is equivalent to if (empty) z else reduce. But there's another way it could work too. If the operation # is also commutative, and z is not actually a zero but just occupies a fixed point of # (meaning z#z == z but z#a is not necessarily just a), then you can run the same thing, and since commutivity lets you switch the order around, you conceptually can reorder all the z's together at the beginning, and then merge them all together.
And this is a parallel fold, which is really a rather different beast than a sequential fold.
(Note that neither fold nor aggregate are strictly generalizations of reduce even for unordered collections where operations have to be associative and commutative, as some operations do not have a sensible zero! For instance, reducing strings by shortest length has as its "zero" the longest possible string, which conceptually doesn't exist, and practically is an absurd waste of memory.)
fold requires an associative reduce operation plus either a zero value or a reduce operation that's commutative plus a fixed-point value
Now, when would you ever use a parallel fold that wasn't just a reduceOrElse(zero)? Probably never, actually, though they can exist. For example, if you have a ring, you often have fixed points of the type we need. For instance, 10 % 45 == (10*10) % 45, and * is both associative and commutative in integers mod 45. Thus, if our collection is numbers mod 45, we can fold with a "zero" of 10 and an operation of *, and parallelize however we please while still getting the same result. Pretty weird.
However, note that you can just plug the zero and operation of fold into aggregate and get exactly the same result, so aggregate is a proper generalization of fold.
So, bottom line:
Reduce requires only an associative merge operation, but doesn't change the type, and doesn't work on empty collecitons.
Parallel fold tries to extend reduce but requires a true zero, or a fixed point and the merge operation must be commutative.
Aggregate changes the type by (conceptually) running sequential folds followed by a (parallel) reduce, but there are complex relationships between the reduce operation and the fold operation--basically they have to be doing "the same thing".
An unordered collection (e.g. a set) always requires an associative and commutative operation for any of the above.
With regard to the byKey stuff: it's just the same as this, except it only applies it to the collection of values associated with a (potentially repeated) key.
If Spark actually requires commutativity where the above analysis does not suggest it's needed, one could reasonably consider that a bug (or at least an unnecessary limitation of the implementation, given that operations like map and filter preserve order on ordered RDDs).
the function passed to fold only has to be associative, while the one for reduce has to be commutative additionally.
It is not correct. fold on RDDs requires the function to be commutative as well. It is not the same operation as fold on Iterable what is pretty well described in the official documentation:
This behaves somewhat differently from fold operations implemented for non-distributed
collections in functional languages like Scala.
This fold operation may be applied to
partitions individually, and then fold those results into the final result, rather than
apply the fold to each element sequentially in some defined ordering. For functions
that are not commutative, the result may differ from that of a fold applied to a
non-distributed collection.
As you can see order of merging partial values is not part of the contract hence function which is used for fold has to be commutative.
I read that the more generalised methods can be more efficient
Technically speaking there should be no significant difference. For fold vs reduce you can check my answers to reduce() vs. fold() in Apache Spark and Why is the fold action necessary in Spark?
Regarding *byKey methods all are implemented using the same basic construct which is combineByKeyWithClassTag and can be reduced to three simple operations:
createCombiner - create "zero" value for a given partition
mergeValue - merge values into accumulator
mergeCombiners - merge accumulators created for each partition.
I am trying to test some of the lambda calculus functions that I wrote using Racket but not having much luck with the testcases. For example given a definition
; successor function
(define my_succ (λ (one)
(λ (two)
(λ (three)
(two ((one two) three))))))
I am trying to apply it to 1 2 3, expecting the successor of 2 to be 3 by doing
(((my_succ 1) 2) 3)
logic being that since my_succ is a function that takes one arg and passes it to another function that takes one arg which passes it to the third function that takes one arg. But I get
application: not a procedure;
expected a procedure that can be applied to arguments
given: 1
arguments.:
I tried Googling and found a lot of code for the rules, but no examples of application of these rules. How should I call the above successor function in order to test it?
You are mixing two completely different things: lambda terms and functions in Racket.
In Racket you can have anonymous functions, that can be written in the λ notation (like (λ(x) (+ x 1)) that returns the successor of an integer, so that ((λ(x) (+ x 1)) 1) returns 2),
in Pure Lambda Calculus you have only lambda terms, that are written with in a similar notation, and that can be interpreted as functions.
In the second domain, you do not have natural numbers like 0, 1, 2, ..., but you have only lambda terms, and represent numbers as such. For instance, if you use the so-called Church numerals, you represent (in technical term encode) the number 0 with the lambda term λf.λx.x, 1 with λf.λx.f x, 2 with λf.λx.f (f x) and so on.
So, the function successor (for numbers represented with this encoding) correspond to a term which, in Racket notation, is the function that you have written, but you cannot apply it to numbers like 0, 1, etc., but only to other lambda expressions, that is you could write something like this:
(define zero (λ(f) (λ (x) x))) ; this correspond to λf.λx.x
(successor zero)
The result in Racket is a procedure (it will be printed as: #<procedure>), but if you try to test that your result is correct, comparing it with the functional encoding of 1, you will find something strange. In fact:
(equal? (successor zero) (λ(f) (λ(x) (f x))))
produces #f, since if you compare two procedures in Racket you obtain always false (e.g. (equal? (λ(x)x) (λ(x)x)) produces #f), unless you compare the “identical” (in the sense of “same memory cell”) value ((equal? zero zero) gives #t). This is due to the fact that, for comparing correctly two functions, you should compare infinite sets of couples (input, output)!
Another possibility would be representing lambda terms as some kind of structure in Racket, so you can represent Church numerals, as well as "normal" lambda terms, and define a function apply (or better reduce) the perform lambda-reduction.
You are trying to apply currying.
(define my_succ
(lambda(x)(
lambda(y)(
lambda(z)(
(f x y z)))))
(define (add x y z)
(+ x y z))
((( (my_succ add)1)2)3)
Implementation in DR Racket:
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
What is the algorithm used by programming languages to eval their ASTs?
That is, suppose we have 4 basic functions, /*+-. What is a basic algorithm that will correctly eval any AST in the form of, for example:
(+ (- (* 3 2) (+ (/ 5 2))) (* 2 4))
My doubt is actually what happens if the evaluation of a node returns something that still have to be evaluated. For example, in Scheme, the evaluation of ((lambda (a) (+ a 2)) 3) would be (+ 3 2). But this could be evaluated again into 5. So how does the language determine when to stop evaluating a form?
You're totally misunderstanding how Scheme/Lisp evaluation works. I'll use the example you gave:
(+ (- (* 3 2) (+ (/ 5 2))) (* 2 4))
To evaluate a list, we evaluate each of the elements. The first is expected to return a procedure (I'm ignoring the special case of syntax operators), the rest can return arbitrary values. We call the procedure with the rest as arguments.
At the top level, this is a list of 3 elements:
+
(- (* 3 2) (+ (/ 5 2)))
(* 2 4)
Each of these is evaluated. The first is a variable whose value is a procedure (Scheme's built-in addition function). The others, being lists, require recursion into the evaluation algorithm. I'm going to skip describing the second one, because of its complexity, and go to the third: (* 2 4).
This is a list of 3 elements: *, 2, and 4. As above, * is the multiplication function. 2 and 4 are literals, so they evaluate to themselves. So we call the multiplication function with the arguments 2 and 4, and it returns 8.
The complicated second argument goes through the same process, just with several more levels of recursion. It eventually returns 4. So we then call the multiplication function with the arguments 4 and 8, and it returns 32.
Your second example is processed similarly. At the top, you have a list of two elements:
(lambda (a) (+ a 2))
3
Each of these is evaluated. Lambda is special syntax that parses its contents and returns a procedure that evaluates its body in a context where the parameter variables are bound to arguments, so the first returns a procedure that adds 2 to its argument and returns that. 3 is a literal, so it just returns the number 3. We then call the procedure with the argument 3, it adds 2 to it and returns 5.
In the case you give, the execution will stop at 5, since it is a literal value and represents itself. This is not hard to test for. You might as well ask how a function that traverses a list in depth knows how to stop (in fact, you should, since in Scheme this is the same thing).
In Scheme, any compound expression should eventually resolve to one of the 7 primitive datatypes or the empty list, unless it becomes trapped in an infinite loop. If you want to know in advance if the expression will resolve, well, that's an interesting problem: http://en.wikipedia.org/wiki/Halting_problem
I think you may be asking the wrong question, but I will try:
Until it gets a result that it can work with. In your example you're asking about when an interpeter stops evaluating an expression... its 100% language depedent and would be a completely different answer if you were to ask about a compiler. For your Scheme example, you would need to read the Scheme specification (R5RS).
So it is defined by the writer of the interpreter. If a single literal (or even variable) is the expected result of an expression in my language, then it would stop there.
There are many different algorithms.
Alternative 1: You could compile the AST to an intermediate representation which is more linear. Your code could be compiled to something like the following:
a <- 3 * 2
b <- 5 / 2
c <- a - b
d <- 2 * 4
e <- c + d
return e
This is easy to evaluate, since it is just a sequence of instructions. Most of the instructions have the same format: X <- Y OP Z, so the evaluator will be very simple.
Alternative 2: You can compile alternative #1 to machine code or byte code.
li r3, 3
muli r3, 2
li r4, 5
divi r4, r5, 2
subf r3, r3, r4
li r4, 2
muli r4, r4, 4
add r3, r3, r4
blr
Alternative 3: You can compile alternative #1 to a special form called SSA, or "single static assignment", which is similar to #1 but the LHS of every assignment is unique, and special "phi" nodes are used to combine values from different branches. SSA can then be compiled to machine code or byte code.
Alternative 4: You can evaluate the AST by recursive descent. This is covered thoroughly in most books on Scheme / Lisp.
Alternative 5: You can use recursive descent to convert the code to stack machine code, and then evaluate that. Something like:
push 3
push 2
mul
push 5
push 2
div
sub
push 2
push 4
mul
add
ret
Alternative ∞: There are plenty of other techniques. The books written on this subject are thick.
I have this problem to work on:
The sum higher order procedure can be generalised even further to capture the idea of combining terms with a fixed operator. The mathematical product operator is a specific example of this idea, with multiplication replacing the addition of the summation operator.
The procedure accumulate, started below, is intended to capture this idea. The combiner parameter represents the operator that is used to reduce the terms, and the base parameter represents the value that is returned when there are no terms left to be combined. For example, if we have already implemented the accumulate procedure, then we could define the sum procedure as:
(define sum (accumulate + 0))
Complete the definition of accumulate so that it behaves according to this description.
(define accumulate
(lambda (combiner base)
(lambda (term start next stop)
(if (> start stop)
...
...))))
I inserted as the last two lines:
base
(combiner base (accumulate (combiner start stop) start next stop))
but, I have no idea if this is correct nor how to actually use the sum procedure to call accumulate and hence sum up numbers.
This is a great way to learn how to fish. Much better
than being given a fish.
Until then, here's how to approach the problem. Write a
function which would do what (accumulate + 0) would do. Don't use the accumulate function; just write a defun which which does what your homework asks. Next, write a function which would do what (accumulate * 1) would do. What are the similarities, what are the differences between the two functions. For the most part, they should be identical except for the occurrence of the + and * operators.
Next, note that the accumulate function is to return a function which will look a lot like the two functions you wrote earlier. Now, using the insight that two functions you wrote are very similar, think how to apply that to the function which (defun accumulate ...) is to return.