This weekend I decided to try my hand at some Scala and Clojure. I'm proficient with object oriented programming, and so Scala was easy to pick up as a language, but wanted to try out functional programming. This is where it got hard.
I just can't seem to get my head into a mode of writing functions. As an expert functional programmer, how do you approach a problem?
Given a list of values and a defined period of summation, how would you generate a new list of the simple moving average of the list?
For example: Given the list values (2.0, 4.0, 7.0, 6.0, 3.0, 8.0, 12.0, 9.0, 4.0, 1.0), and the period 4, the function should return: (0.0, 0.0, 0.0, 4.75, 5.0, 6.0, 7.25, 8.0, 8.25, 6.5)
After spending a day mulling it over, the best I could come up with in Scala was this:
def simpleMovingAverage(values: List[Double], period: Int): List[Double] = {
(for (i <- 1 to values.length)
yield
if (i < period) 0.00
else values.slice(i - period, i).reduceLeft(_ + _) / period).toList
}
I know this is horribly inefficient, I'd much rather do something like:
where n < period: ma(n) = 0
where n = period: ma(n) = sum(value(1) to value(n)) / period
where n > period: man(n) = ma(n -1) - (value(n-period) / period) + (value(n) / period)
Now that would be easily done in a imperative style, but I can't for the life of me work out how to express that functionally.
Interesting problem. I can think of many solutions, with varying degrees of efficiency. Having to add stuff repeatedly isn't really a performance problem, but let's assume it is. Also, the zeroes at the beginning can be prepended later, so let's not worry about producing them. If the algorithm provides them naturally, fine; if not, we correct it later.
Starting with Scala 2.8, the following would give the result for n >= period by using sliding to get a sliding window of the List:
def simpleMovingAverage(values: List[Double], period: Int): List[Double] =
List.fill(period - 1)(0.0) ::: (values sliding period map (_.sum) map (_ / period))
Nevertheless, although this is rather elegant, it doesn't have the best performance possible, because it doesn't take advantage of already computed additions. So, speaking of them, how can we get them?
Let's say we write this:
values sliding 2 map sum
We have a list of the sum of each two pairs. Let's try to use this result to compute the moving average of 4 elements. The above formula made the following computation:
from d1, d2, d3, d4, d5, d6, ...
to (d1+d2), (d2+d3), (d3+d4), (d4+d5), (d5+d6), ...
So if we take each element and add it to the second next element, we get the moving average for 4 elements:
(d1+d2)+(d3+d4), (d2+d3)+(d4+d5), (d3+d4)+(d5+d6), ...
We may do it like this:
res zip (res drop 2) map Function.tupled(_+_)
We could then compute the moving average for 8 elements, and so on. Well, there is a well known algorithm to compute things that follow such pattern. It's most known for its use on computing the power of a number. It goes like this:
def power(n: Int, e: Int): Int = e match {
case 0 => 1
case 1 => n
case 2 => n * n
case odd if odd % 2 == 1 => power(n, (odd - 1)) * n
case even => power(power(n, even / 2), 2)
}
So, let's apply it here:
def movingSum(values: List[Double], period: Int): List[Double] = period match {
case 0 => throw new IllegalArgumentException
case 1 => values
case 2 => values sliding 2 map (_.sum)
case odd if odd % 2 == 1 =>
values zip movingSum(values drop 1, (odd - 1)) map Function.tupled(_+_)
case even =>
val half = even / 2
val partialResult = movingSum(values, half)
partialResult zip (partialResult drop half) map Function.tupled(_+_)
}
So, here's the logic. Period 0 is invalid, period 1 is equal to the input, period 2 is sliding window of size 2. If greater than that, it may be even or odd.
If odd, we add each element to the movingSum of the next (odd - 1) elements. For example, if 3, we add each element to the movingSum of the next 2 elements.
If even, we compute the movingSum for n / 2, then add each element to the one n / 2 steps afterwards.
With that definition, we can then go back to the problem and do this:
def simpleMovingAverage(values: List[Double], period: Int): List[Double] =
List.fill(period - 1)(0.0) ::: (movingSum(values, period) map (_ / period))
There's a slight inefficiency with regards to the use of :::, but it's O(period), not O(values.size). It can be made more efficient with a tail recursive function. And, of course, the definition of "sliding" I provided is horrendous performance-wise, but there will be a much better definition of it on Scala 2.8. Note that we can't make an efficient sliding method on a List, but we can do it on an Iterable.
Having said all that, I'd go with the very first definition, and optimize only if a critical path analysis pinpointed this as a big deal.
To conclude, let's consider how I went about the problem. We have a moving average problem. A moving average is the sum of a moving "window" on a list, divided by the size of that window. So, first, I try to get a sliding window, sum everything on it, and then divide by the size.
The next problem was to avoid repetition of already computed additions. In this case, I went to the smallest addition possible, and tried to figure out how to compute bigger sums reusing such results.
Finally, let's try to solve the problem the way you figured it, by adding and subtracting from the previous result. Getting the first average is easy:
def movingAverage(values: List[Double], period: Int): List[Double] = {
val first = (values take period).sum / period
Now we make two lists. First, the list of elements to be subtracted. Next, the list of elements to be added:
val subtract = values map (_ / period)
val add = subtract drop period
We can add these two lists by using zip. This method will only produce as many elements as the smaller list has, which avoids the problem of subtract being bigger than necessary:
val addAndSubtract = add zip subtract map Function.tupled(_ - _)
We finish by composing the result with a fold:
val res = (addAndSubtract.foldLeft(first :: List.fill(period - 1)(0.0)) {
(acc, add) => (add + acc.head) :: acc
}).reverse
which is the answer to be returned. The whole function looks like this:
def movingAverage(values: List[Double], period: Int): List[Double] = {
val first = (values take period).sum / period
val subtract = values map (_ / period)
val add = subtract drop period
val addAndSubtract = add zip subtract map Function.tupled(_ - _)
val res = (addAndSubtract.foldLeft(first :: List.fill(period - 1)(0.0)) {
(acc, add) => (add + acc.head) :: acc
}).reverse
res
}
I know Clojure better than Scala, so here goes. As I write this the other Clojure entry here is imperative; that's not really what you're after (and isn't idiomatic Clojure). The first algorithm that comes to my mind is repeatedly taking the requested number of elements from the sequence, dropping the first element, and recurring.
The following works on any kind of sequence (vector or list, lazy or not) and gives a lazy sequence of averages---which could be helpful if you're working on a list of indefinite size. Note that it takes care of the base case by implicitly returning nil if there aren't enough elements in the list to consume.
(defn moving-average [values period]
(let [first (take period values)]
(if (= (count first) period)
(lazy-seq
(cons (/ (reduce + first) period)
(moving-average (rest values) period))))))
Running this on your test data yields
user> (moving-average '(2.0, 4.0, 7.0, 6.0, 3.0, 8.0, 12.0, 9.0, 4.0, 1.0) 4)
(4.75 5.0 6.0 7.25 8.0 8.25 6.5)
It doesn't give "0" for the first few elements in the sequence, though that could easily be handled (somewhat artificially).
The easiest thing of all is to see the pattern and be able to bring to mind an available function that fits the bill. partition gives a lazy view of portions of a sequence, which we can then map over:
(defn moving-average [values period]
(map #(/ (reduce + %) period) (partition period 1 values))
Someone asked for a tail recursive version; tail recursion vs. laziness is a bit of a tradeoff. When your job is building up a list then making your function tail recursive is usually pretty simple, and this is no exception---just build up the list as an argument to a subfunction. We'll accumulate to a vector instead of a list because otherwise the list will be built up backwards and will need to be reversed at the end.
(defn moving-average [values period]
(loop [values values, period period, acc []]
(let [first (take period values)]
(if (= (count first) period)
(recur (rest values) period (conj acc (/ (reduce + first) period)))
acc))))
loop is a way to make an anonymous inner function (sort of like Scheme's named let); recur must be used in Clojure to eliminate tail calls. conj is a generalized cons, appending in the manner natural for the collection---the beginning of lists and the end of vectors.
Here is another (functional) Clojure solution:
(defn avarage [coll]
(/ (reduce + coll)
(count coll)))
(defn ma [period coll]
(map avarage (partition period 1 coll)))
The zeros at the beginning of the sequence must still be added if that is a requirement.
Here's a purely functional solution in Clojure. More complex than those already provided, but it is lazy and only adjusts the average at each step, instead of recalculating it from scratch. It's actually slower than a simple solution which calculates a new average at each step if the period is small; for larger periods, however, it experiences virtually no slowdown, whereas something doing (/ (take period ...) period) will perform worse for longer periods.
(defn moving-average
"Calculates the moving average of values with the given period.
Returns a lazy seq, works with infinite input sequences.
Does not include initial zeros in the output."
[period values]
(let [gen (fn gen [last-sum values-old values-new]
(if (empty? values-new)
nil
(let [num-out (first values-old)
num-in (first values-new)
new-sum (+ last-sum (- num-out) num-in)]
(lazy-seq
(cons new-sum
(gen new-sum
(next values-old)
(next values-new)))))))]
(if (< (count (take period values)) period)
nil
(map #(/ % period)
(gen (apply + (take (dec period) values))
(cons 0 values)
(drop (dec period) values))))))
Here's a partially point-free one line Haskell solution:
ma p = reverse . map ((/ (fromIntegral p)) . sum . take p) . (drop p) . reverse . tails
First it applies tails to the list to get the "tails" lists, so:
Prelude List> tails [2.0, 4.0, 7.0, 6.0, 3.0]
[[2.0,4.0,7.0,6.0,3.0],[4.0,7.0,6.0,3.0],[7.0,6.0,3.0],[6.0,3.0],[3.0],[]]
Reverses it and drops the first 'p' entries (taking p as 2 here):
Prelude List> (drop 2 . reverse . tails) [2.0, 4.0, 7.0, 6.0, 3.0]
[[6.0,3.0],[7.0,6.0,3.0],[4.0,7.0,6.0,3.0],[2.0,4.0,7.0,6.0,3.0]]
In case you aren't familiar with the (.) dot/nipple symbol, it is the operator for 'functional composition', meaning it passes the output of one function as the input of another, "composing" them into a single function. (g . f) means "run f on a value then pass the output to g", so ((f . g) x) is the same as (g(f x)). Generally its usage leads to a clearer programming style.
It then maps the function ((/ (fromIntegral p)) . sum . take p) onto the list. So for every list in the list it takes the first 'p' elements, sums them, then divides them by 'p'. Then we just flip the list back again with "reverse".
Prelude List> map ((/ (fromIntegral 2)) . sum . take 2) [[6.0,3.0],[7.0,6.0,3.0]
,[4.0,7.0,6.0,3.0],[2.0,4.0,7.0,6.0,3.0]]
[4.5,6.5,5.5,3.0]
This all looks a lot more inefficient than it is; "reverse" doesn't physically reverse the order of a list until the list is evaluated, it just lays it out onto the stack (good ol' lazy Haskell). "tails" also doesn't create all those separate lists, it just references different sections of the original list. It's still not a great solution, but it one line long :)
Here's a slightly nicer but longer solution that uses mapAccum to do a sliding subtraction and addition:
ma p l = snd $ mapAccumL ma' a l'
where
(h, t) = splitAt p l
a = sum h
l' = (0, 0) : (zip l t)
ma' s (x, y) = let s' = (s - x) + y in (s', s' / (fromIntegral p))
First we split the list into two parts at "p", so:
Prelude List> splitAt 2 [2.0, 4.0, 7.0, 6.0, 3.0]
([2.0,4.0],[7.0,6.0,3.0])
Sum the first bit:
Prelude List> sum [2.0, 4.0]
6.0
Zip the second bit with the original list (this just pairs off items in order from the two lists). The original list is obviously longer, but we lose this extra bit:
Prelude List> zip [2.0, 4.0, 7.0, 6.0, 3.0] [7.0,6.0,3.0]
[(2.0,7.0),(4.0,6.0),(7.0,3.0)]
Now we define a function for our mapAccum(ulator). mapAccumL is the same as "map", but with an extra running state/accumulator parameter, which is passed from the previous "mapping" to the next one as map runs through the list. We use the accumulator as our moving average, and as our list is formed of the element that has just left the sliding window and the element that just entered it (the list we just zipped), our sliding function takes the first number 'x' away from the average and adds the second number 'y'. We then pass the new 's' along and return 's' divided by 'p'. "snd" (second) just takes the second member of a pair (tuple), which is used to take the second return value of mapAccumL, as mapAccumL will return the accumulator as well as the mapped list.
For those of you not familiar with the $ symbol, it is the "application operator". It doesn't really do anything but it has a has "low, right-associative binding precedence", so it means you can leave out the brackets (take note LISPers), i.e. (f x) is the same as f $ x
Running (ma 4 [2.0, 4.0, 7.0, 6.0, 3.0, 8.0, 12.0, 9.0, 4.0, 1.0]) yields [4.75, 5.0, 6.0, 7.25, 8.0, 8.25, 6.5] for either solution.
Oh and you'll need to import the module "List" to compile either solution.
Here are 2 more ways to do moving average in Scala 2.8.0(one strict and one lazy). Both assume there are at least p Doubles in vs.
// strict moving average
def sma(vs: List[Double], p: Int): List[Double] =
((vs.take(p).sum / p :: List.fill(p - 1)(0.0), vs) /: vs.drop(p)) {(a, v) =>
((a._1.head - a._2.head / p + v / p) :: a._1, a._2.tail)
}._1.reverse
// lazy moving average
def lma(vs: Stream[Double], p: Int): Stream[Double] = {
def _lma(a: => Double, vs1: Stream[Double], vs2: Stream[Double]): Stream[Double] = {
val _a = a // caches value of a
_a #:: _lma(_a - vs2.head / p + vs1.head / p, vs1.tail, vs2.tail)
}
Stream.fill(p - 1)(0.0) #::: _lma(vs.take(p).sum / p, vs.drop(p), vs)
}
scala> sma(List(2.0, 4.0, 7.0, 6.0, 3.0, 8.0, 12.0, 9.0, 4.0, 1.0), 4)
res29: List[Double] = List(0.0, 0.0, 0.0, 4.75, 5.0, 6.0, 7.25, 8.0, 8.25, 6.5)
scala> lma(Stream(2.0, 4.0, 7.0, 6.0, 3.0, 8.0, 12.0, 9.0, 4.0, 1.0), 4).take(10).force
res30: scala.collection.immutable.Stream[Double] = Stream(0.0, 0.0, 0.0, 4.75, 5.0, 6.0, 7.25, 8.0, 8.25, 6.5)
The J programming language facilitates programs such as moving average. Indeed, there are fewer characters in (+/ % #)\ than in their label, 'moving average.'
For the values specified in this question (including the name 'values') here is a straightforward way to code this:
values=: 2 4 7 6 3 8 12 9 4 1
4 (+/ % #)\ values
4.75 5 6 7.25 8 8.25 6.5
We can describe this by using labels for components.
periods=: 4
average=: +/ % #
moving=: \
periods average moving values
4.75 5 6 7.25 8 8.25 6.5
Both examples use exactly the same program. The only difference is the use of more names in the second form. Such names can help readers who don't know the J primaries.
Let's look a bit further into what's going on in the subprogram, average. +/ denotes summation (Σ) and % denotes division (like the classical sign ÷). Calculating a tally (count) of items is done by # . The overall program, then, is the sum of values divided by the tally of values: +/ % #
The result of the moving-average calculation written here does not include the leading zeros expected in the original question. Those zeros are arguably not part of the intended calculation.
The technique used here is called tacit programming. It is pretty much the same as the point-free style of functional programming.
Here is Clojure pretending to be a more functional language. This is fully tail-recursive, btw, and includes leading zeroes.
(defn moving-average [period values]
(loop [[x & xs] values
window []
ys []]
(if (and (nil? x) (nil? xs))
;; base case
ys
;; inductive case
(if (< (count window) (dec period))
(recur xs (conj window x) (conj ys 0.0))
(recur xs
(conj (vec (rest window)) x)
(conj ys (/ (reduce + x window) period)))))))
(deftest test-moving-average
(is (= [0.0 0.0 0.0 4.75 5.0 6.0 7.25 8.0 8.25 6.5]
(moving-average 4 [2.0 4.0 7.0 6.0 3.0 8.0 12.0 9.0 4.0 1.0]))))
Usually I put the collection or list parameter last to make the function easier to curry. But in Clojure...
(partial moving-average 4)
... is so cumbersome, I usually end up doing this ...
#(moving-average 4 %)
... in which case, it doesn't really matter what order the parameters go.
Here's a clojure version:
Because of the lazy-seq, it's perfectly general and won't blow stack
(defn partialsums [start lst]
(lazy-seq
(if-let [lst (seq lst)]
(cons start (partialsums (+ start (first lst)) (rest lst)))
(list start))))
(defn sliding-window-moving-average [window lst]
(map #(/ % window)
(let [start (apply + (take window lst))
diffseq (map - (drop window lst) lst)]
(partialsums start diffseq))))
;; To help see what it's doing:
(sliding-window-moving-average 5 '(1 2 3 4 5 6 7 8 9 10 11))
start = (+ 1 2 3 4 5) = 15
diffseq = - (6 7 8 9 10 11)
(1 2 3 4 5 6 7 8 9 10 11)
= (5 5 5 5 5 5)
(partialsums 15 '(5 5 5 5 5 5) ) = (15 20 25 30 35 40 45)
(map #(/ % 5) (20 25 30 35 40 45)) = (3 4 5 6 7 8 9)
;; Example
(take 20 (sliding-window-moving-average 5 (iterate inc 0)))
This example makes use of state, since to me it's a pragmatic solution in this case, and a closure to create the windowing averaging function:
(defn make-averager [#^Integer period]
(let [buff (atom (vec (repeat period nil)))
pos (atom 0)]
(fn [nextval]
(reset! buff (assoc #buff #pos nextval))
(reset! pos (mod (+ 1 #pos) period))
(if (some nil? #buff)
0
(/ (reduce + #buff)
(count #buff))))))
(map (make-averager 4)
[2.0, 4.0, 7.0, 6.0, 3.0, 8.0, 12.0, 9.0, 4.0, 1.0])
;; yields =>
(0 0 0 4.75 5.0 6.0 7.25 8.0 8.25 6.5)
It is still functional in the sense of making use of first class functions, though it is not side-effect free. The two languages you mentioned both run on top of the JVM and thus both allow for state-management when necessary.
This solution is in Haskell, which is more familiar to me:
slidingSums :: Num t => Int -> [t] -> [t]
slidingSums n list = case (splitAt (n - 1) list) of
(window, []) -> [] -- list contains less than n elements
(window, rest) -> slidingSums' list rest (sum window)
where
slidingSums' _ [] _ = []
slidingSums' (hl : tl) (hr : tr) sumLastNm1 = sumLastN : slidingSums' tl tr (sumLastN - hl)
where sumLastN = sumLastNm1 + hr
movingAverage :: Fractional t => Int -> [t] -> [t]
movingAverage n list = map (/ (fromIntegral n)) (slidingSums n list)
paddedMovingAverage :: Fractional t => Int -> [t] -> [t]
paddedMovingAverage n list = replicate (n - 1) 0 ++ movingAverage n list
Scala translation:
def slidingSums1(list: List[Double], rest: List[Double], n: Int, sumLastNm1: Double): List[Double] = rest match {
case Nil => Nil
case hr :: tr => {
val sumLastN = sumLastNm1 + hr
sumLastN :: slidingSums1(list.tail, tr, n, sumLastN - list.head)
}
}
def slidingSums(list: List[Double], n: Int): List[Double] = list.splitAt(n - 1) match {
case (_, Nil) => Nil
case (firstNm1, rest) => slidingSums1(list, rest, n, firstNm1.reduceLeft(_ + _))
}
def movingAverage(list: List[Double], n: Int): List[Double] = slidingSums(list, n).map(_ / n)
def paddedMovingAverage(list: List[Double], n: Int): List[Double] = List.make(n - 1, 0.0) ++ movingAverage(list, n)
A short Clojure version that has the advantage of being O(list length) regardless of your period:
(defn moving-average [list period]
(let [accums (let [acc (atom 0)] (map #(do (reset! acc (+ #acc %1 ))) (cons 0 list)))
zeros (repeat (dec period) 0)]
(concat zeros (map #(/ (- %1 %2) period) (drop period accums) accums))))
This exploits the fact that you can calculate the sum of a range of numbers by creating a cumulative sum of the sequence (e.g. [1 2 3 4 5] -> [0 1 3 6 10 15]) and then subtracting the two numbers with an offset equal to your period.
It looks like you are looking for a recursive solution. In that case, I would suggest to slightly change the problem and aim for getting (4.75, 5.0, 6.0, 7.25, 8.0, 8.25, 6.5, 0.0, 0.0, 0.0) as a solution.
In that case, you can write the below elegant recursive solution in Scala:
def mavg(values: List[Double], period: Int): List[Double] = {
if (values.size < period) List.fill(values.size)(0.0) else
if (values.size == period) (values.sum / values.size) :: List.fill(period - 1)(0.0) else {
val rest: List[Double] = mavg(values.tail, period)
(rest.head + ((values.head - values(period))/period)):: rest
}
}
I know how I would do it in python (note: the first 3 elements with the values 0.0 are not returned since that is actually not the appropriate way to represent a moving average). I would imagine similar techniques will be feasible in Scala. Here are multiple ways to do it.
data = (2.0, 4.0, 7.0, 6.0, 3.0, 8.0, 12.0, 9.0, 4.0, 1.0)
terms = 4
expected = (4.75, 5.0, 6.0, 7.25, 8.0, 8.25, 6.5)
# Method 1 : Simple. Uses slices
assert expected == \
tuple((sum(data[i:i+terms])/terms for i in range(len(data)-terms+1)))
# Method 2 : Tracks slots each of terms elements
# Note: slot, and block mean the same thing.
# Block is the internal tracking deque, slot is the final output
from collections import deque
def slots(data, terms):
block = deque()
for datum in data :
block.append(datum)
if len(block) > terms : block.popleft()
if len(block) == terms :
yield block
assert expected == \
tuple(sum(slot)/terms for slot in slots(data, terms))
# Method 3 : Reads value one at a time, computes the sums and throws away read values
def moving_average((avgs, sums),val):
sums = tuple((sum + val) for sum in sums)
return (avgs + ((sums[0] / terms),), sums[1:] + (val,))
assert expected == reduce(
moving_average,
tuple(data[terms-1:]),
((),tuple(sum(data[i:terms-1]) for i in range(terms-1))))[0]
# Method 4 : Semantically same as method 3, intentionally obfuscates just to fit in a lambda
assert expected == \
reduce(
lambda (avgs, sums),val: tuple((avgs + ((nsum[0] / terms),), nsum[1:] + (val,)) \
for nsum in (tuple((sum + val) for sum in sums),))[0], \
tuple(data[terms-1:]),
((),tuple(sum(data[i:terms-1]) for i in range(terms-1))))[0]
Being late on the party, and new to functional programming too, I came to this solution with an inner function:
def slidingAvg (ixs: List [Double], len: Int) = {
val dxs = ixs.map (_ / len)
val start = (0.0 /: dxs.take (len)) (_ + _)
val head = List.make (len - 1, 0.0)
def addAndSub (sofar: Double, from: Int, to: Int) : List [Double] =
if (to >= dxs.length) Nil else {
val current = sofar - dxs (from) + dxs (to)
current :: addAndSub (current, from + 1, to + 1)
}
head ::: start :: addAndSub (start, 0, len)
}
val xs = List(2, 4, 7, 6, 3, 8, 12, 9, 4, 1)
slidingAvg (xs.map (1.0 * _), 4)
I adopted the idea, to divide the whole list by the period (len) in advance.
Then I generate the sum to start with for the len-first-elements.
And I generate the first, invalid elements (0.0, 0.0, ...) .
Then I recursively substract the first and add the last value.
In the end I listify the whole thing.
In Haskell pseudocode:
group4 (a:b:c:d:xs) = [a,b,c,d] : group4 (b:c:d:xs)
group4 _ = []
avg4 xs = sum xs / 4
running4avg nums = (map avg4 (group4 nums))
or pointfree
runnig4avg = map avg4 . group4
(Now one really should abstract the 4 out ....)
Using Haskell:
movingAverage :: Int -> [Double] -> [Double]
movingAverage n xs = catMaybes . (fmap avg . take n) . tails $ xs
where avg list = case (length list == n) -> Just . (/ (fromIntegral n)) . (foldl (+) 0) $ list
_ -> Nothing
The key is the tails function, which maps a list to a list of copies of the original list, with the property that the n-th element of the result is missing the first n-1 elements.
So
[1,2,3,4,5] -> [[1,2,3,4,5], [2,3,4,5], [3,4,5], [4,5], [5], []]
We apply fmap (avg . take n) to the result, which means we take the n-length prefix from the sublist, and compute its avg. If the length of the list we are avg'ing is not n, then we do not compute the average (since it is undefined). In that case, we return Nothing. If it is, we do, and wrap it in "Just". Finally, we run "catMaybes" on the result of fmap (avg . take n), to get rid of the Maybe type.
I was (surprised and) disappointed by the performance of what seemed to me the most idiomatic Clojure solutions, #JamesCunningham 's lazy-seq solutions.
(def integers (iterate inc 0))
(def coll (take 10000 integers))
(def n 1000)
(time (doall (moving-average-james-1 coll n)))
# "Elapsed time: 3022.862 msecs"
(time (doall (moving-average-james-2 coll n)))
# "Elapsed time: 3433.988 msecs"
So here's a combination of James' solution with #DanielC.Sobral 's idea of adapting fast-exponentiation to moving sums :
(defn moving-average
[coll n]
(letfn [(moving-sum [coll n]
(lazy-seq
(cond
(= n 1) coll
(= n 2) (map + coll (rest coll))
(odd? n) (map + coll (moving-sum (rest coll) (dec n)))
:else (let [half (quot n 2)
hcol (moving-sum coll half)]
(map + hcol (drop half hcol))))))]
(cond
(< n 1) nil
(= n 1) coll
:else (map #(/ % n) (moving-sum coll n)))))
(time (doall (moving-average coll n)))
# "Elapsed time: 42.034 msecs"
Edit: this one -based on #mikera 's solution- is even faster.
(defn moving-average
[coll n]
(cond
(< n 1) nil
(= n 1) coll
:else (let [sums (reductions + 0 coll)]
(map #(/ (- %1 %2) n) (drop n sums) sums))))
(time (doall (moving-average coll n)))
# "Elapsed time: 9.184 msecs"
Related
Without using case expressions (that comes in the next section of the class), I can't see why the following doesn't do a quicksort. It goes into a loop somewhere and never ends.
splitAt and append have already been thorughly tested, but here are the codes for them.
fun append(xs, ys) =
if null xs
then ys
else (hd xs) :: append(tl xs, ys)
fun splitAt(xs : int list, x : int) =
let
fun sp(ys : int list, more : int list, less : int list) =
if null ys
then (more, less)
else
if hd ys < x
then sp(tl ys, more, append(less, [hd ys]))
else sp(tl ys, append(more, [hd ys]), less)
in
sp(xs, [], [])
end
fun qsort(xs : int list) =
if length xs <= 1
then xs
else
let
val s = splitAt(xs, hd xs)
in
qsort(append(#2 s, #1 s))
end
And I get the same problem using append(qsort(#2 s), qsort(#1 s)), but I though the former was better style since it only require a single recursion with each round.
I guess I should say that 'splitAt' divides the list into greater than or equal to the second argument, and less than, and creates a tuple). Append concatenates 2 lists.
PS: This is only a practice problem, not a test or homework.
It goes into a loop somewhere and never ends.
Your problem is most likely qsort being called on a list that does not reduce in size upon recursive call. Perhaps go with append(qsort (#2 s), qsort (#1 s)). But even then, can you be sure that each of #1 s and #2 s will always reduce in size?
Ideally you should supply splitAt and append since they're not library functions. You might consider using the built-in append called # and the built-in List.partition to form splitAt.
Compare against this one found somewhere on the interwebs:
fun quicksort [] = []
| quicksort (x::xs) =
let
val (left, right) = List.partition (fn y => y < x) xs
in
quicksort left # [x] # quicksort right
end
Since this isn't homework ...
Note that if xs = [1,2] then splitAt(xs hd xs) returns ([1,2],[]) so the attempt to sort [1,2] by this version of qsort reduces to ... sorting [1,2] again. That will never terminate.
Instead, I would advocate applying splitAt to the tail of the xs. A minimal tweak to your code is to leave append and splitAt alone but to rewrite qsort as:
fun qsort(xs : int list) =
if length xs <= 1
then xs
else
let
val s = splitAt(tl xs, hd xs)
in
append(qsort(#2 s), (hd xs) :: qsort(#1 s))
end;
Then, for example,
- qsort [4,2,1,2,3,6,4,7];
val it = [1,2,2,3,4,4,6,7] : int list
It is crucial that you apply qsort twice then append the results. Tryng to apply qsort once to the appended split would be trying to reduce qsort to applying qsort to a list which is the same size as the original list.
SML really becomes fun when you get to pattern matching. You should enjoy the next chapter.
I'm taking a functional programming languages course and I'm having difficulty understanding recursion within the context of 'functions as arguments'
fun n_times(f , n , x) =
if n=0
then x
else f (n_times(f , n - 1 , x))
fun double x = x+x;
val x1 = n_times(double , 4 , 7);
the value of x1 = 112
This doubles 'x' 'n' times so 7 doubled 4 times = 112
I can understand simpler recursion patterns such as adding numbers in a list, or 'power of' functions but I fail to understand how this function 'n_times' evaluates by calling itself ? Can provide an explanation of how this function works ?
I've tagged with scala as I'm taking this course to improve my understanding of scala (along with functional programming) and I think this is a common pattern so may be able to provide advice ?
If n is 0, x is returned.
Otherwise, f (n_times(f , n - 1 , x)) is returned.
What does n_times do? It takes the result of calling f with x, n times, or equivalently: calls f with the result of n_times(f, n - 1, x) (calling f n-1 times on x).
Note by calling f i mean for example:
calling f 3 times: f(f(f(x)))
calling f 2 times: f(f(x))
Just expand by hand. I'm going to call n_times nx to save space.
The core operation is
nx(f, n, x) -> f( nx(f, n-1, x))
terminating with
nx(f, 0, x) -> x
So, of course,
nx(f, 1, x) -> f( nx(f, 0, x) ) -> f( x )
nx(f, 2, x) -> f( nx(f, 1, x) ) -> f( f( x ) )
...
nx(f, n, x) -> f( nx(f,n-1,x) ) -> f( f( ... f( x ) ... ) )
Function n_times has a base case when n = 0 and an inductive case otherwise. You recurse on the inductive case until terminating on the base case.
Here is an illustrative trace:
n_times(double, 4, 7)
~> double (n_times(double, 3, 7)) (* n = 4 > 0, inductive case *)
~> double (double (n_times(double, 2, 7))) (* n = 3 > 0, inductive case *)
~> double (double (double (n_times(double, 1, 7)))) (* n = 2 > 0, inductive case *)
~> double (double (double (double (n_times(double, 0, 7))))) (* n = 1 > 0, inductive case *)
~> double (double (double (double 7))) (* n = 0, base case *)
~> double (double (double 14))
~> double (double 28)
~> double 56
~> 112
It is the same recursion thinking what you know already, just mixed with another concept: higher order functions.
n_times gets a function (f) as a parameter, so n_times is a higher order function, which in turn is capable to apply this f function in his body. In effect that is his job, apply f n times to x.
So how you apply f n times to x? Well, if you applied n-1 times
n_times(f , n - 1 , x)
, then you apply once more.
f (n_times(f , n - 1 , x))
You have to stop the recursion, as usual, that is the n=0 case with x.
Does anyone have a Scala implementation of Kadane's algorithm done in a functional style?
Edit Note: The definition on the link has changed in a way that invalidated answers to this question -- which goes to show why questions (and answers) should be self-contained instead of relying on external links. Here's the original definition:
In computer science, the maximum subarray problem is the task of finding the contiguous subarray within a one-dimensional array of numbers (containing at least one positive number) which has the largest sum. For example, for the sequence of values −2, 1, −3, 4, −1, 2, 1, −5, 4; the contiguous subarray with the largest sum is 4, −1, 2, 1, with sum 6.
What about this, if an empty subarray is allowed or the input array cannot be all negative:
numbers.scanLeft(0)((acc, n) => math.max(0, acc + n)).max
Or, failing the conditions above this (which assumes the input is non-empty):
numbers.tail.scanLeft(numbers.head)((acc, n) => (acc + n).max(n)).max
I prefer the folding solution to the scan solution -- though there's certainly elegance to the latter. Anyway,
numbers.foldLeft(0 -> 0) {
case ((maxUpToHere, maxSoFar), n) =>
val maxEndingHere = 0 max maxUpToHere + n
maxEndingHere -> (maxEndingHere max maxSoFar)
}._2
The following code returns the start and end index as well as the sum:
import scala.math.Numeric.Implicits.infixNumericOps
import scala.math.Ordering.Implicits.infixOrderingOps
case class Sub[T: Numeric](start: Index, end: Index, sum: T)
def maxSubSeq[T](arr: collection.IndexedSeq[T])(implicit n: Numeric[T]) =
arr
.view
.zipWithIndex
.scanLeft(Sub(-1, -1, n.zero)) {
case (p, (x, i)) if p.sum > n.zero => Sub(p.start, i, p.sum + x)
case (_, (x, i)) => Sub(i, i, x)
}
.drop(1)
.maxByOption(_.sum)
Here is some imperative code:
var sum = 0
val spacing = 6
var x = spacing
for(i <- 1 to 10) {
sum += x * x
x += spacing
}
Here are two of my attempts to "functionalize" the above code:
// Attempt 1
(1 to 10).foldLeft((0, 6)) {
case((sum, x), _) => (sum + x * x, x + spacing)
}
// Attempt 2
Stream.iterate ((0, 6)) { case (sum, x) => (sum + x * x, x + spacing) }.take(11).last
I think there might be a cleaner and better functional way to do this. What would be that?
PS: Please note that the above is just an example code intended to illustrate the problem; it is not from the real application code.
Replacing 10 by N, you have spacing * spacing * N * (N + 1) * (2 * N + 1) / 6
This is by noting that you're summing (spacing * i)^2 for the range 1..N. This sum factorizes as spacing^2 * (1^2 + 2^2 + ... + N^2), and the latter sum is well-known to be N * (N + 1) * (2 * N + 1) / 6 (see Square Pyramidal Number)
I actually like idea of lazy sequences in this case. You can split your algorithm in 2 logical steps.
At first you want to work on all natural numbers (ok.. not all, but up to max int), so you define them like this:
val naturals = 0 to Int.MaxValue
Then you need to define knowledge about how numbers, that you want to sum, can be calculated:
val myDoubles = (naturals by 6 tail).view map (x => x * x)
And putting this all together:
val naturals = 0 to Int.MaxValue
val myDoubles = (naturals by 6 tail).view map (x => x * x)
val mySum = myDoubles take 10 sum
I think it's the way mathematician will approach this problem. And because all collections are lazily evaluated - you will not get out of memory.
Edit
If you want to develop idea of mathematical notation further, you can actually define this implicit conversion:
implicit def math[T, R](f: T => R) = new {
def ∀(range: Traversable[T]) = range.view map f
}
and then define myDoubles like this:
val myDoubles = ((x: Int) => x * x) ∀ (naturals by 6 tail)
My personal favourite would have to be:
val x = (6 to 60 by 6) map {x => x*x} sum
Or given spacing as an input variable:
val x = (spacing to 10*spacing by spacing) map {x => x*x} sum
or
val x = (1 to 10) map (spacing*) map {x => x*x} sum
There are two different directions to go. If you want to express yourself, assuming that you can't use the built-in range function (because you actually want something more complicated):
Iterator.iterate(spacing)(x => x+spacing).take(10).map(x => x*x).foldLeft(0)(_ + _)
This is a very general pattern: specify what you start with and how to get the next given the previous; then take the number of items you need; then transform them somehow; then combine them into a single answer. There are shortcuts for almost all of these in simple cases (e.g. the last fold is sum) but this is a way to do it generally.
But I also wonder--what is wrong with the mutable imperative approach for maximal speed? It's really quite clear, and Scala lets you mix the two styles on purpose:
var x = spacing
val last = spacing*10
val sum = 0
while (x <= last) {
sum += x*x
x += spacing
}
(Note that the for is slower than while since the Scala compiler transforms for loops to a construct of maximum generality, not maximum speed.)
Here's a straightforward translation of the loop you wrote to a tail-recursive function, in an SML-like syntax.
val spacing = 6
fun loop (sum: int, x: int, i: int): int =
if i > 0 then loop (sum+x*x, x+spacing, i-1)
else sum
val sum = loop (0, spacing, 10)
Is this what you were looking for? (What do you mean by a "cleaner" and "better" way?)
What about this?
def toSquare(i: Int) = i * i
val spacing = 6
val spaceMultiples = (1 to 10) map (spacing *)
val squares = spaceMultiples map toSquare
println(squares.sum)
You have to split your code in small parts. This can improve readability a lot.
Here is a one-liner:
(0 to 10).reduceLeft((u,v)=>u + spacing*spacing*v*v)
Note that you need to start with 0 in order to get the correct result (else the first value 6 would be added only, but not squared).
Another option is to generate the squares first:
(1 to 2*10 by 2).scanLeft(0)(_+_).sum*spacing*spacing
I'm halfway through figuring out a solution to my question, but I have a feeling that it won't be very efficient. I've got a 2 dimensional cell structure of variable length arrays that is constructed in a very non-functional way in Matlab that I would like to convert to Clojure. Here is an example of what I'm trying to do:
pre = cell(N,1);
aux = cell(N,1);
for i=1:Ne
for j=1:D
for k=1:length(delays{i,j})
pre{post(i, delays{i, j}(k))}(end+1) = N*(delays{i, j}(k)-1)+i;
aux{post(i, delays{i, j}(k))}(end+1) = N*(D-1-j)+i; % takes into account delay
end;
end;
end;
My current plan for implementation is to use 3 loops where the first is initialized with a vector of N vectors of an empty vector. Each subloop is initialized by the previous loop. I define a separate function that takes the overall vector and the subindices and value and returns the vector with an updated subvector.
There's got to be a smarter way of doing this than using 3 loop/recurs. Possibly some reduce function that simplifies the syntax by using an accumulator.
I'm not 100% sure I understand what your code is doing (I don't know Matlab) but this might be one approach for building a multi-dimensional vector:
(defn conj-in
"Based on clojure.core/assoc-in, but with vectors instead of maps."
[coll [k & ks] v]
(if ks
(assoc coll k (conj-in (get coll k []) ks v))
(assoc coll k v)))
(defn foo []
(let [w 5, h 4, d 3
indices (for [i (range w)
j (range h)
k (range d)]
[i j k])]
(reduce (fn [acc [i j k :as index]]
(conj-in acc index
;; do real work here
(str i j k)))
[] indices)))
user> (pprint (foo))
[[["000" "001" "002"]
["010" "011" "012"]
["020" "021" "022"]
["030" "031" "032"]]
[["100" "101" "102"]
["110" "111" "112"]
["120" "121" "122"]
["130" "131" "132"]]
[["200" "201" "202"]
["210" "211" "212"]
["220" "221" "222"]
["230" "231" "232"]]
[["300" "301" "302"]
["310" "311" "312"]
["320" "321" "322"]
["330" "331" "332"]]
[["400" "401" "402"]
["410" "411" "412"]
["420" "421" "422"]
["430" "431" "432"]]]
This only works if indices go in the proper order (increasing), because you can't conj or assoc onto a vector anywhere other than one-past-the-end.
I also think it would be acceptable to use make-array and build your array via aset. This is why Clojure offers access to Java mutable arrays; some algorithms are much more elegant that way, and sometimes you need them for performance. You can always dump the data into Clojure vectors after you're done if you want to avoid leaking side-effects.
(I don't know which of this or the other version performs better.)
(defn bar []
(let [w 5, h 4, d 3
arr (make-array String w h d)]
(doseq [i (range w)
j (range h)
k (range d)]
(aset arr i j k (str i j k)))
(vec (map #(vec (map vec %)) arr)))) ;yikes?
Look to Incanter project that provide routines for work with data sets, etc.