Chapter 7 in FP in Scala deals with creating a purely functional library for handling concurrency. To that end, it defines a type
type Par[A] = ExecutorService => Future[A]
and a set of useful functions such as fork
def fork[A] (a: => Par[A]): Par[A] =
es => es.submit(new Callable[A] {
def call = a(es).get
})
One of the exercises is about the function sequence with the following signature
def sequence[A](ps: List[Par[A]]): Par[List[A]]
The solution using foldRight is straightforward. However the authors included two other versions as answers, one of which states the following
// This implementation forks the recursive step off to a new logical thread,
// making it effectively tail-recursive. However, we are constructing
// a right-nested parallel program, and we can get better performance by
// dividing the list in half, and running both halves in parallel.
// See `sequenceBalanced` below.
def sequenceRight[A](as: List[Par[A]]): Par[List[A]] =
as match {
case Nil => unit(Nil)
case h :: t => map2(h, fork(sequenceRight(t)))(_ :: _)
}
I am not quite sure what is meant by "effectively tail recursive". From the definition of fork it is clear that it accepts a by name parameter and so map2(h, fork(sequenceRight(t)))(_ :: _) will not evaluate the second parameter (until the executor service is provided). But that doesn't tell me how and why it is "effectively tail recursive".
Let's take some List(a, b, c). After passing it into sequenceRight it will turn into:
map2(
a,
fork(
map2(
b,
fork(
map2(
c,
fork(unit(Nil)
)(_ :: _)
)
)(_ :: _)
)
)(_ :: _)
This isn't tail recursive at all and compiler cannot treat it as one. However, when you would evaluate how it would be executed:
fork would make whatever you pass to it async, so it would return immediately,
map2 implementation will not block the execution until fork is executed to apply the function passed to map2, instead it would asynchronously transform the result calculated in fork to prepend the value
since recursion is done asynchronously, posting things to ExecutorService and appending operation to Future let you treat ExecutorService+Future like a trampoline
As a result what actually happens is:
sequenceRight(List(a, b, c)) call `map2(a, fork(sequenceRight(List(b, c))(_ :: _)
a will complete when it will complete but we can hold it as value even now
fork(sequenceRight(List(b, c)) is scheduled, but we aren't waiting until it complete, we can pass it around already
we can create a Future that will combine the result of the 2 above (and return it) without waiting for any of them completing!
as a result, this Future is returned immediately! It still runs, but this one call is finished!
same is true for recursively created Futures
once c and fork(unit(Nil)) completes, rResult :: Nil is computed
this allows completion of bResult :: cResult :: Nil
and this finally allows computation of the final result
In other words tail-recursive refers to recursive calls being rewritten into while loop to avoid stack overflow. But stack overflow is only an issue if recursive calls are being made synchronously. If you are returning something async, then the backtracing is shoved to ExecutionServices and Futures' observers, so they are hidden in a heap. From that point of view they solve the same issue of stack-overflow as tail-recursive calls, so "in spirit" they could be considered somewhat similar.
This is certainly not tail-recursion, and I think, they know it – that's why they added "effectively" :).
What they must mean is that this implementation does not create additional frames on stack for recursive invocations, which is true, since those invocations happen asynchronously, as you noted.
Now, whether this situation is even deserves to be called a "recursion" at all is a good question. I don't think there is a single accepted answer to that. I personally lean toward "yes", because the definition of recursion as "referencing itself" definitely includes this situation, and I don't really know how else to define it so that async invocations are excluded, but tail-recursion is not.
BTW, I am not much of an expert in Javascript, but I hear that the term "asynchronous recursion" is used fairly widely there.
Related
for {
i <- 1 to 5
} yield Future(println(i))
Desugared to:
List(1,2,3,4,5).map {i => Future(println(i))}
The above code prints numbers in random order.
Now, if we see the multiple definitions of Monad:
a) Monad is a wrapper over an object
b) Monad is a mechanism for sequencing computations
The question that I'm trying to answer is that shouldn't map operation on List monad wait for the first element in the list to be printed and only then go for the computation of the second element regardless of Future?
Sorry, it might be simple and I'm complicating it but it gets trickier for me to find simple reasoning. Answers will be much appreciated:)
Compare:
for {
_ <- Future(println(1))
_ <- Future(println(2))
_ <- Future(println(3))
_ <- Future(println(4))
_ <- Future(println(5))
} yield ()
or
Future(println(1)).flatMap { _ =>
Future(println(2))
}.flatMap { _ =>
Future(println(3))
}.flatMap { _ =>
Future(println(4))
}.flatMap { _ =>
Future(println(5))
}
with
List(
Future(println(1)),
Future(println(2)),
Future(println(3)),
Future(println(4)),
Future(println(5))
)
The first two create the next Future only after the former completed and made the result available. The last one creates all Futures at once (and it doesn't differ much in this regard from your example with List[Future]).
Future (as opposed to IO from Cats Effect, Monix's Task or ZIO) is eager, so it starts execution the moment you create it. For that reason you have sequential result in the first two examples, and random order (race condition) in the third example.
If you used IO instead of Future it would be more apparent because you wouldn't be able to just have List[IO[Unit]] and execute side effects - you would have to somehow combine the different IOs into one, and the way you would do it would make it obvious whether the effects will be sequential or parallel.
The bottom line is - whether or not Future is a monad depends on how the .flatMap behaves (and how it behaves with combination with Future.successful), so your results doesn't invalidate the claim that Future is a monad. (You can have some doubts if you start checking its behavior with exceptions, but that is another topic).
The execution of map is sequential indeed, but when you wrap it to a Future it gets executed in an asynchronous manner, I mean it is evaluated in another thread and because of that, it is not possible to know what thread is going to finish earlier because it depends also in the thread management of the operating system and other considerations.
Both of your code snippets are still Monads in loose terms. When you did .map() on your object, the map picks element one by one in orderly fashion (from index 0 to index 4). And then it passes on that to an operation block (which is body of map - map is a higher order function that accepts a function of type f:This => That).
So monad operation's responsibility is picking it up and passing it as paramater to a function.
In your case the actual function type is:
f: Int => Future[Unit]
For clarity, your function actually looks like this:
def someFunction(i: Int): Future[Unit] = {
Future {
println(i)
}
}
So, what the map operation did here is that it picked on item from your object (in sequence, one by one) and called the someFunction(i). And that's all a monad does.
Now to answer why your println are random, it's because of JVM threads.
If you re-define the body of you map like this
List(1,2,3,4,5)
.map {i =>
println(s"Going to invoke the println in another thread for $i")
Future(println(i))
}
You'll see that the first println will be in sequence - always! It proves that .map() picks your elements in sequence. While the next println may or may not be out of sequence. This out of order fashion is not because of monad operation map but because of multithreading nature in multi core CPUs.
I have been looking into FP languages (off and on) for some time and have played with Scala, Haskell, F#, and some others. I like what I see and understand some of the fundamental concepts of FP (with absolutely no background in Category Theory - so don't talk Math, please).
So, given a type M[A] we have map which takes a function A=>B and returns a M[B]. But we also have flatMap which takes a function A=>M[B] and returns a M[B]. We also have flatten which takes a M[M[A]] and returns a M[A].
In addition, many of the sources I have read describe flatMap as map followed by flatten.
So, given that flatMap seems to be equivalent to flatten compose map, what is its purpose? Please don't say it is to support 'for comprehensions' as this question really isn't Scala-specific. And I am less concerned with the syntactic sugar than I am in the concept behind it. The same question arises with Haskell's bind operator (>>=). I believe they both are related to some Category Theory concept but I don't speak that language.
I have watched Brian Beckman's great video Don't Fear the Monad more than once and I think I see that flatMap is the monadic composition operator but I have never really seen it used the way he describes this operator. Does it perform this function? If so, how do I map that concept to flatMap?
BTW, I had a long writeup on this question with lots of listings showing experiments I ran trying to get to the bottom of the meaning of flatMap and then ran into this question which answered some of my questions. Sometimes I hate Scala implicits. They can really muddy the waters. :)
FlatMap, known as "bind" in some other languages, is as you said yourself for function composition.
Imagine for a moment that you have some functions like these:
def foo(x: Int): Option[Int] = Some(x + 2)
def bar(x: Int): Option[Int] = Some(x * 3)
The functions work great, calling foo(3) returns Some(5), and calling bar(3) returns Some(9), and we're all happy.
But now you've run into the situation that requires you to do the operation more than once.
foo(3).map(x => foo(x)) // or just foo(3).map(foo) for short
Job done, right?
Except not really. The output of the expression above is Some(Some(7)), not Some(7), and if you now want to chain another map on the end you can't because foo and bar take an Int, and not an Option[Int].
Enter flatMap
foo(3).flatMap(foo)
Will return Some(7), and
foo(3).flatMap(foo).flatMap(bar)
Returns Some(15).
This is great! Using flatMap lets you chain functions of the shape A => M[B] to oblivion (in the previous example A and B are Int, and M is Option).
More technically speaking; flatMap and bind have the signature M[A] => (A => M[B]) => M[B], meaning they take a "wrapped" value, such as Some(3), Right('foo), or List(1,2,3) and shove it through a function that would normally take an unwrapped value, such as the aforementioned foo and bar. It does this by first "unwrapping" the value, and then passing it through the function.
I've seen the box analogy being used for this, so observe my expertly drawn MSPaint illustration:
This unwrapping and re-wrapping behavior means that if I were to introduce a third function that doesn't return an Option[Int] and tried to flatMap it to the sequence, it wouldn't work because flatMap expects you to return a monad (in this case an Option)
def baz(x: Int): String = x + " is a number"
foo(3).flatMap(foo).flatMap(bar).flatMap(baz) // <<< ERROR
To get around this, if your function doesn't return a monad, you'd just have to use the regular map function
foo(3).flatMap(foo).flatMap(bar).map(baz)
Which would then return Some("15 is a number")
It's the same reason you provide more than one way to do anything: it's a common enough operation that you may want to wrap it.
You could ask the opposite question: why have map and flatten when you already have flatMap and a way to store a single element inside your collection? That is,
x map f
x filter p
can be replaced by
x flatMap ( xi => x.take(0) :+ f(xi) )
x flatMap ( xi => if (p(xi)) x.take(0) :+ xi else x.take(0) )
so why bother with map and filter?
In fact, there are various minimal sets of operations you need to reconstruct many of the others (flatMap is a good choice because of its flexibility).
Pragmatically, it's better to have the tool you need. Same reason why there are non-adjustable wrenches.
The simplest reason is to compose an output set where each entry in the input set may produce more than one (or zero!) outputs.
For example, consider a program which outputs addresses for people to generate mailers. Most people have one address. Some have two or more. Some people, unfortunately, have none. Flatmap is a generalized algorithm to take a list of these people and return all of the addresses, regardless of how many come from each person.
The zero output case is particularly useful for monads, which often (always?) return exactly zero or one results (think Maybe- returns zero results if the computation fails, or one if it succeeds). In that case you want to perform an operation on "all of the results", which it just so happens may be one or many.
The "flatMap", or "bind", method, provides an invaluable way to chain together methods that provide their output wrapped in a Monadic construct (like List, Option, or Future). For example, suppose you have two methods that produce a Future of a result (eg. they make long-running calls to databases or web service calls or the like, and should be used asynchronously):
def fn1(input1: A): Future[B] // (for some types A and B)
def fn2(input2: B): Future[C] // (for some types B and C)
How to combine these? With flatMap, we can do this as simply as:
def fn3(input3: A): Future[C] = fn1(a).flatMap(b => fn2(b))
In this sense, we have "composed" a function fn3 out of fn1 and fn2 using flatMap, which has the same general structure (and so can be composed in turn with further similar functions).
The map method would give us a not-so-convenient - and not readily chainable - Future[Future[C]]. Certainly we can then use flatten to reduce this, but the flatMap method does it in one call, and can be chained as far as we wish.
This is so useful a way of working, in fact, that Scala provides the for-comprehension as essentially a short-cut for this (Haskell, too, provides a short-hand way of writing a chain of bind operations - I'm not a Haskell expert, though, and don't recall the details) - hence the talk you will have come across about for-comprehensions being "de-sugared" into a chain of flatMap calls (along with possible filter calls and a final map call for the yield).
Well, one could argue, you don't need .flatten either. Why not just do something like
#tailrec
def flatten[T](in: Seq[Seq[T], out: Seq[T] = Nil): Seq[T] = in match {
case Nil => out
case head ::tail => flatten(tail, out ++ head)
}
Same can be said about map:
#tailrec
def map[A,B](in: Seq[A], out: Seq[B] = Nil)(f: A => B): Seq[B] = in match {
case Nil => out
case head :: tail => map(tail, out :+ f(head))(f)
}
So, why are .flatten and .map provided by the library? Same reason .flatMap is: convenience.
There is also .collect, which is really just
list.filter(f.isDefinedAt _).map(f)
.reduce is actually nothing more then list.foldLeft(list.head)(f),
.headOption is
list match {
case Nil => None
case head :: _ => Some(head)
}
Etc ...
I am teaching myself Scala by trying to implement the operations on List[T]. I just implemented dropWhile and it made me wonder about how tail recursion optimization works when the recursive call appears in different cases.
def dropWhile[T](list: List[T])(predicate: T => Boolean): List[T] = list match {
case head :: tail if predicate(head) => dropWhile(tail)(predicate)
case _ => list
}
Does it matter that the recursive call appears in the first case?
As some one said in the comments, you can apply the #tailrec annotations to your function and it will give a compiler error if the recursion can not be optimized to a loop.
Where the recursive call is does not matter. The important part is that there is no need for a stack frame to stay allocated waiting for the return from the recursive calls.
I know streams are supposed to be lazily evaluated sequences in Scala, but I think I am suffering from some sort of fundamental misunderstanding because they seem to be more eager than I would have expected.
In this example:
val initial = Stream(1)
lazy val bad = Stream(1/0)
println((initial ++ bad) take 1)
I get a java.lang.ArithmeticException, which seems to be cause by zero division. I would expect that bad would never get evaluated since I only asked for one element from the stream. What's wrong?
OK, so after commenting other answers, I figured I could as well turn my comments into a proper answer.
Streams are indeed lazy, and will only compute their elements on demand (and you can use #:: to construct a stream element by element, much like :: for List). By example, the following will not throw any exception:
(1/2) #:: (1/0) #:: Stream.empty
This is because when applying #::, the tail is passed by name so as to not evaluate it eagerly, but only when needed (see ConsWrapper.# ::, const.apply and class Cons in Stream.scala for more details).
On the other hand, the head is passed by value, which means that it will always be eagerly evaluated, no matter what (as mentioned by Senthil). This means that doing the following will actually throw a ArithmeticException:
(1/0) #:: Stream.empty
It is a gotcha worth knowing about streams. However, this is not the issue you are facing.
In your case, the arithmetic exception happens before even instantiating a single Stream. When calling Stream.apply in lazy val bad = Stream(1/0), the argument is eagerly executed because it is not declared as a by name parameter. Stream.apply actually takes a vararg parameter, and those are necessarily passed by value.
And even if it was passed by name, the ArithmeticException would be triggered shortly after, because as said earlier the head of a Stream is always early evaluated.
The fact that Streams are lazy doesn't change the fact that method arguments are evaluated eagerly.
Stream(1/0) expands to Stream.apply(1/0). The semantics of the language require that the arguments are evaluated before the method is called (since the Stream.apply method doesn't use call-by-name arguments), so it attempts to evaluate 1/0 to pass as the argument to the Stream.apply method, which causes your ArithmeticException.
There are a few ways you can get this working though. Since you've already declared bad as a lazy val, the easiest is probably to use the also-lazy #::: stream concatenation operator to avoid forcing evaluation:
val initial = Stream(1)
lazy val bad = Stream(1/0)
println((initial #::: bad) take 1)
// => Stream(1, ?)
The Stream will evaluate the head & remaining tail is evaluated lazily. In your example, both the streams are having only the head & hence giving an error.
I was reading about fold techniques in Programming in Scala book and came across this snippet:
def reverseLeft[T](xs:List[T]) = (List[T]() /: xs) {
(y,ys) => ys :: y
}
As you can see, it was done using foldLeft or /: operator. Curious how it would look like if I did it using :\, I came up with this:
def reverseRight[T](xs:List[T]) = (xs :\ List[T]()) {
(y,ys) => ys ::: List(y)
}
As I understand it, ::: doesn't seem to be as fast as :: and has a linear cost depending on the size of the operand list. Admittedly, I don't have a background in CS and no prior FP experience. So my questions are:
How do you recognise/distinguish between foldLeft/foldRight in problem approaches?
Is there a better way of doing this without using :::?
Since foldRight on List in the standard library is strict and implemented using linear recursion, you should avoid using it, as a rule. An iterative implementation of foldRight would be as follows:
def foldRight[A,B](f: (A, B) => B, z: B, xs: List[A]) =
xs.reverse.foldLeft(z)((x, y) => f(y, x))
A recursive implementation of foldLeft could be this:
def foldLeft[A,B](f: (B, A) => B, z: B, xs: List[A]) =
xs.reverse.foldRight(z)((x, y) => f(y, x))
So you see, if both are strict, then one or the other of foldRight and foldLeft is going to be implemented (conceptually anyway) with reverse. Since the way lists are constructed with :: associates to the right, the straightforward iterative fold is going to be foldLeft, and foldRight is simply "reverse then foldLeft".
Intuitively, you might think that this would be a slow implementation of foldRight, since it folds the list twice. But:
"Twice" is a constant factor anyway, so it's asymptotically equivalent to folding once.
You have to go over the list twice anyway. Once to push computations onto the stack and again to pop them off the stack.
The implementation of foldRight above is faster than the one in the standard library.
Operations on a List are intentionally not symmetric. The List data structure is a singly-linked list where each node (both data and pointer) are immutable. The idea behind this data structure is that you perform modifications on the front of the list by taking references to internal nodes and adding new nodes that point to them -- different versions of the list will share the same nodes for the end of the list.
The ::: operator which appends a new element on to the end of the list has to create a new copy of the entire list, because otherwise it would modify other lists that share nodes with the list you're appending to. This is why ::: takes linear time.
Scala has a data structure called a ListBuffer that you can use instead of the ::: operator to make appending to the end of a list faster. Basically, you create a new ListBuffer and it starts with an empty list. The ListBuffer maintains a list completely separate from any other list that the program knows about, so it's safe to modify it by adding on to the end. When you're finished adding on to the end, you call ListBuffer.toList, which releases the list into the world, at which point you can no longer add on to the end without copying it.
foldLeft and foldRight also share a similar assymmetry. foldRight requires you to walk the entire list to get to the end of the list, and keep track of everywhere you've visited on the way there, so that you an visit them in reverse order. This is usually done recursively, and it can lead to foldRight causing stack overflows on large lists. foldLeft on the other hand, deals with nodes in the order they appear in the list, so it can forget the ones it's visited already and only needs to know about one node at a time. Though foldLeft is also usually implemented recursively, it can take advantage of an optimization called tail recursion elimination, in which the compiler transforms the recursive calls into a loop because the function doesn't do anything after returning from the recursive call. Thus, foldLeft doesn't overflow the stack even on very long lists. EDIT: foldRight in Scala 2.8 is actually implemented by reversing the list and running foldLeft on the reversed list -- so the tail recursion issue is not an issue -- both data structures optimize tail recursion correctly, and you could choose either one (You do get into the issue now that you're defining reverse in terms of reverse -- you don't need to worry if you're defining your own reverse method for the fun of it, but you wouldn't have the foldRight option at all if you were defining Scala's reverse method.)
Thus, you should prefer foldLeft and :: over foldRight and :::.
(In an algorithm that would combine foldLeft with ::: or foldRight with ::, then you need to make a decision for yourself about which is more important: stack space or running time. Or you should use foldLeft with a ListBuffer.)