While trying to become familiar with FS2, I came across a nifty recursive implementation using the Scala collections' Stream, and thought I'd have a go at trying it in FS2:
import fs2.{Pure, Stream}
val fibs: Stream[Pure, Int] = Stream[Pure, Int](0) ++ fibs.fold[Int](1)(_ + _)
println(fibs take 10 toList) // This will hang
What is the reason this hangs in FS2, and what is the best way to get a similar, working solution?
Your issue is that Stream.fold consumes all elements of the stream, producing a single final value from the fold. Note that it only emits one element.
The recursive stream only terminates when 10 elements have been emitted (this is specified by take 10). Since this stream is not productive enough, fold continues to add values without stopping.
The simplest way to fix this is to use a combinator that emits the partial results from the fold; this is scan.
Also, FS2 can infer most of the types in this code, so you don't necessarily need as many type annotations.
The following implementation should work fine:
import fs2.{Pure, Stream}
val fibs: Stream[Pure, Int] = Stream(0) ++ fibs.scan(1)(_ + _)
println(fibs take 10 toList)
Related
I'm migrating a project from Scala 2.12.1 to 2.13.6, and find that SeqView#flatMap now returns a View, which doesn't have a distinct method. I thus have one bit of code that does not compile anymore:
val nodes = debts.view
.flatMap { case Debt(from, to, _) => List(from, to) }
.distinct
.map(name => (name, new Node(name)))
.toMap
There's a dumb way to fix it, by converting the view to a seq and then back to a view:
val nodes = debts.view
.flatMap { case Debt(from, to, _) => List(from, to) }.toSeq.view
.distinct
.map(name => (name, new Node(name)))
.toMap
However, this is obviously not great because it forces the view to be collected, and also it's just super inelegant to have to go back-and-forth between types. I found another way to fix it, with is to use a LazyList:
val nodes = debts.to(LazyList)
.flatMap { case Debt(from, to, _) => List(from, to) }
.distinct
.map(name => (name, new Node(name)))
.toMap
Now that's what I want, it basically behaves like a Java stream. Sure, some operations have O(n) memory usage like distinct, but at least all operations after it get to be streamed, without reconstructing the data structure.
With this, it gets me thinking about why we should ever need a view, given that they're much less powerful than before (even if I can believe 2.13 has fixed some other issues this power was introducing). I looked for the answer and found hints, but nothing that I found comprehensive enough. Below is my research:
Description of views in 2.13
StackOverflow: What is the difference between List.view and LazyList?
Another comparison on an external website
It might be me, but even after reading these references, I don't find a clear upside in using views, for most if not all use cases. Anyone more enlightened than me?
There are actually 3 basic possibilities for lazy sequences in Scala 2.13: View, Iterator and LazyList.
View is the simplest lazy sequence with very little additional costs. It's good to use by default in general case to avoid allocations for intermediate results when working with large sequences.
It's possible to traverse the View several times (using foreach, foldLeft, toMap, etc.). Transformations (map, flatMap, filter, etc.) will be executed separately for every traversal. So care has to be applied either to avoid time-consuming transformations, or to traverse the View only once.
Iterator can be traversed only once. It's similar to Java Streams or Python generators. Most transformation methods on Iterator require that you only use the returned Iterator and discard the original object.
It's also fast like a View and supports more operations, including distinct.
LazyList is basically a real strict structure, which can be expanded automatically on the fly. LazyList memoizes all the generated elements. If you have a val with a LazyList, the memory will be allocated for all the generated elements. But if you traverse it on the fly and don't store in a val, the garbage collector can clean up the traversed elements.
Stream in Scala 2.12 was considerably slower than Views or Iterators. I'm not sure if this applies to LazyList in Scala 2.13.
So every lazy sequence has some caveat:
View can execute transformations multiple times.
Iterator can be consumed only once.
LazyList can allocate the memory for all the sequence elements.
In your use case I believe, it's Iterator that's the most appropriate:
val nodes = debts.iterator
.flatMap { case Debt(from, to, _) => List(from, to) }
.distinct
.map(name => (name, new Node(name)))
.toMap
Why do Scala and frameworks like Spark and Scalding have both reduce and foldLeft? So then what's the difference between reduce and fold?
reduce vs foldLeft
A big big difference, not mentioned in any other stackoverflow answer relating to this topic clearly, is that reduce should be given a commutative monoid, i.e. an operation that is both commutative and associative. This means the operation can be parallelized.
This distinction is very important for Big Data / MPP / distributed computing, and the entire reason why reduce even exists. The collection can be chopped up and the reduce can operate on each chunk, then the reduce can operate on the results of each chunk - in fact the level of chunking need not stop one level deep. We could chop up each chunk too. This is why summing integers in a list is O(log N) if given an infinite number of CPUs.
If you just look at the signatures there is no reason for reduce to exist because you can achieve everything you can with reduce with a foldLeft. The functionality of foldLeft is a greater than the functionality of reduce.
But you cannot parallelize a foldLeft, so its runtime is always O(N) (even if you feed in a commutative monoid). This is because it's assumed the operation is not a commutative monoid and so the cumulated value will be computed by a series of sequential aggregations.
foldLeft does not assume commutativity nor associativity. It's associativity that gives the ability to chop up the collection, and it's commutativity that makes cumulating easy because order is not important (so it doesn't matter which order to aggregate each of the results from each of the chunks). Strictly speaking commutativity is not necessary for parallelization, for example distributed sorting algorithms, it just makes the logic easier because you don't need to give your chunks an ordering.
If you have a look at the Spark documentation for reduce it specifically says "... commutative and associative binary operator"
http://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.rdd.RDD
Here is proof that reduce is NOT just a special case of foldLeft
scala> val intParList: ParSeq[Int] = (1 to 100000).map(_ => scala.util.Random.nextInt()).par
scala> timeMany(1000, intParList.reduce(_ + _))
Took 462.395867 milli seconds
scala> timeMany(1000, intParList.foldLeft(0)(_ + _))
Took 2589.363031 milli seconds
reduce vs fold
Now this is where it gets a little closer to the FP / mathematical roots, and a little trickier to explain. Reduce is defined formally as part of the MapReduce paradigm, which deals with orderless collections (multisets), Fold is formally defined in terms of recursion (see catamorphism) and thus assumes a structure / sequence to the collections.
There is no fold method in Scalding because under the (strict) Map Reduce programming model we cannot define fold because chunks do not have an ordering and fold only requires associativity, not commutativity.
Put simply, reduce works without an order of cumulation, fold requires an order of cumulation and it is that order of cumulation that necessitates a zero value NOT the existence of the zero value that distinguishes them. Strictly speaking reduce should work on an empty collection, because its zero value can by deduced by taking an arbitrary value x and then solving x op y = x, but that doesn't work with a non-commutative operation as there can exist a left and right zero value that are distinct (i.e. x op y != y op x). Of course Scala doesn't bother to work out what this zero value is as that would require doing some mathematics (which are probably uncomputable), so just throws an exception.
It seems (as is often the case in etymology) that this original mathematical meaning has been lost, since the only obvious difference in programming is the signature. The result is that reduce has become a synonym for fold, rather than preserving it's original meaning from MapReduce. Now these terms are often used interchangeably and behave the same in most implementations (ignoring empty collections). Weirdness is exacerbated by peculiarities, like in Spark, that we shall now address.
So Spark does have a fold, but the order in which sub results (one for each partition) are combined (at the time of writing) is the same order in which tasks are completed - and thus non-deterministic. Thanks to #CafeFeed for pointing out that fold uses runJob, which after reading through the code I realised that it's non-deterministic. Further confusion is created by Spark having a treeReduce but no treeFold.
Conclusion
There is a difference between reduce and fold even when applied to non-empty sequences. The former is defined as part of the MapReduce programming paradigm on collections with arbitrary order (http://theory.stanford.edu/~sergei/papers/soda10-mrc.pdf) and one ought to assume operators are commutative in addition to being associative to give deterministic results. The latter is defined in terms of catomorphisms and requires that the collections have a notion of sequence (or are defined recursively, like linked lists), thus do not require commutative operators.
In practice due to the unmathematical nature of programming, reduce and fold tend to behave in the same way, either correctly (like in Scala) or incorrectly (like in Spark).
Extra: My Opinion On the Spark API
My opinion is that confusion would be avoided if use of the term fold was completely dropped in Spark. At least spark does have a note in their documentation:
This behaves somewhat differently from fold operations implemented for
non-distributed collections in functional languages like Scala.
If I am not mistaken, even though the Spark API does not require it, fold also requires for the f to be commutative. Because the order in which the partitions will be aggregated is not assured.
For example in the following code only the first print out is sorted:
import org.apache.spark.{SparkConf, SparkContext}
object FoldExample extends App{
val conf = new SparkConf()
.setMaster("local[*]")
.setAppName("Simple Application")
implicit val sc = new SparkContext(conf)
val range = ('a' to 'z').map(_.toString)
val rdd = sc.parallelize(range)
println(range.reduce(_ + _))
println(rdd.reduce(_ + _))
println(rdd.fold("")(_ + _))
}
Print out:
abcdefghijklmnopqrstuvwxyz
abcghituvjklmwxyzqrsdefnop
defghinopjklmqrstuvabcwxyz
fold in Apache Spark is not the same as fold on not-distributed collections. In fact it requires commutative function to produce deterministic results:
This behaves somewhat differently from fold operations implemented for non-distributed
collections in functional languages like Scala. This fold operation may be applied to
partitions individually, and then fold those results into the final result, rather than
apply the fold to each element sequentially in some defined ordering. For functions
that are not commutative, the result may differ from that of a fold applied to a
non-distributed collection.
This has been shown by Mishael Rosenthal and suggested by Make42 in his comment.
It's been suggested that observed behavior is related to HashPartitioner when in fact parallelize doesn't shuffle and doesn't use HashPartitioner.
import org.apache.spark.sql.SparkSession
/* Note: standalone (non-local) mode */
val master = "spark://...:7077"
val spark = SparkSession.builder.master(master).getOrCreate()
/* Note: deterministic order */
val rdd = sc.parallelize(Seq("a", "b", "c", "d"), 4).sortBy(identity[String])
require(rdd.collect.sliding(2).forall { case Array(x, y) => x < y })
/* Note: all posible permutations */
require(Seq.fill(1000)(rdd.fold("")(_ + _)).toSet.size == 24)
Explained:
Structure of fold for RDD
def fold(zeroValue: T)(op: (T, T) => T): T = withScope {
var jobResult: T
val cleanOp: (T, T) => T
val foldPartition = Iterator[T] => T
val mergeResult: (Int, T) => Unit
sc.runJob(this, foldPartition, mergeResult)
jobResult
}
is the same as structure of reduce for RDD:
def reduce(f: (T, T) => T): T = withScope {
val cleanF: (T, T) => T
val reducePartition: Iterator[T] => Option[T]
var jobResult: Option[T]
val mergeResult = (Int, Option[T]) => Unit
sc.runJob(this, reducePartition, mergeResult)
jobResult.getOrElse(throw new UnsupportedOperationException("empty collection"))
}
where runJob is performed with disregard of partition order and results in need of commutative function.
foldPartition and reducePartition are equivalent in terms of order of processing and effectively (by inheritance and delegation) implemented by reduceLeft and foldLeft on TraversableOnce.
Conclusion: fold on RDD cannot depend on order of chunks and needs commutativity and associativity.
One other difference for Scalding is the use of combiners in Hadoop.
Imagine your operation is commutative monoid, with reduce it will be applied on the map side also instead of shuffling/sorting all data to reducers. With foldLeft this is not the case.
pipe.groupBy('product) {
_.reduce('price -> 'total){ (sum: Double, price: Double) => sum + price }
// reduce is .mapReduceMap in disguise
}
pipe.groupBy('product) {
_.foldLeft('price -> 'total)(0.0){ (sum: Double, price: Double) => sum + price }
}
It is always good practice to define your operations as monoid in Scalding.
I have been looking into scala and AKKA for managing an obviously parallelisable algorithm.
I have some knowledge of functional programming and mostly do Java, so my FP might not be the best yet.
The algorithm I am working with is pretty simple, there is a top computation:
def computeFull(...): FullObject
This computation calls sub computations and then sum it up (to simplify):
def computePartial(...): Int
and computeFull does something like this (again simplifying):
val partials = for(x <- 1 to 10
y <- 1 to 10) yield computePartial(x, y)
partials.foldLeft(0)(_ + _)
So, it's very close to the AKKA example, doing the PI computation. I have many computeFull to call and many computePartial within each of them. So I can wrap all of this in AKKA actors, or to simplify in Futures, calling each computeFull and each computePartial in separate threads. I then can use the fold, zip and map functions of http://doc.akka.io/docs/akka/snapshot/scala/futures.html to combile the futures.
However, this implies that computeFull and computePartial will have to return Futures wrapping the actual results. They thus become dependent on AKKA and assuming that things are run in parallel. In fact, I also have to implicitly pass down the execution contexts within my functions.
I think that this is weird and that the algorithm "shouldn't" know the details of how it is parallelised, or if it is.
After reading about Futures in scala (and not the AKKA one) and looking into Code Continuation. It seems like the Responder monad that is provided by scala (http://www.scala-lang.org/api/current/scala/Responder.html) seems like the right way to abstract how the function calls are run.
I have this vague intuition that computeFull and computePartial could return Responders instead of futures and that when the monad in executed, it decides how the code embedded within the Responder gets executed (if it spawns a new actor or if it is executed on the same thread).
However, I am not really sure how to get to this result. Any suggestions? Do you think I am on the right way?
If you don’t want to be dependent on Akka (but note that Akka-style futures will be moved and included with Scala 2.10) and your computation is a simple fold on a collection you can simply use Scala’s parallel collections:
val partials = for { x <- (1 to 10).par
y <- 1 to 10
} yield computePartial(x, y)
// blocks until everything is computed
partials.foldLeft(0)(_ + _)
Of course, this will block until partials is ready, so it may not be a appropriate situation when you really need futures.
With Scala 2.10 style futures you can make that completely asynchronous without your algorithms ever noticing it:
def computePartial(x: Int, y: Int) = {
Thread.sleep((1000 * math.random).toInt)
println (x + ", " + y)
x * y
}
import scala.concurrent.future
import scala.concurrent.Future
val partials: IndexedSeq[Future[Int]] = for {
x <- 1 to 10
y <- 1 to 10
} yield future(computePartial(x, y))
val futureResult: Future[Int] = Future.sequence(partials).map(_.fold(0)(_ + _))
def useResult(result: Int) = println(s"The sum is $result")
// now I want to use the result of my computation
futureResult map { result => // called when ready
useResult(result)
}
// still no blocking
println("This is probably printed before the sum is calculated.")
So, computePartial does not need to know anything about how it is being executed. (It should not have any side-effects though, even though for the purpose of this example, a println side-effect was included.)
A possible computeFull method should manage the algorithm and as such know about Futures or parallelism. After all this is part of the algorithm.
(As for the Responder: Scala’s old futures use it so I don’t know where this is going. – And isn’t an execution context exactly the means of configuration you are looking for?)
The single actor in akka knows not if he runs in parrallel or not. That is how akka is designed. But if you don't want to rely on akka you can use parrallel collections like:
for (i <- (0 until numberOfPartialComputations).par) yield (
computePartial(i)
).sum
The sum is called on a parrallel collection and is performed in parrallel.
Passing messages around with actors is great. But I would like to have even easier code.
Examples (Pseudo-code)
val splicedList:List[List[Int]]=biglist.partition(100)
val sum:Int=ActorPool.numberOfActors(5).getAllResults(splicedList,foldLeft(_+_))
where spliceIntoParts turns one big list into 100 small lists
the numberofactors part, creates a pool which uses 5 actors and receives new jobs after a job is finished
and getallresults uses a method on a list. all this done with messages passing in the background. where maybe getFirstResult, calculates the first result, and stops all other threads (like cracking a password)
With Scala Parallel collections that will be included in 2.8.1 you will be able to do things like this:
val spliced = myList.par // obtain a parallel version of your collection (all operations are parallel)
spliced.map(process _) // maps each entry into a corresponding entry using `process`
spliced.find(check _) // searches the collection until it finds an element for which
// `check` returns true, at which point the search stops, and the element is returned
and the code will automatically be done in parallel. Other methods found in the regular collections library are being parallelized as well.
Currently, 2.8.RC2 is very close (this or next week), and 2.8 final will come in a few weeks after, I guess. You will be able to try parallel collections if you use 2.8.1 nightlies.
You can use Scalaz's concurrency features to achieve what you want.
import scalaz._
import Scalaz._
import concurrent.strategy.Executor
import java.util.concurrent.Executors
implicit val s = Executor.strategy[Unit](Executors.newFixedThreadPool(5))
val splicedList = biglist.grouped(100).toList
val sum = splicedList.parMap(_.sum).map(_.sum).get
It would be pretty easy to make this prettier (i.e. write a function mapReduce that does the splitting and folding all in one). Also, parMap over a List is unnecessarily strict. You will want to start folding before the whole list is ready. More like:
val splicedList = biglist.grouped(100).toList
val sum = splicedList.map(promise(_.sum)).toStream.traverse(_.sum).get
You can do this with less overhead than creating actors by using futures:
import scala.actors.Futures._
val nums = (1 to 1000).grouped(100).toList
val parts = nums.map(n => future { n.reduceLeft(_ + _) })
val whole = (0 /: parts)(_ + _())
You have to handle decomposing the problem and writing the "future" block and recomposing it in to a final answer, but it does make executing a bunch of small code blocks in parallel easy to do.
(Note that the _() in the fold left is the apply function of the future, which means, "Give me the answer you were computing in parallel!", and it blocks until the answer is available.)
A parallel collections library would automatically decompose the problem and recompose the answer for you (as with pmap in Clojure); that's not part of the main API yet.
I'm not waiting for Scala 2.8.1 or 2.9, it would rather be better to write my own library or use another, so I did more googling and found this: akka
http://doc.akkasource.org/actors
which has an object futures with methods
awaitAll(futures: List[Future]): Unit
awaitOne(futures: List[Future]): Future
but http://scalablesolutions.se/akka/api/akka-core-0.8.1/
has no documentation at all. That's bad.
But the good part is that akka's actors are leaner than scala's native ones
With all of these libraries (including scalaz) around, it would be really great if scala itself could eventually merge them officially
At Scala Days 2010, there was a very interesting talk by Aleksandar Prokopec (who is working on Scala at EPFL) about Parallel Collections. This will probably be in 2.8.1, but you may have to wait a little longer. I'll lsee if I can get the presentation itself. to link here.
The idea is to have a collections framework which parallelizes the processing of the collections by doing exactly as you suggest, but transparently to the user. All you theoretically have to do is change the import from scala.collections to scala.parallel.collections. You obviously still have to do the work to see if what you're doing can actually be parallelized.
In Scala there is a Stream class that is very much like an iterator. The topic Difference between Iterator and Stream in Scala? offers some insights into the similarities and differences between the two.
Seeing how to use a stream is pretty simple but I don't have very many common use-cases where I would use a stream instead of other artifacts.
The ideas I have right now:
If you need to make use of an infinite series. But this does not seem like a common use-case to me so it doesn't fit my criteria. (Please correct me if it is common and I just have a blind spot)
If you have a series of data where each element needs to be computed but that you may want to reuse several times. This is weak because I could just load it into a list which is conceptually easier to follow for a large subset of the developer population.
Perhaps there is a large set of data or a computationally expensive series and there is a high probability that the items you need will not require visiting all of the elements. But in this case an Iterator would be a good match unless you need to do several searches, in that case you could use a list as well even if it would be slightly less efficient.
There is a complex series of data that needs to be reused. Again a list could be used here. Although in this case both cases would be equally difficult to use and a Stream would be a better fit since not all elements need to be loaded. But again not that common... or is it?
So have I missed any big uses? Or is it a developer preference for the most part?
Thanks
The main difference between a Stream and an Iterator is that the latter is mutable and "one-shot", so to speak, while the former is not. Iterator has a better memory footprint than Stream, but the fact that it is mutable can be inconvenient.
Take this classic prime number generator, for instance:
def primeStream(s: Stream[Int]): Stream[Int] =
Stream.cons(s.head, primeStream(s.tail filter { _ % s.head != 0 }))
val primes = primeStream(Stream.from(2))
It can be easily be written with an Iterator as well, but an Iterator won't keep the primes computed so far.
So, one important aspect of a Stream is that you can pass it to other functions without having it duplicated first, or having to generate it again and again.
As for expensive computations/infinite lists, these things can be done with Iterator as well. Infinite lists are actually quite useful -- you just don't know it because you didn't have it, so you have seen algorithms that are more complex than strictly necessary just to deal with enforced finite sizes.
In addition to Daniel's answer, keep in mind that Stream is useful for short-circuiting evaluations. For example, suppose I have a huge set of functions that take String and return Option[String], and I want to keep executing them until one of them works:
val stringOps = List(
(s:String) => if (s.length>10) Some(s.length.toString) else None ,
(s:String) => if (s.length==0) Some("empty") else None ,
(s:String) => if (s.indexOf(" ")>=0) Some(s.trim) else None
);
Well, I certainly don't want to execute the entire list, and there isn't any handy method on List that says, "treat these as functions and execute them until one of them returns something other than None". What to do? Perhaps this:
def transform(input: String, ops: List[String=>Option[String]]) = {
ops.toStream.map( _(input) ).find(_ isDefined).getOrElse(None)
}
This takes a list and treats it as a Stream (which doesn't actually evaluate anything), then defines a new Stream that is a result of applying the functions (but that doesn't evaluate anything either yet), then searches for the first one which is defined--and here, magically, it looks back and realizes it has to apply the map, and get the right data from the original list--and then unwraps it from Option[Option[String]] to Option[String] using getOrElse.
Here's an example:
scala> transform("This is a really long string",stringOps)
res0: Option[String] = Some(28)
scala> transform("",stringOps)
res1: Option[String] = Some(empty)
scala> transform(" hi ",stringOps)
res2: Option[String] = Some(hi)
scala> transform("no-match",stringOps)
res3: Option[String] = None
But does it work? If we put a println into our functions so we can tell if they're called, we get
val stringOps = List(
(s:String) => {println("1"); if (s.length>10) Some(s.length.toString) else None },
(s:String) => {println("2"); if (s.length==0) Some("empty") else None },
(s:String) => {println("3"); if (s.indexOf(" ")>=0) Some(s.trim) else None }
);
// (transform is the same)
scala> transform("This is a really long string",stringOps)
1
res0: Option[String] = Some(28)
scala> transform("no-match",stringOps)
1
2
3
res1: Option[String] = None
(This is with Scala 2.8; 2.7's implementation will sometimes overshoot by one, unfortunately. And note that you do accumulate a long list of None as your failures accrue, but presumably this is inexpensive compared to your true computation here.)
I could imagine, that if you poll some device in real time, a Stream is more convenient.
Think of an GPS tracker, which returns the actual position if you ask it. You can't precompute the location where you will be in 5 minutes. You might use it for a few minutes only to actualize a path in OpenStreetMap or you might use it for an expedition over six months in a desert or the rain forest.
Or a digital thermometer or other kinds of sensors which repeatedly return new data, as long as the hardware is alive and turned on - a log file filter could be another example.
Stream is to Iterator as immutable.List is to mutable.List. Favouring immutability prevents a class of bugs, occasionally at the cost of performance.
scalac itself isn't immune to these problems: http://article.gmane.org/gmane.comp.lang.scala.internals/2831
As Daniel points out, favouring laziness over strictness can simplify algorithms and make it easier to compose them.