Cannot execute parallel computation with functional API - scala

Question

I am trying to implement the countWords function from the red book on the parallelism chapter. When I pass a thread pool to the function and I modify the function to print the thread counting the words, I can only see the main thread printed. This indicates me that I am not able to make this function execute in parallel.
What I currently have:
type Par[A] = ExecutorService => Future[A]
def asyncF[A, B](f: A => B): A => Par[B] = a => lazyUnit(f(a))
def lazyUnit[A](a: => A): Par[A] = fork(unit(a))
def unit[A](a: A): Par[A] = (_: ExecutorService) => UnitFuture(a)
def fork[A](a: => Par[A]): Par[A] =
es => es.submit(new Callable[A] {
def call = a(es).get
})
def countWords(l: List[String]): Par[Int] = map(sequence(l.map(asyncF {
println(Thread.currentThread())
s => s.split(" ").length
})))(_.sum)
When I run:
val listPar = List("ab cd", "hg ks", "lh ks", "lh hs")
val es = Executors.newFixedThreadPool(4)
val counts = countWords(listPar)(es)
println(counts.get(100, SECONDS))
I get:
Thread[main,5,main]
8
I would expect to see a thread printed per each element of the list (as there are four elements and a thread pool of size 4) however I can only see the main thread printed.
Any suggestions?
Thanks

Answer 1

I want to start with one piece of advice when asking questions - you should always provide a MCVE. Your code doesn't compile; for example, I have no idea where UnitFuture comes from, I have no idea what's the implementation of sequence that you're using etc.
Here is a snippet that works with standard Scala. First, the explanation:
Method countWords takes a list of strings to count, and also two services - one for handling Java Futures on different threads, and one for handling Scala Futures on different threads. Scala one is derived from Java one via ExecutionContext.fromExecutor method.
Why both Java and Scala? Well, I wanted to preserve Java because that's how you initially wrote your code, but I don't know how to sequence a Java Future. So what I did was:
for each substring:
fork a Java Future task
turn it into a Scala Future
sequence the obtained list of Scala Futures
In case you're not familiar with implicits, you will (if you intend to work with Scala). Here I used the execution context implicitly because it removes a lot of boilerplate - this way I don't have to explicitly pass it when converting to Scala future, when mapping / sequencing etc.
And now the code itself:
import java.util.concurrent.{Callable, ExecutorService, Executors}
import java.util.concurrent.{Future => JFuture}
import scala.concurrent.{ExecutionContext, Future}
def scalaFromJavaFuture[A](
javaFuture: JFuture[A]
)(implicit ec: ExecutionContext): Future[A] =
Future { javaFuture.get }(ec)
def fork(s: String)(es: ExecutorService): java.util.concurrent.Future[Int] =
es.submit(new Callable[Int] {
def call = {
println(s"Thread: ${Thread.currentThread()}, processing string: $s")
s.split(" ").size
}
})
def countWords(l: List[String])(es: ExecutorService)(implicit ec: ExecutionContext): Future[Int] = {
val listOfFutures = l.map(elem => scalaFromJavaFuture(fork(elem)(es)))
Future.sequence(listOfFutures).map(_.sum)
}
val listPar = List("ab cd", "hg ks", "lh ks", "lh hs")
val es = Executors.newFixedThreadPool(4)
implicit val ec = ExecutionContext.fromExecutor(es)
val counts = countWords(listPar)(es)
counts.onComplete(println)
Example output:
Thread: Thread[pool-1-thread-1,5,main], processing string: ab cd
Thread: Thread[pool-1-thread-3,5,main], processing string: hg ks
Thread: Thread[pool-1-thread-2,5,main], processing string: lh ks
Thread: Thread[pool-1-thread-4,5,main], processing string: lh hs
Success(8)
Note that it's up to execution context to determine the threads. Run it a couple of times and you will see for yourself - you might end up with e.g. only two threads being used:
Thread: Thread[pool-1-thread-1,5,main], processing string: ab cd
Thread: Thread[pool-1-thread-3,5,main], processing string: hg ks
Thread: Thread[pool-1-thread-1,5,main], processing string: lh ks
Thread: Thread[pool-1-thread-1,5,main], processing string: lh hs
Success(8)