Tackling recursion while using Scala Futures - scala

In my working code is recursion involved. I try to avoid this, because the recursion could be too deep if a predicate don't hold.
case class Chan[T]() {
private var promise: Promise[T] = Promise[T]()
/** Binds a handler with the "write" in casu update() */
def this(handler: (T => Unit) => Unit) {
this
handler(update)
}
def filter(p: (T) => Boolean): Future[T] = {
apply().flatMap(value => if (p(value)) Future(value) else filter(p))
}
def apply(): Future[T] = {
promise = Promise[T]()
promise.future
}
def update(t: T): Unit = {
if (!promise.isCompleted) promise.success(t)
}
}
The problem lays in the filter method. When the predicate is not met, filter will be called again, hunting for an event which is conform the parameter. By that the stack will be filled, ahead for a StackOverflow :-).
How can the code refactored, in a loop or tail recursive calls, avoiding excessive stack usage?

So here's the solution as suggested above.
I hope that fits your purpose:
case class Chan[T]() {
private var pendingFilters: List[(T => Boolean, Promise[T])] = List.empty
/** Binds a handler with the "write" in casu update() */
def this(handler: (T => Unit) => Unit) {
this
handler(update)
}
def filter(p: (T) => Boolean): Future[T] = {
val promise = Promise[T]()
// add both the predicate and the promise to your pending filters
synchronized(pendingFilters = p -> promise :: pendingFilters)
promise.future
}
def update(t: T): Unit = {
synchronized {
// partition your pending filters into completed and uncompleted ones
val (completed, pending) = pendingFilters.partition(_._1(t))
pendingFilters = pending
completed
}.foreach{ case (_, promise) =>
// and finally complete the promises
promise.trySuccess(t)
}
}
}

Related

Why are Futures in scala realised using a collection in the companion object?

I'm not sure whether I chose the right title for my question..
I'm interested as to why the collection in the companion object is defined. Am I mistaken that this collection will have only one f in it? What I am seeing is a collection with exactly one element.
Here's the Future I'm dealing with:
trait Future[+T] { self =>
def onComplete(callback: Try[T] => Unit): Unit
def map[U](f: T => U) = new Future[U] {
def onComplete(callback: Try[U] => Unit) =
self onComplete (t => callback(t.map(f)))
}
def flatMap[U](f: T => Future[U]) = new Future[U] {
def onComplete(callback: Try[U] => Unit) =
self onComplete { _.map(f) match {
case Success(fu) => fu.onComplete(callback)
case Failure(e) => callback(Failure(e))
} }
}
def filter(p: T => Boolean) =
map { t => if (!p(t)) throw new NoSuchElementException; t }
}
Its companion object:
object Future {
def apply[T](f: => T) = {
val handlers = collection.mutable.Buffer.empty[Try[T] => Unit]
var result: Option[Try[T]] = None
val runnable = new Runnable {
def run = {
val r = Try(f)
handlers.synchronized {
result = Some(r)
handlers.foreach(_(r))
}
}
}
(new Thread(runnable)).start()
new Future[T] {
def onComplete(f: Try[T] => Unit) = handlers.synchronized {
result match {
case None => handlers += f
case Some(r) => f(r)
}
}
}
}
}
In my head I was imagining something like the following instead of the above companion object (notice how I replaced the above val handlers .. with var handler ..):
object Future {
def apply[T](f: => T) = {
var handler: Option[Try[T] => Unit] = None
var result: Option[Try[T]] = None
val runnable = new Runnable {
val execute_when_ready: Try[T] => Unit = r => handler match {
case None => execute_when_ready(r)
case Some(f) => f(r)
}
def run = {
val r = Try(f)
handler.synchronized {
result = Some(r)
execute_when_ready(r)
}
}
}
(new Thread(runnable)).start()
new Future[T] {
def onComplete(f: Try[T] => Unit) = handler.synchronized {
result match {
case None => handler = Some(f)
case Some(r) => f(r)
}
}
}
}
}
So why does the function execute_when_ready leads to stackoverflow, but that's not the case with handlers.foreach? what is the collection is offering me which I can't do without it? And is it possible to replace the collection with something else in the companion object?
The collection is not in the companion object, it is in the apply method, so there is a new instance for each Future. It is there because there can be multiple pending onComplete handlers on the same Future.
Your implementation only allows a single handler and silently removes any existing handler in onComplete which is a bad idea because the caller has no idea if a previous function has added an onComplete handler or not.
As noted in the comments, the stack overflow is because execute_when_ready calls itself if handler is None with no mechanism to stop the recursion.

Scala Future flatMap implementation (chaining)

Trying to wrap my head around how async tasks are chained together, for ex futures and flatMap
val a: Future[Int] = Future { 123 }
val b: Future[Int] = a.flatMap(a => Future { a + 321 })
How would i implement something similar where i build a new Future by waiting for a first and applying a f on the result. I tried looking a scala's source code but got stuck here: https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/Future.scala#L216
i would imagine some code like:
def myFlatMap(a: Future[Int])(f: a => Future[Int]): Future[Int] = {
Future {
// somehow wait for a to complete and the apply 'f'
a.onComplete {
case Success(x) => f(a)
}
}
}
but i guess above will return a Future[Unit] instead. I would just like to understand the "pattern" of how async tasks are chained together.
(Disclaimer: I'm the main maintainer of Scala Futures)
You wouldn't want to implement flatMap/recoverWith/transformWith yourself—it is a very complex feature to implement safely (stack-safe, memory-safe, and concurrency-safe).
You can see what I am talking about here:
https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/impl/Promise.scala#L442
https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/impl/Promise.scala#L307
https://github.com/scala/scala/blob/2.13.x/src/library/scala/concurrent/impl/Promise.scala#L274
There is more reading on the topic here: https://viktorklang.com/blog/Futures-in-Scala-2.12-part-9.html
I found that looking at the source for scala 2.12 was easier for me to understand how Future / Promise was implemented, i learned that Future is in fact a Promise and by looking at the implementation i now understand how "chaining" async tasks can be implemented.
Im posting the code that i wrote to help me better understand what was going on under the hood.
import java.util.concurrent.{ScheduledThreadPoolExecutor, TimeUnit}
import scala.collection.mutable.ListBuffer
trait MyAsyncTask[T] {
type Callback = T => Unit
var result: Option[T]
// register a new callback to be called when task is completed.
def onComplete(callback: Callback): Unit
// complete the given task, and invoke all registered callbacks.
def complete(value: T): Unit
// create a new task, that will wait for the current task to complete and apply
// the provided map function, and complete the new task when completed.
def flatMap[B](f: T => MyAsyncTask[B]): MyAsyncTask[B] = {
val wrapper = new DefaultAsyncTask[B]()
onComplete { x =>
f(x).onComplete { y =>
wrapper.complete(y)
}
}
wrapper
}
def map[B](f: T => B): MyAsyncTask[B] = flatMap(x => CompletedAsyncTask(f(x)))
}
/**
* task with fixed pre-calculated result.
*/
case class CompletedAsyncTask[T](value: T) extends MyAsyncTask[T] {
override var result: Option[T] = Some(value)
override def onComplete(callback: Callback): Unit = {
// we already have the result just call the callback.
callback(value)
}
override def complete(value: T): Unit = () // noop nothing to complete.
}
class DefaultAsyncTask[T] extends MyAsyncTask[T] {
override var result: Option[T] = None
var isCompleted = false
var listeners = new ListBuffer[Callback]()
/**
* register callback, to be called when task is completed.
*/
override def onComplete(callback: Callback): Unit = {
if (isCompleted) {
// already completed just invoke callback
callback(result.get)
} else {
// add the listener
listeners.addOne(callback)
}
}
/**
* trigger all registered callbacks to `onComplete`
*/
override def complete(value: T): Unit = {
result = Some(value)
listeners.foreach { listener =>
listener(value)
}
}
}
object MyAsyncTask {
def apply[T](body: (T => Unit) => Unit): MyAsyncTask[T] = {
val task = new DefaultAsyncTask[T]()
// pass in `complete` as callback.
body(task.complete)
task
}
}
object MyAsyncFlatMap {
/**
* helper to simulate async task.
*/
def delayedCall(body: => Unit, delay: Int): Unit = {
val scheduler = new ScheduledThreadPoolExecutor(1)
val run = new Runnable {
override def run(): Unit = {
body
}
}
scheduler.schedule(run, delay, TimeUnit.SECONDS)
}
def main(args: Array[String]): Unit = {
val getAge = MyAsyncTask[Int] { cb =>
delayedCall({ cb(66) }, 1)
}
val getName = MyAsyncTask[String] { cb =>
delayedCall({ cb("John") }, 2)
}
// same as: getAge.flatMap(age => getName.map(name => (age, name)))
val result: MyAsyncTask[(Int, String)] = for {
age <- getAge
name <- getName
} yield (age, name)
result.onComplete {
case (age, name) => println(s"hello $name $age")
}
}
}

Scala promise does not work if I encapsulate it in a commodity function

I have this code that I use in a spray handler
get {
def callService = {
val p = Promise[Option[DocumentRef]]
val fut = p.future
archiveService.getByHash(ZeroHash, {
result => p success result
})
fut
}
onComplete(OnCompleteFutureMagnet(callService)){
case Success(docRef) => {
val doc = docRef map {
x => x.title
} getOrElse "nothing"
complete("Done with " + doc)
}
case Failure(ex) => complete("error ${ex.getMessage}")
}
}
so I had the bright idea of writing the following function to encapsulate the work done to create a future out of a promise:
def callback2Future[T](funToCall: (T => Unit) => Any): Future[T] = {
val p = Promise[T]
val resultFuture = p future
def callbacklistener(arg: T): Unit = {
arg: T => p success arg
}
funToCall(callbacklistener)
resultFuture
}
And restructure the onComplete as:
onComplete(OnCompleteFutureMagnet(callback2Future(archiveService.getByHash(ZeroHash, _: Option[DocumentRef] => Unit)))) {
case Success(docRef) => {
...
}
In the original implementation with callservice, it works great (with great throughput too), with the callback2Future implementation I get a forever wait and it eventually times out. They seem the same to me, can anyone spot the error?
I believe that your problem is due to the infamous auto-Unit feature of Scala. Your function:
def callbacklistener(arg: T): Unit = {
arg: T => p success arg
}
will probably be interpreted as:
def callbacklistener(arg: T): Unit = {
{ arg: T => p success arg }
()
}
What you really want is probably:
def callbacklistener(arg: T): Unit = p success arg
To be clear, in your implementation you are defining a function callbackListener with return type Unit; in the body of this function you have an expression, { arg: T => p success arg }, whose value is of type T => Unit and is discarded; the Scala compiler will then put a free () in your code as the return type of the callbackListener is supposed to be Unit.

transforming a Seq[Future[X]] into an Enumerator[X]

Is there a way to turn a Seq[Future[X]] into an Enumerator[X] ? The use case is that I want to get resources by crawling the web. This is going to return a Sequence of Futures, and I'd like to return an Enumerator that will push the futures in the order in which they are first finished on to the Iteratee.
It looks like Victor Klang's Future select gist could be used to do this - though it looks pretty inefficient.
Note: The Iteratees and Enumerator's in question are those given by the play framework version 2.x, ie with the following imports: import play.api.libs.iteratee._
Using Victor Klang's select method:
/**
* "Select" off the first future to be satisfied. Return this as a
* result, with the remainder of the Futures as a sequence.
*
* #param fs a scala.collection.Seq
*/
def select[A](fs: Seq[Future[A]])(implicit ec: ExecutionContext):
Future[(Try[A], Seq[Future[A]])] = {
#scala.annotation.tailrec
def stripe(p: Promise[(Try[A], Seq[Future[A]])],
heads: Seq[Future[A]],
elem: Future[A],
tail: Seq[Future[A]]): Future[(Try[A], Seq[Future[A]])] = {
elem onComplete { res => if (!p.isCompleted) p.trySuccess((res, heads ++ tail)) }
if (tail.isEmpty) p.future
else stripe(p, heads :+ elem, tail.head, tail.tail)
}
if (fs.isEmpty) Future.failed(new IllegalArgumentException("empty future list!"))
else stripe(Promise(), fs.genericBuilder[Future[A]].result, fs.head, fs.tail)
}
}
I can then get what I need with
Enumerator.unfoldM(initialSeqOfFutureAs){ seqOfFutureAs =>
if (seqOfFutureAs.isEmpty) {
Future(None)
} else {
FutureUtil.select(seqOfFutureAs).map {
case (t, seqFuture) => t.toOption.map {
a => (seqFuture, a)
}
}
}
}
A better, shorter and I think more efficient answer is:
def toEnumerator(seqFutureX: Seq[Future[X]]) = new Enumerator[X] {
def apply[A](i: Iteratee[X, A]): Future[Iteratee[X, A]] = {
Future.sequence(seqFutureX).flatMap { seqX: Seq[X] =>
seqX.foldLeft(Future.successful(i)) {
case (i, x) => i.flatMap(_.feed(Input.El(x)))
}
}
}
}
I do realise that the question is a bit old already, but based on Santhosh's answer and the built-in Enumterator.enumerate() implementation I came up with the following:
def enumerateM[E](traversable: TraversableOnce[Future[E]])(implicit ec: ExecutionContext): Enumerator[E] = {
val it = traversable.toIterator
Enumerator.generateM {
if (it.hasNext) {
val next: Future[E] = it.next()
next map {
e => Some(e)
}
} else {
Future.successful[Option[E]] {
None
}
}
}
}
Note that unlike the first Viktor-select-based-solution this one preserves the order, but you can still start off all computations asynchronously before. So, for example, you can do the following:
// For lack of a better name
def mapEachM[E, NE](eventuallyList: Future[List[E]])(f: E => Future[NE])(implicit ec: ExecutionContext): Enumerator[NE] =
Enumerator.flatten(
eventuallyList map { list =>
enumerateM(list map f)
}
)
This latter method was in fact what I was looking for when I stumbled on this thread. Hope it helps someone! :)
You could construct one using the Java Executor Completeion Service (JavaDoc). The idea is to use create a sequence of new futures, each using ExecutorCompletionService.take() to wait for the next result. Each future will start, when the previous future has its result.
But please b e aware, that this might be not that efficient, because a lot of synchronisation is happening behind the scenes. It might be more efficient, to use some parallel map reduce for calculation (e.g. using Scala's ParSeq) and let the Enumerator wait for the complete result.
WARNING: Not compiled before answering
What about something like this:
def toEnumerator(seqFutureX: Seq[Future[X]]) = new Enumerator[X] {
def apply[A](i: Iteratee[X, A]): Future[Iteratee[X, A]] =
Future.fold(seqFutureX)(i){ case (i, x) => i.flatMap(_.feed(Input.El(x)))) }
}
Here is something I found handy,
def unfold[A,B](xs:Seq[A])(proc:A => Future[B])(implicit errorHandler:Throwable => B):Enumerator[B] = {
Enumerator.unfoldM (xs) { xs =>
if (xs.isEmpty) Future(None)
else proc(xs.head) map (b => Some(xs.tail,b)) recover {
case e => Some((xs.tail,errorHandler(e)))
}
}
}
def unfold[A,B](fxs:Future[Seq[A]])(proc:A => Future[B]) (implicit errorHandler1:Throwable => Seq[A], errorHandler:Throwable => B) :Enumerator[B] = {
(unfold(Seq(fxs))(fxs => fxs)(errorHandler1)).flatMap(unfold(_)(proc)(errorHandler))
}
def unfoldFutures[A,B](xsfxs:Seq[Future[Seq[A]]])(proc:A => Future[B]) (implicit errorHandler1:Throwable => Seq[A], errorHandler:Throwable => B) :Enumerator[B] = {
xsfxs.map(unfold(_)(proc)).reduceLeft((a,b) => a.andThen(b))
}
I would like to propose the use of a Broadcast
def seqToEnumerator[A](futuresA: Seq[Future[A]])(defaultValue: A, errorHandler: Throwable => A): Enumerator[A] ={
val (enumerator, channel) = Concurrent.broadcast[A]
futuresA.foreach(f => f.onComplete({
case Success(Some(a: A)) => channel.push(a)
case Success(None) => channel.push(defaultValue)
case Failure(exception) => channel.push(errorHandler(exception))
}))
enumerator
}
I added errorHandling and defaultValues but you can skip those by using onSuccess or onFailure, instead of onComplete

how to implement this future/state concept as a monad in scala

I'm trying to implement a container for a match (like in sports) result so that I can create matches between the winners of other matches. This concept is close to what a future monads is as it contains a to be defined value, and also close to a state monad as it hides state change. Being mostly a begginer on the topic I have implemented an initial version in scala that is surely improvable. I added a get method that I'm not sure was a good idea, and so far the only way to create a value would be Unknown(null) which is not as elegant as I'd hoped. What do you think I could do to improve this design?
case class Unknown[T](t : T) {
private var value : Option[T] = Option(t)
private var applicatives: List[T => Unit] = Nil
def set(t: T) {
if (known) {
value = Option(t)
applicatives.foreach(f => f(t))
applicatives = Nil
} else {
throw new IllegalStateException
}
}
def get : T = value.get
def apply(f: T => Unit) = value match {
case Some(x) => f(x);
case None => applicatives ::= f
}
def known = value == None
}
UPDATE: a usage example of the current implementation follows
case class Match(val home: Unknown[Team], val visit: Unknown[Team], val result: Unknown[(Int, Int)]) {
val winner: Unknown[Team] = Unknown(null)
val loser: Unknown[Team] = Unknown(null)
result.apply(result => {
if (result._1 > result._2) {
home.apply(t => winner.set(t))
visit.apply(t => loser.set(t))
} else {
home.apply(t => loser.set(t))
visit.apply(t => winner.set(t))
}
})
}
And a test snippet:
val definedUnplayedMatch = Match(Unknown(Team("A")), Unknown(Team("B")), Unknown(null));
val definedPlayedMatch = Match(Unknown(Team("D")), Unknown(Team("E")), Unknown((1,0)));
val undefinedUnplayedMatch = Match(Unknown(null), Unknown(null), Unknown(null));
definedUnplayedMatch.winner.apply(undefinedUnplayedMatch.home.set(_))
definedPlayedMatch.winner.apply(undefinedUnplayedMatch.visit.set(_))
undefinedUnplayedMatch.result.set((3,1))
definedUnplayedMatch.result.set((2,4))
undefinedUnplayedMatch.winner.get must be equalTo(Team("B"));
undefinedUnplayedMatch.loser.get must be equalTo(Team("D"));
UPDATE - CURRENT IDEA : I haven't had much time to work on this because my laptop broke down, but I though it would be useful to write the monad I have so far for those who are interested:
sealed abstract class Determine[+A] {
def map[B](f: A => B): Determine[B]
def flatMap[B](f: A => Determine[B]): Determine[B]
def filter(p: A => Boolean): Determine[A]
def foreach(b: A => Unit): Unit
}
final case class Known[+A](value: A) extends Determine[A] {
def map[B](f: A => B): Determine[B] = Known(f(value))
def flatMap[B](f: A => Determine[B]): Determine[B] = f(value)
def filter(p: A => Boolean): Determine[A] = if (p(value)) this else Unknown
def foreach(b: A => Unit): Unit = b(value)
}
final case class TBD[A](definer: () => A) extends Determine[A] {
private var value: A = _
def map[B](f: A => B): Determine[B] = {
def newDefiner(): B = {
f(cachedDefiner())
}
TBD[B](newDefiner)
}
def flatMap[B](f: A => Determine[B]): Determine[B] = {
f(cachedDefiner())
}
def filter(p: A => Boolean): Determine[A] = {
if (p(cachedDefiner()))
this
else
Unknown
}
def foreach(b: A => Unit): Unit = {
b(cachedDefiner())
}
private def cachedDefiner(): A = {
if (value == null)
value = definer()
value
}
}
case object Unknown extends Determine[Nothing] {
def map[B](f: Nothing => B): Determine[B] = this
def flatMap[B](f: Nothing => Determine[B]): Determine[B] = this
def filter(p: Nothing => Boolean): Determine[Nothing] = this
def foreach(b: Nothing => Unit): Unit = {}
}
I got rid of the set & get and now the TBD class receives instead a function that will define provide the value or null if still undefined. This idea works great for the map method, but the rest of the methods have subtle bugs.
For a simple approach, you don't need monads, with partial application is enough:
//some utilities
type Score=(Int,Int)
case class MatchResult[Team](winner:Team,loser:Team)
//assume no ties
def playMatch[Team](home:Team,away:Team)(score:Score)=
if (score._1>score._2) MatchResult(home,away)
else MatchResult(away,home)
//defined played match
val dpm= playMatch("D","E")(1,0)
//defined unplayed match, we'll apply the score later
val dum= playMatch("A","B")_
// a function that takes the dum score and applies it
// to get a defined played match from an undefined one
// still is a partial application of match because we don't have the final result yet
val uumWinner= { score:Score => playMatch (dpm.winner,dum(score).winner) _ }
val uumLoser= { score:Score => playMatch (dpm.loser,dum(score).loser) _}
//apply the scores
uumWinner (2,4)(3,1)
uumLoser (2,4)(0,1)
//scala> uumWinner (2,4)(3,1)
//res6: MatchResult[java.lang.String] = MatchResult(D,B)
//scala> uumLoser (2,4)(0,1)
//res7: MatchResult[java.lang.String] = MatchResult(A,E)
This is a starting point, I'm pretty sure it can be further refined. Maybe there we'll find the elusive monad. But I think an applicative functor will be enough.
I'll give another pass later...