transforming a Seq[Future[X]] into an Enumerator[X] - scala

Question

Is there a way to turn a Seq[Future[X]] into an Enumerator[X] ? The use case is that I want to get resources by crawling the web. This is going to return a Sequence of Futures, and I'd like to return an Enumerator that will push the futures in the order in which they are first finished on to the Iteratee.
It looks like Victor Klang's Future select gist could be used to do this - though it looks pretty inefficient.
Note: The Iteratees and Enumerator's in question are those given by the play framework version 2.x, ie with the following imports: import play.api.libs.iteratee._

Answer 1

Using Victor Klang's select method:
/**
* "Select" off the first future to be satisfied. Return this as a
* result, with the remainder of the Futures as a sequence.
*
* #param fs a scala.collection.Seq
*/
def select[A](fs: Seq[Future[A]])(implicit ec: ExecutionContext):
Future[(Try[A], Seq[Future[A]])] = {
#scala.annotation.tailrec
def stripe(p: Promise[(Try[A], Seq[Future[A]])],
heads: Seq[Future[A]],
elem: Future[A],
tail: Seq[Future[A]]): Future[(Try[A], Seq[Future[A]])] = {
elem onComplete { res => if (!p.isCompleted) p.trySuccess((res, heads ++ tail)) }
if (tail.isEmpty) p.future
else stripe(p, heads :+ elem, tail.head, tail.tail)
}
if (fs.isEmpty) Future.failed(new IllegalArgumentException("empty future list!"))
else stripe(Promise(), fs.genericBuilder[Future[A]].result, fs.head, fs.tail)
}
}
I can then get what I need with
Enumerator.unfoldM(initialSeqOfFutureAs){ seqOfFutureAs =>
if (seqOfFutureAs.isEmpty) {
Future(None)
} else {
FutureUtil.select(seqOfFutureAs).map {
case (t, seqFuture) => t.toOption.map {
a => (seqFuture, a)
}
}
}
}

Answer 2

A better, shorter and I think more efficient answer is:
def toEnumerator(seqFutureX: Seq[Future[X]]) = new Enumerator[X] {
def apply[A](i: Iteratee[X, A]): Future[Iteratee[X, A]] = {
Future.sequence(seqFutureX).flatMap { seqX: Seq[X] =>
seqX.foldLeft(Future.successful(i)) {
case (i, x) => i.flatMap(_.feed(Input.El(x)))
}
}
}
}

Answer 3

I do realise that the question is a bit old already, but based on Santhosh's answer and the built-in Enumterator.enumerate() implementation I came up with the following:
def enumerateM[E](traversable: TraversableOnce[Future[E]])(implicit ec: ExecutionContext): Enumerator[E] = {
val it = traversable.toIterator
Enumerator.generateM {
if (it.hasNext) {
val next: Future[E] = it.next()
next map {
e => Some(e)
}
} else {
Future.successful[Option[E]] {
None
}
}
}
}
Note that unlike the first Viktor-select-based-solution this one preserves the order, but you can still start off all computations asynchronously before. So, for example, you can do the following:
// For lack of a better name
def mapEachM[E, NE](eventuallyList: Future[List[E]])(f: E => Future[NE])(implicit ec: ExecutionContext): Enumerator[NE] =
Enumerator.flatten(
eventuallyList map { list =>
enumerateM(list map f)
}
)
This latter method was in fact what I was looking for when I stumbled on this thread. Hope it helps someone! :)

Answer 4

You could construct one using the Java Executor Completeion Service (JavaDoc). The idea is to use create a sequence of new futures, each using ExecutorCompletionService.take() to wait for the next result. Each future will start, when the previous future has its result.
But please b e aware, that this might be not that efficient, because a lot of synchronisation is happening behind the scenes. It might be more efficient, to use some parallel map reduce for calculation (e.g. using Scala's ParSeq) and let the Enumerator wait for the complete result.

Answer 5

WARNING: Not compiled before answering
What about something like this:
def toEnumerator(seqFutureX: Seq[Future[X]]) = new Enumerator[X] {
def apply[A](i: Iteratee[X, A]): Future[Iteratee[X, A]] =
Future.fold(seqFutureX)(i){ case (i, x) => i.flatMap(_.feed(Input.El(x)))) }
}

Answer 6

Here is something I found handy,
def unfold[A,B](xs:Seq[A])(proc:A => Future[B])(implicit errorHandler:Throwable => B):Enumerator[B] = {
Enumerator.unfoldM (xs) { xs =>
if (xs.isEmpty) Future(None)
else proc(xs.head) map (b => Some(xs.tail,b)) recover {
case e => Some((xs.tail,errorHandler(e)))
}
}
}
def unfold[A,B](fxs:Future[Seq[A]])(proc:A => Future[B]) (implicit errorHandler1:Throwable => Seq[A], errorHandler:Throwable => B) :Enumerator[B] = {
(unfold(Seq(fxs))(fxs => fxs)(errorHandler1)).flatMap(unfold(_)(proc)(errorHandler))
}
def unfoldFutures[A,B](xsfxs:Seq[Future[Seq[A]]])(proc:A => Future[B]) (implicit errorHandler1:Throwable => Seq[A], errorHandler:Throwable => B) :Enumerator[B] = {
xsfxs.map(unfold(_)(proc)).reduceLeft((a,b) => a.andThen(b))
}

Answer 7

I would like to propose the use of a Broadcast
def seqToEnumerator[A](futuresA: Seq[Future[A]])(defaultValue: A, errorHandler: Throwable => A): Enumerator[A] ={
val (enumerator, channel) = Concurrent.broadcast[A]
futuresA.foreach(f => f.onComplete({
case Success(Some(a: A)) => channel.push(a)
case Success(None) => channel.push(defaultValue)
case Failure(exception) => channel.push(errorHandler(exception))
}))
enumerator
}
I added errorHandling and defaultValues but you can skip those by using onSuccess or onFailure, instead of onComplete