execute DAG like operations in scala Future - scala

Question

I am working on use case where in I have to execute inter-dependent operations (defined as a Directed Acyclic Graph) using scala Future. Basically every operation (say node of DAG) will be executed in a Future and subsequent dependent nodes will be triggered (they should be in a Future too) once the current node Future completes. This will go on until every node has finished processing or one of them fails. So far I have (minimal code):
def run(node: Node, result: Result): Unit = {
val f: Future[(Node, Result)] = Future {
// process current Node
...
}
f onComplete {
case Success(x) =>
val n = x._1 // Current Node
val r = x._2 // Result of current Node
if (!n.isLeaf()) {
n.children.foreach { z =>
run(z, r)
}
}
case Failure(e) => throw e
}
}
Is this correct way to tackle this problem (Calling another Future in a callback)? Again I don't have proper way stop other running future once one of the node fails processing.
Can this be solved using Future composition? If so, how can I achieve that?
Thanks,
Pravin

Answer 1

Here is a more functional approach: instead of using Unit as a result of evaluation of the run/Future we can have a generic type. Usually you would want to work with results of the Future functionally, rather than with its side effects.
I've added type annotations and descriptive variable names so that it would be easier to understand. I also added a few cases to show how it will fail. You can also chose to recover rather than fail everything when failure occurs. However, for this problem if the child computation relies on parent value it's probably more reasonable to fail.
import scala.concurrent.{Await, Future}
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.duration._
import scala.util.Try
case class Node[T](value: T, children: List[Node[T]])
object DagFuture extends App {
def run[A, B](node: Node[A], result: B)(nodeEval: (Node[A], B) => B)(aggregator: List[B] => B): Future[B] = {
val nodeResult: Future[B] = Future(nodeEval(node, result))
val allResults: Future[List[B]] = nodeResult.flatMap(r => Future.sequence(nodeResult :: node.children.map(x => run(x, r)(nodeEval)(aggregator))))
val finalResult: Future[B] = allResults.map(cl => aggregator(cl))
finalResult
}
val debugSum = (l: List[Int]) => {
println(s"aggregating: $l")
l.sum
}
def debugNodeEval(f: (Node[Int], Int) => Int)(n: Node[Int], r: Int) = {
val eval = Try { f(n, r) }
println(s"node: $n, result: $r, eval: $eval")
eval.get
}
val debugNodeEvalDefault = debugNodeEval((n, r) => n.value + r) _
val singleNodeDag = Node(1, Nil)
val multiNodeDag = Node(1, List(Node(20, Nil), Node(300, Nil)))
println("\nSINGLE NODE DAG EXAMPLE:")
val singleNodeFuture = run(singleNodeDag, 0)(debugNodeEvalDefault)(debugSum)
val singleNodeResult = Await.result(singleNodeFuture, 5 seconds)
println(s"Single node result: $singleNodeResult")
println("\nDAG PATH LENGTH EXAMPLE:")
val pathLengthFuture = run(multiNodeDag, 0)(debugNodeEvalDefault)(debugSum)
val pathLengthResult = Await.result(pathLengthFuture, 5 seconds)
println(s"Path length: $pathLengthResult")
println("\nFAILED DAG ROOT NODE EXAMPLE:")
val failedRootNodeFuture = run(multiNodeDag, 0)(debugNodeEval((n, r) => throw new Exception))(debugSum)
val failedRootNodePromise = Await.ready(failedRootNodeFuture, 5 seconds)
println(s"Failed root node: ${failedRootNodePromise.value}")
println("\nFAILED DAG CHILD NODE EXAMPLE:")
val failedChildNodeFuture = run(multiNodeDag, 0)(debugNodeEval((n, r) => if (n.value == 300) throw new Exception else n.value + r))(debugSum)
val failedChildNodePromise = Await.ready(failedChildNodeFuture, 5 seconds)
println(s"Failed child node: ${failedChildNodePromise.value}")
}
Prints this:
SINGLE NODE DAG EXAMPLE:
node: Node(1,List()), result: 0, eval: Success(1)
aggregating: List(1)
Single node result: 1
DAG PATH LENGTH EXAMPLE:
node: Node(1,List(Node(20,List()), Node(300,List()))), result: 0, eval: Success(1)
node: Node(20,List()), result: 1, eval: Success(21)
node: Node(300,List()), result: 1, eval: Success(301)
aggregating: List(301)
aggregating: List(21)
aggregating: List(1, 21, 301)
Path length: 323
FAILED DAG ROOT NODE EXAMPLE:
node: Node(1,List(Node(20,List()), Node(300,List()))), result: 0, eval: Failure(java.lang.Exception)
Failed root node: Some(Failure(java.lang.Exception))
FAILED DAG CHILD NODE EXAMPLE:
node: Node(1,List(Node(20,List()), Node(300,List()))), result: 0, eval: Success(1)
node: Node(20,List()), result: 1, eval: Success(21)
aggregating: List(21)
node: Node(300,List()), result: 1, eval: Failure(java.lang.Exception)
Failed child node: Some(Failure(java.lang.Exception))
TL;DR
def run[A, B](node: Node[A], result: B)(nodeEval: (Node[A], B) => B)(aggregator: Traversable[B] => B): Future[B] = {
val nodeResult = Future(nodeEval(node, result))
val allResults = nodeResult flatMap { r => Future.sequence(nodeResult :: node.children.map { x => run(x, r)(nodeEval)(aggregator) }) }
allResults map aggregator
}
Loosely speaking it's just a Future.flatMap(result => Future.sequence(children ...)). When the parent Future completes its result is passed in flatMap to children computation. If parent Future fails the whole computation fails as well. sequence combines result from list of Futures into a single Future. A child Future is a parent to it's children, and so on recursively. Thus the same failure mode applies.