How to handle recursion with monix's observable? - scala

Question

Using monix I'm trying to traverse a graph by building an Observable[Node] and using a breadth first algorithm.
However there I have a bit of a recursion problem. Here is a snippet illustrating my problem:
package gp
import monix.eval.Task
import monix.execution.Scheduler.Implicits.global
import monix.reactive._
object HelloObservable {
type Node = Int
//real case fetch next node across the network so the signature
//has to be Node -> List[Task[Node]]
def nexts(i : Node) : List[Task[Node]] =
List(Task(i), Task(i+1))
def go(i :Node) : Task[Iterator[List[Node]]] =
Task.sequence(nexts(i).sliding(100,100).map(Task.gatherUnordered))
def explore(r: Node): Observable[Node] = {
val firsts = for {
ilr <- Observable.fromTask(go(r))
lr <- Observable.fromIterator(ilr)
r <- Observable.fromIterable(lr)
} yield r
firsts ++ firsts.flatMap(explore)
}
def main(args : Array[String]) : Unit = {
val obs = explore(0)
val cancelable = obs
.dump("O")
.subscribe()
scala.io.StdIn.readLine()
}
}
The observable stop after the first iteration. Can anyone hint me why ?

Answer 1

I think the issue is not related to recursion. I think it comes from the fact that you use sliding which returns an Iterator. The major difference between Iterator and Iterable is that you can consume Iterator only once and after that all you are left with is an empty Iterator. It means when you do firsts.flatMap there is nothing left in the Observable.fromIterator(ilr) and so nothing is produced.
Fundamentally I don't think you can do a breadth-first search if you can't hold (most part of) the prefix in the memory. But since your nexts already returns List, I assume that you can afford having two copies of that list in the memory. And the second copy is a materialized result of the sliding. So your fixed code would be something like this:
object HelloObservable {
import monix.eval.Task
import monix.execution.Scheduler.Implicits.global
import monix.reactive._
type Node = Int
//real case fetch next node across the network so the signature
//has to be Node -> List[Task[Node]]
def nexts(i: Node): List[Task[Node]] = List(Task(i), Task(i + 1))
def go(i: Node): Task[List[List[Node]]] =
Task.sequence(nexts(i).sliding(100, 100).toList.map(Task.gatherUnordered))
def explore(r: Node): Observable[Node] = {
val firsts = for {
ilr <- Observable.fromTask(go(r))
lr <- Observable.fromIterable(ilr)
r <- Observable.fromIterable(lr)
} yield r
firsts ++ firsts.flatMap(explore)
}
def main(args: Array[String]): Unit = {
val obs = explore(0)
val cancelable = obs
.dump("O")
.subscribe()
scala.io.StdIn.readLine()
}
}