Combining Futures dependent on each other - scala

Question

I'm using Scala to make HTTP GET requests to an API (Play Framework's WS, to be exact) which responds with a JSON response that looks like;
{
data: [
{text: "Hello there", id: 1},
{text: "Hello there again", id: 2}
],
next_url: 'http://request-this-for-more.com/api?page=2' //optional
}
So, the next_url field in the returned JSON may or may not be present.
What my method needs to do is start with calling the first URL, check if the response has a next_url and then do a GET on that. In the end, I should have all the data fields from the responses combined into one single future of all the data fields. I terminate when the response has no next_url present in it.
Now, doing this in a blocking way is easier, but I don't want to do that. What is the best way do tackle a problem like this?

Answer 1

There's probably a method to do this somewhere in scalaz, but if you don't know a specific solution it's usually possible to construct one with recursion and flatMap. Something like:
//Assume we have an async fetch method that returns Result and Option of next Url
def fetch(url: Url): Future[(Result, Option[Url])] = ...
//Then we can define fetchAll with recursion:
def fetchAll(url: Url): Future[Vector[Result]] =
fetch(url) flatMap {
case (result, None) => Future.successful(Vector(result))
case (result, Some(nextUrl)) =>
fetchAll(nextUrl) map {results => result +: results}
}
(Note that this uses a stack frame for each call - if you want to do thousands of fetches then we need to write it a little more carefully so that it's tail-recursive)

Answer 2

the Future.flatMap method exists exaclty for cases like that
Suppose you have such things:
case class Data(...)
def getContent(url:String):Future[String]
def parseJson(source:String):Try[JsValue]
def getData(value: JsValue):Seq[Data]
and JsValue type have methods inspired by play json library
def \ (fieldName: String): JsValue
def as[T](implicit ...):T //probably throwing exception
you could compose final result like
def innerContent(url:String):Future[Seq[Data]] = for {
first <- getContent(url)
json <- Future.fromTry(parseJson(first))
nextUrlAttempt = Try((json \ "next_url").as[String])
dataAttempt = Try(getData(json \ "data"))
data <- Future.fromTry(dataAttempt)
result <- nextUrlAttempt match {
case Success(nextUrl) => innerContent(nextUrl)
case Failure(_) => Future.successful(Seq())
} yield data ++ result
Also check out libraries that are targeted for complex asynchronous streams like your one:
play iteratees
scalaz iteratees
scalaz stream