scala, play, futures: combining results from multiple futures - scala

I am using:
Scala 2.10
Play 2.1
Currently, I am using the Future class from scala.concurrent._, but I'm open to trying another API.
I am having trouble combining the results of multiple futures into a single List[(String, String)].
The following Controller method successfully returns the results of a single Future to an HTML template:
def test = Action { implicit request =>
queryForm.bindFromRequest.fold(
formWithErrors => Ok("Error!"),
query => {
Async {
getSearchResponse(query, 0).map { response =>
Ok(views.html.form(queryForm,
getAuthors(response.body, List[(String, String)]())))
}
}
})
}
The method getSearchResult(String, Int) performs a web service API call and returns a Future[play.api.libs.ws.Response]. The method getAuthors(String, List[(String, String)]) returns a List[(String, String)] to the HTML template.
Now, I am trying to call getSearchResult(String, Int) in a for loop to get several Response bodies. The following should give an idea of what I'm trying to do, but I get a compile-time error:
def test = Action { implicit request =>
queryForm.bindFromRequest.fold(
formWithErrors => Ok("Error!"),
query => {
Async {
val authors = for (i <- 0 to 100; if i % 10 == 0) yield {
getSearchResponse(query, i)
}.map { response =>
getAuthors(response.body, List[(String, String)]())
}
Ok(views.html.form(queryForm, authors))
}
})
}
type mismatch; found : scala.collection.immutable.IndexedSeq[scala.concurrent.Future[List[(String, String)]]] required: List[(String, String)]
How can I map the responses of several Future objects to a single Result?

Create a Future parametrized by a List or other Collection of the Result type.
From here:
In Play 1 you can do this:
F.Promise<List<WS.HttpResponse>> promises = F.Promise.waitAll(remoteCall1, remoteCall2, remoteCall3);
// where remoteCall1..3 are promises
List<WS.HttpResponse> httpResponses = await(promises); // request gets suspended here
In Play 2 less direct:
val httpResponses = for {
result1 <- remoteCall1
result2 <- remoteCall2
} yield List(result1, result2)

Related

Evaluating a Future Boolean asynchronously using Scala and Play

I have a method that returns a Future[Boolean] in a Play controller and i want to evaluate that using async but i can't seem to get it to compile.
The following will work:
def health = Action {
logger.info("Endpoint method: health")
val isHealthy = healthCheckService.checkDynamo()
val b: Boolean = Await.result(isHealthy, scala.concurrent.duration.Duration(5, "seconds"))
Ok(Json.toJson(HealthCheckResponse(b.toString)))
}
But i don't think i want that Await in there. So i'm trying things like this with no success:
def health =
Action.async {
Future {
logger.info("Endpoint method: health")
healthCheckService.checkDynamo() match {
case Future.successful(true) => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
val r = healthCheckService.checkDynamo() match {
case true => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
}
}
I can't even get those to compile to test them out.
Any suggestions?
Try this:
def health = Action.async {
healthCheckService.checkDynamo().map {
case true => Ok(Json.toJson("false"))
case false => Ok(Json.toJson("true"))
}
}
Let Play handle the awaiting for you under the hood. That is, Action.async accepts a Future, which checkDynamo() already returns. All you have to do is map it to the appropriate result.
With Futures you have to use combinators like map and flatMap to express the final value. For example:
Action.async {
healthCheckService.checkDynamo()
.map { result => // boolean
HealthCheckResponse(result.toString)
}
.map(Json.toJson(_))
.map(Ok(_))
}
(You can merge maps above to one map and construct the final Ok value there; it is more or less a matter of taste)
If you have, say, two async calls which you want to execute and return a result based on their results, you can use flatMap, which could be easily expressed using a for comprehension:
Action.async {
for {
result1 <- someService.someCall()
result2 <- anotherService.anotherCall(result1.someProperty)
finalResult = SomeFinalResultType(result1, result2)
} yield Ok(Json.toJson(finalResult))
}
If you are not familiar with futures, you might want to read some tutorial which explains their nature, how to combine them and how to get useful results from them, like this one: http://hello-scala.com/920-scala-futures.html

Scala - Batched Stream from Futures

I have instances of a case class Thing, and I have a bunch of queries to run that return a collection of Things like so:
def queries: Seq[Future[Seq[Thing]]]
I need to collect all Things from all futures (like above) and group them into equally sized collections of 10,000 so they can be serialized to files of 10,000 Things.
def serializeThings(Seq[Thing]): Future[Unit]
I want it to be implemented in such a way that I don't wait for all queries to run before serializing. As soon as there are 10,000 Things returned after the futures of the first queries complete, I want to start serializing.
If I do something like:
Future.sequence(queries)
It will collect the results of all the queries, but my understanding is that operations like map won't be invoked until all queries complete and all the Things must fit into memory at once.
What's the best way to implement a batched stream pipeline using Scala collections and concurrent libraries?
I think that I managed to make something. The solution is based on my previous answer. It collects results from Future[List[Thing]] results until it reaches a treshold of BatchSize. Then it calls serializeThings future, when it finishes, the loop continues with the rest.
object BatchFutures extends App {
case class Thing(id: Int)
def getFuture(id: Int): Future[List[Thing]] = {
Future.successful {
List.fill(3)(Thing(id))
}
}
def serializeThings(things: Seq[Thing]): Future[Unit] = Future.successful {
//Thread.sleep(2000)
println("processing: " + things)
}
val ids = (1 to 4).toList
val BatchSize = 5
val future = ids.foldLeft(Future.successful[List[Thing]](Nil)) {
case (acc, id) =>
acc flatMap { processed =>
getFuture(id) flatMap { res =>
val all = processed ++ res
val (batch, rest) = all.splitAt(5)
if (batch.length == BatchSize) { // if futures filled the batch with needed amount
serializeThings(batch) map { _ =>
rest // process the rest
}
} else {
Future.successful(all) //if we need more Things for a batch
}
}
}
}.flatMap { rest =>
serializeThings(rest)
}
Await.result(future, Duration.Inf)
}
The result prints:
processing: List(Thing(1), Thing(1), Thing(1), Thing(2), Thing(2))
processing: List(Thing(2), Thing(3), Thing(3), Thing(3), Thing(4))
processing: List(Thing(4), Thing(4))
When the number of Things isn't divisible by BatchSize we have to call serializeThings once more(last flatMap). I hope it helps! :)
Before you do Future.sequence do what you want to do with individual future and then use Future.sequence.
//this can be used for serializing
def doSomething(): Unit = ???
//do something with the failed future
def doSomethingElse(): Unit = ???
def doSomething(list: List[_]) = ???
val list: List[Future[_]] = List.fill(10000)(Future(doSomething()))
val newList =
list.par.map { f =>
f.map { result =>
doSomething()
}.recover { case throwable =>
doSomethingElse()
}
}
Future.sequence(newList).map ( list => doSomething(list)) //wait till all are complete
instead of newList generation you could use Future.traverse
Future.traverse(list)(f => f.map( x => doSomething()).recover {case th => doSomethingElse() }).map ( completeListOfValues => doSomething(completeListOfValues))

Calling multiple web services

I'm using Play 2.5 with Scala, I created a class that will call multiple times external web services.
External web services are called on some condition and get a simple response of ok or nok. If it is ok then I should update internal objects status and if nok I do nothing for now.
Here is my class, it takes a list of list of String as paramters and return a Future list of object to be handled in the controller.
def callWSAndGetResponse(listOfList: List[List[String]]): Future[List[MyObject]] = {
val res = listOfList map { listOfIds =>
listOfIds map { id =>
val foundObj = allMyObject.find(obj => obj.id == id)
if(foundObj.isDefined) {
foundObj.get.urls map { url =>
val futureReponse: Future[WSResponse] = ws.url(url).get()
futureResponse map { response =>
(response.json \ "response").as[String]
}
}
}
}
}
// if responses are ok create a list of MyObject to return for example.
}
val res is of type list of list but I would like it to be just a simple list of response.
1) How to simplify and correct my code in order to get just a list of response, for later check if ok or not ?
2) How to check if responses are ok or have failed ?
It looks like you want this block to return a List[Future[Json]], from then you can use "List[Future] to Future[List] disregarding failed futures" or similar.
To do this you should use map and flatMap (rather than isDefined/get):
val res = listOfList map {
_.map { id =>
allMyObject
.find(obj => obj.id == id)
.map {
_.flatMap {url =>
val futureReponse: Future[WSResponse] = ws.url(url).get()
futureResponse map { response =>
(response.json \ "response").as[String]
}
}
}
}
}
if blocks assign to AnyVal rather than to a specific type (which will cause you issues here):
scala> if (true) 23
23: AnyVal
scala> if (false) 23
(): AnyVal
Using map keeps you in the Option monad:
scala> (None: Option[Int]).map(_ * 2)
None: Option[Int]

Scala extract result from future

I am new to Scala, Play and using Futures.
I have the following Play class which makes an API call and encapsulates the result in a Future.
How can I extract the result from the Future?
class WikiArticle(url : String) {
var future : Future[WSResponse] = null
def queryApi(): Unit = {
val holder : WSRequest = WS.url(url)
future = {
holder.get()
}
future.onSuccess({
//How do I extract the result here?
});
}
Try to avoid extracting result from future. For this you can chain future calls using for comprehensions:
val chainResult = for {
result1 <- apiCallReturningFuture1;
result2 <- apiCallReturningFuture2(result1)
} yield result2
In given example result1 is 'extracted' result of Future apiCallReturningFuture1. Once result1 is obtained it is passed to call to apiCallReturningFuture2 and 'unwrapped' to result2. Finally chainResult is a future wrapping result2 and it is still Future! Through your API you can chain and transform your futures without awaiting it's result
In the long run you may want to return result of the future in controller. In Play Framework you can do it by using Action.async:
def load(id:Long) = Action.async {
repository.load(id)
.map {
case Some(x) => Ok(Json.toJson(x))
case None => NotFound
}
}
So I would not recommend await on futures except waiting in tests
future.onSuccess({
case result => result.json
})

type mismatch; found : scala.concurrent.Future[play.api.libs.ws.Response] required: play.api.libs.ws.Response

I am trying to make a post request to Pusher api, but I am having trouble returning the right type, I a type mismatch; found : scala.concurrent.Future[play.api.libs.ws.Response] required: play.api.libs.ws.Response
def trigger(channel:String, event:String, message:String): ws.Response = {
val domain = "api.pusherapp.com"
val url = "/apps/"+appId+"/channels/"+channel+"/events";
val body = message
val params = List(
("auth_key", key),
("auth_timestamp", (new Date().getTime()/1000) toInt ),
("auth_version", "1.0"),
("name", event),
("body_md5", md5(body))
).sortWith((a,b) => a._1 < b._1 ).map( o => o._1+"="+URLEncoder.encode(o._2.toString)).mkString("&");
val signature = sha256(List("POST", url, params).mkString("\n"), secret.get);
val signatureEncoded = URLEncoder.encode(signature, "UTF-8");
implicit val timeout = Timeout(5 seconds)
WS.url("http://"+domain+url+"?"+params+"&auth_signature="+signatureEncoded).post(body
}
The request you are making with post is asynchronous. That call returns immediately, but does not return a Response object. Instead, it returns a Future[Response] object, which will contain the Response object once the http request is completed asynchronously.
If you want to block execution until the request is completed, do:
val f = Ws.url(...).post(...)
Await.result(f)
See more about futures here.
Just append a map:
WS.url("http://"+domain+url+"?"+params+"&auth_signature="+signatureEncoded).post(body).map(_)
Assuming you don't want to create a blocking app, your method should also return a Future[ws.Response]. Let your futures bubble up to the Controller where you return an AsyncResult using Async { ... } and let Play handle the rest.
Controller
def webServiceResult = Action { implicit request =>
Async {
// ... your logic
trigger(channel, event, message).map { response =>
// Do something with the response, e.g. convert to Json
}
}
}