How to handle asynchronous API response in scala - scala

I have an API that I need to query in scala. API returns a code that would be equal to 1 when results are ready.
I thought about an until loop to handle as the following:
var code= -1
while(code!=1){
var response = parse(Http(URL).asString.body)
code = response.get("code").get.asInstanceOf[BigInt].toInt
}
println(response)
But this code returns:
error: not found: value response
So I thought about doing the following:
var code = -1
var res = null.asInstanceOf[Map[String, Any]]
while(code!=1){
var response = parse(Http(URL).asString.body)
code = response.get("code").get.asInstanceOf[BigInt].toInt
res = response
}
println(res)
And it works. But I would like to know if this is really the best scala-friendly way to do so ?
How can I properly use a variable that outside of an until loop ?

When you say API, do you mean you use a http api and you're using a http library in scala, or do you mean there's some class/api written up in scala? If you have to keep checking then you have to keep checking I suppose.
If you're using a Scala framework like Akka or Play, they'd have solutions to asyncrhonously poll or schedule jobs in the background as a part of their solutions which you can read about.
If you're writing a Scala script, then from a design perspective I would either run the script every 1 minute and instead of having the while loop I'd just quit until code = 1. Otherwise I'd essentially do what you've done.
Another library that could help a scala script might be fs2 or ZIO which can allow you to setup tasks that periodically poll.
This appears to be a very open question about designing apps which do polling. A specific answer is hard to give.

You can just use simple recursion:
def runUntil[A](block: => A)(cond: A => Boolean): A = {
#annotation.tailrec
def loop(current: A): A =
if (cond(current)) current
else loop(current = block)
loop(current = block)
}
// Which can be used like:
val response = runUntil {
parse(Http(URL).asString.body)
} { res =>
res.getCode == 1
}
println(response)
An, if your real code uses some kind of effect type like IO or Future
// Actually cats already provides this, is called iterateUntil
def runUntil[A](program: IO[A])(cond: A => Boolean): IO[A] = {
def loop: IO[A] =
program.flatMap { a =>
if (cond(a)) IO.pure(a)
else loop
}
loop
}
// Used like:
val request = IO {
parse(Http(URL).asString.body)
}
val response = runUntil(request) { res =>
res.getCode == 1
}
response.flatMap(IO.println)
Note, for Future you would need to use () => Future instead to be able to re-execute the operation.
You can see the code running here.

Related

Forcing an AsyncStream[A] into a Seq[A] in scala

I am specifically using twitter's AsyncStream and I need to take the results of concurrent processing, and make it into a Seq, but the only way I've managed to get working is horrendous. This feels like it should be a one liner, but all my attempts with Await and force have either hung or not processed the work.
Here's what I have working - what's a more idiomatic way of doing this?
def processWork(work: AsyncStream[Work]): Seq[Result] = {
// TODO: make less stupid
val resultStream = work.flatMap { processWork }
var results : Seq[Result] = Nil
resultStream.foreach {
result => {
results = results :+ result
}
}
results
}
Like #MattFowler pointed out - you can force the stream and wait until it's complete using:
Await.result(resultStream.toSeq, 1.second)
toSeq will start realizing the stream and return a Future[Seq[A]] that completes once all the element are resolved, as documentated here.
You can then block until the future completes by using Await.result.
Make sure your stream is finite! Calling Await.result(resultStream.toSeq, 1.second) would hang forever.

Trying to understand Scala enumerator/iteratees

I am new to Scala and Play!, but have a reasonable amount of experience of building webapps with Django and Python and of programming in general.
I've been doing an exercise of my own to try to improve my understanding - simply pull some records from a database and output them as a JSON array. I'm trying to use the Enumarator/Iteratee functionality to do this.
My code follows:
TestObjectController.scala:
def index = Action {
db.withConnection { conn=>
val stmt = conn.createStatement()
val result = stmt.executeQuery("select * from datatable")
logger.debug(result.toString)
val resultEnum:Enumerator[TestDataObject] = Enumerator.generateM {
logger.debug("called enumerator")
result.next() match {
case true =>
val obj = TestDataObject(result.getString("name"), result.getString("object_type"),
result.getString("quantity").toInt, result.getString("cost").toFloat)
logger.info(obj.toJsonString)
Future(Some(obj))
case false =>
logger.warn("reached end of iteration")
stmt.close()
null
}
}
val consume:Iteratee[TestDataObject,Seq[TestDataObject]] = {
Iteratee.fold[TestDataObject,Seq[TestDataObject]](Seq.empty[TestDataObject]) { (result,chunk) => result :+ chunk }
}
val newIteree = Iteratee.flatten(resultEnum(consume))
val eventuallyResult:Future[Seq[TestDataObject]] = newIteree.run
eventuallyResult.onSuccess { case x=> println(x)}
Ok("")
}
}
TestDataObject.scala:
package models
case class TestDataObject (name: String, objtype: String, quantity: Int, cost: Float){
def toJsonString: String = {
val mapper = new ObjectMapper()
mapper.registerModule(DefaultScalaModule)
mapper.writeValueAsString(this)
}
}
I have two main questions:
How do i signal that the input is complete from the Enumerator callback? The documentation says "this method takes a callback function e: => Future[Option[E]] that will be called each time the iteratee this Enumerator is applied to is ready to take some input." but I am unable to pass any kind of EOF that I've found because it;s the wrong type. Wrapping it in a Future does not help, but instinctively I am not sure that's the right approach.
How do I get the final result out of the Future to return from the controller view? My understanding is that I would effectively need to pause the main thread to wait for the subthreads to complete, but the only examples I've seen and only things i've found in the future class is the onSuccess callback - but how can I then return that from the view? Does Iteratee.run block until all input has been consumed?
A couple of sub-questions as well, to help my understanding:
Why do I need to wrap my object in Some() when it's already in a Future? What exactly does Some() represent?
When I run the code for the first time, I get a single record logged from logger.info and then it reports "reached end of iteration". Subsequent runs in the same session call nothing. I am closing the statement though, so why do I get no results the second time around? I was expecting it to loop indefinitely as I don't know how to signal the correct termination for the loop.
Thanks a lot in advance for any answers, I thought I was getting the hang of this but obviously not yet!
How do i signal that the input is complete from the Enumerator callback?
You return a Future(None).
How do I get the final result out of the Future to return from the controller view?
You can use Action.async (doc):
def index = Action.async {
db.withConnection { conn=>
...
val eventuallyResult:Future[Seq[TestDataObject]] = newIteree.run
eventuallyResult map { data =>
OK(...)
}
}
}
Why do I need to wrap my object in Some() when it's already in a Future? What exactly does Some() represent?
The Future represents the (potentially asynchronous) processing to obtain the next element. The Option represents the availability of the next element: Some(x) if another element is available, None if the enumeration is completed.

How to handle an unknown number of futures that depend on the previous result?

I have a method that makes an API call and returns a Future[Seq[Post]], with each post having a sequential id. The API call returns 200 posts per call. In order to get a complete sequence of posts, I submit the smallest id (as maxId) in subsequent calls. Each Post object also contains a User object that contains a field specifying the total number of posts made by the user. What's the best way to accomplish getting all of a user's posts in a non-blocking manner?
Thus far, I have the following blocking code:
def completeTimeline(userId: Long) = {
def getTimeline(maxId = Option[Long] = None): Seq[Post] = {
val timeline: Future[Seq[Post]] = userPosts(userId) // calls method that returns Future[Seq[Post]]
Await.result(timeline, 5.seconds)
}
def recurseTimeline(timeline: Seq[Post]): Seq[Post] = {
if (timeline.length < timeline.head.user.map(user => user.post_count).getOrElse(0)) {
val maxId = timeline.min(Ordering.by((p: Post) => p.id)).id - 1 // API call with maxId is inclusive, subtract 1 to avoid duplicates
val nextTimeline = getTimeline(maxId = Some(maxId))
val newTimeline = timeline ++ nextTimeline
recurseTimeline(newTimeline)
} else timeline
}
recurseTimeline(getTimeline())
}
Something like this:
def completeTimeline(userId: Long): Future[Seq[Post]] = {
def getTimeline(maxId: Option[Long] = None): Future[Seq[Post]] = {
val timeline: Future[Seq[Post]] = userPosts(userId) // calls method that returns Future[Seq[Post]]
timeline
}
def recurseTimeline(timeline: Seq[Post]): Future[Seq[Post]] = {
if (timeline.length < timeline.head.user.map(user => user.post_count).getOrElse(0)) {
val maxId = timeline.min(Ordering.by((p: Post) => p.id)).id - 1 // API call with maxId is inclusive, subtract 1 to avoid duplicates
for {
nextTimeline <- getTimeline(maxId = Some(maxId))
full <- recurseTimeline(timeline ++ nextTimeline)
} yield full
} else Future.successful(timeline)
}
getTimeline() flatMap recurseTimeline
}
Essentially, you are scheduling things to happen once each future completes (map, flatMap, and for-comprehension), instead of waiting for them to complete.
The code can probably be simplified if i understood a bit better what you're trying to do.
When you say non-blocking that is incompatible with I need to check all of the Posts: you do need to wait for all of the Futures to complete - and thus be able to find the minimum id before proceeding. Correct? or would you like to further explain your intent?
Proceeding on that assumption - you can add each of the Future's to a list and then iterate on the list and calling Await on each one. This is in fact the standard blocking mechanism for obtaining a set of futures.
If there were further considerations and/or assumptions being made then please provide them - to allow tweaking this general approach for your particulars.

Slick - What if database does not contain result

I am trying to build a simple RESTful service that performs CRUD operations on a database and returns JSON. I have a service adhering to an API like this
GET mydomain.com/predictions/some%20string
I use a DAO which contains the following method that I have created to retrieve the associated prediction:
def getPrediction(rawText: String): Prediction = {
val predictionAction = predictions.filter{_.rawText === rawText}.result
val header = predictionAction.head
val f = db.run(header)
f.onComplete{case pred => pred}
throw new Exception("Oops")
}
However, this can't be right, so I started reading about Option. I changed my code accordingly:
def getPrediction(rawText: String): Option[Prediction] = {
val predictionAction = predictions.filter{_.rawText === rawText}.result
val header = predictionAction.headOption
val f = db.run(header)
f.onSuccess{case pred => pred}
None
}
This still doesn't feel quite right. What is the best way to invoke these filters, return the results, and handle any uncertainty?
I think the best way to rewrite your code is like this:
def getPrediction(rawText: String): Future[Option[Prediction]] = {
db.run(users.filter(_.rawText === rawText).result.headOption)
}
In other words, return a Future instead of the plain result. This way, the database actions will execute asynchronously, which is the preferred way for both Play and Akka.
The client code will then work with the Future. Per instance, a Play action would be like:
def prediction = Action.async {
predictionDao.getPrediction("some string").map { pred =>
Ok(views.html.predictions.show(pred))
}.recover {
case ex =>
logger.error(ex)
BadRequest()
}
}

Maintaining a growing list functionally

I'm writing this application (while learning scala and functional programing) that has a growing list that needs to be accessed. I have a way of implementing it but it doesn't seem like a good way of doing it.
I have a class that in another thread polls another source and appends the response to a list. There is another function that will once in a while use the latest response from that list and append to another list.
The only way Ive been able to come up with is having the polling function mutate the list. I do something like declaring it when the class is created. Either by var stream = Stream() or val list = collection.mutable.list. Then have the polling function update that with var stream = stream :+ response.
This doesn't seem like a good way of doing this?
case class example extends Runnable {
var responses = Stream[response] = Stream()
var validResponses = Stream[response] = Stream()
def run() {
while(true){
val r = pollFunction
responses = responses :+ r
Thread.sleep(time)
}
}
def get: response = {
lazy val r = responses.last
if( isValid(r) ) validResponses = validResponses :+ r
validResponses.last
}
}
I just happened to use stream but that can be replaced with collection.mutable or something. Im not sure if there is a way to do something like this within a more functional way?