Play WS API: throttling request rates - scala

Question

I'm using the async Play WS Scala API to query a RESTful service. I wonder how I could process a List containing request URLs to be called via WSClient, but not more than one request per second (the service allows "only" 1 request per second per client). From a logical standpoint, the idea is to get an element (URL) from the list, make a request, then wait a certain amount of time before proceeding with the next element in the list.
using good old Thread.sleep in a non-blocking and asynchronous framework like Play is certainly a bad idea.
the same is probably true for things like ScheduledThreadPoolExecutor or other methods that require to spawn new threads.
How could I throttle the request rate without having a negative impact on the asynchronous and "as-less-threads-as-possible" nature of Play?

Answer 1

Suppose you have a list of URLs you watch to fetch:
val urls = List(
"http://www.google.com",
"http://stackoverflow.com",
"http://www.bing.com"
)
In Play 2.5.x, we can process these sequentially, and use akka.pattern.after to force an asynchronous delay between each call. We flatMap the Future result of a webservice call to something that will return the same value after one second.
Future.traverse(urls) { url =>
wsClient.url(url).get().flatMap { result =>
// potentially process `result` here
akka.pattern.after(1.second, actorSystem.scheduler)(Future.successful(result))
}
} // returns Future[List[WSResponse]]
This requires that you have a WSClient and ActorSystem component available, as well as in implicit ExecutionContext in scope.
In Play 2.4.x and earlier, you could do the same using Promise.timeout:
Future.traverse(urls) { url =>
wsClient.url(url).get().flatMap { result =>
// potentially process `result` here
Promise.timeout(result, 1.second)
akka.pattern.after(1.second, actorSystem.scheduler)(Future.successful(result))
}
}

Answer 2

Akka has a handy scheduler functionality here: http://doc.akka.io/docs/akka/current/scala/scheduler.html
Since Akka is already in Play, you don't need to import anything else.
It wouldn't be the cleanest or easily testable but you could something like:
val webserviceCall : Runnable = new Runnable {
override def run(): Unit = {
// do webservice call work
// figure out if you need to make more webservice calls, and if you do:
actorSystem.scheduler.scheduleOnce(0 seconds, 1 seconds, webserviceCall)
}
}
actorSystem.scheduler.scheduleOnce(0 seconds, webserviceCall)
Alternatively, you can use this Akka message throttler that someone made a while ago: http://doc.akka.io/docs/akka/snapshot/contrib/throttle.html
I have used it before (i think it was Akka 2.3 last year) but not sure if it will still work.