I have two methods:
def getNextJob: Future[Option[Job]]
def process(job: Job): Future[Unit]
I would like to process all Jobs until there are no jobs remaining.
I can do this with Await e.g.
private def process()(implicit ctx: ExecutionContext): Future[Unit] = {
var job: Option[Job] = Await.result(service.getNextJob, FiniteDuration(2, TimeUnit.SECONDS))
while(job.isDefined) {
Await.result(process(job.get), FiniteDuration(2, TimeUnit.SECONDS))
job = Await.result(service.getNextJob, FiniteDuration(2, TimeUnit.SECONDS))
}
Future.successful()
}
But this is ugly and doesn't use Futures properly. Is there a way I could chain the futures somehow to replace this?
def go()(implicit ctx: ExecutionContext): Future[Unit] =
getNextJob.flatMap { maybeJob ⇒
if(maybeJob.isDefined) process(maybeJob.get).flatMap(_ ⇒ go())
else Future.unit
}
Note: It is not tail recursive.
def processAll()(implicit ec: ExecutionContext): Future[Unit] =
getNextJob.flatMap {
case Some(job) => process(job).flatMap(_ => processAll())
case None => Future.unit
}
To process them all possibly concurrently:
def processAll()(implicit ec: ExecutionContext): Future[Unit] =
getNextJob.flatMap {
case Some(job) => process(job).zipWith(processAll())((_,_) => ())
case None => Future.unit
}
Related
In my Play application, I service my requests usings cats-effect's IO, instead of Future in the controller, like this (super-simplified):
def handleServiceResult(serviceResult: ServiceResult): Result = ...
def serviceMyRequest(request: Request): IO[ServiceResult] = ...
def myAction = Action { request =>
handleServiceResult(
serviceMyRequest(request).unsafeRunSync()
)
}
Requests are then processed (asynchronously) on Play's default thread pool. Now, I want to implement multiple thread pools to handle different sorts of requests. Were I using Futures, I could do this:
val myCustomExecutionContext: ExecutionContext = ...
def serviceMyRequest(request: Request): Future[ServiceResult] = ...
def myAction = Action.async { request =>
Future(serviceMyRequest(request))(myCustomExecutionContext)
.map(handleServiceResult)(defaultExecutionContext)
}
But I'm not using Futures, I'm using IO, and I'm not sure about the right way to go about implementing it. This looks promising, but seems a bit clunky:
def serviceMyRequest(request: Request): IO[ServiceResult] = ...
def myAction = Action { request =>
val ioServiceResult = for {
_ <- IO.shift(myCustomExecutionContext)
serviceResult <- serviceMyRequest(request)
_ <- IO.shift(defaultExecutionContext)
} yield {
serviceResult
}
handleServiceResult(ioServiceResult.unsafeRunSync())
}
Is this the right way to implement it? Is there a best practice here? Am I screwing up badly? Thanks.
Ok, so since this doesn't seem to be well-trodden ground, this is what I ended up implementing:
trait PlayIO { self: BaseControllerHelpers =>
implicit class IOActionBuilder[A](actionBuilder: ActionBuilder[Request, A]) {
def io(block: Request[A] => IO[Result]): Action[A] = {
actionBuilder.apply(block.andThen(_.unsafeRunSync()))
}
def io(executionContext: ExecutionContext)(block: Request[A] => IO[Result]): Action[A] = {
val shiftedBlock = block.andThen(IO.shift(executionContext) *> _ <* IO.shift(defaultExecutionContext))
actionBuilder.apply(shiftedBlock.andThen(_.unsafeRunSync()))
}
}
}
Then (using the framework from the question) if I mix PlayIO into the controller, I can do this,
val myCustomExecutionContext: ExecutionContext = ...
def handleServiceResult(serviceResult: ServiceResult): Result = ...
def serviceMyRequest(request: Request): IO[ServiceResult] = ...
def myAction = Action.io(myCustomExecutionContext) { request =>
serviceMyRequest(request).map(handleServiceResult)
}
such that I execute the action's code block on myCustomExecutionContext and then, once complete, thread-shift back to Play's default execution context.
Update:
This is a bit more flexible:
trait PlayIO { self: BaseControllerHelpers =>
implicit class IOActionBuilder[R[_], A](actionBuilder: ActionBuilder[R, A]) {
def io(block: R[A] => IO[Result]): Action[A] = {
actionBuilder.apply(block.andThen(_.unsafeRunSync()))
}
def io(executionContext: ExecutionContext)(block: R[A] => IO[Result]): Action[A] = {
if (executionContext == defaultExecutionContext) io(block) else {
val shiftedBlock = block.andThen(IO.shift(executionContext) *> _ <* IO.shift(defaultExecutionContext))
io(shiftedBlock)
}
}
}
}
Update2:
Per the comment above, this will ensure we always shift back to the default thread pool:
trait PlayIO { self: BaseControllerHelpers =>
implicit class IOActionBuilder[R[_], A](actionBuilder: ActionBuilder[R, A]) {
def io(block: R[A] => IO[Result]): Action[A] = {
actionBuilder.apply(block.andThen(_.unsafeRunSync()))
}
def io(executionContext: ExecutionContext)(block: R[A] => IO[Result]): Action[A] = {
if (executionContext == defaultExecutionContext) io(block) else {
val shiftedBlock = block.andThen { ioResult =>
IO.shift(executionContext).bracket(_ => ioResult)(_ => IO.shift(defaultExecutionContext))
}
io(shiftedBlock)
}
}
}
}
I have the feeling that I can abstract my code further but I am a little stuck. I am doing this more as a challenge, so don't worry about premature optimisation. It's for my open source library Reactive Kraken.
If you look at this code, I'm sure you'll get what I mean.
case class Response[T](error: List[String], result: Option[Map[String, T]])
case class OrderResponse[T](error: List[String], result: Option[T])
def handleRequest[T: JsonFormat](request: HttpRequest): Future[Response[T]] =
fireRequest(request)
.map(_.parseJson.convertTo[Response[T]])
.recover { case t: Throwable => Response[T](List(t.getMessage), None) }
def handleOrderRequest[A: JsonFormat](request: HttpRequest): Future[OrderResponse[A]] =
fireRequest(request)
.map(_.parseJson.convertTo[OrderResponse[A]])
.recover { case t: Throwable => OrderResponse[A](List(t.getMessage), None) }
The difference between these two functions is simply the type that they return. How would you go about this? Maybe using higher
kinded types?
One way would be to unify Response and OrderResponse:
case class Response[T] (error: List[String], result: Option[Map[String, T]])
case class OrderResponse[T](error: List[String], result: Option[T])
// ===
type Response[T] = OrderResponse[Map[String, T]]
case class OrderResponse[T](error: List[String], result: Option[T])
and then the function on OrderResponse is also one on Response. However, I don't know about the semantics of these types.
If that's not satisfactory, a typeclass will do:
trait Recovering[T] {
def recover(error: Throwable): T
}
object Recovering {
def apply[T: Recovering] = implicitly[Recovering[T]]
}
implicit def requestsAreRecovering[T]: Recovering[Request[T]] = new Recovering[Request[T]] {
override def recover(error: Throwable) = Request[T](List(t.getMessage), None)
}
implicit def orderRequestsAreRecovering[T]: Recovering[OrderRequest[T]] = new Recovering[OrderRequest[T]] {
override def recover(error: Throwable) = OrderRequest[T](List(t.getMessage), None)
}
def handleRequest[T: JsonFormat: Recovering](request: HttpRequest): Future[T] =
fireRequest(request)
.map(_.parseJson.convertTo[T])
.recover(Recovering[T].recover)
Without abstracting, you can still factorize your code :
case class Response[T](error: List[String], result: Option[Map[String, T]])
case class OrderResponse[T](error: List[String], result: Option[T])
private def handle[T: JsonFormat, U](request: HttpRequest, recovery: Throwable => U): Future[U] = {
fireRequest(request)
.map(_.parseJson.convertTo[U])
.recover { case t: Throwable => recovery(t) }
}
def handleRequest[T: JsonFormat](request: HttpRequest): Future[Response[T]] =
handle(request, {t: Throwable => Response[T](List(t.getMessage), None)})
def handleOrderRequest[T: JsonFormat](request: HttpRequest): Future[Response[T]] =
handle(request, {t: Throwable => OrderResponse[A](List(t.getMessage), None)})
My old code looks something like below, where all db calls blocking.
I need help converting this over to using Futures.
def getUserPoints(username: String): Option[Long]
db.getUserPoints(username) match {
case Some(userPoints) => Some(userPoints.total)
case None => {
if (db.getSomething("abc").isEmpty) {
db.somethingElse("asdf") match {
case Some(pointId) => {
db.setPoints(pointId, username)
db.findPointsForUser(username)
}
case _ => None
}
} else {
db.findPointsForUser(username)
}
}
}
}
My new API is below where I am returning Futures.
db.getUserPoints(username: String): Future[Option[UserPoints]]
db.getSomething(s: String): Future[Option[Long]]
db.setPoints(pointId, username): Future[Unit]
db.findPointsForUser(username): Future[Option[Long]]
How can I go about converting the above to use my new API that uses futures.
I tried using a for-compr but started to get wierd errors like Future[Nothing].
var userPointsFut: Future[Long] = for {
userPointsOpt <- db.getUserPoints(username)
userPoints <- userPointsOpt
} yield userPoints.total
But it gets a bit tricky with all the branching and if clauses and trying to convert it over to futures.
I would argue that the first issue with this design is that the port of the blocking call to a Future should not wrap the Option type:
The blocking call:
def giveMeSomethingBlocking(for:Id): Option[T]
Should become:
def giveMeSomethingBlocking(for:Id): Future[T]
And not:
def giveMeSomethingBlocking(for:Id): Future[Option[T]]
The blocking call give either a value Some(value) or None, the non-blocking Future version gives either a Success(value) or Failure(exception) which fully preserves the Option semantics in a non-blocking fashion.
With that in mind, we can model the process in question using combinators on Future. Let's see how:
First, lets refactor the API to something we can work with:
type UserPoints = Long
object db {
def getUserPoints(username: String): Future[UserPoints] = ???
def getSomething(s: String): Future[UserPoints] = ???
def setPoints(pointId:UserPoints, username: String): Future[Unit] = ???
def findPointsForUser(username: String): Future[UserPoints] = ???
}
class PointsNotFound extends Exception("bonk")
class StuffNotFound extends Exception("sthing not found")
Then, the process would look like:
def getUserPoints(username:String): Future[UserPoints] = {
db.getUserPoints(username)
.map(userPoints => userPoints /*.total*/)
.recoverWith{
case ex:PointsNotFound =>
(for {
sthingElse <- db.getSomething("abc")
_ <- db.setPoints(sthingElse, username)
points <- db.findPointsForUser(username)
} yield (points))
.recoverWith{
case ex: StuffNotFound => db.findPointsForUser(username)
}
}
}
Which type-checks correctly.
Edit
Given that the API is set in stone, a way to deal with nested monadic types is to define a MonadTransformer. In simple words, let's make Future[Option[T]] a new monad, let's call it FutureO that can be composed with other of its kind. [1]
case class FutureO[+A](future: Future[Option[A]]) {
def flatMap[B](f: A => FutureO[B])(implicit ec: ExecutionContext): FutureO[B] = {
val newFuture = future.flatMap{
case Some(a) => f(a).future
case None => Future.successful(None)
}
FutureO(newFuture)
}
def map[B](f: A => B)(implicit ec: ExecutionContext): FutureO[B] = {
FutureO(future.map(option => option map f))
}
def recoverWith[U >: A](pf: PartialFunction[Throwable, FutureO[U]])(implicit executor: ExecutionContext): FutureO[U] = {
val futOtoFut: FutureO[U] => Future[Option[U]] = _.future
FutureO(future.recoverWith(pf andThen futOtoFut))
}
def orElse[U >: A](other: => FutureO[U])(implicit executor: ExecutionContext): FutureO[U] = {
FutureO(future.flatMap{
case None => other.future
case _ => this.future
})
}
}
And now we can re-write our process preserving the same structure as the future-based composition.
type UserPoints = Long
object db {
def getUserPoints(username: String): Future[Option[UserPoints]] = ???
def getSomething(s: String): Future[Option[Long]] = ???
def setPoints(pointId: UserPoints, username:String): Future[Unit] = ???
def findPointsForUser(username: String): Future[Option[Long]] = ???
}
class PointsNotFound extends Exception("bonk")
class StuffNotFound extends Exception("sthing not found")
def getUserPoints2(username:String): Future[Option[UserPoints]] = {
val futureOpt = FutureO(db.getUserPoints(username))
.map(userPoints => userPoints /*.total*/)
.orElse{
(for {
sthingElse <- FutureO(db.getSomething("abc"))
_ <- FutureO(db.setPoints(sthingElse, username).map(_ => Some(())))
points <- FutureO(db.findPointsForUser(username))
} yield (points))
.orElse{
FutureO(db.findPointsForUser(username))
}
}
futureOpt.future
}
[1] with acknowledgements to http://loicdescotte.github.io/posts/scala-compose-option-future/
Let's say I have a function
def request(url: String): Future[String]
and need to write a function
def requestFirst(urls: List[String]): Future[String]
that calls request(url) in sequence untill it completes successfully (in this case the successful value is returned) or requests for all urls fail (in this case all failures are returned).
How can this be done in Scala?
def requestFirst(urls: List[String]): Future[String] = {
val default: Future[String] = Future.failed(new scala.Exception("all failed"))
urls.foldLeft(default)((prevFuture, currentUrl) => {
prevFuture fallbackTo (request(currentUrl))
})
}
OR
def requestFirst(urls: List[String]): Future[String] = {
def requestFirstInternal(urlSubset: List[String]): Future[String] = {
if(urlSubset.isEmpty) {
Future.failed(new Exception("Exhausted all urlSubset"))
} else {
request(urlSubset.head) fallbackTo {
requestFirstInternal(urlSubset.tail)
}
}
}
requestFirstInternal(urls)
}
This is what I came up with based on #parnav-shukla answer:
def requestFirst(urls: List[String]): Future[String] = {
def requestFirstInternal(urlSubset: List[String], errors: Seq[String]): Future[String] = {
if(urlSubset.isEmpty) {
Future.failed(new Exception(errors.mkString("\n")))
} else {
request(urlSubset.head) recoverWith { t =>
requestFirstInternal(urlSubset.tail, errors ++ Seq(t.getMessage))
}
}
}
requestFirstInternal(urls, Seq.empty)
}
I'm writing scala <-> java interop wrappers for Futures and I don't know the Right Way to implement scala.concurrent.Future.onComplete (http://www.scala-lang.org/api/current/index.html#scala.concurrent.Future). This probably works:
def onComplete[U](func: Try[T] => U)(implicit executor: ExecutionContext): Unit = {
executor.execute(new Runnable {
#tailrec
def run = value match {
case Some(t) => func(t)
case None => { Thread.sleep(100); run }
}
})
}
but Asynchronous IO in Scala with futures suggests that when I have to block I should pass the relevant part of the code to scala.concurrent.blocking to let the ExecutionContext know what's up. The problem is that when I surround the value match{...} with blocking {} it's no longer a tail call.
What's the proverbial right way to do this?
Edit: for completeness here is the entire wrapping class:
class JavaFutureWrapper[T](val jf: java.util.concurrent.Future[T]) extends scala.concurrent.Future[T] {
def isCompleted = jf.isDone
def result(atMost: Duration)(implicit permit: CanAwait): T =
atMost match { case Duration(timeout, units) => jf.get(timeout, units) }
def onComplete[U](func: Try[T] => U)(implicit executor: ExecutionContext): Unit = {
executor.execute(new Runnable {
#tailrec
def run = value match {
case Some(t) => func(t)
case None => { Thread.sleep(100); run }
}
})
}
def ready(atMost: Duration)(implicit permit: CanAwait): this.type = atMost match {
case Duration(timeout, units) => {
jf.get(timeout, units)
this
}
}
def value: Option[Try[T]] = (jf.isCancelled, jf.isDone) match {
case (true, _) => Some(Failure(new Exception("Execution was cancelled!")))
case (_, true) => Some(Success(jf.get))
case _ => None
}
}
I would just wait for the Java future to complete:
import scala.util.{Try, Success, Failure}
import scala.concurrent._
import java.util.concurrent.TimeUnit
class JavaFutureWrapper[T](val jf: java.util.concurrent.Future[T])
extends scala.concurrent.Future[T] {
...
def onComplete[U](func: Try[T] => U)(implicit executor: ExecutionContext): Unit =
executor.execute(new Runnable {
def run: Unit = {
val result = Try(blocking(jf.get(Long.MaxValue, TimeUnit.MILLISECONDS)))
func(result)
}
})
...
}
Hmm, my edit to 0__ 's answer didn't get approved, so for the sake of future readers, here's the solution I'm going with (which is simplified from 0__'s)
def onComplete[U](func: Try[T] => U)(implicit executor: ExecutionContext): Unit = {
executor.execute(new Runnable {
def run = func(Try( blocking { jf.get } ))
})
}