I have two scalaz.concurrent.Tasks which are performing a HTTP request to different servers.
I want to compose them in a manner similar to Future.firstCompletedOf, that is: run them both in parallel and get the result of the first one that successfully completes.
Unfortunately Task.gatherUnordered does not what I want since it runs every task to completion before returning the result.
Not sure how to do it in scalaz.concurrent natively, but this one works for me:
import scalaz.Nondeterminism._
import scalaz.std.either.eitherInstance
import scalaz.syntax.bitraverse._
def race[A, B](t1: Task[A], t2: Task[B]): Task[A \/ B] = {
Nondeterminism[Task].choose(t1, t2).map {
_.bimap(_._1, _._2)
}
}
In fs2 - successor of scalaz.concurrent - it is fs2.async#race
While using bimap is indeed correct, there's an alternate implementation:
import scalaz.concurrent.Task
import scalaz.Nondeterminism
def firstOf[A, B, C](ta: Task[A], tb: Task[B])(fa: A => C, fb: B => C): Task[C] =
Nondeterminism[Task].chooseAny(ta.map(fa), Seq(tb.map(fb))).map(_._1)
val task1 = Task { Thread.sleep(10000); 4 }
val task2 = Task { Thread.sleep(5000); "test" }
firstOf(task1, task2)(_.toString, identity).unsafePerformSync // test
Here I'm assuming that non-deterministic retrieval of results is used to obtain equivalent values for which exact computation time is unknown. So the function incorporates concurrently-performed conversions fa and fb to the common type. It's good in the cases when conversion time is difficult to compute as well - it selects first result after conversion, for example, some request data extraction in the case of HTTP. For simpler cases, variant of race function that performs mapping in parallel is retrieved from firstOf as follows:
def race[A, B](ta: Task[A], tb: Task[B]): Task[A \/ B] = firstOf(ta, tb)(-\/(_), \/-(_))
Related
I'm looking for a way to lazily compose two effects without first executing their results in Zio. My program takes the following form:
/**
* Returns a reference to an effectful singleton cron scheduler backed by akka
* See https://github.com/philcali/cronish for more info on the API
*/
def scheduled: UManaged[Ref[Scheduled]] = ???
def schedule[R, E, A](e: => ZIO[R, E, A], crondef: String) =
(for {
resource <- scheduled
task <- ZManaged.fromEffect(e) // I need to lift the underlying effect here, not access its result
} yield resource.modify(schedule => schedule(job(task), crondef.cron) -> schedule)).flattenM
def scheduleEffect[A](e: => A, description: String = "")(crondef: String) =
(for {
resource <- scheduled
} yield resource.modify(schedule => schedule(job(e), crondef.cron) -> schedule)).flattenM
// Program which schedules cron jobs to increment/decrement x and y, respectively
def run(args: List[String]): URIO[ZEnv, ExitCode] = {
var x = 0
var y = 100
(for {
_ <- Scheduler.schedule(UIO({ x += 1; println(x) }), "every second")
_ <- Scheduler.scheduleEffect({ y -= 1; println(y) }, "every second")
} yield ())
.provideCustomLayer(???)
.as(ExitCode.success)
.useForever
}
In this current formulation, the decrementing of y runs every second until the program terminates, while the incrementing of x only runs once. I know that ZIO provides a Schedule utility, but for legacy compatibility reasons I have to stick with the effectful singleton used by the Cronish library. Basically job takes a pass-by-reference effect of type A and suspends it in a CronTask for execution within the Scheduled singleton according to the schedule defined by crondef.
What I am wondering is if it is possible to compose the effects themselves, rather than their results in the context of ZIO? I've basically wrapped the legacy cron scheduler in ZIO data types to manage the concurrency properly, but I still need the suspended effect from other ZIO-signature methods in my code to be available for me to pass down into the scheduler.
I ultimately found the solution by reading through the source code for ZIO.effectAsyncM, specifically noting its reference to ZIO.runtime[R]:
/**
* Imports an asynchronous effect into a pure `ZIO` value. This formulation is
* necessary when the effect is itself expressed in terms of `ZIO`.
*/
def effectAsyncM[R, E, A](
register: (ZIO[R, E, A] => Unit) => ZIO[R, E, Any]
): ZIO[R, E, A] =
for {
p <- Promise.make[E, A]
r <- ZIO.runtime[R] // This line right here!
a <- ZIO.uninterruptibleMask { restore =>
val f = register(k => r.unsafeRunAsync_(k.to(p)))
restore(f.catchAllCause(p.halt)).fork *> restore(p.await)
}
} yield a
Though I was not able to find a direct reference to this method in the ZIO docs, the Scaladocs were clear enough:
Returns an effect that accesses the runtime, which can be used to (unsafely) execute tasks. This is useful for integration with legacy code that must call back into ZIO code.
With this, my new implementation works beautifully as follows:
def schedule[R, E, A](e: => ZIO[R, E, A], crondef: String) =
(for {
resource <- scheduled
runtime <- ZManaged.fromEffect(ZIO.runtime[R])
} yield resource.modify({
schedule => schedule(job(runtime.unsafeRun(e)), crondef.cron) -> schedule
})).flattenM
I have the following scenario:
a List[Building] collection I'll call buildings
a def getAddress(building: Building): Future[Address] function
the result should be a List[BuildingWithAddress] collection
I want to iterate over the List[Building] and for each element call getAddress and wait for it to complete so that I can create a new object of type BuildingWithAddress and store it in a collection that I'll then return to the caller.
I thought I'd use a for-comprehension but it turns out that something along the line of this won't really work:
for {
building <- listOfBuildings
address <- getAddress(building)
buildingWithAddress = BuildingWithAddress(name = building.name, town = address.town)
} yield buildingWithAddress
I also thought on using flatMap to iterate over the list and then do the same for the address but the types are different and it won't work.
I tried with a forEach but then again the forEach isn't waiting on the Future to complete.
What's the solution for such a simple use case?
You cannot usually combine different monads in a single for comprehension (except for scala's collection-likes). Here you want to combine the Future and List monads, which cannot be done this way.
If you want to do this in a "sequential" way (waiting for previous future operations to finish before starting new ones) you need to use a ListT monad transformer from scalaz (or cats) like this:
import scalaz.ListT
import scalaz.std.scalaFuture._
implicit executor: ExecutionContext = ...
def fromList[A](x: List[A]) = ListT(Future.successful(x))
def single[A](x: Future[A]) = ListT(x.map(List(_)))
(for {
building <- fromList(listOfBuildings)
address <- single(getAddress(building))
buildingWithAddress = BuildingWithAddress(name = building.name, town = address.town)
} yield buildingWithAddress).run
Which will result in a Future[List[...]] as you need.
Alternative solution if you are fine with calling the getAddress functions in parallel:
Future.traverse(listOfBuildings)(building =>
getAddress(building).map(address =>
BuildingWithAddress(name = building.name, town = address.town)))
This is traverses the List "applicatively" (meaning: in parallel).
use Future.sequence
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
val l = List(1,2,3)
val futureList: Seq[Future[Int]] = l.map(e=>Future(e))
//conversion List[Future[T]] => Future[List[T]]
val singleFuture: Future[Seq[Int]] = Future.sequence(futureList)
singleFuture.map(_.length)
I am trying to build a new control structure which create a thread for each of its argument, and run them in parallel. The code seems to be fine when I built two future manually for each input, because I see the fast thread finishes before the slow thread.
Here is output:
fast
slow
However, if I use List(a, b).map(f => Future {f}) then I always see fast thread is executed after slow is done. Here is the output:
slow
fast
Can someone explain this?
code pasted here:
import java.util.concurrent.Executors
import scala.concurrent.{ExecutionContext, Future}
object ExecInParallel extends App {
def run(a: => Unit, b: => Unit): Unit = {
val executorService = Executors.newFixedThreadPool(2)
implicit val executionContext =
ExecutionContext.fromExecutorService(executorService)
// af and bf are executed in parallel
val af = Future(a)
val bf = Future(b)
// however, the following code is not parallel
List(a, b).map(f => Future(f))
Thread.sleep(3000)
executorService.shutdown
}
run(
{
Thread.sleep(2000)
println("slow")
},
{
Thread.sleep(1000)
println("fast")
}
)
}
That's because a and b are evaluated every time they are referenced in a non-by-name position and List(a, b) arguments are not by-name. From https://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_name:
Call by name is an evaluation strategy where the arguments to a function are not evaluated before the function is called—rather,... then left to be evaluated whenever they appear in the function. If an argument is not used in the function body, the argument is never evaluated; if it is used several times, it is re-evaluated each time it appears.
Effectively that is equivalent to this code:
List({
Thread.sleep(2000)
println("slow")
},
{
Thread.sleep(1000)
println("fast")
}).map(f => Future(f))
Since the List constructor doesn't take by-name arguments, these values are evaluated before the list itself is even constructed
This happens because you first create list of two call-by name values:
List(a, b)...
and until a and b not fully computed the map operation is not executed.
When List(a,b) is ready you wrap it in Futures:
List(a, b).map(f => Future(f))
Your by-name a and b is executed (sequentially) in List(a, b), before the construction of Future in map. If you check the inferred type of List(a, b) you'd see it's a List[Unit].
To achieve what you intented, you need a list of functions rather than list of results.
The following will work:
List(a _, b _).map(f => Future(f()))
I have a sequence of parameters. For each parameter I have to perform DB query, which may or may not return a result. Simply speaking, I need to stop after the first result is non-empty. Of course, I would like to avoid doing unnecessary calls. The caveat is - I need to have this operation(s) contained as a another Future - or any "most reactive" approach.
Speaking of code:
//that what I have
def dbQuery(p:Param): Future[Option[Result]] = {}
//my list of params
val input = Seq(p1,p2,p3)
//that what I need to implements
def getFirstNonEmpty(params:Seq[Param]): Future[Option[Result]]
I know I can possibly just wrap entire function in yet another Future and execute code sequentially (Await? Brrr...), but that not the cleanest solution.
Can I somehow create lazy initialized collection of futures, like
params.map ( p => FutureWhichWontStartUnlessAskedWhichWrapsOtherFuture { dbQuery(p) }).findFirst(!_.isEmpty())
I believe it's possible!
What do you think about something like this?
def getFirstNonEmpty(params: Seq[Param]): Future[Option[Result]] = {
params.foldLeft(Future.successful(Option.empty[Result])) { (accuFtrOpt, param) =>
accuFtrOpt.flatMap {
case None => dbQuery(param)
case result => Future.successful(result)
}
}
}
This might be overkill, but if you are open to using scalaz we can do this using OptionT and foldMap.
With OptionT we sort of combine Future and Option into one structure. We can get the first of two Futures with a non-empty result using OptionT.orElse.
import scalaz._, Scalaz._
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
val someF: Future[Option[Int]] = Future.successful(Some(1))
val noneF: Future[Option[Int]] = Future.successful(None)
val first = OptionT(noneF) orElse OptionT(someF)
first.run // Future[Option[Int]] = Success(Some(1))
We could now get the first non-empty Future from a List with reduce from the standard library (this will however run all the Futures) :
List(noneF, noneF, someF).map(OptionT.apply).reduce(_ orElse _).run
But with a List (or other collection) we can't be sure that there is at least one element, so we need to use fold and pass a start value. Scalaz can do this work for us by using a Monoid. The Monoid[OptionT[Future, Int]] we will use will supply the start value and combine the Futures with the orElse used above.
type Param = Int
type Result = Int
type FutureO[x] = OptionT[Future, x]
def query(p: Param): Future[Option[Result]] =
Future.successful{ println(p); if (p > 2) Some(p) else None }
def getFirstNonEmpty(params: List[Param]): Future[Option[Result]] = {
implicit val monoid = PlusEmpty[FutureO].monoid[Result]
params.foldMap(p => OptionT(query(p))).run
}
val result = getFirstNonEmpty(List(1,2,3,4))
// prints 1, 2, 3
result.foreach(println) // Some(3)
This is an old question, but if someone comes looking for an answer, here is my take. I solved it for a use case that required me to loop through a limited number of futures sequentially and stop when the first of them returned a result.
I did not need a library for my use-case, a light-weight combination of recursion and pattern matching was sufficient. Although the question here does not have the same problem as a sequence of futures, looping through a sequence of parameters would be similar.
Here would be the pseudo-code based on recursion.
I have not compiled this, fix the types being matched/returned.
def getFirstNonEmpty(params: Seq[Param]): Future[Option[Result]] = {
if (params.isEmpty) {
Future.successful(None)
} else {
val head = params.head
dbQuery(head) match {
case Some(v) => Future.successful(Some(v))
case None => getFirstNonEmpty(params.tail)
}
}
}
I've got an ADT that's essentially a cross between Option and Try:
sealed trait Result[+T]
case object Empty extends Result[Nothing]
case class Error(cause: Throwable) extends Result[Nothing]
case class Success[T](value: T) extends Result[T]
(assume common combinators like map, flatMap etc are defined on Result)
Given an Iteratee[A, Result[B] called inner, I want to create a new Iteratee[Result[A], Result[B]] with the following behavior:
If the input is a Success(a), feed a to inner
If the input is an Empty, no-op
If the input is an Error(err), I want inner to be completely ignored, instead returning a Done iteratee with the Error(err) as its result.
Example Behavior:
// inner: Iteratee[Int, Result[List[Int]]]
// inputs:
1
2
3
// output:
Success(List(1,2,3))
// wrapForResultInput(inner): Iteratee[Result[Int], Result[List[Int]]]
// inputs:
Success(1)
Success(2)
Error(Exception("uh oh"))
Success(3)
// output:
Error(Exception("uh oh"))
This sounds to me like the job for an Enumeratee, but I haven't been able to find anything in the docs that looks like it'll do what I want, and the internal implementations are still voodoo to me.
How can I implement wrapForResultInput to create the behavior described above?
Adding some more detail that won't really fit in a comment:
Yes it looks like I was mistaken in my question. I described it in terms of Iteratees but it seems I really am looking for Enumeratees.
At a certain point in the API I'm building, there's a Transformer[A] class that is essentially an Enumeratee[Event, Result[A]]. I'd like to allow clients to transform that object by providing an Enumeratee[Result[A], Result[B]], which would result in a Transformer[B] aka an Enumeratee[Event, Result[B]].
For a more complex example, suppose I have a Transformer[AorB] and want to turn that into a Transformer[(A, List[B])]:
// the Transformer[AorB] would give
a, b, a, b, b, b, a, a, b
// but the client wants to have
a -> List(b),
a -> List(b, b, b),
a -> Nil
a -> List(b)
The client could implement an Enumeratee[AorB, Result[(A, List[B])]] without too much trouble using Enumeratee.grouped, but they are required to provide an Enumeratee[Result[AorB], Result[(A, List[B])] which seems to introduce a lot of complication that I'd like to hide from them if possible.
val easyClientEnumeratee = Enumeratee.grouped[AorB]{
for {
_ <- Enumeratee.dropWhile(_ != a) ><> Iteratee.ignore
headResult <- Iteratee.head.map{ Result.fromOption }
bs <- Enumeratee.takeWhile(_ == b) ><> Iteratee.getChunks
} yield headResult.map{_ -> bs}
val harderEnumeratee = ??? ><> easyClientEnumeratee
val oldTransformer: Transformer[AorB] = ... // assume it already exists
val newTransformer: Transformer[(A, List[B])] = oldTransformer.andThen(harderEnumeratee)
So what I'm looking for is the ??? to define the harderEnumeratee in order to ease the burden on the user who already implemented easyClientEnumeratee.
I guess the ??? should be an Enumeratee[Result[AorB], AorB], but if I try something like
Enumeratee.collect[Result[AorB]] {
case Success(ab) => ab
case Error(err) => throw err
}
the error will actually be thrown; I actually want the error to come back out as an Error(err).
Simplest implementation of such would be Iteratee.fold2 method, that could collect elements until something is happened.
Since you return single result and can't really return anything until you verify there is no errors, Iteratee would be enough for such a task
def listResults[E] = Iteratee.fold2[Result[E], Either[Throwable, List[E]]](Right(Nil)) { (state, elem) =>
val Right(list) = state
val next = elem match {
case Empty => (Right(list), false)
case Success(x) => (Right(x :: list), false)
case Error(t) => (Left(t), true)
}
Future(next)
} map {
case Right(list) => Success(list.reverse)
case Left(th) => Error(th)
}
Now if we'll prepare little playground
import scala.concurrent.ExecutionContext.Implicits._
import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
val good = Enumerator.enumerate[Result[Int]](
Seq(Success(1), Empty, Success(2), Success(3)))
val bad = Enumerator.enumerate[Result[Int]](
Seq(Success(1), Success(2), Error(new Exception("uh oh")), Success(3)))
def runRes[X](e: Enumerator[Result[X]]) : Result[List[X]] = Await.result(e.run(listResults), 3 seconds)
we can verify those results
runRes(good) //res0: Result[List[Int]] = Success(List(1, 2, 3))
runRes(bad) //res1: Result[List[Int]] = Error(java.lang.Exception: uh oh)