I am trying to get the grips of using Iteratees in Play 2 to stream comet results. I have got the handle go creating an enumerator from a callback and an enumeratee from a map.
My issue is with the Enumeratee.map, this takes a function that takes a pure input and returns a pure output (e.g. the String to Int conversion in the doc). What I would like to do is take a pure input and return a promise of a result.
After all, an enumerator feeds an enumeratee with promises, an enumeratee converts one enumerator to the other, so there should be a way to make an enumeratee that maps to promises.
Now, let me make an example to make this a bit clearer. Let's say that I have an http request coming in with a list of IDs to query in my database. Let's say that these ids represent rows in a database table and the request does a set of (long) computations on these rows and then returns a set of json objects representing the computation.
As I have long blocking stuff to do, it would be cool to stream that one ID at a time, so I would like to have an enumeratee pipeline that does:
query a row in the database (returns a promise of the row)
make a long computation on the row (takes a row and returns a promise of a computation)
convert the long computation to JSON
&> this out to the Comet enumeratee provided by Play 2
1 is kinda easy, I can construct an enumerator with a fromCallback that will return a promise of the query result.
3 is also kinda easy, as it's a simple Enumeratee.map
But I can't wrap my head around how to implement the applyOn of the enumeratee of step 2. I can of understood that I have bo build a new iteratee that get the promise from the "inner" iteratee, flatMap the long calculation and return the new promise. What I don't get is how to make this given the odd applyOn signature: def applyOn[A](it: Iteratee[To, A]): Iteratee[From, Iteratee[To, A]]
Can someone help me with that?
Thanks
There is a method on master Enumeratee.mapM[E] that takes f: E => Promise[NE] and returns Enumeratee[E, NE]
https://github.com/playframework/Play20/blob/master/framework/src/play/src/main/scala/play/api/libs/iteratee/Enumeratee.scala#L150
The signature of applyOn makes more sense when you think that the enumeratee combines with an iteratee on the right like in enumerator |>> (enumeratee &> iteratee). iteratee has type Iteratee[E, A] and enumerator expects a Iteratee[Promise[E], Iteratee[E, A] so that the inner iteratee can be extracted. So applyOn will have to a Iteratee[E, A] and return a Iteratee[Promise[E], Iteratee[E, A].
Here is the outline of an implementation. It defines a step function that takes an input and return the expected result. Then recursively steps through the promise elements.
import play.api.libs.concurrent._
import play.api.libs.iteratee._
def unpromise[E]: Enumeratee[Promise[E], E] = new Enumeratee[Promise[E], E] {
def applyOn[A](inner: Iteratee[E, A]): Iteratee[Promise[E], Iteratee[E, A]] = {
def step(input: Input[Promise[E]], i: Iteratee[E, A]): Iteratee[Promise[E], Iteratee[E, A]] = {
input match {
case Input.EOF => Done(i, Input.EOF)
case Input.Empty => Cont(step(_, i))
case Input.El(pe) =>
val pe2 = pe.map(e => i.feed(Input.El(e))).flatMap(identity)
val i2 = Iteratee.flatten(pe2)
i2.pureFlatFold(
(a, e2) => Done(i2, Input.Empty),
k => Cont(step(_, i2)),
(msg, e2) => Done(i2, Input.Empty))
}
}
// should check that inner is not done or error - skipped for clarity
Cont(step(_, inner))
}
}
I'm discarding e2, so there is probably more code to ensure some input is not lost.
Related
Given two futures f1 and f2, I'm trying to return the one that completes its execution first using this method:
def first[A](f1: Future[A], f2: Future[A]): Future[A] = ???
As an example of an implementation, a promise is used to fetch the first future that finish its computation, something like this:
def first[A](f1: Future[A], f2: Future[A]): Future[A] =
val promise = Promise[A]()
f1.onComplete(result1 => promise.tryComplete(result1))
f2.onComplete(result2 => promise.tryComplete(result2))
promise.future
But I was wondering, is there any other approach to this problem that doesn't use the onComplete() method? (e.g., using the map() or flatMap() methods instead).
How would this implementation look like?
That's it, thanks so much for the help! I'm a newbie in this futures topic in Scala so I'm struggling a lil' bit with this. Have a nice day!
I have a method which returns some ZIO:
def method(...): ZIO[Any with clock, SomeError, Unit]
The method which calls this return Task[Unit]:
def otherMethod(..): Task[Unit] = {
ZIO.effect(method(...))
}
The problem is when I call it with ZIO.effect I don't get result.
How I should convert ZIO to Task to get result?
With ZIO.effect(method(...)) you get back a Task[ZIO[...]] that is rarely what you want (it's conceptually similar to a nested Future[Future[A]]).
In order to turn a ZIO[R, E, A] into a Taks[A], you have to understand that the latter is just a type alias for ZIO[Any, Throwable, A], which suggest that you have to
eliminate the dependency on the environment R (by providing it)
turn the error type E to Throwable if it's not already a sub-type of it (e.g. by .mapError)
This should work:
def otherMethod(..): Task[Unit] =
method(...)
.mapError(someError => new RuntimeException(s"failed with: $someError"))
.provideLayer(Clock.live)
// Simulated external API that synchronously returns elements one at a time indefinitely.
def externalApiGet[A](): A = ???
// This wraps with the proper fs2 stream that will indefinitely return values.
def wrapGetWithFS2[A](): Stream[Task, A] = Stream.eval(Task.delay(externalApiGet))
// Simulated external API that synchronously returns "chunks" of elements at a time indefinitely.
def externalApiGetSeq[A](): Seq[A] = ???
// How do I wrap this with a stream that hides the internal chunks and just provides a stream of A values.
// The following doesn't compile. I need help fixing this.
def wrapGetSeqWithFS2[A](): Stream[Task, A] = Stream.eval(Task.delay(externalApiGetSeq))
You need to mark the sequence as a Chunk and then use flatMap to flatten the stream.
def wrapGetSeqWithFS2[A](): Stream[Task, A] =
Stream.eval(Task.delay(externalApiGetSeq()))
.flatMap(Stream.emits)
(Edited to simplify solution)
Taking in account that any pattern matching on Future result matches Success[A] and Failure[A] (onComplete(), andThen()) (because they expect Try[A] as an argument, directly or through partial function), could there be a case when I would want to say explicitly that a function is of type Future[Try[A]]?
There are various constructs in Scala which carry failure case in them. There are Option, Either, Try and Future. (Futures main point is to abstract asynchronous operations, an error handling is there for convinience). Scalaz have even more: Validation (Disjunction and Maybe are better Either and Option).
They all have a bit different treatment of erroneous values. Yet Try and Future have very similar, both wrap Throwable. So IMHO, Future[Try[A]] doesn't add much information (about the error). Compare to having Future[Future[A]] or Try[Try[A]]. OTOH Future[Option[A]] or Future[Either[MyError, A]] make sense to me.
There might be sitatuon where you have for example potentially failing f: A => B and g: B => C and you'd like to avoid creating too much tasks in the ExecutionContext:
val a: Future[A] = ???
val c: Future[C] = a.map(f).map(g) // two tasks, not good
val c2: Future[Try[C]] = a.map(x => Try { f(x) } map g ) // one task, maybe better
// get will throw, but exception will be catched by Future's map
// Almost no information is lost compared to `c2`
// You cannot anymore distinguish between possible error in `a`
// and exceptions thrown by `f` or `g`.
val c3: Future[C] = a.map(x => Try { f (x) }.map(g).get)
In this case, I'd rather refactor f and g to have better types, at least: f: A => Option[B] and g: B => Option[C] then, ending up with Future[Option[C]].
Try[A] represents a synchronous computation that may fail.
Future[A] represents an asynchronous computation that may fail.
Under this definitions, Future[Try[A]] represents the result of a synchronous computation (that may fail) executed inside of an asynchronous computation (that may fail).
Does it make sense? Not to me, but I'm open to creative interpretations.
I'm fairly new to scalaz and I've started out with validations.
I have some validation functions of the form:
def validateXyz(...): ValidationNEL[String, String] = ...
I'm then using the applicative style to combine multiple validations and then call another function that also returns a validation:
(validateXyz(...) |#| validateAbc(...)) { (first, second) =>
otherFunction(first, second)
}
where,
def otherFunction(first: String, second: String): ValidationNEL[String, String] = ...
However, when calling the above the resulting type is:
val result: ValidationNEL[String, ValidationNEL[String, String]] = ...
I can unpack this by calling fold on the result with two functions, the first which just propagates the NEL as a fail and the second which just propagates its argument:
def propagateF(result: NonEmptyList[String]): ValidationNEL[String, String] = result.fail
def propagateV(result: ValidationNEL[String, String]) = result
result.fold(propagateF, propagateV)
// result type: ValidationNEL[String, String]
This works and returns the correct types and results. However it doesn't feel like the correct solution so I must be missing something. What do I need to do to avoid this horrible fold at the end?
What you are looking for here is monadic join.
The thing is that Validation itself is not really a monad, since the error side carries a Semigroup structure that cannot be preserved by Monad. But you can always drop down into the Either monad if you need to. This functionality is provided by flatMap.
(validateXyz(...) |#| validateAbc(...))(otherFunction).flatMap(x => x)
If you have an error on the outside, the result will be that error. If you have an error inside of a success, the result will be the inner error. Otherwise the result will be a success. Note the impossibility of having an error both on the inside and outside. This is why you have to use Applicative rather than Monad if you want to combine errors.