// Simulated external API that synchronously returns elements one at a time indefinitely.
def externalApiGet[A](): A = ???
// This wraps with the proper fs2 stream that will indefinitely return values.
def wrapGetWithFS2[A](): Stream[Task, A] = Stream.eval(Task.delay(externalApiGet))
// Simulated external API that synchronously returns "chunks" of elements at a time indefinitely.
def externalApiGetSeq[A](): Seq[A] = ???
// How do I wrap this with a stream that hides the internal chunks and just provides a stream of A values.
// The following doesn't compile. I need help fixing this.
def wrapGetSeqWithFS2[A](): Stream[Task, A] = Stream.eval(Task.delay(externalApiGetSeq))
You need to mark the sequence as a Chunk and then use flatMap to flatten the stream.
def wrapGetSeqWithFS2[A](): Stream[Task, A] =
Stream.eval(Task.delay(externalApiGetSeq()))
.flatMap(Stream.emits)
(Edited to simplify solution)
Related
Given two futures f1 and f2, I'm trying to return the one that completes its execution first using this method:
def first[A](f1: Future[A], f2: Future[A]): Future[A] = ???
As an example of an implementation, a promise is used to fetch the first future that finish its computation, something like this:
def first[A](f1: Future[A], f2: Future[A]): Future[A] =
val promise = Promise[A]()
f1.onComplete(result1 => promise.tryComplete(result1))
f2.onComplete(result2 => promise.tryComplete(result2))
promise.future
But I was wondering, is there any other approach to this problem that doesn't use the onComplete() method? (e.g., using the map() or flatMap() methods instead).
How would this implementation look like?
That's it, thanks so much for the help! I'm a newbie in this futures topic in Scala so I'm struggling a lil' bit with this. Have a nice day!
I would like to define a function with the following signature using fs2 Streams, cats EitherT and cats-effect IO.
def list2Stream[A,B,F[_],S](vs: List[A],
f: A => EitherT[IO,S,Stream[IO,B]]
): EitherT[IO,S,Stream[IO,B]] = {
???
}
which would map each value from vs to a stream of values and collect all those values in a new stream.
I tried something like:
vs.map(f).sequence.flatten
but it seems there is no implicit definition for Stream.
The following answer was provided in the gitter channel by Michael Pilquist:
vs.traverse(f).map(s => Stream.emits(s).flatten)
What's the use and meaning of second type parameter in akka.Source?
Sample Code:-
def stream: ServiceCall[Source[String, NotUsed], Source[String, NotUsed]]
According to as much code I have seen up to now by default second type parameter is set to akka.NotUsed. But I don't know what's the significance of it.
The second parameter is the materialized value, that is, when you run the source, it's the value that gets returned by the run method to you. All stream shapes in Akka streams have them, that is, sources, sinks, flows, bidiflows, etc. With sinks, it's really obvious, if you're folding the stream, you're going to end up with a single value, that value is given to you through the materialized value, which is a future of the result:
def fold[U, T](zero: U)(f: (U, T) ⇒ U): Sink[T, Future[U]]
For sources though, it's less obvious, but here's one example:
def actorRef[T](bufferSize: Int, overflowStrategy: OverflowStrategy): Source[T, ActorRef]
This is a Source that, when you run it, materializes to an ActorRef. Every message that you send to the actor will be emitted from the source. If you wanted to use this in Lagom, you would do something like this:
def stream = ServiceCall { _ =>
Source.actorRef[String](16, OverflowStrategy.dropHead)
.mapMaterializedValue { actor =>
// send messages here, or maybe pass the actor to somewhere else
actor ! "Hello world"
// And return NotUsed so that it now materializes to `NotUsed`, as required by Lagom
NotUsed
}
}
I already have a Source[T], but I need to pass it to a function that requires a Stream[T].
I could .run the source and materialize everything to a list and then do a .toStream on the result but that removes the lazy/stream aspect that I want to keep.
Is this the only way to accomplish this or am I missing something?
EDIT:
After reading Vladimir's comment, I believe I'm approaching my issue in the wrong way. Here's a simple example of what I have and what I want to create:
// Given a start index, returns a list from startIndex to startIndex+10. Stops at 50.
def getMoreData(startIndex: Int)(implicit ec: ExecutionContext): Future[List[Int]] = {
println(s"f running with $startIndex")
val result: List[Int] = (startIndex until Math.min(startIndex + 10, 50)).toList
Future.successful(result)
}
So getMoreData just emulates a service which returns data by the page to me.
My first goal it to create the following function:
def getStream(startIndex: Int)(implicit ec: ExecutionContext): Stream[Future[List[Int]]]
where the next Future[List[Int]] element in the stream depends on the previous one, taking the last index read from the previous Future's value in the stream. Eg with a startIndex of 0:
getStream(0)(0) would return Future[List[0 until 10]]
getStream(0)(1) would return Future[List[10 until 20]]
... etc
Once I have that function, I then want to create a 2nd function to further map it down to a simple Stream[Int]. Eg:
def getFlattenedStream(stream: Stream[Future[List[Int]]]): Stream[Int]
Streams are beginning to feel like the wrong tool for the job and I should just write a simple loop instead. I liked the idea of streams because the consumer can map/modify/consume the stream as they see fit without the producer needing to know about it.
Scala Streams are a fine way of accomplishing your task within getStream; here is a basic way to construct what you're looking for:
def getStream(startIndex : Int)
(implicit ec: ExecutionContext): Stream[Future[List[Int]]] =
Stream
.from(startIndex, 10)
.map(getMoreData)
Where things get tricky is with your getFlattenedStream function. It is possible to eliminate the Future wrapper around your List[Int] values, but it will require an Await.result function call which is usually a mistake.
More often than not it is best to operate on the Futures and allow asynchronous operations to happen on their own. If you analyze your ultimate requirement/goal it is usually not necessary to wait on a Future.
But, iff you absolutely must drop the Future then here is the code that can accomplish it:
val awaitDuration = 10 Seconds
def getFlattenedStream(stream: Stream[Future[List[Int]]]): Stream[Int] =
stream
.map(f => Await.result(f, awaitDuration))
.flatten
I am trying to get the grips of using Iteratees in Play 2 to stream comet results. I have got the handle go creating an enumerator from a callback and an enumeratee from a map.
My issue is with the Enumeratee.map, this takes a function that takes a pure input and returns a pure output (e.g. the String to Int conversion in the doc). What I would like to do is take a pure input and return a promise of a result.
After all, an enumerator feeds an enumeratee with promises, an enumeratee converts one enumerator to the other, so there should be a way to make an enumeratee that maps to promises.
Now, let me make an example to make this a bit clearer. Let's say that I have an http request coming in with a list of IDs to query in my database. Let's say that these ids represent rows in a database table and the request does a set of (long) computations on these rows and then returns a set of json objects representing the computation.
As I have long blocking stuff to do, it would be cool to stream that one ID at a time, so I would like to have an enumeratee pipeline that does:
query a row in the database (returns a promise of the row)
make a long computation on the row (takes a row and returns a promise of a computation)
convert the long computation to JSON
&> this out to the Comet enumeratee provided by Play 2
1 is kinda easy, I can construct an enumerator with a fromCallback that will return a promise of the query result.
3 is also kinda easy, as it's a simple Enumeratee.map
But I can't wrap my head around how to implement the applyOn of the enumeratee of step 2. I can of understood that I have bo build a new iteratee that get the promise from the "inner" iteratee, flatMap the long calculation and return the new promise. What I don't get is how to make this given the odd applyOn signature: def applyOn[A](it: Iteratee[To, A]): Iteratee[From, Iteratee[To, A]]
Can someone help me with that?
Thanks
There is a method on master Enumeratee.mapM[E] that takes f: E => Promise[NE] and returns Enumeratee[E, NE]
https://github.com/playframework/Play20/blob/master/framework/src/play/src/main/scala/play/api/libs/iteratee/Enumeratee.scala#L150
The signature of applyOn makes more sense when you think that the enumeratee combines with an iteratee on the right like in enumerator |>> (enumeratee &> iteratee). iteratee has type Iteratee[E, A] and enumerator expects a Iteratee[Promise[E], Iteratee[E, A] so that the inner iteratee can be extracted. So applyOn will have to a Iteratee[E, A] and return a Iteratee[Promise[E], Iteratee[E, A].
Here is the outline of an implementation. It defines a step function that takes an input and return the expected result. Then recursively steps through the promise elements.
import play.api.libs.concurrent._
import play.api.libs.iteratee._
def unpromise[E]: Enumeratee[Promise[E], E] = new Enumeratee[Promise[E], E] {
def applyOn[A](inner: Iteratee[E, A]): Iteratee[Promise[E], Iteratee[E, A]] = {
def step(input: Input[Promise[E]], i: Iteratee[E, A]): Iteratee[Promise[E], Iteratee[E, A]] = {
input match {
case Input.EOF => Done(i, Input.EOF)
case Input.Empty => Cont(step(_, i))
case Input.El(pe) =>
val pe2 = pe.map(e => i.feed(Input.El(e))).flatMap(identity)
val i2 = Iteratee.flatten(pe2)
i2.pureFlatFold(
(a, e2) => Done(i2, Input.Empty),
k => Cont(step(_, i2)),
(msg, e2) => Done(i2, Input.Empty))
}
}
// should check that inner is not done or error - skipped for clarity
Cont(step(_, inner))
}
}
I'm discarding e2, so there is probably more code to ensure some input is not lost.