Given:
AtoB: GraphStageWithMaterializedValue[A, B, Future[AtoBRuntimeApi]];
BtoC: GraphStageWithMaterializedValue[B, C, Future[BtoCRuntimeApi]].
Wanted: AtoC: GraphStageWithMaterializedValue[A, C, Future[AtoCRuntimeApi]].
In my particular case it's really convenient to implement AtoCRuntimeApi in terms of both AtoBRuntimeApi and BtoARuntimeApi.
So I would like to define AtoCRuntimeApi as case class AtoCRuntimeApi(a2b: AtoBRuntimeApi, b2c: BtoCRuntimeApi);
And to define the new compound stage as stageAtoB.viaMat(stageBtoC)(combineIntoAtoC) where
combineIntoAtoC: (Future[AtoBRuntimeApi], Future[B2CRuntimeApi]) => Future[AtoCRuntimeApi].
Obviously the implementation of combineIntoAtoC requires some instance of ExecutionContext in order to map the futures.
The question: what execution context should I use in the described case?
Options, I would rather avoid:
bake in an instance that is currently available (while composition of stages) — the "blueprint" will not be safe to materialise if that execution-context becomes unavailable;
ExecutionContext.global — well, it's global. It seems terribly wrong to use it (probably, once per materialisation — is not that big deal).
The most wanted execution-context is the one that is available as the property of materializer (mat.executionContext). But there is no way I can access it within that combine-function.
I usually use the actor system context in such cases, which is indeed by default can be referred to from the materializer. As a general solution, though, you can pass the execution context to whatever function you construct the stream graph in with an implicit parameter:
def makeGraph(...)(implicit ec: ExecutionContext): RunnableGraph = {
def combineIntoAtoC(...) = ... // uses ec implicitly
}
This allows you to push the decision about which context to use up the call stack. At the appropriate level there most certainly will be some kind of access to the actor system's dispatcher.
The reason why I prefer to use the actor system's dispatcher instead of the global one is because it reduces the surface of dependencies - all execution contexts in this case come from one source, and you know how to configure them if the need arises.
Related
Need to fill in the methods next and hasNext and preserve laziness
new Iterator[T] {
val stream: fs2.Stream[IO, T] = ...
def next(): T = ???
def hasNext(): Boolean = ???
}
But cannot figure out how an earth to do this from a fs2.Stream? All the methods on a Stream (or on the "compiled" thing) are fairly useless.
If this is simply impossible to do in a reasonable amount of code, then that itself is a satisfactory answer and we will just rip out fs2.Stream from the codebase - just want to check first!
fs2.Stream, while similar in concept to Iterator, cannot be converted to one while preserving laziness. I'll try to elaborate on why...
Both represent a pull-based series of items, but the way in which they represent that series and implement the laziness differs too much.
As you already know, Iterator represents its pull in terms of the next() and hasNext methods, both of which are synchronous and blocking. To consume the iterator and return a value, you can directly call those methods e.g. in a loop, or use one of its many convenience methods.
fs2.Stream supports two capabilities that make it incompatible with that interface:
cats.effect.Resource can be included in the construction of a Stream. For example, you could construct a fs2.Stream[IO, Byte] representing the contents of a file. When consuming that stream, even if you abort early or do some strange flatMap, the underlying Resource is honored and your file handle is guaranteed to be closed. If you were trying to do the same thing with iterator, the "abort early" case would pose problems, forcing you to do something like Iterator[Byte] with Closeable and the caller would have to make sure to .close() it, or some other pattern.
Evaluation of "effects". In this context, effects are types like IO or Future, where the process of obtaining the value may perform some possibly-asynchronous action, and may perform side-effects. Asynchrony poses a problem when trying to force the process into a synchronous interface, since it forces you to block your current thread to wait for the asynchronous answer, which can cause deadlocks if you aren't careful. Libraries like cats-effect strongly discourage you from calling methods like unsafeRunSync.
fs2.Stream does allow for some special cases that prevent the inclusion of Resource and Effects, via its Pure type alias which you can use in place of IO. That gets you access to Stream.PureOps, but that only gets you methods that consume the whole stream by building a collection; the laziness you want to preserve would be lost.
Side note: you can convert an Iterator to a Stream.
The only way to "convert" a Stream to an Iterator is to consume it to some collection type via e.g. .compile.toList, which would get you an IO[List[T]], then .map(_.iterator) that to get an IO[Iterator[T]]. But ultimately that doesn't fit what you're asking for since it forces you to consume the stream to a buffer, breaking laziness.
#Dima mentioned the "XY Problem", which was poorly-received since they didn't really elaborate (initially) on the incompatibility, but they're right. It would be helpful to know why you're trying to make a Stream-to-Iterator conversion, in case there's some other approach that would serve your overall goal instead.
The use case is actually fairly typical. A lot of web services use authorization tokens that you retrieve at the start of a session and you need to send those back on subsequent requests.
I know I can do it like this:
lazy val myData = {
val request = ws.url("/some/url").withAuth(user, password, WSAuthScheme.BASIC).withHeaders("Accept" -> "application/json")
Await.result(request.get().map{x => x.json }, 120.seconds)
}
That just feels wrong as all the docs say never us Await.
Is there a Future/Promise Scala style way of handling this?
I've found .onComplete which allows me to run code upon the completion of a Promise however without using a (mutable) var I see no way of getting a value in that scope into a lazy val in a different scope. Even with a var there is a possible timing issue -- hence the evils of mutable variables :)
Any other way of doing this?
Unfortunately, there is no way to make this non-blocking - lazy vals are designed to be synchronous and to block any thread accessing them until they are completed with a value (internally a lazy val is represented as a simple synchronized block).
A Future/Promise Scala way would be to use a Future[T] or a Promise[T] instead of a val x: T, but that way implies a great deal of overhead with executionContexts and maps upon each use of the val, and more optimal resource utilization may not be worth the decreased readability in all cases, so it may be OK to leave the Await there if you extensively use the value in many parts of your application.
I'm trying to find the proper pattern for initializing an actor asynchronously, so that I can look up dependent ActorRefs it needs. I want to avoid using ActorSelection, since it's
ambiguous as to the number of actors it points to, and
has some overhead that's undesirable for many tells
Looking at the Actor LifeCycle it seems to be all pretty much synchronous until the message loop starts, including preStart, etc., which leads me to think that I have only one of two choices:
Use a factory method with a signature of Future[ActorRef]
All dependencies for constructing the actor are resolved asynchronously and passed in via Props.
The main problem with this approach is that you cannot use this factory to construct an actor inside of another actor, since it then has the same problem, i.e. it's turtles all the way down, wiring up the hierarchy of all actors and their dependencies asynchronously.
Use become and stash to transition the actor
The actor is created with actorOf, immediately resulting in an ActorRef but it starts in an Initialization state, does it's dependency resolution, stashing incoming messages in the meantime, and finally becomeing the Running state and unstashAlling.
This feels a lot more idiomatic for actors, even though my dependencies will all be var instead of val.
Both seem like a lot of overhead, making me wondering if I these are the best options or if I just haven't found the proper pattern in the docs.
There's no reason your dependencies have to be vars when using become:
val initializing: Actor.Receive = {
case Dependencies(d1, d2) => context.become(working(d1, d2))
}
def working(d1: Dependency, d2: Dependency): Actor.Receive = {
case msg => d1.fetch(...) // whatever
}
def receive = initializing
Also, actorFor is a) deprecated and b) doesn't create an actor.
I have an actor which takes the result from another actor and applies some check on it.
class Actor1(actor2:Actor2) {
def receive = {
case SomeMessage =>
val r = actor2 ? NewMessage()
r.map(someTransform).pipeTo(sender)
}
}
now if I make an ask of Actor1, we now have 2 futures generated, which doesnt seem overly efficient. Is there a way to provide a foward with some kind of continuation, or some other approach I could use here?
case SomeMessage => actor2.forward(NewMessage, someTransform)
Futures are executed in an ExecutionContext, which are like thread pools. Creating a new future is not as expensive as creating a new thread, but it has its cost. The best way to work with futures is to create as much as needed and compose then in a way that things that can be computed in parallel are computed in parallel if the necessary resources are available. This way you will make the best use of your machine.
You mentioned that akka documentation discourages excessive use of futures. I don't know where you read this, but what I think it means is to prefer transforming futures rather than creating your own. This is exactly what you are doing by using map. Also, it may mean that if you create a future where it is not needed you are adding unnecessary overhead.
In your case you have a call that returns a future and you need to apply sometransform and return the result. Using map is the way to go.
Background
i want to send a closure to a remote actor. remote actor should run the closure on its data and send back the result. May be it is not advisable, but for curiosity's sake that's i want to do now
But i observe that if a closure is created as an anonymous function, it captures the outer object also and tries to marshal it, which fails if the outer object is not serializable, as in this case.
class Client(server: ActorRef) extends Actor {
var every = 2
override def preStart() = {
println("client started. sending message....")
server ! new Message((x) => x % every == 0)
}
}
the above code generates exception while calling the remote actor. i could define a local variable in the method preStart()
val every_ = every
and use it in place of actor member variable. But i feel it is a workaround not a solution. and i will have to be very careful if the closure is any bit more complex.
Alternative is to define a class inheriting from Function1[A,B] and send its instances as closure.
class MyFunc(every : Int) extends Function1[Int,Boolean] with Serializable {
def apply(v1 :Int) : Boolean = {
v1 % every == 0
}
}
server ! new Message(new MyFunc(every))
But this separates the closure definition from the place it is used, and defeats the whole purpose of using a functional language. and also makes defining the closure logic more difficult.
Specific Query
Is there a way i can defer defining the body of the Function1.apply and assign the body of apply when i create the instance of MyFunc from a locally defined closure?
e.g.
server ! new Message(new MyFunc(every){ // not valid scala code
x % every == 0
})
where every is a local variable?
basically i want to combine the two approaches i.e. send an object of Function1 over to remote actor with the body of Function1 defined by an anon function defined in place where Function1 instance is created.
Thanks,
Sure, you could send behaviour to actor, but it considered to be a bad practice, and your questions is a good answer on question: "why".
As BGR pointed out there is special section in documentation on this question, but it has no example.
So, when you sending a closure as message you sending some extra "implicit" state with it. It could be not mutable as said in documentation, but even in this case it can create problems.
The problem with scala here is that it not strictly functional language - it is multiparadigm language. In other words you could have code in functional paradigm side by side with code in imperative style. There is no such problems in, for example haskell, which is purely functional.
In case of your "specific query" I'll suggest you to use set of predefined functions. This is full equivalent of variant with closures but with a bit chatty syntax. Since you do not generate code during runtime all functions you use are defined in limited set and (looks like) parameterized by value. This makes your code not so flexible like with closures, but in the end it will be equivalent cases.
So, as a leitmotif of all my post: if you going to send behaviour to actor it should be rock solid atomic (in meaning have no any dependencies)