I need to add a WebSocket-to-TCP proxy to my Play 2.3 application, but while the outgoing TCP connection using Akka I/O supports back-pressure, I don't see anything for the WebSocket. There's clearly no support in the actor-based API, but James Roper says:
Iteratees handle this by design, you can't feed a new element into an
iteratee until last future it returns has been redeemed, because you
don't have a reference to it until then.
However, I don't see what he's referring to. Iteratee.foreach, as used in the examples, seems too simple. The only futures I see in the iteratee API are for completing the result of the computation. Should I be completing a Future[Unit] for each message or what?
Iteratee.foldM lets to pass a state along to each step, much like the regular fold operation, and return a future. If you do not have such a state you can just pass Unit and it will behave as a foreach that will not accept the next step until the future completes.
Here is an example of a utility function that does exactly that:
def foreachM[E](f: E => Future[Unit])(implicit ec: ExecutionContext): Iteratee[E, Unit] =
Iteratee.foldM[E, Unit](Unit)((_, e) => f(e))
Iteratee is not the same as Iterator. An Iteratee does indeed inherently support back-pressure (in fact you'll find yourself with the opposite problem - by default they don't do any buffering (at least within the pipeline - of course async sockets still have receive buffers), so you sometimes have to add an explicit buffering step to an enumerator/iteratee pipeline to get reasonable performance). The examples look simple but that just means the framework is doing what a framework does and making things easy. If you're doing a significant amount of work, or making async calls, in your handlers, then you shouldn't use the simple Iteratee.foreach, but instead use an API that accepts a Future-based handler; if you're blocking within an Iteratee then you block the whole thing, waste your threads, and defeat the point of using them at all.
Related
I have an HTTP API endpoint that I need constantly check for new values. Luckily, it's supports long polling. So the idea is that I need to implement 'an infinite loop' where I do a request, wait for a response (at most 10mins), get some value from response and produce a side-effect by storing them somewhere, make another request.
Given that I have some function the call to which will start this 'infinite loop' I also need to return a Closable to satisfy Finagle API I'm integrating with so the process can be interrupted. If HTTP request fails I need to re-try immediately.
Now I need to figure out how to implement this with Futures in Finagle. I wonder whether I can user a recursion by applying transform to response Future?.. Or am I missing something and there is a more straightforward way to do it in Finagle?
Thanks!
I am not sure I can imagine how it (what you described) can be made any more straightforward than recursive:
def keepCalling: Future[Unit] = makeRequest
.flatMap { response =>
processResponse(response)
if(cancelled) Future.Unit else keepCalling
}
Note, that this is actually not recursive in the traditional sense, as we should normally expect (with some reservations) only one instance of keepCalling to be on stack at any given time, since the "recursive" invocation happens on a different thread.
I am reading this blog post that claims Futures are not "functional" since they are just wrappers of side-effectful computations. For instance, they contain RPC calls, HTTP requests, etc. Is it correct ?
The blog post gives the following example:
def twoUsersFeed(a: UserHandle, b: UserHandle)
(implicit ec: ExecutionContext): Future[Html] =
for {
feedA <- usersFeed(a)
feedB <- usersFeed(b)
} yield feedA ++ feedB
you lose the desired property: consistent results (the referential transparency). Also you lose the property of making as few requests as possible. It is difficult to use multi-valued requests and have composable code.
I am afraid I don't get it. Could you explain how we lose the consistent result in this case ?
The blog post fails to draw a proper distinction between Future itself and the way it's commonly used, IMO. You could write pure-functional code with Future, if you only ever wrote Futures that called pure, total functions; such code would be referentially transparent and "functional" in every remotely reasonable sense of the word.
What is true is that Futures give you limited control of side effects, if you use them with methods that have side effects. If you create a Futurewrapping webClient.get, then creating that Future will send a HTTP call. But that's not a fact about Future, that's a fact about webClient.get!
There is a grain of truth in this blog post. Separating expressing your computation from executing it, completely, via e.g. the Free monad, can result in more efficient and more testable code. E.g. you can create a "query language", where you express an operation like "fetch the profile photos of all the mutual friends of A and B" without actually running it. This makes it easier to test if your logic is correct (because it's very easy to make e.g. a test implementation that can "run" the same queries - or even just inspect the "query object" directly) and, as I think the blog post is trying to suggest, means you could e.g. combine multiple requests to fetch the same profile. (This isn't even purely a functional-programming concern - some OO books have the idea of a "command pattern" - though IME functional programming tools like for/yield syntax make it much easier to work in this way). Whereas if all you have is a fetchProfile method that, when run, immediately fires off a HTTP request, then if your code logic requests the same profile twice, there's no way to avoid fetching the same profile twice.
But that isn't really about Future per se, and IMO this blog post is more confusing than helpful.
I'm really having trouble understanding the fundamental purposes for Play's Iteratees and Enumerators.
I've read
Understand Play2 Iteratees for Normal Humans
Iteratees
Enumerators
I read this answer to find that instead of relying on a pulling model, like InputStream, it uses a pushing model.
Iteratees are an interesting beast -- on one hand, it "pushes" data to the handler, instead of relying to the handler to pull data, and, therefore, has better performance. On the other hand, it allows the handler to control when the flow should be stopped.
But then Play's documentation on Iteratees says
Or more generally enumerating a java.io.InputStream using Enumerator.fromStream. It is important to note that input won’t be read until the iteratee this Enumerator is applied on is ready to take more input.
Wait....so what is going on?
Does the data get pushed by the Enumerator or pulled by the Iteratee? (i.e. who decides when it is time to calculate more data)
Both. It is entirely non-blocking for both ends of the stream.
The Enumerator doesn't push any data until the Iteratee is ready to receive it, and it doesn't push any more data until the Iteratee signals it is ready for more. At the same time, the Enumerator can take as long as it wants to push data. Neither process will block on the other.
This method, on Iteratee, is critical to understanding how it works:
abstract def fold[B](folder: (Step[E, A]) ⇒ Future[B])(implicit ec: ExecutionContext): Future[B]
This is the sole abstract method in Iteratee, the only one that must be implemented. All the rest of the methods are defined in terms of fold. When an Enumerator is applied to an Iteratee, it invokes this method, providing the folder which is effectively a callback. Once the Iteratee is ready, it invokes folder providing the state it is currently in, Cont if it can receive more data, Done if it doesn't need any more, or Error if something has gone wrong. And because folder returns a Future, it can take as long as it needs to provide further input if the Iteratee is in the Cont state.
I'm building a library that, as part of its functionality, makes HTTP requests. To get it to work in the multiple environments it'll be deployed in I'd like it to be able to work with or without Futures.
One option is to have the library parametrise the type of its response so you can create an instance of the library with type Future, or an instance with type Id, depending on whether you are using an asynchronous HTTP implementation. (Id might be an Identity monad - enough to expose a consistent interface to users)
I've started with that approach but it has got complicated. What I'd really like to do instead is use the Future type everywhere, boxing synchronous responses in a Future where necessary. However, I understand that using Futures will always entail some kind of threadpool. This won't fly in e.g. AppEngine (a required environment).
Is there a way to create a Future from a value that will be executed on the current thread and thus not cause problems in environments where it isn't possible to spawn threads?
(p.s. as an additional requirement, I need to be able to cross build the library back to Scala v2.9.1 which might limit the features available in scala.concurrent)
From what I understand you wish to execute something and then wrap the result with Future. In that case, you can always use Promise
val p = Promise[Int]
p success 42
val f = p.future
Hence you now have a future wrapper containing the final value 42
Promise is very well explained here .
Take a look at Scalaz version of Future trait. That's based on top of Trampoline mechanism which will be executing by the current thread unless fork or apply won't be called + that completely removes all ExecutionContext imports =)
I've been studying about how I could develop a distributed architecture that implements the protocol request/response using the concept of concurrency through actors.
I concluded that the best way to do this is by creating a response system with synchronous handling of Futures/Promises, and soon after the response, leaving an open channel to receive notifications.
Thus an architecture that would work exactly like a inbox message.
It has some problems.
Thus I would have to maintain two endpoints (actors in the two layers)?
The Problem:
The view module requests that a particular element is processed. She sends this command to be processed via RemoteActor on the application server. This server should immediately return the promise that it will notify you when the element is processed. After this, the view module will be waiting the notification of completion of processing.
How do you see this problem?
I am using Scala, Akka and Google Guice.
I believe it is a generic problem that everyone can make use of their solutions. Excuse me if I'm hurting the terms of stackoverflow site.
Thanks in advance
I don't want to distract from any good answers you may get on Akka, because I unfortunately don't know much about Akka and it's distributed actors features, but I'd like to ask if you've considered any alternatives.
It seems that you basically need an asynchronous RPC library. There are two powerful libraries written in Scala that I know of that may satisfy your requirements - http://sna-projects.com/norbert/ and http://twitter.github.com/finagle/. Finagle gives some really powerful combinators for expressing asynchronous computation dependencies and registering listeners on futures. I'm currently maintaining Norbert, which we use at LinkedIn for some of our distributed systems, such as search.
//On "server" node
class SomeActor extends Actor {
def receive = {
case messageToRemoteActor => self reply_? TypeOfResponseMessage()
}
}
Actor.remote.register("myRemoteActor", actorOf[SomeActor])
//On "client" node
val remoteActor = Actor.remote.actorFor("myRemoteActor", "hostnameOrIpOfNodeContainingTheActorAbove", port)
val f: Future[TypeOfResponseMessage] = remoteActor !!! messageToRemoteActor
f.onComplete( _.value.get match {
case Left(exception) => handle(exception)
case Right(result) => handle(result)
})
Why not just use one of 0MQ's REQ-REP sockets?
https://github.com/zcox/akka-zeromq-java
That way you solve your immediate problem and at the same time, begin learn an architecture that will take you a long way and supports communications with clients written in many languages.
For an example of where this might lead look at http://blog.getintheloop.eu/2009/05/22/lift-amqp-with-rabbitmq-and-scala-tutorial-and-screencast/
Note that I am NOT suggesting that you use AMQP today since a RabbitMQ broker would be overkill for your immediate problem. Rather, I am suggesting that you invest your time in using an architecture (message queueing) that will pay you dividends in the long run.