How to achieve composing of Vert.x AsyncResult<T>? - vert.x

While performing a operation which might fail or succeed I have been using the Handler<AsyncResult<T>> so that result can be asynchronously passed when ready.
For example operation may be something like adding an entry into cache, like below
public void putEnvironmentVariable(String variableName, EnvVariable variable, Handler<AsyncResult<Long>> rsHandler) {
redisClient.hset(ENV_CACHE, variableName, Json.encode(variable), rsHandler);
}
In the above example I'm making use of Vert.x-redis add entry into a set.
Now if the same has to be done to add multiple entries into cache, I'm doing it like mentioned below
list.forEach(envVariable -> {
redisUtils.putEnvironmentVariable(envVariable.getName(), envVariable, result -> {
});
});
Questions:
1) Is this approach right ? or can it be done in a simpler/better way?
2) Here I've taken an example, but in general if there is a need to perform some operation like mentioned above and result of all operations combined to be passed on to the calling method how can that be achieved in vert.x?
I'm using
Vert.x 3.6.3

When you have to perform several async operations and combine the result of them into one result, you have two choices:
Using Vert.x Future
You wrap the redisClient.hset() async result in a Vert.x Future and then join all that async operations using Vert.x CompositeFuture:
Future<Long> fut = Future.future();
redisClient.hset(ENV_CACHE, variableName, Json.encode(variable), fut);
Then when you have a list of all futures you can use CompositeFuture:
CompositeFuture
.all(futuresList)
.map(cf -> /*aggregate all results in one single result*/)
Using Vert.x with RxJava 2
If you are using Vert.x with RxJava 2, you can merge all Single using combinators and then compose the final result with aggregators
The choice depends on you, anyway I suggest you give a look at Vert.x with RxJava 2. The RxJava 2 operators play well with Vert.x async operations and allow you to easily solve problems like these

Related

Scala Requests module: Is it possible to make concurrent requests?

I am using Scala concurrent module but I am not sure if it works asynchronously or synchronously ? I have a list of urls which are image urls and I want to make concurrent requests and get Array[Bytes]:
object Runner extends App {
def getBytesFromUrl(url:String) = {
requests.get(url).bytes
}
val urls = Seq(
"https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg",
"https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg",
"https://cdn.pixabay.com/photo/2014/02/27/16/10/flowers-276014__340.jpg"
)
// Would this make concurrent or sequential requests ?
val result = urls.map(url => getBytesFromUrl(url))
}
Would the above code make concurrent or sequential requests? And if it makes sequential requests then what's the right way to make concurrent requests?
I am using Scala concurrent module
You are not, at least not in the code shown.
Maybe you meant "request-scala library" in there?
but I am not sure if it works asynchronously or synchronously
For concurrency to exist you need asynchronously.
Now, again, if you were wondering about requests-scala then it is synchronously, that should be clear from the README and from the type signature.
Would the above code make concurrent or sequential requests?
Sequential, as the library explicitly states.
And if it makes sequential requests then what's the right way to make concurrent requests?
For one, you may consider using a different ecosystem since the author has been very clear that he doesn't think you need to be asynchronous at all to be efficient most of the time.
Another thing to consider is if you really need to make three HTTP calls concurrently.
Anyways, you can just use Future.traverse to do what you want.
object Runner {
def getBytesFromUrl(URL: String): Array[Byte] = {
requests.get(url).bytes
}
def getBytesFromUrls(urls: List[String]): Future[List[Array[Byte]]] =
Future.traverse(urls)(url => Future(getBytesFromUrl(url)))
}
You can then either compose that Future or Await it if you want to return to synchronous-land.
Personal disclaimer, if you actually have a lot of URLs and you need to do other asynchronous things with those bytes, like writing them to disk.
Then, I personally would recommend you to look to other ecosystems like typelevel, since those provide tools to write that kind of program in a more principled way.

What does it mean that Vert.x is "based on callbacks" (instead of futures)?

When talking about futures and callbacks, the documentation says that
The Vert.x core APIs are based on callbacks to notify of asynchronous
events. The seasoned developer will naturally think that this opens
the door to the so-called "callback hell" where multiple levels of
nested callbacks render the code difficult to comprehend as
illustrated by this fictional code:
foo.a(1, res1 -> {
if (res1.succeeded()) {
bar.b("abc", 1, res2 -> {
if (res.succeeded()) {
baz.c(res3 -> {
dosomething(res1, res2, res3, res4 -> {
// (...)
});
});
}
});
}
});
While the core APIs could have been designed to favor promises and futures, the choice of callbacks as the cardinal model is
actually interesting since it allows different programming
abstractions to be used.
What does it mean that "The Vert.x core APIs are based on callbacks"? What would be different if it was based on futures?
What does "callbacks as the cardinal model" mean?
It means that the Vert.x core asynchronous APIs all take callbacks as input, and use that callback to handle reacting to the asynchronous result of the API call. If it was based on Futures, instead of an API like this:
void deployVerticle(Verticle verticle, Handler<AsyncResult<String>>
completionHandler);
It would look like this:
Future<String> deployVerticle(Verticle verticle);
So instead of taking a callback that gets an AsyncResult<String> as input, it returns a Future<String>. However, the Vert.x team is moving to a hybrid Callback/Future-based model in Vert.x 4.0, so both of these kinds of APIs will be part of the core Vert.x API in the near-future.
I believe cardinal is used here to mean "fundamental". So it just means that Vert.x itself uses callbacks in its own implementation. It then uses code generation and/or other techniques to derive other kinds of API abstractions. Like APIs that return Futures, or return RxJava types, or can be used as Kotlin coroutines, etc.
In general, Futures are easier to work with than callbacks, especially when you have to compose several asynchronous operations together. For example, the code you included in your original question could probably be written like this, using Futures:
CompositeFuture.all(foo.a(1), bar.b("abc", 1), baz.c())
.compose(fut -> doSomething(fut.resultAt(0), fut.resultAt(1),
fut.resultAt(2)))
.compose(res4 -> ...)

Scala Fork Join Pool, How to implement for multiple tasks that Do not have a Concurrent Structure

Ok, so I am incredibly new to Scala (started yesterday). I have been reading the documents on concurrency and failed to find how to run larger tasks with Callables in a fork Join Pool. This is what I have as a sketch of what I would use in Java in Scala.
private Object fjp{
val fjp:ForkJoinPool=new ForkJoinPool(Runtime.getRuntime.availableProcessors()*2)
var w:Int=0
def invokeObjects(collection:Collection[Callable[Map[String:Int]]]){
var futures=fjp.invokeAll(collection)
w=0
while(fjp.isQuiescent()==false && fjp.getActiveThreadCount()==0){
w=w+1
}
println("Checked "+w+" times")
for(i<-0 to futures.size()){
var mp=futures.get(i).get()
//add keys to a common list
//submit count with frequency to sparse matrix
//avoid a ton of locking
}
}
}
How would I turn the code into a forkjoin pool that I can continually call. If possible, could I use a foreach without another List to get the results? Thank you for the help. It will point me in the right direction with Scala as well.
In general, I would not bother to follow exactly that path. Scala has a very clean paradigm in order to run parallel computations that just feels more idiomatic.
If you are new to Async computation in Scala, I would recommend you to start reading this.
In particular, you can define and/or reuse several kinds of ExecutorContext in order to get the kind of threadpool you need. Or you can use the default one if you are not blocking the threads (it has one per core only, by default)

Playframework non-blocking Action

Came across a problem I did not find an answer yet.
Running on playframework 2 with Scala.
Was required to write an Action method that performs multiple Future calls.
My question:
1) Is the attached code non-blocking and hence looking the way it should be ?
2) Is there a guarantee that both DAO results are caught at any given time ?
def index = Action.async {
val t2:Future[Tuple2[List[PlayerCol],List[CreatureCol]]] = for {
p <- PlayerDAO.findAll()
c <- CreatureDAO.findAlive()
}yield(p,c)
t2.map(t => Ok(views.html.index(t._1, t._2)))
}
Thanks for your feedback.
Is the attached code non-blocking and hence looking the way it should be ?
That depends on a few things. First, I'm going to assume that PlayerDAO.findAll() and CreatureDAO.findAlive() return Future[List[PlayerCol]] and Future[List[CreatureCol]] respectively. What matters most is what these functions are actually calling themselves. Are they making JDBC calls, or using an asynchronous DB driver?
If the answer is JDBC (or some other synchronous db driver), then you're still blocking, and there's no way to make it fully "non-blocking". Why? Because JDBC calls block their current thread, and wrapping them in a Future won't fix that. In this situation, the most you can do is have them block a different ExecutionContext than the one Play is using to handle requests. This is generally a good idea, because if you have several db requests running concurrently, they can block Play's internal thread pool used for handling HTTP requests, and suddenly your server will have to wait to handle other requests (even if they don't require database calls).
For more on different ExecutionContexts see the thread pools documentation and this answer.
If you're answer is an asynchronous database driver like reactive mongo (there's also scalike-jdbc, and maybe some others), then you're in good shape, and I probably made you read a little more than you had to. In that scenario your index controller function would be fully non-blocking.
Is there a guarantee that both DAO results are caught at any given time ?
I'm not quite sure what you mean by this. In your current code, you're actually making these calls in sequence. CreatureDAO.findAlive() isn't executed until PlayerDAO.findAll() has returned. Since they are not dependent on each other, it seems like this isn't intentional. To make them run in parallel, you should instantiate the Futures before mapping them in a for-comprehension:
def index = Action.async {
val players: Future[List[PlayerCol]] = PlayerDAO.findAll()
val creatures: Future[List[CreatureCol]] = CreatureDAO.findAlive()
val t2: Future[(List[PlayerCol], List[CreatureCol])] = for {
p <- players
c <- creatures
} yield (p, c)
t2.map(t => Ok(views.html.index(t._1, t._2)))
}
The only thing you can guarantee about having both results being completed is that yield isn't executed until the Futures have completed (or never, if they failed), and likewise the body of t2.map(...) isn't executed until t2 has been completed.
Further reading:
Are there any benefits in using non-async actions in Play Framework 2.2?
Understanding the Difference Between Non-Blocking Web Service Calls vs Non-Blocking JDBC

A little help on understanding Scalaz Future and Task

I'm trying to understand the idea and purpose behind scalaz concurrent package, primarily Future and Task classes, but when using them in some application, it's now far from simple sequential analog, whereas scala.concurrent.Future, works more then better. Can any one share with his experience on writing concurrent/asynchronous application with scalaz, basically how to use it's async method correctly? As i understand from the sources async doesn't use a separate thread like the call to standard future, or fork/apply methods from scalaz works, so why it is called async then? Does it mean that in order to get real concurrency with scalaz i always have to call fork(now(...)) or apply?
I'm not a scalaz expert, but I'll try to help you a little bit. Let me try answer your questions one by one:
1) Can any one share with his experience on writing concurrent/asynchronous application with scalaz, basically how to use it's async method correctly?
Let's first take a look at async signature:
def async[A](listen: (A => Unit) => Unit): Future[A]
This could be a bit cryptic at first, so as always it's good idea to look at tests to understands possible use cases. In https://github.com/scalaz/scalaz/blob/scalaz-seven/tests/src/test/scala/scalaz/concurrent/FutureTest.scala
you can find the following code:
"when constructed from Future.async" ! prop{(n: Int) =>
def callback(call: Int => Unit): Unit = call(n)
Future.async(callback).run must_==
}
As we know from signature Future.async just construct new Future using function of signature (A => Unit) => Unit. What this really means is that Future.async takes as parameter function which for given callback makes all required computations and pass the result to that callback.
What is important to note it that Future.async does not run any computations on itself, it only prepare structure to run them later.
2) As i understand from the sources async doesn't use a separate thread like the call to standard future, or fork/apply methods from scalaz works, so why it is called async then?
You are correct. Only fork and apply seems to be running anything using threads, which is easy to notice looking at the signatures which contains implicit pool: ExecutorService. I cannot speak for the authors here, but I guess async is related to the callback. It means that rather than blocking on Future to get it result at the end you will use asynchronous callback.
3) Does it mean that in order to get real concurrency with scalaz i always have to call fork(now(...)) or apply?
From what I can say, yes. Just notice that when you are creating Future using syntax Future(x) you are using apply method here, so this is kind of default behavior (which is fine).
If you want to better understand design of Scalaz Futures I can recommend you reading "Functional Programming in Scala". I believe this book is written by main Scalaz contributors and chapter 7 discusses designing API for purely functional parallelism library. It's not exactly the same as Scalaz Future, but you can see many similarities.
You can also read wonderful Timothy Perrett blog post about Scalaz Task and Future which covers many not so obvious details.
async is used to adapt an async, callback-based API as a Future. It's called async because it's expected that it will be used with something that runs asynchronously, perhaps calling the callback from another thread somewhere further down the line. This is "real" concurrency, provided the API you're calling really uses it asynchronously (e.g. I use Future.async with the async parts of the AWS SDK like AmazonSimpleDBAsyncClient).
If you want "real" concurrency from the scalaz Task API directly you need to use things like fork or gatherUnordered, as many of the APIs default towards being safe/deterministic and restartable, with concurrency only when explicitly requested.
When composing Tasks with map and flatMap you can get a performance win by not using fork, see:
http://blog.higher-order.com/blog/2015/06/18/easy-performance-wins-with-scalaz/