I have a back-end Scala application that needs to integrate with RabbitMQ. The back-end Scala app executes long running tasks asynchronously. Messages to execute the tasks are queued into RabbitMQ by a web client. The back-end application then consumes each of these messages, executing the corresponding long-running tasks.
Should Scala app directly consume the message from RabbitMQ and simply have the corresponding tasks be processed using Futures? Or is it better to use Akka Actors to receive these messages from RabbitMQ, and then execute the long-running tasks?
What are the pro's and con's of each approach?
Futures sound like a simpler approach for your use case combined with the RabbitMQ Java client.
My model for choosing actors v. futures is: prefer futures, switching to actors when I feel I have a good use case for them (see Good use case for Akka for some examples). For example, if you were trying to divide-and-conquer the batch workloads (as the linked answer states), actors may serve your purposes well.
Use the RabbitMQ Java examples below as a starting point, modifying to do work in futures so that the thread polling the work queue is not blocked. I included links to both work queue and RPC examples in case you need to return some response (RabbitMQ is good at this case as it has a concept of correlationId built in).
Java RabbitMQ examples:
Work Queues
Remote procedure call (RPC)
Related
I am evaluating the vert.x framework to see if I can reduce the Kafka based communications between my microservices developed using spring boot.
The question is:
Can I replace
Kafka with vert.x event bus and
spring boot microservices with vert.x based verticles
To answer quickly, I would say it depends on your needs.
Yes, the eventbus can be a good way to handle natively communication between microservices verticles using an asynchronous and non-blocking paradigm.
But in some cases you could need:
to handle some common enterprises patterns like replay mechanisms, persistence of messages, transactional reading
to be able to process some kind of messages in a chronological order
to handle communication between multiples kind of microservices that aren't all written with the same framework/toolkit or even programming language
to handle reliability, resilience and
failure recovery when all your consumers/microservices/verticles are died
to handle dynamic horizontal scalability and monitoring of your consumers/microservices/verticles
to be able to work with a single cluster deployed in multi-datacenters and multi-regions
In those cases I'd prefer to choose Apache Kafka over the native eventbus or an old fascioned JMS compliant system.
It's not forbidden to use both eventbus and kafka in the same microservices architecture according to your real needs. For example, you could have one kafka consumers group reading a kafka topic to handle scaling, monitoring, failure recovery and reply mechanism and then handle communication between your sub-verticles through the eventbus.
I'll clarify a little bit for the scalability and monitoring part and explain why I think it's more simple to handle that with Kafka over the native eventbus and cluster mode with vert.x : Kafka allow us to know in real time (through JMX metrics and the describe command):
the "lag" of a topic which corresponds to
the number of unread messages
the number of consumers of each group that are listening a topic
the number of partitions of a topic affected of each consumers
i/o metrics
So it's possible to use an ElasticStack or Prometheus+Grafana solution to monitor those metrics and use them to handle a dynamic scalability (when you know that there's a need to increase temporarily the number of consumers for example according to the lag metric and the number of partitions and the cpu/ram/swap metrics of your hosts).
To answer the second question vert.x or SpringBoot my answer will be not very objective but I'd vote for vert.x for its performances on the JVM and especially for its simplicity. I'm a little tired of the Spring factory and its big layers of abstraction that hides a lot of issues under a mountain of annotations triggering a mountain of AOP.
Moreover, In the Java world of microservices, there's other alternatives to SpringBoot like the different implementations of Microprofile (thorntail project for example).
The event-bus is not persistent. You should use it for fast verticle-to-verticle communications, and more generally to dispatch events where you know that you can loose them if you have some crash.
Kafka streams are persistent, and you should send events there because either you want other (possibly non-Vert.x) applications to consume them, and/or because you want to ensure that these events are not being lost in case of failure.
A reactive (read "scalable and fault-tolerant") Vert.x application typically uses a combination of both the event-bus and some replicable messaging systems like AMQP / Kafka / etc.
On the question:
Can I replace spring boot microservices with vert.x based verticles?
Yes, definitely, although the 2 have different programming models.
If you want a more progressive approach and use Spring for structuring your application while using Vert.x for resource efficiency over your I/O and event processing then you can mix them, see https://github.com/vert-x3/vertx-examples/tree/master/spring-examples for examples.
Take a look at the Quarkus framework: in the workshop section you'll find Vert.x and Apache Kafka combined!
We know Akka is one implementation of actor pattern. Without Akka, I usually implement a simple actor pattern using ThreadPool+BlockingQueue. So the message is offered into the queue, and the works(actors) take the message from the Queue, then do what they should do. Of course, this kind of implementation can be only in just ONE process.
So as to in one process,
What's the essential difference between these two(Akka vs.
ThreadPool+BlockingQueue)
Moreover, what's the difference between actor pattern and producer-consumer model?
Actor model is indeed quite similar to producer-consumer model (P-C).
However, if you use a blocking queue with P-C your application won't be completely non-blocking and asynchronous. The promise of actor model and Akka is that all messages are sent asynchronously and don't block the sender.
Another aspect of it is managing these queues gets quite cumbersome once you have many consumers and producers. With actors you simply send a message and don't have to think about these low level details. Under the hood Akka will keep a message queue aka mailbox per actor with a dispatcher assigning actors to the thread pool to process those messages.
It's much easier to use Akka to achieve highly performant and resilient application than coding it yourself. You get fault tolerance, resource management, location transparency, routing, distributed, async processing, hierarchical supervision out of the box. Not to mention other frameworks and libraries leveraging these features to give you even more (reactive streams, akka http, etc). There are lot's of patterns developed for you already there, so why bother with your own.
I am going to keep it short, we have a product that uses BPM and internal queue with lots of EJBs (pojo implementation). We decided to add REST to the product and we zeroed in to JAX-RS and Swagger for documentation.
Now, we created endpoint pointing to a async scenario in a such a way that when REST request arrives we start the BPMN flow asynchronously and then we wait for agreed timeout duration for flows to finish so that we can parallelly send a response to internal queue, which receive message when BPMN flow finished processing and then can construct REST response.
I am looking for some enterprise pattern or some utility framework to help me achieve this and not invent it myself. I know Camel has lots of such patterns but I am not so sure I am looking for something available on JDK 1.6 compatible framework to simulate this synchronous behavior.
I would have something like a RxJava or some observer notifier pattern probably no internal JMS queues to pass message between threads. A concurrent and thread-safe soilutuion is what I am looking for.
I would have something like a RxJava or some observer notifier pattern probably no internal JMS queues to pass message between threads. A concurrent and thread-safe solution is what I am looking for.
If you are to be using JAX-RS, then you should probably become familiar with the Asynchronous Server API. For a slow but synchronous operation, you would simply dispatch a task to your executor, and resume the suspended request when you have a result.
Another approach is to store the suspended request in a shared data structure, with a worker responsible for observing the completed flows, looking up the suspended request and dispatching the response.
The ResponseServlet from Michael Barker's ticketing demonstration shows this basic idea (Barker's code uses servlets rather than JAX-RS, and Disruptor rather than RxJava, so you'll need to translate).
Additional resources on async response processing
https://dennis-xlc.gitbooks.io/restful-java-with-jax-rs-2-0-2rd-edition/content/en/part1/chapter13/server_asynchronous_response_processing.html
http://www.nurkiewicz.com/2014/12/asynchronous-timeouts-with.html
I need to stream data between a couple hundred KB and many MBs between akka cluster nodes. Simplest approach would be to split it up as chunked messages, but that appears to be in advisable because it might interfere with housekeeping chatter of the cluster.
Alternatively, I could use messages to communicate one time urls and use http.
However, I'd prefer a persistent connection approach, so I was thinking using zeromq and chunked messages.
But rather than rolling my own approach, I'd like to use an existing way of accomplishing this but I have not found one.
One more requirement: most of the time the consumption of that stream is going straight out via Play, so an approach that created an iteratee that could be used to proxy the steam to http would be preferable.
Akka Streams 2.5.12 has StreamRefs for what I believe to be your use case.
Iteratees can't communicate across machine boundaries, so iteratees alone are probably not the tool you are looking for.
I would pursue one of the following approaches:
Using remote rpc akka Actors to send chunks of your data across the wire. Actors can be used to create iteratees and enumerators on either side (Enumerator.unicast and Iteratee.foreach) of the wire so that the fact that you are using Actors is just an implementation detail and not visible in your interface of these streams.
Use Akka Streams. This library has support for TCP connections, and while this is a different streaming library from iteratees, I have found that it is more robust in the stream operations it supports. It looks like Play is looking to move towards a tighter integration with Akka Streams as they are looking at replacing their netty HTTP backend with Akka Http Streams
I'm new to Scala Actors. What I plan to build is an application that has several cartridges that each do a specific http call and retrieve+persist some info periodically. Robustness is what matters the most. So far these are the ways I've thought of:
Build the app around a TimerTask,
extend cartridges from Actor and
call their .act s periodically (or
should I send them messages instead?
what's the difference?)
Extend from Actor and use Timeouts
to periodically run them.
Can someone shed some light on the differences?
Scala Actors will be merged with Akka so take a look at http://akka.io,
You can use Akka's "Scheduler" to schedule messages to be sent to actors at certain intervals, it's all in the docs:
http://akka.io/docs/akka/1.1.3/
Hope this helps,
Cheers,
√