We are going to be developing a client which subscribes to an AMQP channel, but the client is going to be clustered (in Kubernetes) and we want only one of the clustered client to process the subscribed message.
For example, if we have a replica set of 3, we only want one to get the message, not all 3.
In JMS 2.0 this is possible using the shared consumers: https://www.oracle.com/technical-resources/articles/java/jms2messaging.html
1 message is sent to RabbitMQ Channel 1:
Consumer 1 (with 3 replicas) <----- RabbitMQ Channel 1
Consumer 2 (with 3 replicas) <----- RabbitMQ Channel 1
Only 2 messages would be processed
Is something similar possible with AMQP? The client will be developed either in C# or MuleSoft.
Cheers,
Steve
AMQP is designed for this. If you have three clients consuming from the same queue, RabbitMQ will round-robin delivery of messages to them. You may also be interested in the Single Active Consumer feature.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
Related
Description of proposed cluster setup
2 Data centres and each having 5 node Kafka cluster
Clusters are having the same topics and same producer/consumer instances working with it
There is no data replication across the clusters. So data in Cluster 1 and 2 is distinct
There is no message affinity required. [It will not make any difference functionally if either the Producer 1 were to start posting message to Cluster 2 and vice versa]
What we want to achieve is, Lets say Producer 1 posts a message asynchronously to Cluster 1, but receives a negative acknowledgment ( after all the retry timeout has occurred). This is easily detected in the producer callback method
On receiving this failure, We use another KafkaTemplate (having details of Cluster 2) to be used by producer. Now producer tries posting the same message on to cluster 2 [ It applies other way round as well, if producer 2 unable to post locally , it will send message to cluster 1]
The advantage that we get here is
message is not lost and posted automatically to the other cluster
Since this activity occurs for each message, so once the Cluster 1 is back up, automatically Producer 1 is able to send messages to cluster 1
One down side we see is, We are handling the failover logic ourselves by producing to secondary cluster in exception handling block of either Metadata fetch timeout or on Negative acknowledgment
I could not find any where on the net showing a similar setup. Is there is something fundamentally wrong with this approach
Sure; just configure 2 sets of infrastructure beans - producer and consumer factories, container factories, templates.
You can't use Boot's auto configuration for that, but you can define the beans yourself.
I plan to use MQTT to provide 2 way data exchange between a PC and Raspberry Pi on the same network. I know how MQTT works in terms of subscribe/publish and topics. What I am a little unsure about is whether I need 1 or 2 brokers?
As I want to provide 2 way communication do I need a broker on the PC and Raspberry Pi or do I only need one and have a client on each?
MQTT is a hub/spoke protocol
Clients connect to a single broker, where they publish messages to topics and subscribe to those topics to receive messages.
There is no direct communication between 2 clients, just messages published to the broker and the broker then distributes those messages to any subscribed clients.
There is no need for more than one broker in most cases.
I have written few c++ services which have the MQTT Client. Based on the message received on the MQTT topic the c++ service will take some actions like sending an MQTT message to another topic or saving the message to the database etc.
I have set up a few MQTT Brokers on Dockers and attached those MQTT Brokers to an HA Load balancer. All these MQTT Brokers also clustered.
So, if client 1 connected broker-1 ( through Load balancer ) can send message to client x connected broker -x. Due to the clustering of the MQTT Brokers.
So, How can I set the load balancer to my c++ services with HA or similar load balancers?
Update:
In the case of HTTP / REST APIs, the request will be transferred to only one web application at any point of time. But in case of MQTT, the message will be published, and If I run multiple c++ service of Same ABC then all the services will process that message. How I should make sure only one service will process the message. I want to establish High Availability for the C++ service
This is not possible under MQTT 3.x. The reason being that prior to MQTT 5, every message is sent to every subscriber to that topic making it very difficult to load balance correctly. Subscribers would need receive everything then discard decide for themselves to discard some messages, leaving them for other subscribers. It's one of the limitations of MQTT 3.x.
There are those who have worked around this by connecting their MQTT broker into an Apache Kafka cluster, routing all messages from MQTT to Kafka and then attaching their subscribers (like your c++ services) to Kafka instead of MQTT. Kafka supports the type of load balancing you are asking for.
This may be about to change with MQTT 5.0. There are still a lot of clients and brokers which don't support this. However if both your client and broker support MQTT version 5 then there is a new 1 concept of "Shared Subscriptions":
Shared Subscriptions – If the message rate on a subscription is high, shared subscriptions can be used to load balance the messages across a number of receiving clients
You haven't stated your client library. But your first steps should be:
investigate if both your broker and subscriber support MQTT 5
Check the API for your client to discover how to use subscriber groups
1 New to MQTT, Kafka already has it.
I am studying the internals of Apache Kafka; how it works.
The Kafka brokers deal with the requests from the multiple producers and consumers.
I want to know how Kafka schedules those requests. (i.e. FCFS)
- Is it First-Come-First-Served (FCFS) or Processor Sharing (PS) ?
- Does the producers have the higher priorities than consumers?
The Kafka official documentation does not have explanation on it.
Can anyone give me an idea on this?
Thanks,
There is a TCP connection per client at the broker (the client can either be a consumer or producer or any number of producers &/or consumers)
The way CPU resources are shared between different connections is not a property controlled by Kafka. This depends on the OS on which your broker is running. Specifically, the scheduler implementation of your OS (which decide how processes are schedules on cores), will decide this.
If the scheduler is FCFS, this will very well be FCFS. More generally, the scheduler implementation in most OS is some version of Multi Level Feedback Queue.
Thus, this has got nothing to do with Kafka.
I am trying to count messages in HornetQ clustered queue on JBoss EAP 6.4 (domain mode)
Obtaining number of messages in particular HornetQ instance is not an problem (here is the way I do it), but what I am actually want, is to get cumulative/total number of messages of given queue in whole cluster.
Right now when I send to given queue 24604 messages, they are being nicely distributed to 3 nodes:
Node A: 8201 messages
Node B: 8202 messages
Node C: 8201 messages
Is there a way to count all messages of given queue in a cluster?
I have finally found a solution to obtain total number of messages in cluster by invoking broadcast ejb call on all cluster members, where each cluster member gets number of messages from InVm jms sender.
I have described it here:
http://jeefix.com/how-to-invoke-broadcast-ejb-at-all-jboss-eap6-ejb-cluster-members/
http://jeefix.com/managing-hornetq-queues-via-jms-api/