I have written few c++ services which have the MQTT Client. Based on the message received on the MQTT topic the c++ service will take some actions like sending an MQTT message to another topic or saving the message to the database etc.
I have set up a few MQTT Brokers on Dockers and attached those MQTT Brokers to an HA Load balancer. All these MQTT Brokers also clustered.
So, if client 1 connected broker-1 ( through Load balancer ) can send message to client x connected broker -x. Due to the clustering of the MQTT Brokers.
So, How can I set the load balancer to my c++ services with HA or similar load balancers?
Update:
In the case of HTTP / REST APIs, the request will be transferred to only one web application at any point of time. But in case of MQTT, the message will be published, and If I run multiple c++ service of Same ABC then all the services will process that message. How I should make sure only one service will process the message. I want to establish High Availability for the C++ service
This is not possible under MQTT 3.x. The reason being that prior to MQTT 5, every message is sent to every subscriber to that topic making it very difficult to load balance correctly. Subscribers would need receive everything then discard decide for themselves to discard some messages, leaving them for other subscribers. It's one of the limitations of MQTT 3.x.
There are those who have worked around this by connecting their MQTT broker into an Apache Kafka cluster, routing all messages from MQTT to Kafka and then attaching their subscribers (like your c++ services) to Kafka instead of MQTT. Kafka supports the type of load balancing you are asking for.
This may be about to change with MQTT 5.0. There are still a lot of clients and brokers which don't support this. However if both your client and broker support MQTT version 5 then there is a new 1 concept of "Shared Subscriptions":
Shared Subscriptions – If the message rate on a subscription is high, shared subscriptions can be used to load balance the messages across a number of receiving clients
You haven't stated your client library. But your first steps should be:
investigate if both your broker and subscriber support MQTT 5
Check the API for your client to discover how to use subscriber groups
1 New to MQTT, Kafka already has it.
Related
I plan to use MQTT to provide 2 way data exchange between a PC and Raspberry Pi on the same network. I know how MQTT works in terms of subscribe/publish and topics. What I am a little unsure about is whether I need 1 or 2 brokers?
As I want to provide 2 way communication do I need a broker on the PC and Raspberry Pi or do I only need one and have a client on each?
MQTT is a hub/spoke protocol
Clients connect to a single broker, where they publish messages to topics and subscribe to those topics to receive messages.
There is no direct communication between 2 clients, just messages published to the broker and the broker then distributes those messages to any subscribed clients.
There is no need for more than one broker in most cases.
We have a network of Red Hat AMQ 7.2 brokers with Master/Slave configuration. The client application publish / subscribe to topics on the broker cluster.
How do we handle the situation wherein the network connectivity between the client application and the broker cluster goes down? Does Red Hat AMQ have a native solution like client local queue and a jms to jms bridge between local queue and remote broker so that network connectivity failure will not result in loss of messages.
It would be possible for you to craft a solution where your clients use a local broker and that local broker bridges messages to the remote broker. The local broker will, of course, never lose network connectivity with the local clients since everything is local. However, if the local broker loses connectivity with the remote broker it will act as a buffer and store messages until connectivity with the remote broker is restored. Once connectivity is restored then the local broker will forward the stored messages to the remote broker. This will allow the producers to keep working as if nothing has actually failed. However, you would need to configure all this manually.
That said, even if you don't implement such a solution there is absolutely no need for any message loss even when clients encounter a loss of network connectivity. If you send durable (i.e. persistent) messages then by default the client will wait for a response from the broker telling the client that the broker successfully received and persisted the message to disk. More complex interactions might require local JMS transactions and even more complex interactions may require XA transactions. In any event, there are ways to eliminate the possibility of message loss without implementing some kind of local broker solution.
I have some IoT devices which need to be updated sometimes, based on configuration done in web or mobile clients. So I need to give capability to be updated based on a configuration.
I have the following architecture when Clients communicates over HTTPS to an API Gateway. This Gateway is responsible to fetch data from several micro-services that interact with Kafka and some databases.
In this context, it is a good idea to create a Kafka consumer in IoT devices that will consumes messages from a Kafka Configuration Topic ?
Based on each new message received on this topic, the IoT device will be responsible to apply the change on the configuration.
Any advise ?
Usually, IoT devices have strong CPU/RAM and/or battery restrictions. The most widely used solution for messaging over IoT is MQTT and https://mosquitto.org/ the currently most widespread MQTT broker, so I would try to use https://mosquitto.org/ on the IoT devices and link it with Kafka through a "Confluent MQTT Proxy", you have more information at https://www.confluent.io/confluent-mqtt-proxy/
It is also not difficult to create your own "MQTT proxy" in python (or the language you prefer)
Kafka does not push. Consumers poll.
You can embed Kafka consumers in IoT devices, yes (assuming you are able to deploy such apps into them), however, MQTT is often documented as more used in those environments, and you could route Kafka events to an MQTT broker through various methods.
We are going to be developing a client which subscribes to an AMQP channel, but the client is going to be clustered (in Kubernetes) and we want only one of the clustered client to process the subscribed message.
For example, if we have a replica set of 3, we only want one to get the message, not all 3.
In JMS 2.0 this is possible using the shared consumers: https://www.oracle.com/technical-resources/articles/java/jms2messaging.html
1 message is sent to RabbitMQ Channel 1:
Consumer 1 (with 3 replicas) <----- RabbitMQ Channel 1
Consumer 2 (with 3 replicas) <----- RabbitMQ Channel 1
Only 2 messages would be processed
Is something similar possible with AMQP? The client will be developed either in C# or MuleSoft.
Cheers,
Steve
AMQP is designed for this. If you have three clients consuming from the same queue, RabbitMQ will round-robin delivery of messages to them. You may also be interested in the Single Active Consumer feature.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I have a use case where my Kafka cluster will have 1000 brokers and I am writing Kafka client.
In order to write client, i need to provide brokers list.
Question is, what are the recommended guidelines to provide brokers list in client?
Is there any proxy like service available in kafka which we can give to client?
- that proxy will know all the brokers in cluster and connect client to appropriate broker.
- like in redis world, we have twemproxy (nutcracker)
- confluent-rest-api can act as proxy?
Is it recommended to provide any specific number of brokers in client, for example provide list of 3 brokers even though cluster has 1000 nodes?
- what if provided brokers gets crashed?
- what if provided brokers restarts and there location/ip changes?
The list of broker URL you pass to the client are only to bootstrap the client. Thus, the client will automatically learn about all other available brokers automatically, and also connect to the correct brokers it need to "talk to".
Thus, if the client is already running, the those brokers go down, the client will not even notice. Only if all those brokers are down at the same time, and you startup the client, the client will "hang" as it cannot connect to the cluster and eventually time out.
It's recommended to provide at least 3 broker URLs to "survive" the outage of 2 brokers. But you can also provide more if you need a higher level of resilience.