I have a question about ZeroMQ PUB/SUB. Does the publisher create a separate queue for each individual subscriber or is there one queue for all subscribers (and hence limited by the slowest subscriber)?
Each connected subscriber gets its own queue. When a subscriber is slow, its own queue will fill up and then overflow (high water mark is by default 1,000), and its messages will get dropped. This won't affect other subscribers.
Related
The picture below depicts my basic use-case using message groups and Spring-based JMS consumers.
Please note, here the concurrency refers to the config set as shown below:
defaultJmsListenerContainerFactory.setConcurrency("3-10");
Would the G1 and G2 listener receive messages concurrently for the respective groups?
Would in any case the message from one group wait for the dispatch of any message in another group?
Generally speaking, multiple consumers receiving grouped messages can receive them concurrently. However, there are caveats...
The core JMS client implementations actually consumes messages from a local data structure that is filled with messages asynchronously based on the consumerWindowSize which is 1 MiB (1024 * 1024 bytes) by default. If a consumer is receiving messages from a large, contiguous group and its "window" fills up then the broker will not be able to dispatch any more messages to it and will have to wait for the consumer to acknowledge messages in order to dispatch more. Once that block of grouped messages is dispatched then the broker will be able to dispatch messages from other groups to other consumers.
This is also explained in the documentation (although in a bit less detail).
We have application where multiple subscribers are writing to publisher Kafka topic This data is then propagated to specific subscriber topic then subscriber consumes this data from specific topic assigned to it.
We want to use SQS for same purpose but issue is we will again need an SQS for each subscriber.
Handling these multiple SQS will create an issue and if there is time when no data is published to subscriber the queue assigned to it will be idle.
Is there any way i can use single SQS from which all subscribers can consumed messages base don message identifier.
Challenges needs to be cover in this design are:
Each subscriber can get its message based on identifier
Latency must not be there in case one publisher publish very few messages and other one is publishing it in millions.
We can have one SQS for each publisher but single SQS for all subscribers of this publisher.
Can any one suggest any architecture using similar implementation.
Thanks
I think you can achieve it by setting up a single SQS queue. You would want to set up a Lambda trigger on that queue which will serve as a Service Manager (SM). SM will have a static JSON file that define the mapping between message identifier and its subscriber/worker. SM will receive an SQS message event, find the message attribute used for identifier, and then look up in JSON to find the corresponding subscriber. If subscriber is found, SM will invoke it.
Consider using SQS batch trigger.
Hi, I have the system as captured in the image. I'm planning to adopt a reliable messaging system, but I'm bit confused over which one to use. Below explained the detail flow of data and my requirement.
Step 1: data from System is given to Publisher.
Step 2: Publisher simply pushes the data to the Topic based Messaging
system.
Step 3: There will be more than one subscribers for each topic and
subscribers should get notified as soon there are some entries in
messaging system.
Step 4: Subscribers process the data and update the status back to messaging
system.
Step 5: Publisher should get notified for the processed messages and
acknowledge the System which gave the data.
So, my question is can I use RabbitMq or Kafka for "Topic Based Messaging System" ? my main requirement here is to update the status back from subscribers and also publisher should get notification for the status update. (I'm not much bothered about the throughput, performance, scalable AT THIS POINT of TIME). Also my another concern is data recovery/HA.
How about having a N+1 topic system, one for publishing messages which would be consumed by N subscribers, and N topics for acknowledgements, one per subscriber.
Your "System" could subscribe to all these N acknowledgment topics, and can verify if all the subscribers processed the original message which was published by the producer.
Each message in Kafka for eg. has a message key, and the same message key can be used to co-relate the original message with its subscriber specific acknowledgement.
Does this achieve what you want in your system ?
I have a distributed system that reads messages from RabbitMQ. In my situation now I need to process no more than N msgs/s.
For example: Imagine service A that sends text messages (SMS). This service can only handle 100 msgs/s. My system has multiple consumers that reads data from RabbitMQ. Each message needs to be processed first and than be send to service A. Processing time is always different.
So the question:
Is it possible to configure queue to dispatch no more than 100 msgs/s to multiple consumers?
You can use the prefetch_size parameter of the qos method to limit throughput to your consumers.
channel.basic_qos(100);
See also: http://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/
I'm new to messaging and a little unclear as to whether it is possible for MSMQ to deliver out-of-order messages for transactional queues. I suppose it must be because if a message is not processed correctly (and since we will be using multiple "competing consumers"), then other consumers could continue to process messages while the failed message is placed back on queue. Just can't seem to find a black-and-white answer anywhere on this.
Negative black-and-white answer are hard to find( they don't often exist).
You are confusing two terms here( I think). delivery is from the sender to the queue. consuming is from the queue to the consumer. Those two action can't be put in the same transaction. They are totally separate action ( this is one of the points of queuing )
More to the point: from "Microsoft Message Queuing Services (MSMQ) Tips"
That these messages will either be sent together, in the order they were sent, or not at all. In addition, consecutive transactions initiated from the same machine to the same queue will arrive in the order they were committed relative to each other.
This is the only case of order in msmq.
Sadly you won't find anything about ordered consuming because its not relevant. You can consume messages from msmq any way you want.
Update: If you must have ordered processing, than I don't see the reason to use many consumers. You will have to implement the order in your code.
Do your messages need to be processed in order because:
1) They are different steps of a workflow? If so, you should create different queues to handle the different steps. Process 1 reads Queue 1, does its thing, then writes to Queue 2, and so forth.
2) They have different priorities? If the priority levels are fairly coarse (and the order of messages within priorities doesn't matter), you should create high-priority and low-priority queues. Consumers read from the higher priority queues first.
3) A business rule specifies it. For example, "customer orders must be processed in the order they are received." Message queues are not appropriate for this kind of sequencing since they only convey the order in which messages are received. A process that periodically polls a database for an ordered list of tasks would be more suitable.