Environment: Windows or Linux
ActiveMQ-artemis version: any
There are two properties in the attributes of an address. For instance as below. What is the difference between "Message count" and "Number of messages" in the attribute of an address ?
If I want to know how many messages come in the a topic even there is no consumer subscribe it, which one or what else property I should monitor?
This is a bug in the way Number of messages is calculated. It mistakenly counts messages from both "local" and "remote" queue bindings when it should only count messages from "local" queue bindings.
To be clear, "remote" queue bindings represent queues for this address which exist on other brokers in the cluster. The address needs to know about these remote queues so it can distribute messages around the cluster properly. They're really just an implementation detail that shouldn't influence the Number of messages metric.
I will fix this as part of ARTEMIS-1964.
Related
Let`s imagine we have server A with publisher and servers B and C with consumers.
Also we have got 5 different subjects; foo1, foo2,... foo5.
We always want to send a message only to one consumer and receive only one response.
So we utilize the requestOne function from the JS SDK at the publisher side and subscribe function with the {queue: "default"} option.
So both servers B and C has been subscribed one time for each subject.
But every time they subscribe they use queue with name "default" to prevent multiple consumers receive the same message as mentioned in docs.
So the question is:
Will this queue with name "default" be shared across all the subjects? Or each subject will have his own queue with name "default" and it is just shared between the subscribers of particular subject.
For example: producer generates 10 messages 2 for each subject.
Will we have 10 messages processed at the same time or only 2 messages since all the subscription share the same queue with name "default"?
You form a queue group based on the queue name that you specify and the subject. So a queue group of "foo" is different than a queue group on "bar".
That being said, with wildcards, you could have multiple subjects being part of the same queue group. That is, 2 members of the group "bar" listening on "foo.*" would split processing of messages sent on "foo.bar", "foo.baz", etc..
The same queue name in different subjects is separate.
You can test it with the examples in the link below.
https://nats.io/documentation/additional_documentation/nats-queueing/
start nats server
gnatsd
sub subject1
go run nats-qsub.go subject1 default
...
sub subject2
go run nats-qsub.go subject2 default
...
pub subject1&2
go run nats-pub.go subject1 "message"
...
go run nats-pub.go subject2 "message"
...
[org.jgroups.protocols.pbcast.NAKACK] (requester=, local_addr=) message ::port not found in retransmission table of :port:
(size=xxxx, missing=x, highest stability=xxxxx)]
NAKACK (or its newer cousin, NAKACK2) provide reliable transmission of messages to the cluster. To do this, every messages gets a sequence number (seqno) and receivers deliver the message to the application in seqno order.
Every cluster member has a table of all other members and their messages (conceptually a list). When member P sends messages P21, P22 and P23, a receiver R first looks up the message list for R, then adds P21-P23 to the list.
However, in your case, the list for R was not found. This means that R was not a cluster member (anymore).
For example, if we have cluster {P,Q,R,T}, and member R leaves or is excluded because it was suspected (e.g. we didn't receive a heartbeat for a period of time), then messages P21-23 will be dropped by any receiver.
This is because JGroups only allows cluster members to send and receive messages.
How can a member get excluded?
This is likely done by on of the failure detection protocols (e.g. FD_ALL or FD).
Another possibility is that your thread pools were clogged and failure detection heartbeat messages were dropped, leading to false suspicions.
Also, long GC pauses can cause this.
Fixes:
Increase the timeouts in FD_ALL or FD. The timeout should be longer than the longest GC cycle. Note that it will now take longer to detect hung members.
Size your thread pools, e.g. make sure that the max number of threads are big and the queue is disabled.
Note that false suspicions can happen, but MERGE3 should rememdy a split cluster later on.
I need to choose a new Queue broker for my new project.
This time I need a scalable queue that supports pub/sub, and keeping message ordering is a must.
I read Alexis comment: He writes:
"Indeed, we think RabbitMQ provides stronger ordering than Kafka"
I read the message ordering section in rabbitmq docs:
"Messages can be returned to the queue using AMQP methods that feature
a requeue
parameter (basic.recover, basic.reject and basic.nack), or due to a channel
closing while holding unacknowledged messages...With release 2.7.0 and later
it is still possible for individual consumers to observe messages out of
order if the queue has multiple subscribers. This is due to the actions of
other subscribers who may requeue messages. From the perspective of the queue
the messages are always held in the publication order."
If I need to handle messages by their order, I can only use rabbitMQ with an exclusive queue to each consumer?
Is RabbitMQ still considered a good solution for ordered message queuing?
Well, let's take a closer look at the scenario you are describing above. I think it's important to paste the documentation immediately prior to the snippet in your question to provide context:
Section 4.7 of the AMQP 0-9-1 core specification explains the
conditions under which ordering is guaranteed: messages published in
one channel, passing through one exchange and one queue and one
outgoing channel will be received in the same order that they were
sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Messages can be returned to the queue using AMQP methods that feature
a requeue parameter (basic.recover, basic.reject and basic.nack), or
due to a channel closing while holding unacknowledged messages. Any of
these scenarios caused messages to be requeued at the back of the
queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release
2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure. (emphasis added)
So, it is clear that RabbitMQ, from 2.7.0 onward, is making a rather drastic improvement over the original AMQP specification with regard to message ordering.
With multiple (parallel) consumers, order of processing cannot be guaranteed.
The third paragraph (pasted in the question) goes on to give a disclaimer, which I will paraphrase: "if you have multiple processors in the queue, there is no longer a guarantee that messages will be processed in order." All they are saying here is that RabbitMQ cannot defy the laws of mathematics.
Consider a line of customers at a bank. This particular bank prides itself on helping customers in the order they came into the bank. Customers line up in a queue, and are served by the next of 3 available tellers.
This morning, it so happened that all three tellers became available at the same time, and the next 3 customers approached. Suddenly, the first of the three tellers became violently ill, and could not finish serving the first customer in the line. By the time this happened, teller 2 had finished with customer 2 and teller 3 had already begun to serve customer 3.
Now, one of two things can happen. (1) The first customer in line can go back to the head of the line or (2) the first customer can pre-empt the third customer, causing that teller to stop working on the third customer and start working on the first. This type of pre-emption logic is not supported by RabbitMQ, nor any other message broker that I'm aware of. In either case, the first customer actually does not end up getting helped first - the second customer does, being lucky enough to get a good, fast teller off the bat. The only way to guarantee customers are helped in order is to have one teller helping customers one at a time, which will cause major customer service issues for the bank.
It is not possible to ensure that messages get handled in order in every possible case, given that you have multiple consumers. It doesn't matter if you have multiple queues, multiple exclusive consumers, different brokers, etc. - there is no way to guarantee a priori that messages are answered in order with multiple consumers. But RabbitMQ will make a best-effort.
Message ordering is preserved in Kafka, but only within partitions rather than globally. If your data need both global ordering and partitions, this does make things difficult. However, if you just need to make sure that all of the same events for the same user, etc... end up in the same partition so that they are properly ordered, you may do so. The producer is in charge of the partition that they write to, so if you are able to logically partition your data this may be preferable.
I think there are two things in this question which are not similar, consumption order and processing order.
Message Queues can -to a degree- give you a guarantee that messages will get consumed in order, they can't, however, give you any guarantees on the order of their processing.
The main difference here is that there are some aspects of message processing which cannot be determined at consumption time, for example:
As mentioned a consumer can fail while processing, here the message's consumption order was correct, however, the consumer failed to process it correctly, which will make it go back to the queue. At this point the consumption order is intact, but the processing order is not.
If by "processing" we mean that the message is now discarded and finished processing completely, then consider the case when your processing time is not linear, in other words processing one message takes longer than the other. For example, if message 3 takes longer to process than usual, then messages 4 and 5 might get consumed and finish processing before message 3 does.
So even if you managed to get the message back to the front of the queue (which by the way violates the consumption order) you still cannot guarantee they will also be processed in order.
If you want to process the messages in order:
Have only 1 consumer instance at all times, or a main consumer and several stand-by consumers.
Or don't use a messaging queue and do the processing in a synchronous blocking method, which might sound bad but in many cases and business requirements it is completely valid and sometimes even mission critical.
There are proper ways to guarantuee the order of messages within RabbitMQ subscriptions.
If you use multiple consumers, they will process the message using a shared ExecutorService. See also ConnectionFactory.setSharedExecutor(...). You could set a Executors.newSingleThreadExecutor().
If you use one Consumer with a single queue, you can bind this queue using multiple bindingKeys (they may have wildcards). The messages will be placed into the queue in the same order that they were received by the message broker.
For example you have a single publisher that publishes messages where the order is important:
try (Connection connection2 = factory.newConnection();
Channel channel2 = connection.createChannel()) {
// publish messages alternating to two different topics
for (int i = 0; i < messageCount; i++) {
final String routingKey = i % 2 == 0 ? routingEven : routingOdd;
channel2.basicPublish(exchange, routingKey, null, ("Hello" + i).getBytes(UTF_8));
}
}
You now might want to receive messages from both topics in a queue in the same order that they were published:
// declare a queue for the consumer
final String queueName = channel.queueDeclare().getQueue();
// we bind to queue with the two different routingKeys
final String routingEven = "even";
final String routingOdd = "odd";
channel.queueBind(queueName, exchange, routingEven);
channel.queueBind(queueName, exchange, routingOdd);
channel.basicConsume(queueName, true, new DefaultConsumer(channel) { ... });
The Consumer will now receive the messages in the order that they were published, regardless of the fact that you used different topics.
There are some good 5-Minute Tutorials in the RabbitMQ documentation that might be helpful:
https://www.rabbitmq.com/tutorials/tutorial-five-java.html
I understand that in HornetQ you can do live-backup pairs type of clustering. I also noticed from the documentation that you can do load balancing between two or more nodes in a cluster. Are those the only two possible topologies? How would you implement a clustered queue pattern?
Thanks!
Let me answer this using two terminologies: One the core queues from hornetq:
When you create a cluster connection, you are setting an address used to load balance hornetq addresses and core-queues (including its direct translation into jms queues and jms topics), for the addresses that are part of the cluster connection basic address (usually the address is jms)
When you load balance a core-queue, it will be load balanced among different nodes. That is each node will get one message at the time.
When you have more than one queue on the same address, all the queues on the cluster will receive the messages. In case one of these queues are in more than one node.. than the previous rule on each message being load balanced will also apply.
In JMS terms:
Topic subscriptions will receive all the messages sent to the topic. Case a topic subscription name / id is present in more than one node (say same clientID and subscriptionName on different nodes), they will be load balanced.
Queues will be load balanced through all the existent queues.
Notice that there is a setting on forward when no consumers. meaning that you may not get a message if you don't have a consumer. You can use that to configure that as well.
How would you implement a clustered queue pattern?
Tips for EAP 6.1/HornetQ 2.3 To implement a distributed queue/topic:
Read the official doc for your version: e.g. for 2.3 https://docs.jboss.org/hornetq/2.3.0.Final/docs/user-manual/html/clusters.html
Note that the old setting clusterd=true is deprecated, defining the cluster connection is enough, check that internal core bridges are created automatically / clustered=true is deprecated in 2.3+
take the full-ha configuration as a baseline or make sure you have jgroups properly set. This post goes deeply into the subject: https://developer.jboss.org/thread/253574
Without it, no errors are shown, the core bridge connection is
established... but messages are not being distributed, again no errors
or warnings at all...
make sure security domain and security realms, users, passwords, roles are properly set.
E.g. I confused the domain id ('other') with the realm id
('ApplicationRealm') and got auth errors, but the errors were
generic, so I wasted time checking users, passwords, roles... until I
eventually found out.
debug by enabling debug (logger.org.hornetq.level=DEBUG)
I have been working with qpid and now i am trying to move to broker less messaging system , but I am really confused about network traffic in a Pub Sub pattern. I read the following document :
http://www.250bpm.com/pubsub#toc4
and am really confused how subscription forwarding is actually done ?
I thought zero mq has to be agnostic for the underlying network topology but it seems it is not. How does every node knows what to forward and what to not (for e.g. : in eth network , where there can be millions subscriber and publisher , message tree does not sound a feasible to me . What about the hops that do not even know about the existence of zero mq , how would they forward packets to subscribers connected to them , for them it would be just a normal packet , so they would just forward multiple copies of data packets even if its the same packet ?
I am not networking expert so may be I am missing something obvious about message tree and how it is even created ?
Could you please give certain example cases how this distribution tree is created and exactly which nodes are xpub and xsub sockets created ?
Is device (term used in the link) something like a broker , in the whole article it seemed like device is just any general intermediary hop which does not know anything about zero mq sockets (just a random network hop) , if it is indeed a broker kind of thing , does that mean for pub sub , all nodes in messaging tree have to satisfy the definition of being a device and hence it is not a broke less design ?
Also in the tree diagram (from the link , which consist P,D,C) , I initially assumed C and C are two subscribers and P the only publisher (D just random hop), but now it seems that we have D as the zero mq . Does C subscribes to D and D subscribes to P ? or both the C just subscribe to P (To be more generic , does each node subscribe to its parent only in the ). Sorry for the novice question but it seems i am missing on something obvious here, it would be nice if some one can give more insights.
zeromq uses the network to establish a connection between nodes directly (e.g via tcp), but only ever between 1 sender and 1-n receivers. These are connected "directly" and can exchange messages using the underlying protocol.
Now when you subscribe to only certain events in a pub-sub scenario, zeromq used to filter out messages subscriber side causing unnecessary network traffic from the publisher to at least a number of subscribers.
In newer versions of zeromq (3.0 and 3.1) the subscriber process sends its subscription list to the publisher, which manages a list of subscribers and the topics they are interested in. Thus the publisher can discard messages that are not subscribed too by any subscriber and potentially send targeted messages at only interested subscribers.
When the publisher is itself a subscriber of events (e.g. a forwarding or routing device service) it might forward those subscriptions again by similarly subscribing to its connected publishers.
I am not sure whether zeromq still does client side filtering in newer versions even if it "forwards" its subscriptions though.
A more efficient mechanism for pub/sub to multiple subscribers is to use multicast whereby a single message traverses the network and is received by all subscribers (who can then filter what they wish).
ZeroMQ supports a standardised reliable multicast called Pragmatic General Multicast.
These references should give you an idea how it all works. Note that multicast generally only works on a single subLAN and may need router configuration or TCP bridges to span multiple subLANs.