I'm very new to messaging system, and I was trying to find my answer on http://www.rabbitmq.com/tutorials/, and I'm pretty sure it should be over there, but so far I got little bit confused with all bindings, queues, exchanges.
So I'm looking an answer for the question how to specify what type of "queue" (sorry if i have to use other word for this) is it. On producer side. To be more clear I'll give you an example:
So I want my consumer to subscribe to one "queue" and than once it receives it perform some operation based on what's inside this queue. Lets say if message contains a picture than do something, if it is a text, than do something else.
I was thinking my producer should add something like type:foo to the payload, and than consumer will look for this type. But I hope there is a better solution for this. Something like add a header to the queue.
Thank you.
If your consumer have to do different tasks for different types of message, then it would be better to create one distinct consumer per task.
That way, you can easily create one queue for each type of message and make each consumer consume messages from the right queue.
Your producer can send the message to the correct queue either directly or by using RabbitMQ routing.
Take a look at the "Routing" tutorial on the RabbitMQ website, it seems to match your use-case : http://www.rabbitmq.com/tutorials/tutorial-four-python.html
Related
i was wondering if there is a way to implement metadata or even multiple metadata to a service bus queue message to be used later on in an application to sort on but still maintaining FIFO in the queue.
So in short, what i want to do is:
Maintaining Fifo, that s First in First Out structure in the queue, but as the messages are coming and inserted to the queue from different Sources i want to be able to sort from which source the message came from with for example metadata.
I know this is possible with Topics where you can insert a property to the message, but also i am unsure if it is possible to implement multiple properties into the topic message.
Hope i made my self clear on what i am asking is possible.
I assume you use .NET API. If this case you can use Properties dictionary to write and read your custom metadata:
BrokeredMessage message = new BrokeredMessage(body);
message.Properties.Add("Source", mySource);
You are free to add multiple properties too. This is the same for both Queues and Topics/Subscriptions.
i was wondering if there is a way to implement metadata or even multiple metadata to a service bus queue message to be used later on in an application to sort on but still maintaining FIFO in the queue.
To maintain FIFO in the queue, you'd have to use Message Sessions. Without message sessions you would not be able to maintain FIFO in the queue itself. You would be able to set a custom property and use it in your application and sort out messages once they are received out of order, but you won't receive message in FIFO order as were asking in your original question.
If you drop the requirement of having an order preserved on the queue, the the answer #Mikhail has provided will be suitable for in-process sorting based on custom property(s). Just be aware that in-process sorting will be not a trivial task.
We're trying to build a platform using microservices that communicate async over kafka.
It would seem natural, the way i understood it, to have 1 topic per aggregate type in each microservice. So a microservice implementing user registration would publish user related events into the topic "users".
Other microservices would listen to events created from the "users" microservices and implement their own logic and fill their DBs accordingly. The problem is that other microservices might not be interested in all the events generated by the user microservice but rather a subset of these events, like UserCreated only (without UsernameChanged... for example).
Using RabbitMq is easy since event handlers are invoked based on message type.
Did you ever implement message based routing/filtering over kafka?
Should we consume all the messages, deserialize them and ignore unneeded ones by the consumer? (sounds like an overhead)
Should we forward these topics to storm and redirect these messages to consumer targeted topics? (sounds like an overkill and un-scalable)
Using partitions doesn't seem logical as a routing mechanism
Use a different topic for each of the standard object actions: Create, Read, Update, and Delete, with a naming convention like "UserCreated", "UserRead", etc. If you think about it, you will likely have a different schema for the objects in each. Created will require a valid object; Read will require some kind of filter; Update you might want to handle incremental updates (add 10 to a specific field, etc).
If the different actions have different schemas it makes deserialization difficult. If you're in a loosey-goosey language like JavaScript, ok -- no big deal. But a strictly typed language like Scala and having different schemas in this same topic is problematic.
It'll also solve you're problem -- you can listen for exactly the types of actions you want, no more, not less.
I have a Java web-service that I am going to reimplement from scratch in Scala. I have an actor-based design for the new code, with around 10-20 actors. One of the use-cases has a flow like this:
Actor A gets a message a, creates tens of b messages to be handled by Actor B (possibly multiple instances, for load balancing), producing multiple c messages for Actor C, and so on.
In the scenario above, one message a could lead to a few thousand messages being sent back and forth, but I don't expect more than a handful of a messages a day (yes, it is not a busy service at the moment).
I have the following requirements:
Messages should not be lost or repeated. I mean if the system is restarted in the middle of processing b messages, the unprocessed ones should be picked up after restart. On the other hand, the processed ones should not be taken again (these messages will in the end start some big computation, and repeating them is costly).
It should be easily extensible. I mean in the future, I may want to add some other components to the system that can read all the communication (or parts of it) and for example make a log of what has happened, or count how many b messages were processed, or do something new with the b messages (next to what is already happening), etc. Note that these "components" could be independent applications written in other languages.
I am new to message bus technologies, but from what I have read, these requirements sound to me like what "message buses" offer, like RabbitMQ, Kafka, Kestrel, but I also see that akka also offers some means for persistence.
My problem is, given the huge range of possibilities, I am lost which technology to use. I read that something like Kafka is probably an overkill for my application. But I am also not sure if akka persistence answers my two requirements (especially the extensibility).
My question is: Should I go for an enterprise message bus? Something like Kafka? Or something like akka persistence will do?
Or would it be just faster and more appropriate if I implement something myself (with support for, say, AMQP to allow extensibility)?
Of course, specific technology suggestions are also welcome if you know of something that fits this purpose.
A Message Bus (typically called Message Brokers) like RabbitMQ can handle "out of the box" all of the messaging mechanisms you describe in your question. Specifically:
RabbitMQ has the ability "Out of the Box":
To deliver messages without repeating the message.
To extend the system and add logging and have statistics like you describe.
The scenario is publisher/subscriber, and I am looking for a solution which can give the feasibility of sending one message generated by ONE producer across MULTIPLE consumers in real-time. the light weight this scenario can be handled by one solution, the better!
In case of AMQP servers I've only checked out Rabbitmq and using rabbitmq server for pub/sub pattern each consumer should declare an anonymous, private queue and bind it to an fanout exchange, so in case of thousand users consuming one message in real-time there will be thousands or so anonymous queue handling by rabbitmq.
But I really do not like the approach by the rabbitmq, It would be ideal if rabbitmq could handle this pub/sub scenario with one queue, one message , many consumers listening on one queue!
what I want to ask is which AMQP server or other type of solutions (anyone similar including XMPP servers or Apache Kafka or ...) handles the pub/sub pattern/scenario better and much more efficient than RabbitMQ with consuming (of course) less server resource?
preferences in order of interest:
in case of AMQP enabled server handling the pub/sub scenario with only ONE or LESS number of queues (as explained)
handling thousands of consumers in a light-weight manner, consuming less server resource comparing to other solutions in pub/sub pattern
clustering, tolerating failing of nodes
Many Language Bindings ( Python and Java at least)
easy to use and administer
I know my question may be VERY general but I like to hear the ideas and suggestions for the pub/sub case.
thanks.
In general, for RabbitMQ, if you put the user in the routing key, you should be able to use a single exchange and then a small number of queues (even a single one if you wanted, but you could divide them up by server or similar if that makes sense given your setup).
If you don't need guaranteed order (as one would for, say, guaranteeing that FK constraints wouldn't get hit for a sequence of changes to various SQL database tables), then there's no reason you can't have a bunch of consumers drawing from a single queue.
If you want a broadcast-message type of scenario, then that could perhaps be handled a bit differently. Instead of the single user in the routing key, which you could use for non-broadcast-type messages, have a special user type, say, __broadcast__, that no user could actually have, and have the users to broadcast to stored in the payload of the message along with the message itself.
Your message processing code could then take care of depositing that message in the database (or whatever the end destination is) across all of those users.
Edit in response to comment from OP:
So the routing key might look something like this message.[user] where [user] could be the actual user if it were a point-to-point message, and a special __broadcast__ user (or similar user name that an actual user would not be allowed to register) which would indicate a broadcast style message.
You could then place the users to which the message should be delivered in the payload of the message, and then that message content (which would also be in the payload) could be delivered to each user. The mechanism for doing that would depend on what your end destination is. i.e. do the messages end up getting stored in Postgres, or Mongo DB or similar?
I am using activemq to pass requests between different processes. In some cases, I have multiple, duplicate message (which are requests) in the queue. I would like to have only one. Is there a way to send a message in a way that it will replace an older message with similar attributes? If there isn't, is there a way to inspect the queue and check for a message with specific attributes (in this case I will not send the new message if an older one exists).
Clarrification (based on Dave's answer): I am actually trying to make sure that there aren't any duplicate messages on the queue to reduce the amount of processing that is happening whenever the consumer gets the message. Hence I would like either to replace a message or not even put it on the queue.
Thanks.
This sounds like an ideal use case for the Idempotent Consumer which removes duplicates from a queue or topic.
The following example shows how to do this with Apache Camel which is the easiest way to implement any of the Enterprise Integration Patterns, particularly if you are using ActiveMQ which comes with Camel integrated out of the box
from("activemq:queueA").
idempotentConsumer(memoryMessageIdRepository(200)).
header("myHeader").
to("activemq:queueB");
The only trick to this is making sure there's an easy way to calculate a unique ID expression on each message - such as pulling out an XPath from the document or using as in the above example some unique message header
You could browse the queue and use selectors to identify the message. However, unless you have a small amount of messages this won't scale very well. Instead, you message should just be a pointer to a database-record (or set of records). That way you can update the record and whoever gets the message will then access the latest version of the record.