Lets say we have 4 different kind of messages: Type1, Type2, Type3 and Type 4.
We have a service bus queue with sessions because of nessesaty of FIFO functionality.
We also have two message writers. First message writer takes care of creating messages of type type1 and type2, where writer #2 takes care of creating messages of type type3 and type4. This is the easy part.
Now i want to be able to create two readers, reader1 and reader2.
Reader1 must read messages of type type1 and type2.
Reader2 must read messages of type type3 and type4.
But i cant figure out how to achieve this. Because if reader1 reades message of type3, it needs to abanadon it, since reader1 cant only take care of messages of type type1 and type2. In worst case reader1 will be stuck with the message until it is thrown to deadletter queue.
Reason to create two different readers is because they are in two completly different domains, with completly different functionality.
How can i achieve the requered functionality?
You need to use Service Bus Topics and Subscriptions for this.
Writers publish messages to a Topic. Then both readers subscribe to that topic, and each subscription can filter based on message type.
See "Topics and subscriptions" in the docs.
How different are the different message types from each other?
You could possibly set up 4 different topics, one for each message type. The listeners would only need to listen to the topics that they are interested in.
This may be overkill if the message body for each message type is basically the same.
Related
Suppose I have a Kafka topic named account with several message types (each one with a different Avro schema), like account.created, account.deleted and so on.
I would like to understand if it is feasible (and it makes sense) to publish/receive different types on the same topics with Spring Cloud Stream. In particular, it would be very useful to have several #StreamListener, each one dedicated to a particular type. According to this blog post this is really useful when having the need to order messages because they are related to the same entity. What is an example of the configuration in this case?
I think you are talking about content-based routing which allows messages to be delivered to a specific StreamListener for cases where there are multiple.
You do so by using condition attribute. Please refer to this section for more details and let us know if it is still unclear or not what you're looking for.
I have a producer producer who wants to send some_persistent_message to dbConsumer and some_Notification_message to notificationConsumer
Producer1 is sending the message with keys
db_key: some_persistent_message
notify_key:some_Notifiction_message
On the consumer side, I have a consumer group App1_group with the two consumers dbConsumer and notificationConsumer
At this point in time, my dbConsumer is always getting messages of some_Notification_message because of my consumer ending up owning a specific partition which is always getting notify_key:some_Notifiction_message
Is it possible to send some_persistent_message to dbConsumer
and some_Notification_message to notification_consumer?
If you must mix multiple message types in a single topic (to maintain chronological order of the messages, for example), then I'd suggest having a single overloaded message type which can represent each of your disparate message types and which is able to answer the kind of message it holds. Then set up a separate consumer group for each message type and associated processing logic. Finally, modify the logic in each consumer to only process appropriate messages.
Having two different consumer groups assumes that you have roughly the same number of messages of each type, otherwise there could be a lot of wasted CPU. If one message type occurs orders of magnitudes more often that the other, you might be better off merging the two consumers into a single consumer with a branch to perform the appropriate processing.
Widely differing numbers of messages can also interfere with processing the two types of messages chronologically with two different consumer groups.
The best way is to simply use KafkaConsumer#assign() and specify the proper partitions.
Alternatively if you are okay with twice the I/O, simply use two consumer groups - both consumers will get all the messages, and in your consumer logic you can simply skip the messages of unwanted type.
In scenario where multiple single domain event types are produced to single topic and only subset of event types are consumed by consumer i need a good way to read the event type before taking action.
I see 2 options:
Put event type (example "ORDER_PUBLISHED") into message body (payload) itself which would be like broker agnostic approach and have other advantages. But would involve parsing of every message just to know the event type.
Utilize Kafka message headers which would allow to consume messages without extra payload parsing.
The context is event-sourcing. Small commands, small payloads. There are no huge bodies to parse. Golang. All messages are protobufs. gRPC.
What is typical workflow in such scenario.
I tried to google on this topic, but didn't found much on Headers use-cases and good practices.
Would be great to hear when and how to use Kafka message headers and when not to use.
Clearly the same topic should be used for different event types that apply to the same entity/aggregate (reference). Example: BookingCreated, BookingConfirmed, BookingCancelled, etc. should all go to the same topic in order to (excuse the pun) guarantee ordering of delivery (in this case the booking ID is the message key).
When the consumer gets one of these events, it needs to identify the event type, parse the payload, and route to the processing logic accordingly. The event type is the piece of message metadata that allows this identification.
Thus, I think a custom Kafka message header is the best place to indicate the type of event. I'm not alone:
Felipe Dutra: "Kafka allow you to put meta-data as header of your message. So use it to put information about the message, version, type, a correlationId. If you have chain of events, you can also add the correlationId of opentracing"
This GE ERP system has a header labeled "event-type" to show "The type of the event that is published" to a kafka topic (e.g., "ProcessOrderEvent").
This other solution mentions that "A header 'event' with the event type is included in each message" in their Kafka integration.
Headers are new in Kafka. Also, as far as I've seen, Kafka books focus on the 17 thousand Kafka configuration options and Kafka topology. Unfortunately, we don't easily find much on how an event-driven architecture can be mapped with the proper semantics onto elements of the Kafka message broker.
I have a validation service which takes in validation-requests and publishes them to a SQS queue. Now based on the type of validation request, I want to forward the message to that specific service.
So basically, I have one producer and multiple consumers, but essentially, one message is to be consumed by only one consumer.
What approach should I use? Should I have a different SQS queue for each service or I can do this using a single queue based on message type?
As I see it, you have three options;
The first option, like you say is to have a unique consumer for each message type. This is the approach we use and we have thousands of queues and many different messages types.
The second option would be to decorate the message being pushed onto SQS with something that would indicate it's desired consume, then have a generic consumer in your application that can forward the message on to the right consumer. Though this approach is generally seen as an anti pattern, I would personally agree.
Thirdly, you could take advantage of SNS filtering but that's only if you use SNS right now otherwise you'd have to invest in some time to setup it up and make it work.
Hope that helps!
I'm trying to replace rabbit mq with apache-kafka and while planning, I bumped in to several conceptual planning problem.
First we are using rabbit mq for per user queue policy meaning each user uses one queue. This suits our need because each user represent some job to be done with that particular user, and if that user causes a problem, the queue will never have a problem with other users because queues are seperated ( Problem meaning messages in the queue will be dispatch to the users using http request. If user refuses to receive a message (server down perhaps?) it will go back in retry queue, which will result in no loses of message (Unless queue goes down))
Now kafka is fault tolerant and failure safe because it write to a disk.
And its exactly why I am trying to implement kafka to our structure.
but there are problem to my plannings.
First, I was thinking to create as many topic as per user meaning each user would have each topic (What problem will this cause? My max estimate is that I will have around 1~5 million topics)
Second, If I decide to go for topics based on operation and partition by random hash of users id, if there was a problem with one user not consuming message currently, will the all user in the partition have to wait ? What would be the best way to structure this situation?
So as conclusion, 1~5 millions users. We do not want to have one user blocking large number of other users being processed. Having topic per user will solve this issue, it seems like there might be an issue with zookeeper if such large number gets in (Is this true? )
what would be the best solution for structuring? Considering scalability?
First, I was thinking to create as many topic as per user meaning each user would have each topic (What problem will this cause? My max estimate is that I will have around 1~5 million topics)
I would advise against modeling like this.
Google around for "kafka topic limits", and you will find the relevant considerations for this subject. I think you will find you won't want to make millions of topics.
Second, If I decide to go for topics based on operation and partition by random hash of users id
Yes, have a single topic for these messages and then route those messages based on the relevant field, like user_id or conversation_id. This field can be present as a field on the message and serves as the ProducerRecord key that is used to determine which partition in the topic this message is destined for. I would not include the operation in the topic name, but in the message itself.
if there was a problem with one user not consuming message currently, will the all user in the partition have to wait ? What would be the best way to structure this situation?
This depends on how the users are consuming messages. You could set up a timeout, after which the message is routed to some "failed" topic. Or send messages to users in a UDP-style, without acks. There are many ways to model this, and it's tough to offer advice without knowing how your consumers are forwarding messages to your clients.
Also, if you are using Kafka Streams, make note of the StreamPartitioner interface. This interface appears in KStream and KTable methods that materialize messages to a topic and may be useful in a chat applications where you have clients idling on a specific TCP connection.