Add a type to messages in Kafka? - apache-kafka

Question

We are starting to use Kafka in a backend redevelopment, and have a quick question about how to structure the messages that we produce and consume.
Imagine we have a user microservice that handles CRUD operations on users. The two structures that have been put forward as a possibility are:
1) Four kafka topics, one for each operation. The message value would just contain the data needed to perform the operation, i.e.
topic: user_created
message value: {
firstName: 'john'
surname: 'smith'
}
topic: user_deleted
message value: c73035d0-6dea-46d2-91b8-d557d708eeb1 // A UUID
and so on
2) A single topic for user related events, with a property on the message describing the action to be taken, as well as the data needed, i.e.
// User created
topic: user_events
message value: {
type: 'user_created',
payload: {
firstName: 'john'
surname: 'smith'
}
}
// User deleted
topic: user_events
message value: {
type: 'user_deleted',
payload: c73035d0-6dea-46d2-91b8-d557d708eeb1 // A UUID
}
I am in favour of the first system described, although my inexperience with Kafka renders me unable to argue strongly why. We would greatly value any input from more experienced users.

Answer 1

Kafka messages don't have a type associated with them.
With a topic-per-event-type you would have to worry about ordering of events pertaining to the same entity read from the different topics. For this reason alone I would recommend putting all the events in the same topic. That way clients just have to consume a single topic to be able to fully track the state of each entity.

Answer 2

I worked on this kind of architecture recently.
We used an API Gateway, Which was the Webservice that communicates with our front end (ReactJS in our case). This API gateway used REST protocol. That microservice, developed with Spring Boot, acts as a producer and consumer in a separate thread.
1- Producing Message: Send message to Kafka broker on topic "producer_topic"
2- Consuming Message: Listen to the incoming messages from Kafka on topic "consumer_topic"
For consuming there was a pool of threads that handled the incoming messages and execution service which listen to Kafka stream and send assign the message handling to a thread from the pool.
Bottom to that there was a DAO Microservice that handle kafka messages and did the CRUD stuff.
Messages format looked really like your second approach.
//content of messages in the consumer_topic
{
event_type: 'delete'
message: {
first_name: 'John Doe'
user_id: 'c73035d0-6dea-46d2-91b8-d557d708eeb1'
}
}
This is why I should recommend you the second approach. There is less complexity as you handle all crud operations with only one topic. It's really fast due to partitions parallelism and you can add replication for being more fault tolerant.
The first approach sounds good in term of dematerialization and separation of concerns, but it's not really scalable. For instance let's say you want to add additional operation, it's one more topic to add. Also look at the replication. You will have more replicas to do and that's pretty bad I think.

Answer 3

Following the Tom advice, remember that even if you use a single topic you could chose to have more than one partitions for consumer scalability. Kafka provides you ordering at partition level so not at topic level. It means that you should use a "key" for identifying a resource you are creating, deleting, updating in order to have the message related to this "key" always in the same partition so with the right order otherwise even with a single topic you could lose the messages order having messages sent on different partitions.

Answer 4

Kafka 0.11 adds Message Headers which is an easy way of indicating different message types for the body of the message, even if they are all using the same serializer.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers