We are working on a appliation in which we are using kafka.
The components of the Application are as follows,
We have a microservice which gets the request's and pushes the messages to a kafka topic. (Lets say ServiceA)
Another microservice which consumes the messages from topics and push the data to a datastore. (Lets say ServiceB)
I am clear with ServiceA part of the application but have some design confusions in the ServiceB part.
As ServiceB we are planning for REST API,
Is it good to bundle Consumer and controllers in a single application ?
For consumer i am planning to go with ConsumerGroup with multiple Consumer's to acheive more throughput. Is there any better and efficent approach ?
Should i take out the Consumer part of ServiceB and make it as a separate service which is independent ?
If we are bundling it inside the ServiceB should i configure Consumer as a Listener ? (We are going with spring boot for microservice)
Thanks in Advance !
Is it good to bundle Consumer and controllers in a single application
?
Its good to bundle together by context, having a listener, wich forwards to another service to controll makes no sense in my opionion. But consider splitting up controller by different context if necessary. Like Martin Fowler says: start with a monolith first and than split up (https://martinfowler.com/bliki/MonolithFirst.html)
For consumer i am planning to go with ConsumerGroup with multiple
Consumer's to acheive more throughput. Is there any better and
efficent approach ?
A consumer group makes sense if you think about scale your service B out. If you want to have this possibility in the future, start with one instance of ServiceB inside the consumer group. If you use something like kubernetes, its simple do later on deploy more instances of your service if required. But do not invest to much on in an eventual future. Start simple and do some monitoring, and if you figure out some bottle necks, than act. One more thing to keep in mind, is that kafka by default keeps message for a long time (i guess 7 days by default) so if you thing in a classical message broker style you could get a lot of duplicates of your messages. Think about a update message, if somethings change, which is raised when your ServiceA starts. Maybe reducing the retention.ms would be an option, but take care not to loose messages.
Should i take out the Consumer part of ServiceB and make it as a
separate service which is independent ?
No i think not.
If we are bundling it inside the ServiceB should i configure Consumer
as a Listener ? (We are going with spring boot for microservice)
Yes :-)
Related
This is more of a design/architecture question.
We have a microservice A (MSA) with multiple instances (say 2) running of it behind LB.
The purpose of this microservice is to get the messages from Kafka topic and send to end users/clients. Both instances use same consumer group id for a particular client/user so as messages are not duplicated. And we have 2 (or =#instances) partitions of Kafka topic
End users/clients connect to LB to fetch the message from MSA. Long polling is used here.
Request from client can land to any instance. If it lands to MSA1, it will pull the data from kafka partion1 and if it lands to MSA2, it will pull the data from partition2.
Now, a producer is producing the messages, we dont have high messages count. So, lets say producer produce msg1 and it goes to partition1. End user/client will not get this message unless it's request lands to MSA1, which might not happen always as there are other requests coming to LB.
We want to solve this issue. We want that client gets the message near realtime.
One of the solution can be having a distributed persistent queue (e.g. ActiveMQ) where both MSA1 and MSA2 keep on putting the messages after reading from Kafka and client just fetch the message from queue. But this will cause separate queue for every end-user/client/groupid.
Is this a good solution, can we go ahead with this? Anything that we should change here. We are deploying our system on AWS, so if any AWS managed service can help here e.g. SNS+SQS combination?
Some statistics:
~1000 users, one group id per user
2-4 instances of microservice
long polling every few seconds (~20s)
average message size ~10KB
Broadly you have three possible approaches:
You can dispense with using Kafka's consumer group functionality and allow each instance to consume from all partitions.
You can make the instances of each service aware of each other. For example, an instance which gets a request which can be fulfilled by another instance will forward the request there. This is most effective if the messages can be partitioned by client on the producer end (so that a request from a given client only needs to be routed to an instance). Even then, the consumer group functionality introduces some extra difficulty (rebalances mean that the consumer currently responsible for a given partition might not have seen all the messages in the partition). You may want to implement your own variant of the consumer group coordination protocol, only on rebalance, the instance starts from some suitably early point regardless of where the previous consumer got to.
If you can't reliably partition by client in the producer (e.g. the client is requesting a stream of all messages matching arbitrary criteria) then Kafka is really not going to be a fit and you probably want a database (with all the expense and complexity that implies).
There is a Consumer A consuming from Topic A.
Consumer A is going to be deprecated (as it belongs to a different team and they dont handle the use case anymore). It is not possible to just change the ownership of the component to our team as we already have a Consumer B. Consumer B currently listens to Topic B. We want to leverage the Consumer B to read from Topic A as well.
So, we want to migrate the business logic in Consumer A to Consumer B.
Before going ahead with the full fledged migration, we want to validate the logic for a small part of the traffic.
Is there a standard way of migrating business logic from Consumer A to Consumer B ?
For the small part of traffic, is it possible to just process the traffic in Consumer B ?
In an ideal world, your business logic code should be separated from the consumer code. Because your business logic most likely going to outlive your broker technology's life span. If you are in that ideal world, you can route a subset of the messages to the new business logic library. Say for example if you decided to route 10% of the traffic, do message offset % 10 = 0 then route it to the new logic. If you are not in an ideal world, you can try manual partition assignment and take only those messages coming into a specific partition. Try this.
From an event sourcing/CQRS perspective: Say I have a consumer group of 2 instances, that's subscribed to a topic. On startup/subscription, each instance processes its share of the event stream, and builds a local view of the data.
When an external request comes in with a command to update the data, how would that request be routed to the correct instance in the group? If the data were partitioned by entity ID so that odd-numbered IDs went to consumer 1 and evens to consumer 2, how would that be communicated to the consumers? Or, for that matter, whatever reverse-proxy or service-mesh is responsible for sending that incoming request to the correct instance?
And what happens when the consumer group is re-balanced due to the addition or subtraction of consumers? Is that somehow automatically communicated the routing mechanism?
Is there a gap in service while the consumers all rebuild their local model from their new set of events from the given topics?
This seems to apply to both the command and query side of things, if they're both divided between multiple instances with partitioned data...
Am I even thinking about this correctly?
Thank you
Kafka partitioning is great for sharding streams of commands and events by the entity they affect, but not for using this sharding in other means (e.g. for routing requests).
The broad technique for sharding the entity state I'd recommend is to not rely on Kafka partitioning for that (only using the topic partitions to ensure ordering of commands/events for an entity, i.e. by having all commands/events for a given entity be in one partition), but instead using something external to coordinate those shards (candidates would include leases in zookeeper/etcd/consul or cluster sharding from akka (JVM) or akka.net or cloudstate/akka serverless (more polyglot)). From there, there are two broad approaches you can take:
(most really applicable if the number of entity shards for state and processing happens to equal the number of Kafka partitions) move part of the consumer group protocol into your application and have the instance which owns a particular shard consume a particular partition
have the instances ingesting from Kafka resolve the shard for an entity and which instance owns that shard and then route a request to that instance. The same pattern would also allow things like HTTP requests for an entity to be handled by any instance. By doing this you're making a service implemented in a stateful manner present to things like a service mesh/container scheduler/load balancer as a more stateless service would present.
I have the following situation:
I have 5 instances of the same service, all in the same kafka consumer group. One of them has a websocket connection to the client (the graphql subscription). I use graphql-java and Spring Boot.
When that connection is opened, I produce events from any of the 5 instances (with a message key defined so they go to the same partition and ordered) and I need for all those events to be consumed by the same instance that opened that connection. Not by the other 4.
Even if the partition assignment plays in my favor, a reassignment can by done at any time, leaving me without luck
My implementation is using reactor-kafka but I think it's just an implementation detail.
The options I see are:
Start to listen on that topic with a new group id each time, so that service always receives the messages from that topic (but the 5 in the other group id too)
Create a new topic for each websocket connection, so only the producer knows that topic (but the topic id should be sent in the kafka events so that the producers of those events know where to publish them)
If I receive the message and I'm not the one with the connection, don't ACK it. But this would make things slow and seems hacky
Start using something different altogether like Redis PubSub to receive all messages in all consumers and check for the connection.
I see there's an implementation for node but I don't see how it is solving the problem.
A similar question explains how to program a subscription but doesn't talk about this distributed thing.
Is the cleanest approach any of the one I suggested? Is there an approach with Kafka that I'm not seeing? Or am I misunderstanding some piece?
I ended up using 1 consumer group id per listener with a topic specifically for those events.
Let's say I have an application which consumes logs from kafka cluster. I want the application to periodically check for the availability of the cluster and based on that perform certain actions. I thought of a few approaches but was not sure which one is better or what is the best way to do this:
Create a MessageProducer and MessageConsumer. The producer publishes heartbeatTopic to the cluster and the consumer looks for it. The issue that I think for this is, where the application is concerned with only consuming, healthcheck has both producing and consuming part.
Create a MessageConsumer with a new groupId which continuously pools for new messages. This way the monitoring/healthcheck is doing the same thing which the application is supposed to do, which I think is good.
Create a MessageConsumer which does something different from actually consuming the messages. Something like listTopics (https://stackoverflow.com/a/47477448/2094963) .
Which of these methods is more preferable and why?
Going down a slightly different route here, you could potentially poll zookeeper (znode path - /brokers/ids) for this information by using the Apache Curator library.
Here's an idea that I tried and worked - I used the Curator's Leader Latch recipe for a similar requirement.
You could create an instance of LeaderLatch and invoke the getLeader() method. If at every invocation, you get a leader then it is safe to assume that the cluster is up and running otherwise there is something wrong with it.
I hope this helps.
EDIT: Adding the zookeeper node path where the leader information is stored.