I have two applications one is a regular Kafka consumer and the other is a gRPC based microservice. Kafka consumer is only responsible for consumption of messages and the business logic resides within the microservice. Also the key for messages within our Kafka topic is null, so Kafka does round-robin assignment of messages to the partitions which distributes the incoming messages evenly to all partitions. At the end of the day I am dealing with non-transactional storage (BigTable) so I have to make sure that there is only one thread responsible for reading, updating and writing a row-key into the storage in order to avoid race-conditions. My gRPC microservice is running within a Kubernetes cluster on multiple pods, how can I make sure that a message object belonging to a particular row-key goes to the same pod within the Kubernetes cluster so that there are no race-conditions?? My microservice is responsible for writing the final output to the BigTable and the microservice is sitting behind a load balancer.
It might not be a solution if you already have a (big) code base, but streaming frameworks like Apache Flink handle this pretty gracefully.
It has an operator keyBy() that does exactly what you want. It will 'sort' the messages by a key defined by you and will guarantee messages with the same key get processed by the same thread.
Related
I have built a micro service platform based on kubernetes, but Kafka is used as MQ in the service. Now a very confusing question has arisen. Kubernetes is designed to facilitate the expansion of micro services. However, when the expansion exceeds the number of Kafka partitions, some micro services cannot be consumed. What should I do?
This is a Kafka limitation and has nothing to do with your service scheduler.
Kafka consumer groups simply cannot scale beyond the partition count. So, if you have a single partitioned topic because you care about strict event ordering, then only one replica of your service can be active and consuming from the topic, and you'd need to handle failover in specific ways that is outside the scope of Kafka itself.
If your concern is the k8s autoscaler, then you can look into the KEDA autoscaler for Kafka services
Kafka, as OneCricketeer notes, bounds the parallelism of consumption by the number of partitions.
If you couple processing with consumption, this limits the number of instances which will be performing work at any given time to the number of partitions to be consumed. Because the Kafka consumer group protocol includes support for reassigning partitions consumed by a crashed (or non-responsive...) consumer to a different consumer in the group, running more instances of the service than there are partitions at least allows for the other instances to be hot spares for fast failover.
It's possible to decouple processing from consumption. The broad outline of could be to have every instance of your service join the consumer group. Up to the number of instances consuming will actually consume from the topic. They can then make a load-balanced network request to another (or the same) instance based on the message they consume to do the processing. If you allow the consumer to have multiple requests in flight, this expands your scaling horizon to max-in-flight-requests * number-of-partitions.
If it happens that the messages in a partition don't need to be processed in order, simple round-robin load-balancing of the requests is sufficient.
Conversely, if it's the case that there are effectively multiple logical streams of messages multiplexed into a given partition (e.g. if messages are keyed by equipment ID; the second message for ID A needs to be processed after the first message, but could be processed in any order relative to messages from ID B), you can still do this, but it needs some care around ensuring ordering. Additionally, given the amount of throughput you should be able to get from a consumer of a single partition, needing to scale out to the point where you have more processing instances than partitions suggests that you'll want to investigate load-balancing approaches where if request B needs to be processed after request A (presumably because request A could affect the result of request B), A and B get routed to the same instance so they can leverage local in-memory state rather than do a read-from-db then write-to-db pas de deux.
This sort of architecture can be implemented in any language, though maintaining a reasonable level of availability and consistency is going to be difficult. There are frameworks and toolkits which can deliver a lot of this functionality: Akka (JVM), Akka.Net, and Protoactor all implement useful primitives in this area (disclaimer: I'm employed by Lightbend, which maintains and provides commercial support for one of those, though I'd have (and actually have) made the same recommendations prior to my employment there).
When consuming messages from Kafka in this style of architecture, you will definitely have to make the choice between at-most-once and at-least-once delivery guarantees and that will drive decisions around when you commit offsets. Note particularly that you need to be careful, if doing at-least-once, to not commit until every message up to that offset has been processed (or discarded), lest you end up with "at-least-zero-times", which isn't a useful guarantee. If doing at-least-once, you may also want to try for effectively-once: at-least-once with idempotent processing.
We are thinking about using the Strimzi Kafka-Bridge(https://strimzi.io/docs/bridge/latest/#proc-creating-kafka-bridge-consumer-bridge) as HTTP(s) Gateway to an existing Kafka Cluster.
The documentation mentions the creation of consumers using arbitrary names for taking part in a consumer-group. These names can subsequently be used to consume messages, seek or sync offsets,...
The question is: Am I right in assuming the following?
The bridge-consumers seem to be created and maintained just in one Kafka-Bridge instance.
If I want to use more than one bridge because of fault-tolerance-requirements, the name-information about a specific consumer will not be available on the other nodes, since there is no synchronization or common storage between the bridge-nodes.
So if the clients of the kafka-bridge are not sticky, as soon as a it communicates (e.g. because of round-robin handling by a load-balancer) with another node, the consumer-information will not be available and the http(s)-clients must be prepared to reconfigure the consumers on the new communicating node.
The offsets will be lost. Worst case the fetching of messages and syncing their offsets will always happen on different nodes.
Or did I overlook anything?
You are right. The state and the Kafka connections are currently not shared in any way between the bridge instances. The general recommendation is that when using consumers, you should run the bridge only with single replica (and if needed deploy different bridge instances for different consumer groups).
From an event sourcing/CQRS perspective: Say I have a consumer group of 2 instances, that's subscribed to a topic. On startup/subscription, each instance processes its share of the event stream, and builds a local view of the data.
When an external request comes in with a command to update the data, how would that request be routed to the correct instance in the group? If the data were partitioned by entity ID so that odd-numbered IDs went to consumer 1 and evens to consumer 2, how would that be communicated to the consumers? Or, for that matter, whatever reverse-proxy or service-mesh is responsible for sending that incoming request to the correct instance?
And what happens when the consumer group is re-balanced due to the addition or subtraction of consumers? Is that somehow automatically communicated the routing mechanism?
Is there a gap in service while the consumers all rebuild their local model from their new set of events from the given topics?
This seems to apply to both the command and query side of things, if they're both divided between multiple instances with partitioned data...
Am I even thinking about this correctly?
Thank you
Kafka partitioning is great for sharding streams of commands and events by the entity they affect, but not for using this sharding in other means (e.g. for routing requests).
The broad technique for sharding the entity state I'd recommend is to not rely on Kafka partitioning for that (only using the topic partitions to ensure ordering of commands/events for an entity, i.e. by having all commands/events for a given entity be in one partition), but instead using something external to coordinate those shards (candidates would include leases in zookeeper/etcd/consul or cluster sharding from akka (JVM) or akka.net or cloudstate/akka serverless (more polyglot)). From there, there are two broad approaches you can take:
(most really applicable if the number of entity shards for state and processing happens to equal the number of Kafka partitions) move part of the consumer group protocol into your application and have the instance which owns a particular shard consume a particular partition
have the instances ingesting from Kafka resolve the shard for an entity and which instance owns that shard and then route a request to that instance. The same pattern would also allow things like HTTP requests for an entity to be handled by any instance. By doing this you're making a service implemented in a stateful manner present to things like a service mesh/container scheduler/load balancer as a more stateless service would present.
As far as I understand, the workload that needs to be executed by KSQL, will be stored in a meta topic (Command Topic), to which all of the KSQL Server nodes are subscribed as Kafka consumers. Incoming new workload in the form of a query, or more granular, singular tasks of a complex query, are written into that topic and all the consumers are obviously notified. But how do the KSQL Servers elect the "worker" for that specific task?
I found following KSQL Server Elastic Scaling in Kubernetes SO answer, as well as this this Confluent deep dive on that topic, but both imply that all KSQL Servers take the task, not just one of them. So how does KSQL ensure the same data is not processed twice, both from data consistency and load efficiency perspective?
My guess would be that all of the KSQL server nodes are within the same Kafka consumer group, so the same Kafka message is not interpreted twice, but each KSQL server node is responsible for one partition of that topic, which leads to effective distribution of load. Is my assumption right? Is this the same, how multiple Kafka Connect instances behave?
I have two kafka clusters say A and B, B is replica of A. I would like to consume messages from cluster B only if A is down and viceversa. Nevertheless consuming messages from both the clusters would result in duplicate messages. So is there any way I can configure my kafka consumer to receive messages from only one cluster.
Thanks--
So is there any way I can configure my kafka consumer to receive messages from only one cluster.
Yes: a Kafka consumer instance will always receive messages from one Kafka cluster only. That is, there's no built-in option to use the same consumer instance for reading from 2+ clusters. But I think you are looking for something different, see below.
I would like to consume messages from cluster B only if A is down and viceversa. Nevertheless consuming messages from both the clusters would result in duplicate messages.
There's no built-in failover support such as "switch to cluster B if cluster A fails" in Kafka's consumer API. If you need such behavior (as in your case), you would need to do so in your application that uses the Kafka consumer API.
For example, you could create a consumer instance to read from cluster A, monitor that instance and/or that cluster to determine whether failover to cluster B is required, and (if needed) perform the failover to B by creating another consumer instance to read from B in the event that A fails.
There are a few gotchas however that makes this failover behavior more complex than my simplified example. One difficulty is to know which messages from cluster A have already been read when switching over to B: this is tricky because, typically, the message offsets differ between clusters so determining whether the "copy" of a message (in B) was already read (from A) is not trivial.
Note: Sometimes you can simplify such an application / such a failover logic in situations where e.g. message processing is idempotent (i.e. where duplicate messages / duplicate processing of messages will not alter the processing outcome).