I use Confluent.Kafka 1.9.2 C# library to create single Kafka consumer for listening topic with several partitions. Currently consumer drain out all messages from first partition and only then goes to next. As I know from KIP, I can avoid such behavior and achieve round-robin by changing max.partition.fetch.bytes parameter to lower value. I changed this value to 5000 bytes and pushed 10000 messages to first partition and 1000 to second, average size of messages is 2000 bytes, so consumer should to move between partitions every 2-3 messages (if I understand correctly). But it still drains out first partition before consuming second one. My only guess why it don't work as should is latest comment here that such approach can't work with several brokers, btw Kafka server that I use just has 6 brokers. Could it be the reason or maybe something else?
Related
When a Spring Kafka MessageListener is consuming messages from multiple partitions, it keeps processing messages from one partition until there are no more messages and only after that it continues with the next partition. (based on my observations)
Is it possible to set a max number of messages/batches and tell the Listener to switch faster to the next partition rather than later?
This would improve fairness and consume evenly from all assigned partitions.
switch faster to the next partition, consume evenly from all assigned partitions
I don't think Kafka has any properties for this. kafka consumer config
It's weird. You could see a partition replica in Kafka as a log file. Your consumer poll runs in one thread, for better performance, it should consume from one file, and the next poll will consume from another file rather than separate it and consume evenly from many partitions for each poll, right? Eventually, you still need to consume all of the messages on the topic.
Is there any limit on the number of consumers or consumer groups in Kafka?
I am planning to push 200 MB of data every 10 mins to a topic and have 200+ distinct consumers listen and consume from this topic. Is there any other recommended way to do this?
As Rohit answer states, there' no such limit.
Regarding your issue, it seems like you want to achieve some kind of paralellization of consumption. If you send 200 consumers with 200 different consumer groups, each consumer will read all the data independently, so you'll have 200 threads reading the same 200MB every 10 minutes (200x200 MB = 40GB received every 10 minutes). I guess you wanted every consumer to read 1MB every 10 mins with your approach, but that's not how it works.
If the logic implemented by each consumer is the same, you shouldn't declare more than a consumer group. If you declare two consumer groups, each one will read the same data, and you'll just repeat the job done, duplicating the output. Set different consumer groups if the job to be done on the topic's records is different: for example, one consumer group must store the records into a DDBB. The other consumer group must visualize the data into Grafana. Those are two different processing mechanisms, so each one must read all the data at its own. This is not the only reason to declare different consumer groups, but one example of them. There are multiple justifications for declaring more than a consumer group for a topic.
Imagine an scenario where the only job to be done is storing the messages into a DDBB. If you declare two consumer groups and launch your consumers, what you'll get is duplicate values stored in your database, as the first consumer group is just doing the same work than the second. Not only you are re-reading from kafka, you are re-storing the same messages into the ddbb.
In order to achieve launching multiple consumers that efficiently share the work (so for example, launching 4 consumers each one reads 50MB), you must partition your topic.
Only one consumer thread from the same consumer group can read from an specific partition. If you have 4 partitions in that topic, and 4 consumer threads that share the same consumer group, launching them will lead to each thread reading from one partition. If you launch two consumers, both will be assigned 2 partitions. Works like this:
And in this scenario, you do have a limit in the number of consumers concurrently reading if they share the same consumer group, which is, the number of partitions of that topic. If you launch a 5th consumer thread, one of them will block/wait, because it wasn't assigned any partition. In the example, consumer 5 waits until a partition is avaliable for him (so maybe waits forever).
What I suggest is: decide how many consumer threads you'll need to consume the data and partition the topic in base of that. If you, for example, partition the topic to 8 different partitions, you'll be able to launch 8 consumers from the same consumer group. Each one will then read, more or less, (depending on the producer partitioner) 25MB (200/8) of the incoming data, efficiently sharing the work load: Each consumer will read from its own partition.
If you launch 200 consumers with 200 different consumer groups,
you'll just multiply the work to be done x200, as every single consumer will read the data from start to end.
If you launch 200 consumers with the same consumer group and the topic has a single partition,
you'll have one thread doing all the job and 199 stale consumers.
In Kafka, there is no limit on the number of Consumer groups for a particular topic. However, the increase in consumer groups increases network utilization.
Worth nothing that newer versions of Kafka, store offsets in the internal Kafka topic called __consumer_offsets.
Lets say we have one topic "topic-1" in kafka with partition 5.
Consumer Group-A with 5 consumer attached to "topic-1" each partition. Due to large workload large number of message get publish. Now we want to scale up consumer / add more consumer in Group-A to process message.
How can we increase consumer ON_DEMAND in same group?
Is any way to do it from coding ? so that single message get consumed by each consumer.
Once load is decrease shut-down few consumer from same group.
What I would suggest is having some partitions as buffer for when the
load increases.
For eg. if having 5 partitions is enough for normal load, I would
suggest having 15 partitions for that topic but only 5 consumers
for them at the start.
Then, when the load increases, keep adding consumers, preferably in other machines, until the load decreases
You can have kubernetes do the autoscaling for you
Kafka framework suggest that the number of the consumers corresponds to the number of the partitions. Increasing the number of the consumers will not help as you will have one consumer per partition anyway and the rest will remain idle. If you need to speed it up you can read the data from Kafka and process them in another thread. You can scale with this number of processing threads and you will need to program it yourself.
I am using the Kafka Streams Processor API to construct a Kafka Streams application to retrieve messages from a Kafka topic. I have two consumer applications with the same Kafka Streams configuration. The difference is only in the message size. The 1st one has messages with 2000 characters (3KB) while 2nd one has messages with 34000 characters (60KB).
Now in my second consumer application I am getting too much lag which increases gradually with the traffic while my first application is able to process the messages at the same time without any lag.
My Stream configuration parameters are as below,
application.id=Application1
default.key.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
default.value.serde=org.apache.kafka.common.serialization.Serdes$StringSerde
num.stream.threads=1
commit.interval.ms=10
topology.optimization=all
Thanks
In order to consume messages faster, you need to increase the number of partitions (if it's not yet done, depending on the current value), and do one of the following two options:
1) increase the value for the config num.stream.threads within your application
or
2) start several applications with the same consumer group (the same application.id).
as for me, increasing num.stream.threads is preferable (until you reach the number of CPUs of the machine your app runs on). Try gradually increasing this value, e.g go from 4 over 6 to 8, and monitor the consumer lag of your application.
By increasing num.stream.threads your app will be able to consume messages in parallel, assuming you have enough partitions.
We started to use Apache Kafka to persist Timeseries data into a Timeseries database. What we started with was to just have a single topic, a producer writing to this topic and a single consumer reading from this topic and dumping the data to the Timeseries database.
We had 3 broker instances and what we noticed in the first try was that the producer was pretty fast in writing messages to the topic. Within a matter of 30 minutes, we had around 1.5 million messages. The consumer was just doing 300 messages per second.
Our next approach was to partition the topic and have more consumer instances (equal to the number of partitions). This definitely improved on the consumer write speed. Now my questions are:
What happens if I set my topic partition to 6, but I have only 3 broker instances. Which broker instance would be the leader for partition 1 to 6?
Is there a formula to determine how many partitions would I be needing? Since this was our test environment, we could play with it and scale it. We might not be able to do the same on our production environment. So how to determine the partition size?
The partitions get distributed amongst your brokers. It's impossible to know which broker will be elected leader of a given partition -- and it can change over time. Depending on which version of Kafka and which Consumer API you use, your consumer may or may not discover partition leaders on its own. With the SimpleConsumer you have to find partition leaders on your own, and respond to new leader election in your code (instead of having it handled by the API automatically).
As to the number of partitions -- there's no real "formula" other than this: you can have no more parallelism than you have partitions. If you have 4 partitions and 5 consumers, one of the consumers will starve. I usually use numbers like 12 or 60 or multiples thereof for the number of partitions for large topics. Something that divides easily and cleanly among variable numbers of consumers.
Also, note that you can later on change the number of partitions, with some caveats. See this answer for how and what the caveats are.