Is there any API or client library exist which can tell me how much percent topic is filled with data so that I can figure out is there any way to check whether partitions are balanced or not
This is a good strategy to discuss before designing and development on Kafka.
The first point you need to consider how you are defining your key and whats exactly partitioner you are planning to use while producing a message to the topics.
Thumb-rule:
If you not bothering collecting messages in different groups based on key just pass the key as null to redistribute your messages in a round-robin manner.
You can also use a custom partitioner to define partitioning in case you need to do some more refinement.
To check the partition distribution, the best approach is to check the lagging on each partition and rate byte/sec
There many ways to monitor
1.You can use simple API to get various matrices like lagging, rate, etc
You can refer here Kafka Metrices
kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica
2.I generally prefer Grafana with JMX exported it will visualize matrices
Grafana
3.We can also use CLI to identify each partition offset and lagging, and really give you the overall figure instantly
bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group consumer-group
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test 1 10 30 20 consumer-group
You can also do with programmatically
How to identify partition lagging
Confluent Control-Center is paid one but a very interesting tool to monitor overall Kafka including consumer and its partitions/
Confluent control center
Assume that you created a topic X. Your producers started to push tons of data into your topic. Your topic is growing exponentially. Depending on the configuration log.segment.bytes, Kafka will create a new segment and start writing data into it. Old segment will be kept for log.retention.ms milliseconds. Because of this, 100% of a topic itself is tricky calculate.
However, if you are looking for a tool that can allocate partitions depending on the load on each broker then I would recommend looking into Kafka-kit (https://www.datadoghq.com/blog/engineering/introducing-kafka-kit-tools-for-scaling-kafka/).
Related
I'm relatively new to working with Kafka, below is a sample of what my current set up is.
Kafka Setup
Multiple topics that all have one partition each. 2 Consumer Groups with each group containing one consumer.
The issue I am seeing is that the Lag is enormous, sometimes upwards of 8-10 hours waiting for consuming, the load is about 100-200 million messages a day
What steps should I look at in order to address this? Is it as simple as reassigning partitions or creating new partitions for the 3 topics that are being consumed by the two consumers? - I've also looked at compressing the contents of the producer with gzip but it doesn't really help in terms of the lag. I've looked at network connections and don't feel that it is anything got to do with this. If anyone could point me in the direction of Kafka and Low Latency documents that would be good also.
Generally the flow is to parallelize your consumption through the increase on the number of partitions and consumers in consumer groups that subscribe to those topics with increased partitions (Nconsumers <= Npartitions).
And distribute your topics with increase on the number of brokers in your cluster.
So from topic considerations:
Less partition per topic result:
in producer and/or consumer lag
starved or overloaded brokers and consumers.
(But take into account) More partition per topic result in:
More broker resources – file handlers and memory.
There is an overhead with each additional partition and a number of partitions a broker can handle is limited.
Overhead of replication load
Then increase the number of consumers in that consumer groups.
Try increasing partition per topic, but by itself it should not help! You also will need to increase the number of consumers in your consumer group. Is that single consumers or consumer groups on your diagram? How many consumers in your consumer group vs partitions on the topic that they are subscibed to.
From this in your message:
I've also looked at compressing the contents of the producer with gzip but it doesn't really help in terms of the lag.
I get an idead that your messages may be huge! Is it so? In case yes, try to keep messages small (for example by excluding BLOBs and keep external links to them)
Still the issue may be somewhere else like bad configs, consumer commit messages (acknowledgment handling), etc.
So, I highly advice you to read article Fine-tune Kafka performance with the Kafka optimization theorem
I also advise you to go through Apache Kafka courses on Confluent web-page
This should be added as a comment, but I haven't had permissions to do so. The provided info is very limited with incorrect diagram, which limits the ability to provide an adequate helpfull answer. If possible please correct your diagram and add more details about your set-up like:
broker configuration, file attached;
consumer set-up (Consumer commit messages);
producer set-up;
topic set-up;
kafka version (the defaults differ with major/minor versions)
The provided diagram is not correct in the notion of topic - partition relationship, so I assume it is a mistype and Partition 0 must be substituded with Broker 0, right?
Kafka's topics are divided into several partitions. While the topic is a logical concept in Kafka, a partition is the smallest storage unit that holds a subset of records owned by a topic...
Then there is an open question on the number of partiotions in each topic and the number of topics in each broker, as well as the number of brokers in your cluster!
I am investigating on Kafka to assess its suitability for our use case. Can you please help me understand how flexible is Kafka with changing the number of partitions for an existing Topic?
Specifically,
Is it possible to change the number of partitions without tearing down the cluster?
And is it possible to do that without bringing down the topic?
Will adding/removing partitions automatically take care of redistributing messages across the new partitions?
Ideally, I would want the change to be transparent to the producers and consumers. Does Kafka ensure this?
Update:
From my understanding so far, it looks like Kafka's design cannot allow this because it mapping of consumer groups to partitions will have to be altered. Is that correct?
1.Is it possible to change the number of partitions without tearing down the cluster?
Yes kafka supports increasing the number of partitions at runtime but doesn't support decreasing number of partitions due to its design
2.And is it possible to do that without bringing down the topic?
Yes provided you are increasing partitions.
3.Will adding/removing partitions automatically take care of redistributing messages across the new partitions?
As mentioned earlier removing partitions is not supported .
When you increase the number of partitions, the existing messages will remain in the same partitions as before only the new messages will be considered for new partitions (also depending on you partitioner logic). Increasing the partitions for a topic will trigger a cluster rebalance , where in the consumers and producers will get notified with the updated metadata of the topics. Producers will start sending messages to new partitions after receiving updated metadata and consumer rebalancer will redistribute the partitions among the consumers groups and resume consumption from the last committed offset.All this will happen under the hood , so you wont have to do any changes at client side
Yes, it it perfectly possible. You just execute the following command against the topic of your choice: bin/kafka-topics.sh --zookeeper zk_host:port --alter --topic <your_topic_name> --partitions <new_partition_count>. Remember, Kafka only allows increasing the number of partitions, because decreasing it would cause in data loss.
There's a catch here. Kafka doc says the following:
Be aware that one use case for partitions is to semantically partition
data, and adding partitions doesn't change the partitioning of
existing data so this may disturb consumers if they rely on that
partition. That is if data is partitioned by hash(key) %
number_of_partitions then this partitioning will potentially be
shuffled by adding partitions but Kafka will not attempt to
automatically redistribute data in any way.
Yes, if by bringing down the topic you mean deleting the topic.
Once you've increased the partition count, Kafka would trigger a rebalance, for consumers who are subscribing to that topic, and on subsequent polls, the partitions would get distributed across the consumers. It's transparent to the client code, you don't have to worry about it.
NOTE: As I mentioned before, you can only add partitions, removing is not possible.
+one more thing, if you are using stateful operations in clients like aggregations(making use of statestore), change in partition will kill all the streams thread in consumer. This is expected as increase in partition may corrupt stateful applications. So beware changing partition size, it may break stateful consumers connected to the topic.
Good read: Why does kafka streams threads die when the source topic partitions changes ? Can anyone point to reading material around this?
We are working on Confluent Platform and we are still getting to know the internals. But we have implemented generic use cases . We are trying to optimizing our cluster
In my use case, I need to increase the number of partitions of a topic . What is the impact of it ? Can you please share of it
Sure, you can increase partitions.
However,
Increasing partitions does not move existing data. If using Confluent Enterprise, you could use confluent-rebalancer, or if not, then kafka-reassign-partitions CLI tool. So, you'll definitely want to rebalance a topic to "optimize" the cluster.
During the retention period of the topic (read: for the existing data), if you previously had a producer sending data to partition N, and now had N+1 partitions, then you lose ordering of those messages that solely existed in partition N. New messages could be spread across new partitions that a new producer calculates with the DefaultPartitioner. If you don't send keys with messages, then this isn't a problem.
What is maximum limit of topics can a consumer subscribe to in Kafka. Am not able to find this value documented anywhere.
If consumer subscribes 500000 or more topics, will there be downgrade in performance.
500,000 or more topics in a single Kafka cluster would be a bad design from the broker point of view. You typically want to keep the number of topic partitions down to the low tens of thousands.
If you find yourself thinking you need that many topics in Kafka you might instead want to consider creating a smaller number of topics and having 500,000 or more keys instead. The number of keys in Kafka is unlimited.
To be technical the "maximum" number of topics you could be subscribed to would be constrained by the available memory space for your consumer process (if your topics are listed explicitly then a very large portion of the Java String pool will be your topics). This seems the less likely limiting factor (listing that many topics explicitly is prohibitive).
Another consideration is how the Topic assignment data structures are setup at Group Coordinator Brokers. They could run out of space to record the topic assignment depending on how they do it.
Lastly, which is the most plausible, is the available memory on your Apache Zookeeper node. ZK keeps ALL data in memory for fast retrieval. ZK is also not sharded, meaning all data MUST fit onto one node. This means there is a limit to the number of topics you can create, which is constrained by the available memory on a ZK node.
Consumption is initiated by the consumers. The act of subscribing to a topic does not mean the consumer will start receiving messages for that topic. So as long as the consumer can poll and process data for that many topics, Kafka should be fine as well.
Consumer is fairly independent entity than Kafka cluster, unless you are talking about build in command line consumer that is shipped with Kafka
That said logic of subscribing to a kafka topic, how many to subscribe to and how to handle that data is upto the consumer. So scalability issue here lies with consumer logic
Last but not the least, I am not sure it is a good idea to consumer too many topics within a single consumer. The vary purpose of pub sub mechanism that Kafka provides through the segregation of messages into various topics is to facilitate the handling of specific category of messages using separate consumers. So I think if you want to consume many topics like few 1000s of them using a single consumer, why divide the data into separate topics first using Kafka.
We are planning to expand cluster from 2 node to 8 node. The partition reassignment tool has the option to move topic or partition.
For re-distribution of partitions I am planning to follow the below steps.
Irrespective of number of node additions,If I give all the topics in the topic-to-move.json and all the brokers in the below command then it will give equal distribution of partition among nodes correct ?
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3,4,5,6,7" --generate
After this I am planning to apply the json
--execute --reassignment-json-file generated-json file
Will this cause any problem ?
This step seems to be more general but why it is not documented this way?
There are few things to be aware of:
Evenly distributing partitions does not necessarily evenly distribute data. Some partitions hold more data than others so you need to look at how much data is in each partition to make a plan to spread the data evenly across the brokers. This is particularly true is you have single partition topics or unevenly balanced keys.
Be "rack aware". If the 8 brokers are in 3 Amazon availability zones or on two different power supplies or network switches in your data center then be careful not to distribute the leader and all it's replicas into the same Rack ID or you lose your high availability.
Consider using replication quotas. When you move lots of data between brokers it can take away network bandwidth from active producers and consumers. Kafka 0.10+ added separate replication quotas (bandwidth throttling) so that you could reduce the bandwidth used during reassignment so it will not negatively impact you live client traffic. Just do t throttle too low or you reassignment might not ever catch up to the new changes coming from producers.
You may want to consider using a third party tool to help to automatically build a reassignment plan. Yahoo!'s Kafka Manager has a reassignment feature (see https://github.com/yahoo/kafka-manager/blob/master/README.md) and Confluent has a 30 day free trial for their Auto Rebalancer that allows both expansion and reduction of broker nodes with rack awareness and throttled reassignment (see http://docs.confluent.io/current/kafka/rebalancer/rebalancer.html)
By passing the full topic list to the tool, all your partitions are likely to be reassigned.
In an already large cluster (> 1000s topics) this would cause a lot of unnecessary data copy and leader elections. So typically you would only provide a subset of your topics and only specify the new brokers as destinations to minimize the work required to complete the reassignment.
If your cluster is small enough and without GBs/TBs of data, passing all topics to the reassignment tool should be fine and it's probably the easiest/fastest.