Related
I have Kafka cluster of 3 nodes. I am using kafkacat to list data from Kafka. I configure PLAINTEXT and VPN_PLAINTEXT listeners:
listeners=PLAINTEXT://0.0.0.0:6667,VPN_PLAINTEXT://0.0.0.0:6669
advertised.listeners=PLAINTEXT://hadoop-kafka1-stg.local.company.cloud:6667,VPN_PLAINTEXT://hadoop-kafka1-stg-vip.local.company.cloud:6669
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL,VPN_PLAINTEXT:PLAINTEXT
We find out, we cannot consume data from node 1 (only) - from topics where partition leader is node 1 with error:
kafkacat -C -b hadoop-kafka1-stg-vip.local.company.cloud:6669 -t <topic-name> -o beginning -e -q -p 11
% ERROR: Topic <topic-name> [11] error: Broker: Not leader for partition
I can see, node 1 is leader for this partition:
Metadata for <topic-name> (from broker 3: hadoop-kafka3-stg-vip.local.company.cloud:6669/3):
3 brokers:
broker 2 at hadoop-kafka2-stg-vip.local.company.cloud:6669
broker 3 at hadoop-kafka3-stg-vip.local.company.cloud:6669 (controller)
broker 1 at hadoop-kafka1-stg-vip.local.company.cloud:6669
1 topics:
topic "<topic-name>" with 12 partitions:
partition 0, leader 2, replicas: 2,1,3, isrs: 3,2,1
partition 1, leader 3, replicas: 3,2,1, isrs: 3,2,1
partition 2, leader 1, replicas: 1,3,2, isrs: 3,2,1
partition 3, leader 2, replicas: 2,3,1, isrs: 3,2,1
partition 4, leader 3, replicas: 3,1,2, isrs: 3,2,1
partition 5, leader 1, replicas: 1,2,3, isrs: 3,2,1
partition 6, leader 2, replicas: 2,1,3, isrs: 3,2,1
partition 7, leader 3, replicas: 3,2,1, isrs: 3,2,1
partition 8, leader 1, replicas: 1,3,2, isrs: 3,2,1
partition 9, leader 2, replicas: 2,3,1, isrs: 3,2,1
partition 10, leader 3, replicas: 3,1,2, isrs: 3,2,1
partition 11, leader 1, replicas: 1,2,3, isrs: 3,2,1
I thought the data on node could be corrupted, so I remove everything from data directory kafka_data_dir for Kafka. When I start the daemon, I could see it syncing. After that, the issue persists. There is nothing suspicious in logs.
Could anybody describ and help to find out where is the root cause? Only node number 1 encounter this issue. When I ask the same node on port 6667, it works smoothly.
After deeper investigation of traffic with tcpdump I find out that the Kafka configuration was without any problem. When I asked node1 for topic partition, tcpdump on node1 did not catch any packets. Requests has been sent to node3. Requests should be forwarded based on DNS to the right Kafka nodes over Citrix, but the configuration was wrong:
hadoop-kafka1-stg-vip.local.company.cloud -> node 3
hadoop-kafka2-stg-vip.local.company.cloud -> node 2
hadoop-kafka3-stg-vip.local.company.cloud -> node 3
That's the reason, why requests for partition where node1 is not leader works, and when asked for partition where node1 was leader failed with message Broker: Not leader for partition because it was always forwared to node3 by Citrix.
My application runs over a Kubernetes cluster of 3 nodes and uses Kafka to stream data.
I am trying to check my system's ability to recover from node failure, so I deliberately fail one of the nodes for 1 minute.
Around 50% of the times, I experience data loss of a single data record after the node failure.
If the controller Kafka broker was running on the failed node, I see that a new controller broker was elected as expected.
When the data loss occur, I see the following error in the new controller broker log:
ERROR [Controller id=2 epoch=13] Controller 2 epoch 13 failed to
change state for partition __consumer_offsets-45 from OfflinePartition
to OnlinePartition (state.change.logger) [controller-event-thread]
I am not sure if that's the problem, but searching the web for information about this error made me suspect that I need to configure Kafka to have more than 1 replica for each topic.
This is how my topics/partitions/replicas configuration looks like:
My questions:
Is my suspicion that more replicas are required is correct?
If yes, how do I increase the number of topics replicas? I played around with a few broker parameters such as default.replication.factor and replication.factor but I did not see the number of replicas change.
If no, what is the meaning of this error log?
Thanks!
Yes, if the broker hosting the single replica goes down, then you can expect an unclean topic. If you have unclean leader election disabled, however, you shouldn't lose data that's already been persisted to the broker.
To modify existing topics, you must use kafka-reassign-partitions tool, not any of the broker settings, as those only apply for brand new topics.
Kafka | Increase replication factor of multiple topics
Ideally, you should disable auto topic creation, as well, to force clients to use Topic CRD resources in Strimzi that include a replication factor, and you can use other k8s tools to verify that they have values greater than 1.
Yes, you're right, you need to set the replication factor to more than 1 to be able to sustain the broker-level failures.
Once you add this value as default, the new topics will start having the configured number of replicas. But for existing topics, you need to follow the below instruction-
Describe the topic
$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic one
Topic: one PartitionCount: 3 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: one Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: one Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: one Partition: 2 Leader: 2 Replicas: 2 Isr: 2
Create the json file with the topic reassignment details
$ cat >>increase.json <<EOF
{
"version":1,
"partitions":[
{"topic":"one","partition":0,"replicas":[0,1,2]},
{"topic":"one","partition":1,"replicas":[1,0,2]},
{"topic":"one","partition":2,"replicas":[2,1,0]},
]
}
EOF
Execute this reassignment plan
$ ./bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file increase.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"one","partition":0,"replicas":[0,1,2],"log_dirs":["any","any"]},{"topic":"one","partition":1,"replicas":[1,0,2],"log_dirs":["any","any"]},{"topic":"one","partition":2,"replicas":[2,1.0],"log_dirs":["any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for one-0,one-1,one-2
Describe the topic again
$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic one
Topic: one PartitionCount: 3 ReplicationFactor: 3 Configs: segment.bytes=1073741824
Topic: one Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
Topic: one Partition: 1 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: one Partition: 2 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Meaning
rf =3 , byt Min.insyce.reaplica = 1, and When procuder setup ack=1, I want to make sure only need leader get data then cosnidering good.
However, I do see it comes with default even I setup it lower with the commmand. It shows automatically like this.
Topic:isr-set1 PartitionCount:6 ReplicationFactor:3 Configs:min.insync.replicas=1
Topic: isr-set1 Partition: 0 Leader: 3 Replicas: 3,1,5 Isr: 3,1,5
Topic: isr-set1 Partition: 1 Leader: 4 Replicas: 4,5,3 Isr: 4,5,3
Topic: isr-set1 Partition: 2 Leader: 0 Replicas: 0,3,4 Isr: 0,3,4
Topic: isr-set1 Partition: 3 Leader: 2 Replicas: 2,4,0 Isr: 2,4,0
I expect something like:
Topic:isr-set1 PartitionCount:6 ReplicationFactor:3 Configs:min.insync.replicas=1
Topic: isr-set1 Partition: 0 Leader: 3 Replicas: 3,1,5 Isr: 3
Topic: isr-set1 Partition: 1 Leader: 4 Replicas: 4,5,3 Isr: 4
Topic: isr-set1 Partition: 2 Leader: 0 Replicas: 0,3,4 Isr: 0
Topic: isr-set1 Partition: 3 Leader: 2 Replicas: 2,4,0 Isr: 2
ISR means in sync replicas, It can equal or lower than replication factor, but you can not control it.
If you create a topic with replication factor = 3, that means Kafka keeps that topic-partition log in 3 different places. To keep in 3 different places, follower replicas need to get synced with leader replica for the partition. In Sync Replicas or ISR means this replica sync process. How many replicas are in synced with the leader.
But in your case you configured Min ISR to the one. That means when you produce data with ack=all it look in Kafka if there is atleast min ISR number of replicas are synced with the leader including leader. If not, it returns an error and not produced.
If you only need Leader availability, this Min ISR is not considered. ISR is also not considered in producing messages. So if you don't need replication (In sync replicas) just create topic with replication factor = 1. But it is too risky because there is no fault tolerance for that topic partitions.
Replication
Kafka replicates the log for each topic's partitions across a
configurable number of servers (you can set this replication factor on
a topic-by-topic basis). This allows automatic failover to these
replicas when a server in the cluster fails so messages remain
available in the presence of failures.
min.insync.replicas
When a producer sets acks to "all" (or "-1"), this configuration
specifies the minimum number of replicas that must acknowledge a write
for the write to be considered successful. If this minimum cannot be
met, then the producer will raise an exception (either
NotEnoughReplicas or NotEnoughReplicasAfterAppend). When used
together, min.insync.replicas and acks allow you to enforce greater
durability guarantees. A typical scenario would be to create a topic
with a replication factor of 3, set min.insync.replicas to 2, and
produce with acks of "all". This will ensure that the producer raises
an exception if a majority of replicas do not receive a write.
I have an kafka cluster (3 machine with 1 zookeeper and 1 broker run on each machine)
I am using kafka_exporter to monitoring consumer lag metric, it's work fine in normal case.
But, when i kill 1 broker, the Prometheus cannot get metric from http://machine1:9308/metric (kafka_exporter metric endpoint), because it take a long time to get data (1,5m), so it will be timeout.
Now, if I restart kafka_exporter I will see some error:
Cannot get leader of topic __consumer_offsets partition 20: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes
When I run the command: kafka-topics.bat --describe --zookeeper machine1:2181,machine2:2181,machine3:2181 --topic __consumer_offsets
The result are:
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:1 Configs:compression.type=producer,cleanup.policy=compact,segment.bytes=104857600
Topic: __consumer_offsets Partition: 0 Leader: -1 Replicas: 1 Isr: 1
Topic: __consumer_offsets Partition: 1 Leader: 2 Replicas: 2 Isr: 2
Topic: __consumer_offsets Partition: 49 Leader: 2 Replicas: 2 Isr: 2
Is this a configuration error? And how can I get the consumer lag in this case? The "Leader: -1" is an error? if I shutdown the machine 1 forever, it's still work fine?
The leader being -1 means that there is no other broker in the cluster that has a copy of the data for the partition.
The problem in your case is that the replication factor for your topic __consumer_offsets is 1, which means that there is only one broker that hosts the data of any partition in the topic. If you lose any one of the brokers, all the partitions on the broker become unavailable resulting in the topic becoming unavailable. So, your kafka_exporter will fail to read from this topic.
The fix to this if you want to continue exporting consumer offsets on a broker loss, is to reconfigure the topic __consumer_offsets to have replication factor more than 1.
Advised Config - Replication factor - 3, min.insync.replicas - 2.
I have a Kafka cluster with 5 partitions.
On scaling down to 3, leader election took place several times.
Finally only one broker became the leader for all the 3 partitions of one of my topics.
Topic: test PartitionCount:3 ReplicationFactor:3
Topic: test Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,1,0
Topic: test Partition: 1 Leader: 2 Replicas: 3,1,2 Isr: 2,1
Topic: test Partition: 2 Leader: 2 Replicas: 4,2,3 Isr: 2
2,1,0 are the brokers that are running.
partition 0 is available with 2, 0, 1. All the brokers are available.
So, isr=2,1,0
partition 1 is available with 3, 1, 2 but 3 is removed broker. So
isr=2,1
partition 2 is available with 4,2,3 but both 4,3 are removed brokers. So
isr=2
Note that, only 2 has been elected as the leader. Even if we assume that it has the highest watermark among the other brokers, all the ISRs for a given partition could have been in sync, therefore all have the same offsets for a given partition (otherwise they would have been removed from the ISR).
I have waited for a lot of time (there is a time after which if one of the replicas is not up to the mark, it will be removed from ISR) but still that is the leader election.
Leaders can be evenly distributed (load balanced).
For example, partition-0 leader can be 0
partition 1 leader can be 1
partition 2 leader can be 2
Why is this not so?
Note: I did not enable unclean leader election. It is the default value only.
If we assume that 0,1 came up after the leader election happened, why
is not there a re-election then? If the ISRs are updated, ideally the
leaders should also be. Isn't it?
i.e. if Kafka knows that 0,1 are up
and have in-sync replicas, it SHOULD have conducted one more leader
election.
Is there any specific reason why is it not so?
Kafka has the concept of a preferred leader, meaning that if possible it will elect that replica to serve as the leader. The first replica listed in the replicas list is the preferred leader. Now looking at the current cluster state:
Topic: test Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,1,0
Topic: test Partition: 1 Leader: 2 Replicas: 3,1,2 Isr: 2,1
Topic: test Partition: 2 Leader: 2 Replicas: 4,2,3 Isr: 2
Partition 0, broker 2 is the preferred leader and is the current leader
Partition 1, broker 3 is the preferred leader but it's not in-sync, so a random leader is picked between 2 and 1
Partition 2, broker 4 is the preferred leader but again 4 is not in-sync. Only 2 is in-sync, so it's elected.
If all your brokers were to go back in-sync, by default Kafka would re-elect the preferred leaders (or it can be forced using the kafka-preferred-replica-election.sh tool, see Balancing leadership).
If the missing brokers are not going to be restarted, you can change the replica assignments for the partitions, to balance the leadership manually using the kafka-reassign-partitions.sh tool. Just make sure you put the preferred leader as the first entry in the replicas list.