I started seeing the following error
[2020-06-12 20:09:01,324] ERROR [ReplicaManager broker=3] Error processing append operation on partition __consumer_offsets-10 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(3) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-10
My setup is having three brokers and all brokers are up. Couple of
things i did before this error was about pop up
I configured min.isr to be 2 in all the brokers. I created a topic
with replication factor 3 and starting producing the message from a
producer with ack = 1 with two brokers down. I brought up all the
brokers and started consumer.
How to go about troubleshooting this error
Consumer is also NOT be able to see this message ( not sure why, the message is supposed to be treated as "committed" as one broker was up when the producer was running)
Couple of facts
It is interesting to see rebalancing didnt happen yet WRT preferred leader starategy
$ kafka-topics --zookeeper 127.0.0.1:2181 --topic stock-prices --describe
Topic: stock-prices PartitionCount: 3 ReplicationFactor: 3 Configs: min.insync.replicas=2
Topic: stock-prices Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 1,2,3
Topic: stock-prices Partition: 1 Leader: 1 Replicas: 2,1,3 Isr: 1,2,3
Topic: stock-prices Partition: 2 Leader: 1 Replicas: 3,2,1 Isr: 1,2,3
Here's your problem:
You have set min.insync.replicas=2, which means you need at least two broker up and running to publish a message to a topic. If you let down 2 brokers, then you have only one left. Which means your insync.replica requirement is not fulfilled.
This has nothing to do with the Consumers, since this is about the brokers. When you set acks=1 that means your producer gets the acknowledgement when the message is published to one broker. (It will not acknowledge all the replicas are created).
So the problem is, you have your Producer, which gets acknowledged that the message is received, when a single broker (The leader) gets the message. But the leader cannot put replicas since there aren't any brokers up to sync.
One way to get this done is to set the acks=all, so your producer won't get acknowledged until all the replicas are done. It'll retry until the at least 2 in sync replicas are online.
Related
So I've been using Kafka 3.1.0 in production environment. One of the VMs had to be live migrated, but due to some issues live migration failed and node has been forcefully migrated, involving full VM restart.
After that VM booted up, Kafka stopped working "completely" - clients were not able to connect and produce/consume anything. JMX metrics were still showing up, but that node showed many partitions as "Offline partitions".
Looking into the logs, that particular node kept showing A LOT of INCONSISTENT_TOPIC_ID errors. Example:
WARN [ReplicaFetcher replicaId=4, leaderId=2, fetcherId=0] Received INCONSISTENT_TOPIC_ID from the leader for partition my-topic-3. This error may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread)
WARN [ReplicaFetcher replicaId=4, leaderId=2, fetcherId=0] Received INCONSISTENT_TOPIC_ID from the leader for partition my-topic-2. This error may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread)
WARN [ReplicaFetcher replicaId=4, leaderId=2, fetcherId=0] Received INCONSISTENT_TOPIC_ID from the leader for partition my-topic-3. This error may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread)
WARN [ReplicaFetcher replicaId=4, leaderId=2, fetcherId=0] Received INCONSISTENT_TOPIC_ID from the leader for partition my-topic-2. This error may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread)
WARN [ReplicaFetcher replicaId=4, leaderId=2, fetcherId=0] Received INCONSISTENT_TOPIC_ID from the leader for partition my-topic-3. This error may be returned transiently when the partition is being created or deleted, but it is not expected to persist. (kafka.server.ReplicaFetcherThread)
However, if you take a look at other Kafka brokers, they were showing a bit different errors (I don't have a logs sample) - UNKNOWN_TOPIC_ID...
Another interesting issue - I've described Kafka topic and this is what I've got:
Topic: my-topic TopicId: XXXXXXXXXXXXXXXXXXXXXX PartitionCount: 4 ReplicationFactor: 4 Configs: segment.bytes=214748364,unclean.leader.election.enable=true,retention.bytes=214748364
Topic: my-topic Partition: 0 Leader: 2 Replicas: 5,2,3,0 Isr: 2
Topic: my-topic Partition: 1 Leader: 0 Replicas: 0,1,2,3 Isr: 0
Topic: my-topic Partition: 2 Leader: 2 Replicas: 1,2,3,4 Isr: 2
Topic: my-topic Partition: 3 Leader: 2 Replicas: 2,3,4,5 Isr: 2
Why does it show only 1 ISR when there should be 4 per partition? Why did it happen in the first place?
I've added additional partition and this is what it shows now:
Topic: my-topic TopicId: XXXXXXXXXXXXXXXXXXXXXX PartitionCount: 5 ReplicationFactor: 4 Configs: segment.bytes=214748364,unclean.leader.election.enable=true,retention.bytes=214748364
Topic: my-topic Partition: 0 Leader: 2 Replicas: 5,2,3,0 Isr: 2
Topic: my-topic Partition: 1 Leader: 0 Replicas: 0,1,2,3 Isr: 0
Topic: my-topic Partition: 2 Leader: 2 Replicas: 1,2,3,4 Isr: 2
Topic: my-topic Partition: 3 Leader: 2 Replicas: 2,3,4,5 Isr: 2
Topic: my-topic Partition: 4 Leader: 3 Replicas: 3,4,5,0 Isr: 3,4,5,0
I know there is kafka-reassign-partitions.sh script and it fixed similar issue in preproduction environment, but I am more interested why did it happen in the first place?
Could this be related? I've set the parameter replica.lag.time.max.ms=5000 (over default 500) and even after restarting all nodes it didn't help.
This normally happens when the topic ID in the session does not match the topic ID in the log. To fix this issue you will have to make sure that the topic ID remains consistent across your cluster.
If you are using zookeeper, run this command in zkCli.sh, on one of your nodes that are still in sync and note the topic_id -
[zk: localhost:2181(CONNECTED) 10] get /brokers/topics/my-topic
{"partitions":{"0":[5,1,2],"1":[5,1,2],"2":[5,1,2],"3":[5,1,2],"4":
[5,1,2],"5":[5,1,2],"6":[5,1,2],"7":[5,1,2],"8":[5,1,2],"9":
[5,1,2]},"topic_id":"s3zoLdMp-T3CIotKlkBpMgL","adding_replicas":
{},"removing_replicas":{},"version":3}
Next, for each node, check the file partition.metadata for all the partitions of the topic my-topic. This file can be found in logs.dir (see server.properties).
For example, if logs.dir is set to /media/kafka-data, you can find it at -
/media/kafka-data/my-topic-1/partition.meta for partition 1.
/media/kafka-data/my-topic-2/partition.meta for partition 2, and so on.
The contents of the file may look like this, (you see it matches the topic_id that zookeeper has) -
version: 0
topic_id: s3zoLdMp-T3CIotKlkBpMgL
You'll need to make sure that the value of topic_id in all the parition.metadata files across your cluster for my-topic is the same. If you come across a different topic ID in any of the partitions you can edit it with any text editor (or write a script to do this for you).
Once done, you may need to restart your brokers one at a time for this change to take effect.
I will try to answer why the topic Id in Zookeeper may differ from topic Id stored in partition.metadata:
In certain situations, it is possible that the TopicId stored locally on a broker for a topic differs from the topicId stored for that topic on Zk. Currently, such situation arises when users use a <2.8 client to alterPartitions for a topic on a >=2.8 (including latest 3.4) brokers AND they use --zookeeper flag from the client. Note that --zookeeper has been marked deprecated for a long time and has been replaced by --bootstrap-server which doesn't face this problem.
The result of topic Id discrepancy leads to availability loss for the topic until user performs the mitigation steps listed in KAFKA-14190.
The exact sequence of steps are:
User uses pre 2.8 client to create a new topic in zookeeper directly
No TopicId is generated in Zookeeper
KafkaController listens to the ZNode, and a TopicChange event is created, During handling on this event, controller notices that there is no TopicId, it generated a new one and updates Zk.
At this stage, Zk has a TopicId.
User uses pre 2.8 client to increase the number of partitions for this topic
The client will replace/overwrite the entire existing Znode with new placement information. This will delete the existing TopicId in Zk (that was created by controller in step 3).
Next time KafkaController interacts with this ZNode, it will generate a new TopicId.
Note that we now have two different TopicIds for this topic name.
Broker may have a different topicId (older one) in metadata file and will complain about the mismatch when they encounter a new TopicId.
If i set the min.insync.replicas to 1, the leader will wait for a follower to fetch this record or he will send an ack to producer considering himself one of the ISR list.
If you set min.insync.replicas to 1 it is sufficient that the leader will acknowledge the receival of data. The leader will not wait for any followers to acknowledge the data.
Maybe it is worth mentioning that the min.insync.replicas only comes into effect when you have set your producer configuration acks set to all (or -1).
The minimal allowed value for min.insync.replicas is 1, so even if your replication factor of the topic is set to 1 you can't get below 1.
Is the leader part of ISR list?
Yes, the leader is part of the ISR list. You can see this when calling the command line tool kafka-topics. In the result you will notice that the same broker number will show up as "Leader" and in "ISR":
./bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic test
Topic: test PartitionCount: 1 ReplicationFactor: 3
Topic: test Partition: 0 Leader: 3 Replicas: 2,3,1 Isr: 3
I noticed that few of my kafka topics are behaving in a manner i cannot explain clearly.
For eg:
./kafka-topics.sh --describe --zookeeper ${ip}:2181 --topic test
Topic:test PartitionCount:3 ReplicationFactor:1 Configs:retention.ms=1209600000
Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: test Partition: 1 Leader: -1 Replicas: 2 Isr: 2
Topic: test Partition: 2 Leader: 3 Replicas: 3 Isr: 3
I am particularly concerned about Partition: 1 which shows Leader '-1'.
I also notice that roughly 1/3 of the messages produced to this topic fail due to a 'Timeout'. This I believe is a consequence of one partition not having a leader.
I was wondering if anyone has insights into why this issue occurs and how to recover from this in a Production scenario without losing data?
EDIT:
I am using the librdkafka based python producer; and the error message I see is Message failed delivery: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}
Most probably your second kafka broker is down.
In order to check active Kafka brokers you need to run
./zookeeper-shell.sh localhost:2181 <<< "ls /brokers/ids"
And the output should be similar to the one below:
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
[0, 1, 2]
[zk: localhost:2181(CONNECTED) 1]
If the second broker is not listed in the active brokers then you need to figure out why is not up and running (logs should tell you if something went wrong). I would also suggest to increase the replication-factor since you have a multi-broker configuration.
This often indicates that the broker leading that partition is offline. I would check the offline partitions metric to confirm this, but also check whether broker 2 is currently functional.
I am testing resilience of Kafka (apache; kafka_2.12-1.1.0). What i expect is that ISR of a topic should increase it self (i.e. replicate to available node) when ever a node crashes. I spent 4 days googling for possible solutions, but was of no use.
Have 3 node cluster, and created 3 brokers, 3 zoo keepers on it (1node = 1broker + 1 zookeeper) using docker (wurstmeister)
updated the below in server.properties
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
min.insync.replicas=2
default.replication.factor=3
started all brokers; waited a minute; created topic with replication3, min in sync replication 2
bin/kafka-topics.sh --create --zookeeper 172.31.31.142:2181,172.31.26.102:2181,172.31.17.252:2181 --config 'min.insync.replicas=2' --replication-factor 3 --partitions 1 --topic test2
when i describe the topic i see the below data
bash-4.4# bin/kafka-topics.sh --describe --zookeeper zookeeper:2181 --topic test2
Topic:test2 PartitionCount:1 ReplicationFactor:3 Configs:min.insync.replicas=2
Topic: test2 Partition: 0 Leader: 2 Replicas: 2,3,1 Isr: 2,3,1
So far so good, Now i start consuers; followd by producers. When the consumpmtion is in full throttle i kill the broker #2. Now when i describe the same topic i see the below ([Edit-1])
bash-4.4# bin/kafka-topics.sh --describe --zookeeper zookeeper:2181 --topic test2
Topic:test2 PartitionCount:1 ReplicationFactor:3 Configs:min.insync.replicas=2
Topic: test2 Partition: 0 Leader: 3 Replicas: 2,3,1 Isr: 3,1
bash-4.4# bin/kafka-topics.sh --describe --zookeeper zookeeper:2181 --topic __consumer_offsets
Topic:__consumer_offsets PartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer Topic: __consumer_offsets Partition: 0 Leader: 1 Replicas: 1,2,3 Isr: 1,3
Topic: __consumer_offsets Partition: 1 Leader: 3 Replicas: 2,3,1 Isr: 1,3
.. .. ..
[end of edit-1]
I let the kafka producer, consumer continue for couple of minutes; Q1: why does Replicas still show 2 when the broker 2 is down?
Now i added 2 more brokers to the cluster. While the producer, consumers continue i keep observing ISR; the no of ISR replicas dont increase they stick to 3,1 only. Q2: why is ISR not increasing even though 2 more brokers are available?.
Then i stopped the producer, consumer; waited couple of minutes; re-ran the describe command again --stillthe same result. when does ISR expand its replication?. Where there are 2 more nodes available, why did ISR not replicate?
i crreate my producer as follows
props.put("acks", "all");
props.put("retries", 4);
props.put("batch.size", new Integer(args[2]));// 60384
props.put("linger.ms", new Integer(args[3]));// 1
props.put("buffer.memory", args[4]);// 33554432
props.put("bootstrap.servers", args[6]);// host:port,host:port,host:port etc
props.put("max.request.size", "10485760");// 1048576
and consumer as follows
props.put("group.id", "testgroup");
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", args[2]);// 1000
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.ByteArrayDeserializer");
props.put("max.partition.fetch.bytes", args[3]);// 52428800
props.put("fetch.max.bytes", args[4]);// 1048576
props.put("fetch.message.max.bytes", args[5]);// 1048576
props.put("bootstrap.servers", args[6]);
props.put("max.poll.records", args[7]);
props.put("max.poll.interval.ms", "30000");
props.put("auto.offset.reset", "latest");
In a separate experiment, when i removed another broker the i started seeing errors that total in sync replications are less than the minimum required. Surprizingly in this state the producer is not blocked; but i see the error on the broker server.log. No new messages are getting enqueued. Q4:Shouldnt producer be blocked? instead of throwing error on broker side? or is my understanding wrong?
Any help please?
If I understand correctly, Kafka does not auto rebalance when you add brokers. A down replica will not be reassigned unless you use the repartition tool
It's not clear what difference are between your environments, but it looks like you didn't really kill a broker if it's still listed as a leader.
if you had two brokers down with min ISR as 2, then, yes you'll see errors. The producer should still be able to reach at least one broker, though, so I don't think it'll be completely blocked unless you set the ack value to all. The errors at the broker end are more related to placing replicas
Recap of the meaning of replica: all partition replicas are replicas, even the leader one; in other words 2 replicas means you have the leader and one follower.
When you describe the topic, for your only partition you see: "Replicas: 2,3,1 Isr: 3,1" which means that when the topic was created the leader partition was assigned to broker 2 (the first in the replicas list), and followers where assigned to brokers 3 and 1; now the broker 2 is the "preferred leader" for that partition.
This assignment is not going to change from itself (the leader may change, but not the "preferred leader"), so your followers will not move to other brokers, only the leader role can be given to another in-sync replica. (There is a property auto.leader.rebalance.enable which set to true will allow the leader role to go back to the preferred leader if it is up again, otherwise the leader role will be kept by the newly elected leader...
Next time try to kill the leader broker and you will see that a new leader will be elected and used, but the "Replicas: 2,3,1" will stay.
And if you set replication-factor=3 acks=all and min.insync.replicas=2, you can produce as long as 2 replicas acknowledge the writes (the leader and one follower), but will get logs on the broker if if is not possible to maintain 3 ISRs...
Hope this helps...
I have two broker kafka 0.10.2.0 cluster.Replication factor is 2. I am running 1.0.0 kafka stream application against this Kafka. In my kafka stream application, producer config has retries = 10 and retry.backoff.ms = 100
After running few minutes, I observed following logs in Kakfa server.log. Due to this Kafka stream application is throwing 'NOT_LEADER_FOR_PARTITION' exception.
What may be the possible reason? Please help me.
[2017-12-12 10:26:02,583] ERROR [ReplicaFetcherThread-0-1], Error for partition [__consumer_offsets,22] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
Each topic is served by one or multiple Brokers - one is leader and the remaining brokers are followers.
A producer needs to send new messages to the leader Broker which internally replicate the data to all followers.
I assume, that your producer client does not connect to the correct Broker, its connect to a follower instead of the leader, and this follower rejects your send request.
Try to run ./kafka-topics.sh --zookeeper localhost:2181 --topic your_topic --describe
Topic:your_topic PartitionCount:3 ReplicationFactor:1 Configs:retention.ms=14400000
Topic: your_topic Partition: 0 Leader: 2 Replicas: 2 Isr: 2
Topic: your_topic Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: your_topic Partition: 2 Leader: 1 Replicas: 1 Isr: 1
In this example you can see that your_topic have 3 partitions meaning all 3 brokers are leaders of that topic each on different partition, s.t broker 2 is leader on partition 0 and broker 0 and broker 1 are followers on partition 0.
Try setting these properties and see if it helps resolve the issue:
props.put(ProducerConfig.RETRIES_CONFIG, 10); //increase to 10 from default of 0
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG,
Integer.toString(Integer.MAX_VALUE)); // increase to infinity from default of 300 s
(Source)