We are tackling with production issue which might take few days to fix. Majority of Kafka nodes are active. One node is down. We will bring it up after the bugs are fixed. Our Kafka version is 2.1.X.
I was curious what are the impacts of starting an inactive broker after few days.
Are there any issues we might observe ? (Especially impacts on consumer after replicas are catching up on restarted broker.)
What are the contingencies to rollout safely ?
Whenever a broker is down, it's recommended to restore as quickly as you can. The consumer offsets expire and log-end offsets are also getting cleaned regularly in an active cluster.
We were able to restore node after 4 days but it wasn't easy operation. We restore the Kafka cluster by enabling unclean leader election. We were having controlled shutdowns due to bad leader assignments. After the inactive node was restored, we disabled the unclean leader election.
Things to take into account:
In prod usually the clients can't have any downtime. Monitor consumer
groups for any long rebalances or lagging commits beyond SLA's.
Running a preferred replica election if the offset on restored nodes
are live.
Reset offsets on consumer group. This does require a short
downtime.
Rollback:
You can rollback topic partition using reassignment tool but there are no easy rollback.
Related
I was performing a rolling kafka broker upgrade from 2.2.1 to 3.2.0(from cp 5.2.2 to 7.2.1).This was done as a rolling upgrade on kafka cluster.There are three pods i.e, kafka-0,kafka-1 and kafka-2.The replication factor for this topic is 2.During this rolling upgrade I am facing a downtime as leader is not assigned for "test" partition.
Cannot get leader of topic test partition 4: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes
when I added log I can see Partition leader=-1
I tried increasing the replication factor for one of the topic it worked but no concrete reason why did that happen
Probably you have min.in.sync.replicas=2, and unclean.leader.election=false. Therefore, when the current leader went down, it wasn't fully replicated, so no other replica will take its place.
You have 3 brokers, so there's no reason not to have replication factor of 3, other than to save cost, with the tradeoff of unavailability when losing any one replica.
The producer should retry until a new leader is elected. Make sure you have acks= all and min.insync.replicas=2 as #OneCricketeer suggested.
You should expect to see the "NotLeaderException" during upgrades or broker failures though. It's normal and as long as the producer retries and the above configurations are set there won't be data loss.
We have experimented an issue that could be caused by the parameter auto.leader.rebalance.enable, which is set to true by default on brokers.
In detail, when the automatic rebalance occurs, for example after a broker restart, some partition leaders are moved to match the preferred leader.
After this event, some stateful Kafka Streams applications blocks on the source partitions whose leader has been moved and the consumer lag start to grow.
Is it a known issue? Why don't applications receive the information regarding the change of leader?
The tactical solution we found in case we need to execute a rolling restart of brokers is:
Stop stateful applications
Perform brokers rolling restarts.
Wait 5 minutes (default value) untile the automatic leader rebalance occurs
Start stateful applications.
We are using Confluent Platform Community 5.2.2, deployed on a 3 node on prem cluster.
We are trying to recreate what happened in the test environment but without success. is it possible that it is influenced by the load of the cluster, much lower in test?
Thanks in Advance!
Giorgio
Based of the answer given in Is Zookeeper a must for Kafka?.
It is clear that what is the responsibility of Zookeeper in Kafka 0.9 and above
I just wanted to understand what will be the impact if zookeeper cluster goes down completely?
kafka uses ZK for membership (figure out what brokers exist and which of them are alive) and leader election (elect the one broker that is controller for the cluster at any moment).
simply put - if ZK fails kafka dies.
if ZK sneezes (say a particularly long GC pause or a short network connectivity issue) a kafka cluster may temporarily "lose" any number of members and/or the controller. by the time this settles you may have a new controller and new leader brokers for all partitions (which may or may not cause loss of acknowledged data, see "unclean leader election"). I'm not sure if all ongoing transactions would be rolled back - never tried.
In our production environment, we often see that the partitions go under-replicated while consuming the messages from the topics. We are using Kafka 0.11. From the documentation what is understand is
Configuration parameter replica.lag.max.messages was removed. Partition leaders will no longer consider the number of lagging messages when deciding which replicas are in sync.
Configuration parameter replica.lag.time.max.ms now refers not just to the time passed since last fetch request from the replica, but also to time since the replica last caught up. Replicas that are still fetching messages from leaders but did not catch up to the latest messages in replica.lag.time.max.ms will be considered out of sync.
How do we fix this issue? What are the different reasons for replicas go out of sync? In our scenario, we have all the Kafka brokers in the single RACK of the blade servers and all are using the same network with 10GBPS Ethernet(Simplex). I do not see any reason for the replicas to go out of sync due to the network.
We faced the same issue:
Solution was:
Restart the Zookeeper leader.
Restart the broker\brokers that are not replicating some of the partitions.
No data lose.
The issue is due to a faulty state in ZK, there was an opened issue on ZK for this, don't remember the number.
I faced the same issue on Kafka 2.0,
On restart Kafka controller node everything caught-up on the replicas.
But still looking for the reasons why few partitions are under-replicated whereas the other partitions on the same nodes for the same topic works good, and this issue i see on a random partitions.
Do NOT run reassignment for all topics together, consider running it for small portions.
Find the topic that has under-replicated partitions and where reassignment process can't be completed.
Set unclean.leader.election.enable to true for this topic.
Find under-replicated partition that stuck for this topic. Check its leader ID.
Stop the broker (just the service, not the instance).
Execute Preferred Replica Election (in yahoo/kafka-manager or manually).
Start the broker back.
Repeat for the rest of topics that have the same problem.
Also I tried this advice, it didn't help me: https://stackoverflow.com/a/51063607/1929406
I am very new to Kafka. So this question might be very basic.
What i am trying to achieve is to find out the time it takes to rebalance when a broker fails and is then added back.
From my reading up of the documentation(http://kafka.apache.org/documentation/#basic_ops_restarting). When a broker fails or is taken down for maintenance
It will sync all its logs to disk to avoid needing to do any log recovery when it restarts (i.e. validating the checksum for all messages in the tail of the log). Log recovery takes time so this speeds up intentional restarts.
It will migrate any partitions the server is the leader for to other replicas prior to shutting down. This will make the leadership transfer faster and minimize the time each partition is unavailable to a few milliseconds.
What i want to do is find out the time taken to migrate any partitions that the server is the leader for to other replicas
My kafka setup is 3 broker nodes and 3 zk nodes.
Also, when i add this node back to the property of auto.rebalance=true the rebalance again kicks in, and it re-elects a leader.
How do i measure this time as well?
There is no "migration" as in data copy. When shutting down a broker cleanly, the controller will simply elect a new leader from the available replicas for all partitions the broker was the leader, making the transition fast.
There are a few metrics you can monitor the leader elections.
Since 0.11.0.0, the broker exposes a number of Controller metrics including:
kafka.controller:type=ControllerStats,name=AutoLeaderBalanceRateAndTimeMs
This tracks the rate and duration of auto leader rebalance. The full list of controller metrics that were added in 0.11 is available in the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-143%3A+Controller+Health+Metrics#KIP-143:ControllerHealthMetrics-ControllerMetrics
If you are running an older version (< 0.11.0.0), you'll have to rely on metrics like:
kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs
This include any leader elections.