During application planned maintenance activities , there is a need to drain all the messages a in kafka topic.
In MQ , we can monitor the queue depth and start the maintenance activities once all the messages are consumed. In kafka , do we have a similar mechanism to find out if all messages in the topic has been consumed and its safe to shutdown the producer and consumer ?
Using the following command you can monitor the LAG of your consumer group, once the lag is 0 means no more messages in topic to consume
./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group count_errors --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
count_errors logs 2 2908278 2908278 0 consumer-1_/10.8.0.55
count_errors logs 3 2907501 2907501 0 consumer-1_/10.8.0.43
count_errors logs 4 2907541 2907541 0 consumer-1_/10.8.0.177
count_errors logs 1 2907499 2907499 0 consumer-1_/10.8.0.115
count_errors logs 0 2907469 2907469 0 consumer-1_/10.8.0.126
I am new to Kafa and data ingestion. I know Kafka is fault tolerant, as it keeps the data redundantly on multiple nodes. However, what I don't understand is how can we achieve fault tolerance on the source/producer end. For example, If I have netcat as souce, as given in the example below.
nc -l [some_port] | ./bin/kafka-console-producer --broker-list [kafka_server]:9092 --topic [my_topic]
The producer would fail pushing messages if the node doing netcat goes down. I was thinking if there was a mechanism where Kafka can pull the input itself, so for example, if on one node the netcat fails, another node can take over and start pushing messages using netcat.
My second question is how this is achieved in Flume, as it is a pull based architecture. Would Flume work in this case, that is, if one node doing netcat fails?
Every topic, is a particular stream of data (similar to a table in a database). Topics, are split into partitions (as many as you like) where each message within a partition gets an incremental id, known as offset as shown below.
Partition 0:
+---+---+---+-----+
| 0 | 1 | 2 | ... |
+---+---+---+-----+
Partition 1:
+---+---+---+---+----+
| 0 | 1 | 2 | 3 | .. |
+---+---+---+---+----+
Now a Kafka cluster is composed of multiple brokers. Each broker is identified with an ID and can contain certain topic partitions.
Example of 2 topics (each having 3 and 2 partitions respectively):
Broker 1:
+-------------------+
| Topic 1 |
| Partition 0 |
| |
| |
| Topic 2 |
| Partition 1 |
+-------------------+
Broker 2:
+-------------------+
| Topic 1 |
| Partition 2 |
| |
| |
| Topic 2 |
| Partition 0 |
+-------------------+
Broker 3:
+-------------------+
| Topic 1 |
| Partition 1 |
| |
| |
| |
| |
+-------------------+
Note that data is distributed (and Broker 3 doesn't hold any data of topic 2).
Topics, should have a replication-factor > 1 (usually 2 or 3) so that when a broker is down, another one can serve the data of a topic. For instance, assume that we have a topic with 2 partitions with a replication-factor set to 2 as shown below:
Broker 1:
+-------------------+
| Topic 1 |
| Partition 0 |
| |
| |
| |
| |
+-------------------+
Broker 2:
+-------------------+
| Topic 1 |
| Partition 0 |
| |
| |
| Topic 1 |
| Partition 0 |
+-------------------+
Broker 3:
+-------------------+
| Topic 1 |
| Partition 1 |
| |
| |
| |
| |
+-------------------+
Now assume that Broker 2 has failed. Broker 1 and 3 can still serve the data for topic 1. So a replication-factor of 3 is always a good idea since it allows for one broker to be taken down for maintenance purposes and also for another one to be taken down unexpectedly. Therefore, Apache-Kafka offers strong durability and fault tolerance guarantees.
Note about Leaders:
At any time, only one broker can be a leader of a partition and only that leader can receive and serve data for that partition. The remaining brokers will just synchronize the data (in-sync replicas). Also note that when the replication-factor is set to 1, the leader cannot be moved elsewhere when a broker fails. In general, when all replicas of a partition fail or go offline, the leader will automatically be set to -1.
Having said that, as far as your producer lists all the addresses of the Kafka brokers that are in the cluster (bootstrap_servers), you should be fine. Even when a broker is down, your producer will attempt to write the record to another broker.
Finally, make sure to set acks=all (might have an impact to throughput though) so that the all in-sync replicas acknowledge that they received the message.
During the last maintenance of Kafka, which required a rolling restart of kafka brokers, we witnessed a reset for consumer group offsets for certain partitions.
At 11:14 am, everything is fine for the consumer group and we don't see a consumer lag:
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 0 105130857 105130857 0 st-...
...
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 6 78591770 78591770 0 st-...
However 5 minutes later, during the rolling restart of brokers, we have a reset for one partition and a consumer lag of millions of events.
$ bin/kafka-consumer-groups --bootstrap-server XXX:9093,XXX... --command-config secrets.config --group st-xx --describe
Note: This will not show information about old Zookeeper-based consumers.
[2019-08-26 12:44:13,539] WARN Connection to node -5 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-08-26 12:44:13,707] WARN [Consumer clientId=consumer-1, groupId=st-xx] Connection to node -5 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Consumer group 'st-xx' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 0 105132096 105132275 179
...
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 6 15239401 78593165 63353764 ...
In the last two hours, the offset for the partition hasn't recovered and we need to patch it now manually. We had similar issues during the last rolling restart of brokers.
Has anyone seen something like this before? The only clue we could find is this ticket, however we run Kafka version: 1.0.1-kafka3.1.0,
Kafka topic creation is failing in below scenarios:
Node is kafka cluster: 4
Replication factor: 4
Number of nodes up and running in cluster: 3
Below is the error:
./kafka-topics.sh --zookeeper :2181 --create --topic test_1 --partitions 1 --replication-factor 4
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Replication factor: 4 larger than available brokers: 3.
[2018-10-31 11:58:13,084] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 4 larger than available brokers: 3.
Is it a valid behavior or some known issue in kafka?
If all the nodes in a cluster should be up and running always then what about failure tolerance?
upating json file for increasing the replication factor for already created topic:
$cat /tmp/increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"vHost_v81drv4","partition":0,"replicas":[4,1,2,3]},
{"topic":"vHost_v81drv4","partition":1,"replicas":[4,1,2,3]},
{"topic":"vHost_v81drv4","partition":2,"replicas":[4,1,2,3]},
{"topic":"vHost_v81drv4","partition":3,"replicas":[4,1,2,3]}
{"topic":"vHost_v81drv4","partition":4,"replicas":[4,1,2,3]},
{"topic":"vHost_v81drv4","partition":5,"replicas":[4,1,2,3]},
{"topic":"vHost_v81drv4","partition":6,"replicas":[4,1,2,3]},
{"topic":"vHost_v81drv4","partition":7,"replicas":[4,1,2,3]}
]}
When a new topic is created in Kafka, it is replicated N=replication-factor times across your brokers. Since you have 3 brokers up and running and replication-factor set to 4 the topic cannot be replicated 4 times and thus you get an error.
When creating a new topic you either need to ensure that all of your 4 brokers are up and running or set the replication factor to a smaller value in order to avoid failure on topic creation when one of your brokers is down.
In case you want to create topic with replication factor set to 4 while one broker is down, you can initially create the topic with replication-factor=3 and once your 4th broker is up and running you can modify the configuration of that topic and increase its replication factor by following the steps below (assuming you have a topic example with 4 partitions):
Create a increase-replication-factor.json file with this content:
{"version":1,
"partitions":[
{"topic":"example","partition":0,"replicas":[0,1,2,3]},
{"topic":"example","partition":1,"replicas":[0,1,2,3]},
{"topic":"example","partition":2,"replicas":[0,1,2,3]},
{"topic":"example","partition":3,"replicas":[0,1,2,3]}
]}
Then execute the following command:
kafka-reassign-partitions --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
And finally you'd be able to confirm that your topic is replicated across the 4 brokers:
kafka-topics --zookeeper localhost:2181 --topic signals --describe
Topic:signals PartitionCount:4 ReplicationFactor:4 Configs:retention.ms=1000000000
Topic: signals Partition: 0 Leader: 2 Replicas: 0,1,2,3 Isr: 2,0,1,3
Topic: signals Partition: 1 Leader: 2 Replicas: 0,1,2,3 Isr: 2,0,1,3
Topic: signals Partition: 2 Leader: 2 Replicas: 0,1,2,3 Isr: 2,0,1,3
Topic: signals Partition: 3 Leader: 2 Replicas: 0,1,2,3 Isr: 2,0,1,3
Regarding high availability let me explain how Kafka works:
Every topic, is a particular stream of data (similar to a table in a database). Topics, are split into partitions (as many as you like) where each message within a partition gets an incremental id, known as offset as shown below.
Partition 0:
+---+---+---+-----+
| 0 | 1 | 2 | ... |
+---+---+---+-----+
Partition 1:
+---+---+---+---+----+
| 0 | 1 | 2 | 3 | .. |
+---+---+---+---+----+
Now a Kafka cluster is composed of multiple brokers. Each broker is identified with an ID and can contain certain topic partitions.
Example of 2 topics (each having 3 and 2 partitions respectively):
Broker 1:
+-------------------+
| Topic 1 |
| Partition 0 |
| |
| |
| Topic 2 |
| Partition 1 |
+-------------------+
Broker 2:
+-------------------+
| Topic 1 |
| Partition 2 |
| |
| |
| Topic 2 |
| Partition 0 |
+-------------------+
Broker 3:
+-------------------+
| Topic 1 |
| Partition 1 |
| |
| |
| |
| |
+-------------------+
Note that data is distributed (and Broker 3 doesn't hold any data of topic 2).
Topics, should have a replication-factor > 1 (usually 2 or 3) so that when a broker is down, another one can serve the data of a topic. For instance, assume that we have a topic with 2 partitions with a replication-factor set to 2 as shown below:
Broker 1:
+-------------------+
| Topic 1 |
| Partition 0 |
| |
| |
| |
| |
+-------------------+
Broker 2:
+-------------------+
| Topic 1 |
| Partition 0 |
| |
| |
| Topic 1 |
| Partition 1 |
+-------------------+
Broker 3:
+-------------------+
| Topic 1 |
| Partition 1 |
| |
| |
| |
| |
+-------------------+
Now assume that Broker 2 has failed. Broker 1 and 3 can still serve the data for topic 1. So a replication-factor of 3 is always a good idea since it allows for one broker to be taken down for maintenance purposes and also for another one to be taken down unexpectedly. Therefore, Apache-Kafka offers strong durability and fault tolerance guarantees.
Note about Leaders:
At any time, only one broker can be a leader of a partition and only that leader can receive and serve data for that partition. The remaining brokers will just synchronize the data (in-sync replicas). Also note that when the replication-factor is set to 1, the leader cannot be moved elsewhere when a broker fails. In general, when all replicas of a partition fail or go offline, the leader will automatically be set to -1.
This is a valid behavior. When creating a new topic all nodes should be up and running.
Confluence Replica placements - Initial placement
Only create the topic, make decision-based on current live brokers (manual create topic command);
All nodes must not be up and running while using this topic (after it is created)
Apache documentation about replication factor
The replication factor controls how many servers will replicate each message that is written. If you have a replication factor of 3 then up to 2 servers can fail before you will lose access to your data. We recommend you use a replication factor of 2 or 3 so that you can transparently bounce machines without interrupting data consumption.
I am trying to test a configuration of broker offset.retention.minutes=30. I have changed to this config to 10 mins, instead of 24 hours as default.
however after more than 10 mins the consumer group offset still showing offset in information
ldnpsr000001131$ bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe -group rent_test
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
rent_test rent_test 0 44 44 0 none
Any idea why it is not getting deleted?
offsets.retention.minutes controls the log retention window in minutes for offsets topic, namely __consumer_offsets into which new consumers store offsets. In your case where you are using the old consumer, the offsets are stored in the zookeeper, so setting offsets.retention.minutes has no effect on the ZK-based consumer group.