I am not able to get any solution to print Kafka consumer Lag.
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group test-group --zookeeper localhost:2181 --topic test01
Group Topic Pid Offset logSize Lag Owner
test-group test01 0 7 9 2 test-group_Jitendra-E5530-1497519128391-bdea2e0c-0
Related
I have Kafka setup on my local machine and have started the zookeeper and a single broker server.
Now i have a single topic with following description:
~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-topics.sh --zookeeper 127.0.0.1:2181 --topic edu-topic --describe
Topic:edu-topic PartitionCount:3 ReplicationFactor:1 Configs:
Topic: edu-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: edu-topic Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: edu-topic Partition: 2 Leader: 0 Replicas: 0 Isr: 0
I have a producer which have produced some message before the consumer was started as follows:
~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic edu-topic
>book
>pen
>pencil
>marker
>
and when i started the consumer with --from-beginning option, it does not shows all the messages produced by the producer:
~/Documents/backups/kafka_2.12-2.2.0/data/kafka$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group edu-service --from-beginning
However, it is showing the newly added messages.
What's wrong i am doing here? Any help?
--from-beginning: If the consumer does not already have an established offset to consume from, start with the earliest message
present in the log rather than the latest message.
Kafka consumer uses --from-beginning very first time if you retry which I suspect you did, it will start from where it left. You can consume the message again with any of the below options
reset consumer group offset using below
kafka-streams-application-reset.sh --application-id edu-service
--input-topics edu-topic --bootstrap-servers localhost:9092 --zookeeper 127.0.0.1:2181
then retry again from the beginning
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group edu-service --from-beginning
Use new consumer id which will start consuming from staring points
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic edu-topic --group new-edu-service --from-beginning
You can also use an offset instead to consume the next N messages from a partition
kafka-console-consumer.sh --bootstrap-server localhost:9092 --offset 0 --partition 0 --topic edu-topic
--offset <String: consume offset> : The offset id to consume from (a non- negative number), or 'earliest' which means from beginning, or
'latest' which means from end (default: latest)
--partition <Integer: partition> : The partition to consume from Consumption starts from the end of the partition unless '--offset'
is specified.
Because you are using the old consumer group. --from-beginning only works for the new consumer group which its group name has not been recorded on the Kafka cluster yet.
To re-consume again from the start, either you can:
Start a new consumer group (change the group name) with the flag --from-beginning
Reset the offsets of this consumer group. I haven't tried yet but you can test it here
The flag
--from-begining
will affect the behavior of your GroupConsumer the first time it is started/created , or the stored (last commited consuming) offset is expired (or maybe when you try to reset the stored offset).
Otherwise the GroupConsumer will just continue at the stored (last commited) offset.
Please consider get more message from manual.
Just add
--from-beginning
But do know that messages from the beginning would not be in order
if you have used multiple partitions for the same topic.
Order is only Guaranteed at the partition level. (for the same partition)
I created the new topic 'rahul' with the following command :
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rahul
Created topic "rahul".
I also re-checked the topics with
bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
rahhy
rahul`
Now starting the producer:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rahul
hey
hi
hello
But when the time comes to consumer to show the messages: there is nothing
As of Kafka 0.9, you don't use Zookeeper for consumption or production
Try kafka-console-consumer --topic rahul --bootstrap-server localhost:9092
There are other ways you can check messages were sent to Kafka - by checking that the offsets of the topic partitions have changed using GetOffsetShell
I'm trying to reset consumer offset with latest CLI tools for Kafka.
kafka-consumer-groups.bat --bootstrap-server kafka-host:9092 --group my-group --reset-offsets --to-earliest --all-topics
As a result I see this output:
TOPIC PARTITION NEW-OFFSET
FirstTopic 0 0
SecondTopic 0 0
But running again command:
kafka-consumer-groups.bat --bootstrap-server kafka-host:9092 --group my-group --describe
results in output:
Consumer group 'my-group' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
FirstTopic 0 1230 1230 0
SecondTopic 0 1022 1022 0
I've tried other options like resetting to explicit offset or specifying the topic directly but result is same. The output suggests that operation succeed while checking offsets with describe command or debugging shows that offset has not be changed.
Anyone succeed with resetting consumer offset within non-zookeeper brokers.
By default, --reset-offsets just prints the result of the operation. To actually perform the operation you need to add --execute to your command:
kafka-consumer-groups.bat --bootstrap-server kafka-host:9092 --group
my-group --reset-offsets --to-earliest --all-topics --execute
Though the accepted answer perfectly answers OP question, there are more parameters available to reset offsets. So adding this answer to extend accepted answer.
To reset offset of all topics to earliest in the consumer group
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
<group_name> --reset-offsets --to-earliest --all-topics --execute
To reset offset of specific topic to earliest in the consumer group
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
<group_name> --reset-offsets --to-earliest --topic <my-topic> --execute
To reset offset of specific topic to specific offset in the consumer group
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
<group_name> --reset-offsets --to-offset 1000 --topic <my-topic> --execute
Other supported arguments:
--shift-by [positive or negative integer] - Shifts offset forward or backward from given integer.
--to-current and --to-latest are same as --to-offset and --to-earliest.
--to-datetime [Datetime format is yyyy-MM-ddTHH:mm:ss.xxx]
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
<group_name> --reset-offsets --to-datetime 2017-08-04T00:00:00.000 [ --all-topics or --topic <topic-name> ] --execute
--by-duration [Format is PnDTnHnMnS]
kafka-consumer-groups.sh --bootstrap-server <kafka_broker_host:9091> --group
<group_name> --reset-offsets --by-duration PT0H10M0S [ --all-topics or --topic <topic-name> ] --execute
Reset to offset by duration from current timestamp.
How to validate?
Use below command to check current/end of offsets and to confirm reset made the cahnges.
kafka-consumer-groups.sh --bootstrap-server <kafka_host:port> --group <group_id> --describe
Sample output:
Consumer group 'group1' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
intro 0 0 99 99 - - -
Possible Errors:
Error: Assignments can only be reset if the group '[group_name]' is inactive, but the current state is Stable.
'Stable' means, there is an active consumer running for this group. So first you have to stop the active consumer(s) and retry resetting offsets.
It is not possible to reset offsets if there is an active consumer for the consumer group.
kafka-consumer-groups.sh --bootstrap-server kafka-host:9092 --group my-group --reset-offsets --to-latest --all-topics --execute
Sh
./kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
https://stackoverflow.com/a/41645130/5368856
https://www.hadoopinrealworld.com/how-to-change-or-reset-consumer-offset-in-kafka/#:~:text=topic%20sales_topic%20%2D%2Dexecute-,%E2%80%93to%2Dlatest,-Reset%20offsets%20to
Describe the Kafka consumer group.
cd /home/USER/kafka_2.11-1.0.0/bin
./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group GROUPNAME
https://docs.cloudera.com/runtime/7.2.10/kafka-managing/topics/kafka-manage-cli-cgroups.html
For people who can't figure out what kafka-consumer-groups.sh is or are using kt client
kt group -reset oldest -topic <YOUR_TOPIC_NAME> -group <YOUR_CONSUMER_GROUP_NAME> -partitions all -brokers <YOUR_BROKER>
You can also view its help manual for customization
$ kt -help
kt is a tool for Kafka.
Usage:
kt command [arguments]
The commands are:
consume consume messages.
produce produce messages.
topic topic information.
group consumer group information and modification.
admin basic cluster administration.
Use "kt [command] -help" for for information about the command.
Authentication:
Authentication with Kafka can be configured via a JSON file.
You can set the file name via an "-auth" flag to each command or
set it via the environment variable KT_AUTH.
From 0.8.1.1 release, Kafka provides the provision for storage of offsets in Kafka, instead of Zookeeper (see this).
I'm not able to figure out how to check the details of offsets consumed, as the current tools only provide consumer offset count checks for zookeeper only.(I'm referring to this)
If there are any tools available to check consumer offset, please let me know.
I'am using kafka 0.8.2 with offsets stored in kafka. This tools works good for me:
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
--topic your-topic
--group your-consumer-group
--zookeeper localhost:2181
You get all informations you need: topic size, consumer lag, owner.
The following straight command gives the enough details:
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-second-application
You will get the details like this
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
first_topic 0 4 4 0 consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53 consumer-1
first_topic 1 3 3 0 consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53 consumer-1
first_topic 2 3 3 0 consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53 consumer-1
first-topic 0 4 4 0 - - -
I'm using Kafka 2.1 and I use kafka-consumer-groups command which gives useful details like current offset, log-end offset, lag, etc. The simplest command syntax is
kafka-consumer-groups.sh \
--bootstrap-server localhost:29092 \
--describe --group <consumer group name>
And the sample output looks like this
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
your.topic 1 17721650 17721673 23 consumer-159-beb9050b /1.2.3.4 consumer-159
your.topic 3 17718700 17718719 19 consumer-159-beb9050b /1.2.3.4 consumer-159
your.topic 0 17721700 17721717 17 consumer-159-beb9050b /1.2.3.4 consumer-159
HTH
I'm looking for a Kafka command that shows all of the topics and offsets of partitions. If it's dynamically would be perfect. Right now I'm using java code to see these information, but it's very inconvenient.
Kafka ships with some tools you can use to accomplish this.
List topics:
# ./bin/kafka-topics.sh --list --zookeeper localhost:2181
test_topic_1
test_topic_2
...
List partitions and offsets:
# ./bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --broker-info --group test_group --topic test_topic --zookeeper localhost:2181
Group Topic Pid Offset logSize Lag Owner
test_group test_topic 0 698020 698021 1 test_group-0
test_group test_topic 1 235699 235699 0 test_group-1
test_group test_topic 2 117189 117189 0 test_group-2
Update for 0.9 (and higher) consumer APIs
If you're using the new apis, there's a new tool you can use: kafka-consumer-groups.sh.
./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group count_errors --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
count_errors logs 2 2908278 2908278 0 consumer-1_/10.8.0.55
count_errors logs 3 2907501 2907501 0 consumer-1_/10.8.0.43
count_errors logs 4 2907541 2907541 0 consumer-1_/10.8.0.177
count_errors logs 1 2907499 2907499 0 consumer-1_/10.8.0.115
count_errors logs 0 2907469 2907469 0 consumer-1_/10.8.0.126
You might want to try kt. It's also quite faster than the bundled kafka-topics.
This is the current most complete info description you can get out of a topic with kt:
kt topic -brokers localhost:9092 -filter my_topic_name -partitions -leaders -replicas
It also outputs as JSON, so you can pipe it to jq for further flexibility.
If anyone is interested, you can have the the offset information for all the consumer groups with the following command:
kafka-consumer-groups --bootstrap-server localhost:9092 --all-groups --describe
The parameter --all-groups is available from Kafka 2.4.0
We're using Kafka 2.11 and make use of this tool - kafka-consumer-groups.
$ rpm -qf /bin/kafka-consumer-groups
confluent-kafka-2.11-1.1.1-1.noarch
For example:
$ kafka-consumer-groups --describe --group logstash | grep -E "TOPIC|filebeat"
Note: This will not show information about old Zookeeper-based consumers.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
beats_filebeat 0 20003914484 20003914888 404 logstash-0-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /192.168.1.1 logstash-0
beats_filebeat 1 19992522286 19992522709 423 logstash-0-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /192.168.1.1 logstash-0
beats_filebeat 2 19990597254 19990597637 383 logstash-0-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX /192.168.1.1 logstash-0
beats_filebeat 7 19991718707 19991719268 561 logstash-0-YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY /192.168.1.2 logstash-0
beats_filebeat 8 20015611981 20015612509 528 logstash-0-YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY /192.168.1.2 logstash-0
beats_filebeat 5 19990536340 19990541331 4991 logstash-0-ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ /192.168.1.3 logstash-0
beats_filebeat 6 19990728038 19990733086 5048 logstash-0-ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ /192.168.1.3 logstash-0
beats_filebeat 3 19994613945 19994616297 2352 logstash-0-AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA /192.168.1.4 logstash-0
beats_filebeat 4 19990681602 19990684038 2436 logstash-0-AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA /192.168.1.4 logstash-0
Random Tip
NOTE: We use an alias that overloads kafka-consumer-groups like so in our /etc/profile.d/kafka.sh:
alias kafka-consumer-groups="KAFKA_JVM_PERFORMANCE_OPTS=\"-Djava.security.auth.login.config=$HOME/.kafka_client_jaas.conf\" kafka-consumer-groups --bootstrap-server ${KAFKA_HOSTS} --command-config /etc/kafka/security-enabler.properties"