I am going through the documentation and didn't find a way apart from using "committed" method to fetch the current committed offset per partition of this current consumer , consumer group.
Is there a simple way , command to find out the same committed offset of a consumer ?
You can use the Kafka tool as described in the documentation checking consumer position
> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Related
I want to delete all messages that are contained in the __consumer_offsets table that start with a given key (resetting one particular consumer group without affecting the rest).
Is there a way to do this?
Kafka comes with a ConsumerGroupCommand tool. You cand find some information in the Kafka documentation.
If you plan to reset a particular Consumer Group ("myConsumerGroup") without affecting the rest you can use
> bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --reset-offsets --group myConsumerGroup --topic topic1 --to-latest
Depending on your requirement you can reset the offsets for each partition of the topic with that tool. The help function or documentation explain the options.
I have a logstash kafka consumer group subscribing nearly 20topics which isn't performing well for a particular high priority kafka topic to process, so I've decided to remove one topic out of the consumer group and launch a separate consumer group for high priority topics, but unfortunately with that i'm losing the offset that was there in old consumer group.
Is there anyway I can start the new logstash consumer group with the initial offset from last consumer group?
Thanks
You can use kafka scripts to set offset for new group.
Sample scenario:
Stop application.
Check current offset for group. You can use following command. The output will contains information about current offset, log end offset, lag, etc for each topic.
./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group groupId --describe
Set offset for new group id, that will be used by new application (suppose the offset for topic you would like to switch is 10001)
./bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group newGroupId --to-offset 10001 --topic topicName --reset-offsets --execute
Remove topicName from topics list for old application.
Set up new logstash configuration with newGroupId group id.
Start old and new logstash applications.
I am using auto.offset.reset=earliest in my code and used offset commit in kafka with the help of below code.
val offsetRanges=rdd.asInstanceOf[HasOffsetRanges].offsetRanges
inputStream.asInstanceOf[CanCommitOffsets].commitAsync(offsetRanges)
Now when I run my program it does not get new messages as all messages are committed.
I am testing this code in QA so want to reset the offset to beginning but looks like earliest is not working it is not reading new messages and there are no new messages in the topic. I want to read messages from beginning for testing purpose.
Can someone assist if earliest does not fetch messages from beginning if it is committed?
The property auto.offset.reset is used only if there's no committed offset for the partition. You can reset the offset for the whole group using kafka-consumer-groups (comes as part of Kafka):
kafka-consumer-groups --bootstrap-server <kafkahost:port> --group <group_id> --topic <topic_name> --reset-offsets --to-earliest --execute
I try to shift back the offset of topic within consumer group using following command:
bin/kafka-consumer-groups.sh --bootstrap-server loclahost:9092 --group xxx-0 --topic schedule-changed --reset-offsets --shift-by -2 --execute
(Yes, I'm using kafka version > 1.x) As a result I got a message:
Error: Assignments can only be reset if the group 'xxx-0' is
inactive, but the current state is Stable.
How can I change the state of group from 'stable' to 'inactive'?
The reset-offsets option for kafka-consumer-groups.sh first checks to see if a consumer is active in that group before attempting to shift back your offsets. 'Stable' means you have an active consumer running.
Use the describe-groups option to check on your consumer groups:
bin/kafka-consumer-groups.sh --bootstrap-server $SERVERS --group $GROUP --describe
If you see an entry under 'CONSUMER-ID/HOST/CLIENT-ID' for your topic, that means you still have a consumer running. Once you shut down the app that's keeping the consumer alive your consumer group will be inactive and you'll be able to shift your offsets at will.
I had the same problem, but in contrast to kellanburket's answer there was no consumer running any more. In that case, I had to delete the consumer group:
kafka-consumer-groups --zookeeper a.zookeeper.host:2181 --group the-group-name --delete
In newer versions of kafka, you may need to use:
kafka-consumer-groups --bootstrap-server $SERVERS --group the-group-name --delete
I am using a kafka version where the offset storage is kafka i.e._consumer_offsets
How to retrieve the consumer offset ,when the consumer is down or inactive?
Below Command work but with a flaw that it reads from beginning not the latest one
kafka-console-consumer --consumer.config /tmp/consumer.config --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --zookeeper <> --topic __consumer_offsets --from-beginning
If your consumer is down and you want last committed offset of the topic for the specific group please refer to the below mentioned sources
https://github.com/yahoo/kafka-manager
KafkaConsumerOffsets