kafka-consumer-groups.sh --bootstrap-server hostname:port --describe --group sub1
Consumer group 'sub1' has no active members.
kafka-consumer-groups.sh --bootstrap-server hostname:port --delete --group sub1
Option '[delete]' is only valid with '[zookeeper]'.
Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.
Also when i try to display my consumer details using zookeeper , It tells consumer "sub1" not available.
kafka-consumer-groups.sh --zookeeper hostname:port --describe --group sub1
Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).
Error: The consumer group 'sub1' does not exist.
Group information for consumers that use Kafka to manage offsets instead of Zookeeper cannot be deleted with built-in tools. If you read the whole warning when trying to execute
kafka-consumer-groups.sh --bootstrap-server hostname:port --delete --group sub1
it clearly mentions why group metadata info cannot be deleted:
Option '[delete]' is only valid with '[zookeeper]'.
Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.
Related
When I run two instances of Kafka-console-consumers with the exact same properties (using the default one config/consumer.properties), I get same messages on both the instances.
./bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic test1
If both the instances have the same consumer group id, shouldn't Kafka send a given message to only one of the consumers? How to run them as one consumer group?
From kafka docs i found this
The default for console consumer's enable.auto.commit property when no group.id is provided is now set to false. This is to avoid polluting the consumer coordinator cache as the auto-generated group is not likely to be used by other consumers.
But here is the trick, use this command to list all consumer groups across all topics, as you said i have opened four console consumers and i want to check list of consumer groups consuming from that topic
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
Every console consumer start with different group id, this is the reason always consuming from beginning addition of this property (--from-beginning)
ups.sh --bootstrap-server localhost:9092 --list
Note: This will not show information about old Zookeeper-based consumers.
console-consumer-66835
console-consumer-38647
console-consumer-18983
console-consumer-18365
console-consumer-96734
Okay easiest way to set group.id for console consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=test1
Read up Managing Consumer Groups.
The trick is to use --consumer.config config/consumer.properties or --consumer-property group.id=test1 that would specify the group.id explicitly.
./bin/kafka-console-consumer.sh \
--bootstrap-server localhost:9092 \
--topic test1 \
--consumer.config config/consumer.properties
I try to shift back the offset of topic within consumer group using following command:
bin/kafka-consumer-groups.sh --bootstrap-server loclahost:9092 --group xxx-0 --topic schedule-changed --reset-offsets --shift-by -2 --execute
(Yes, I'm using kafka version > 1.x) As a result I got a message:
Error: Assignments can only be reset if the group 'xxx-0' is
inactive, but the current state is Stable.
How can I change the state of group from 'stable' to 'inactive'?
The reset-offsets option for kafka-consumer-groups.sh first checks to see if a consumer is active in that group before attempting to shift back your offsets. 'Stable' means you have an active consumer running.
Use the describe-groups option to check on your consumer groups:
bin/kafka-consumer-groups.sh --bootstrap-server $SERVERS --group $GROUP --describe
If you see an entry under 'CONSUMER-ID/HOST/CLIENT-ID' for your topic, that means you still have a consumer running. Once you shut down the app that's keeping the consumer alive your consumer group will be inactive and you'll be able to shift your offsets at will.
I had the same problem, but in contrast to kellanburket's answer there was no consumer running any more. In that case, I had to delete the consumer group:
kafka-consumer-groups --zookeeper a.zookeeper.host:2181 --group the-group-name --delete
In newer versions of kafka, you may need to use:
kafka-consumer-groups --bootstrap-server $SERVERS --group the-group-name --delete
I am playing aroung Kafka, when I use
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic test
Kafka will automatically create a consumer group. I am wondering how to get the consumer group name?
You should use kafka-consumer-groups.sh. The following command will list you all consumer groups.
bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
Note: This will only show information about consumers that use the Java Consumer API (non-Zookeeper-based consumers).
I want to delete a Kakfa consumer group so that when the application creates a consumer and subscribes to a topic it can start at the beginning of the topic data.
This is with a single node development vm using the current latest Confluent Platform 3.1.2 which uses Kafka 0.10.1.1.
I try the normal syntax:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --delete --group my_consumer_group
I get the error:
Option [delete] is only valid with [zookeeper]. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.
If I try the zookeeper variant:
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --delete --group my_consumer_group
I get:
Delete for group my_consumer_group failed because group does not exist.
If I list using the "old" consumer, I do not see my consumer group (or any other consumer groups)
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --list
If I list using the "new" consumer, I can see my consumer group but apparently I can't delete it:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --list
This can be done with Kafka 1.1.x. From the documentation:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group
In Kafka 0.11 (or Confluent 3.3) you can reset the offsets of any existing consumer group without having to delete the topic. In fact you can change the offsets to any absolute offset value or timestamp or any relative position as well.
These new functions are all added with the new --reset-offsets flag on the kafka-consumer-groups command line tool.
See KIP-122 details here https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
Upgrading to the just released Confluent Platform 3.2 with Kafka 0.10.2 solved my underlying issue. When I delete a topic, offset information is now correctly reset. So when I create a topic with the same name, consumers start from the beginning of the new data.
I still can't delete new style consumer groups with the kafka-consumer-groups tool, but my underlying issue is solved.
Before Kafka 0.10.2, there were hacks, but no clean solution to this issue.
If you use Java client, you can first get the beginning offset.
TopicPartition partition = new TopicPartition("YOUR_TOPIC", YOUR_PARTITION);
Map<TopicPartition, Long> map = consumer.beginningOffsets(Collections.singleton(partition));
And the offset that consumer using to start processing, (if not delete the consumer group).
Long committedOffset = consumer.committed(partition).offset();
Now, if you think start from committedOffset is ok, just poll records.
if you want the beginning offset,
consumer.seek(partition, map.get(partition));
you can also reset the offset of single topic without deleting the entire consumer group :
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --topic my-topic --reset-offsets --to-earliest --execute
If you're using Windows and you need a JAAS config with password & username to access your kafka cluster, you can use the following commands
#consumer.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="XXXXXXXXXXXXXXX" password="XXXXXXXXXXXXXXXXXXX";
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
.\kafka\bin\windows\kafka-consumer-groups.bat --bootstrap-server xxxxxxxxxx.confluent.cloud:9092 --group cop-group --topic topic_name_ini --reset-offsets --to-earliest --execute --command-config .\kafka\config\consumer.properties
In the command above we used --command-config to pass a properties file for the JAAS config
when i use kafka0.8.x，i can get all group list with zookeeper by path /consumers/group_id, because offsets will commit to zookeeper in 0.8.x.
But in version of 0.10.x, offsets commited to kafka broker like a path /brokers/topics/__consumer_offsets and path /consumers in zookeeper has nothing. so I can not get consumer group list by zookeeper.
To list all consumer groups across all topics:
bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --list
To view offsets for the consumer group:
bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group
See Managing Consumer Groups for detailed information.