I set up a Kafka standalone node(HDF ver 3.1.1, Kafka ver 1.0.0)
To load data from kafka, one can create consumer group using kafka-consumer-groups.sh
The command is like the following:
kafka-consumer-groups.sh --bootstrap-server ip-xxx-xxx-xxx-xxx:6667 (--new-consumer) --describe --group logstash
But it results in the following error:
Note: This will not show information about old Zookeeper-based consumers.
Error: Executing consumer group command failed due to The consumer group command timed out while waiting for group to initialize:
How can i solve this problem?
The following will create a consumer under group-1 (and create the group if it doesn't exist already):
kafka-console-consumer --bootstrap-server localhost:9092 --topic topic --from-beginning--consumer-property group.id=group-1
Also refer the following question:
how to set group name when consuming messages in kafka using command line
Related
I executed
"kafka-consumer-groups --bootstrap-server localhost:9092 --list"
and this results in one group being displayed: console-consumer-961
I then tried to delete this group:
kafka-consumer-groups --bootstrap-server localhost:9092 --delete --group console-consumer-961
But this results in an exception:
Error: Deletion of some consumer groups failed:
* Group 'console-consumer-961' could not be deleted due to: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.GroupNotEmptyException: The group is not empty.
I guess this group was created when I ran kafka-console-consumer.bat, but now this consumer is not running. How can I delete this consumer group?
You can validate the state of a consumer group by using the kafka-consumer-groups command line tool.
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group console-consumer-961 --state
This will show you the state of the consumer. If it is not empty, Kafka will not allow to delete that group. If it still shows that your console consumer is running you need to make sure to properly shut it down (usually by CTRL+C)
When you delete a topic, offset information is now correctly reset. So when you create a topic with the same name, consumers start from the beginning of the new data.
you still can't delete new style consumer groups with the Kafka-consumer-groups tool, but your underlying issue is solved.
Before Kafka 0.10.2, there were hacks, but no clean solution to this issue.
I have been trying to run the following scripts programmatically in java and for the most part they work but for some this error keeps showing up:
"the consumer-group xyz does not exist". Could someone explain why this error is occurring especially if the consumer-group appears in the --list script.
kafka/bin/kafka-consumer-groups.sh --zookeeper
192.xxx.x.xxx:2181 --describe -group TEST_CONSUMER
Returns the error:
Error: The consumer group 'TEST_CONSUMER' does not exist.
But this script will list the consumer group as active:
kafka/bin/kafka-consumer-groups.sh --zookeeper 192.xxx.x.xxx:2181 --list
So why does it if listed, then return an error stating it does not exist? Also related to consumer-groups -- I run this script to get lag info as I was told it is not possible to get this via jmx metrics in kafka 1.0 unless I connect to each consumer server or something. Could someone verify this?
Thanks
Use bootstrap-server not zookeeper here
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
First list the groups in cluster:
bin/kafka-consumer-groups.sh --bootstrap-server $IP_Address:9092 --list
then describe the group you need:
bin/kafka-consumer-groups.sh --bootstrap-server $IP_Address:9092 --group $GROUPID --describe
In case of this not working, I guess you are describing a group which is in ZK(akka old consumer), that's the old kafkaconsumer which stored in ZK. Which version of your kafka consumer client?
I used kafka version 2.10-0.9.0.1, all information goes to zookeeper so i used the following command to checke kafka Lag:
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
group-name --topic topicName --zookeeper zooIp:2181
Now we used latest kafka version 2.11-0.11.0.0, here this command not works.
How we see the kafka lag of topics.
The offsets are not stored anymore in Zookeeper but directly in Kafka now.
You can use the kafka-consumer-groups tool (kafka.admin.ConsumerGroupCommand) to retrieve the position/lag of a group.
bin/kafka-consumer-groups.sh --bootstrap-server KAFKA_HOST:KAFKA_PORT --describe --group GROUP_NAME
I am playing aroung Kafka, when I use
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic test
Kafka will automatically create a consumer group. I am wondering how to get the consumer group name?
You should use kafka-consumer-groups.sh. The following command will list you all consumer groups.
bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
Note: This will only show information about consumers that use the Java Consumer API (non-Zookeeper-based consumers).
I want to delete a Kakfa consumer group so that when the application creates a consumer and subscribes to a topic it can start at the beginning of the topic data.
This is with a single node development vm using the current latest Confluent Platform 3.1.2 which uses Kafka 0.10.1.1.
I try the normal syntax:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --delete --group my_consumer_group
I get the error:
Option [delete] is only valid with [zookeeper]. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.
If I try the zookeeper variant:
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --delete --group my_consumer_group
I get:
Delete for group my_consumer_group failed because group does not exist.
If I list using the "old" consumer, I do not see my consumer group (or any other consumer groups)
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --list
If I list using the "new" consumer, I can see my consumer group but apparently I can't delete it:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --list
This can be done with Kafka 1.1.x. From the documentation:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group
In Kafka 0.11 (or Confluent 3.3) you can reset the offsets of any existing consumer group without having to delete the topic. In fact you can change the offsets to any absolute offset value or timestamp or any relative position as well.
These new functions are all added with the new --reset-offsets flag on the kafka-consumer-groups command line tool.
See KIP-122 details here https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
Upgrading to the just released Confluent Platform 3.2 with Kafka 0.10.2 solved my underlying issue. When I delete a topic, offset information is now correctly reset. So when I create a topic with the same name, consumers start from the beginning of the new data.
I still can't delete new style consumer groups with the kafka-consumer-groups tool, but my underlying issue is solved.
Before Kafka 0.10.2, there were hacks, but no clean solution to this issue.
If you use Java client, you can first get the beginning offset.
TopicPartition partition = new TopicPartition("YOUR_TOPIC", YOUR_PARTITION);
Map<TopicPartition, Long> map = consumer.beginningOffsets(Collections.singleton(partition));
And the offset that consumer using to start processing, (if not delete the consumer group).
Long committedOffset = consumer.committed(partition).offset();
Now, if you think start from committedOffset is ok, just poll records.
if you want the beginning offset,
consumer.seek(partition, map.get(partition));
you can also reset the offset of single topic without deleting the entire consumer group :
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --topic my-topic --reset-offsets --to-earliest --execute
If you're using Windows and you need a JAAS config with password & username to access your kafka cluster, you can use the following commands
#consumer.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="XXXXXXXXXXXXXXX" password="XXXXXXXXXXXXXXXXXXX";
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
.\kafka\bin\windows\kafka-consumer-groups.bat --bootstrap-server xxxxxxxxxx.confluent.cloud:9092 --group cop-group --topic topic_name_ini --reset-offsets --to-earliest --execute --command-config .\kafka\config\consumer.properties
In the command above we used --command-config to pass a properties file for the JAAS config