I used kafka version 2.10-0.9.0.1, all information goes to zookeeper so i used the following command to checke kafka Lag:
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
group-name --topic topicName --zookeeper zooIp:2181
Now we used latest kafka version 2.11-0.11.0.0, here this command not works.
How we see the kafka lag of topics.
The offsets are not stored anymore in Zookeeper but directly in Kafka now.
You can use the kafka-consumer-groups tool (kafka.admin.ConsumerGroupCommand) to retrieve the position/lag of a group.
bin/kafka-consumer-groups.sh --bootstrap-server KAFKA_HOST:KAFKA_PORT --describe --group GROUP_NAME
Related
I am playing aroung Kafka, when I use
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic test
Kafka will automatically create a consumer group. I am wondering how to get the consumer group name?
You should use kafka-consumer-groups.sh. The following command will list you all consumer groups.
bin/kafka-consumer-groups.sh --list --bootstrap-server localhost:9092
Note: This will only show information about consumers that use the Java Consumer API (non-Zookeeper-based consumers).
I want to delete a Kakfa consumer group so that when the application creates a consumer and subscribes to a topic it can start at the beginning of the topic data.
This is with a single node development vm using the current latest Confluent Platform 3.1.2 which uses Kafka 0.10.1.1.
I try the normal syntax:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --delete --group my_consumer_group
I get the error:
Option [delete] is only valid with [zookeeper]. Note that there's no need to delete group metadata for the new consumer as the group is deleted when the last committed offset for that group expires.
If I try the zookeeper variant:
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --delete --group my_consumer_group
I get:
Delete for group my_consumer_group failed because group does not exist.
If I list using the "old" consumer, I do not see my consumer group (or any other consumer groups)
sudo /usr/bin/kafka-consumer-groups --zookeeper localhost:2181 --list
If I list using the "new" consumer, I can see my consumer group but apparently I can't delete it:
sudo /usr/bin/kafka-consumer-groups --new-consumer --bootstrap-server localhost:9092 --list
This can be done with Kafka 1.1.x. From the documentation:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-group --group my-other-group
In Kafka 0.11 (or Confluent 3.3) you can reset the offsets of any existing consumer group without having to delete the topic. In fact you can change the offsets to any absolute offset value or timestamp or any relative position as well.
These new functions are all added with the new --reset-offsets flag on the kafka-consumer-groups command line tool.
See KIP-122 details here https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
Upgrading to the just released Confluent Platform 3.2 with Kafka 0.10.2 solved my underlying issue. When I delete a topic, offset information is now correctly reset. So when I create a topic with the same name, consumers start from the beginning of the new data.
I still can't delete new style consumer groups with the kafka-consumer-groups tool, but my underlying issue is solved.
Before Kafka 0.10.2, there were hacks, but no clean solution to this issue.
If you use Java client, you can first get the beginning offset.
TopicPartition partition = new TopicPartition("YOUR_TOPIC", YOUR_PARTITION);
Map<TopicPartition, Long> map = consumer.beginningOffsets(Collections.singleton(partition));
And the offset that consumer using to start processing, (if not delete the consumer group).
Long committedOffset = consumer.committed(partition).offset();
Now, if you think start from committedOffset is ok, just poll records.
if you want the beginning offset,
consumer.seek(partition, map.get(partition));
you can also reset the offset of single topic without deleting the entire consumer group :
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --topic my-topic --reset-offsets --to-earliest --execute
If you're using Windows and you need a JAAS config with password & username to access your kafka cluster, you can use the following commands
#consumer.properties
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="XXXXXXXXXXXXXXX" password="XXXXXXXXXXXXXXXXXXX";
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
.\kafka\bin\windows\kafka-consumer-groups.bat --bootstrap-server xxxxxxxxxx.confluent.cloud:9092 --group cop-group --topic topic_name_ini --reset-offsets --to-earliest --execute --command-config .\kafka\config\consumer.properties
In the command above we used --command-config to pass a properties file for the JAAS config
I use Ubuntu server 16.04 to try using Kafka. For the command to start a producer and a consumer console I use the following.
producer console :
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic
consumer console :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic
but the command above only subscribes to one topic. How can I subscribe to multiple topics?
First you should connect with the option bootstrap-server to the Kafka server itself not the zookeeper server.
For multiple topics you use the whitelist option. This will get interpreted as a regular expression and has to get quoted, see Kafka documentation.
So a correct command would be:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist 'hello-topic|world-topic|another-topic'
Other expressions are also possible, like
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist '.*'
BE AWARE
For convenience we allow the use of ',' instead of '|' to specify a list of topics.
Does not work with Kafka 2.0, perhaps only when mirroring, which I did not try yet.
Hi The actual problem lies in the syntax, from version to verison it keeps on changing.
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic muleesb
would work for you
Reason
For kafka versions above kafka_2.11-2.1.0: ./kafka-console-consumer.sh --bootstrap-server IP:PORT --from-beginning --topic TOPIC
For Kafka versions kafka_2.11_0.9.0.0 and below: /kafka-console-consumer.sh --zookeeper IP:PORT —-topic TOPIC --from-beginning --whitelist TOPIC
Exactly one of whitelist/blacklist/topic is required
As mentioned by Harald, use whitelist / blacklist option to include / exclude a set of topics for consumption.
sh kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --whitelist Hello,World
How do I get, set, or reset the offset of a Kafka Connect connector/task/sink?
I can use the /usr/bin/kafka-consumer-groups tool which runs kafka.admin.ConsumerGroupCommand to see the offsets for all my regular Kafka consumer groups. However, Kafka Connect tasks and groups do not show up with this tool.
Similarly, I can use the zookeeper-shell to connect to Zookeeper and I can see zookeeper entries for regular Kafka consumer groups, but not for Kafka Connect sinks.
As of 0.10.0.0, Connect doesn't provide an API for managing offsets. It's something we want to improve in the future, but not there yet. The ConsumerGroupCommand would be the right tool to manage offsets for Sink connectors. Note that source connector offsets are stored in a special offsets topic for Connect (they aren't like normal Kafka offsets since they are defined by the source system, see offset.storage.topic in the worker configuration docs) and since sink Connectors uses the new consumer, they won't store their offsets in Zookeeper -- all modern clients use native Kafka-based offset storage. The ConsumerGroupCommand can work with these offsets, you just need to pass the --new-consumer option).
You can't set offsets, but you can use kafka-consumer-groups.sh tool to "scroll" the feed forward.
The consumer group of your connector has a name of connect-*CONNECTOR NAME*, but you can double check:
unset JMX_PORT; ./bin/kafka-consumer-groups.sh --bootstrap-server *KAFKA HOSTS* --list
To view current offset:
unset JMX_PORT; ./bin/kafka-consumer-groups.sh --bootstrap-server *KAFKA HOSTS* --group connect-*CONNECTOR NAME* --describe
To move the offset forward:
unset JMX_PORT; ./bin/kafka-console-consumer.sh --bootstrap-server *KAFKA HOSTS* --topic *TOPIC* --max-messages 10000 --consumer-property group.id=connect-*CONNECTOR NAME* > /dev/null
I suppose you can move the offset backward as well by deleting the consumer group first, using --delete flag.
Don't forget to pause and resume your connector via Kafka Connect REST API.
In my case(testing reading files into producer and consume in console, all in local only), I just saw this in producer output:
offset.storage.file.filename=/tmp/connect.offsets
So I wanted to open it but it is binary, with some hardly recognizable characters.
I deleted it(rename it also works), and then I can write into the same file and get the file content from consumer again. You have to restart the console producer to take effect because it attempts to read the offset file, if not there, create a new one, so that the offset is reset.
If you want to reset it without deletion, you can use:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <group-name> --reset-offsets --to-earliest --topic <topic_name>
You can check all group names by:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
and check details of each group:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group <group_name> --describe
In production environment, this offset is managed by zookeeper, so more steps (and caution) is needed. You can refer to this page:
https://metabroadcast.com/blog/resetting-kafka-offsets
https://community.hortonworks.com/articles/81357/manually-resetting-offset-for-a-kafka-topic.html
Steps:
kafka-topics --list --zookeeper localhost:2181
kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic vital_signs --time -1 // -1 for largest, -2 for smallest
set /consumers/{yourConsumerGroup}/offsets/{yourFancyTopic}/{partitionId} {newOffset}
To launch a Kafka consumer I have to run it with the following params.
kafka-console-consumer.bat --zookeeper localhost:2181 --topic MyTopic
Was wondering why it needs to have a zookeeper as a param, if I use the broker param will it not work similar to how the producer is launched. Consumer needs to be aware of the broker/cluster and not the zookeeper location.
.\kafka-console-producer.bat --broker --list localhost:9092 --topic MyTopic
This is on windows, am not sure how its in Unix flavors.
-Chandra
You're invoking the old Kafka Consumer. Old consumer requires zookeeper to co-ordinate from which broker to fetch data.
The new consumer don't requires the zookeeper parameter.
.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic MyTopic --new-consumer
Optionally, you can add --from-beginning argument to read the old records in the topic.