I’m having an issue with using the Kafka command line tools to get consumer lag for a given group/topic. Currently, I'm trying to use kafka-consumer-groups.sh as mentioned in countless online resources. The following command works just fine: kafka-consumer-groups.sh --bootstrap-server $BROKERS --list
However, if I use kafka-consumer-groups.sh --bootstrap-server $BROKERS --describe --group $GROUP, I get the following output and error:
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
[2018-07-24 00:19:09,139] ERROR admin-client-network-thread exited (kafka.admin.AdminClient) java.lang.NullPointerException
at org.apache.kafka.common.utils.Utils.join(Utils.java:399)
at org.apache.kafka.common.requests.OffsetFetchRequest$Builder.toString(OffsetFetchRequest.java:74)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.apache.kafka.clients.ClientRequest.toString(ClientRequest.java:63)
at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:374)
at org.apache.kafka.clients.NetworkClient.send(NetworkClient.java:332)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:409)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:252)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:208)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:199)
at kafka.admin.AdminClient$$anon$1.run(AdminClient.scala:61)
at java.lang.Thread.run(Thread.java:748) Error: Executing consumer group command failed due to The server experienced an unexpected error when processing the request
I've tried setting GROUP to each and every group available and I've used KafkaTool to confirm that these groups exist and are working properly (consuming messages from various topics). I've tried placing strings in the command directly instead of using an environment variable.
Why am I getting this error and what else can I do to debug?
Related
I am using kafka console consumer, and i want to redirect error thrown from kafka process to a log file. Requirement is such that if a process runs without error i redirect output of my topic to a file and am using command like kafka-console-consumer --topic example --group-id mygroup -bootstrap-server broker:9092 --from-beginning --property print.key=true . Now if there is an error while say connecting to topic or permission on say group id , i want that error to be redirected to a file say /tmp/myerrors.txt ( errors thrown by kafka or system error ) but in case there is no issues and i start getting data for the topic i redirect data to another file \home\abcd\data.txt . Is it feasible ? I want it so that error and data files are separate and due to an issue on mid-day my files which is meant for collecting data does not get the error logged into it.
I am using Kafka Command Line tool to consume data from a Kafka topic, but this is not working as expected.
I am using the following command :
kafka-console-consumer.sh --bootstrap-server totkafka-console-consumer.sh --bootstrap-server localhost1:9092,localhost2:9092, localhost3:9092 --topic topic-name
and I am getting the following error stack:
Exception in thread "main" joptsimple.MultipleArgumentsForOptionException: Found multiple arguments for option bootstrap-server, but you asked for only one
at joptsimple.OptionSet.valueOf(OptionSet.java:179)
at kafka.tools.ConsoleConsumer$ConsumerConfig.<init>(ConsoleConsumer.scala:301)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:52)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
May you please help me with this?
Best regards
Syntax eror in your command, try below:
kafka-console-consumer.sh --bootstrap-server localhost1:9092,localhost2:9092,localhost3:9092 --topic topic-name
i'm trying to start a consumer using Apache Kafka, it used to work well, but i had to format my pc and reinstall everything again, and now when trying to run this:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
this is what i'm getting:
Missing required argument "[zookeeper]"
Option Description
------ -----------
--blacklist <blacklist> Blacklist of topics to exclude from
consumption.
--bootstrap-server <server to connect
to>
--consumer.config <config file> Consumer config properties file.
--csv-reporter-enabled If set, the CSV metrics reporter will
be enabled
--delete-consumer-offsets If specified, the consumer path in
zookeeper is deleted when starting up
--formatter <class> The name of a class to use for
formatting kafka messages for
display. (default: kafka.tools.
DefaultMessageFormatter)
--from-beginning If the consumer does not already have
an established offset to consume
from, start with the earliest
message present in the log rather
than the latest message.
--key-deserializer <deserializer for
key>
--max-messages <Integer: num_messages> The maximum number of messages to
consume before exiting. If not set,
consumption is continual.
--metrics-dir <metrics directory> If csv-reporter-enable is set, and
this parameter isset, the csv
metrics will be outputed here
--new-consumer Use the new consumer implementation.
--property <prop>
--skip-message-on-error If there is an error when processing a
message, skip it instead of halt.
--timeout-ms <Integer: timeout_ms> If specified, exit if no message is
available for consumption for the
specified interval.
--topic <topic> The topic id to consume on.
--value-deserializer <deserializer for
values>
--whitelist <whitelist> Whitelist of topics to include for
consumption.
--zookeeper <urls> REQUIRED: The connection string for
the zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.
my guess is that there's some kind of problem with the zookeeper connection port, because it's telling me to specify the port which zookeeper has to use to get connected to kafka. I'm not sure of this though, and don't know how to figure out the port to specify if this was the problem. Any suggestions??
Thanks in advance for the help
It looks like you are using an old version of the Kafka tools that requires to set --new-consumer if you want to directly connect to the brokers.
I'd recommend picking a recent version of Kafka so you only need to specify --bootstrap-server like in your example: http://kafka.apache.org/downloads
I am getting "Error: Executing consumer group command failed due to null" when running the following command
./kafka-consumer-groups.bat --zookeeper 192.168.99.101:2181 --describe --group console-consumer-62252
Googled it but couldn't find any solution
Not sure if it's still relevant, but the new Kafka using bootstrap-server instead of zookeeper, try this one:
./kafka-consumer-groups.sh --bootstrap-server 192.168.99.101:9092 --describe --group console-consumer-62252
Try to use list of bootstrap and see if --new-consumer work
sh kafka-consumer-groups --bootstrap-server broker1host:port1,broker2host:port2... --new-consumer --describe --group console-consumer-62252
or without --new-consumer if you are using kafka 2.8.0+
sh kafka-consumer-groups --bootstrap-server broker1host:port1,broker2host:port2... --describe --group console-consumer-62252
FWIW since this was the first result when I googled Error: Executing consumer group command failed due to null, I thought I'd record that plain old PEBCAK can also cause this error. I got it and puzzled over it for a few minutes before realising I was trying to reset offsets for a consumer group on a topic, and had a typo in the topic name...
Setting up Kafka first time, Kafka 0.11. Using pretty much default configurations. Produced produced some messages to topic ABC. 2 Consumers are coded to consume messages from the same topic. Each consumer belongs to different group id GROUP.1 and GROUP.2
Want to look into the topic for all the messages and also the offset details.
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group GROUP.1
throws following error,
Error: The consumer group 'GROUP.1' does not exist.
Same error for GROUP.2 also. I got some output without error for one of the group yesterday, but not today. What I'm I missing? Need to configure somewhere to persist consumer group details, or will the command work only when the consumers with given group id is currently running, or?
I tried kafka-consumer-groups --zookeeper localhost:2181 --describe --group GROUP.1 but got the same error.
Also tried Kafka-consumer-offset-checker command.
kafka-consumer-offset-checker --zookeeper localhost:2181 --topic ABC --group GROUP.1
[2017-12-19 19:25:01,654] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/GROUP.1/offsets/ABC/2.
As you said you saw the group details yesterday, it's probably worth noting that by default offsets are only stored for 24 hours. So if you group has not committed offsets in 24 hours, Kafka has no more information about it.
If this is indeed the issue, you can increase the time by setting offsets.retention.minutes to a larger value.