When I run the command:
kafka-consumer-groups -bootstrap-server localhost:9092 -list
It produces a list of consumer groups. Here's some lengthy output:
$ kafka-consumer-groups --bootstrap-server localhost:9092 --list
Note: This will not show information about old Zookeeper-based consumers.
simulate_birdfeeder.e67c034c-a96d-11e9-861a-983b8f0e47c8
xfer_server.e65d3732-a96d-11e9-992f-983b8f0e47c8
xfer_server.e654e596-a96d-11e9-992f-983b8f0e47c8
scheduler_node.e6ba65ce-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e695c336-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e681a4c8-a96d-11e9-861a-983b8f0e47c8
scheduler_node.e6b936f4-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e6956c2e-a96d-11e9-861a-983b8f0e47c8
scheduler_node.e6de0222-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e68ec02c-a96d-11e9-861a-983b8f0e47c8
scheduler_node.e6b48f82-a96d-11e9-b125-983b8f0e47c8
scheduler_node.e6b436cc-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e68fed3a-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e658faf0-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e64f28ea-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e691a710-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e6889f30-a96d-11e9-861a-983b8f0e47c8
74b6a2e3-efe6-4a62-ad6b-a4782038db43
simulate_birdfeeder.e67ec7bc-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e67b9178-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e65b4b2a-a96d-11e9-861a-983b8f0e47c8
scheduler_node.e6dcad1e-a96d-11e9-b125-983b8f0e47c8
water_connector
simulate_birdfeeder.e6924904-a96d-11e9-861a-983b8f0e47c8
reporter
xfer_server.e652ee8a-a96d-11e9-992f-983b8f0e47c8
scheduler_node.e68d3446-a96d-11e9-b125-983b8f0e47c8
simulate_flexcharger.e560bef8-a96d-11e9-a249-983b8f0e47c8
511bcf37-7e6b-4a33-9a57-9ad498f9a089
simulator.e5d05a10-a96d-11e9-9375-983b8f0e47c8
trajectory.queue
simulate_birdfeeder.e662497a-a96d-11e9-861a-983b8f0e47c8
scheduler
simulate_birdfeeder.e6874ca2-a96d-11e9-861a-983b8f0e47c8
85f138e2-bb87-4885-98b8-a6536595cf7c
scheduler_node.e69bcd9e-a96d-11e9-b125-983b8f0e47c8
scheduler_node.e69b0c42-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e6612c7a-a96d-11e9-861a-983b8f0e47c8
scheduler_node.e691649e-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e665d02c-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e6554ab8-a96d-11e9-861a-983b8f0e47c8
xfer_server.e65b81ee-a96d-11e9-992f-983b8f0e47c8
server.e6159012-a96d-11e9-bbdf-983b8f0e47c8
simulate_birdfeeder.e666c9b4-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e67dffb2-a96d-11e9-861a-983b8f0e47c8
KMOffsetCache-my-computer
simulate_birdfeeder.e697c230-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e683e0bc-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e65fbd40-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e69706c4-a96d-11e9-861a-983b8f0e47c8
scheduler_node.e69cc438-a96d-11e9-b125-983b8f0e47c8
xfer_server.e64a93de-a96d-11e9-992f-983b8f0e47c8
scheduler_node.e6a55e72-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e685be50-a96d-11e9-861a-983b8f0e47c8
product_monitor
simulate_birdfeeder.e688f89a-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e66b29fa-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e6811242-a96d-11e9-861a-983b8f0e47c8
scheduler_node.e6a62be0-a96d-11e9-b125-983b8f0e47c8
scheduler_node.e6b9fa58-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e68a2986-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e690480c-a96d-11e9-861a-983b8f0e47c8
server.e60fec02-a96d-11e9-bbdf-983b8f0e47c8
scheduler_node.e6a45856-a96d-11e9-b125-983b8f0e47c8
simulate_birdfeeder.e68385ae-a96d-11e9-861a-983b8f0e47c8
server
simulate_birdfeeder.e668c246-a96d-11e9-861a-983b8f0e47c8
simulate_birdfeeder.e65f11a6-a96d-11e9-861a-983b8f0e47c8
xfer_server.e64f2bb0-a96d-11e9-992f-983b8f0e47c8
scheduler_node.e6b8e834-a96d-11e9-b125-983b8f0e47c8
scheduler_node.e6ddb13c-a96d-11e9-b125-983b8f0e47c8
Of which I can describe them:
$ kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group server
Note: This will not show information about old Zookeeper-based consumers.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
reservationist.queue 0 456 456 0 rdkafka-fa9be154-c7ac-471f-a9d1-5bfd3f42da8c /127.0.0.1 rdkafka
However, for some consumer groups, if I describe them, there is no output at all. I'll get something like the following:
$ kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group scheduler_node.e6ddb13c-a96d-11e9-b125-983b8f0e47c8
Note: This will not show information about old Zookeeper-based consumers.
Aside from the warning, there is no output, but the consumer does exist, because it is listed in the former list command. Now, if I use the zookeeper option, the command complains about the group not existing.
Often the consumers that look like widget has a description of offsets, and the ones that look like widget.some-long-random-string have no output. And consumers that look like some-long-random-string also produce a description with offsets.
(I'm not 100% certain, but I'm pretty sure the missing consumer groups are subscribed to, otherwise my application would be bombing, which isn't the case.)
I am using confluent kafka locally. My code and confluent are all running in my dev box:
$ confluent version kafka
This CLI is intended for development only, not for production
https://docs.confluent.io/current/cli/index.html
1.1.1-cp1
Can any give me some insight as to what is going on? Thanks.
kafka-consumer-groups \
--bootstrap-server localhost:9092 \
--describe \
--group your_consumer_group_name
will return no output if the consumers in the consumer group have not committed any offsets.
If the output of
bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group server
and
bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--new-consumer
--describe \
--group server
matches and the consumer group appears in both consumer groups list output, my guess is that your consumers inside the consumer group server have not committed any offsets and this is why no information appears using the describe command.
Make sure that your consumers inside server group commit their offsets succesfully (either manually or automatically).
I am new people on apache Kafka. When I go through quick start instruction via http://kafka.apache.org/quickstart with latest version kafka_2.12-2.2.0. I got a problem and can't figure it out by myself.
The issue is, on my laptop, I created 3 brokers to simulate cluster situation.
Each broker has its owned server property file. I made below change for each server property file and leave other default value as what it is.
broker.id=1 (server2: broker.id=2; server3: broker.id=3)
listeners=PLAINTEXT://127.0.0.1:9092 (server2: 127.0.0.1:9023; server3: 127.0.0.1:9004)
log.dirs=/tmp/kafka-logs (server2: /tmp/kafka-logs-2; server3: /tmp/kafka-logs-3)
num.partitions=3 (for all servers)
offsets.topic.replication.factor=3 (for all servers)
After I started ZK and those 3 brokers, I (can) create a topic 'TestTopic' with 3 partitions on any broker
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic TestTopic
And then I use below command to start 3 consumers in the same group 'rickygroup'.
//consumer one
bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic TestTopic —group.id rickygroup —group.name rickygroup
//consumer two
bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9093 --from-beginning --topic TestTopic —group.id rickygroup —group.name rickygroup
//consumer three
bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9094 --from-beginning --topic TestTopic —group.id rickygroup —group.name rickygroup
Now, I use another terminal to publish some messages on Topic 'TestTopic'. The issue is, all of the above 3 consumers will get all and exactly the same messages. My understanding is 3 consumers should consume all messages indifference instead of the same. Otherwise, the consumer group shows repeated consuming instead of balance consuming.
Is there any misunderstanding on consumer group concept by me? or anything I did wrong here?
The console consumer uses --group (with two dashes), not -group.id and/or -group.name, which are not parsed options.
I'm trying to delete consumer groups in kafka 10.2.1:
./kafka-consumer-groups.sh --zookeeper myzkhost --list
consumer-1
consumer-2
...
consumer-n
if i do:
./kafka-consumer-groups.sh --zookeeper myzkhost --delete --group consumer-1
Note: This will only show information about consumers that use ZooKeeper (not those using the Java consumer API).
Error: Delete for group 'consumer-1' failed because group does not exist.
but when i go to my zookeeper host:
ls /consumers/consumer-1
[offset]
and that happens with all consumers... is it safe to delete the zookeeper path for those consumers?
Thanks!
I have started a consumer in a consumer group using following command
ldnpsr000001131$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic rent_test --property group.id=rent_test auto.commit.enable=true auto.commit.interval.ms=100
as I understand, above command will create a consumer group named rent_test, and commtted offset every 100 ms. However, when I tried to list all of the consumer group, the group "rent_test" is not presented.
ldnpsr000001131$ bin/kafka-consumer-groups.sh --list --zookeeper localhost:2181
console-consumer-68623
console-consumer-18287
console-consumer-45392
test
console-consumer-9009
KafkaMirror-test
console-consumer-25049
kafka-mirror
console-consumer-61946
console-consumer-940
console-consumer-11318
KafkaMirror
console-consumer-43035
console-consumer-99202
consumer-test
console-consumer-42642
console-consumer-19085
console-consumer-7142
KafkaMirror-test-1
console-consumer-82299
console-consumer-81448
console-consumer-26487
console-consumer-71474
flink
console-consumer-4692
Please advise?
If you are using old consumer, do not specify group.id in property. In 0.10.0.1, you have to specify it in a consumer config file and set consumer.config:
bin/kafka-console-consumer.sh --zookeeper zkHost:2181 --topic test-topic --consumer.config <config file path>
I am doing the Kafka Quickstart for Kafka 0.9.0.0.
I have zookeeper listening at localhost:2181 because I ran
bin/zookeeper-server-start.sh config/zookeeper.properties
I have a single broker listening at localhost:9092 because I ran
bin/kafka-server-start.sh config/server.properties
I have a producer posting to topic "test" because I ran
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
yello
is this thing on?
let's try another
gimme more
When I run the old API consumer, it works by running
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
However, when I run the new API consumer, I don't get anything when I run
bin/kafka-console-consumer.sh --new-consumer --topic test --from-beginning \
--bootstrap-server localhost:9092
Is it possible to subscribe to a topic from the console consumer using the new api? How can I fix it?
I my MAC box I was facing the same issue of console-consumer not consuming any messages when used the command
kafka-console-consumer --bootstrap-server localhost:9095 --from-beginning --topic my-replicated-topic
But when I tried with
kafka-console-consumer --bootstrap-server localhost:9095 --from-beginning --topic my-replicated-topic --partition 0
It happily lists the messages sent. Is this a bug in Kafka 1.10.11?
I just ran into this issue and the solution was to delete /brokers in zookeeper and restart the kafka nodes.
bin/zookeeper-shell <zk-host>:2181
and then
rmr /brokers
Not sure why this solves it.
When I enabled debug logging, I saw this error message over and over again in the consumer:
2017-07-07 01:20:12 DEBUG AbstractCoordinator:548 - Sending GroupCoordinator request for group test to broker xx.xx.xx.xx:9092 (id: 1007 rack: null)
2017-07-07 01:20:12 DEBUG AbstractCoordinator:559 - Received GroupCoordinator response ClientResponse(receivedTimeMs=1499390412231, latencyMs=84, disconnected=false, requestHeader={api_key=10,api_version=0,correlation_id=13,client_id=consumer-1}, responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}}) for group test
2017-07-07 01:20:12 DEBUG AbstractCoordinator:581 - Group coordinator lookup for group test failed: The group coordinator is not available.
2017-07-07 01:20:12 DEBUG AbstractCoordinator:215 - Coordinator discovery failed for group test, refreshing metadata
For me the solution described in this thread worked - https://stackoverflow.com/a/51540528/7568227
Check if
offsets.topic.replication.factor
(or probably other config parameters related to replication)
is not higher than the number of brokers. That was the problem in my case.
There was no need to use --partition 0 anymore after this fix.
Otherwise I recommend to follow the debugging procedure described in the mentioned thread.
In my case, this doesn't work
kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
and this works
kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic --partition 0
because the topic __consumer_offsets was located on the unaccessible broker. Basically, I'd forgotten to replicate it. Relocating __consumer_offsets solved my issue.
Was getting the same issue on my Mac.
I checked the logs and found the following error.
Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor').
This error can be ignored if the cluster is starting up and not all brokers are up yet.
This can be fixed by changing the replication factor to 1. Add the following line in server.properties and restart Kafka/Zookeeper.
offsets.topic.replication.factor=1
I got the same problem, now I have figured out.
When you use --zookeeper, it is supposed to be provided with zookeeper address as parameter.
When you use --bootstrap-server, it is supposed to be provided with broker address as parameter.
Your localhost is the foo here.
if you replace the localhost word for the actual hostname, it should work.
like this:
producer
./bin/kafka-console-producer.sh --broker-list \
sandbox-hdp.hortonworks.com:9092 --topic test
consumer:
./bin/kafka-console-consumer.sh --topic test --from-beginning \
--bootstrap-server bin/kafka-console-consumer.sh --new-consumer \
--topic test --from-beginning \
--bootstrap-server localhost:9092
This problem also impacts ingesting data from the kafka using flume and sink the data to HDFS.
To fix the above issue:
Stop Kafka brokers
Connect to zookeeper cluster and remove /brokers z node
Restart kafka brokers
There is no issue with respect to kafka client version and scala version that we are using the cluster. Zookeeper might have wrong information about broker hosts.
To verify the action:
Create topic in kafka.
$ kafka-console-consumer --bootstrap-server slavenode01.cdh.com:9092 --topic rkkrishnaa3210 --from-beginning
Open a producer channel and feed some messages to it.
$ kafka-console-producer --broker-list slavenode03.cdh.com:9092 --topic rkkrishnaa3210
Open a consumer channel to consume the message from a specific topic.
$ kafka-console-consumer --bootstrap-server slavenode01.cdh.com:9092 --topic rkkrishnaa3210 --from-beginning
To test this from flume:
Flume agent config:
rk.sources = source1
rk.channels = channel1
rk.sinks = sink1
rk.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
rk.sources.source1.zookeeperConnect = ip-20-0-21-161.ec2.internal:2181
rk.sources.source1.topic = rkkrishnaa321
rk.sources.source1.groupId = flume1
rk.sources.source1.channels = channel1
rk.sources.source1.interceptors = i1
rk.sources.source1.interceptors.i1.type = timestamp
rk.sources.source1.kafka.consumer.timeout.ms = 100
rk.channels.channel1.type = memory
rk.channels.channel1.capacity = 10000
rk.channels.channel1.transactionCapacity = 1000
rk.sinks.sink1.type = hdfs
rk.sinks.sink1.hdfs.path = /user/ce_rk/kafka/%{topic}/%y-%m-%d
rk.sinks.sink1.hdfs.rollInterval = 5
rk.sinks.sink1.hdfs.rollSize = 0
rk.sinks.sink1.hdfs.rollCount = 0
rk.sinks.sink1.hdfs.fileType = DataStream
rk.sinks.sink1.channel = channel1
Run flume agent:
flume-ng agent --conf . -f flume.conf -Dflume.root.logger=DEBUG,console -n rk
Observe logs from the consumer that the message from the topic is written in HDFS.
18/02/16 05:21:14 INFO internals.AbstractCoordinator: Successfully joined group flume1 with generation 1
18/02/16 05:21:14 INFO internals.ConsumerCoordinator: Setting newly assigned partitions [rkkrishnaa3210-0] for group flume1
18/02/16 05:21:14 INFO kafka.SourceRebalanceListener: topic rkkrishnaa3210 - partition 0 assigned.
18/02/16 05:21:14 INFO kafka.KafkaSource: Kafka source source1 started.
18/02/16 05:21:14 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: source1: Successfully registered new MBean.
18/02/16 05:21:14 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: source1 started
18/02/16 05:21:41 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false
18/02/16 05:21:42 INFO hdfs.BucketWriter: Creating /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920.tmp
18/02/16 05:21:48 INFO hdfs.BucketWriter: Closing /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920.tmp
18/02/16 05:21:48 INFO hdfs.BucketWriter: Renaming /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920.tmp to /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920
18/02/16 05:21:48 INFO hdfs.HDFSEventSink: Writer callback called.
Use this:
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Note: Remove --new-consumer from your command
For reference see here: https://kafka.apache.org/quickstart
Can you please try like this:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
In my case it didn't worked using either approaches then I also increased the log level to DEBUG at config/log4j.properties, started the console consumer
./bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic MY_TOPIC
Then got the log below
[2018-03-11 12:11:25,711] DEBUG [MetadataCache brokerId=10] Error while fetching metadata for MY_TOPIC-3: leader not available (kafka.server.MetadataCache)
The point here is that I have two kafka nodes but one is down, by some reason by default kafka-console consumer will not consume if there is some partition not available because the node is down (the partition 3 in that case). It doesn't happen in my application.
Possible solutions are
Startup the down brokers
Delete the topic and create it again that way all partitions will be placed at the online broker node
Run the below command from bin:
./kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092
"test" is the topic name
I had this problem that consumer finished executing
in kafka_2.12-2.3.0.tgz.
Tried debugging but no logs were printed.
Try running fine with kafka_2.12-2.2.2
.Works fine.
And try running the zookeeper and kafka from the quickstart guide!
In my case, broker.id=1 in server.properties was problem.
This should be broker.id=0 when you use only one kafka server for development.
Don't forget remove all logs and restart zookeper and kafka
Remove /tmp/kafka-logs (defined in server.properties file)
Remove [your_kafka_home]/logs
Restart Zookeper and Kafka
In kafka_2.11-0.11.0.0 the zookeeper server is deprecated and and it is using bootstrap-server, and it will take broker ip address and port. If you give correct broker parameters you will be able to consume messages.
e.g. $ bin/kafka-console-consumer.sh --bootstrap-server :9093 --topic test --from-beginning
I'm using port 9093, for you it may vary.
regards.
replication factor must be at least 3
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic test