Setting up Kafka first time, Kafka 0.11. Using pretty much default configurations. Produced produced some messages to topic ABC. 2 Consumers are coded to consume messages from the same topic. Each consumer belongs to different group id GROUP.1 and GROUP.2
Want to look into the topic for all the messages and also the offset details.
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group GROUP.1
throws following error,
Error: The consumer group 'GROUP.1' does not exist.
Same error for GROUP.2 also. I got some output without error for one of the group yesterday, but not today. What I'm I missing? Need to configure somewhere to persist consumer group details, or will the command work only when the consumers with given group id is currently running, or?
I tried kafka-consumer-groups --zookeeper localhost:2181 --describe --group GROUP.1 but got the same error.
Also tried Kafka-consumer-offset-checker command.
kafka-consumer-offset-checker --zookeeper localhost:2181 --topic ABC --group GROUP.1
[2017-12-19 19:25:01,654] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$)
Exiting due to: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /consumers/GROUP.1/offsets/ABC/2.
As you said you saw the group details yesterday, it's probably worth noting that by default offsets are only stored for 24 hours. So if you group has not committed offsets in 24 hours, Kafka has no more information about it.
If this is indeed the issue, you can increase the time by setting offsets.retention.minutes to a larger value.
I am running Kafka 0.10.0 on CDH 5.9, cluster is kerborized.
What I am trying to do is to write messages from a remote machine to my Kafka broker.
The cluster (where Kafka is installed) has internal as well as external IP addresses.
The machines' hostnames within the cluster get resolved to the private IPs, the remote machine resolves the same hostnames to the public IP addreses.
I opened the necessary port 9092 (I am using SASL_PLAINTEXT protocol) from remote machine to Kafka Broker, verified that using telnet.
First Step - in addition to the standard properties for the Kafka Broker, I configured the following:
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://<hostname>:9092
I am able to start the console consumer with
kafka-console-consumer --new consumer --topic <topicname> --from-beginning --bootstrap-server <hostname>:9092 --consumer.config consumer.properties
I am able to use my custom producer from another machine within the cluster.
Relevant excerpt of producer properties:
security.protocol=SASL_PLAINTEXT
bootstrap.servers=<hostname>:9092
I am not able to use my custom producer from the remote machine:
Exception org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for <topicname>-<partition>
using the same producer properties. I am able to telnet the Kafka Broker from the machine and /etc/hosts includes hostnames and public IPs.
Second Step - I modified server.properties:
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://<kafkaBrokerInternalIP>:9092
consumer & producer within the same cluster still run fine (bootstrap
servers are now the internal IP with port 9092)
as expected remote producer fails (but that is obvious given that it
is not aware of the internal IP addresses)
Third Step - where it gets hairy :(
listeners=SASL_PLAINTEXT://0.0.0.0:9092
advertised.listeners=SASL_PLAINTEXT://<kafkaBrokerPublicIP>:9092
starting my consumer with
kafka-console-consumer --new-consumer --topic <topicname> --from-beginning --bootstrap-server <hostname>:9092 --consumer.config consumer.properties
gives me a warning, but I don't think this is right...
WARN clients.NetworkClient: Error while fetching metadata with correlation id 1 : {<topicname>=LEADER_NOT_AVAILABLE}
starting my consumer with
kafka-console-consumer --new-consumer --topic <topicname> --from-beginning --bootstrap-server <KafkaBrokerPublicIP>:9092 --consumer.config consumer.properties
just hangs after those log messages:
INFO utils.AppInfoParser: Kafka version : 0.10.0-kafka-2.1.0
INFO utils.AppInfoParser: Kafka commitId : unknown
seems like it cannot find a coordinator as in the normal flow this would be the next log:
INFO internals.AbstractCoordinator: Discovered coordinator <hostname>:9092 (id: <someNumber> rack: null) for group console-consumer-<someNumber>.
starting the producer on a cluster node with bootstrap.servers=:9092
I observe the same as with the producer:
WARN NetworkClient:600 - Error while fetching metadata with correlation id 0 : {<topicname>=LEADER_NOT_AVAILABLE}
starting the producer on a cluster node with bootstrap.servers=:9092 I get
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
starting the producer on my remote machine with either bootstrap.servers=:9092 or bootstrap.servers=:9092 I get
NetworkClient:600 - Error while fetching metadata with correlation id 0 : {<topicname>=LEADER_NOT_AVAILABLE}
I have been struggling for the past three days to get this to work, however I am out of ideas :/ My understanding is that advertised.hostnames serves for exactly this purpose, however either I am doing something wrong, or there is something wrong in the machine setup.
Any hints are very much appreciated!
I met this issue recently.
In my case , I enabled Kafka ACL, and after disable it by comment this 2 configuration, the problem worked around.
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:kafka
And an thread may help you I think:
https://gist.github.com/jorisdevrede/a7933a99251452bb1867
What mentioned in it at the end:
If you only use a SASL_PLAINTEXT listener on the Kafka Broker, you
have to make sure that you have set the
security.inter.broker.protocol=SASL_PLAINTEXT too, otherwise you will
get a LEADER_NOT_AVAILABLE error in the client.
I found something very weird with Kafka.
I have a producer with 3 brokers :
bin/kafka-console-producer.sh --broker-list localhost:9093, localhost:9094, localhost:9095 --topic topic
Then I try to run a consumer with the new API :
bin/kafka-console-consumer.sh --bootstrap-server localhost:9093,localhost:9094,localhost:9095 --topic topic --from-beginning
I got nothing ! BUT if I use the old API :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic topic
I got my messages !
What is wrong with me ?
PS : I am using Kafka 10
I eventually resolved my problem thanks to this similar post : Kafka bootstrap-servers vs zookeeper in kafka-console-consumer
I believe it is a bug / wrong configuration of mine leading to a problem with zookeeper and kafka.
SOLUTION :
First be sure to have enable topic deleting in server.properties files of your brokers :
# Switch to enable topic deletion or not, default value is false
delete.topic.enable=true
Then delete the topic :
bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic myTopic
Remove all the /tmp/log.dir directories of your brokers.
EDIT : I faced again the problem and I had to remove also the log files of zookeeper in /tmp/zookeeper/version-2/.
Finally delete the topic in /brokers/topics in zookeeper as follow :
$ kafka/bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is disabled
rmr /broker/topics/mytopic
And restart your brokers and create your topic again.
After fighting a while with same problem. Specify --partition and console consumer new API works (but hangs..). I have CDH 5.12 + Kafka 0.11 (from parcel).
UPD:
Also find out that Kafka 0.11 (versioned as 3.0.0 in CDH parclel) does not work correctly with consuming messages. After downgrading to Kafka 0.10 it become OK. --partition does not need any more.
I had the same problem, but I was using a single broker instance for my "cluster", and I was getting this error:
/var/log/messages
[2018-04-04 22:29:39,854] ERROR [KafkaApi-20] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
I just added in my server configuration file the setting offsets.topic.replication.factor=1 and restarted. It started to work fine.
I have setup a multi node setup for kafka, everything seems to work well and show no error logs unless i try to push message to one producer. I get a message:
Bootstrap broker host2:2181 disconnected (org.apache.kafka.clients.NetworkClient)
and on the zookeeper logs i am getting:
"WARN Exception causing close of session 0x0 due to java.io.IOException:
Unreasonable length = 1701969920 (org.apache.zookeeper.server.NIOServerCnxn)"
i cleaned up my data directory which is "/var/zookeeper/data" still no luck.
Any help on the the would be much appriciated
Vaibhav looking at this line (Bootstrap broker host2:2181) looks like you are trying to connect to zookeeper instance rather than broker instance. By Default Kafka broker runs on 9092 port. So producer and consumer should be created as per below command
Producer :
bin/kafka-console-producer.sh --broker-list host1:9092,host2:9092 \
--topic "topic_name"
Consumer:
bin/kafka-console-consumer.sh --bootstrap-server <host_ip_of_producer>:9092 \
--topic "topic_name" --from-beginning
I am doing the Kafka Quickstart for Kafka 0.9.0.0.
I have zookeeper listening at localhost:2181 because I ran
bin/zookeeper-server-start.sh config/zookeeper.properties
I have a single broker listening at localhost:9092 because I ran
bin/kafka-server-start.sh config/server.properties
I have a producer posting to topic "test" because I ran
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
yello
is this thing on?
let's try another
gimme more
When I run the old API consumer, it works by running
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
However, when I run the new API consumer, I don't get anything when I run
bin/kafka-console-consumer.sh --new-consumer --topic test --from-beginning \
--bootstrap-server localhost:9092
Is it possible to subscribe to a topic from the console consumer using the new api? How can I fix it?
I my MAC box I was facing the same issue of console-consumer not consuming any messages when used the command
kafka-console-consumer --bootstrap-server localhost:9095 --from-beginning --topic my-replicated-topic
But when I tried with
kafka-console-consumer --bootstrap-server localhost:9095 --from-beginning --topic my-replicated-topic --partition 0
It happily lists the messages sent. Is this a bug in Kafka 1.10.11?
I just ran into this issue and the solution was to delete /brokers in zookeeper and restart the kafka nodes.
bin/zookeeper-shell <zk-host>:2181
and then
rmr /brokers
Not sure why this solves it.
When I enabled debug logging, I saw this error message over and over again in the consumer:
2017-07-07 01:20:12 DEBUG AbstractCoordinator:548 - Sending GroupCoordinator request for group test to broker xx.xx.xx.xx:9092 (id: 1007 rack: null)
2017-07-07 01:20:12 DEBUG AbstractCoordinator:559 - Received GroupCoordinator response ClientResponse(receivedTimeMs=1499390412231, latencyMs=84, disconnected=false, requestHeader={api_key=10,api_version=0,correlation_id=13,client_id=consumer-1}, responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}}) for group test
2017-07-07 01:20:12 DEBUG AbstractCoordinator:581 - Group coordinator lookup for group test failed: The group coordinator is not available.
2017-07-07 01:20:12 DEBUG AbstractCoordinator:215 - Coordinator discovery failed for group test, refreshing metadata
For me the solution described in this thread worked - https://stackoverflow.com/a/51540528/7568227
Check if
offsets.topic.replication.factor
(or probably other config parameters related to replication)
is not higher than the number of brokers. That was the problem in my case.
There was no need to use --partition 0 anymore after this fix.
Otherwise I recommend to follow the debugging procedure described in the mentioned thread.
In my case, this doesn't work
kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
and this works
kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic --partition 0
because the topic __consumer_offsets was located on the unaccessible broker. Basically, I'd forgotten to replicate it. Relocating __consumer_offsets solved my issue.
Was getting the same issue on my Mac.
I checked the logs and found the following error.
Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor').
This error can be ignored if the cluster is starting up and not all brokers are up yet.
This can be fixed by changing the replication factor to 1. Add the following line in server.properties and restart Kafka/Zookeeper.
offsets.topic.replication.factor=1
I got the same problem, now I have figured out.
When you use --zookeeper, it is supposed to be provided with zookeeper address as parameter.
When you use --bootstrap-server, it is supposed to be provided with broker address as parameter.
Your localhost is the foo here.
if you replace the localhost word for the actual hostname, it should work.
like this:
producer
./bin/kafka-console-producer.sh --broker-list \
sandbox-hdp.hortonworks.com:9092 --topic test
consumer:
./bin/kafka-console-consumer.sh --topic test --from-beginning \
--bootstrap-server bin/kafka-console-consumer.sh --new-consumer \
--topic test --from-beginning \
--bootstrap-server localhost:9092
This problem also impacts ingesting data from the kafka using flume and sink the data to HDFS.
To fix the above issue:
Stop Kafka brokers
Connect to zookeeper cluster and remove /brokers z node
Restart kafka brokers
There is no issue with respect to kafka client version and scala version that we are using the cluster. Zookeeper might have wrong information about broker hosts.
To verify the action:
Create topic in kafka.
$ kafka-console-consumer --bootstrap-server slavenode01.cdh.com:9092 --topic rkkrishnaa3210 --from-beginning
Open a producer channel and feed some messages to it.
$ kafka-console-producer --broker-list slavenode03.cdh.com:9092 --topic rkkrishnaa3210
Open a consumer channel to consume the message from a specific topic.
$ kafka-console-consumer --bootstrap-server slavenode01.cdh.com:9092 --topic rkkrishnaa3210 --from-beginning
To test this from flume:
Flume agent config:
rk.sources = source1
rk.channels = channel1
rk.sinks = sink1
rk.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
rk.sources.source1.zookeeperConnect = ip-20-0-21-161.ec2.internal:2181
rk.sources.source1.topic = rkkrishnaa321
rk.sources.source1.groupId = flume1
rk.sources.source1.channels = channel1
rk.sources.source1.interceptors = i1
rk.sources.source1.interceptors.i1.type = timestamp
rk.sources.source1.kafka.consumer.timeout.ms = 100
rk.channels.channel1.type = memory
rk.channels.channel1.capacity = 10000
rk.channels.channel1.transactionCapacity = 1000
rk.sinks.sink1.type = hdfs
rk.sinks.sink1.hdfs.path = /user/ce_rk/kafka/%{topic}/%y-%m-%d
rk.sinks.sink1.hdfs.rollInterval = 5
rk.sinks.sink1.hdfs.rollSize = 0
rk.sinks.sink1.hdfs.rollCount = 0
rk.sinks.sink1.hdfs.fileType = DataStream
rk.sinks.sink1.channel = channel1
Run flume agent:
flume-ng agent --conf . -f flume.conf -Dflume.root.logger=DEBUG,console -n rk
Observe logs from the consumer that the message from the topic is written in HDFS.
18/02/16 05:21:14 INFO internals.AbstractCoordinator: Successfully joined group flume1 with generation 1
18/02/16 05:21:14 INFO internals.ConsumerCoordinator: Setting newly assigned partitions [rkkrishnaa3210-0] for group flume1
18/02/16 05:21:14 INFO kafka.SourceRebalanceListener: topic rkkrishnaa3210 - partition 0 assigned.
18/02/16 05:21:14 INFO kafka.KafkaSource: Kafka source source1 started.
18/02/16 05:21:14 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: source1: Successfully registered new MBean.
18/02/16 05:21:14 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: source1 started
18/02/16 05:21:41 INFO hdfs.HDFSDataStream: Serializer = TEXT, UseRawLocalFileSystem = false
18/02/16 05:21:42 INFO hdfs.BucketWriter: Creating /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920.tmp
18/02/16 05:21:48 INFO hdfs.BucketWriter: Closing /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920.tmp
18/02/16 05:21:48 INFO hdfs.BucketWriter: Renaming /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920.tmp to /user/ce_rk/kafka/rkkrishnaa3210/18-02-16/FlumeData.1518758501920
18/02/16 05:21:48 INFO hdfs.HDFSEventSink: Writer callback called.
Use this:
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Note: Remove --new-consumer from your command
For reference see here: https://kafka.apache.org/quickstart
Can you please try like this:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic
In my case it didn't worked using either approaches then I also increased the log level to DEBUG at config/log4j.properties, started the console consumer
./bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic MY_TOPIC
Then got the log below
[2018-03-11 12:11:25,711] DEBUG [MetadataCache brokerId=10] Error while fetching metadata for MY_TOPIC-3: leader not available (kafka.server.MetadataCache)
The point here is that I have two kafka nodes but one is down, by some reason by default kafka-console consumer will not consume if there is some partition not available because the node is down (the partition 3 in that case). It doesn't happen in my application.
Possible solutions are
Startup the down brokers
Delete the topic and create it again that way all partitions will be placed at the online broker node
Run the below command from bin:
./kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092
"test" is the topic name
I had this problem that consumer finished executing
in kafka_2.12-2.3.0.tgz.
Tried debugging but no logs were printed.
Try running fine with kafka_2.12-2.2.2
.Works fine.
And try running the zookeeper and kafka from the quickstart guide!
In my case, broker.id=1 in server.properties was problem.
This should be broker.id=0 when you use only one kafka server for development.
Don't forget remove all logs and restart zookeper and kafka
Remove /tmp/kafka-logs (defined in server.properties file)
Remove [your_kafka_home]/logs
Restart Zookeper and Kafka
In kafka_2.11-0.11.0.0 the zookeeper server is deprecated and and it is using bootstrap-server, and it will take broker ip address and port. If you give correct broker parameters you will be able to consume messages.
e.g. $ bin/kafka-console-consumer.sh --bootstrap-server :9093 --topic test --from-beginning
I'm using port 9093, for you it may vary.
regards.
replication factor must be at least 3
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic test