I have produce message using Kafka producer in java application and I had consumed message from Kafka Server smoothly..
But suddenly I cant run the command '''kafka-console-consumer.bat''' in the command prompt. It doesn't display any error message..
I have tried to consume message from the Kafka server through the command '''kafka-console-consumer.bat --bootstrap-server localhost:9092 --from-beginning --topic student-details''' where the '''student-details''' is the Topic name
But the command doesn't working.. Is there any alternative way to consume message
The alternative is to write code for a consumer, or use any other tool available. I prefer kcat on the CLI.
To debug your problem, you could edit Kafka's log4j.properties to increase the log level of the consumer.
Or, depending on how much time elapsed between "now", and "working perfectly", your topic is empty.
Related
I'm writing a Go service that works with Kafka. I have a problems with bad commits when broker rebalances. I want to do an experiment forcing Kafka to rebalance and to see how the service behaves.
What I do:
running Kafka in Docker locally (broker, zookeeper, schema registry and control center)
created a topic with 2 partition
running producer that sends messages to both partitions
Then I'm running two consumers with the same groupID, after that I'm closing one. It seems to me that broker should start rebalancing this moment. Or no? Who's logs should I check for it?
You can check that by running the following commands:
bin/kafka-consumer-groups --bootstrap-server host:9092 --list
and to describe:
bin/kafka-consumer-groups --bootstrap-server host:9092 --describe --group foo
Full documentation could be found here: Kafka consumer group
Who's logs should I check for it?
The running consumer's log should be checked, depending on if the library you're using actually logs such information.
I am running Kafka in windows.
I am creating a Kafka console producer with below command
C:\kafka_2.11-2.4.0\bin\windows>kafka-console-producer.bat --broker-list localhost:9092 --topic input-topic
and creating kafka console consumer with below command
C:\kafka_2.11-2.4.0\bin\windows>kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic input-topic
the problem is i am getting the messages continuously. yesterday it was showing only the messages which i am typing to the console producer.
how to stop auto sending messages?
I need to send the messages if I am typing otherwise it should not send blank messages.
By default, console consumer only gets latest offsets of the topic (only new messages)
If you want to read existing data, you must add --from-beginning
If you want to preserve offsets between runs, you need a --group
I am using kafka 1.0.0V In my project. From yesterday on wards. I am unable to listen Messages from command line . In the same time I am able to listen the messages from Kafka tool. In the same time My applications are also not consuming the data. Is there any issue in kafka.
This is the command i am using to consume the messages
/usr/hdp/2.6.0.3-8/kafka/bin/kafka-console-consumer.sh --bootstrap-server hostname:6667 --topic test --from-beginning
topic create,describe,produce command are working from commandline
I have installed Apache Kafka to a Windows system and have tried a console Producer-Consumer example. In case I add new consumers to the topic, only the messages after adding the consumer are printed to the console. Is there any way to get all the messages of the particular topic?
You need to add --from-beginning flag to your console consumer command to get all messages.
Example command:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
I try to read messages in kafka consumer using the following command:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
Here we can read old messages of about 4 days as we have set retention time in kafka server configuration file as 7 days. But while we are trying to read messages using KafkaConsumer of kaka-python client library like following:
cons = KafkaConsumer("localhost:9092", "test","smallest")
cons.fetch_messages()
we are getting messages of today's only with some offset. I don't have any idea how to get oldest message available in Kafka like we got in kafka consumer shell script above. Please help.
The docs show the config passed in via namedtuples.
consumer = KafkaConsumer('topic1', 'topic2',
bootstrap_servers=['localhost:9092'],
group_id='my_consumer_group',
auto_commit_enable=True,
auto_commit_interval_ms=30 * 1000,
auto_offset_reset='smallest')