I am new to kafka!!
Is it possible to exclude a topic from a broker in apache kafka?? How?
To run a producer for a topic we give the command:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic kafkatopic
we do not have the option of broker in this command. Can we exclude broker anyways, either command line or java API??
Also is there any documentation of java API for Kafka? Any tutorials??
Thanks in advance :)
Related
I am very new to Kafka. Following a few tutorials, I have the following questions regarding consuming actual Kafka topics.
The situation: there is a server in my workplace that is streaming Kafka topics. I have the topic name. I would like to consume this topic from my machine (Windows WSL2 Ubuntu). From this tutorial, I am able to
Run zookeeper with this command:
bin/zookeeper-server-start.sh config/zookeeper.properties
Create a broker with:
bin/kafka-server-start.sh config/server.properties
Run a producer console, with a fake topic named quickstart-events at port localhost:9092:
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
Run a consumer console listening to localhost:9092 and receive the streaming data from the producer:
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
Now for my real situation: if I know the topic name, what else do I need in order to apply the same steps above to listen to it as a consumer? What are the steps involved? I read in other threads about tunnelling with Jumphost. How to do that?
I understand this question is rather generic. Appreciate any pointers to any relevant readings or guidance.
Based on your company nameserver the next procedure should be done in your wsl instance
To gain outside connection
unable to access network from WSL2
You need to set bootstrap-server to your company server
--bootstrap-server my.company.com:9092
I have kafka topics, server name,port, and also brokers list.
How can i see the data from consumer with these details..
Try running this from your kafka directory:
bin/kafka-console-consumer.sh --bootstrap-server <BROKER_IP>:<PORT> --topic <TOPIC_NAME> --from-beginning
It's the quickest way to test consuming from a topic.
I've spent some hours to figure out what was going on but didn't manage to find the solution.
Here is my set up on a single machine:
1 zookeeper running
3 broker running (on port 9092/9093/9094)
1 topic with 3 partitions and 3 replications (each partition are properly assigned between brokers)
I'm using kafka console producer to insert messages. If i check the replication offset (cat replication-offset-checkpoint), I see that my messages are properly ingested by Kafka.
Now I use the kafka console consumer (new):
sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic testTopicPartitionned2
I dont see anything consumed. I tried to delete my logs folder (/tmp/kafka-logs-[1,2,3]), create new topics, still nothing.
However when I use the old kafka consumer:
sudo bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopicPartitionned2
I can see my messages.
Am I missing something big here to make this new consumer work ?
Thanks in advance.
Check to see what setting the consumer is using for auto.offset.reset property
This will affect what a consumer group without a previously committed offset will do in terms of setting where to start reading messages from a partition.
Check the Kafka docs for more on this.
Try providing all your brokers to --bootstrap-server argument to see if you notice any differnce:
sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --from-beginning --topic testTopicPartitionned2
Also, your topic name is rather long. I assume you've already made sure you provide the correct topic name.
I'm trying to test run a single Kafka node with 3 brokers & zookeeper. I wish to test using the console tools. I run the producer as such:
kafka-console-producer --broker-list localhost:9092,localhost:9093,localhost:9094 --topic testTopic
Then I run the consumer as such:
kafka-console-consumer --zookeeper localhost:2181 --topic testTopic --from-beginning
And I can enter messages in the producer and see them in the consumer, as expected. However, when I run the updated version of the consumer using bootstrap-server, I get nothing. E.g
kafka-console-consumer --bootstrap-server localhost:9092,localhost:9093,localhost:9094 --topic testTopic --from-beginning
This worked fine when I had one broker running on port 9092 so I'm thoroughly confused. Is there a way I can see what zookeeper is providing as the bootstrap server? Is the bootstrap server different from the broker list? Kafka compiled with Scala 2.11.
I have no idea what was wrong. Likely I put Kafka or Zookeeper in a weird state. After deleting the topics in the log.dir of each broker AND the zookeeper topics in /brokers/topics then recreating the topic, Kafka consumer behaved as expected.
Bootstrap servers are same as kafka brokers. And if you want to see the list of bootstrap server zookeeper is providing, you can query ZNode information via any ZK client. All active brokers are registered under /brokers/ids/[brokerId]. All you need is zkQuorum address. Below command will give you
list of active bootstrap servers :
./zookeeper-shell.sh localhost:2181 <<< "ls /brokers/ids"
I experienced the same problem when using mismatched versions of:
Kafka client libraries
Kafka scripts
Kafka brokers
In my exact scenario I was using Confluent Kafka client libraries version 0.10.2.1 with Confluent Platform 3.3.0 w/ Kafka broker 0.11.0.0. When I downgraded my Confluent Platform to 3.3.2 which matched my client libraries the consumer worked as expected.
My theory is that the latest kafka-console-consumer using the new Consumer API was only retrieving messages using the latest format. There were a number of message format changes introduced in Kafka 0.11.0.0.
I have installed Apache Kafka to a Windows system and have tried a console Producer-Consumer example. In case I add new consumers to the topic, only the messages after adding the consumer are printed to the console. Is there any way to get all the messages of the particular topic?
You need to add --from-beginning flag to your console consumer command to get all messages.
Example command:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning