I created the new topic 'rahul' with the following command :
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic rahul
Created topic "rahul".
I also re-checked the topics with
bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
rahhy
rahul`
Now starting the producer:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic rahul
hey
hi
hello
But when the time comes to consumer to show the messages: there is nothing
As of Kafka 0.9, you don't use Zookeeper for consumption or production
Try kafka-console-consumer --topic rahul --bootstrap-server localhost:9092
There are other ways you can check messages were sent to Kafka - by checking that the offsets of the topic partitions have changed using GetOffsetShell
Related
kafka-topics.bat --list --bootstrap-server localhost:9092 is not returning anything
kafka-topics.bat --list --bootstrap-server localhost:9092 is not showing anything
The above command meant to show me the topic list but it wasn't returning anything.
It can only be two things
The cluster at localhost:9092 does not have any topics
There are ACL's preventing you from listing topics.
Have you tried creating a topic first? You can do this with
kafka-topics.sh --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1
Does anyone knows how to fix the error when creating a topic in Kafka?
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:2181 --replication-factor 1 --partition 1 --topic test
Exception in thread "main" joptsimple.UnrecognizedOptionException: partition is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:567)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
The parameter is partitions
The bootstrap server normally (default) runs in port 9092
C:\kafka\bin\windows>kafka-topics.bat --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
In recent versions, you don't have to create topics on zookeeper. You can directly create topics on the bootstrap servers of Kafka. In the later version, they plan to remove the zookeeper altogether, so they are preparing for that in the current versions.
Use the below to create a new partition. I suggest adding the below parameters as well to control the topic behaviour appropriately.
kafka-topics.bat --create --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1 --config retention.ms=604800000 segment.bytes=26214400 retention.bytes=1048576000 min.insync.replicas=1 --topic test
I have a Kafka topic that seems to simultaneously exist and not exist.
kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic --delete
returns an error:
Topics in [] does not exist
Meanwhile, trying to re-create the topic
kafka-topics.sh --bootstrap-server localhost:9092 --topic my-topic \
--create --replication-factor 1 --partitions 1
returns
Topic already exists
It does not show up in the topics list with
kafka-topics.sh --list
I suspect some form of corruption but it's no clear how I can fully delete the topic so that it can be re-created
Instead of passing --bootstrap-server try with --zookeeper
kafka-topics.sh --zookeeper localhost:2181 --topic my-topic --delete
For this i started Zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
then the kafka server:
bin/kafka-server-start.sh config/server.properties
topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages
then what is the next step for file generation ?
You need to parse the file, and write a Kafka producer from it.
Or you can just pipe the file directly into the Console Producer.
How to write a file to Kafka Producer
Or use KafkaConnect FileSourceConnector
For this same Question i also referred another link and i got the result. The Link is --> https://docs.confluent.io/current/streams/quickstart.html
Followings are the step that i followed-->
Step 1: Start Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Step 2: Start kafka server
bin/kafka-server-start.sh config/server.properties
Step 3: create a topic named as "fast-messages"
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages
step 4: Run Producer
kafka-console-producer.sh --broker-list localhost:9092 --topic fast-messages --new-producer < my_file.txt
step 5: Listened by the Consumer:
/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fast-messages --from-beginning
How to achieve such outcome with messages in kafka topics?
I.e changelog-like functionality - have multiple messages coming into the topic, but I only care about the last one that came in.
Also what happens in the case topic is partitioned?
Is it possible in Kafka?
To achieve this, you should set cleanup.policy for this topic to compact, as shown below:
CREATE TOPIC:
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --partitions 1 --replication-factor 1 --config cleanup.policy=compact
UPDATE TOPIC:
bin/kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name my-topic --alter --add-config cleanup.policy=compact
With compact policy set, you have to assign a key for every message and Kafka producer will partition messages based on that key.