I'm using the following command to start the broker and create the topic:
bin/kafka-server-start.sh config/server.properties & bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic read-singnals
Prior to doing this, I start zoo keeper:
bin/zookeeper-server-start.sh config/zookeer.properties
When I tried to list all the topics that are available, using the following command:
bin/kafka-topics.sh --list --zookeeper localhost:2181
I get nothing printed! Any ideas?
I followed the steps (similar to the ones mentioned) and also mentioned in the documentation link.
I tried out the steps on a vagrant CentoS minimal instance (relativkreativ/centos-7-minimal) and I received the expected output.
I even tried the same name for the topic (just to make sure if it wasn't the hyphen in the name causing the issue) and it worked as expected.
Followed the steps below on vagrant centos instance
sudo yum install java
wget http://a.mbbsindia.com/kafka/0.8.2.0/kafka_2.10-0.8.2.0.tgz
tar -xzf kafka_2.10-0.8.2.0.tgz
cd kafka_2.10-0.8.2.0
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testme
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testme2
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testme2-test1
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic read-singnals
Expected results below
[vagrant#localhost kafka_2.10-0.8.2.0]$ bin/kafka-topics.sh --list --zookeeper localhost:2181
read-singnals
testme
testme2
testme2-test1
Please do let me know if you are able to find any other version differences or if you are able to solve the issue
Related
I am using dockerised wurstmeister/kafka-docker. I created a topic using
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 27 --topic raw-sensor-data --config retention.ms=86400000
After a few days I tried changing retention period by
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type topics --entity-name raw-sensor-data --add-config retention.ms=3600000
I also tried
bin/kafka-topics.sh --zookeeper locahost:2181 --alter --topic raw-sensor-data --config retention.ms=3600000
and
./bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic raw-sensor-data --config cleanup.policy=delete
This also gets reflected in topic describe details
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topics-with-overrides
Topic: raw-sensor-data PartitionCount: 27 ReplicationFactor: 1 Configs: cleanup.policy=delete,retention.ms=3600000
But I can still see old data and data is not getting deleted in 1 hour time.
In server.properties I have
log.retention.check.interval.ms=300000
Only closed log segments will be deleted. The default segment size is 1GB.
So, if you have less data in the topic, it will remain, regardless of the time that has passed.
I cannot create Kafka topic from GCE
in GEC
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
aa#kafka-1-vm:~$ sudo systemctl restart kafka
aa#kafka-1-vm:~$ bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test_cdc
-bash: bin/kafka-topics.sh: No such file or directory
aa#kafka-1-vm:~$ kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --
topic test_cdc
Exception in thread "main" joptsimple.UnrecognizedOptionException: bootstrap-server is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:361)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:44)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
If you use version older than 2.2 you should use "--zookeeper" option and pass connection string to zookeeper (check your kafka version).
Command would be something like this:
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_cdc
[before your command] kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test_cdc
In the latest version kafka_2.12-3.1.0 (2022) after unzipping and setting the properties and logs. keep the Kafka folder on the C drive and always run the command prompt with 'run as administrator'.
The .bat file is for windows
Terminal 1
C:\kafka\bin\windows>zookeeper-server-start.bat
..\..\config\zookeeper.properties
Terminal 2
C:\kafka\bin\windows>kafka-server-start.bat
..\..\config\server.properties
Terminal 3
C:\kafka\bin\windows>kafka-topics.bat --create --topic tutorialGB
--bootstrap-server localhost:9092
Created topic tutorialGB.
To checklist of topic created
C:\kafka\bin\windows>kafka-topics.bat --list --bootstrap-server
localhost:9092
tutorialGB
For this i started Zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
then the kafka server:
bin/kafka-server-start.sh config/server.properties
topic:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages
then what is the next step for file generation ?
You need to parse the file, and write a Kafka producer from it.
Or you can just pipe the file directly into the Console Producer.
How to write a file to Kafka Producer
Or use KafkaConnect FileSourceConnector
For this same Question i also referred another link and i got the result. The Link is --> https://docs.confluent.io/current/streams/quickstart.html
Followings are the step that i followed-->
Step 1: Start Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Step 2: Start kafka server
bin/kafka-server-start.sh config/server.properties
Step 3: create a topic named as "fast-messages"
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic fast-messages
step 4: Run Producer
kafka-console-producer.sh --broker-list localhost:9092 --topic fast-messages --new-producer < my_file.txt
step 5: Listened by the Consumer:
/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic fast-messages --from-beginning
I am new to Kafka and trying to create a new topic on my local machine.
I am following this link.
Here are the steps which i followed:
Start zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Start kafka-server
bin/kafka-server-start.sh config/server.properties
Create a topic
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
but when creating the topic, i am getting the following error:
Exception in thread "main" joptsimple.UnrecognizedOptionException: bootstrap-server is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:358)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:44)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
Is there any other configuration required to create a topic?
What wrong am i doing
If you use version older than 2.2 you should use --zookeeper option and pass connection string to zookeeper
Command would be something like that:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
I'm trying to run official "Kafka010Example.scala", but unortunatelly it doesn't read from input topic and write to output as expected. What am I missing or doing wrong? Any help or hints much appreciated. Here's exactly what I did:
Started kafka in docker container (spotify/kafka:latest)
$ docker run -d -p 2181:2181 -p 9092:9092 spotify/kafka:latest
Started bash session inside of the container:
$ docker exec -it 26d1cfced4cb /bin/bash
Created input and output topics:
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test-input
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test-output
Launched "Kafka010Example.scala" running flink 1.3.2 in local mode (MiniCluster), with flink-connector-kafka-0.10_2.11
with the following arguments:
--input-topic test-input --output-topic test-output --bootstrap.servers localhost:9092 --zookeeper.connect localhost:2181 --group.id myconsumer
Logs:
https://file.io/jWsqI8
Sent some messages to the topic:
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-input
blah
blahh
blahhh
Checked offset on the output topic - NOTHING
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-offset-checker.sh --zookeeper localhost:2181 --group myconsumer --topic test-output
Group Topic Pid Offset logSize
Lag Owner
myconsumer test-output 0 0 0
0 none
Check consumer group offset - NOTHING
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-consumer-groups.sh --zookeeper localhost:2181 --describe --group myconsumer
No topic available for consumer group provided
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
I suggest using the console-consumer to watch the test-output topic. I wouldn't expect the offset for the test-output topic to have advanced, since no one has read from it.
Step 7 should be:
$ /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --group myconsumer --topic test-output