I am referring to a book on Kafka. To create a Kafka topic, it states:
Using the command line utility for creating topics on the Kafka server, let’s create a topic named replicated-kafkatopic with two partitions and two replicas:
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic replicated-kafkatopic
While text states to create 2 partitions and 2 replicas, arguments passed are: 3 for replication-factor and 1 for partitions.
Are arguments correct in context of what text states?
No, it must be a typo. If you want to create a topic with two partitions and two replicas, the command should be as follows:
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic replicated-kafkatopic
The command that you have provided, creates a topic replicated-kafkatopic with 1 partition (--partitions 1) and 3 replicas (--replication-factor 3).
Related
I should create a topic by
sh /opt/client/Kafka/kafka/bin/kafka-topics.sh --zookeeper xxx.xxx.xxx.xxx:24002/kafka --create --partitions 3 --replication-factor 1 --topic order
in kafka rather than
sh /opt/client/Kafka/kafka/bin/kafka-topics.sh --zookeeper xxx.xxx.xxx.xxx:24002 --create --partitions 3 --replication-factor 1 --topic order
Why does the true way have one more word "/kafka"?
The /kafka on Zookeeper is called a chroot. It acts like a folder in Zookeeper for isolation of data. You'd need to ask your server administrators why they did it this way
Worth mentioning, Zookeeper flag is deprecated in latest Kafka.
Also, you should place the Kafka installation bin folder on your PATH so that you don't need to always use the full file path.
I should create a topic by
sh /opt/client/Kafka/kafka/bin/kafka-topics.sh --zookeeper xxx.xxx.xxx.xxx:24002/kafka --create --partitions 3 --replication-factor 1 --topic order
in kafka rather than
sh /opt/client/Kafka/kafka/bin/kafka-topics.sh --zookeeper xxx.xxx.xxx.xxx:24002 --create --partitions 3 --replication-factor 1 --topic order
Why does the true way have one more word "/kafka"?
The /kafka on Zookeeper is called a chroot. It acts like a folder in Zookeeper for isolation of data. You'd need to ask your server administrators why they did it this way
Worth mentioning, Zookeeper flag is deprecated in latest Kafka.
Also, you should place the Kafka installation bin folder on your PATH so that you don't need to always use the full file path.
I need to add another consumer to my topic to increase the speed. I have seen the documentation here and can use
bin/kafka-topics.sh --bootstrap-server broker_host:port --alter --topic my_topic_name --partitions 2
but can I do it directly while the messages arrive?
when we create topic ,where we decide number of partition and replica factor.
Do this topic get created in all the brokers?
Is it specific to any one broker?
You are required to pass replication factor and partitions when creating topics.
One which broker(s) each partition and replica are placed, are randomly decided, although you can later use kafka-reassign-partitions to move replicas around to other brokers.
You can crate a topic with this command:
./bin/kafka-topics.sh --create --zookeeper <ZOOKEEPER_URL:PORT> --replication-factor <NO_OF_REPLICATIONS> --partitions <NO_OF_PARTITIONS> --topic <TOPIC_NAME>
After this command metadata about the topic (number of partitions, replicas, ISR list etc.) is stored in Zookeeper. You can get information about topic by this command:
./bin/kafka-topics.sh --zookeeper localhost:2181 --topic TopicName --describe
Replica list of partitions are created according to round-robin algorithm and Controller broker is responsible for noticing new topic creation and triggering partitions assignment.
Is this any bug with the Apache Kafka 0.9.0.0? I'm using the following command to list the topics and I get nothing even though the server was started with a topic being created!
Joes-MacBook-Pro:kafka_2.11-0.9.0.0 joe$ bin/kafka-topics.sh --list --zookeeper localhost:2181
Joes-MacBook-Pro:kafka_2.11-0.9.0.0 joe$
Here is the command that I used to start Apache Kafka:
bin/kafka-server-start.sh config/server.properties & bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_topic
It is getting harsh for me to just try out the basic stuff around the much hyped Apache Kafka. Here is one another example where I'm pretty much stuck!
Apache Kafka 0.9.0.0 Show all Topics with Partitions
I don't see the zookeeper started in your code.
Please start zookeeper and then kafka in the same order.
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
Try to list the topics with the below command.
bin/kafka-topics.sh --list --zookeeper localhost:2181
You can create the topic with the below command
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --partitions 1 --replication-factor 1
Once the topic is created, you get a message saying "test" created.
Now you can try list command to see your topic listed.
Note: The replication factor depends upon the number of brokers that you run on a single server.