I need to add another consumer to my topic to increase the speed. I have seen the documentation here and can use
bin/kafka-topics.sh --bootstrap-server broker_host:port --alter --topic my_topic_name --partitions 2
but can I do it directly while the messages arrive?
Related
I am referring to a book on Kafka. To create a Kafka topic, it states:
Using the command line utility for creating topics on the Kafka server, let’s create a topic named replicated-kafkatopic with two partitions and two replicas:
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic replicated-kafkatopic
While text states to create 2 partitions and 2 replicas, arguments passed are: 3 for replication-factor and 1 for partitions.
Are arguments correct in context of what text states?
No, it must be a typo. If you want to create a topic with two partitions and two replicas, the command should be as follows:
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic replicated-kafkatopic
The command that you have provided, creates a topic replicated-kafkatopic with 1 partition (--partitions 1) and 3 replicas (--replication-factor 3).
./bin/kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic test
./bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test
Why to consume kakfa needed zookeeper?
could I consume kafka from brokers ?
--zookeeper is an outdated option (for the old Consumer).
Use the newer --bootstrap-server option with kafka-console-consumer.
More info: http://kafka.apache.org/documentation/#newconsumerconfigs
I would like to update the TTL of a specific Kafka topic to 10 days.
How can I do that?
You previously asked about that and I already replied here : Update TTL for a particular topic in kafka using Java
Unless you are asking to do that using Kafka tools? (And not in Java)
In this case there is the kafka-topics.sh command line tool, allowing you to do that using the --alter option.
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic test --config retention.ms=10000
Because altering using kafka-topics script could be removed in next release, you should use the kafka-configs script:
bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-type topics --entity-name test --add-config retention.ms=5000
To launch a Kafka consumer I have to run it with the following params.
kafka-console-consumer.bat --zookeeper localhost:2181 --topic MyTopic
Was wondering why it needs to have a zookeeper as a param, if I use the broker param will it not work similar to how the producer is launched. Consumer needs to be aware of the broker/cluster and not the zookeeper location.
.\kafka-console-producer.bat --broker --list localhost:9092 --topic MyTopic
This is on windows, am not sure how its in Unix flavors.
-Chandra
You're invoking the old Kafka Consumer. Old consumer requires zookeeper to co-ordinate from which broker to fetch data.
The new consumer don't requires the zookeeper parameter.
.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic MyTopic --new-consumer
Optionally, you can add --from-beginning argument to read the old records in the topic.
Is this any bug with the Apache Kafka 0.9.0.0? I'm using the following command to list the topics and I get nothing even though the server was started with a topic being created!
Joes-MacBook-Pro:kafka_2.11-0.9.0.0 joe$ bin/kafka-topics.sh --list --zookeeper localhost:2181
Joes-MacBook-Pro:kafka_2.11-0.9.0.0 joe$
Here is the command that I used to start Apache Kafka:
bin/kafka-server-start.sh config/server.properties & bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_topic
It is getting harsh for me to just try out the basic stuff around the much hyped Apache Kafka. Here is one another example where I'm pretty much stuck!
Apache Kafka 0.9.0.0 Show all Topics with Partitions
I don't see the zookeeper started in your code.
Please start zookeeper and then kafka in the same order.
bin/zookeeper-server-start.sh config/zookeeper.properties &
bin/kafka-server-start.sh config/server.properties &
Try to list the topics with the below command.
bin/kafka-topics.sh --list --zookeeper localhost:2181
You can create the topic with the below command
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic test --partitions 1 --replication-factor 1
Once the topic is created, you get a message saying "test" created.
Now you can try list command to see your topic listed.
Note: The replication factor depends upon the number of brokers that you run on a single server.