Creating a Kafka topic results in no leader - apache-kafka

Question

I am using Kafka v0.9.0.1 (Scala v2.11) and the com.101tec:zkclient v0.7. I am trying to use AdminUtils to create a kafka topic. My code is the following.
String zkServers = "node1:2181,node2:2181,node3:2181,node4:2181";
Integer sessionTimeout = (int)TimeUnit.SECONDS.toMillis(10L);
Integer connectionTimeout = (int)TimeUnit.SECONDS.toMillis(8L);
ZkSerializer zkSerializer = ZKStringSerializer$.MODULE$;
Boolean isSecureKafkaCluster = false;
String topic = "test";
Integer partitions = 1;
Integer replication = 3;
ZkClient zkClient = new ZkClient(zkServers, sessionTimeout, connectionTimeout, zkSerializer);
ZkUtils zkUtils = new ZkUtils(zkClient, new ZkConnection(zkServers), isSecureKafkaCluster)
if(!AdminUtils.topicExists(zkUtils, topic)) {
AdminUtils.createTopic(zkUtils, topic, partitions, replications, new Properties());
}
The topic is actually created as verified by the following command.
bin/kafka-topics.sh --describe --zookeeper node1:2181 --topic test
However, the output is not as expected.
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: -1 Replicas: 4 Isr:
If I use the script.
bin/kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 3 --partitions 1 --topic topic1
Then I see the following.
Topic:test1 PartitionCount:1 ReplicationFactor:3 Configs:
Topic: test1 Partition: 0 Leader: 2 Replicas: 2,3,4 Isr: 2
Any ideas on what I'm doing wrong? The effect is that if I use a Producer to send a ProducerRecord to the topic, nothing shows up on the topic.

Answer 1

I had the same issue.
Solution:
Clean zk meta info (/brokers/topic)
Clean all /data dir to remove all topic-partition folders belongs to that topic
Restart the whole kafka cluster all brokers at once.
Recreate that topic.
This solved my problem. And I think the root cause was the defect from kafka itself failing to handle clean removal topics (this has been fixed since v1.0.0).
Edit:
even with Kafka(>= v1.0.0), sometimes deleting topic will stuck if you are deleting an empty topic or if your kafka cluster is under extreme load.
solution would be as simple as restarting the controller broker. (you can always find the controller broker under ZK: /controller by get /controller). so just restarting one broker instead of the whole kafka cluster.