Kafka consumer returns empty iterator - apache-kafka

Question

In my sample program i try to publish a file and try to consume that immediately. But my consumer iterator returns null.
Any idea what I'm doing wrong?
Test
**main(){**
KafkaMessageProducer producer = new KafkaMessageProducer(topic, file);
producer.generateMessgaes();
MessageListener listener = new MessageListener(topic);
listener.start();
}
MessageListener
public void start() {
Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
topicCountMap.put(topic, new Integer(CoreConstants.THREAD_SIZE));
Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector
.createMessageStreams(topicCountMap);
List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
executor = Executors.newFixedThreadPool(CoreConstants.THREAD_SIZE);
for (KafkaStream<byte[], byte[]> stream : streams) {
System.out.println("The stream is --"+ stream.iterator().makeNext().topic());
executor.submit(new ListenerThread(stream));
}
try { // without this wait the subsequent shutdown happens immediately before any messages are delivered
Thread.sleep(10000);
} catch (InterruptedException ie) {
}
if (consumerConnector != null) {
consumerConnector.shutdown();
}
if (executor != null) {
executor.shutdown();
}
}
ListenerThread
public class ListenerThread implements Runnable {
private KafkaStream<byte[], byte[]> stream;
public ListenerThread(KafkaStream<byte[], byte[]> msgStream) {
this.stream = msgStream;
System.out.println("----------" + stream.iterator().makeNext().topic());
}
public void run() {
try {
ConsumerIterator<byte[], byte[]> it = stream.iterator();
while (it.hasNext()) {
// MessageAndMetadata<byte[], byte[]> messageAndMetadata =
// it.makeNext();
// String topic = messageAndMetadata.topic();
// byte[] message = messageAndMetadata.message();
System.out.println("111111111111111111111111111");
FileProcessor processor = new FileProcessor();
processor.processFile("LOB_TOPIC", it.next().message());
}
in the above iterator it is not going inside while loop , since the iterator is null. But I'm sure I'm publishing a single message to the same topic and consumer listens to that topic.
Any help would be appreciated

Answer 1

I was having this same issue yesterday. After trying to work with it for a while, I couldn't get it to read from my current topic. So I took following steps
a. Stopped my consumer,
b. stopped the producer,
c. stopped the kafka server
bin/zookeeper-server-stop.sh config/zookeeper.properties
d. stopped the zookeeper
bin/zookeeper-server-stop.sh config/zookeeper.properties
After that I deleted my topic.
bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test
I also deleted the files that was created by following the "Setting up a multi-broker cluster" but I don't think it created the issue.
a. Started the Zookeeper
b. started kafka
c. started producer and send some messages to Kafka
it started to work again. I am not sure if this will help you or not. But seems like that somehow my producer must have got disconnected from the consumer. Hope this helps.