Kafka 0.9 cant get Producer and Consumer API working - apache-kafka

Question

Trying the new 0.9 Consumer API and Producer API. But just cant seem to get it working. I have a producer that produces 100 messages per partition of a topic with two paritions. My code reads like
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("acks", "1");
props.put("retries", 0);
props.put("batch.size", 2);
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
System.out.println("Created producer");
for (int i = 0; i < 100; i++) {
for (int j = 0; j < 2; j++) {
producer.send(new ProducerRecord<>("burrow_test_2", j, "M_"+ i + "_" + j + "_msg",
"M_" + i + "_" + j + "_msg"));
System.out.println("Sending msg " + i + " into partition " + j);
}
System.out.println("Sent 200 msgs");
}
System.out.println("Closing producer");
producer.close();
System.out.println("Closed producer");
Now producer.close takes a really long time to close, post which I assume any buffers are flushed, and messages written to the tail of the log.
Now my consumer , I would like to read the specified number of messages before quitting. I chose to manually commit offset as I read. The code reads like
int noOfMessagesToRead = Integer.parseInt(args[0]);
String groupName = args[1];
System.out.println("Reading " + noOfMessagesToRead + " for group " + groupName);
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", groupName);
//props.put("enable.auto.commit", "true");
//props.put("auto.commit.interval.ms", "1000");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("burrow_test_2"));
//consumer.seekToEnd(new TopicPartition("burrow_test_2", 0), new TopicPartition("burrow_test_2", 1));
int noOfMessagesRead = 0;
while (noOfMessagesRead != noOfMessagesToRead) {
System.out.println("Polling....");
ConsumerRecords<String, String> records = consumer.poll(0);
for (ConsumerRecord<String, String> record : records) {
System.out.printf("offset = %d, key = %s, value = %s", record.offset(), record.key(),
record.value());
noOfMessagesRead++;
}
consumer.commitSync();
}
}
But my consumer is always stuck on the poll call (never returns even though I have specified timeout as 0).
Now just to confirm, I tried consuming from command line console consumer provided by Kafka bin/kafka-console-consumer.sh --from-beginning --new-consumer --topic burrow_test_2 --bootstrap-server localhost:9092.
Now this seems to read only to only consume messages produced before I started the java producer.
So questions are
Whats the problem with Java producer above
Why is the consumer not polling anything (I have older messages that are being read just fine by console consumer).
UPDATE: My bad. I had port forwarding enabled from localhost to a remote machine running Kafka, ZK. When this was the case there seems to be some issue. Running on localhost seemed to produce the message.
My only remaining question is that with consumer API, I am unable to seekan offset. I tried both seekToEnd and seekToBeginning methods, both threw an exception saying no record found. So what is the way for consumer to seek an offset. Is auto.offset.reset consumer property the only option ?