Kafka/Zookeeper instance on EC2: How to consume events using public IPs - deployment

Question

I have a Kafka cluster running on EC2. The nodes are having internal as well as public IPs. I have configured the kafka config file server.properties with:
host.name=MY_PUBLIC_KAFKA_NODE_IP
I am looking for a configuration in which a dev-machine can consume the kafka events directly agnostic of internal/public IPs of kafka nodes. Only zookeeper node's public IP is exposed to the developers. With this configuration, when the dev-machine runs:
./bin/kafka-console-consumer.sh --zookeeper MY_PUBLIC_ZOOKEEPER_IP:2181 --topic test --from-beginning
I get following error:
[2016-01-18 19:00:57,388] WARN Fetching topic metadata with correlation id 61 for topics [Set(test)] from broker [id:1,host:ip-172-31-1-91.us-west-2.compute.internal,port:9092] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer.send(SyncProducer.scala:113)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
The error suggests that the broker list contains internal IP of the kafka nodes. Is there a way to fix this configuration?

Answer 1

You need to set advertised.host.name property to an address/IP that will be resolvable by the client, which is in your case the AWS node's public address/IP.
You'll also have to allow inbound access in your AWS security group by adding your client's IP address there.
To do that, go to: Security groups > select the corresponding group > go to Inbound tab in the lower pane > click Edit > Add Rule > Custom TCP rule > TCP > 9092 > select My IP (if accessing AWS console from the client that's going to be used to access Kafka, otherwise add IP manually).