Problems publishing messages with kafka running on mesos DCOS - apache-kafka

Question

I have a small cluster running DCOS. I'm able to successfully install kafka following this guide. running
$ dcos kafka connection
gives
{
"address": [
"10.131.17.126:9475",
"10.131.24.6:9655",
"10.131.14.192:9181"
],
"zookeeper": "master.mesos:2181/dcos-service-kafka",
"dns": [
"broker-0.kafka.mesos:9475",
"broker-1.kafka.mesos:9655",
"broker-2.kafka.mesos:9181"
]
}
I can create topics and I've examined zookeeper with the cli tool and the state appears to be good
get /dcos-service-kafka/brokers/ids/0
{"jmx_port":-1,"timestamp":"1474206074029","endpoints":["PLAINTEXT://10.131.17.126:9475"],"host":"10.131.17.126","version":3,"port":9475}
get /dcos-service-kafka/brokers/ids/1
{"jmx_port":-1,"timestamp":"1474206120002","endpoints":["PLAINTEXT://10.131.24.6:9655"],"host":"10.131.24.6","version":3,"port":9655}
get /dcos-service-kafka/brokers/ids/2
{"jmx_port":-1,"timestamp":"1474206122985","endpoints":["PLAINTEXT://10.131.14.192:9181"],"host":"10.131.14.192","version":3,"port":9181}
However when I try publishing
echo "Hello, World." | ./kafka-console-producer.sh --broker-list 10.131.17.126:9475, 10.131.24.6:9655, 10.131.14.192:9181 --topic topic1
I get
[2016-09-18 18:49:32,909] ERROR Error when sending message to topic topic1 with key: null, value: 13 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
I suspect it might be something to do with private vs. public ip addresses and perhaps host.name in server.properties.
Can anyone give some suggestions as to how I might debug (and hopefully fix!) the problem so I can successfully publish messages?
Thanks
AJ

Answer 1

Update - this does appear to have been caused by missing entries in /etc/hosts. I've updated my terraform script to write these during setup and your example above now works as expected.
Thanks for your help

Answer 2

Edit: For anyone looking in the future. This was a problem in /etc/hosts caused by a terraform script.
Your suspicion is correct. Those are private IP addresses which are not addressable from outside the cluster. In order to communicate with Kafka you will either have to setup a VPN such that those IP addresses become reachable, or run your publishing command on a machine in the cluster.
Also, it looks like you're running on a DC/OS version earlier than 1.8. If you use 1.8, you'll get an easier broker endpoint to use, regardless of the dynamically assigned IP addresses. You can used the named VIP broker.kafka.l4lb.thisdcos.directory:9092 however this is only addressable from machines in the cluster.
Setting up haproxy or nginx to point to the named VIP is also a way to get easy external access to a service (in this case Kafka) running on a DC/OS cluster. You would want to ensure that these proxies run on a public Agent. See here for more details.
Here is an example of installing, producing and consuming from the default Kafka installation:
~ $ dcos package install kafka
Installing Marathon app for package [kafka] version [1.1.11-0.10.0.0]
Installing CLI subcommand for package [kafka] version [1.1.11-0.10.0.0]
New command available: dcos kafka
DC/OS Kafka Service is being installed.
Documentation: https://docs.mesosphere.com/usage/services/kafka/
Issues: https://docs.mesosphere.com/support/
~ $ dcos kafka connection
{
"address": [
"10.0.3.64:9951",
"10.0.3.68:9795",
"10.0.3.66:9531"
],
"zookeeper": "master.mesos:2181/dcos-service-kafka",
"dns": [
"broker-0.kafka.mesos:9951",
"broker-1.kafka.mesos:9795",
"broker-2.kafka.mesos:9531"
],
"vip": "broker.kafka.l4lb.thisdcos.directory:9092"
}
~ $ dcos kafka topic create topic0
{
"message": "Output: Created topic \"topic0\".\n"
}
~ $ dcos node ssh --master-proxy --leader
core#ip-10-0-7-56 ~ $ wget http://download.nextag.com/apache/kafka/0.10.0.1/kafka_2.11-0.10.0.1.tgz
core#ip-10-0-7-56 ~ $ tar xf kafka_2.11-0.10.0.1.tgz
core#ip-10-0-7-56 ~ $ cd kafka_2.11-0.10.0.1
core#ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-producer.sh --broker-list broker.kafka.l4lb.thisdcos.directory:9092 --topic topic0
This is a message
This is another message
^Ccore#ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning
This is a message
This is another message
^CProcessed a total of 2 messages
$ bin/kafka-console-producer.sh --broker-list 10.0.3.64:9951,10.0.3.68:9795,10.0.3.66:9531 --topic topic0
foo
bar
baz
^Ccore#ip-10-0-7-56 ~/kafka_2.11-0.10.0.1 $ bin/kafka-console-consumer.sh --zookeeper master.mesos:2181/dcos-service-kafka --topic topic0 --from-beginning
This is a message
This is another message
foo
bar
baz
^CProcessed a total of 5 messages