Running Kafka Cluster with multiple Brokers on local minikube - apache-kafka

Question

For testing purposes I try to create a Kafka Cluster on my local minikube. The Cluster must be reachable from outside of the Kubernetes.
When I produce/consume from inside the pods there is no problem, everything works just fine.
When I produce from my local machine with
bin/kafka-console-producer.sh --topic mytopic --broker-list 192.168.99.100:32767
where 192.168.99.100 is my minikube-ip and 32767 is the node port of the kafka service.
I get the following Error Message:
>testmessage
>[2018-04-30 11:55:04,604] ERROR Error when sending message to topic ams_stream with key: null, value: 11 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for ams_stream-0: 1506 ms has passed since batch creation plus linger time
When I consume from my local machine I get the following warnings:
[2018-04-30 10:22:30,680] WARN Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-04-30 10:23:46,057] WARN Connection to node 8 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-04-30 10:25:01,542] WARN Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2018-04-30 10:26:17,008] WARN Connection to node 5 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
The Broker IDs are right, so it looks like I can at least reach the brokers
Edit:
I think that the Problem may be, that the service is routing me "randomly" to any of my brokers, but he needs to route me to the leader of the topic.
Could this be the Problem? Does anybody know a way around this Problem?
Additional Information:
I'm using the wurstmeister/kafka and digitalwonderland/zookeeper images
I started using the DellEMC Tutorial (and the linked one from defuze.org)
This did not work out for me so I made some changes in the kafka-service.yml (1) and the kafka-cluster.yml (2)
kafka-service.yml
added a fixed NodePort
removed id from the selector
kafka-cluster.yml
added replicas to the specification
removed id from the label
changed the broker id to be generated by the last number from the IP
replaced deprecated values advertised_host_name / advertised_port with
listeners ( pod-ip:9092 ) for communication inside the k8s
advertised_listeners ( minikube-ip:node-port ) for communication with applications outside the kubernetes
1 - kafka-service.yml:
---
apiVersion: v1
kind: Service
metadata:
name: kafka-service
labels:
name: kafka
spec:
type: NodePort
ports:
- port: 9092
nodePort: 32767
targetPort: 9092
protocol: TCP
selector:
app: kafka
type: LoadBalancer
2 - kafka-cluster.yml:
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: kafka-b
spec:
replicas: 3
template:
metadata:
labels:
app: kafka
spec:
containers:
- name: kafka
image: wurstmeister/kafka
ports:
- containerPort: 9092
env:
- name: HOSTNAME_COMMAND
value: "ifconfig |grep 'addr:172' |cut -d':' -f 2 |cut -d ' ' -f 1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: zk1:2181
- name: BROKER_ID_COMMAND
value: "ifconfig |grep 'inet addr:172' | cut -d'.' -f '4' | cut -d' ' -f '1'"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL://192.168.99.100:32767"
- name: KAFKA_LISTENERS
value: "INTERNAL://_{HOSTNAME_COMMAND}:9092"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL:PLAINTEXT"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL"
- name: KAFKA_CREATE_TOPICS
value: mytopic:1:3