Kafka deployment on minikube - apache-kafka

Question

Hi i am new to kubernets, i am using minikube single node cluster for local development and testing.
Host: Ubuntu 16.04 LTS.
Minikube: Virtual box running minikube cluster
My requirement is i need to deploy kafka and zookeeper on minikube and should be used to produce or consume messages.
I followed this link and successfully deployed it on minikube its details are below
$ kubectl get services
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kafka-service 10.0.0.15 <pending> 9092:30244/TCP 46m
kubernetes 10.0.0.1 <none> 443/TCP 53m
zoo1 10.0.0.43 <none> 2181/TCP,2888/TCP,3888/TCP 50m
zoo2 10.0.0.226 <none> 2181/TCP,2888/TCP,3888/TCP 50m
zoo3 10.0.0.6 <none> 2181/TCP,2888/TCP,3888/TCP 50m
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-deployment-3583985961-f2301 1/1 Running 0 48m
zookeeper-deployment-1-1598963595-vgx1l 1/1 Running 0 52m
zookeeper-deployment-2-2038841231-tdsff 1/1 Running 0 52m
zookeeper-deployment-3-2478718867-5vjcj 1/1 Running 0 52m
$ kubectl describe service kafka-service
Name: kafka-service
Namespace: default
Labels: app=kafka
Annotations: <none>
Selector: app=kafka
Type: LoadBalancer
IP: 10.0.0.15
Port: kafka-port 9092/TCP
NodePort: kafka-port 30244/TCP
Endpoints: 172.17.0.7:9092
Session Affinity: None
Events: <none>
and i have set KAFKA_ADVERTISED_HOST_NAME to minikube ip(192.168.99.100).
Now for message producer i am using $cat textfile.log | kafkacat -b $(minikube ip):30244 -t mytopic its not publishing the message giving below message
% Auto-selecting Producer mode (use -P or -C to override)
% Delivery failed for message: Local: Message timed out
can any one help how to publish and consume message.

Answer 1

I know that this is quite an old post. Were you able to resolve and run kafka + zookeeper within minikube? I was able to run a simple single cluster kafka and zookeeper deployment successfully using minikube v0.17.1 and produce and consume messages using kafkacat producer and consumer respectively. I was able to run these successfully on uBuntu and Mac OSX. The deployment and service yamls are as below:
zookeeper-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: zookeeper
name: zookeeper
spec:
replicas: 1
template:
metadata:
labels:
app: zookeeper
spec:
containers:
- image: wurstmeister/zookeeper
imagePullPolicy: IfNotPresent
name: zookeeper
ports:
- containerPort: 2181
zookeeper-service.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: zookeeper-service
name: zookeeper-service
spec:
type: NodePort
ports:
- name: zookeeper-port
port: 2181
nodePort: 30181
targetPort: 2181
selector:
app: zookeeper
kafka-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: kafka
name: kafka
spec:
replicas: 1
template:
metadata:
labels:
app: kafka
spec:
containers:
- env:
- name: KAFKA_ADVERTISED_HOST_NAME
value: "192.168.99.100"
- name: KAFKA_ADVERTISED_PORT
value: "30092"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: 192.168.99.100:30181
- name: KAFKA_CREATE_TOPICS
value: "test-topic:1:1"
image: wurstmeister/kafka
imagePullPolicy: IfNotPresent
name: kafka
ports:
- containerPort: 9092
kafka-service.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: kafka-service
name: kafka-service
spec:
type: NodePort
ports:
- name: kafka-port
port: 9092
nodePort: 30092
targetPort: 9092
selector:
app: kafka
You can test your deployment by installing kafkacat client and running the following commands on separate terminal windows:
echo "Am I receiving this message?" | kafkacat -P -b 192.168.99.100:30092 -t test-topic
kafkacat -C -b 192.168.99.100:30092 -t test-topic
% Reached end of topic test-topic [0] at offset 0
Am I receiving this message?
I was able to successfully run this on minikube versions v0.17.1 and v0.19.0. If you want to run this on minikube versions v0.21.1 and v0.23.0, please refer to my reply to the post here: Kafka inaccessible once inside Kubernetes/Minikube
Thanks.

Answer 2

I followed as shown below.
$ git clone https://github.com/d1egoaz/minikube-kafka-cluster
$ cd minikube-kafka-cluster
$ kubectl apply -f 00-namespace/
$ kubectl apply -f 01-zookeeper/
$ kubectl apply -f 02-kafka/
$ kubectl apply -f 03-yahoo-kafka-manager/
$ kubectl get svc -n kafka-ca1 (get the port of kafka manager like 31445)
$ minikube ip
Put http://minikube-ip:port in browser to get ui page of kafka manager.
source https://technology.amis.nl/2019/03/24/running-apache-kafka-on-minikube/

Answer 3

You used a service with type = LoadBalancer which is used for cloud provider (you can see the service waiting for an external ip address .. pending state ... which will never happen). In your case you should try with NodePort.

Answer 4

Going off the previous answer and your comment here is very basic sample code for a Kafka service with type=Nodeport.
apiVersion: v1
kind: Service
metadata:
name: kafka-service
labels:
name: kafka
spec:
ports:
- port: 9092
name: kafka-port
protocol: TCP
selector:
app: kafka
type: NodePort