I am running my Kafka on my local k8s cluster. I have 1 instance of broker and zookeeper each. I have logged into Kafka broker container and I am able to create topic, send/receive message on the topic.
Now I want to send message from another pod which is running in a different namespace. Below are services in my Kafka cluster:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kafka-service LoadBalancer 10.108.3.192 localhost 9092:30785/TCP 56m app=kafka,id=0
zoo1 ClusterIP 10.108.132.149 <none> 2181/TCP,2888/TCP,3888/TCP 56m app=zookeeper-1
I am trying to curl the services from a different pod.
I am able to access the zookeeper service.
bash-5.1# curl zoo1.queues.svc.cluster.local:2181
curl: (52) Empty reply from server
But when I try to access kafka-service.queues.svc.cluster.local:9092, I get the below error:
bash-5.1# curl kafka-service.queues.svc.cluster.local:9092
curl: (56) Recv failure: Connection reset by peer
How to access the LoadBalancer service from another living in a different namespace?
Below is the description of the service:
Name: kafka-service
Namespace: queues
Labels: name=kafka
Annotations: <none>
Selector: app=kafka,id=0
Type: LoadBalancer
IP: 10.108.3.192
LoadBalancer Ingress: localhost
Port: kafka-port 9092/TCP
TargetPort: 9092/TCP
NodePort: kafka-port 30785/TCP
Endpoints: 10.1.1.95:9092
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Related
Followed https://strimzi.io/quickstarts/ and https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/ to use GKE internal loadbalancer with Strimzi. After adding the internal load balancer Strimzi provisioned two loadbalancer service with external IP.
Kafka % k get svc -n kafka
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-cluster-kafka-0 LoadBalancer xx.xxx.xx.xxx bb.bb.bbb.bb 9094:30473/TCP 3d1h
my-cluster-kafka-bootstrap ClusterIP xx.xxx.xx.xxx <none> 9091/TCP,9092/TCP,9093/TCP 25d
my-cluster-kafka-brokers ClusterIP None <none> 9090/TCP,9091/TCP,9092/TCP,9093/TCP 25d
my-cluster-kafka-external-bootstrap LoadBalancer xx.xxx.xx.xxx aa.aa.aaa.aa 9094:30002/TCP 3d1h
my-cluster-zookeeper-client ClusterIP xx.xxx.xx.xxx <none> 2181/TCP 25d
my-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 25d
The producer/consumer flow is working inside the cluster using my-cluster-kafka-bootstrap and I can also curl the my-cluster-kafka-external-bootstrap addess aa.aa.aaa.aa:9094 from outside the cluster. However after producing to aa.aa.aaa.aa:9094 from outside the cluster my producer logged the error below.
Connection to node 0 (bb.bb.bbb.bb:9094) could not be established. Broker may not be available.
which seem to indicate my-cluster-kafka-external-bootstrap is forwarding the traffic to my-cluster-kafka-0. And per kubectl get svc -o yaml output only my-cluster-kafka-external-bootstrap was setup as a GKE internal LB. Since there are various firewall rules in our enviroment I suspect that my-cluster-kafka-0 needs to be set up as a GKE internal LB as well for the producer to work. Does this seem to be the issue? How do I update Strimzi to make both LB internal? Thanks.
A relevant question before Strimzi kafka accessing it privately with in GKE. But it didn't help after I turn off tls.
Answering own question. Appearntly Strimzi provision one LB per broker which is the my-cluster-kafka-0 here. The listener config can specify these per broker LBs like this https://strimzi.io/blog/2019/05/13/accessing-kafka-part-4/
# ...
listeners:
# ...
- name: external
port: 9094
type: loadbalancer
tls: true
authentication:
type: tls
configuration:
bootstrap:
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
brokers:
- broker: 0
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
- broker: 1
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
- broker: 2
annotations:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
# ...
I made a Minio deployment in my 2 Node Kubernetes cluster using YAML files.
I had used an NFS server for the corresponding persistent volume and pvc associated with the same.
Once the pod is running, I created a service to access the console from the browser.
But when tried the URL "http://<host-ip-address:nodePort>", the same was getting redirected to the port 45893 with the message "This site cannot be reached."
Regards,
Vivek
After many tries, got a solution with the help of my friend.
We created a copy of the service and changed the Port to the port to which my Minio console was getting redirected and Nodeport to some random port allowed in the firewall. This resolved the issue.
service.yaml
type: LoadBalancer
ports:
- port: 9000
nodePort: 32767
protocol: TCP
selector:
service_copy.yaml
ports:
- port: 45893
nodePort: 32766
protocol: TCP
selector:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP X.X.X.X <none> 443/TCP 25d
minio-xxx-service NodePort X.X.X.X <none> 9000:32767/TCP 3d23h
minio-xxxx-service-cp NodePort X.X.X.X <none> 45893:32766/TCP 146m
After doing the same, I was able to access the console.
Regards,
Vivek
I've deployed https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka on my on prem k8s cluster.
I'm trying to expose it my using a TCP controller with nginx.
My TCP nginx configmap looks like
data:
"<zookeper-tcp-port>": <namespace>/cp-zookeeper:2181
"<kafka-tcp-port>": <namespace>/cp-kafka:9092
And i've made the corresponding entry in my nginx ingress controller
- name: <zookeper-tcp-port>-tcp
port: <zookeper-tcp-port>
protocol: TCP
targetPort: <zookeper-tcp-port>-tcp
- name: <kafka-tcp-port>-tcp
port: <kafka-tcp-port>
protocol: TCP
targetPort: <kafka-tcp-port>-tcp
Now I'm trying to connect to my kafka instance.
When i just try to connect to the IP and port using kafka tools, I get the error message
Unable to determine broker endpoints from Zookeeper.
One or more brokers have multiple endpoints for protocol PLAIN...
Please proved bootstrap.servers value in advanced settings
[<cp-broker-address-0>.cp-kafka-headless.<namespace>:<port>][<ip>]
When I enter, what I assume are the correct broker addresses (I've tried them all...) I get a time out. There are no logs coming from the nginx controler excep
[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:14 +0000]TCP200000.001
From the pod kafka-zookeeper-0 I'm gettting loads of
[2020-04-08 15:52:02,415] INFO Accepted socket connection from /<ip:port> (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-08 15:52:02,415] WARN Unable to read additional data from client sessionid 0x0, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-04-08 15:52:02,415] INFO Closed socket connection for client /<ip:port> (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
Though I'm not sure these have anything to do with it?
Any ideas on what I'm doing wrong?
Thanks in advance.
TL;DR:
Change the value nodeport.enabled to true inside cp-kafka/values.yaml before deploying.
Change the service name and ports in you TCP NGINX Configmap and Ingress object.
Set bootstrap-server on your kafka tools to <Cluster_External_IP>:31090
Explanation:
The Headless Service was created alongside the StatefulSet. The created service will not be given a clusterIP, but will instead simply include a list of Endpoints.
These Endpoints are then used to generate instance-specific DNS records in the form of:
<StatefulSet>-<Ordinal>.<Service>.<Namespace>.svc.cluster.local
It creates a DNS name for each pod, e.g:
[ root#curl:/ ]$ nslookup my-confluent-cp-kafka-headless
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: my-confluent-cp-kafka-headless
Address 1: 10.8.0.23 my-confluent-cp-kafka-1.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 2: 10.8.1.21 my-confluent-cp-kafka-0.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 3: 10.8.3.7 my-confluent-cp-kafka-2.my-confluent-cp-kafka-headless.default.svc.cluster.local
This is what makes this services connect to each other inside the cluster.
I've gone through a lot of trial and error, until I realized how it was supposed to be working. Based your TCP Nginx Configmap I believe you faced the same issue.
The Nginx ConfigMap asks for: <PortToExpose>: "<Namespace>/<Service>:<InternallyExposedPort>".
I realized that you don't need to expose the Zookeeper, since it's a internal service and handled by kafka brokers.
I also realized that you are trying to expose cp-kafka:9092 which is the headless service, also only used internally, as I explained above.
In order to get outside access you have to set the parameters nodeport.enabled to true as stated here: External Access Parameters.
It adds one service to each kafka-N pod during chart deployment.
Then you change your configmap to map to one of them:
data:
"31090": default/demo-cp-kafka-0-nodeport:31090
Note that the service created has the selector statefulset.kubernetes.io/pod-name: demo-cp-kafka-0 this is how the service identifies the pod it is intended to connect to.
Edit the nginx-ingress-controller:
- containerPort: 31090
hostPort: 31090
protocol: TCP
Set your kafka tools to <Cluster_External_IP>:31090
Reproduction:
- Snippet edited in cp-kafka/values.yaml:
nodeport:
enabled: true
servicePort: 19092
firstListenerPort: 31090
Deploy the chart:
$ helm install demo cp-helm-charts
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
demo-cp-control-center-6d79ddd776-ktggw 1/1 Running 3 113s
demo-cp-kafka-0 2/2 Running 1 113s
demo-cp-kafka-1 2/2 Running 0 94s
demo-cp-kafka-2 2/2 Running 0 84s
demo-cp-kafka-connect-79689c5c6c-947c4 2/2 Running 2 113s
demo-cp-kafka-rest-56dfdd8d94-79kpx 2/2 Running 1 113s
demo-cp-ksql-server-c498c9755-jc6bt 2/2 Running 2 113s
demo-cp-schema-registry-5f45c498c4-dh965 2/2 Running 3 113s
demo-cp-zookeeper-0 2/2 Running 0 112s
demo-cp-zookeeper-1 2/2 Running 0 93s
demo-cp-zookeeper-2 2/2 Running 0 74s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-cp-control-center ClusterIP 10.0.13.134 <none> 9021/TCP 50m
demo-cp-kafka ClusterIP 10.0.15.71 <none> 9092/TCP 50m
demo-cp-kafka-0-nodeport NodePort 10.0.7.101 <none> 19092:31090/TCP 50m
demo-cp-kafka-1-nodeport NodePort 10.0.4.234 <none> 19092:31091/TCP 50m
demo-cp-kafka-2-nodeport NodePort 10.0.3.194 <none> 19092:31092/TCP 50m
demo-cp-kafka-connect ClusterIP 10.0.3.217 <none> 8083/TCP 50m
demo-cp-kafka-headless ClusterIP None <none> 9092/TCP 50m
demo-cp-kafka-rest ClusterIP 10.0.14.27 <none> 8082/TCP 50m
demo-cp-ksql-server ClusterIP 10.0.7.150 <none> 8088/TCP 50m
demo-cp-schema-registry ClusterIP 10.0.7.84 <none> 8081/TCP 50m
demo-cp-zookeeper ClusterIP 10.0.9.119 <none> 2181/TCP 50m
demo-cp-zookeeper-headless ClusterIP None <none> 2888/TCP,3888/TCP 50m
Create the TCP configmap:
$ cat nginx-tcp-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: kube-system
data:
31090: "default/demo-cp-kafka-0-nodeport:31090"
$ kubectl apply -f nginx-tcp.configmap.yaml
configmap/tcp-services created
Edit the Nginx Ingress Controller:
$ kubectl edit deploy nginx-ingress-controller -n kube-system
$kubectl get deploy nginx-ingress-controller -n kube-system -o yaml
{{{suppressed output}}}
ports:
- containerPort: 31090
hostPort: 31090
protocol: TCP
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
My ingress is on IP 35.226.189.123, now let's try to connect from outside the cluster. For that I'll connect to another VM where I have a minikube, so I can use kafka-client pod to test:
user#minikube:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-client 1/1 Running 0 17h
user#minikube:~$ kubectl exec kafka-client -it -- bin/bash
root#kafka-client:/# kafka-console-consumer --bootstrap-server 35.226.189.123:31090 --topic demo-topic --from-beginning --timeout-ms 8000 --max-messages 1
Wed Apr 15 18:19:48 UTC 2020
Processed a total of 1 messages
root#kafka-client:/#
As you can see, I was able to access the kafka from outside.
If you need external access to Zookeeper as well I'll leave a service model for you:
zookeeper-external-0.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: cp-zookeeper
pod: demo-cp-zookeeper-0
name: demo-cp-zookeeper-0-nodeport
namespace: default
spec:
externalTrafficPolicy: Cluster
ports:
- name: external-broker
nodePort: 31181
port: 12181
protocol: TCP
targetPort: 31181
selector:
app: cp-zookeeper
statefulset.kubernetes.io/pod-name: demo-cp-zookeeper-0
sessionAffinity: None
type: NodePort
It will create a service for it:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-cp-zookeeper-0-nodeport NodePort 10.0.5.67 <none> 12181:31181/TCP 2s
Patch your configmap:
data:
"31090": default/demo-cp-kafka-0-nodeport:31090
"31181": default/demo-cp-zookeeper-0-nodeport:31181
Add the Ingress rule:
ports:
- containerPort: 31181
hostPort: 31181
protocol: TCP
Test it with your external IP:
pod/zookeeper-client created
user#minikube:~$ kubectl exec -it zookeeper-client -- /bin/bash
root#zookeeper-client:/# zookeeper-shell 35.226.189.123:31181
Connecting to 35.226.189.123:31181
Welcome to ZooKeeper!
JLine support is disabled
If you have any doubts, let me know in the comments!
I have created a Kafka cluster using the Strimzi Kafka operator on minikube to learn the basics. I am trying to access Kafka inside the minikube environment from my Host and for this I created a Kafka Node port:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: kafka-cluster
spec:
kafka:
version: 2.4.0
replicas: 3
listeners:
plain: {}
tls: {}
external:
type: nodeport
tls: false
overrides:
bootstrap:
nodePort: 32100
brokers:
- broker: 0
nodePort: 32000
- broker: 1
nodePort: 32001
- broker: 2
nodePort: 32002
...
...
...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kafka-cluster-kafka-0 NodePort 10.96.40.176 <none> 9094:32000/TCP 7d
kafka-cluster-kafka-1 NodePort 10.96.138.2 <none> 9094:32001/TCP 7d
kafka-cluster-kafka-2 NodePort 10.96.209.16 <none> 9094:32002/TCP 7d
kafka-cluster-kafka-bootstrap ClusterIP 10.96.216.169 <none> 9091/TCP,9092/TCP,9093/TCP,9404/TCP 7d
kafka-cluster-kafka-brokers ClusterIP None <none> 9091/TCP,9092/TCP,9093/TCP 7d
kafka-cluster-kafka-exporter ClusterIP 10.96.17.45 <none> 9404/TCP 47d
kafka-cluster-kafka-external-bootstrap NodePort 10.96.252.97 <none> 9094:32100/TCP 7d
kafka-cluster-zookeeper-client ClusterIP 10.96.155.34 <none> 9404/TCP,2181/TCP 7d
kafka-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 7d
All seems to work well so far and I am able to publish messages to the Kafka topic using kafka-cluster-kafka-external-bootstrap (withBootstrapServers("192.168.99.107:32100"). Since I am learning things step-by-step, I wanted to see if I could assign a name instead of referring by the IP address.
Is there an easy way to do it in the Nodeport configuration? I have been stuck on this issue for a week. Appreciate a nudge in the right direction!
That mainly depends on your infrastructure. Minikube is using IP addresses as both node address and loadbalancer address. That is why Strimzi always gives you the address based on the IP address. If you would run it on Kubernetes for example somewhere on AWS, you would out of the box get DNS names instead of the IPs.
Even with Minikube you can configure Strimzi to use some names you specify: https://strimzi.io/docs/latest/full.html#con-kafka-broker-external-listeners-addresses-deployment-configuration-kafka ... but you would need to make sure these route to the Minikube IP address for example by adding them anually to /etc/hosts. But TBH, normally this is not worth it with Minikube.
You should install some Kubernetes bare-metal load balancer, like MetalLB, then use LoadBalancer service instead of NodePort service, get EXTERNAL-IP value of LoadBalancer service and point DNS to this IP.
I create cluster with kops utility on AWS EC2. Right now I am trying to configure ingress-nginx controller so it routes all traffic in my cluster. I need it handles HTTP, HTTPS and WebSocket connections. Based on this guide I made all required things:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l4.yaml
When I am trying to get all items in ingress-nginx namespace with kubectl -n ingress-nginx get all:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx LoadBalancer 100.71.94.9 a7d3fe1383e344c1d8cb2de671xxxxxx-810xxxxxx.eu-central-1.elb.amazonaws.com 80:32389/TCP,443:31803/TCP 16m
When I open AWS console -> EC2 Instances -> Load Balancer, I can see that ELB has been created, but there OutOfService status on each node under "Instances" tab. So I can't get reach my ELB URL: a7d3fe1383e344c1d8cb2de671xxxxxx-810xxxxxx.eu-central-1.elb.amazonaws.com:
There is some more details about service using kubectl -n ingress-nginx describe service/ingress-nginx
Name: ingress-nginx
Namespace: ingress-nginx
Labels: app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout":"60"...
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 60
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: *
Selector: app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type: LoadBalancer
IP: 100.71.94.9
LoadBalancer Ingress: a7d3fe1383e344c1d8cb2de671xxxxxx-810xxxxxx.eu-central-1.elb.amazonaws.com
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 32389/TCP
Endpoints: <none>
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31803/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 15m service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 15m service-controller Ensured load balancer
Am I missed something?
UPD #1
If I do the same things in EKS cluster, everything works well and ingress-controller appears on each node. Any ideas..?
You need to add security group on the EC2 instances(kubernetes worker nodes) where you have the nginx deployed to allow port 80 and 443 for the security group that was created for ELB.
Edit:
The endpoints section of the service/ingress-nginx service does not have IPs of the nginx pods. Hence when ELB sends a health check request but the request can not reach the pods so health check fails and ELB marks the backend as outofservice.