I made a Minio deployment in my 2 Node Kubernetes cluster using YAML files.
I had used an NFS server for the corresponding persistent volume and pvc associated with the same.
Once the pod is running, I created a service to access the console from the browser.
But when tried the URL "http://<host-ip-address:nodePort>", the same was getting redirected to the port 45893 with the message "This site cannot be reached."
Regards,
Vivek
After many tries, got a solution with the help of my friend.
We created a copy of the service and changed the Port to the port to which my Minio console was getting redirected and Nodeport to some random port allowed in the firewall. This resolved the issue.
service.yaml
type: LoadBalancer
ports:
- port: 9000
nodePort: 32767
protocol: TCP
selector:
service_copy.yaml
ports:
- port: 45893
nodePort: 32766
protocol: TCP
selector:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP X.X.X.X <none> 443/TCP 25d
minio-xxx-service NodePort X.X.X.X <none> 9000:32767/TCP 3d23h
minio-xxxx-service-cp NodePort X.X.X.X <none> 45893:32766/TCP 146m
After doing the same, I was able to access the console.
Regards,
Vivek
Related
I am following the instructions here:
https://z2jh.jupyter.org/en/stable/jupyterhub/installation.html
to install locally with kubernetes and minikube a jupyter hub.
Its almost done as can be seen in the pic
the namespace is called k8s-namespace-jose
I had to run the command:
kubectl --namespace get service proxy-public --output jsonpath='{.status.loadBalancer.ingress[].ip}'
In order to get the EXTERNAL-IP shown above.
The thing is that going to:
http://104.196.41.97 does not work (server not responding)
nor the folowing works:
http://104.196.41.97:80
the error that I get is:
What can I do in order to get my jupyterhub in my local server?
EDIT:
In order to have all the info about the loadbalancer:
Name: proxy-public
Namespace: k8s-namespace-jose
Labels: app=jupyterhub
app.kubernetes.io/managed-by=Helm
chart=jupyterhub-2.0.0
component=proxy-public
heritage=Helm
release=helm-release-name-jose
Annotations: meta.helm.sh/release-name: helm-release-name-jose
meta.helm.sh/release-namespace: k8s-namespace-jose
Selector: component=proxy,release=helm-release-name-jose
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.103.131.233
IPs: 10.103.131.233
External IPs: 104.196.41.97
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 32297/TCP
Endpoints: 10.244.0.13:8000
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Due to the minikube installation, probably the address is the address of minikube cluster and not the External-IP
Can you try an alternative to hit the minikube IP instead of the External IP?
sgrigori#sgrigori-vbox:~/Dev/jupyterhub$ minikube ip
192.168.49.2
and use your node port 32297
http://192.168.49.2:32297
In my namespace I have services
k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
blue-service NodePort 10.107.127.118 <none> 80:32092/TCP 60m
demo ClusterIP 10.111.134.22 <none> 80/TCP 3d
I added blue-service to /etc/hosts
It failes again
wget -O- blue-service
--2022-06-13 11:11:32-- http://blue-service/
Resolving blue-service (blue-service)... 10.107.127.118
Connecting to blue-service (blue-service)|10.107.127.118|:80... failed: Connection timed out.
Retrying.
I decided to chech with describe
Name: blue-service
Namespace: default
Labels: app=blue
Annotations: <none>
Selector: app=blue
Type: NodePort
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.107.127.118
IPs: 10.107.127.118
Port: <unset> 80/TCP
TargetPort: 8080/TCP
NodePort: <unset> 32092/TCP
Endpoints: 172.17.0.39:8080,172.17.0.40:8080,172.17.0.41:8080
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Why?
The services you are referring to do not have an external IP (the External IP field is empty) so you cannot access those services.
If you want to access those services, you either need to
Make them a LoadBalancer service type which will give them an external IP
or
Use kubectl port-forward to connect a local port on your machine to the service then use localhost:xxxx to access the service
If you want to map a DNS name to the service, you should look at the External DNS project as mentioned in this answer which will allow you to create DNS entries in your provider's DNS service (if you are running the cluster on a managed platform)
OR, use nip.io if you're only testing
I am running my Kafka on my local k8s cluster. I have 1 instance of broker and zookeeper each. I have logged into Kafka broker container and I am able to create topic, send/receive message on the topic.
Now I want to send message from another pod which is running in a different namespace. Below are services in my Kafka cluster:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kafka-service LoadBalancer 10.108.3.192 localhost 9092:30785/TCP 56m app=kafka,id=0
zoo1 ClusterIP 10.108.132.149 <none> 2181/TCP,2888/TCP,3888/TCP 56m app=zookeeper-1
I am trying to curl the services from a different pod.
I am able to access the zookeeper service.
bash-5.1# curl zoo1.queues.svc.cluster.local:2181
curl: (52) Empty reply from server
But when I try to access kafka-service.queues.svc.cluster.local:9092, I get the below error:
bash-5.1# curl kafka-service.queues.svc.cluster.local:9092
curl: (56) Recv failure: Connection reset by peer
How to access the LoadBalancer service from another living in a different namespace?
Below is the description of the service:
Name: kafka-service
Namespace: queues
Labels: name=kafka
Annotations: <none>
Selector: app=kafka,id=0
Type: LoadBalancer
IP: 10.108.3.192
LoadBalancer Ingress: localhost
Port: kafka-port 9092/TCP
TargetPort: 9092/TCP
NodePort: kafka-port 30785/TCP
Endpoints: 10.1.1.95:9092
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
I've deployed https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka on my on prem k8s cluster.
I'm trying to expose it my using a TCP controller with nginx.
My TCP nginx configmap looks like
data:
"<zookeper-tcp-port>": <namespace>/cp-zookeeper:2181
"<kafka-tcp-port>": <namespace>/cp-kafka:9092
And i've made the corresponding entry in my nginx ingress controller
- name: <zookeper-tcp-port>-tcp
port: <zookeper-tcp-port>
protocol: TCP
targetPort: <zookeper-tcp-port>-tcp
- name: <kafka-tcp-port>-tcp
port: <kafka-tcp-port>
protocol: TCP
targetPort: <kafka-tcp-port>-tcp
Now I'm trying to connect to my kafka instance.
When i just try to connect to the IP and port using kafka tools, I get the error message
Unable to determine broker endpoints from Zookeeper.
One or more brokers have multiple endpoints for protocol PLAIN...
Please proved bootstrap.servers value in advanced settings
[<cp-broker-address-0>.cp-kafka-headless.<namespace>:<port>][<ip>]
When I enter, what I assume are the correct broker addresses (I've tried them all...) I get a time out. There are no logs coming from the nginx controler excep
[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:14 +0000]TCP200000.001
From the pod kafka-zookeeper-0 I'm gettting loads of
[2020-04-08 15:52:02,415] INFO Accepted socket connection from /<ip:port> (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-08 15:52:02,415] WARN Unable to read additional data from client sessionid 0x0, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-04-08 15:52:02,415] INFO Closed socket connection for client /<ip:port> (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
Though I'm not sure these have anything to do with it?
Any ideas on what I'm doing wrong?
Thanks in advance.
TL;DR:
Change the value nodeport.enabled to true inside cp-kafka/values.yaml before deploying.
Change the service name and ports in you TCP NGINX Configmap and Ingress object.
Set bootstrap-server on your kafka tools to <Cluster_External_IP>:31090
Explanation:
The Headless Service was created alongside the StatefulSet. The created service will not be given a clusterIP, but will instead simply include a list of Endpoints.
These Endpoints are then used to generate instance-specific DNS records in the form of:
<StatefulSet>-<Ordinal>.<Service>.<Namespace>.svc.cluster.local
It creates a DNS name for each pod, e.g:
[ root#curl:/ ]$ nslookup my-confluent-cp-kafka-headless
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: my-confluent-cp-kafka-headless
Address 1: 10.8.0.23 my-confluent-cp-kafka-1.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 2: 10.8.1.21 my-confluent-cp-kafka-0.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 3: 10.8.3.7 my-confluent-cp-kafka-2.my-confluent-cp-kafka-headless.default.svc.cluster.local
This is what makes this services connect to each other inside the cluster.
I've gone through a lot of trial and error, until I realized how it was supposed to be working. Based your TCP Nginx Configmap I believe you faced the same issue.
The Nginx ConfigMap asks for: <PortToExpose>: "<Namespace>/<Service>:<InternallyExposedPort>".
I realized that you don't need to expose the Zookeeper, since it's a internal service and handled by kafka brokers.
I also realized that you are trying to expose cp-kafka:9092 which is the headless service, also only used internally, as I explained above.
In order to get outside access you have to set the parameters nodeport.enabled to true as stated here: External Access Parameters.
It adds one service to each kafka-N pod during chart deployment.
Then you change your configmap to map to one of them:
data:
"31090": default/demo-cp-kafka-0-nodeport:31090
Note that the service created has the selector statefulset.kubernetes.io/pod-name: demo-cp-kafka-0 this is how the service identifies the pod it is intended to connect to.
Edit the nginx-ingress-controller:
- containerPort: 31090
hostPort: 31090
protocol: TCP
Set your kafka tools to <Cluster_External_IP>:31090
Reproduction:
- Snippet edited in cp-kafka/values.yaml:
nodeport:
enabled: true
servicePort: 19092
firstListenerPort: 31090
Deploy the chart:
$ helm install demo cp-helm-charts
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
demo-cp-control-center-6d79ddd776-ktggw 1/1 Running 3 113s
demo-cp-kafka-0 2/2 Running 1 113s
demo-cp-kafka-1 2/2 Running 0 94s
demo-cp-kafka-2 2/2 Running 0 84s
demo-cp-kafka-connect-79689c5c6c-947c4 2/2 Running 2 113s
demo-cp-kafka-rest-56dfdd8d94-79kpx 2/2 Running 1 113s
demo-cp-ksql-server-c498c9755-jc6bt 2/2 Running 2 113s
demo-cp-schema-registry-5f45c498c4-dh965 2/2 Running 3 113s
demo-cp-zookeeper-0 2/2 Running 0 112s
demo-cp-zookeeper-1 2/2 Running 0 93s
demo-cp-zookeeper-2 2/2 Running 0 74s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-cp-control-center ClusterIP 10.0.13.134 <none> 9021/TCP 50m
demo-cp-kafka ClusterIP 10.0.15.71 <none> 9092/TCP 50m
demo-cp-kafka-0-nodeport NodePort 10.0.7.101 <none> 19092:31090/TCP 50m
demo-cp-kafka-1-nodeport NodePort 10.0.4.234 <none> 19092:31091/TCP 50m
demo-cp-kafka-2-nodeport NodePort 10.0.3.194 <none> 19092:31092/TCP 50m
demo-cp-kafka-connect ClusterIP 10.0.3.217 <none> 8083/TCP 50m
demo-cp-kafka-headless ClusterIP None <none> 9092/TCP 50m
demo-cp-kafka-rest ClusterIP 10.0.14.27 <none> 8082/TCP 50m
demo-cp-ksql-server ClusterIP 10.0.7.150 <none> 8088/TCP 50m
demo-cp-schema-registry ClusterIP 10.0.7.84 <none> 8081/TCP 50m
demo-cp-zookeeper ClusterIP 10.0.9.119 <none> 2181/TCP 50m
demo-cp-zookeeper-headless ClusterIP None <none> 2888/TCP,3888/TCP 50m
Create the TCP configmap:
$ cat nginx-tcp-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: kube-system
data:
31090: "default/demo-cp-kafka-0-nodeport:31090"
$ kubectl apply -f nginx-tcp.configmap.yaml
configmap/tcp-services created
Edit the Nginx Ingress Controller:
$ kubectl edit deploy nginx-ingress-controller -n kube-system
$kubectl get deploy nginx-ingress-controller -n kube-system -o yaml
{{{suppressed output}}}
ports:
- containerPort: 31090
hostPort: 31090
protocol: TCP
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
My ingress is on IP 35.226.189.123, now let's try to connect from outside the cluster. For that I'll connect to another VM where I have a minikube, so I can use kafka-client pod to test:
user#minikube:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-client 1/1 Running 0 17h
user#minikube:~$ kubectl exec kafka-client -it -- bin/bash
root#kafka-client:/# kafka-console-consumer --bootstrap-server 35.226.189.123:31090 --topic demo-topic --from-beginning --timeout-ms 8000 --max-messages 1
Wed Apr 15 18:19:48 UTC 2020
Processed a total of 1 messages
root#kafka-client:/#
As you can see, I was able to access the kafka from outside.
If you need external access to Zookeeper as well I'll leave a service model for you:
zookeeper-external-0.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: cp-zookeeper
pod: demo-cp-zookeeper-0
name: demo-cp-zookeeper-0-nodeport
namespace: default
spec:
externalTrafficPolicy: Cluster
ports:
- name: external-broker
nodePort: 31181
port: 12181
protocol: TCP
targetPort: 31181
selector:
app: cp-zookeeper
statefulset.kubernetes.io/pod-name: demo-cp-zookeeper-0
sessionAffinity: None
type: NodePort
It will create a service for it:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-cp-zookeeper-0-nodeport NodePort 10.0.5.67 <none> 12181:31181/TCP 2s
Patch your configmap:
data:
"31090": default/demo-cp-kafka-0-nodeport:31090
"31181": default/demo-cp-zookeeper-0-nodeport:31181
Add the Ingress rule:
ports:
- containerPort: 31181
hostPort: 31181
protocol: TCP
Test it with your external IP:
pod/zookeeper-client created
user#minikube:~$ kubectl exec -it zookeeper-client -- /bin/bash
root#zookeeper-client:/# zookeeper-shell 35.226.189.123:31181
Connecting to 35.226.189.123:31181
Welcome to ZooKeeper!
JLine support is disabled
If you have any doubts, let me know in the comments!
I have created a Kafka cluster using the Strimzi Kafka operator on minikube to learn the basics. I am trying to access Kafka inside the minikube environment from my Host and for this I created a Kafka Node port:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: kafka-cluster
spec:
kafka:
version: 2.4.0
replicas: 3
listeners:
plain: {}
tls: {}
external:
type: nodeport
tls: false
overrides:
bootstrap:
nodePort: 32100
brokers:
- broker: 0
nodePort: 32000
- broker: 1
nodePort: 32001
- broker: 2
nodePort: 32002
...
...
...
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kafka-cluster-kafka-0 NodePort 10.96.40.176 <none> 9094:32000/TCP 7d
kafka-cluster-kafka-1 NodePort 10.96.138.2 <none> 9094:32001/TCP 7d
kafka-cluster-kafka-2 NodePort 10.96.209.16 <none> 9094:32002/TCP 7d
kafka-cluster-kafka-bootstrap ClusterIP 10.96.216.169 <none> 9091/TCP,9092/TCP,9093/TCP,9404/TCP 7d
kafka-cluster-kafka-brokers ClusterIP None <none> 9091/TCP,9092/TCP,9093/TCP 7d
kafka-cluster-kafka-exporter ClusterIP 10.96.17.45 <none> 9404/TCP 47d
kafka-cluster-kafka-external-bootstrap NodePort 10.96.252.97 <none> 9094:32100/TCP 7d
kafka-cluster-zookeeper-client ClusterIP 10.96.155.34 <none> 9404/TCP,2181/TCP 7d
kafka-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 7d
All seems to work well so far and I am able to publish messages to the Kafka topic using kafka-cluster-kafka-external-bootstrap (withBootstrapServers("192.168.99.107:32100"). Since I am learning things step-by-step, I wanted to see if I could assign a name instead of referring by the IP address.
Is there an easy way to do it in the Nodeport configuration? I have been stuck on this issue for a week. Appreciate a nudge in the right direction!
That mainly depends on your infrastructure. Minikube is using IP addresses as both node address and loadbalancer address. That is why Strimzi always gives you the address based on the IP address. If you would run it on Kubernetes for example somewhere on AWS, you would out of the box get DNS names instead of the IPs.
Even with Minikube you can configure Strimzi to use some names you specify: https://strimzi.io/docs/latest/full.html#con-kafka-broker-external-listeners-addresses-deployment-configuration-kafka ... but you would need to make sure these route to the Minikube IP address for example by adding them anually to /etc/hosts. But TBH, normally this is not worth it with Minikube.
You should install some Kubernetes bare-metal load balancer, like MetalLB, then use LoadBalancer service instead of NodePort service, get EXTERNAL-IP value of LoadBalancer service and point DNS to this IP.