Why can I not use port 80 when using K3s Kubernetes - kubernetes

I have a simple NodeJS project running on a K3s cluster on a Raspberry Pi 4. The cluster has a service to expose it. The code is as follows...
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 3000
targetPort: 3000
I want to try and use port 80 instead of 3000 so I try...
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 80
targetPort: 3000
But it can't use the port.
Warning FailedScheduling 5m 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
Why am I having issues?
Update
Per the answer I tried...
pi#raspberrypi:~ $ sudo netstat -tulpn | grep :80
pi#raspberrypi:~ $ sudo ss -tulpn | grep :80
pi#raspberrypi:~ $
My guess is this is a K3s or Pi limitation.
Update 2
When I run kubectl get service --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 24d
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 24d
kube-system metrics-server ClusterIP 10.43.48.200 <none> 443/TCP 24d
kube-system traefik-prometheus ClusterIP 10.43.89.96 <none> 9100/TCP 24d
kube-system traefik LoadBalancer 10.43.65.154 192.168.x.xxx 80:31065/TCP,443:32574/TCP 24d
test-namespace app-tier LoadBalancer 10.43.190.179 192.168.x.xxx 3000:31500/TCP 4d

k3s comes with a pre-installed traefik ingress controller which binds to 80, 443 and 8080 on the host, alhtough you should have seen that with ss or netstat
You should see this service if you run:
kubectl get service --all-namespaces
Although you should have seen it with netstat or ss if something is using the port if this is the case. But mb this service also failed to deploy but somehow blocks k3s from taking the port.
Another thing I can think of: Are you running the experimental rootless setup?

Related

Rancher Desktop port forwarding not working

I created a service exposing port 8000/TCP and trying to get that port forwarded to my Mac.
$ kubectl describe svc ddb
Name: ddb
Namespace: default
Labels: <none>
Annotations: <none>
Selector: app=local-ddb
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.215.24
IPs: 10.43.215.24
Port: <unset> 8000/TCP
TargetPort: ddb-http-port/TCP
Endpoints: 10.42.0.91:8000
Session Affinity: None
Events: <none>
Port seems indeed open on my Mac (NOTE I found the local port number by looking at the Rancher Desktop GUI)
lsof | grep 56107
Rancher 50831 victorbarbu 84u IPv4 0x8f04a06a4fb87541 0t0 TCP localhost:56107 (LISTEN)
Doing curl http://localhost:56107 just hangs, however running:
$ kubectl run bb -i --tty --image alpine
# curl http://ddb:8000
{"__type":"com.[TRUNCATED]
Works as expected. Why could that be? How do I fix it?

Cannot access external IP of Load Balancer in a Kubernetes cluster

I created a load balancer service and the describe command returns the following:
Name: minio-service
Namespace: minio
Labels: app=minio
Annotations: <none>
Selector: app=minio
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.43.150.49
IPs: 10.43.151.50
LoadBalancer Ingress: 192.168.31.12, 192.168.32.13, 192.168.33.14
Port: <unset> 9012/TCP
TargetPort: 9011/TCP
NodePort: <unset> 30777/TCP
Endpoints: 10.42.10.110:9011,10.42.10.111:9011
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
If I try curl http://192.168.31.12:9012 it returns:
curl: (7) Failed to connect to 192.168.31.12 port 9012: Connection
timed out
Furthermore, I observed something strange.
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
antonis-dell Ready control-plane,master 4h42m v1.21.2+k3s1 192.168.31.12 <none> Ubuntu 18.04.1 LTS 4.15.0-147-generic containerd://1.4.4-k3s2
knodea Ready <none> 4h9m v1.21.2+k3s1 192.168.32.13 <none> Raspbian GNU/Linux 10 (buster) 5.10.17-v7l+ containerd://1.4.4-k3s2
knodeb Ready <none> 4h2m v1.21.2+k3s1 192.168.33.14 <none> Raspbian GNU/Linux 10 (buster) 5.4.51-v7l+ containerd://1.4.4-k3s2
which means that LoadBalancer Ingress ips are the same with internal ips of nodes in the cluster.
Does anyone know why I have three LoadBalancer Ingress ips which are the same as the internal node ips and how to fix this?

ClusterIP not reachable within the Cluster

I'm struggling with kubernates configurations. What I want to get it's just to reach a deployment within the cluster. The cluster is on my dedicated server and I'm deploying it by using Kubeadm.
My nodes:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 9d v1.19.3
k8s-worker1 Ready <none> 9d v1.19.3
I've a deployment running (nginx basic example)
$ kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 2/2 2 2 29m
I've created a service
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
my-service ClusterIP 10.106.109.94 <none> 80/TCP 20m
The YAML file for my service is the following:
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: nginx-deployment
ports:
- protocol: TCP
port: 80
Now I should expect, if I run curl 10.106.109.94:80 on my k8s-master to get the http answer.. but what I got is:
curl: (7) Failed to connect to 10.106.109.94 port 80: Connection refused
I've tried with NodePort as well and with targetPort and nodePort but the result is the same.
The cluster ip can not be reachable from outside of your cluster that means you will not get any response from the host machine that host your k8s cluster as this ip is not a part of your machine or any other machine rather than its a cluster ip which is used by your cluster CNI network like flunnel,weave.
So to get your services accessible from the outside or atleast from the host machine you have to change the type of your service like NodePort,LoadBalancer,K8s port-forward.
If you can change the service type NodePort then you will get response with any of your host machine ip and the allocated nodeport.
For example,if your k8s-master is 192.168.x.x and nodePort is 33303 then you can get response by
curl http://192.168.x.x:33303
or
curl http://worker_node_ip:33303
if your cluster is in locally installed, then you can install metalLB to get the privilege of load balancer.
You can also use port-forward to get your service accessible from the host that has kubectl client with k8s cluster access.
kubectl port-forward svc/my-service 80:80
kubectl -n namespace port-forward svc/service_name Port:Port

Not able to connect to kafka brokers

I've deployed https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka on my on prem k8s cluster.
I'm trying to expose it my using a TCP controller with nginx.
My TCP nginx configmap looks like
data:
"<zookeper-tcp-port>": <namespace>/cp-zookeeper:2181
"<kafka-tcp-port>": <namespace>/cp-kafka:9092
And i've made the corresponding entry in my nginx ingress controller
- name: <zookeper-tcp-port>-tcp
port: <zookeper-tcp-port>
protocol: TCP
targetPort: <zookeper-tcp-port>-tcp
- name: <kafka-tcp-port>-tcp
port: <kafka-tcp-port>
protocol: TCP
targetPort: <kafka-tcp-port>-tcp
Now I'm trying to connect to my kafka instance.
When i just try to connect to the IP and port using kafka tools, I get the error message
Unable to determine broker endpoints from Zookeeper.
One or more brokers have multiple endpoints for protocol PLAIN...
Please proved bootstrap.servers value in advanced settings
[<cp-broker-address-0>.cp-kafka-headless.<namespace>:<port>][<ip>]
When I enter, what I assume are the correct broker addresses (I've tried them all...) I get a time out. There are no logs coming from the nginx controler excep
[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:12 +0000]TCP200000.000
[08/Apr/2020:15:51:14 +0000]TCP200000.001
From the pod kafka-zookeeper-0 I'm gettting loads of
[2020-04-08 15:52:02,415] INFO Accepted socket connection from /<ip:port> (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-08 15:52:02,415] WARN Unable to read additional data from client sessionid 0x0, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-04-08 15:52:02,415] INFO Closed socket connection for client /<ip:port> (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)
Though I'm not sure these have anything to do with it?
Any ideas on what I'm doing wrong?
Thanks in advance.
TL;DR:
Change the value nodeport.enabled to true inside cp-kafka/values.yaml before deploying.
Change the service name and ports in you TCP NGINX Configmap and Ingress object.
Set bootstrap-server on your kafka tools to <Cluster_External_IP>:31090
Explanation:
The Headless Service was created alongside the StatefulSet. The created service will not be given a clusterIP, but will instead simply include a list of Endpoints.
These Endpoints are then used to generate instance-specific DNS records in the form of:
<StatefulSet>-<Ordinal>.<Service>.<Namespace>.svc.cluster.local
It creates a DNS name for each pod, e.g:
[ root#curl:/ ]$ nslookup my-confluent-cp-kafka-headless
Server: 10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
Name: my-confluent-cp-kafka-headless
Address 1: 10.8.0.23 my-confluent-cp-kafka-1.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 2: 10.8.1.21 my-confluent-cp-kafka-0.my-confluent-cp-kafka-headless.default.svc.cluster.local
Address 3: 10.8.3.7 my-confluent-cp-kafka-2.my-confluent-cp-kafka-headless.default.svc.cluster.local
This is what makes this services connect to each other inside the cluster.
I've gone through a lot of trial and error, until I realized how it was supposed to be working. Based your TCP Nginx Configmap I believe you faced the same issue.
The Nginx ConfigMap asks for: <PortToExpose>: "<Namespace>/<Service>:<InternallyExposedPort>".
I realized that you don't need to expose the Zookeeper, since it's a internal service and handled by kafka brokers.
I also realized that you are trying to expose cp-kafka:9092 which is the headless service, also only used internally, as I explained above.
In order to get outside access you have to set the parameters nodeport.enabled to true as stated here: External Access Parameters.
It adds one service to each kafka-N pod during chart deployment.
Then you change your configmap to map to one of them:
data:
"31090": default/demo-cp-kafka-0-nodeport:31090
Note that the service created has the selector statefulset.kubernetes.io/pod-name: demo-cp-kafka-0 this is how the service identifies the pod it is intended to connect to.
Edit the nginx-ingress-controller:
- containerPort: 31090
hostPort: 31090
protocol: TCP
Set your kafka tools to <Cluster_External_IP>:31090
Reproduction:
- Snippet edited in cp-kafka/values.yaml:
nodeport:
enabled: true
servicePort: 19092
firstListenerPort: 31090
Deploy the chart:
$ helm install demo cp-helm-charts
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
demo-cp-control-center-6d79ddd776-ktggw 1/1 Running 3 113s
demo-cp-kafka-0 2/2 Running 1 113s
demo-cp-kafka-1 2/2 Running 0 94s
demo-cp-kafka-2 2/2 Running 0 84s
demo-cp-kafka-connect-79689c5c6c-947c4 2/2 Running 2 113s
demo-cp-kafka-rest-56dfdd8d94-79kpx 2/2 Running 1 113s
demo-cp-ksql-server-c498c9755-jc6bt 2/2 Running 2 113s
demo-cp-schema-registry-5f45c498c4-dh965 2/2 Running 3 113s
demo-cp-zookeeper-0 2/2 Running 0 112s
demo-cp-zookeeper-1 2/2 Running 0 93s
demo-cp-zookeeper-2 2/2 Running 0 74s
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-cp-control-center ClusterIP 10.0.13.134 <none> 9021/TCP 50m
demo-cp-kafka ClusterIP 10.0.15.71 <none> 9092/TCP 50m
demo-cp-kafka-0-nodeport NodePort 10.0.7.101 <none> 19092:31090/TCP 50m
demo-cp-kafka-1-nodeport NodePort 10.0.4.234 <none> 19092:31091/TCP 50m
demo-cp-kafka-2-nodeport NodePort 10.0.3.194 <none> 19092:31092/TCP 50m
demo-cp-kafka-connect ClusterIP 10.0.3.217 <none> 8083/TCP 50m
demo-cp-kafka-headless ClusterIP None <none> 9092/TCP 50m
demo-cp-kafka-rest ClusterIP 10.0.14.27 <none> 8082/TCP 50m
demo-cp-ksql-server ClusterIP 10.0.7.150 <none> 8088/TCP 50m
demo-cp-schema-registry ClusterIP 10.0.7.84 <none> 8081/TCP 50m
demo-cp-zookeeper ClusterIP 10.0.9.119 <none> 2181/TCP 50m
demo-cp-zookeeper-headless ClusterIP None <none> 2888/TCP,3888/TCP 50m
Create the TCP configmap:
$ cat nginx-tcp-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: kube-system
data:
31090: "default/demo-cp-kafka-0-nodeport:31090"
$ kubectl apply -f nginx-tcp.configmap.yaml
configmap/tcp-services created
Edit the Nginx Ingress Controller:
$ kubectl edit deploy nginx-ingress-controller -n kube-system
$kubectl get deploy nginx-ingress-controller -n kube-system -o yaml
{{{suppressed output}}}
ports:
- containerPort: 31090
hostPort: 31090
protocol: TCP
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
My ingress is on IP 35.226.189.123, now let's try to connect from outside the cluster. For that I'll connect to another VM where I have a minikube, so I can use kafka-client pod to test:
user#minikube:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kafka-client 1/1 Running 0 17h
user#minikube:~$ kubectl exec kafka-client -it -- bin/bash
root#kafka-client:/# kafka-console-consumer --bootstrap-server 35.226.189.123:31090 --topic demo-topic --from-beginning --timeout-ms 8000 --max-messages 1
Wed Apr 15 18:19:48 UTC 2020
Processed a total of 1 messages
root#kafka-client:/#
As you can see, I was able to access the kafka from outside.
If you need external access to Zookeeper as well I'll leave a service model for you:
zookeeper-external-0.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: cp-zookeeper
pod: demo-cp-zookeeper-0
name: demo-cp-zookeeper-0-nodeport
namespace: default
spec:
externalTrafficPolicy: Cluster
ports:
- name: external-broker
nodePort: 31181
port: 12181
protocol: TCP
targetPort: 31181
selector:
app: cp-zookeeper
statefulset.kubernetes.io/pod-name: demo-cp-zookeeper-0
sessionAffinity: None
type: NodePort
It will create a service for it:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-cp-zookeeper-0-nodeport NodePort 10.0.5.67 <none> 12181:31181/TCP 2s
Patch your configmap:
data:
"31090": default/demo-cp-kafka-0-nodeport:31090
"31181": default/demo-cp-zookeeper-0-nodeport:31181
Add the Ingress rule:
ports:
- containerPort: 31181
hostPort: 31181
protocol: TCP
Test it with your external IP:
pod/zookeeper-client created
user#minikube:~$ kubectl exec -it zookeeper-client -- /bin/bash
root#zookeeper-client:/# zookeeper-shell 35.226.189.123:31181
Connecting to 35.226.189.123:31181
Welcome to ZooKeeper!
JLine support is disabled
If you have any doubts, let me know in the comments!

Map service on minikube to host IP

This is my first time running through the Kubernetes tutorial.
I installed Docker, Kubectl and Minikube on a headless Ubuntu server (18.04).
I ran Minikube like this -
minikube start --vm-driver=none
I have a local docker image that run a restful service on port 9110. I create a deployment and expose it like this -
kubectl run hello-node --image=dbtemplate --port=9110 --image-pull-policy=Never
kubectl expose deployment hello-node --type=NodePort
status of my service -
# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-node NodePort 10.98.104.45 <none> 9110:32651/TCP 39m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h2m
# kubectl describe services hello-node
Name: hello-node
Namespace: default
Labels: run=hello-node
Annotations: <none>
Selector: run=hello-node
Type: NodePort
IP: 10.98.104.45
Port: <unset> 9110/TCP
TargetPort: 9110/TCP
NodePort: <unset> 32651/TCP
Endpoints: 172.17.0.5:9110
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
# minikube ip
192.168.1.216
As you can see, the service is available on the internal IP of 172.17.0.5.
Is there some way for me to get this service mapped to/exposed on the IP of the parent host, which is 192.168.1.216. I would like my restful service at 192.168.1.216:9110.
I think minikube tunnel might be what you're looking for. https://github.com/kubernetes/minikube/blob/master/docs/networking.md
Services of type LoadBalancer can be exposed via the minikube tunnel command.