I am making my first steps with Kubernetes and I have some difficulties.
I have a pod with it's service defined as NodePort at port 30010.
I have a load balancer configured in front of this Kubernetes cluster where port 8443 directs traffic to this 30010 port.
when I try to access this pod from outside the cluster to port 8443 the pod is not getting any connections but I can see the incoming connections via tcptrack in the host in port 30010 with means the load balancer is doing it's job.
when I do curl -k https://127.0.0.1:30010 in the host I get a response from the pods.
what am I missing?
how can I debug it?
thanks
Related
Whats the difference between kubectl port-forwarding (which forwards port from local host to the pod in the cluster to gain access to cluster resources) and NodePort Service type ?
You are comparing two completely different things. You should compare ClusterIP, NodePort, LoadBalancer and Ingress.
The first and most important difference is that NodePort expose is persistent while by doing it using port-forwarding, you always have to run kubectl port-forward ... and kept it active.
kubectl port-forward is meant for testing, labs, troubleshooting and not for long term solutions. It will create a tunnel between your machine and kubernetes so this solution will serve demands from/to your machine.
NodePort can give you long term solution and it can serve demands from/to anywhere inside the network your nodes reside.
If you use port forwarding kubectl port forward svc/{your_service} -n {service_namespace} you just need a clusterIP, kubectl will handle the traffic for you. Kubectl will be the proxy for your traffic
If you use nodeport for access your service means you need to open port on the worker nodes.
when you use port forwarding, that is going to cause our cluster to essentially behave like it has a node port service running inside of it without creating a service. This is strictly for the development setting. with one command you will have a node port service.
// find the name of the pod that running nats streaming server
kubectl get pods
kubectl port-forward nats-Pod-5443532542c8-5mbw9 4222:4222
kubectl will set up proxy that will forward any trafic on your local machine to a port on that specific pod.
however, to create a node port you need to write a YAML config file to set up a service. It will expose the port permanently and performs load balancing.
I have a service which is accessible on 8081. If I do via docker-compose or swarm without any specific changing on port it's work.
http://$(minikube ip):8081
but when i run my app via Kubernetes(minikube) is assign a nodePort in range of 30000-32767.
Then i have to call as follow:
http://$(minikube ip):30546
which is not acceptable from my service. Is there any way to map randomly given port to my own defined port?
When call second url then i am getting connection refused
I also used
kubectl port forward my-service 8081
but still no success.
kubectl port-forward command is incorrect. try below one
kubectl port-forward svc/my-service 8081:8081
then you should be able to access the service at http//:127.0.0.1:8081
This answer is not specific to Minikube but applicable to any Kubernetes cluster running inside a docker container.
In order to send a request from the host machine to the Kubernetes pod running in a container, you have to map ports from host machine to all the way to the pod.
Here is how you do it:
Publish the NodePort you want to use inside container to the host machine using --publish or -p.
# Map port 8080 on host machine to 31080 inside the container
docker run -p 8080:31080 ...
Use a custom NodePort when creating the service:
# You need to specify the exposed port as the nodePort value
# Otherwise Kubernetes will generate a random nodePort for you
kubectl create service nodeport myservice --node-port=31080 --tcp=3000:80
The application inside the pod listens to port 80 which is exposed as a service at port 3000. The traffic received at port 31080 on Kubernetes node will be directed at this service.
The query you send to 8080 on your host machine will follow this path:
Request -> Host Machine -> Docker Container -> Kubernetes Node -> Service -> Pod
↑ ↑ ↑ ↑ ↑
localhost:8080 :31080 :31080 :3000 :80
References:
https://docs.docker.com/engine/reference/commandline/run/
https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#-em-service-nodeport-em-
I have an AKS cluster with an nginx ingress controller. Controller has created a service with a type LoadBalancer and Ports section looks like this (from kubectl get service):
80:31141/TCP
If I understand things correctly port 80 is a ClusterIp port that is not reachable from the outside but port 31141 is a port that is a NodePort and is reachable from outside. So I would assume that an Azure Load Balancer is sending traffic to this 31141 port.
I was surprised to find that Azure Load Balancer is set up with a rule:
frontendPort: 80
backendPort: 80
probe (healthCheck): 31141
So it actually does use the nodeport but only as a healthcheck and all traffic is sent to port 80 which presumably functions the same way as 31141.
A curious note is that if I try to reach the node IP at port 80 from a pod I only get "connection refused", but I suppose it does work if traffic comes from a load balancer.
I was not able to find any information about this on internet, so the question is how this really works and why is ALB doing it this way?
P.S. I don't have troubles with connectivity, it works. I am just trying to understand how and why it does behind the scenes.
I think I have figured out how that works (disclaimer: my understanding might not be correct, please correct me if it's wrong).
What happens is that load balanced traffic does not reach the node itself on port 80 nor does it reach it on an opened node port (31141 in my case). Instead the traffic that is sent to the node is not "handled" by the node itself but rather routed further with the help of iptables. So if some traffic hits the node with the destination IP of the LB frontendIP and port 80 it goes to the service and further to the pod.
As for health check I suppose it does not use the same port 80 because the request would not have a destination equal to the external IP (LB frontend IP) and rather the node itself directly, then it uses the service nodePort for that reason.
As I see, you have some misunderstandings about the ingress ports. Let me show you some details about the ingress in AKS.
Ingress info:
From the screenshot, the ports 80 and 443 are the ports of the Azure LB which you can access from the Internet with the public IP associated with the LB, here the public IP is 40.121.64.51. And the ports 31282 and 31869 are the ports of the AKS nodes which you cannot access from the Internet, you can only access them from the vnet through the node private IP.
Azure LB info:
heath probes:
lb rules:
From the screenshots, you can see the health probes and the rules of the Azure LB. It uses them to redirect the traffic from the Internet to the AKS nodes' ports, and the nodes are the backend of the Azure LB.
Hope it helps you understand the traffic of the ingress in AKS.
Update:
The LB rules info:
kubectl exposes commands that can be used to create a Service for an application and assigns an IP address to access it from internet.
As far as I understand, to access any application within Kubernetes cluster there should be a Service resource created and that should have an IP address which is accessible from an external network.
But in case of port-forward how does kubectl create a connection to the application without an IP address which is accessible externally?
To start, it's useful to note and remember that in Kubernetes, every pod gets its own ip address from 10.*, that is usable only within the cluster. Now, the port-forward feature of kubectl simply tunnels the traffic from a specified port at your local host machine to the specified port on the specified pod. API server then becomes, in a sense, a temporary gateway between your local port and the Kubernetes cluster.
kubectl port-forward forwards connections to a local port to a port on a pod. Compared to kubectl proxy, kubectl port-forward is more generic as it can forward TCP traffic while kubectl proxy can only forward HTTP traffic.
kubectl port-forward is useful for testing/debugging purposes so you can access your service locally without exposing it.
Below is the name of the pod and it will forward it's port 6379 to localhost:6379.
kubectl port-forward redis-master-765d459796-258hz 6379:6379
which is the same as
kubectl port-forward pods/redis-master-765d459796-258hz 6379:6379
or
kubectl port-forward deployment/redis-master 6379:6379
or
kubectl port-forward rs/redis-master 6379:6379
or
kubectl port-forward svc/redis-master 6379:6379
kubectl port-forward makes a specific Kubernetes API request. That means the system running it needs access to the API server, and any traffic will get tunneled over a single HTTP connection.
Having this is really useful for debugging (if one specific pod is acting up you can connect to it directly; in a microservice environment you can talk to a back-end service you wouldn't otherwise expose) but it's not an alternative to setting up service objects. When I've worked with kubectl port-forward it's been visibly slower than connecting to a pod via a service, and I've found seen the command just stop after a couple of minutes. Again these aren't big problems for debugging, but they're not what I'd want for a production system.
If you want to forward to a different port in localhost. Try this
kubectl port-forward <pod-name> <locahost-port>:<pod-port>
kubectl port-forward sample-pod-sadasds-sxawdd 8090:6379
The above command forwards to localhost 8090 from pod 6379
The port-forward command, Forwards one (or more) local ports to a pod.
This command is very useful for example in blue/green deployments where you would want to troubleshoot a misbehaving pod.
To take things even further, you could even execute some preliminary tests to the pods you feel could be more error-prone right inside your CI/CD pipeline in Jenkins by using multiple conditions, declarative pipeline.
Usage examples:
Listen on port 8888 locally, forwarding to 5000 in the pod
kubectl port-forward pod/mypod 8888:5000
Listen on port 8888 on all addresses, forwarding to 5000 in the pod
kubectl port-forward --address 0.0.0.0 pod/mypod 8888:5000
Listen on a random port locally, forwarding to 5000 in the pod
kubectl port-forward pod/mypod :5000
Listen on port 8888 on localhost and selected IP, forwarding to 5000 in the pod
kubectl port-forward --address localhost,10.19.21.23 pod/mypod 8888:5000
Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod
kubectl port-forward pod/mypod 5000 6000
Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in a pod selected by the deployment
kubectl port-forward deployment/mydeployment 5000 6000
Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in a pod selected by the service
kubectl port-forward service/myservice 5000 6000
To access something inside the cluster, there ae a couple of different options available to,
Cluster IP service with Ingress-Nginx
NodePort Service to expose the pod directly to the outside world.
Above both approach will require to write config file, In case if you want to access a pod without writing a config file then it comes to third option.
Port Forward: We can run a command at our terminal that tells our kubernets cluster to port-forward a port off a very specific pod inside of our cluster when we use this port forwarding thing that's going to cause our cluster to essentially behaves as though it has a node port service running inside it. It's going to expose this pod or a very specific port on it to the outside world and allow us to connect to it directly from our local machine.
Let's go by an example:
const stan = nats.connect('ticketing', 'abc', {
url: 'http://localhost:5000',
});
Our goal is to establish a connection between stan and a pod inside a kubernets cluster.
first we will need the pod name, you can get the name by command kubectl get pods
kubectl get pods
I am assuming my pod name is nats-depl-855d477f4d-xgbd7, and it is accessiable via a cluster IP service
apiVersion: v1
kind: Service
metadata:
name: nats-srv
spec:
selector:
app: nats
ports:
- name: client
protocol: TCP
port: 4222
targetPort: 4222
now to establish the connection run the below command:
kubectl port-forward nats-depl-855d477f4d-xgbd7 5000:4222
5000: is the port of my local machine
4222 : is the port of the pod I want to get access
kubectl port-forward is the easiest communication method with the Pod, but under the hood, this method is much more complicated. The communication goes through several Kubernetes components, so if anything is broken in the communication path, you will not be able to talk to the pod, even if the pod itself is accessible via regular communication channels.
The server proxy runs via kubectl port-forward command forwards the connection to the Kubernetes API Server living in Master node, then the API Server delivers the connection to the Kubelet on the node hosting your Pod, and then the Kubelet forwards the connection to the application running in the pod's container.
NOTE
The application in the container must be bound to a port on the
loopback device for the Kubelet to reach it. If it listens only on the
pod’s eth0 network interface, you won’t be able to reach it with the
kubectl port-forward command.
Marko Lukša "Kubernetes in Action, Second Edition"
TIP
kubectl port-forward allows you to forward communication also to Services and has several other useful features. Run kubectl port-forward --help to learn more.
I've a nginx service exposed via NodePort. According to the documentation, I should now be able to hit the service on $NODE_IP:$NODE_PORT for all my K8 worker IPs. However, I'm able to access the service via curl on only the node that hosts the actual nginx pod. Any idea why?
I did verify using netstat that kube-proxy is listening on $NODE_PORT on all the hosts. Somehow, the request is not being forwarded to the actual pod by kube-proxy.
This turned out to be an issue with the security group associated with the workers. I had opened only the ports in the --service-node-port-range. This was not enough because I was deploying nginx on port 80 and kube-proxy tried to forward the request to the pod's IP on port 80 but was being blocked.