When I deploy my Application my loadBalancer works just fine.But after few days, my loadBalancer External IP just doesn't work.My pod is running just fine and there is no issue with the logs as well.
I have had this issue with two different Application multiple times now.Not able to debug what's the issue.Have someone faced this type of error before?
For troubleshoot, you can check endpoints of your LoadBalancer service.
You need to see your pod's ip addresses and forwarded ports according your service like below.
$ kubectl get ep nginx
NAME ENDPOINTS AGE
nginx 100.96.5.44:80 17s
And describe
$ kubectl describe ep nginx
Then check logs of affected pod.
$ kubectl get pods
$ kubectl logs -f <pod-name>
ExternalIP addresses created to forward traffic to internal backends(pods)
If there is something, please provide your Yaml files, and outputs.
Related
I Installed K8S with Helm Charts on EKS but the Loadbalancer EXTERNAL IP is in pending state , I see that EKS does support the service Type : LoadBalancer now.
Is it something I will have to check at the network outgoing traffic level ? Please share your experience if any.
Tx,
The Loadbalancer usually takes some seconds or a few minutes to provision you an IP.
If after 5 minutes the IP isn't provisioned:
- run kubectl get svc <SVC_NAME> -o yaml and if there is any different annotation set.
By default services with Type:LoadBalancer are provisioned with Classic Load Balancers automatically. Learn more here.
If you wish to use Network load Balancers you have to use the annotation:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
The process is really automatic, you don't have to check for network traffic.
You can check if there is any issue with the Helm Chart you are deploying by manually creating a service with loadbalancer type and check if it gets provisioned:
$ kubectl run --generator=run-pod/v1 nginx --image=nginx --port=80
pod/nginx created
$ kubectl get pod nginx
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 34s
$ kubectl expose pod nginx --type=LoadBalancer
service/nginx exposed
$ kubectl get svc nginx -w
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.1.63.178 <pending> 80:32522/TCP 7s
nginx LoadBalancer 10.1.63.178 35.238.146.136 80:32522/TCP 42s
In this example the LoadBalancer took 42s to be provisioned. This way you can verify if the issue is on the Helm Chart or something else.
If Kubernetes is running in an environment that doesn't support LoadBalancer services, the load balancer will not be provisioned, but the service will still behave like a NodePort service, your cloud/K8 engine should support LoadBalancer Service.
In that case, if you manage to add EIP or VIP to your node then you can attach to the EXTERNAL-IP of your TYPE=LoadBalancer in the K8 cluster, for example attaching the EIP/VIP address to the node 172.16.2.13.
kubectl patch svc ServiceName -p '{"spec": {"type": "LoadBalancer", "externalIPs":["172.16.2.13"]}}'
How do I use minikube's (cluster's) DNS? I want to receive all IP addresses associated with all pods for selected headless service? I don’t want to expose it outside the cluster. I am currently creating back-end layer.
As stated in the following answer:
What exactly is a headless service, what does it do/accomplish, and what are some legitimate use cases for it?
„Instead of returning a single DNS A record, the DNS server will return multiple A records for the service, each pointing to the IP of an individual pod backing the service at that moment.”
Thus the pods in back-end layer can communicate to each other.
I can’t use dig command. It is not installed in minikube. Eventually how do I install it? There is no apt available.
I hope this explains more accurately what I want to achieve.
You mentioned that you want to receive IP addresses associated with pods for selected service name for testing how does headless service work.
For only testing purposes you can use port-forwarding. You can forward traffic from your local machine to dns pod in your cluster. To do this, you need to run:
kubectl port-forward svc/kube-dns -n kube-system 5353:53
and it will expose kubs-dns service on your host. Then all you need is to use dig command (or alternative) to query the dns server.
dig #127.0.0.1 -p 5353 +tcp +short <service>.<namespace>.svc.cluster.local
You can also test your dns from inside of cluster e.g. by running a pod with interactive shell:
kubectl run --image tutum/dnsutils dns -it --rm -- bash
root#dns:/# dig +search <service>
Let me know it it helped.
As illustrated in kubernetes/minikube issue 4397
Containers don't have an IP address by default.
You'll want to use minikube service or minikube tunnel to get endpoint information.
See "hello-minikube/ Create a service":
By default, the Pod is only accessible by its internal IP address within the Kubernetes cluster.
To make the hello-node Container accessible from outside the Kubernetes virtual network, you have to expose the Pod as a Kubernetes Service.
Expose the Pod to the public internet using the kubectl expose command:
kubectl expose deployment hello-node --type=LoadBalancer --port=8080
The --type=LoadBalancer flag indicates that you want to expose your Service outside of the cluster.
View the Service you just created:
kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-node LoadBalancer 10.108.144.78 <pending> 8080:30369/TCP 21s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
On Minikube, the LoadBalancer type makes the Service accessible through the minikube service command.
Run the following command:
minikube service hello-node
I have run a Hello World application using the below command.
kubectl run hello-world --replicas=2 --labels="run=load-balancer-example" --image=gcr.io/google-samples/node-hello:1.0 --port=8080
Created a service as below
kubectl expose deployment hello-world --type=NodePort --name=example-service
The pods are running
NAME READY STATUS RESTARTS AGE
hello-world-68ff65cf7-dn22t 1/1 Running 0 2m20s
hello-world-68ff65cf7-llvjt 1/1 Running 0 2m20s
Service:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
example-service NodePort 10.XX.XX.XX <none> 8080:32023/TCP 66s
Here, I am able to test it through curl inside the cluster.
curl http://10.XX.XX.XX:8080
Hello Kubernetes!
How can I access this service outside my cluster? (example, through laptop browser)
you shoud try
http://IP_OF_KUBERNETES:32023
IP_OF_KUBERNETES can be your master IP your worker IP
when you expose a port in kubernetes .It expose that port in all of your server in cluster.Imagine you have two worker node with IP1 and IP2
and one pode is running in IP1 and in worker2 there is no pods but you can access your pod by
http://IP1:32023
http://IP2:32023
You should be able to access it outside the cluster using NodePort assingned(32023). Please paste following http://<IP>:<Port> in your browser and you will able to access your app:
http://<MASTER/WORKER_IP>:32023
There are answers already provided, but I felt like this topic needed some consolidation.
This seems to be fairly easy. NodePort actually exposes your application as the name says on the port of each node. So all you have to do is just find the IP address of the Node on which the pod is. You can do it by running:
kubectl get pods -o wide so you can find the IP or name of the node on which the pod is, then just follow what previous answers state: so http://<MASTER/WORKER_IP>:PORT
There is more methods:
You can deploy Ingress Controller and configure Ingress so the application will be reachable through the internet.
You can also use kubectl proxy to expose ClusterIP service outside of the cluster. Like in this example with Dashboard.
Another way is to use LoadBalancer type, which requires underlying cloud infrastructure.
If you are using minikube you can try to run minikube service list to check your exposed services and their IP.
You can access your service using MasterIP or WorkerIP. If you are planning to use it in production or in a more reliable way you should create a service with type LoadBalancer. And use load balancers IP to access it.
If you are using any cloud env, make sure the firewall rules allow incoming traffic.
This will take care of redirecting request to which ever node the pod is running on. Else you will have to manually hit masterIP or workerIP depending on where the pod is running. If the pod gets moved to different node, you will have to change the ip you are hitting
Service Load Balancer
I've deployed the Rancher Helm chart to my Kubernetes cluster and want to access the Rancher API/UI from another pod (i.e. a pod running an ingress-controller).
when I list the services and the endpoints. The IP addresses differ:
$ kubectl get ep | grep rancher
release-name-rancher 10.200.23.13:80 18h
and
$ kubectl get services | grep rancher
release-name-rancher ClusterIP 10.100.200.253 <none> 80/TCP 18h
Within the container of the client (i.e. the ingress controller), I see the service beeing represented with the service's ClusterIP:
$ env | grep RELEASE_NAME_RANCHER_SERVICE_HOST
RELEASE_NAME_RANCHER_SERVICE_HOST=10.100.200.253
Trying to reach the backend via the IP address in the Env does not work (curl 10.100.200.253 just delivers no response and blocks forever).
Trying to reach the backend via the endpoint address works:
$ curl 10.200.23.13
Found.
I'm quite confused why the endpoint IP address and the ClusterIP address differ and why is it not possible to connect to the ClusterIP address. Any hints to polish my understanding?
In Kubernetes, every Pod and Service gets its own IP address. The kubectl get services IP address is the Kubernetes-internal address of the Service; the get ep address address of the Pod behind it. The Service actually acts like a load balancer, and there can be multiple Pods attached to it. The Kubernetes Service documentation goes into a lot of detail about what exactly is happening here.
Kubernetes also provides an internal DNS service that can resolve Service names. You generally shouldn't use any of these IP addresses directly; instead, use the host name release-name-rancher.default.svc.cluster.local (or replace "default" if you're running in some other Kubernetes namespace).
While the ..._SERVICE_HOST environment variable you reference is supported and documented, I'd avoid using it. Of particular note, if you helm install or kubectl apply a large set of resources at once and the Pod gets created before the Service, you'll be in a consistent state except that the Pod won't actually have this environment variable. In a Helm land where Services don't have fixed names, the environment variable name won't be fixed either. Prefer the DNS name.
For development purposes I try to use Minikube. I want to test how my application will catch an event of exposing a service and assigning an External-IP.
When I exposed a service in Google Container Engine quick start tutorial I could see an event of External IP assignment with:
kubectl get services --watch
I want to achieve the same with Minikube (if possible).
Here is how I try to set things up locally on my OSX development machine:
minikube start --vm-driver=xhyve
minikube addons enable ingress
kubectl run echoserver --image=gcr.io/google_containers/echoserver:1.4 --port=8080
kubectl expose deployment echoserver --type="LoadBalancer"
kubectl get services --watch
I see the following output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver LoadBalancer 10.0.0.138 <pending> 8080:31384/TCP 11s
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 4m
External-Ip field never gets updated and shows pending phase. Is it possible to achieve external IP assignment with Minikube?
On GKE or AWS installs, the external IP comes from the cloud support that reports back to kube API the address that the created LB was assigned.
To have the same on minikube you'd have to run some kind of an LB controller, ie. haproxy one, but honestly, for minikube it makes little sense, as you have single IP that you know in advance by minikube ip so you can use NodePort with that knowledge. LB solution would require setting some IP rangethat can be mapped to particular nodeports, as this is effectively what LB will do - take traffic from extIP:extPort and proxy it to minikubeIP:NodePort.
Unless your use case prevents you from it, you should consider Ingress as the way of ingesting traffic to your minikube.
If you want to emulate external IP assignment event (like the one you can observe using GKE or AWS), this can be achieved by applying the following patch on your sandbox kubernetes:
kubectl run minikube-lb-patch --replicas=1 --image=elsonrodriguez/minikube-lb-patch:0.1 --namespace=kube-system
https://github.com/elsonrodriguez/minikube-lb-patch#assigning-external-ips