Find why i am getting 502 Bad gateway error on kubernetes - kubernetes

I am using kubernetes. I have Ingress service which talks my container service. We have exposed a webapi which works all fine. But we keep getting 502 bad gateway error. I am new to kubernetes and i have no clue how to go about debugging this issue. Server is a nodejs server connected to database. Is there anything wrong with configuration?
My Deployment file--
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-pod
spec:
replicas: 1
template:
metadata:
labels:
app: my-pod
spec:
containers:
- name: my-pod
image: my-image
ports:
- name: "http"
containerPort: 8086
resources:
limits:
memory: 2048Mi
cpu: 1020m
---
apiVersion: v1
kind: Service
metadata:
name: my-pod-serv
spec:
ports:
- port: 80
targetPort: "http"
selector:
app: my-pod
My Ingress Service:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: gateway
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: abc.test.com
http:
paths:
- path: /abc
backend:
serviceName: my-pod-serv
servicePort: 80

In Your case:
I think that you get this 502 gateway error because you don't have Ingress controller configured correctly.
Please try do do it with installed Ingress like in example below. It will do all automatically.
Nginx Ingress step by step:
1) Install helm
2) Install nginx controller using helm
$ helm install stable/nginx-ingress --name nginx-ingress
It will create 2 services. You can get their details via
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.39.240.1 <none> 443/TCP 29d
nginx-ingress-controller LoadBalancer 10.39.243.140 35.X.X.15 80:32324/TCP,443:31425/TCP 19m
nginx-ingress-default-backend ClusterIP 10.39.252.175 <none> 80/TCP 19m
nginx-ingress-controller - in short, it's dealing with requests to Ingress and directing
nginx-ingress-default-backend - in short, default backend is a service which handles all URL paths and hosts the nginx controller doesn't understand
3) Create 2 deployments (or use yours)
$ kubectl run my-pod --image=nginx
deployment.apps/my-pod created
$ kubectl run nginx1 --image=nginx
deployment.apps/nginx1 created
4) Connect to one of the pods
$ kubectl exec -ti my-pod-675799d7b-95gph bash
And add additional line to the output to see which one we will try to connect later.
$ echo "HELLO THIS IS INGRESS TEST" >> /usr/share/nginx/html/index.html
$ exit
5) Expose deployments.
$ kubectl expose deploy nginx1 --port 80
service/nginx1 exposed
$ kubectl expose deploy my-pod --port 80
service/my-pod exposed
This will automatically create service and will looks like
apiVersion: v1
kind: Service
metadata:
labels:
app: my-pod
name: my-pod
selfLink: /api/v1/namespaces/default/services/my-pod
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: my-pod
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
6) Now its the time to create Ingress.yaml and deploy it. Each rule in ingress need to be specified. Here I have 2 services. Each service specification starts with -host under rule parameter.
Ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: two-svc-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: my.pod.svc
http:
paths:
- path: /pod
backend:
serviceName: my-pod
servicePort: 80
- host: nginx.test.svc
http:
paths:
- path: /abc
backend:
serviceName: nginx1
servicePort: 80
$ kubectl apply -f Ingress.yaml
ingress.extensions/two-svc-ingress created
7) You can check Ingress and hosts
$ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
two-svc-ingress my.pod.svc,nginx.test.svc 35.228.230.6 80 57m
8) Eplanation why I installed Ingress.
Connect to the ingress controller pod
$ kubectl exec -ti nginx-ingress-controller-76bf4c745c-prp8h bash
www-data#nginx-ingress-controller-76bf4c745c-prp8h:/etc/nginx$ cat /etc/nginx/nginx.conf
Because I have installed nginx ingress earlier, after deploying Ingress.yaml, the nginx-ingress-controller found changes and automatically added necessary code.
In this file you should be able to find whole configuration for two services. I will not copy configuration but only headers.
start server my.pod.svc
start server nginx.test.svc
www-data#nginx-ingress-controller-76bf4c745c-prp8h:/etc/nginx$ exit
9) Test
$ kubectl get svc to get your nginx-ingress-controller external IP
$ curl -H "HOST: my.pod.svc" http://35.X.X.15/
default backend - 404
$ curl -H "HOST: my.pod.svc" http://35.X.X.15/pod
<!DOCTYPE html>
...
</html>
HELLO THIS IS INGRESS TEST
Please keep in mind Ingress needs to be in the same namespace like services. If you have a few services in many namespace you need to create Ingress for each namespace.

I would need to set up a cluster in order to test your yml files.
Just to help you debugging, follow this steps:
1- get the logs of the my-pod container using kubectl logs my-pod-container-name, make sure everything is working
2- Use port-forward to expose your container and test it.
3- Make sure the service is working properly, change its type to load balancer, so you can reach it from outside the cluster.
If the three things are working there is a problem with your ingress configuration.
I am not sure if I explained it in a detailed way, let me know if something is not clear

Related

Why not working request to deployment via service request and via ingress request?

Install minikube version: v1.29.0 on MacOs.
I create API endpoint on flask and build in docker image
FROM debian:latest
COPY . /app
WORKDIR /app
RUN pip3 install --no-cache-dir -r requirements.txt
CMD ["uwsgi", "--socket", "0.0.0.0:5001", "--protocol=http", "-w", "wsgi:app", "--ini", "wsgi.ini"]
after load docker image into minikube
minikube image load drnoreg/devops_blog:0.0.1
check minikube
% minikube image ls
docker.io/drnoreg/devops_blog:0.0.1
create deployment, service and ingress yaml
app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: devops-blog
spec:
selector:
matchLabels:
run: devops-blog
replicas: 1
template:
metadata:
labels:
run: devops-blog
spec:
containers:
- name: devops-blog
image: docker.io/drnoreg/devops_blog:0.0.1
ports:
- name: pod-port
containerPort: 5001
---
apiVersion: v1
kind: Service
metadata:
name: devops-blog
labels:
run: devops-blog
spec:
type: NodePort
ports:
- name: pod-port
port: 5001
targetPort: 5001
protocol: TCP
nodePort: 30001
selector:
run: devops-blog
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: devops-blog
namespace: devops-blog
spec:
rules:
- host: devops-blog.cluster.local
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: devops-blog
port:
number: 5001
execute create namespace
kubectl create namespace devops-blog
set current namespace
kubectl config set-context --current --namespace=devops-blog
and create deployment, service and ingress
kubectl create -f app.yaml
after try forwarding port for check working flask API
kubectl port-forward devops-blog-f666d8cd7-njp95 5001:5001
Forwarding from 127.0.0.1:5001 -> 5001
Forwarding from [::1]:5001 -> 5001
Handling connection for 5001
Handling connection for 5001
flask API service in minikube is working.
% kubectl get service -n devops-blog -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
devops-blog NodePort 10.99.37.126 <none> 5001:30001/TCP 45s run=devops-blog
% kubectl get pod -n devops-blog -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
devops-blog-f666d8cd7-b9n7j 1/1 Running 0 57s 10.244.0.34 minikube <none> <none>
% kubectl get node -n devops-blog -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready control-plane 16h v1.26.1 192.168.49.2 <none> Ubuntu 20.04.5 LTS 5.10.47-linuxkit docker://20.10.23
Now I try to check working API via minikube service
% telnet 192.168.49.2 30001
Trying 192.168.49.2...
not working
add to /etc/hosts
127.0.0.1 devops-blog.cluster.local
try to check working API via ingress minikube
% telnet devops-blog.cluster.local 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
not working too.
Why not working request to deployment via service request and via ingress request?
How solve this problem?
In case you did not enable the ingress addon try enable it by executing the following command
$ minikube addons enable ingress
Instead of NodePort service try using the clusterIP service for the app and when you are creating ingress you can give this service as backend like this
service.yaml
apiVersion: v1
kind: Service
metadata:
name: devops-blog
labels:
run: devops-blog
spec:
type: ClusterIP
ports:
- name: pod-port
port: 5001
targetPort: 5001
protocol: TCP
selector:
run: devops-blog
ingres.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: devops-blog
namespace: devops-blog
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false" #Since you are using localhost
spec:
rules:
- host: devops-blog.cluster.local
http:
paths:
- pathType: ImplementationSpecific
backend:
service:
name: devops-blog
port:
number: 5001
path: /
Once the ingress is generated the IP try opening it in local browser with http://devops-blog.cluster.local/ or curl it like curl $ curl http://devops-blog.cluster.local/.
Note: In case you are deploying this app in the cloud try LoadBalancer as a service.
Try this tutorial as it explained in detail

ERR_CONNECTION_TIMED_OUT kubernetes minikube service

I am getting ERR_CONNECTION_TIMED_OUT when trying to access minikube service in localhost.
apiVersion: apps/v1
kind: Deployment
metadata:
name: identityserver
spec:
selector:
matchLabels:
app: identityserver
template:
metadata:
labels:
app: identityserver
spec:
containers:
- name: identityserver
image: identityserver:0
ports:
- containerPort: 5001
imagePullPolicy: "Never"
I have created service as following.
apiVersion: v1
kind: Service
metadata:
name: identityserver
spec:
type: NodePort
selector:
app: identityserver
ports:
- port: 5001
nodePort: 30002
I am trying to load in my local browser using following command. But it is not getting accessible in localhost. Internal kubernetes apps are able to communicate with service but not externally.
minikube service identityserver
I tried making type as clusterip and then it worked with port forwarding and only nodeport is having issue accessing.
kubectl port-forward service/identityserver 18080:5001 --address 0.0.0.0
This seems to be an issue with the Docker driver. I was able to run this with VirtualBox driver.
So I just had to start using VirtualBox driver (Even though virtualization was enabled in my machine it was giving an error. so had to append the --no-vtx-check flag, you can skip that if not facing an error without that flag)
minikube start --driver=virtualbox --no-vtx-check
There are several ways of trying minikube on Windows + docker:
Docker Desktop app (with Enable Kubernetes option)
Docker Desktop app (without enabling Kubernetes option) and installing minikube to wsl2
No Docker Desktop at all, installing docker and minikube in wsl2
Let's test it with the link you gave in comments - Set up Ingress on Minikube with the NGINX Ingress Controller.
Docker Desktop v.20.10.12 (with Enable Kubernetes option v.1.22.5), Win10, wsl2 backend.
Enable Kubernetes in Docker Desktop.
Check if ingress-controller is installed:
$ kubectl get pods -n ingress-nginx
The output should be similar to:
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-g9g49 0/1 Completed 0 11m
ingress-nginx-admission-patch-rqp78 0/1 Completed 1 11m
ingress-nginx-controller-59b45fb494-26npt 1/1 Running 0 11m
Create a Deployment using the following command:
kubectl create deployment web --image=gcr.io/google-samples/hello-app:1.0
Expose the Deployment:
kubectl expose deployment web --type=NodePort --port=8080
Create example-ingress.yaml from the following file:
$ kubectl apply -f example-ingress.yaml
$ cat example-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
ingressClassName: nginx # this line is essential!
rules:
- host: hello-world.info
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 8080
Verify the IP address is set:
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress <none> hello-world.info localhost 80 38s
Add the following line to the bottom of the C:\Windows\System32\drivers\etc\hosts file on your computer (you will need administrator access):
127.0.0.1 hello-world.info
DONE. Open hello-world.info in a browser.
How to access the NodePort service? In C:\Windows\System32\drivers\etc\hosts find these lines:
# Added by Docker Desktop
192.168.1.179 host.docker.internal
192.168.1.179 gateway.docker.internal
Use this IP and node port: curl 192.168.1.179:portNumber

Ingress with single service on docker on mac

Completely new do K8s here, I'm trying to deploy/connect Ingress to a sample single service on docker for local (docker for mac) and be able to hit it to run the service
1- sample service looks like this (my_service.js) :
const express = require('express')
let app = express()
app.get('/', (req, res) => {
res.send('Hello service')
})
const PORT = process.env.PORT || 9000
app.listen(
PORT,
() => console.log('Listening')
)
and I build the image (with a docker file):
docker build . -t sample-service-image
2- Now from K8s perspective:
here's the deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-service
labels:
app: sample-service
spec:
replicas: 1
selector:
matchLabels:
app: sample-service
template:
metadata:
labels:
app: sample-service
spec:
containers:
- name: sample-service
image: sample-service-image
imagePullPolicy: Never
ports:
- containerPort: 9000
kubectl apply -f deployment.yaml
and expose it as a service:
kubectl expose deploy sample-service --type=NodePort
3- Now from Ingress (ingress-sample.yaml) :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-sample
spec:
backend:
serviceName: sample-service
servicePort: 80
kubectl create -f ingress-sample.yaml
[UPDATE] I deployed an ingress controller from here:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml
this is what I get
$kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
ingress-sample * localhost 80 8m
kubectl get services --all-namespaces | grep nginx
ingress-nginx ingress-nginx NodePort 10.106.47.48 <none> 80:31448/TCP,443:31512/TCP 2h
tried to hit it on http://localhost:31448/ but I get 503 from NGINX ... so I guess it's not properly connected to sample-service.
how should I hit Ingress to be able to call the sample-service?
You need to specify Host header while hitting your service. For example
curl -H "HOST: abc" http://localhost:31448/
Since your ingress definition did not specify any host configuration, you can pass any value.
For more information - see.

Kubernetes - Ingress with Minikube

I am learning kubernetes by playing with minikube.
This is my pod deployment file which is fine.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
spec:
replicas: 2
selector:
matchLabels:
component: web
template:
metadata:
labels:
component: web
spec:
containers:
- name: myapp
image: myid/myimage
I am exposing the above pods using NodePort. I am able to access using minikube IP at port 30002.
apiVersion: v1
kind: Service
metadata:
name: my-ip-service
spec:
type: NodePort
externalIPs:
- 192.168.99.100
selector:
component: web
ports:
- port: 3000
nodePort: 30002
targetPort: 8080
Now i would like to use ingress to access the application at port 80 which will forward the request the ip-service at port 3000. It does NOT work
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
spec:
backend:
serviceName: my-ip-service
servicePort: 3000
If i try to access to ingress, address is blank.
NAME HOSTS ADDRESS PORTS AGE
test-ingress * 80 41m
How to use ingress with minikube? Or how to bind the minikube ip with ingress service - so that the app can be exposed outside without using nodeport
You can get your minikube node's IP address with:
minikube ip
The ingress' IP address will not populate in minikube because minikube lacks a load balancer. If you'd like something that behaves like a load balancer for your minikube cluster, https://github.com/knative/serving/blob/master/docs/creating-a-kubernetes-cluster.md#loadbalancer-support-in-minikube suggests running the following commands to patch your cluster:
sudo ip route add $(cat ~/.minikube/profiles/minikube/config.json | jq -r ".KubernetesConfig.ServiceCIDR") via $(minikube ip)
kubectl run minikube-lb-patch --replicas=1 --image=elsonrodriguez/minikube-lb-patch:0.1 --namespace=kube-system
I think you are missing the ingress controller resource on minikube itself. There are many possible ways to create an ingress-controller resource on K8s , but i think for you the best way to start on minikube is to follow this documentation.
Don't forget to read about Ingress in general once you get this working.

How to verify working Traefik installation?

I'm in the process of setting up Traefik on a Kubernetes cluster, but I can't get it to work, so I need some troubleshooting help. The first thing I would like to verify is that the basic installation is successful.
The guide I'm following is this one:
https://docs.traefik.io/user-guide/kubernetes/
But, I'm installing on a 3-machine cluster (Master + 2x Nodes).
I have setup RBAC and create a Deployment / Service for Traefik. The Pod is up and running:
$ kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
traefik-ingress-controller-7cf98d69cf-n2trx 1/1 Running 0 1h
This is the Service:
$ kubectl get services --namespace kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
traefik-ingress-service NodePort 10.107.17.76 <none> 80:30820/TCP,8080: 31362/TCP 1h
Should I be able to access the Traefik Web UI now?
I tried to access "http://192.168.1.11:31362" from a web browser and it behaves a bit strange. I get a "404 page not found" error in the browser window, but the address bar in the browser changes to: "http://192.168.1.11:31362/dashboard/". That tells me that something is responding at that address / port.
This is the result of a Curl to the same address:
$ curl http://192.168.1.11:31362/
Found.
Is this normal behaviour at this step in the process?
I have also tried to test with an Service / Ingress like this:
apiVersion: v1
kind: Service
metadata:
name: test-service
spec:
selector:
app: homeautomationweb
ports:
- port: 80
targetPort: 31047
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test-ingress
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: test.no
http:
paths:
- backend:
serviceName: test-service
servicePort: 80
I have a working web application running in the cluster exposed on a node port and is accessible outside the cluster at http://http://192.168.1.11:31047/.
The DNS name "test.no" is defined in /etc/hosts as 192.168.1.11
But, when I try to access http://test.no, I get:
"test.no refused to connect"
The details of what I'm doing and the exact content of the Kubernetes Yaml files can be found at the end of this article:
https://github.com/olavt/KubernetesRaspberryPI