I am building a microservice application. I am unable to send request to one of the services using postman:
Endpoint I am sending POST request to using postman:
http://cultor.dev/api/project
Error : "project-srv" does not have any active Endpoint (ingress-nginx returns 503 error)
Note
All the other microservices are running fine which use exact same config.
ingress-nginx config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
nginx.ingress.kubernetes.io/default-backend: ingress-nginx-controller
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: cultor.dev
http:
paths:
- path: /api/project/?(.*)
backend:
serviceName: project-srv
servicePort: 3000
- path: /api/profile/?(.*)
backend:
serviceName: profile-srv
servicePort: 3000
- path: /api/users/?(.*)
backend:
serviceName: auth-srv
servicePort: 3000
- path: /?(.*)
backend:
serviceName: client-srv
servicePort: 3000
ClusterIP services:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
auth-srv ClusterIP 10.245.52.208 <none> 3000/TCP 40m
client-srv ClusterIP 10.245.199.94 <none> 3000/TCP 39m
kubernetes ClusterIP 10.245.0.1 <none> 443/TCP 24d
nats-srv ClusterIP 10.245.1.58 <none> 4222/TCP,8222/TCP 39m
profile-srv ClusterIP 10.245.208.174 <none> 3000/TCP 39m
project-srv ClusterIP 10.245.131.56 <none> 3000/TCP 39m
LOGS
inress-nginx:
45.248.29.8 - - [02/Oct/2020:15:16:52 +0000] "POST /api/project/507f1f77bcf86cd799439011 HTTP/1.1" 503 197 "-" "PostmanRuntime/7.26.5" 591 0.000 [default-project-srv-3000] [] - - - - e1ae0615f49091786d56cab2bb9c94c6
W1002 15:17:59.712320 8 controller.go:916] Service "default/project-srv" does not have any active Endpoint.
I1002 15:17:59.814364 8 main.go:115] successfully validated configuration, accepting ingress ingress-service in namespace default
W1002 15:17:59.827616 8 controller.go:916] Service "default/project-srv" does not have any active Endpoint.
It was error with the labels as suggested by #Kamol Hasan.
The pod selector label in the 'Deployment' config was not matching selector in it's 'Service' config.
Related
I have Keyclock installed on my Kubernetes cluster.
Default ingress which Keycloak creates looks like this.
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
route.openshift.io/termination: passthrough
creationTimestamp: "2022-11-09T13:08:00Z"
generation: 1
labels:
app: keycloak
app.kubernetes.io/managed-by: keycloak-operator
name: keycloak-kc-ingress
namespace: default
ownerReferences:
- apiVersion: k8s.keycloak.org/v2alpha1
blockOwnerDeletion: true
controller: true
kind: Keycloak
name: keycloak-kc
uid: 67a18d00-4bee-4587-b330-cdaf21b39084
resourceVersion: "155002"
uid: 87c2aff4-1489-4ba9-bdf6-9fe1a288c800
spec:
defaultBackend:
service:
name: keycloak-kc-service
port:
number: 8443
rules:
- host: keycloak.example.com
http:
paths:
- backend:
service:
name: keycloak-kc-service
port:
number: 8443
pathType: ImplementationSpecific
status:
loadBalancer:
ingress:
- ip: 10.0.0.3
After installing ingress-nginx and adding kubernetes.io/ingress.class=nginx annotation, everything works.
For some reasons, however, I need to use nginx-ingress.
My new ingress looks like this.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
# nginx.ingress.kubernetes.io/backend-protocol: HTTPS
# route.openshift.io/termination: passthrough
labels:
app: keycloak
app.kubernetes.io/managed-by: keycloak-operator
# target: keycloak-kc-service
name: keycloak-kc-ingress
namespace: default
spec:
defaultBackend:
service:
name: keycloak-kc-service
port:
number: 8443
rules:
- host: accounts.example.com
http:
paths:
- backend:
service:
name: keycloak-kc-service
port:
number: 8443
path: /
pathType: Prefix
tls:
- hosts:
- accounts.example.com
secretName: keycloak-tls-secret
Unfortunately, this ingress returns the error "502 Bad Gateway".
We can't handle it. Please help.
Information for debugging
kubectl get deployments -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default keycloak-operator 2/2 2 2 141m
kube-system cilium-operator 1/1 1 1 148m
kube-system coredns 2/2 2 2 148m
kube-system konnectivity-agent 2/2 2 2 148m
kube-system metrics-server 2/2 2 2 148m
kubernetes-dashboard dashboard-metrics-scraper 2/2 2 2 148m
nginx-ingress nginx-ingress-nginx-ingress-nginx-ingress 1/1 1 1 127m
olm catalog-operator 1/1 1 1 142m
olm olm-operator 1/1 1 1 142m
olm packageserver 2/2 2 2 142m
kubectl get services -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default keycloak-kc-discovery ClusterIP None <none> 7800/TCP 114m
default keycloak-kc-service ClusterIP 10.240.18.67 <none> 8443/TCP 114m
default keycloak-operator ClusterIP 10.240.24.103 <none> 80/TCP 141m
default kubernetes ClusterIP 10.240.16.1 <none> 443/TCP 149m
default postgres-db ClusterIP 10.240.18.157 <none> 5432/TCP 140m
kube-system hcloud-csi-controller-metrics ClusterIP 10.240.30.190 <none> 9189/TCP 149m
kube-system hcloud-csi-node-metrics ClusterIP 10.240.26.123 <none> 9189/TCP 149m
kube-system kube-dns ClusterIP 10.240.16.10 <none> 53/TCP,53/UDP 149m
kube-system metrics-server ClusterIP 10.240.31.184 <none> 443/TCP 149m
kubernetes-dashboard dashboard-metrics-scraper ClusterIP 10.240.25.29 <none> 8000/TCP 149m
nginx-ingress nginx-ingress-nginx-ingress-nginx-ingress LoadBalancer 10.240.26.173 10.0.0.3,167.235.123.123,2a01:4f8:1c1f:6484::1 80:31670/TCP,443:30557/TCP 128m
olm operatorhubio-catalog ClusterIP 10.240.22.30 <none> 50051/TCP 142m
olm packageserver-service ClusterIP 10.240.23.246 <none>
Unfortunately, this ingress returns the error "502 Bad Gateway".
We can't handle it. Please help.
Created a kubernetes cluster, and installed ingress-nginx controller. I am getting a 404 not found if i go to the ingress-nginx-controller load balancer external ip that is aa************.us-east-1.elb.amazonaws.com
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
kubectl create namespace ingress-nginx
helm install ingress-nginx -n ingress-nginx ingress-nginx/ingress-nginx
to get the service:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.100.219.162 aa************.us-east-1.elb.amazonaws.com 80:32091/TCP,443:32305/TCP 154m
ingress-nginx-controller-admission ClusterIP 10.100.208.135 <none> 443/TCP 154m
my ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: factory
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: factory.**.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: factory
port:
number: 80
- host: api.factory.**.com # myfactoryapi-factorydomain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: factory
port:
number: 8082
all my namespace
kubectl get namespace
NAME STATUS AGE
default Active 34d
ingress-nginx Active 160m
kerberos-factory Active 34d
kube-node-lease Active 34d
kube-public Active 34d
kube-system Active 34d
mongodb Active 8d
to get my ingress
kubectl get ing -n kerberos-factory
NAME CLASS HOSTS ADDRESS PORTS AGE
factory <none> factory.**.com,api.factory.**.com a****.us-east-1.elb.amazonaws.com 80 65m
to describe the ingress
kubectl describe ing -n kerberos-factory
Name: factory
Namespace: kerberos-factory
Address: a********.us-east-1.elb.amazonaws.com
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
factory.***.com
/ factory:80 (192.168.34.220:80)
api.factory.***.com
/ factory:8082 (192.168.34.220:8082)
Annotations: kubernetes.io/ingress.class: nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 3m8s (x3 over 4m51s) nginx-ingress-controller Scheduled for sync
why am i getting 404 not found
I have two ingress routes created as below,
kubesphere-console
tekton-pipelines
My manifest files are as like below.
**cat ingress-tekton-dashboard.yaml **
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tekton-dashboard
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
ingressClassName: nginx
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: tekton-dashboard
port:
number: 9097
#host: *
...
cat ../kubesphere/ingress-route-kubesphere.yaml
cat ../kubesphere/ingress-route-kubesphere.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: kubesphere-console
annotations:
kubesphere.io/creator: admin
spec:
ingressClassName: nginx
rules:
- host: crashandburn.australiaeast.cloudapp.azure.com
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: ks-console
port:
number: 80
---
My ingress are as below:
k get ing -nkubesphere-system
NAME CLASS HOSTS ADDRESS PORTS AGE
kubesphere-console nginx * 20.92.133.79 80 28h
ameya#Azure:~/tekton$ k get ing -ntekton-pipelines
NAME CLASS HOSTS ADDRESS PORTS AGE
tekton-dashboard <none> * 80 2m32s
My service for tekton dashboard is like below:
k get svc -n tekton-pipelines
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
tekton-dashboard ClusterIP 10.0.202.127 <none> 9097/TCP 2d3h
tekton-pipelines-controller ClusterIP 10.0.53.46 <none> 9090/TCP,8008/TCP,8080/TCP 2d6h
tekton-pipelines-webhook ClusterIP 10.0.222.121 <none> 9090/TCP,8008/TCP,443/TCP,8080/TCP 2d6h
However, I get the error like below:
k apply -f ingress-tekton-dashboard.yaml -ntekton-pipelines
Error from server (BadRequest): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"ingress.kubernetes.io/ssl-redirect\":\"false\",\"ingressClassName\":\"nginx\",\"nginx.ingress.kubernetes.io/ssl-redirect\":\"false\"},\"name\":\"tekton-dashboard\",\"namespace\":\"tekton-pipelines\"},\"spec\":{\"ingressClassName\":\"nginx\",\"rules\":[{\"http\":{\"paths\":[{\"backend\":{\"service\":{\"name\":\"tekton-dashboard\",\"port\":{\"number\":9097}}},\"path\":\"/\",\"pathType\":\"Prefix\"}]}}]}}\n"}},"spec":{"ingressClassName":"nginx"}}
to:
Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind: "networking.k8s.io/v1, Kind=Ingress"
Name: "tekton-dashboard", Namespace: "tekton-pipelines"
for: "ingress-tekton-dashboard.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: host "_" and path "/" is already defined in ingress kubesphere-system/kubesphere-console
However, I am using two differnt ports for tekton-dashboard and kubesphere-console so wondering why it is complaining.
I am able to access my django app deployment using LoadBalancer service type but I'm trying to switch to ClusterIP service type and ingress-nginx but I am getting 503 Service Temporarily Unavailable when I try to access the site via the host url. Describing the ingress also shows error: endpoints "django-service" not found and error: endpoints "default-http-backend" not found. What am I doing wrong?
This is my service and ingress yaml:
---
apiVersion: v1
kind: Service
metadata:
name: django-service
spec:
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: django-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
spec:
tls:
- hosts:
- django.example.com
rules:
- host: django.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: django-service
port:
number: 80
ingressClassName: nginx
kubectl get all
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/django-app-5bdd8ffff9-79xzj 1/1 Running 0 7m44s
pod/postgres-58fffbb5cc-247x9 1/1 Running 0 7m44s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/django-service ClusterIP 10.233.29.58 <none> 80/TCP 7m44s
service/pg-service ClusterIP 10.233.14.137 <none> 5432/TCP 7m44s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/django-app 1/1 1 1 7m44s
deployment.apps/postgres 1/1 1 1 7m44s
NAME DESIRED CURRENT READY AGE
replicaset.apps/django-app-5bdd8ffff9 1 1 1 7m44s
replicaset.apps/postgres-58fffbb5cc 1 1 1 7m44s
describe ingress
$ kubectl describe ing django-ingress
Name: django-ingress
Labels: <none>
Namespace: django
Address: 10.10.30.50
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
SNI routes django.example.com
Rules:
Host Path Backends
---- ---- --------
django.example.com
/ django-service:80 (<error: endpoints "django-service" not found>)
Annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: true
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 5m28s (x2 over 6m5s) nginx-ingress-controller Scheduled for sync
Normal Sync 5m28s (x2 over 6m5s) nginx-ingress-controller Scheduled for sync
I think you forgot to make the link with your deployment in your service.
apiVersion: v1
kind: Service
metadata:
name: django-service
spec:
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8000
selector:
app: your-deployment-name
Your label must be set in your deployment as well:
spec:
selector:
matchLabels:
app: your-deployment-name
template:
metadata:
labels:
app: your-deployment-name
I want to deploy different versions of the same application in production:
kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0 -n test-a
kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:2.0 -n test-b
kubectl expose deployment hello-server --port 80 --target-port 8080 -n test-a
kubectl expose deployment hello-server --port 80 --target-port 8080 -n test-b
I've tried to load-balance it like this:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: test
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/server-snippets: |
upstream test-servers-upstream {
server hello-server.test-a:80;
server hello-server.test-b:80;
}
spec:
rules:
- host: test-servers.example.com
http:
paths:
- path: /
backend:
serviceName: test-servers-upstream
servicePort: 80
but it returns an error:
/ test-servers-upstream:80 (<error: endpoints "test-servers-upstream" not found>)
Services:
$ kubectl get services -n test-b
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-server ClusterIP 10.128.199.113 <none> 80/TCP 75m
$ kubectl get services -n test-a
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-server ClusterIP 10.128.207.6 <none> 80/TCP 75m
How can I load-balance it?
Fresh ingress-nginx supports weight-based canary deployments out of box.
Looks like what you're asking for.
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary.
Great example here.
https://medium.com/#domi.stoehr/canary-deployments-on-kubernetes-without-service-mesh-425b7e4cc862
Basically, you create 2 identical ingresses, one for each namespace.
They are different only in annotations and in pointing to different services.
# ingress for a service A in namespace A
kind: Ingress
metadata:
name: demo-ingress
namespace: demo-prod
spec:
rules:
- host: canary.example.com
http:
paths:
- backend:
serviceName: demo-prod
servicePort: 80
path: /
# ingress for a service B in namespace B
# note canary annotations
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "20"
name: demo-ingress
namespace: demo-canary
spec:
rules:
- host: canary.example.com
http:
paths:
- backend:
serviceName: demo-canary
servicePort: 80
path: /
Note, that ingress-nginx canary implementation actually requires your services to be in different namespaces.