Ingress - default backend - 404 on TLS - kubernetes

Question

I'm using Rancher 2.4.7 to deploy applications on customer's environments.
I successfully created a new cluster for a production environment however whenever I create an Ingress resource with a specific hostname (the certificate is valid) : I get the default backend - 404 message when accessing my applications.
If I create an Ingress resource using the embed xip.io hostname, I can access my applications like it's supposed to.
I tried creating my resource using kubectl and the Rancher UI.
When getting the default backend - 404 error, no logs are registered on my deployment.
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: default
spec:
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
workload.user.cattle.io/workloadselector: deployment-default-nginx
spec:
containers:
- image: nginx:latest
imagePullPolicy: Always
name: nginx
ports:
- containerPort: 80
name: nginx
protocol: TCP
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities: {}
privileged: false
readOnlyRootFilesystem: false
runAsNonRoot: false
stdin: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
tty: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
Service :
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: default
spec:
ports:
- name: nginx
port: 80
protocol: TCP
targetPort: 80
selector:
workload.user.cattle.io/workloadselector: deployment-default-nginx
sessionAffinity: None
type: ClusterIP
Ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx
namespace: default
spec:
rules:
- host: mydomain.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
path: /
tls:
- hosts:
- mydomain.com
secretName: mycertificate
Any thoughts what is wrong?
EDIT:
I'm using nginx-ingress shipped with Rancher. I'm not using any Cloud Provider. I installed a RKE Cluster using the Rancher UI.
Ingress controller logs:
I0111 16:13:02.868872 6 flags.go:204] Watching for Ingress class: nginx
W0111 16:13:02.869187 6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0111 16:13:02.869225 6 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0111 16:13:02.869357 6 main.go:220] Creating API client for https://10.43.0.1:443
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.32.0
Build: git-29dd9760f
Repository: https://github.com/rancher/ingress-nginx.git
nginx version: nginx/1.17.10
-------------------------------------------------------------------------------
I0111 16:13:02.879836 6 main.go:264] Running in Kubernetes cluster version v1.18 (v1.18.6) - git (clean) commit dff82dc0de47299ab66c83c626e08b245ab19037 - platform linux/amd64
I0111 16:13:02.884023 6 main.go:94] Validated ingress-nginx/default-http-backend as the default backend.
I0111 16:13:03.022518 6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0111 16:13:03.023350 6 main.go:113] Enabling new Ingress features available since Kubernetes v1.18
E0111 16:13:03.024271 6 main.go:122] Unexpected error searching IngressClass: ingressclasses.networking.k8s.io "nginx" is forbidden: User "system:serviceaccount:ingress-nginx:nginx-ingress-serviceaccount" cannot get resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
W0111 16:13:03.024285 6 main.go:125] No IngressClass resource with name nginx found. Only annotation will be used.
I0111 16:13:03.060623 6 nginx.go:263] Starting NGINX Ingress controller
I0111 16:13:03.077607 6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"eb5e7318-e99c-428f-82a0-4321925fdc4e", APIVersion:"v1", ResourceVersion:"610", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0111 16:13:03.077660 6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"91590425-b422-4c4b-829d-a0be04973a9d", APIVersion:"v1", ResourceVersion:"611", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0111 16:13:03.077677 6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"20b82b4a-e536-4a63-9fd2-20e17489b1ab", APIVersion:"v1", ResourceVersion:"607", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I0111 16:13:04.168739 6 backend_ssl.go:66] Adding Secret "default/mycertificate" to the local store
I0111 16:13:04.261564 6 nginx.go:307] Starting NGINX process
I0111 16:13:04.261672 6 leaderelection.go:242] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0111 16:13:04.262436 6 controller.go:139] Configuration changes detected, backend reload required.
I0111 16:13:04.363051 6 leaderelection.go:252] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I0111 16:13:04.363157 6 status.go:86] new leader elected: nginx-ingress-controller-5dwhm
I0111 16:13:04.385580 6 controller.go:155] Backend successfully reloaded.
I0111 16:13:04.385608 6 controller.go:164] Initial sync, sleeping for 1 second.
E0111 16:14:14.950751 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:161: Failed to watch *v1.Pod: Get "https://10.43.0.1:443/api/v1/namespaces/ingress-nginx/pods?allowWatchBookmarks=true&labelSelector=app%3Dingress-nginx%2Ccontroller-revision-hash%3D55dd558cd5%2Cpod-template-generation%3D1&resourceVersion=1028734&timeout=9m32s&timeoutSeconds=572&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0111 16:14:14.950798 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:159: Failed to watch *v1.Service: Get "https://10.43.0.1:443/api/v1/services?allowWatchBookmarks=true&resourceVersion=1028611&timeout=5m1s&timeoutSeconds=301&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0111 16:14:14.950816 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:157: Failed to watch *v1.Secret: Get "https://10.43.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=1028785&timeout=9m23s&timeoutSeconds=563&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0111 16:14:14.950847 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:182: Failed to watch *v1beta1.Ingress: Get "https://10.43.0.1:443/apis/networking.k8s.io/v1beta1/ingresses?allowWatchBookmarks=true&resourceVersion=1028611&timeout=8m58s&timeoutSeconds=538&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0111 16:14:14.951438 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:160: Failed to watch *v1.ConfigMap: Get "https://10.43.0.1:443/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=1028928&timeout=9m58s&timeoutSeconds=598&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0111 16:14:14.951936 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:158: Failed to watch *v1.Endpoints: Get "https://10.43.0.1:443/api/v1/endpoints?allowWatchBookmarks=true&resourceVersion=1028926&timeout=9m48s&timeoutSeconds=588&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
I0112 09:02:55.888418 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 09:02:55.969141 6 controller.go:155] Backend successfully reloaded.
I0112 09:22:55.591923 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 09:22:55.591920 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173601", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/nginx
I0112 09:22:55.676508 6 controller.go:155] Backend successfully reloaded.
I0112 09:23:04.372633 6 status.go:275] updating Ingress default/nginx status from [] to [{10.107.129.176 }]
I0112 09:23:04.380700 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173626", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx
I0112 09:23:04.388779 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173628", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx
I0112 09:23:04.402679 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1173629", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx
I0112 09:23:07.714377 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 09:23:07.791810 6 controller.go:155] Backend successfully reloaded.
10.117.131.141 - - [12/Jan/2021:09:23:11 +0000] "GET / HTTP/1.1" 200 414 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36" 504 0.006 [default-nginx-80] [] 10.42.0.168:80 612 0.004 200 0bb06b53bbc47c83854cf8e19fde2e4a
I0112 09:24:06.033945 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 09:24:06.114217 6 controller.go:155] Backend successfully reloaded.
E0112 09:35:32.853089 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:161: Failed to watch *v1.Pod: Get "https://10.43.0.1:443/api/v1/namespaces/ingress-nginx/pods?allowWatchBookmarks=true&labelSelector=app%3Dingress-nginx%2Ccontroller-revision-hash%3D55dd558cd5%2Cpod-template-generation%3D1&resourceVersion=1173526&timeout=8m3s&timeoutSeconds=483&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0112 09:35:32.853089 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:160: Failed to watch *v1.ConfigMap: Get "https://10.43.0.1:443/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=1175379&timeout=6m4s&timeoutSeconds=364&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0112 09:35:32.853156 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:182: Failed to watch *v1beta1.Ingress: Get "https://10.43.0.1:443/apis/networking.k8s.io/v1beta1/ingresses?allowWatchBookmarks=true&resourceVersion=1173916&timeout=6m15s&timeoutSeconds=375&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0112 09:35:32.853566 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:159: Failed to watch *v1.Service: Get "https://10.43.0.1:443/api/v1/services?allowWatchBookmarks=true&resourceVersion=1173495&timeout=9m31s&timeoutSeconds=571&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0112 09:35:32.854106 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:158: Failed to watch *v1.Endpoints: Get "https://10.43.0.1:443/api/v1/endpoints?allowWatchBookmarks=true&resourceVersion=1175377&timeout=6m14s&timeoutSeconds=374&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0112 09:35:32.854378 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:157: Failed to watch *v1.Secret: Get "https://10.43.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=1173561&timeout=9m27s&timeoutSeconds=567&watch=true": dial tcp 10.43.0.1:443: connect: connection refused
E0112 09:35:36.334273 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:157: Failed to watch *v1.Secret: unknown (get secrets)
E0112 09:35:36.334349 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:160: Failed to watch *v1.ConfigMap: unknown (get configmaps)
E0112 09:35:36.334520 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:159: Failed to watch *v1.Service: unknown (get services)
E0112 09:35:36.334538 6 reflector.go:382] k8s.io/ingress-nginx/internal/ingress/controller/store/store.go:158: Failed to watch *v1.Endpoints: unknown (get endpoints)
I0112 10:18:11.687109 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 10:18:11.775731 6 controller.go:155] Backend successfully reloaded.
I0112 10:48:25.950829 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 10:48:26.033626 6 controller.go:155] Backend successfully reloaded.
W0112 10:49:21.912963 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 11:05:53.875160 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 11:05:53.875312 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 11:05:53.942794 6 controller.go:155] Backend successfully reloaded.
I0112 11:05:53.990374 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"5121fa4a-a2c1-445a-a65a-da5ad64088f1", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1188099", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/nginx
I0112 11:05:57.208715 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 11:05:57.297730 6 controller.go:155] Backend successfully reloaded.
I0112 11:06:15.724049 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"c230bb3e-7e0c-4ffa-b8a3-24e437a11b7c", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1188154", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/nginx
W0112 11:06:15.725433 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 11:06:15.725612 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 11:06:15.808897 6 controller.go:155] Backend successfully reloaded.
I0112 11:07:04.375801 6 status.go:275] updating Ingress default/nginx status from [] to [{10.107.129.176 }]
I0112 11:07:04.386737 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"c230bb3e-7e0c-4ffa-b8a3-24e437a11b7c", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1188268", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx
W0112 11:07:04.386900 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 11:07:04.399808 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"c230bb3e-7e0c-4ffa-b8a3-24e437a11b7c", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1188270", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx
W0112 11:07:07.720456 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 12:37:09.436273 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 12:37:09.436524 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"c230bb3e-7e0c-4ffa-b8a3-24e437a11b7c", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1200926", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx
W0112 12:37:32.812935 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 12:37:43.751781 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 12:37:47.085248 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 13:29:48.126896 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 13:29:48.127046 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 13:29:48.194086 6 controller.go:155] Backend successfully reloaded.
W0112 13:34:18.155900 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 13:34:18.156070 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 13:34:18.231656 6 controller.go:155] Backend successfully reloaded.
W0112 13:50:31.577308 6 main.go:60] Protocol "HTTPS" is not a valid value for the backend-protocol annotation. Using HTTP as protocol
W0112 13:50:31.578553 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 13:50:50.699484 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 14:34:11.399326 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 14:34:11.399493 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 14:34:11.494728 6 controller.go:155] Backend successfully reloaded.
W0112 14:35:04.385107 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 14:35:07.718580 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
I0112 14:35:07.718815 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 14:35:07.819621 6 controller.go:155] Backend successfully reloaded.
W0112 14:35:11.052037 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 14:35:33.306202 6 controller.go:909] Service "default/nginx" does not have any active Endpoint.
W0112 14:37:02.428966 6 controller.go:822] Error obtaining Endpoints for Service "default/nginx": no object matching key "default/nginx" in local store
I0112 14:37:02.429111 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 14:37:02.493923 6 controller.go:155] Backend successfully reloaded.
I0112 14:37:05.762583 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 14:37:05.859639 6 controller.go:155] Backend successfully reloaded.
10.117.131.141 - - [12/Jan/2021:14:37:07 +0000] "GET / HTTP/1.1" 200 414 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36" 618 0.002 [default-nginx-80] [] 10.42.0.168:80 612 0.000 200 4e6feb07a8d5e16a9d2592272ca338fd
I0112 15:24:28.246019 6 controller.go:139] Configuration changes detected, backend reload required.
I0112 15:24:28.316464 6 controller.go:155] Backend successfully reloaded.
I0112 15:25:31.398134 6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx", UID:"c230bb3e-7e0c-4ffa-b8a3-24e437a11b7c", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1224608", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx

Answer 1

spec:
defaultBackend:
serviceName: ingress-nginx-controller
servicePort: 80
Before rules add this

Answer 2

I am not sure if I am able to understand since it would require some RnD.
Meanwhile, you can checkout the ingress config that I used on my project :
Note: What version of Ingress I used and also the annotations for it
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
nginx.ingress.kubernetes.io/default-backend: ingress-nginx-controller
## tells ingress to check for regex in the config file
nginx.ingress.kubernetes.io/use-regex: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/configuration-snippet: |
add_header Access-Control-Allow-Methods "POST, GET, OPTIONS";
add_header Access-Control-Allow-Credentials true;
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "PUT, GET, POST, OPTIONS"
nginx.ingress.kubernetes.io/from-to-www-redirect: "true" ## IMPORTANT
spec:
## TLS config
tls:
- hosts:
- www.mydomain.com
- mydomain.com
secretName: cultor-test-tls
## our custom routing rules
rules:
- host: www.mydomain.com
http:
paths:
- path: /api/upload/?(.*)
backend:
serviceName: aws-srv
servicePort: 3000
- path: /api/project/?(.*)
backend:
serviceName: project-srv
servicePort: 3000
- path: /api/profile/?(.*)
backend:
serviceName: profile-srv
servicePort: 3000
- path: /api/users/?(.*)
backend:
serviceName: auth-srv
servicePort: 3000
- path: /?(.*)
backend:
serviceName: client-srv
servicePort: 3000
Important
Make sure you've tried accessing the domain with and without WWW and add this in your config as shown above
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"

Answer 3

Your service and deployment labels are not matching. That's why any of the requests for this service are not finding the backend and there is nothing to respond to these requests. For service you use:
selector:
name: nginx
When for deployment you use:
labels:
app: nginx
You have to change labels in either of the deployment so they will match.