certificate is valid for ingress.local, not gitlab.mydomain - kubernetes

Question

I'm new to Kubernetes.
I have installed a freshly new Kubernetes Cluster by using RKE (rancher tool for creating k8 clusters).
I added the gitlab chart (https://charts.gitlab.io/) and launch it.
Being on several issues with PersistentStorage, etc that I managed to resolve.
But I'm now stuck on one last issue: the pod for gitlab-runner is failing with the following logs:
ERROR: Registering runner... failed runner=Mk5hMxa5 status=couldn't execute POST against https://gitlab.mydomain.com/api/v4/runners: Post https://gitlab.mydomain.com/api/v4/runners: x509: certificate is valid for ingress.local, not gitlab.mydomain.com
PANIC: Failed to register this runner. Perhaps you are having network problems
Description of the certificate using kubectl describe certificate gitlab-gitlab-tls -n gitlab:
Name: gitlab-gitlab-tls
Namespace: gitlab
Labels: app=unicorn
chart=unicorn-2.4.6
heritage=Tiller
io.cattle.field/appId=gitlab
release=gitlab
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-11-13T13:49:10Z
Generation: 3
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: gitlab-unicorn
UID: 5640645f-550b-4073-bdf0-df8b089b0c94
Resource Version: 6824
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/certificates/gitlab-gitlab-tls
UID: 30ac32bd-c7f3-4f9b-9e3b-966b6090e1a9
Spec:
Acme:
Config:
Domains:
gitlab.mydomain.com
http01:
Ingress Class: gitlab-nginx
Dns Names:
gitlab.mydomain.com
Issuer Ref:
Kind: Issuer
Name: gitlab-issuer
Secret Name: gitlab-gitlab-tls
Status:
Conditions:
Last Transition Time: 2019-11-13T13:49:10Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events: <none>
Description of the issuer using kubectl describe issuer gitlab-issuer -n gitlab:
Name: gitlab-issuer
Namespace: gitlab
Labels: app=certmanager-issuer
chart=certmanager-issuer-0.1.0
heritage=Tiller
release=gitlab
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Issuer","metadata":{"annotations":{},"creationTimestamp":"2019-11-13T13:49:10Z","gener...
API Version: certmanager.k8s.io/v1alpha1
Kind: Issuer
Metadata:
Creation Timestamp: 2019-11-13T13:49:10Z
Generation: 4
Resource Version: 24537
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/issuers/gitlab-issuer
UID: b9971d7a-5220-47ca-a7f9-607aa3f9be4f
Spec:
Acme:
Email: mh#mydomain.com
http01:
Private Key Secret Ref:
Name: gitlab-acme-key
Server: https://acme-v02.api.letsencrypt.org/directory
Status:
Acme:
Last Registered Email: mh#mydomain.com
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/71695690
Conditions:
Last Transition Time: 2019-11-13T13:49:12Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
Description of the challenge using kubectl describe challenges.certmanager.k8s.io -n gitlab gitlab-gitlab-tls-3386074437-0:
Name: gitlab-gitlab-tls-3386074437-0
Namespace: gitlab
Labels: acme.cert-manager.io/order-name=gitlab-gitlab-tls-3386074437
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Challenge
Metadata:
Creation Timestamp: 2019-11-13T13:49:15Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 4
Owner References:
API Version: certmanager.k8s.io/v1alpha1
Block Owner Deletion: true
Controller: true
Kind: Order
Name: gitlab-gitlab-tls-3386074437
UID: 1f01771e-2e38-491f-9b2d-ab5f4fda60e2
Resource Version: 6915
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/challenges/gitlab-gitlab-tls-3386074437-0
UID: 4c115a6f-a76f-4859-a5db-6acd9c039d71
Spec:
Authz URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/1220588820
Config:
http01:
Ingress Class: gitlab-nginx
Dns Name: gitlab.mydomain.com
Issuer Ref:
Kind: Issuer
Name: gitlab-issuer
Key: lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8.lPWns02SmS3zXwFzHdma_RyhwwlzWLRDkdlugFXDlZY
Token: lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/1220588820/AwsnPw
Wildcard: false
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
State: pending
Events: <none>
Logs found in cert-manager pod:
I1113 14:20:21.857235 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "level"=0 "msg"="found one existing HTTP01 solver pod" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ttkmj" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
I1113 14:20:21.857458 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-sdlw7" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
I1113 14:20:21.857592 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-7jzwk" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
E1113 14:20:21.864785 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="gitlab.mydomain.com" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
The DNS gitlab.mydomain.com is set to point to the IP of my LoadBalancer where NGINX is running.
If I go to https://gitlab.mydomain.com in the browser:
The browser is saying the connexion is not secure
The result is "default backend - 404".
Edits
Description of the ingress-controller by using kubectl describe svc gitlab-nginx-ingress-controller -n gitlab:
Name: gitlab-nginx-ingress-controller
Namespace: gitlab
Labels: app=nginx-ingress
chart=nginx-ingress-0.30.0-1
component=controller
heritage=Tiller
io.cattle.field/appId=gitlab
release=gitlab
Annotations: field.cattle.io/ipAddresses: null
field.cattle.io/targetDnsRecordIds: null
field.cattle.io/targetWorkloadIds: null
Selector: <none>
Type: ExternalName
IP:
External Name: gitlab.mydomain.com
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31487/TCP
Endpoints: 10.42.0.7:80,10.42.1.9:80,10.42.2.12:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31560/TCP
Endpoints: 10.42.0.7:443,10.42.1.9:443,10.42.2.12:443
Port: gitlab-shell 22/TCP
TargetPort: gitlab-shell/TCP
NodePort: gitlab-shell 30539/TCP
Endpoints: 10.42.0.7:22,10.42.1.9:22,10.42.2.12:22
Session Affinity: None
Events: <none>
Running kubectl get ingress -n gitlab gives me a bunch of ingress:
NAME HOSTS ADDRESS PORTS AGE
cm-acme-http-solver-5rjg4 minio.mydomain.com gitlab.mydomain.com 80 4d23h
cm-acme-http-solver-7jzwk gitlab.mydomain.com gitlab.mydomain.com 80 4d23h
cm-acme-http-solver-tzs25 registry.mydomain.com gitlab.mydomain.com 80 4d23h
gitlab-minio minio.mydomain.com gitlab.mydomain.com 80, 443 4d23h
gitlab-registry registry.mydomain.com gitlab.mydomain.com 80, 443 4d23h
gitlab-unicorn gitlab.mydomain.com gitlab.mydomain.com 80, 443 4d23h
Description of the gitlab-unicorn by using kubectl describe ingress gitlab-unicron -n gitlab
Name: gitlab-unicorn
Namespace: gitlab
Address: gitlab.mydomain.com
Default backend: default-http-backend:80 (<none>)
TLS:
gitlab-gitlab-tls terminates gitlab.mydomain.com
Rules:
Host Path Backends
---- ---- --------
gitlab.mydomain.com
/ gitlab-unicorn:8181 (10.42.0.9:8181,10.42.1.8:8181)
/admin/sidekiq gitlab-unicorn:8080 (10.42.0.9:8080,10.42.1.8:8080)
Annotations:
certmanager.k8s.io/issuer: gitlab-issuer
field.cattle.io/publicEndpoints: [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"gitlab:gitlab-unicorn","ingressName":"gitlab:gitlab-unicorn","hostname":"gitlab.mydomain.com","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"gitlab:gitlab-unicorn","ingressName":"gitlab:gitlab-unicorn","hostname":"gitlab.mydomain.com","path":"/admin/sidekiq","allNodes":false}]
kubernetes.io/ingress.class: gitlab-nginx
kubernetes.io/ingress.provider: nginx
nginx.ingress.kubernetes.io/proxy-body-size: 512m
nginx.ingress.kubernetes.io/proxy-connect-timeout: 15
nginx.ingress.kubernetes.io/proxy-read-timeout: 600
Events: <none>
Description of cm-acme-http-solver-7jzwk by using kubectl describe ingress cm-acme-http-solver-7jzwk -n gitlab:
Name: cm-acme-http-solver-7jzwk
Namespace: gitlab
Address: gitlab.mydomain.com
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
gitlab.mydomain.com
/.well-known/acme-challenge/lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8 cm-acme-http-solver-sdlw7:8089 (10.42.2.19:8089)
Annotations:
field.cattle.io/publicEndpoints: [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"gitlab:cm-acme-http-solver-sdlw7","ingressName":"gitlab:cm-acme-http-solver-7jzwk","hostname":"gitlab.mydomain.com","path":"/.well-known/acme-challenge/lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8","allNodes":false}]
kubernetes.io/ingress.class: gitlab-nginx
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
Events: <none>
Ports open on my LoadBalancer and on every nodes of my cluster (I know I should close somes but I will first manage to make my gitlab setup working):
80/tcp ALLOW Anywhere
443/tcp ALLOW Anywhere
22/tcp ALLOW Anywhere
2376/tcp ALLOW Anywhere
2379/tcp ALLOW Anywhere
2380/tcp ALLOW Anywhere
6443/tcp ALLOW Anywhere
6783/tcp ALLOW Anywhere
6783:6784/udp ALLOW Anywhere
8472/udp ALLOW Anywhere
4789/udp ALLOW Anywhere
9099/tcp ALLOW Anywhere
10250/tcp ALLOW Anywhere
10254/tcp ALLOW Anywhere
30000:32767/tcp ALLOW Anywhere
30000:32767/udp ALLOW Anywhere
80/tcp (v6) ALLOW Anywhere (v6)
443/tcp (v6) ALLOW Anywhere (v6)
22/tcp (v6) ALLOW Anywhere (v6)
2376/tcp (v6) ALLOW Anywhere (v6)
2379/tcp (v6) ALLOW Anywhere (v6)
2380/tcp (v6) ALLOW Anywhere (v6)
6443/tcp (v6) ALLOW Anywhere (v6)
6783/tcp (v6) ALLOW Anywhere (v6)
6783:6784/udp (v6) ALLOW Anywhere (v6)
8472/udp (v6) ALLOW Anywhere (v6)
4789/udp (v6) ALLOW Anywhere (v6)
9099/tcp (v6) ALLOW Anywhere (v6)
10250/tcp (v6) ALLOW Anywhere (v6)
10254/tcp (v6) ALLOW Anywhere (v6)
30000:32767/tcp (v6) ALLOW Anywhere (v6)
30000:32767/udp (v6) ALLOW Anywhere (v6)
kubectl get pods -n gitlab
cm-acme-http-solver-4d8s5 1/1 Running 0 5d
cm-acme-http-solver-ttkmj 1/1 Running 0 5d
cm-acme-http-solver-ws7kv 1/1 Running 0 5d
gitlab-certmanager-57bc6fb4fd-6rfds 1/1 Running 0 5d
gitlab-gitaly-0 1/1 Running 0 5d
gitlab-gitlab-exporter-57b99467d4-knbgk 1/1 Running 0 5d
gitlab-gitlab-runner-64b74bcd59-mxwvm 0/1 CrashLoopBackOff 10 55m
gitlab-gitlab-shell-cff8b68f7-zng2c 1/1 Running 0 5d
gitlab-gitlab-shell-cff8b68f7-zqvfr 1/1 Running 0 5d
gitlab-issuer.1-lqs7c 0/1 Completed 0 5d
gitlab-migrations.1-c4njn 0/1 Completed 0 5d
gitlab-minio-75567fcbb6-jjxhw 1/1 Running 6 5d
gitlab-minio-create-buckets.1-6zljh 0/1 Completed 0 5d
gitlab-nginx-ingress-controller-698fbc4c64-4wt97 1/1 Running 0 5d
gitlab-nginx-ingress-controller-698fbc4c64-5kv2h 1/1 Running 0 5d
gitlab-nginx-ingress-controller-698fbc4c64-jxljq 1/1 Running 0 5d
gitlab-nginx-ingress-default-backend-6cd54c5f86-2jrkd 1/1 Running 0 5d
gitlab-nginx-ingress-default-backend-6cd54c5f86-cxlmx 1/1 Running 0 5d
gitlab-postgresql-66d8d9574b-hbx78 2/2 Running 0 5d
gitlab-prometheus-server-6fb685b9c7-c8bqj 2/2 Running 0 5d
gitlab-redis-7668c4d476-tcln5 2/2 Running 0 5d
gitlab-registry-7bb984c765-7ww6j 1/1 Running 0 5d
gitlab-registry-7bb984c765-t5jjq 1/1 Running 0 5d
gitlab-sidekiq-all-in-1-8fd95bf7b-hfnjz 1/1 Running 0 5d
gitlab-task-runner-5cd7bf5bb9-gnv8p 1/1 Running 0 5d
gitlab-unicorn-864bd864f5-47zxg 2/2 Running 0 5d
gitlab-unicorn-864bd864f5-gjms2 2/2 Running 0 5d
Their are 3 acme-http-solver:
One for registry.mydomain.com
One for minio.mydomain.com
One for gitlab.mydomain.com
The logs for the one pointing to gitlab.mydomain.com:
I1113 13:49:21.207782 1 solver.go:39] cert-manager/acmesolver "level"=0 "msg"="starting listener" "expected_domain"="gitlab.mydomain.com" "expected_key"="lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8.lPWns02SmS3zXwFzHdma_RyhwwlzWLRDkdlugFXDlZY" "expected_token"="lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8" "listen_port"=8089
Results of kubectl get svc -n gitlab:
cm-acme-http-solver-48b2j NodePort 10.43.58.52 <none> 8089:30090/TCP 5d23h
cm-acme-http-solver-h42mk NodePort 10.43.23.141 <none> 8089:30415/TCP 5d23h
cm-acme-http-solver-sdlw7 NodePort 10.43.86.27 <none> 8089:32309/TCP 5d23h
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 5d23h
gitlab-gitlab-exporter ClusterIP 10.43.187.247 <none> 9168/TCP 5d23h
gitlab-gitlab-shell ClusterIP 10.43.246.124 <none> 22/TCP 5d23h
gitlab-minio-svc ClusterIP 10.43.117.249 <none> 9000/TCP 5d23h
gitlab-nginx-ingress-controller ExternalName <none> gitlab.mydomain.com 80:31487/TCP,443:31560/TCP,22:30539/TCP 5d23h
gitlab-nginx-ingress-controller-metrics ClusterIP 10.43.152.252 <none> 9913/TCP 5d23h
gitlab-nginx-ingress-controller-stats ClusterIP 10.43.173.191 <none> 18080/TCP 5d23h
gitlab-nginx-ingress-default-backend ClusterIP 10.43.116.121 <none> 80/TCP 5d23h
gitlab-postgresql ClusterIP 10.43.97.139 <none> 5432/TCP 5d23h
gitlab-prometheus-server ClusterIP 10.43.67.220 <none> 80/TCP 5d23h
gitlab-redis ClusterIP 10.43.36.138 <none> 6379/TCP,9121/TCP 5d23h
gitlab-registry ClusterIP 10.43.54.244 <none> 5000/TCP 5d23h
gitlab-unicorn ClusterIP 10.43.76.61 <none> 8080/TCP,8181/TCP 5d23h
Logs of the pod gitlab-nginx-ingress-controller-698fbc4c64-jxljq (others nginx-ingress-controller gives same logs): https://textuploader.com/1o9we
Any hint on what could be wrong in my configuration ?
Fell free to ask for more information on my setup.
Many thanks.

Answer 1

Well the issue is, Gitlab requires a valid SSL certificate for the domain in question. Which you do not seem to have according to the output of:
E1113 14:20:21.864785 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="gitlab.mydomain.com" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01"
Status:
Presented: true
Processing: true
Reason: Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
State: pending
The http-01 challenge is where it will try to do a web request to your domain, and it should return a 200 HTTP response. When you said yourself that https://gitlab.mydomain.com gives you a 404 response (hence it will fail to issue a valid certificate). To further diagnose this, check the output of the ingress responsible for the domain, and follow it down the "chain" until you identify where the 404 is being responded by.

Answer 2

The http01 challenge relies on port 80 (http) to be exposed to be able to answer the challenge. The option controller.service.enableHttp configures http, and is enabled by default (see here. But even if you've not touched this config, there might be an upstream component (i.e. a firewall) that blocks traffic on port 80.
Could you check if your ingress Service is listening on port 80, and reachable from the internet? You can try to go to your public IP on port 80 via a browser to check if you get a response from the ingress controller (or a backend).