Cert Manager Challenge Pending Kubernetes - kubernetes

Question

I have it working on one site application I already set up and now I am just trying to replicate the exact same thing for a different site/domain in another namespace.
So staging.correct.com is my working https domain
and staging.example.com is my not working https domain (http works - just not https)
When I do the following it shows 3 certs, the working one for correct and then 2 for the example.com when it should only have one for example:
kubectl get -A certificate
correct staging-correct-com True staging-correct-com-tls 10d
example staging-example-com False staging-example-com-tls 16h
example staging-example-website-com False staging-example-com-tls 17h
When I do:
kubectl get -A certificaterequests
It shows 2 certificate requests for the example
example staging-example-com-nl46v False 15h
example staging-example-website-com-plhqb False 15h
When I do:
kubectl get ingressroute -A
NAMESPACE NAME AGE
correct correct-ingress-route 10d
correct correct-secure-ingress-route 6d22h
kube-system traefik-dashboard 26d
example example-website-ingress-route 15h
example example-website-secure-ingress-route 15h
routing dashboard 29d
routing traefik-dashboard 6d21h
When I do:
kubectl get secrets -A (just showing the relevant ones)
correct default-token-bphcm kubernetes.io/service-account-token
correct staging-correct-com-tls kubernetes.io/tls
example default-token-wx9tx kubernetes.io/service-account-token
example staging-example-com-tls Opaque
example staging-example-com-wf224 Opaque
example staging-example-website-com-rzrvw Opaque
Logs from cert manager pod:
1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="staging.example.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-bqjsj" "related_resource_namespace”=“example” "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="staging-example-com-ltjl6-1661100417-771202110" "resource_namespace”=“example” "resource_version"="v1" "type"="HTTP-01"
When I do:
kubectl get challenge -A
example staging-example-com-nl46v-1661100417-2848337980 staging.example.com 15h
example staging-example-website-com-plhqb-26564845-3987262508 pending staging.example.com
When I do: kubectl get order -A
NAMESPACE NAME STATE AGE
example staging-example-com-nl46v-1661100417 pending 17h
example staging-example-website-com-plhqb-26564845 pending 17h
My yml files:
My ingress route:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
namespace: example
name: example-website-ingress-route
annotations:
kubernetes.io/ingress.class: "traefik"
cert-manager.io/issuer: example-issuer-staging
traefik.ingress.kubernetes.io/router.entrypoints: web
traefik.frontend.redirect.entryPoint: https
spec:
entryPoints:
- web
routes:
- match: Host(`staging.example.com`)
middlewares:
- name: https-only
kind: Rule
services:
- name: example-website
namespace: example
port: 80
my issuer:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: example-issuer-staging
namespace: example
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: example#example.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: staging-example-com-tls
# Enable the HTTP-01 challenge provider
solvers:
# An empty 'selector' means that this solver matches all domains
- http01:
ingress:
class: traefik
my middleware:
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
name: https-only
namespace: example
spec:
redirectScheme:
scheme: https
permanent: true
my secure ingress route:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
namespace: example
name: example-website-secure-ingress-route
annotations:
kubernetes.io/ingress.class: "traefik"
cert-manager.io/issuer: example-issuer-staging
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.frontend.redirect.entryPoint: https
spec:
entryPoints:
- websecure
routes:
- match: Host(`staging.example.com`)
kind: Rule
services:
- name: example-website
namespace: example
port: 80
tls:
domains:
- main: staging.example.com
options:
namespace: example
secretName: staging-example-com-tls
my service:
apiVersion: v1
kind: Service
metadata:
namespace: example
name: 'example-website'
spec:
type: ClusterIP
ports:
- protocol: TCP
name: http
port: 80
targetPort: 80
- protocol: TCP
name: https
port: 443
targetPort: 80
selector:
app: 'example-website'
my solver:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: staging-example-com
namespace: example
spec:
secretName: staging-example-com-tls
issuerRef:
name: example-issuer-staging
kind: Issuer
commonName: staging.example.com
dnsNames:
- staging.example.com
my app:
apiVersion: apps/v1
kind: ReplicaSet
metadata:
namespace: example
name: 'example-website'
labels:
app: 'example-website'
tier: 'frontend'
spec:
replicas: 1
selector:
matchLabels:
app: 'example-website'
template:
metadata:
labels:
app: 'example-website'
spec:
containers:
- name: example-website-container
image: richarvey/nginx-php-fpm:1.10.3
imagePullPolicy: Always
env:
- name: SSH_KEY
value: 'secret'
- name: GIT_REPO
value: 'url of source code for site'
- name: GIT_EMAIL
value: 'example#example.com'
- name: GIT_NAME
value: 'example'
ports:
- containerPort: 80
How can I delete all these secrets, orders, certificates and stuff in the example namespace and try again? Does cert-manager let you do this without restarting them continuously?
EDIT:
I deleted the namespace and redeployed, then:
kubectl describe certificates staging-example-com -n example
Spec:
Common Name: staging.example.com
Dns Names:
staging.example.com
Issuer Ref:
Kind: Issuer
Name: example-issuer-staging
Secret Name: staging-example-com-tls
Status:
Conditions:
Last Transition Time: 2020-09-26T21:25:06Z
Message: Issuing certificate as Secret does not contain a certificate
Reason: MissingData
Status: False
Type: Ready
Last Transition Time: 2020-09-26T21:25:07Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: staging-example-com-gnbl4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 3m10s cert-manager Issuing certificate as Secret does not exist
Normal Reused 3m10s cert-manager Reusing private key stored in existing Secret resource "staging-example-com-tls"
Normal Requested 3m9s cert-manager Created new CertificateRequest resource "staging-example-com-qrtfx"
So then I did:
kubectl describe certificaterequest staging-example-com-qrtfx -n example
Status:
Conditions:
Last Transition Time: 2020-09-26T21:25:10Z
Message: Waiting on certificate issuance from order example/staging-example-com-qrtfx-1661100417: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal OrderCreated 8m17s cert-manager Created Order resource example/staging-example-com-qrtfx-1661100417
Normal OrderPending 8m17s cert-manager Waiting on certificate issuance from order example/staging-example-com-qrtfx-1661100417: ""
So I did:
kubectl describe challenges staging-example-com-qrtfx-1661100417 -n example
Status:
Presented: true
Processing: true
Reason: Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 11m cert-manager Challenge scheduled for processing
Normal Presented 11m cert-manager Presented challenge using HTTP-01 challenge mechanism

Answer 1

I figured it out. The issue seems to be that IngressRoute (which is used in traefik) does not work with cert mananger. I just deployed this file, then the http check was confirmed, then I could delete it again. Hope this helps others with same issue.
Seems cert manager does support IngressRoute which is in Traefik? I opened the issue here so let's see what they say: https://github.com/jetstack/cert-manager/issues/3325
kubectl apply -f example-ingress.yml
File:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: example
name: example-ingress
annotations:
kubernetes.io/ingress.class: "traefik"
cert-manager.io/issuer: example-issuer-staging
spec:
rules:
- host: staging.example.com
http:
paths:
- path: /
backend:
serviceName: example-website
servicePort: 80
tls:
- hosts:
- staging.example.com
secretName: staging-example-com-tls