Issue with Self-signed certificate with Cert-Manager in Kubernetes - kubernetes

Question

I'm trying to add a self-signed certificate in my AKS cluster using Cert-Manager.
I created a ClusterIssuer for the CA certificate (to sign the certificate) and a second ClusterIssuer for the Certificate (self-signed) I want to use.
I am not sure if the certificate2 is being used correctly by Ingress as it looks like it is waiting for some event.
Am I following the correct way to do this?
This is the first ClusterIssuer "clusterissuer.yml":
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
This is the CA certificate "certificate.yml":
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: selfsigned-certificate
spec:
secretName: hello-deployment-tls-ca-key-pair
dnsNames:
- "*.default.svc.cluster.local"
- "*.default.com"
isCA: true
issuerRef:
name: selfsigned
kind: ClusterIssuer
This is the second ClusterIssuer "clusterissuer2.yml" for the certificate I want to use:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: hello-deployment-tls
spec:
ca:
secretName: hello-deployment-tls-ca-key-pair
and finally this is the self-signed certificate "certificate2.yml":
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: selfsigned-certificate2
spec:
secretName: hello-deployment-tls-ca-key-pair2
dnsNames:
- "*.default.svc.cluster.local"
- "*.default.com"
isCA: false
issuerRef:
name: hello-deployment-tls
kind: ClusterIssuer
I am using this certificate in an Ingress:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: "hello-deployment-tls"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
name: sonar-ingress
spec:
tls:
- secretName: "hello-deployment-tls-ca-key-pair2"
rules:
- http:
paths:
- pathType: Prefix
path: "/"
backend:
serviceName: sonarqube
servicePort: 80
As I do not have any registered domain name I just want to use the public IP to access the service over https://<Public_IP>.
When I access to the service https://<Public_IP> I can see that "Kubernetes Ingress Controller Fake Certificate" so i guess this is because the certificate is not globally recognize by the browser.
The strange thing is here. Theoretically the Ingress deployment is using the selfsigned-certificate2 but looks like it is not ready:
kubectl get certificate
NAME READY SECRET AGE
selfsigned-certificate True hello-deployment-tls-ca-key-pair 4h29m
selfsigned-certificate2 False hello-deployment-tls-ca-key-pair2 3h3m
selfsigned-secret True selfsigned-secret 5h25m
kubectl describe certificate selfsigned-certificate2
.
.
.
Spec:
Dns Names:
*.default.svc.cluster.local
*.default.com
Issuer Ref:
Kind: ClusterIssuer
Name: hello-deployment-tls
Secret Name: hello-deployment-tls-ca-key-pair2
Status:
Conditions:
Last Transition Time: 2021-10-15T11:16:15Z
Message: Waiting for CertificateRequest "selfsigned-certificate2-3983093525" to complete
Reason: InProgress
Status: False
Type: Ready
Events: <none>
Any idea?
Thank you in advance.

Answer 1

ApiVersions
First I noticed you're using v1alpha2 apiVersion which is depricated and will be removed in 1.6 cert-manager:
$ kubectl apply -f cluster-alpha.yaml
Warning: cert-manager.io/v1alpha2 ClusterIssuer is deprecated in v1.4+, unavailable in v1.6+; use cert-manager.io/v1 ClusterIssuer
I used apiVersion: cert-manager.io/v1 in reproduction.
Same for v1beta1 ingress, consider updating it to networking.k8s.io/v1.
What happens
I started reproducing your setup step by step.
I applied clusterissuer.yaml:
$ kubectl apply -f clusterissuer.yaml
clusterissuer.cert-manager.io/selfsigned created
$ kubectl get clusterissuer
NAME READY AGE
selfsigned True 11s
Pay attention that READY is set to True.
Next I applied certificate.yaml:
$ kubectl apply -f cert.yaml
certificate.cert-manager.io/selfsigned-certificate created
$ kubectl get cert
NAME READY SECRET AGE
selfsigned-certificate True hello-deployment-tls-ca-key-pair 7s
Next step is to add the second ClusterIssuer which is referenced to hello-deployment-tls-ca-key-pair secret:
$ kubectl apply -f clusterissuer2.yaml
clusterissuer.cert-manager.io/hello-deployment-tls created
$ kubectl get clusterissuer
NAME READY AGE
hello-deployment-tls False 6s
selfsigned True 3m50
ClusterIssuer hello-deployment-tls is not ready. Here's why:
$ kubectl describe clusterissuer hello-deployment-tls
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ErrGetKeyPair 10s (x5 over 75s) cert-manager Error getting keypair for CA issuer: secret "hello-deployment-tls-ca-key-pair" not found
Warning ErrInitIssuer 10s (x5 over 75s) cert-manager Error initializing issuer: secret "hello-deployment-tls-ca-key-pair" not found
This is expected behaviour since:
When referencing a Secret resource in ClusterIssuer resources (eg
apiKeySecretRef) the Secret needs to be in the same namespace as the
cert-manager controller pod. You can optionally override this by using
the --cluster-resource-namespace argument to the controller.
Reference
Answer - how to move forward
I edited the cert-manager deployment so it will look for secrets in default namespace (this is not ideal, I'd use issuer instead in default namespace):
$ kubectl edit deploy cert-manager -n cert-manager
spec:
containers:
- args:
- --v=2
- --cluster-resource-namespace=default
It takes about a minute for cert-manager to start. Redeployed clusterissuer2.yaml:
$ kubectl delete -f clusterissuer2.yaml
clusterissuer.cert-manager.io "hello-deployment-tls" deleted
$ kubectl apply -f clusterissuer2.yaml
clusterissuer.cert-manager.io/hello-deployment-tls created
$ kubectl get clusterissuer
NAME READY AGE
hello-deployment-tls True 3s
selfsigned True 5m42s
Both are READY. Moving forward with certificate2.yaml:
$ kubectl apply -f cert2.yaml
certificate.cert-manager.io/selfsigned-certificate2 created
$ kubectl get cert
NAME READY SECRET AGE
selfsigned-certificate True hello-deployment-tls-ca-key-pair 33s
selfsigned-certificate2 True hello-deployment-tls-ca-key-pair2 6s
$ kubectl get certificaterequest
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
selfsigned-certificate-jj98f True True selfsigned system:serviceaccount:cert-manager:cert-manager 52s
selfsigned-certificate2-jwq5c True True hello-deployment-tls system:serviceaccount:cert-manager:cert-manager 25s
Ingress
When host is not added to ingress, it doesn't create any certificates and seems to used some fake one from ingress which is issued by CN = Kubernetes Ingress Controller Fake Certificate.
Events from ingress:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BadConfig 5s cert-manager TLS entry 0 is invalid: secret "example-cert" for ingress TLS has no hosts specified
When I added DNS to ingress:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateCertificate 4s cert-manager Successfully created Certificate "example-cert"
Answer, part 2 (about ingress, certificates and issuer)
You don't need to create a certificate if you're referencing to issuer in ingress rule. Ingress will issue certificate for you when all details are presented, such as:
annotation cert-manager.io/cluster-issuer: "hello-deployment-tls"
spec.tls part with host within
spec.rules.host
OR
if you want to create certificate manually and ask ingress to use it, then:
remove annotation cert-manager.io/cluster-issuer: "hello-deployment-tls"
create certificate manually
refer to it in ingress rule.
You can check certificate details in browser and find that it no longer has issuer as CN = Kubernetes Ingress Controller Fake Certificate, in my case it's empty.
Note - cert-manager v1.4
Initially I used a bit outdated cert-manager v1.4 and got this issue which has gone after updating to 1.4.1.
It looks like:
$ kubectl describe certificaterequest selfsigned-certificate2-45k2c
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal cert-manager.io 41s cert-manager Certificate request has been approved by cert-manager.io
Warning DecodeError 41s cert-manager Failed to decode returned certificate: error decoding certificate PEM block
Useful links:
Setting up self-singed Issuer
Setting up CA issuers
Cluster Issuers