Cert-Manager k8s, Renewing certificate process - kubernetes

Question

I have installed cert manager on a k8s cluster:
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.5.3 --set installCRDs=true
My objective is to do mtls communication between micro-services running in same name-space.
For this purpose I have created a ca issuer .i.e..
kubectl get issuer -n sandbox -o yaml
apiVersion: v1
items:
- apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"cert-manager.io/v1","kind":"Issuer","metadata":{"annotations":{},"name":"ca-issuer","namespace":"sandbox"},"spec":{"ca":{"secretName":"tls-internal-ca"}}}
creationTimestamp: "2021-09-16T17:24:58Z"
generation: 1
managedFields:
- apiVersion: cert-manager.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:ca:
.: {}
f:secretName: {}
manager: HashiCorp
operation: Update
time: "2021-09-16T17:24:58Z"
- apiVersion: cert-manager.io/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:conditions: {}
manager: controller
operation: Update
time: "2021-09-16T17:24:58Z"
name: ca-issuer
namespace: sandbox
resourceVersion: "3895820"
selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/issuers/ca-issuer
uid: 90f0c811-b78d-4346-bb57-68bf607ee468
spec:
ca:
secretName: tls-internal-ca
status:
conditions:
message: Signing CA verified
observedGeneration: 1
reason: KeyPairVerified
status: "True"
type: Ready
kind: List
metadata:
resourceVersion: ""
selfLink: ""
Using this ca issuer, I have created certificates for my two micro-service .i.e.
kubectl get certificate -n sandbox
NAME READY SECRET Age
service1-certificate True service1-certificate 3d
service2-certificate True service2-certificate 2d23h
which is configured as
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
annotations:
meta.helm.sh/release-name: service1
meta.helm.sh/release-namespace: sandbox
creationTimestamp: "2021-09-17T10:20:21Z"
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
managedFields:
- apiVersion: cert-manager.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
.: {}
f:commonName: {}
f:dnsNames: {}
f:duration: {}
f:issuerRef:
.: {}
f:kind: {}
f:name: {}
f:renewBefore: {}
f:secretName: {}
f:subject:
.: {}
f:organizations: {}
f:usages: {}
manager: Go-http-client
operation: Update
time: "2021-09-17T10:20:21Z"
- apiVersion: cert-manager.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:privateKey: {}
f:status:
.: {}
f:conditions: {}
f:notAfter: {}
f:notBefore: {}
f:renewalTime: {}
f:revision: {}
manager: controller
operation: Update
time: "2021-09-20T05:14:12Z"
name: service1-certificate
namespace: sandbox
resourceVersion: "5177051"
selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/certificates/service1-certificate
uid: 0cf1ea65-92a1-4b03-944e-b847de2c80d9
spec:
commonName: example.com
dnsNames:
- service1
duration: 24h0m0s
issuerRef:
kind: Issuer
name: ca-issuer
renewBefore: 12h0m0s
secretName: service1-certificate
subject:
organizations:
- myorg
usages:
- client auth
- server auth
status:
conditions:
- lastTransitionTime: "2021-09-20T05:14:13Z"
message: Certificate is up to date and has not expired
observedGeneration: 1
reason: Ready
status: "True"
type: Ready
notAfter: "2021-09-21T05:14:13Z"
notBefore: "2021-09-20T05:14:13Z"
renewalTime: "2021-09-20T17:14:13Z"
revision: 5
Now as you could see in the configuration I have configured to renew them in 12 hours. However, the secrets created via this custom certificate resource are still aged to two days (the first it was created). I was thinking this tls secret will be renewed via cert manager each day) .i.e.
kubectl get secrets service1-certificate service2-certificate -n sandbox -o wide
NAME TYPE DATA AGE
service1-certificate kubernetes.io/tls 3 2d23h
service2-certificate kubernetes.io/tls 3 3d1h
Is there is some wrong in my understanding? In the certmangager pod logs I do see some error around renewing .i.e.
I0920 05:14:04.649158 1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key”=“sandbox/service1-certificate" "message"="Renewing certificate as renewal was scheduled at 2021-09-19 08:24:13 +0000 UTC" "reason"="Renewing"
I0920 05:14:04.649235 1 conditions.go:201] Setting lastTransitionTime for Certificate “service1-certificate" condition "Issuing" to 2021-09-20 05:14:04.649227766 +0000 UTC m=+87949.327215532
I0920 05:14:04.652174 1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="sandbox/service2 "message"="Renewing certificate as renewal was scheduled at 2021-09-19 10:20:22 +0000 UTC" "reason"="Renewing"
I0920 05:14:04.652231 1 conditions.go:201] Setting lastTransitionTime for Certificate “service2-certificate" condition "Issuing" to 2021-09-20 05:14:04.652224302 +0000 UTC m=+87949.330212052
I0920 05:14:04.671111 1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671094596 +0000 UTC m=+87949.349082328
I0920 05:14:04.671344 1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671332206 +0000 UTC m=+87949.349319948
 
I0920 05:14:12.703039 1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
 
I0920 05:14:12.703896 1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.7038803 +0000 UTC m=+87957.381868045
 
I0920 05:14:12.749502 1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
 
I0920 05:14:12.750096 1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.750082572 +0000 UTC m=+87957.428070303
I0920 05:14:13.009032 1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0920 05:14:13.117843 1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0920 05:14:13.119366 1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:13.119351795 +0000 UTC m=+87957.797339520
I0920 05:14:13.122820 1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox\service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service-certificate\": the object has been modified; please apply your changes to the latest version and try again"
I0920 05:14:13.123907 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service2-certificate-t92qh" condition "Approved" to 2021-09-20 05:14:13.123896104 +0000 UTC m=+87957.801883833
I0920 05:14:13.248082 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Approved" to 2021-09-20 05:14:13.248071551 +0000 UTC m=+87957.926059296
I0920 05:14:13.253488 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “serivce2-certificate-t92qh" condition "Ready" to 2021-09-20 05:14:13.253474153 +0000 UTC m=+87957.931461871
I0920 05:14:13.388001 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Ready" to 2021-09-20 05:14:13.387983783 +0000 UTC m=+87958.065971525


Answer 1

Short answer
Based on logs and details from certificate you provided it's safe to say it's working as expected.
Pay attention to revision: 5 in your certificate, which means that certificate has been renewed 4 times already. If you try to look there now, this will be 6 or 7 because certificate is updated every 12 hours.
Logs
First thing which can be really confusing is error messages in cert-manager pod. This is mostly noisy messages which are not really helpful by itself.
See about it here Github issue comment and here github issue 3667.
In case logs are really needed, verbosity level should be increased by setting args to --v=5 in the cert-manager deployment. To edit a deployment run following command:
kubectl edit deploy cert-manager -n cert-manager
How to check certificate/secret
When certificate is renewed, secret's and certificate's age are not changed, but content is edited, for instance resourceVersion in secret and revision in certificate.
Below are options to check if certificate was renewed:
Check this by getting secret in yaml before and after renew:
kubectl get secret example-certificate -o yaml > secret-before
And then run diff between them. It will be seen that tls.crt as well as resourceVersion is updated.
Look into certificate revision and dates in status
(I set duration to minimum possible 1h and renewBefore 55m, so it's updated every 5 minutes):
$ kubectl get cert example-cert -o yaml
notAfter: "2021-09-21T14:05:24Z"
notBefore: "2021-09-21T13:05:24Z"
renewalTime: "2021-09-21T13:10:24Z"
revision: 7
Check events in the namespace where certificate/secret are deployed:
$ kubectl get events
117s Normal Issuing certificate/example-cert The certificate has been successfully issued
117s Normal Reused certificate/example-cert Reusing private key stored in existing Secret resource "example-staging-certificate"
6m57s Normal Issuing certificate/example-cert Renewing certificate as renewal was scheduled at 2021-09-21 13:00:24 +0000 UTC
6m57s Normal Requested certificate/example-cert Created new CertificateRequest resource "example-cert-bs8g6"
117s Normal Issuing certificate/example-cert Renewing certificate as renewal was scheduled at 2021-09-21 13:05:24 +0000 UTC
117s Normal Requested certificate/example-cert Created new CertificateRequest resource "example-cert-7x8cf" UTC
Look at certificaterequests:
$ kubectl get certificaterequests
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
example-cert-2pxdd True True ca-issuer system:serviceaccount:cert-manager:cert-manager 14m
example-cert-54zzc True True ca-issuer system:serviceaccount:cert-manager:cert-manager 4m29s
example-cert-8vjcm True True ca-issuer system:serviceaccount:cert-manager:cert-manager 9m29s
Check logs in cert-manager pod to see four stages:
I0921 12:45:24.000726 1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="default/example-cert" "message"="Renewing certificate as renewal was scheduled at 2021-09-21 12:45:24 +0000 UTC" "reason"="Renewing"
I0921 12:45:24.000761 1 conditions.go:201] Setting lastTransitionTime for Certificate "example-cert" condition "Issuing" to 2021-09-21 12:45:24.000756621 +0000 UTC m=+72341.194879378
I0921 12:45:24.120503 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Approved" to 2021-09-21 12:45:24.12049391 +0000 UTC m=+72341.314616684
I0921 12:45:24.154092 1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Ready" to 2021-09-21 12:45:24.154081971 +0000 UTC m=+72341.348204734
Note
Very important that not all issuers support duration and renewBefore flags. E.g. letsencrypt still doesn't work with it and have 90 default days.
Refence.