ClusterIssuer with Cloudflare based DNS solver - kubernetes

Question

I'm trying to install the cert-manager ClusterIssuer on a AKS, and because the cluster is behind Azure Application Gateway I've gone down the route of using a DNS solver rather the HTTP. However, the challenge fails with an error calling the Cloudflare API. I've redacted emails and domains through the code snippets, the output of kubectl describe challenge rabt-cert-tls-g4mcl-1991965707-2468967546 is:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 72s cert-manager Challenge scheduled for processing
Warning PresentError 3s (x5 over 71s) cert-manager Error presenting challenge: Cloudflare API Error for GET "/zones?name=<domain>"
Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
I have followed the guide at https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/ and the issues at https://github.com/jetstack/cert-manager/issues/3021 and https://github.com/jetstack/cert-manager/issues/2384 but can't see any differences beyond the apiVersion of the issuer. I've checked this against the official documentation and there are no changes from what appears in these guides.
The relationship between ingress and cluster issuer seems fine; if I delete and recreate the ingress a new certificate, order and challenge are created. I've verified the secret is populated and I can print it to console, so it shouldn't be sending a blank string in the header. The token is valid, I can use the example CURL request from CloudFlare to check its validity.
Is there somewhere I can see logs and find out exactly what is being sent?
ClusterIssuer
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token-secret
namespace: cert-manager
type: Opaque
stringData:
api-token: ${CLOUDFLARE_API_TOKEN}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: rabt-letsencrypt
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: <email>
# ACME server URL for Let’s Encrypt’s staging environment.
# The staging environment will not issue trusted certificates but is
# used to ensure that the verification process is working properly
# before moving to production
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: rabt-letsencrypt-key
# Enable the HTTP-01 challenge provider
# you prove ownership of a domain by ensuring that a particular
# file is present at the domain
solvers:
- dns01:
cloudflare:
email: <email>
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-key
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rabt-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-protocol: https
appgw.ingress.kubernetes.io/ssl-redirect: "true"
cert-manager.io/cluster-issuer: rabt-letsencrypt
cert-manager.io/acme-challenge-type: dns01
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
tls:
- hosts:
- "*.rabt.<domain>"
secretName: rabt-cert-tls
rules:
- host: "mq.rabt.<domain>"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rabt-mq
port:
number: 15672
- host: es.rabt.<domain>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rabt-db-es-http
port:
number: 9200
- host: "kibana.rabt.<domain>"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rabt-kb-http
port:
number: 5601

Answer 1

As Harsh Manvar guessed, it was an issue with the secret. I wasn't running the kubectl apply command through envsubst so it was encoding the literal string "${CLOUDFLARE_API_TOKEN}"