Cert-Manager is not issuing any certificates Google Cloud Kubernetes - kubernetes

I am struggling to get cert-manager to issue certificates from let's encrypt which I can use with my ingress in Google Cloud Kubernetes.
I have a cluster running with a deployment, service and ingress in the default namespace. I also registered a domain at namecheap and added an A record to the IP address from my ingress. Now I can access the website with http and everything is fine. Now I want to move to https and things do not work.
I have installed cert-manager:
kubectl create namespace cert-manager
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.13.1/cert-manager.yaml
I verified that cert-manager is running:
kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-5655447474-kw9k7 1/1 Running 0 99m
cert-manager-cainjector-59c9dfd4f7-fjzbf 1/1 Running 0 99m
cert-manager-webhook-865b8fb666-7kmx2 1/1 Running 0 99m
Now I created a ClusterIssuer kubectl apply -f letsencrypt-prod.yaml in the default namespace:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: 'my#email.com'
privateKeySecretRef:
name: letsencrypt-prod
http01: {}
And added a certificate with kubectl apply -f certificate.yaml:
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: my-certs
spec:
secretName: my-certs
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
commonName: "www.mydomain.de"
dnsNames:
- "www.mydomain.de"
acme:
config:
- dns01:
provider: cloud-dns
domains:
- "www.mydomain.de"
This is my ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "my-ip"
spec:
rules:
- host: www.mydomain.de
http:
paths:
- backend:
serviceName: my-service
servicePort: 80
tls:
- hosts:
- www.mydomain.de
secretName: my-certs
Now when I run kubectl describe certificate my-certs there are no events:
kubectl describe certificate my-certs
Name: my-certs
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"my-certs","namespace":"default"},"sp...
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2020-02-23T13:30:15Z
Generation: 1
Resource Version: 787204
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/my-certs
UID: a027a698-5640-11ea-bce8-42010a9c00dc
Spec:
Acme:
Config:
Dns 01:
Provider: cloud-dns
Domains:
www.mydomain.de
Common Name: www.mydomain.de
Dns Names:
www.mydomain.de
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-prod
Secret Name: my-certs
Events: <none>
And in Google Cloud Console I see the message "Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP" for the ingress.
What is wrong here or what am I missing?

Instead of creating the certificate resource manually you can add below annotation to the ingress resource:
cert-manager.io/issuer: "letsencrypt-prod"
Cert-manager will read the annotation and use them to create a certificate.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "my-ip"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
rules:
- host: www.mydomain.de
http:
paths:
- backend:
serviceName: my-service
servicePort: 80
tls:
- hosts:
- www.mydomain.de
secretName: my-certs
After this check if a secret and certificate with name my-certs got created or not.
Also check the related issue

Related

ALB Ingress Controller, Terminating TLS via AWS ACM. How to go End-To-End?

We currently use the AWS ALB Ingress Controller to front out ingress, and terminate SSL using a certificate from AWS ACM. This work fine.
Is there a way, to also encrypt the traffic from the Load balancer to the cluster?
Here is what I attempted
Install/Configure Cert Manager
Add a TLS Secret to the Ingress
Change the ingress annotation to set the backend protocol to HTTPS alb.ingress.kubernetes.io/backend-protocol: HTTPS
This... results in a 502 gateway error
here is my current, working ingress, with only the relevant parts still shown.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/certificate-arn: real-cert-arn
alb.ingress.kubernetes.io/group.name: public.monitor
alb.ingress.kubernetes.io/group.order: "40"
alb.ingress.kubernetes.io/healthcheck-path: /
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/load-balancer-name: monitoring-public-qa
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-FS-1-2-Res-2020-10
cert-manager.io/cluster-issuer: cert-manager-r53-qa
kubernetes.io/ingress.class: alb
spec:
rules:
- host: goldilocks.qa.realdomain.com
http:
paths:
- backend:
service:
name: goldilocks-dashboard
port:
name: http
path: /*
pathType: ImplementationSpecific
tls:
- hosts:
- goldilocks.qa.realdomain.com
secretName: goldilocks-qa-cert
status:
loadBalancer:
ingress:
- hostname: real-lb-address.us-gov-west-1.elb.amazonaws.com
my (staging) cert exists, and appears fine.
[ec2-user#ip-10-17-2-102 ~]$ kubectl get cert -n goldilocks goldilocks-qa-cert -o yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: goldilocks-qa-cert
namespace: goldilocks
ownerReferences:
- apiVersion: networking.k8s.io/v1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: goldilocks-dashboard
uid: 2494e5be-e624-471b-afd3-c8d56f5dc853
resourceVersion: "81003934"
uid: d494a1ec-7509-474e-b154-d5db8ceb86c0
spec:
dnsNames:
- goldilocks.qa.realdomain.com
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: cert-manager-r53-qa
secretName: goldilocks-qa-cert
usages:
- digital signature
- key encipherment
status:
conditions:
- lastTransitionTime: "2023-02-08T16:06:56Z"
message: Certificate is up to date and has not expired
The fact I can not figure out what to google to find this answer, leads me to think that I'm attempting to do something weird? I understand I could just terminate the TLS on the pod, but I didn't want to rely on the lets encrypt to provide me good/valid certs, I just want the traffic encrypted.

kubectl get certificates : No resources found using cert-manager

I don't undestand why i can't get certificates on K8S using cert-manager
I installed cert-manager : https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml
I created ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: user#example.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: example-issuer-account-key
solvers:
- http01:
ingress:
class: nginx
I created ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
rules:
- host: mytest.example.fr
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: webapp
port:
number: 80
tls:
- hosts:
- mytest.example.fr
secretName: letsencrypt-staging
But when i try to get an certificate i get 'no resources found'
Any idea ?
Thank you for your help
If you don't want to create kind certificate you can use
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: cluster-issuer-name
namespace: development
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: harsh#example.com
privateKeySecretRef:
name: secret-name
solvers:
- http01:
ingress:
class: nginx-class-name
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx-class-name
cert-manager.io/cluster-issuer: cluster-issuer-name
nginx.ingress.kubernetes.io/rewrite-target: /
name: example-ingress
spec:
rules:
- host: sub.example.com
http:
.
. #Path and service configs
.
.
tls:
- hosts:
- sub.example.com
secretName: secret-name
ingress will call clusterisser and it will auto-create certificate for you.
Update ingress resources as per need if you are higher version 1.18 or above
Notes
Make sure you are using the URL https://acme-v02.api.letsencrypt.org/directory in clusterissue or else you will get fake certificate in browser.
For refrence you can read more here :
https://stackoverflow.com/a/55183209/5525824
Make sure also you ingress pointing to proper clusterissuer if
you have created new.
Also don't use same privateKeySecretRef:name: secret-name you
need to delete it or use the new name as fake certificate
now stored in that secret so.
Certificates are not created automatically by cert-manager.
You have to create a YAML yourself. And use the issuer name that you have already created
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: my-certificate
namespace: default
spec:
secretName: set-a-new-name-here
issuerRef:
name: letsencrypt-staging
kind: ClusterIssuer
commonName: mytest.example.fr
dnsNames:
- mytest.example.fr

Certificate always in 'False' state using LetsEncrypt with cluster issuer in k8s

I am unable to issue a working certificate for my ingress host in k8s. I use a ClusterIssuer to issue certificates and the same ClusterIssuer has issued certificates in the past for my ingress hosts under my domain name *xyz.com. But all of a sudden neither i can issue new Certificate with state 'True' for my host names nor a proper certificate secret (kubernetes.io/tls) gets created (but instead an Opaque secret gets created).
**strong text**
**kubectl describe certificate ingress-cert -n abc**
Name: ingress-cert
Namespace: abc
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1beta1
Kind: Certificate
Metadata:
Creation Timestamp: 2021-09-08T07:48:32Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: test-ingress
UID: c03ffec0-df4f-4dbb-8efe-4f3550b9dcc1
Resource Version: 146643826
Self Link: /apis/cert-manager.io/v1beta1/namespaces/abc/certificates/ingress-cert
UID: 90905ab7-22d2-458c-b956-7100c4c77a8d
Spec:
Dns Names:
abc.xyz.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt
Secret Name: ingress-cert
Status:
Conditions:
Last Transition Time: 2021-09-08T07:48:33Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2021-09-08T07:48:33Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: ingress-cert-gdq7g
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 11m cert-manager Issuing certificate as Secret does not exist
Normal Generated 11m cert-manager Stored new private key in temporary Secret resource "ingress-cert-gdq7g"
Normal Requested 11m cert-manager Created new CertificateRequest resource "ingress-cert-dp6sp"
I checked the certificate request and it contains no events. Also i can see no challenges. I have added the logs below. Any help would be appreciated
kubectl describe certificaterequest ingress-cert-dp6sp -n abc
Namespace: abc
Labels: <none>
Annotations: cert-manager.io/certificate-name: ingress-cert
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: ingress-cert-gdq7g
API Version: cert-manager.io/v1beta1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2021-09-08T07:48:33Z
Generate Name: ingress-cert-
Generation: 1
Owner References:
API Version: cert-manager.io/v1alpha2
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: ingress-cert
UID: 90905ab7-22d2-458c-b956-7100c4c77a8d
Resource Version: 146643832
Self Link: /apis/cert-manager.io/v1beta1/namespaces/abc/certificaterequests/ingress-cert-dp6sp
UID: fef72617-fc1d-4384-9f4b-a7e4502582d8
Spec:
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt
Request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2Z6Q0NBV2NDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxMNgphTGhZNjhuNnhmMUprYlF5ek9OV1J4dGtLOXJrbjh5WUtMd2l4ZEFMVUl0TERra0t6Uksyb3loZzRMMThSQmQvCkNJaGJ5RXBYNnlRditKclRTOC84T1A0MWdwTUxBLzROdVhXWWtyeWhtZFdNaFlqa21OOFpiTUk1SlZZcVV2cVkKRWQ1b2cydmVmSjU1QlJPRExsd0o3YjBZa3hXckUwMGJxQ1ExWER6ZzFhM08yQ2JWd1NQT29WV2x6Uy9CdzRYVgpMeVdMS3E4QU52b2dZMUxXRU8xcG9YelRObm9LK2U2YVZueDJvQ1ZLdGxPaG1iYXRHYXNSaTJKL1FKK0dOWHovCnFzNXVBSlhzYVErUzlxOHIvbmVMOXNPYnN2OWd1QmxCK09yQVg2eHhkNHZUdUIwVENFU00zWis2c2MwMFNYRXAKNk01RlY3dkFFeDQyTWpuejVoa0NBd0VBQWFBNk1EZ0dDU3FHU0liM0RRRUpEakVyTUNrd0p3WURWUjBSQkNBdwpIb0ljY25kemMyZHdMbU5zYjNWa1oyRjBaUzV0YVdOeWIyWnBiaTVrWlRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUFTQ0cwTXVHMjZRbVFlTlBFdmphNHZqUUZOVFVINWVuMkxDcXloY2ZuWmxocWpMbnJqZURuL2JTV1hwdVIKTnhXTnkxS0EwSzhtMG0rekNPbWluZlJRS1k2eHkvZU1WYkw4dTgrTGxscDEvRHl3UGxvREE2TkpVOTFPaDM3TgpDQ0E4NWphLy9FYVVvK0p5aHBzaTZuS1d4UXRpYXdmYXhuNUN4SENPWGF5Qzg0Q0IzdGZ2WWp6YUF3Ykx4akxYCmxvd09LUHNxSE51ZktFM0NtcjZmWGgramd5VWhxamYwOUJHeGxCWEFsSVNBNkN5dzZ2UmpWamFBOW82TmhaTXUKbmdheWZON00zUzBBYnAzVFFCZW8xYzc3QlFGaGZlSUE5Sk51SWtFd3EvNXppYVY1RDErNUxSSnR5ZkVpdnJLTwpmVjQ5WkpCL1BGOTdiejhJNnYvVW9CSkc2Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
Status:
Conditions:
Last Transition Time: 2021-09-08T07:48:33Z
Message: Waiting on certificate issuance from order abc/ingress-cert-dp6sp-3843501305: ""
Reason: Pending
Status: False
Type: Ready
Events: <none>
Here is the ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 20m
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt"
spec:
rules:
- host: abc.xyz.com
http:
paths:
- path: /static
backend:
serviceName: app-service
servicePort: 80
- path: /
backend:
serviceName: app-service
servicePort: 8000
tls:
- hosts:
- abc.xyz.com
secretName: ingress-cert
Here is the clusterissuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: example#user.de
privateKeySecretRef:
name: letsencrypt-key
solvers:
- http01:
ingress:
class: nginx
Works only with Nginx Ingress Controller
I was using ClusterIssuer but I changed it to Issuer and it works.
-- Install cert-manager (Installed version 1.6.1) and be sure that the three pods are running
-- Create an Issuer by appling this yml be sure that the issuer is running.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-nginx
namespace: default
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: example#example.com
privateKeySecretRef:
name: letsencrypt-nginx-private-key
solvers:
- http01:
ingress:
class: nginx
-- Add this to your ingress annotations
cert-manager.io/issuer: letsencrypt-nginx
-- Add the secretName to your ingress spec.tls.hosts
spec:
tls:
- hosts:
- yourdomain.com
secretName: letsencrypt-nginx
Notice that the Nginx Ingress Controller is able to generate the Certificate CRD automatically via a special annotation: cert-manager.io/issuer. This saves work and time, because you don't have to create and maintain a separate manifest for certificates as well (only the Issuer manifest is required). For other ingresses you may need to provide the Certificate CRD as well.
Ideally your ingress pointing to the secret which is storing the secret or SSL/TLS key cert.
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 20m
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt"
spec:
rules:
- host: abc.xyz.com
http:
paths:
- path: /static
backend:
serviceName: app-service
servicePort: 80
- path: /
backend:
serviceName: app-service
servicePort: 8000
tls:
- hosts:
- abc.xyz.com
secretName: letsencrypt-key
Your cluster issue storing the key
privateKeySecretRef:
name: letsencrypt-key
You have to use this secret and attach this to ingress.
If secret already storing cert with a domain
test.example.com and you are trying to get a new cert with hello.example.com
in this case using cluster issuer will overwrite the secret and might loss old cert stored inside secret.
You can create the multiple clusterissuer,
One storing and connect to single ingress, first.example.com
Second cluster issuer with different key name
privateKeySecretRef:
name: letsencrypt-key
and new key or secret will get attached to the ingress.

Cert-managed with wilcard and Cloudflare DNS, stuck at 'OrderCreated'

I'm trying to get cert-manager and letsencrypt working for a wildcard domain. I've pointed the wildcard A host to the load balancer IP (GKE). Here is the secret and issuer:
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-key
namespace: cert-manager
type: Opaque
data:
apikey: BASE_64_ENCODED_API_KEY
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
namespace: cert-manager
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: EMAIL
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- dns01:
cloudflare:
email: EMAIL
apiKeySecretRef:
name: cloudflare-api-key
key: apikey
and here is my ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: apps-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
tls:
- hosts:
- "*.sampledomain.com"
secretName: letsencrypt-staging
rules:
- host: phpmyadmin.sampledomain.com
http:
paths:
- backend:
serviceName: phpmyadmin
servicePort: 8081
The events are stuck at 'OrderCreated'. On checking the logs:
E0817 08:42:45.872348 1 base_controller.go:189] cert-manager/controller/challenges "msg"="re-queuing item due to error processing" "error"="Cloudflare API Error \n\t Error: 9103: Unknown X-Auth-Key or X-Auth-Email" "key"="default/letsencrypt-staging-3055668421-0"
There was a typo in my email address :|

Istio: Ingress for ACME-challenge not working (503)

We are running Istio 1.1.3 on 1.12.5-gke.10 cluster-nodes.
We use certmanager for managing our let's encrypt certificates.
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: certs.ourdomain.nl
namespace: istio-system
spec:
secretName: certs.ourdomain.nl
newBefore: 360h # 15d
commonName: operations.ourdomain.nl
dnsNames:
- operations.ourdomain.nl
issuerRef:
name: letsencrypt
kind: ClusterIssuer
acme:
config:
- http01:
ingressClass: istio
domains:
- operations.ourdomain.nl
Next thing we see the acme backend, service (nodeport and ingress) deployed. The ingress (auto-generated) looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: istio
generateName: cm-acme-http-solver-
generation: 1
labels:
certmanager.k8s.io/acme-http-domain: "1734084804"
certmanager.k8s.io/acme-http-token: "1476005735"
name: cm-acme-http-solver-69vzw
namespace: istio-system
ownerReferences:
- apiVersion: certmanager.k8s.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Certificate
name: certs.ourdomain.nl
uid: 751011d2-4fc8-11e9-b20e-42010aa40101
spec:
rules:
- host: operations.ourdomain.nl
http:
paths:
- backend:
serviceName: cm-acme-http-solver-fzk8q
servicePort: 8089
path: /.well-known/acme-challenge/dnrcr-LRRMdXhBaUefjqpHQx8ytYuk-feEfXu9gW-Ck
status:
loadBalancer: {}
However, when we try to access the url operations.ourdomain.nl /.well-known/acme-challenge/dnrcr-LRRMdXhBaUefjqpHQx8ytYuk-feEfXu9gW-Ck we get a 404.
We do have a loadbalancer for istio:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
labels:
app: istio-ingress
chart: gateways-1.1.0
heritage: Tiller
istio: ingress
release: istio
name: istio-ingress
namespace: istio-system
spec:
selector:
app: istio-ingress
servers:
- hosts:
- operations.ourdomain.nl
#port:
# name: http
# number: 80
# protocol: HTTP
#tls:
# httpsRedirect: true
- hosts:
- operations.ourdomain.nl
port:
name: https
number: 443
protocol: HTTPS
tls:
credentialName: certs.ourdomain.nl
mode: SIMPLE
privateKey: sds
serverCertificate: sds
This interesting article gives a good insight in how the acme-challenge is supposed to work. For purpose of testing we have removed the port 80 and redirect to https in our custom gateway. We have added the autogenerated k8s gateway, listening only on port 80.
Istio is supposed to create a virtualservice for the acme-challenge. This seems to be happening, because now, when we request the acme-challenge url we get a 503: upstream connect error or disconnect/reset before headers. I believe this means the request gets to the gateway and is matched by a virtualservice, but there is no service / healthy pod to revert the traffic to.
We do see some possibly interesting logging in the istio-pilot:
“ProxyStatus”: {“endpoint_no_pod”:
{“cm-acme-http-solver-l5j2g.istio-system.svc.cluster.local”:
{“message”: “10.16.57.248”}
I have double checked and the service mentioned above does have a pod it is exposing. So I am not sure whether this line is relevant to this issue.
The acme-challenge pods do not have an istio-sidecar. Could this be the issue? If so: why does it apparently work for others