Error "connect: connection refused" trying to fullfill acme challenge - kubernetes

Question

I am pretty new to k8s and currently I am trying to set up a K8s-Cluster on a baremetal server with a publix IP.
However during the validation process letsencrypt cannot access the cluster. After following the trouble shooting I could find the following error: "Connection refused"
I set up my cluster with kubespray
K8s version: v1.22.4
Cert-Manager:
kubectl create namespace cert-manager
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml
Nginx-Ingress-Controller:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/v1.0.4/deploy/static/provider/baremetal/deploy.yaml
ClusterIssuer:
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-cluster-issuer
#namespace: cert-manager
namespace: default
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: dav#my-company.com
privateKeySecretRef:
name: letsencrypt-cluster-issuer-key
solvers:
- http01:
ingress:
class: nginx
Certificate.yml:
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: example-app
namespace: default
#namespace: cert-manager
spec:
dnsNames:
- {real-dns}.com
secretName: example-app-tls
issuerRef:
name: letsencrypt-cluster-issuer
#name: letsencrypt-staging
kind: ClusterIssuer
After running kubectl get all -n default, i got
root#node1:~# kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/cm-acme-http-solver-2vx5w 1/1 Running 0 8s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cm-acme-http-solver-gp2f6 NodePort 10.233.13.8 <none> 8089:30158/TCP 8s
service/kubernetes ClusterIP 10.233.0.1 <none> 4
I think maybe the cert-manager could not create an issuer for the service cm-acme-http-solver-gp2f6?
Can someone plase give me a hint, how can I solve the problem?
Many thanks in advance.
Edit:
There is also debug step I tried:
clusterissuer:
root#node1:~# kubectl describe clusterissuer letsencrypt-cluster-
issuer
Name: letsencrypt-cluster-issuer
Namespace:
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2021-12-16T17:35:30Z
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:acme:
.:
f:email:
f:privateKeySecretRef:
.:
f:name:
f:server:
f:solvers:
Manager: OpenAPI-Generator
Operation: Update
Time: 2021-12-16T17:35:30Z
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:acme:
.:
f:lastRegisteredEmail:
f:uri:
f:conditions:
Manager: controller
Operation: Update
Time: 2021-12-16T17:35:31Z
Resource Version: 230199
UID: 95d183fb-c50e-49ed-83fb-98ceee0f1b7a
Spec:
Acme:
Email: #######.com
Preferred Chain:
Private Key Secret Ref:
Name: letsencrypt-cluster-issuer-key
Server: https://acme-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: nginx
Status:
Acme:
Last Registered Email: trang#my-company.com
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/322624000
Conditions:
Last Transition Time: 2021-12-16T17:35:31Z
Message: The ACME account was registered with the
ACME server
Observed Generation: 1
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
CertificateRequest:
root#node1:~# kubectl describe CertificateRequest example-app-w59ng
Name: example-app-w59ng
Namespace: default
Labels: <none>
Annotations: cert-manager.io/certificate-name: example-app
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: example-app-qddzv
API Version: cert-manager.io/v1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2021-12-22T14:07:00Z
Generate Name: example-app-
Generation: 1
Managed Fields:
API Version: cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
Manager: controller
Operation: Update
Time: 2021-12-22T14:07:00Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: example-app
UID: 5c47b8a9-c076-4387-b117-bb4b1864448d
Resource Version: 1520892
UID: 2fe3875a-4b4d-4139-bef4-820679f8356e
Spec:
Extra:
authentication.kubernetes.io/pod-name:
cert-manager-7c6f78c46d-8r9n7
authentication.kubernetes.io/pod-uid:
5f7638e9-27e8-4a3f-b149-96e9a88d0c74
Groups:
system:serviceaccounts
system:serviceaccounts:cert-manager
system:authenticated
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-cluster-issuer
Request: [...]
UID: 8dfc1be6-2e0b-4a2e-8087-a274110c4b74
Username: system:serviceaccount:cert-manager:cert-manager
Status:
Conditions:
Last Transition Time: 2021-12-22T14:07:00Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2021-12-22T14:07:00Z
Message: Waiting on certificate issuance from order default/example-app-w59ng-1045993216: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal cert-manager.io 4m42s cert-manager Certificate request has been approved by cert-manager.io
Normal OrderCreated 4m42s cert-manager Created Order resource default/example-app-w59ng-1045993216
Order:
root#node1:~# kubectl describe order example-app-w59ng-1045993216
Name: example-app-w59ng-1045993216
Namespace: default
Labels: <none>
Annotations: cert-manager.io/certificate-name: example-app
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: example-app-qddzv
API Version: acme.cert-manager.io/v1
Kind: Order
Metadata:
Creation Timestamp: 2021-12-22T14:07:00Z
Generation: 1
Managed Fields:
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:finalizeURL:
f:state:
f:url:
Manager: controller
Operation: Update
Time: 2021-12-22T14:07:00Z
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:authorizations:
Manager: controller
Operation: Update
Time: 2021-12-22T14:07:00Z
Owner References:
API Version: cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: CertificateRequest
Name: example-app-w59ng
UID: 2fe3875a-4b4d-4139-bef4-820679f8356e
Resource Version: 1520893
UID: 4bb32a01-71ee-43ce-976d-072a718fcc98
Spec:
Dns Names:
###.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-cluster-issuer
Request: [...]
Status:
Authorizations:
Challenges:
Token: rloCCUdQXQx_r9idENjxwXBfPn0DpJ7S8f5Ca5YTZzs
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/58799104090/PllHuQ
Token: rloCCUdQXQx_r9idENjxwXBfPn0DpJ7S8f5Ca5YTZzs
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/58799104090/AYdl0A
Token: rloCCUdQXQx_r9idENjxwXBfPn0DpJ7S8f5Ca5YTZzs
Type: tls-alpn-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/58799104090/DRK6xg
Identifier: cranberry-soft.de
Initial State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/58799104090
Wildcard: false
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/322624000/47528833830
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/322624000/47528833830
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 21m cert-manager Created Challenge resource "example-app-w59ng-1045993216-247428591" for domain "###.com"
Challenge:
root#node1:~# kubectl describe Challenge example-app-w59ng-1045993216-247428591
Name: example-app-w59ng-1045993216-247428591
Namespace: default
Labels: <none>
Annotations: <none>
API Version: acme.cert-manager.io/v1
Kind: Challenge
Metadata:
Creation Timestamp: 2021-12-22T14:07:01Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 1
Managed Fields:
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:presented:
Manager: controller
Operation: Update
Time: 2021-12-22T14:07:02Z
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:reason:
Manager: controller
Operation: Update
Time: 2021-12-22T14:07:02Z
Owner References:
API Version: acme.cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Order
Name: example-app-w59ng-1045993216
UID: 4bb32a01-71ee-43ce-976d-072a718fcc98
Resource Version: 1520918
UID: 4cde9b13-e7af-4fbb-ac60-b25e977108a3
Spec:
Authorization URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/58799104090
Dns Name: ###.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-cluster-issuer
Key: [...]
Solver:
http01:
Ingress:
Class: nginx
Token: [...]
Type: HTTP-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/58799104090/PllHuQ
Wildcard: false
Status:
Presented: true
Processing: true
Reason: Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://###.com/.well-known/acme-challenge/rloCCUdQXQx_r9idENjxwXBfPn0DpJ7S8f5Ca5YTZzs': Get "http://###.com/.well-known/acme-challenge/rloCCUdQXQx_r9idENjxwXBfPn0DpJ7S8f5Ca5YTZzs": dial tcp xx.xx.xx.xx:80: connect: connection refused
State: pending
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 29m cert-manager Challenge scheduled for processing
Normal Presented 29m cert-manager Presented challenge using HTTP-01 challenge mechanism