Unable to use cert-manager and nginx ingress controller with SSL termination - kubernetes

I am trying out nginx-ingress on GKE with SSL termination for use cases. I've traveled to millions of blogs on this process which uses cert-manager with nginx ingress controller but none of them worked in my case.
This certainly means I am doing something wrong. But I am not sure what. Here's what I did:
Create sample app exposed on ClusterIP
Deploy nginx-ingress
Create issuer
Create nginx ingress with issuer.
Result:
After describing the nginx ingress, the events areas shows none. That means everything is completely blank. Not a single thing happened for requesting certs, http validation, etc.
nginx ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
kubernetes.io/tls-acme: 'true'
spec:
rules:
-
host: wptls.ml
http: {paths: [{path: /, backend: {serviceName: web, servicePort: 80}}]}
tls:
-
secretName: tls-staging-cert
hosts: [wptls.ml]
clusterissuer.yml:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: 'https://acme-staging-v02.api.letsencrypt.org/directory'
email: xyz#gmail.com
privateKeySecretRef:
name: letsencrypt-sec-staging
http01: {}
I am not sure if there's anything else which needs to be done.

Try Ingress extra annotation likes
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/secure-backends: "true"
https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

Related

ALB Ingress Controller, Terminating TLS via AWS ACM. How to go End-To-End?

We currently use the AWS ALB Ingress Controller to front out ingress, and terminate SSL using a certificate from AWS ACM. This work fine.
Is there a way, to also encrypt the traffic from the Load balancer to the cluster?
Here is what I attempted
Install/Configure Cert Manager
Add a TLS Secret to the Ingress
Change the ingress annotation to set the backend protocol to HTTPS alb.ingress.kubernetes.io/backend-protocol: HTTPS
This... results in a 502 gateway error
here is my current, working ingress, with only the relevant parts still shown.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/certificate-arn: real-cert-arn
alb.ingress.kubernetes.io/group.name: public.monitor
alb.ingress.kubernetes.io/group.order: "40"
alb.ingress.kubernetes.io/healthcheck-path: /
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/load-balancer-name: monitoring-public-qa
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-FS-1-2-Res-2020-10
cert-manager.io/cluster-issuer: cert-manager-r53-qa
kubernetes.io/ingress.class: alb
spec:
rules:
- host: goldilocks.qa.realdomain.com
http:
paths:
- backend:
service:
name: goldilocks-dashboard
port:
name: http
path: /*
pathType: ImplementationSpecific
tls:
- hosts:
- goldilocks.qa.realdomain.com
secretName: goldilocks-qa-cert
status:
loadBalancer:
ingress:
- hostname: real-lb-address.us-gov-west-1.elb.amazonaws.com
my (staging) cert exists, and appears fine.
[ec2-user#ip-10-17-2-102 ~]$ kubectl get cert -n goldilocks goldilocks-qa-cert -o yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: goldilocks-qa-cert
namespace: goldilocks
ownerReferences:
- apiVersion: networking.k8s.io/v1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: goldilocks-dashboard
uid: 2494e5be-e624-471b-afd3-c8d56f5dc853
resourceVersion: "81003934"
uid: d494a1ec-7509-474e-b154-d5db8ceb86c0
spec:
dnsNames:
- goldilocks.qa.realdomain.com
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: cert-manager-r53-qa
secretName: goldilocks-qa-cert
usages:
- digital signature
- key encipherment
status:
conditions:
- lastTransitionTime: "2023-02-08T16:06:56Z"
message: Certificate is up to date and has not expired
The fact I can not figure out what to google to find this answer, leads me to think that I'm attempting to do something weird? I understand I could just terminate the TLS on the pod, but I didn't want to rely on the lets encrypt to provide me good/valid certs, I just want the traffic encrypted.

can't get a trusted certificate from Let's encrypt using cert-manager

I am setting up my first K8s cluster in Linode LKE and I have problem getting a trusted certificate from Lets Encrypt. firefox shows unsecure connection with certificate name "Kubernetes Ingress Controller Fake Certificate".
I can't figure out what is missing and how can I troubleshoot this.
Here is my ClusterIssuer and Ingress definitions. I tried the staging and the production acme urls but I couldn't get a trusted certificate.
`
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
email: example#gmail.com
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource that will be used to store the account's private key.
name: letsencrypt-staging
# Add a single challenge solver, HTTP01 using nginx
solvers:
- http01:
ingress:
class: nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tb-https-loadbalancer
namespace: thingsboard
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
ingressClassName: nginx
tls:
- hosts:
- testing.com
secretName: letsencrypt-staging
rules:
- host: testing.com
http:
paths:
...

cert-manager: no configured challenge solvers can be used for this challenge

I followed this instruction to set up a cert manager on my EKS cluster https://cert-manager.io/docs/tutorials/acme/ingress/.
here is my ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: test
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/issuer: "letsencrypt-staging"
spec:
tls:
- hosts:
- '*.test.com'
secretName: test-tls
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: test-service
port:
number: 80
Here is the issuer. I just copied the config from the instruction
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: info#test.com
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: nginx
After deployment, I found the certificate ready state is false
kubectl get certificate
NAME READY SECRET AGE
test-tls False test-tls 2m45s
Then I followed this to troubleshoot https://cert-manager.io/docs/faq/troubleshooting/
I ran kubectl describe certificaterequest <request name>, found error Waiting on certificate issuance from order test-tls-xxx: "pending"
then ran kubectl describe order test-tls-xxx, found error
Warning Solver 20m cert-manager Failed to determine a valid solver configuration for the set of domains on the Order: no configured challenge solvers can be used for this challenge.
Any idea why it couldn't determine a valid solver? how do I test if solver is working?
It's not working due you are using the staging URL in cluster issuer to verify the image.
Please try with the Production URL.
here a simple and proper example of Clusterissuer and ingress YAML (do note you were trying with staging API https://acme-staging-v02.api.letsencrypt.org/directory if possible use the production server address so it works properly with all browsers)
Example:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: cluster-issuer-name
namespace: development
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: harsh#example.com
privateKeySecretRef:
name: secret-name
solvers:
- http01:
ingress:
class: nginx-class-name
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx-class-name
cert-manager.io/cluster-issuer: cluster-issuer-name
nginx.ingress.kubernetes.io/rewrite-target: /
name: example-ingress
spec:
rules:
- host: sub.example.com
http:
paths:
- path: /api
backend:
serviceName: service-name
servicePort: 80
tls:
- hosts:
- sub.example.com
secretName: secret-name
Note : When you are trying again please try deleting the old objects like ingress, Clusterissuer first.
Issuer vs ClusterIssuer
An Issuer is a namespaced resource, and it is not possible to issue
certificates from an Issuer in a different namespace. This means you
will need to create an Issuer in each namespace you wish to obtain
Certificates in.
If you want to create a single Issuer that can be consumed in multiple
namespaces, you should consider creating a ClusterIssuer resource.
This is almost identical to the Issuer resource, however is
non-namespaced so it can be used to issue Certificates across all
namespaces.
Ref : https://cert-manager.io/docs/concepts/issuer/
Wildcard cert
You can use as per requirement, if you are using issuer you can update the ingress annotation line like
cert-manager.io/issuer: issuer-name
If you are trying to get the wildcard * certificate you won't be able to get it using HTTP auth method
solvers:
- http01:
ingress:
class: nginx-class-name
instead of this you have to use the DNS-auth method for wildcard cert.
solvers:
- dns01:
cloudDNS:
project: my-project
serviceAccountSecretRef:
name: prod-clouddns-svc-acct-secret
key: service-account.json
Read more at : https://cert-manager.io/docs/configuration/acme/dns01/
Ref article to get the wildcard cert : https://medium.com/#harsh.manvar111/wild-card-certificate-using-cert-manager-in-kubernetes-3406b042d5a2

Waiting for http-01 challenge propagation: failed to perform self timed out

I setup a kubernetes cluster with currently two nodes and metallb as a loadbalancer.
Currently I would like to use an Ingress and secure it via ssl. For this I decided to use nginx ingress with cert-manager and put it on their site after the tutorial.
But now I get the error " Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://example.....zone/.well-known/acme-challenge/A5lFUj69fDccpXlvlyVw9-ekATEjt_-DKiJUzJSafxs': Get "http://example.....zone/.well-known/acme-challenge/A5lFUj69fDccpXlvlyVw9-ekATEjt_-DKiJUzJSafxs": dial tcp 94.130.150.125:80: connect: connection timed out
"
My current ClusterIssuer looks like this:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: letsencrypt#mymail.de
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
And I am trying to automatically provide a certificate for
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: web-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
cert-manager.io/acme-challenge-type: http01
spec:
tls:
- hosts:
- example.....zone
secretName: example-...-zone-tls
rules:
- host: example.....zone
http:
paths:
- path: /
backend:
serviceName: nginx-service
servicePort: 80
Manually I can reach any address perfectly.
This one worked for me.
Change LoadBalancer in ingress-nginx service.
Add/Change externalTrafficPolicy: Cluster.
Reason being, pod with the certificate-issuer wound up on a different node than the load balancer did, so it couldn’t talk to itself through the ingress.
Below is complete block taken from https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
#CHANGE/ADD THIS
externalTrafficPolicy: Cluster
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
---
The error message you are getting can mean a wide variety of issues. However, there are some things you can check in order to fix it:
Delete the Ingress, the certificates and the cert-manager. After that add them all back to make sure it installs clean. Sometimes stale certs or bad/multi Ingress pathing might be the issue.
Make sure your traffic allows HTTP or has HTTPS with a trusted cert.
Check if hairpin mode of your loadbalancer and make sure it is working.
Add: nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation to the Ingress rule. Wait a moment and see if valid cert will be created.
GKE only: add this Ingress annotation: certmanager.k8s.io/acme-http01-edit-in-place: "true".
Please let me know if that helped.

What is correct way to configure https to my services (kubernetes, nginx-ingress, letsencrypt, cert-manager)?

I just will describe how it configured on my side. I've installed cert-manger on my Kubernetes by using this tutorial :
https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html
I've checked is it installed and it is :
Also I have ingress-resource with the next config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/acme-http01-edit-in-place: "true"
certmanager.k8s.io/cluster-issuer: letsencrypt-issuer
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
name: boonotes-ingress-resource
namespace: default
spec:
rules:
- host: www.bnsfun.com
http:
paths:
- backend:
serviceName: booknotes-front-end-service
servicePort: 80
path: /?(.*)
- host: www.bnsfun.com
http:
paths:
- backend:
serviceName: booknotes-back-end-service
servicePort: 3000
path: /api/?(.*)
tls:
- hosts:
- www.bnsfun.com
secretName: letsencrypt-certs
status:
loadBalancer:
ingress:
- ip: some ip
Also, I've configured the certificate :
kubectl describe certificate booknotes-certificate
Name: booknotes-certificate
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"booknotes-certificate","namespace":"...
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-11-17T04:51:57Z
Generation: 2
Resource Version: 7257970
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/booknotes-certificate
UID: fbe1d9c0-08f5-11ea-82b3-42010a80017a
Spec:
Acme:
Config:
Domains:
www.bnsfun.com
http01:
Ingress: boonotes-ingress-resource
Common Name: www.bnsfun.com
Dns Names:
www.bnsfun.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-issuer
Secret Name: letsencrypt-certs
Events: <none>
I've also created a secret:
Here is my sevice & ingress section:
I've used this tutorial to configure it :
https://medium.com/#betandr/kubernetes-ingress-with-tls-on-gke-744efd37e49e
and official documentation of cert-manager to install cert managed. What do I wrong? How can I check why this doesn't work? I've tried a lot of stuff, but all doesn't work for me. For sure I do something wrong. But what? I've understood that I need cert-manager for updating my lets-encrypt certificate, also I need to create secret to store it, then I need configure my ingress in tls and annotaions. Pls could you help me to find out more what should happen there and what are the main steps to complete it? If you need more info , pls let me know
Here is my issuer:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-issuer
spec:
acme:
email: email
http01: {}
privateKeySecretRef:
name: letsencrypt
server: "https://acme-v02.api.letsencrypt.org/directory"
let's take an another path, Letsencrypt official docs say that they won't be supporting any longer for below 0.8 versions, so I recommend you to install cert-manager provided by Jetstack, that you can find here, to install the helm chart for it.
The follow this stackoverflow post, for configurations, note that if the api version mentioned in that post doesn't support in case of cluster issuer, then rather use
apiVersion: cert-manager.io/v1alpha2
Note that , the tls secret name mentioned in the certificate will be auto-generated by cert-manager, and it automatically starts an acme-challenge to validate the domain, once you patch that secret name to the TLS in your ingress rule.
It shall solve the issue and the certificate's status will change to ready after the domain verification