I run a kubernetes cluster with cert-manager installed for managing ACME (Let's Encrypt) certificates. I'm using DNS domain validation with Route 53 and it works all fine.
The problem comes when I try to issue a certificate for a cluster internal domain. In this case domain validation does not pass since the validation challenge is presented on external Route53 zone only, while cert-manager is trying to look for domain name via cluster internal DNS.
Any hints on how this can be solved are welcome.
Assuming that you don't control public DNS for your cluster internal domain, you will not be able to receive LetsEncrypt certificates for it.
You may however set up another issuer that will grant you certificates for this domain, e.g. the SelfSigned issuer: https://cert-manager.io/docs/configuration/selfsigned/
Then set the issuerRef of your certificate object to point to your SelfSigned issuer:
(...)
issuerRef:
name: selfsigned-issuer
kind: ClusterIssuer
group: cert-manager.io
Related
I'm trying to deploy an app using Kubernetes and Rancher,but I get the error "The certificate for this site is not valid." How can I change the certificate from Kubernetes Ingress Conroller Fake Certificate to another certificate? Should I do it from Rancher or from command line?
Ingress Conroller Fake Certificate to another certificate?
The basic requirement for ingress TLS is a TLS/SSL certificate. You can obtain these certificates in the following ways.
Self-Signed Certificates: TLS certificate created and signed by our own Certificage Authority. It is great optionfor development environments where you can share the rootCA with the team so that browsers can trust the certificate. Check out create self-signed certificate blog to create your own certificates.
Purchase an SSL Certificate: You need to buy an SSL certificate from a well-known certificate authority trusted by browsers & operating systems for production use cases. Check out the top SSL Providers for more information.
Use Letsencrpt Certificate: Letsencrypt is a non-profit trusted certificate authority that provides free TLS certificates.
For more information follow this document .
To update the certificate follow these steps:
To update ssl certificate for your (ingress) https load balancer.
When you created the cluster please executed these commands:
kubectl create secret tls mysecret --key mykey.key --cert mycert.crt
kubectl apply -f ./ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
ingress.kubernetes.io/rewrite-target: /
ingress.kubernetes.io/ssl-redirect: "false"
spec:
tls:
secretName: mysecret
backend:
serviceName: nginx-nodeport-service
servicePort: 80
For more information follow this document. also refer to troubleshooting the Rancher server kubernetes cluster.
We have a dozen of services exposed using a ingress-nginx controller in GKE.
In order to route the traffic correctly on the same domain name, we need to use a rewrite-target rule.
The services worked well without any maintenance since their launch in 2019, that is until recently; when cert-manager suddenly stopped renewing the Let's Encrypt certificates, we "resolved" this by temporarily removing the "tls" section from the ingress definition, forcing our clients to use the http version.
After that we removed all traces of cert-manager attempting to set it up from scratch.
Now, the cert-manager is creating the certificate signing request, spawns an acme http solver pod and adds it to the ingress, however upon accessing its url I can see that it returns an empty response, and not the expected token.
This has to do with the rewrite-target annotation that messes up the routing of the acme challenge.
What puzzles me the most, is that this used to work before. (It was set up by a former employee)
Disabling rewrite-target is unfortunately not an option, because it will stop the routing from working correctly.
Using dns01 won't work because our ISP does not support programmatic changes of the DNS records.
Is there a way to make this work without disabling rewrite-target?
P.S.
Here's a number of similar cases reported on Github:
https://github.com/cert-manager/cert-manager/issues/2826
https://github.com/cert-manager/cert-manager/issues/286
https://github.com/cert-manager/cert-manager/issues/487
None of them help.
Here's the definition of my ClusterIssuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: mail#domain.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
Please share the cluster issuer or issue you are using.
ingressClass
If the ingressClass field is specified, cert-manager will create
new Ingress resources in order to route traffic to the
acmesolver pods, which are responsible for responding to ACME
challenge validation requests.
Ref : https://cert-manager.io/v0.12-docs/configuration/acme/http01/#ingressclass
Mostly we don't see the HTTP solver challenge it comes and get removed if DNS or HTTP working fine.
Also, make sure your ingress doesn't have SSL-redirect annotation that could be also once reason behind certs not getting generated.
Did you try checking the other object of cert-manager like order and certificate status request ? kubectl describe challenge are you getting 404 there ?
If you are trying continuously there could be chance you hit rate limit of let's encrypt to request generating certificates.
Troubleshooting : https://cert-manager.io/docs/faq/troubleshooting/#troubleshooting-a-failed-certificate-request
When you configure an Issuer with http01, the default serviceType is NodePort. This means, it won't even go through the ingress controller. From the docs:
By default, type NodePort will be used when you don't set HTTP01 or when you set serviceType to an empty string. Normally there's no need to change this.
I'm not sure how the rest of your setup looks like, but http01 cause the acme server to make HTTP requests (not https). You need to make sure your nginx has listener for http (80). It does follow redirects, so you can listen on http and redirect all traffic to https, this is legit and working.
The cert-manager creates an ingress resource for validation. It directs traffic to the temporary pod. This ingress has it's own set of rules, and you can control it using this setting. You can try and disable or modify the rewrite-targets on this resource.
Another thing I would try is to access this URL from inside the cluster (bypassing the ingress nginx). If it works directly, then it's an ingress / networking problem, otherwise it's something else.
Please share the relevant nginx and cert-manager logs, it might be useful for debugging or understanding where your problem exist.
The setup in the organization is, the domain is within organization, so is its dns server. The kubernetes cluster is setup within datacenter vmware infrastructure. And this cluster (k8s-dc.<org.domain>) must be internal and so are the applications deployed in the cluster.
Ingress controller is traefik is in use, metallb is provisioned with 10 IPs from same CIDR block of k8s cluster. And there is a wildcard record for LB IP which is pointed to traefik service. Things are working pretty fine so far and plain http applications are accessible through traefik ingress controller.
The issue is all started with enabling tls certs for applications.
The organization cant open either ports 80/443 to internet for LetsEncrypt http01/tls01 challenge. So the idea is to go with dns01 challenge. The organization don't have acme supported external dns provider, so is it advisable to use azureDNS for subdomain delagation?
The sub domain in fact has been delegated to azure and its resolvable from internet. ClusterIssuer has been in place with all necessary azuredns config.
Status:
Acme:
Last Registered Email: xyz#yzx.se
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/452015810
Conditions:
Last Transition Time: 2022-03-15T15:26:23Z
Message: The ACME account was registered with the ACME server
Observed Generation: 1
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events:
certiificaterequest is stuck in pending:
Status:
Conditions:
Last Transition Time: 2022-03-15T15:35:14Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2022-03-15T15:35:14Z
Message: Waiting on certificate issuance from order rbac-test/cert-to-use-8gmj7-3966242034: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
My question here is, the LB IP is internal to organization and the wildcard entry is made within organization. When the nslookup -type=NS k8s-dc.<org.domain> is made from within organization, the nameservers are resolved to internal nameservers but from outside internet, they are resolved to azureDNS. I think I'm missing something in the whole setup, how can I use azuredns for dns01 challenge for the internal domain of any organization?
My goal now is to create a TLS certificate for MinIO in my k8s cluster.
Link to MinIO requirements for TLS connection - up to date.
MinIO running through port-forward to get into the service in the cluster.
There is a cert-manager chart installed via terraform in the cluster which I want to use it for.
I would be happy to get all info on how to actually create, check the certificate, assign it and understand the core concepts of TLS secure connection. many of the guides I have read/watch so far got me a bit confused.
Our k8s is working as Helm charts overall so please be aware not to get into local commands.
Those certificates are supposed to be the simplest ones to create and assign. It will be self-signed which means the CA will be part of the cluster itself and not Third Party CA.
MinIO service expects for public.crt and private.key insdie this path:
/etc/minio/certs/
or this path:
${HOME}/.minio/certs
values.yaml snippet of TLS configuration:
## TLS Settings for MinIO
tls:
enabled: true
## Create a secret with private.key and public.crt files and pass
that here. Ref:
https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-
create-kubernetes-secret
certSecret: "tls-minio"
publicCrt: public.crt
privateKey: private.key
## Trusted Certificates Settings for MinIO. Ref:
https://docs.minio.io/docs/how-to-secure-access-to-minio-server-
with-tls#install-certificates-from-third-party-cas
## Bundle multiple trusted certificates into one secret and pass that here. Ref:
https://github.com/minio/minio/tree/master/docs/tls/kubernetes#2-
create-kubernetes-secret
## When using self-signed certificates, remember to include MinIO's own certificate in the bundle with key public.crt.
## If certSecret is left empty and tls is enabled, this chart installs the public certificate from .Values.tls.certSecret.
trustedCertsSecret: ""
Ask me for any more info about this.
Thanks!
Is there a common (or any) workflow to issue and renew LE certificates for apps configured in an Istio VirtualService & Gateway? The Istio docs only cover an Ingress use case, and I don't think it covers handling renewals.
My real world use case is about making this work with a wildcard cert and custom applications, but for the sake of simplicity, I want to figure this out using the Prometheus service installed with the Istio demo. The VirtualService and Gateway are necessary for my real world use case.
Here is how I am currently serving Prometheus over https with a self-signed cert. I am running Istio version 1.5.2 on GKE K8s version 1.15.11. Cert Manager is installed as well.
So how would I adapt this to use Cert Manager for issuing and renewing an LE cert for prom.example.com?
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: prometheus-gateway
#namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: http-prom
protocol: HTTPS
hosts:
- "prom.example.com"
tls:
mode: SIMPLE # enables HTTPS on this port
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
privateKey: /etc/istio/ingressgateway-certs/tls.key
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: prometheus-vs
spec:
hosts:
- "prom.example.com"
gateways:
- prometheus-gateway
http:
- match:
- port: 443
route:
- destination:
host: prometheus
port:
number: 9090
TL;DR
Configure cert-manager with DNS domain verification to issue certificate, renewal is handled automatically.
Few notes on the example in Istio docs that hopefully will clarify the workflow:
cert-manager knows nothing about Istio, it is key role is to issue and renew certificate then save them to a secret object in kubernetes.
LE ACME verification is typically done with DNS e.g. AWS Route53
Issued Certificate secret will be in a specific k8s namespace and not visible outside that.
Istio knows nothing about cert-manager, all what it needs is the issued certificate secrets which is configured in the gateway with SDS. This means two things:
The name of the SDS secret must match the one cert-manager produces (this is the only link between them)
The secret must be in the same namespace where the Istio gateway will be.
Finally, your VirtualServices just need a gateway that is configured properly as above. The good news is, VirtualService can link to gateway in any namespace if you used the full qualified name.
So you can have your gateway(s) in the same namespace where you issue the Certificate object to avoid copying secrets around, then your VirtualServices can be in any namespace just use the full gateway name.
There is an example for this in istio documentation:
This example demonstrates the use of Istio as a secure Kubernetes Ingress controller with TLS certificates issued by Let’s Encrypt.
You will start with a clean Istio installation, create an example service, expose it using the Kubernetes Ingress resource and get it secured by instructing cert-manager (bundled with Istio) to manage issuance and renewal of TLS certificates that will be further delivered to the Istio ingress gateway and hot-swapped as necessary via the means of Secrets Discovery Service (SDS).
Hope it helps.