How to make ingress use my TLS Certificate in Microk8s - kubernetes

I have the following Ingress configuration:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: http-ingress
spec:
rules:
- host: example-adress.com
http:
paths:
- path: /apple
pathType: Prefix
backend:
service:
name: apple-service
port:
number: 80
- path: /banana
pathType: Prefix
backend:
service:
name: banana-service
port:
number: 80
tls:
- hosts:
- example-adress.com
secretName: testsecret-tls
And i also created the Secret:
apiVersion: v1
kind: Secret
metadata:
name: testsecret-tls
namespace: default
data:
tls.crt: path to .crt
tls.key: Zpath to .key
type: kubernetes.io/tls
But when i connect to one of my services and check the certificate it says that it uses a cert created by Kubernetes Ingress Controller Fake certificate.
When i run microk8s kubectl describe ingress i get the following output:
Name: http-ingress
Namespace: default
Address: 127.0.0.1
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
testsecret-tls terminates example-adress.com
Rules:
Host Path Backends
---- ---- --------
example-adress.com
/apple apple-service:80 (10.1.55.17:5678)
/banana banana-service:80 (10.1.55.10:5678)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 28m nginx-ingress-controller Ingress default/http-ingress
Normal UPDATE 20m (x2 over 28m) nginx-ingress-controller Ingress default/http-ingress
What do i need to change to make my Ingress use my Cert instead of generating a new one everytime?

Posting this out of comment as it works.
Based on your tls secret yaml, you tried to add certificate and private key using paths, which is not supported currently (reference)
Fragment from reference:
When using this type of Secret, the tls.key and the tls.crt key must be provided in the data (or stringData) field of the Secret configuration, although the API server doesn't actually validate the values for each key.
Therefore there are two suggestions how to move forward:
Add base64 encrypted values for key and certificate to tls secret
Allow kubernetes do it for you with the following command:
kubectl create secret tls testsecret-tls --cert=tls.cert --key=tls.key

Related

the ingress resource stop to sync? what event indicates there?

I run the ingress controller and put one annotation for round_robin algorithm to run. but seems there is no event run there. is it ok if no event in my ingress description? what event indicates there?
# kubectl describe ingress -n ingress
Name: nginx-ingress
Namespace: ingress
Address: 192.168.10.10,192.168.10.45
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
website.com
/ website1:80 (10.42.0.139:80,10.42.1.223:80,10.42.2.98:80 + 1 more...)
/website2 website2:80 (10.42.0.140:80,10.42.1.232:80,10.42.2.74:80 + 1 more...)
/website3 website3:80 (10.42.0.141:80,10.42.1.215:80,10.42.2.58:80 + 1 more...)
Annotations: nginx.ingress.kubernetes.io/load-balance: round_robin
Events: <none>
I deploy my ingress with this bro, and I have my ingress class. update 1.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
namespace: ingress
annotations:
nginx.ingress.kubernetes.io/load-balance: ewma
spec:
ingressClassName: nginx
rules:
- host: service.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service1
port:
number: 80
- path: /service2
pathType: Prefix
backend:
service:
name: service2
port:
number: 80
- path: /service3
pathType: Prefix
backend:
service:
name: service3
port:
number: 80
with another annotation i get the event in my ingress
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 25m (x4 over 113m) nginx-ingress-controller Scheduled for sync
Normal Sync 22m (x39 over 7d2h) nginx-ingress-controller Scheduled for sync
Normal Sync 22m (x41 over 7d2h) nginx-ingress-controller Scheduled for sync
Normal Sync 22m (x22 over 5d10h) nginx-ingress-controller Scheduled for sync
Normal Sync 8m42s (x2 over 9m21s) nginx-ingress-controller Scheduled for sync
# kubectl describe ingress -n ingress
Name: nginx-ingress
Namespace: ingress
Address: 192.168.10.10,192.168.10.45
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
website1.com
/ nginx-deployment:80 (10.42.0.222:80,10.42.1.32:80,10.42.2.155:80 + 1 more...)
/website2 nginx-video:80 (10.42.0.220:80,10.42.1.30:80,10.42.2.153:80 + 1 more...)
/website3 nginx-document:80 (10.42.0.221:80,10.42.1.31:80,10.42.2.154:80 + 1 more...)
Annotations: nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/affinity-mode: persistent
nginx.ingress.kubernetes.io/session-cookie-expires: 172800
nginx.ingress.kubernetes.io/session-cookie-max-age: 172800
nginx.ingress.kubernetes.io/session-cookie-name: route
nginx.ingress.kubernetes.io/upstream-hash-by: ewma
Events: <none>
This means your cluster has no ingress controller installed. When there's a ingress controller (such as ingress-nginx) installed in your cluster, a series of events will triggered to process your ingress request. These events will show in your describe command.
If you do have ingress controller but it is not registered as the default ingress class for your cluster, you can add the annotation kubernetes.io/ingress.class: <name of your IngressClass, example "nginx"> to your ingress spec or:
apiVersion: networking.k8s.io/v1
kind: Ingress
...
spec:
ingressClassName: <name of your IngressClass, example "nginx">
...
in fact, the sync is not always displayed. when we do changes at the beginning the ingress controller does sync. However, after some time the sync will stop automatically.

Certificate always in 'False' state using LetsEncrypt with cluster issuer in k8s

I am unable to issue a working certificate for my ingress host in k8s. I use a ClusterIssuer to issue certificates and the same ClusterIssuer has issued certificates in the past for my ingress hosts under my domain name *xyz.com. But all of a sudden neither i can issue new Certificate with state 'True' for my host names nor a proper certificate secret (kubernetes.io/tls) gets created (but instead an Opaque secret gets created).
**strong text**
**kubectl describe certificate ingress-cert -n abc**
Name: ingress-cert
Namespace: abc
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1beta1
Kind: Certificate
Metadata:
Creation Timestamp: 2021-09-08T07:48:32Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: test-ingress
UID: c03ffec0-df4f-4dbb-8efe-4f3550b9dcc1
Resource Version: 146643826
Self Link: /apis/cert-manager.io/v1beta1/namespaces/abc/certificates/ingress-cert
UID: 90905ab7-22d2-458c-b956-7100c4c77a8d
Spec:
Dns Names:
abc.xyz.com
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt
Secret Name: ingress-cert
Status:
Conditions:
Last Transition Time: 2021-09-08T07:48:33Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2021-09-08T07:48:33Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: ingress-cert-gdq7g
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 11m cert-manager Issuing certificate as Secret does not exist
Normal Generated 11m cert-manager Stored new private key in temporary Secret resource "ingress-cert-gdq7g"
Normal Requested 11m cert-manager Created new CertificateRequest resource "ingress-cert-dp6sp"
I checked the certificate request and it contains no events. Also i can see no challenges. I have added the logs below. Any help would be appreciated
kubectl describe certificaterequest ingress-cert-dp6sp -n abc
Namespace: abc
Labels: <none>
Annotations: cert-manager.io/certificate-name: ingress-cert
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: ingress-cert-gdq7g
API Version: cert-manager.io/v1beta1
Kind: CertificateRequest
Metadata:
Creation Timestamp: 2021-09-08T07:48:33Z
Generate Name: ingress-cert-
Generation: 1
Owner References:
API Version: cert-manager.io/v1alpha2
Block Owner Deletion: true
Controller: true
Kind: Certificate
Name: ingress-cert
UID: 90905ab7-22d2-458c-b956-7100c4c77a8d
Resource Version: 146643832
Self Link: /apis/cert-manager.io/v1beta1/namespaces/abc/certificaterequests/ingress-cert-dp6sp
UID: fef72617-fc1d-4384-9f4b-a7e4502582d8
Spec:
Issuer Ref:
Group: cert-manager.io
Kind: ClusterIssuer
Name: letsencrypt
Request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2Z6Q0NBV2NDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQUxMNgphTGhZNjhuNnhmMUprYlF5ek9OV1J4dGtLOXJrbjh5WUtMd2l4ZEFMVUl0TERra0t6Uksyb3loZzRMMThSQmQvCkNJaGJ5RXBYNnlRditKclRTOC84T1A0MWdwTUxBLzROdVhXWWtyeWhtZFdNaFlqa21OOFpiTUk1SlZZcVV2cVkKRWQ1b2cydmVmSjU1QlJPRExsd0o3YjBZa3hXckUwMGJxQ1ExWER6ZzFhM08yQ2JWd1NQT29WV2x6Uy9CdzRYVgpMeVdMS3E4QU52b2dZMUxXRU8xcG9YelRObm9LK2U2YVZueDJvQ1ZLdGxPaG1iYXRHYXNSaTJKL1FKK0dOWHovCnFzNXVBSlhzYVErUzlxOHIvbmVMOXNPYnN2OWd1QmxCK09yQVg2eHhkNHZUdUIwVENFU00zWis2c2MwMFNYRXAKNk01RlY3dkFFeDQyTWpuejVoa0NBd0VBQWFBNk1EZ0dDU3FHU0liM0RRRUpEakVyTUNrd0p3WURWUjBSQkNBdwpIb0ljY25kemMyZHdMbU5zYjNWa1oyRjBaUzV0YVdOeWIyWnBiaTVrWlRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DCkFRRUFTQ0cwTXVHMjZRbVFlTlBFdmphNHZqUUZOVFVINWVuMkxDcXloY2ZuWmxocWpMbnJqZURuL2JTV1hwdVIKTnhXTnkxS0EwSzhtMG0rekNPbWluZlJRS1k2eHkvZU1WYkw4dTgrTGxscDEvRHl3UGxvREE2TkpVOTFPaDM3TgpDQ0E4NWphLy9FYVVvK0p5aHBzaTZuS1d4UXRpYXdmYXhuNUN4SENPWGF5Qzg0Q0IzdGZ2WWp6YUF3Ykx4akxYCmxvd09LUHNxSE51ZktFM0NtcjZmWGgramd5VWhxamYwOUJHeGxCWEFsSVNBNkN5dzZ2UmpWamFBOW82TmhaTXUKbmdheWZON00zUzBBYnAzVFFCZW8xYzc3QlFGaGZlSUE5Sk51SWtFd3EvNXppYVY1RDErNUxSSnR5ZkVpdnJLTwpmVjQ5WkpCL1BGOTdiejhJNnYvVW9CSkc2Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
Status:
Conditions:
Last Transition Time: 2021-09-08T07:48:33Z
Message: Waiting on certificate issuance from order abc/ingress-cert-dp6sp-3843501305: ""
Reason: Pending
Status: False
Type: Ready
Events: <none>
Here is the ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 20m
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt"
spec:
rules:
- host: abc.xyz.com
http:
paths:
- path: /static
backend:
serviceName: app-service
servicePort: 80
- path: /
backend:
serviceName: app-service
servicePort: 8000
tls:
- hosts:
- abc.xyz.com
secretName: ingress-cert
Here is the clusterissuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: example#user.de
privateKeySecretRef:
name: letsencrypt-key
solvers:
- http01:
ingress:
class: nginx
Works only with Nginx Ingress Controller
I was using ClusterIssuer but I changed it to Issuer and it works.
-- Install cert-manager (Installed version 1.6.1) and be sure that the three pods are running
-- Create an Issuer by appling this yml be sure that the issuer is running.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-nginx
namespace: default
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: example#example.com
privateKeySecretRef:
name: letsencrypt-nginx-private-key
solvers:
- http01:
ingress:
class: nginx
-- Add this to your ingress annotations
cert-manager.io/issuer: letsencrypt-nginx
-- Add the secretName to your ingress spec.tls.hosts
spec:
tls:
- hosts:
- yourdomain.com
secretName: letsencrypt-nginx
Notice that the Nginx Ingress Controller is able to generate the Certificate CRD automatically via a special annotation: cert-manager.io/issuer. This saves work and time, because you don't have to create and maintain a separate manifest for certificates as well (only the Issuer manifest is required). For other ingresses you may need to provide the Certificate CRD as well.
Ideally your ingress pointing to the secret which is storing the secret or SSL/TLS key cert.
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: test-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: 20m
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: "letsencrypt"
spec:
rules:
- host: abc.xyz.com
http:
paths:
- path: /static
backend:
serviceName: app-service
servicePort: 80
- path: /
backend:
serviceName: app-service
servicePort: 8000
tls:
- hosts:
- abc.xyz.com
secretName: letsencrypt-key
Your cluster issue storing the key
privateKeySecretRef:
name: letsencrypt-key
You have to use this secret and attach this to ingress.
If secret already storing cert with a domain
test.example.com and you are trying to get a new cert with hello.example.com
in this case using cluster issuer will overwrite the secret and might loss old cert stored inside secret.
You can create the multiple clusterissuer,
One storing and connect to single ingress, first.example.com
Second cluster issuer with different key name
privateKeySecretRef:
name: letsencrypt-key
and new key or secret will get attached to the ingress.

ClusterIssuer with Cloudflare based DNS solver

I'm trying to install the cert-manager ClusterIssuer on a AKS, and because the cluster is behind Azure Application Gateway I've gone down the route of using a DNS solver rather the HTTP. However, the challenge fails with an error calling the Cloudflare API. I've redacted emails and domains through the code snippets, the output of kubectl describe challenge rabt-cert-tls-g4mcl-1991965707-2468967546 is:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Started 72s cert-manager Challenge scheduled for processing
Warning PresentError 3s (x5 over 71s) cert-manager Error presenting challenge: Cloudflare API Error for GET "/zones?name=<domain>"
Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
I have followed the guide at https://blog.darkedges.com/2020/05/04/cert-manager-kubernetes-cloudflare-dns-update/ and the issues at https://github.com/jetstack/cert-manager/issues/3021 and https://github.com/jetstack/cert-manager/issues/2384 but can't see any differences beyond the apiVersion of the issuer. I've checked this against the official documentation and there are no changes from what appears in these guides.
The relationship between ingress and cluster issuer seems fine; if I delete and recreate the ingress a new certificate, order and challenge are created. I've verified the secret is populated and I can print it to console, so it shouldn't be sending a blank string in the header. The token is valid, I can use the example CURL request from CloudFlare to check its validity.
Is there somewhere I can see logs and find out exactly what is being sent?
ClusterIssuer
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token-secret
namespace: cert-manager
type: Opaque
stringData:
api-token: ${CLOUDFLARE_API_TOKEN}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: rabt-letsencrypt
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: <email>
# ACME server URL for Let’s Encrypt’s staging environment.
# The staging environment will not issue trusted certificates but is
# used to ensure that the verification process is working properly
# before moving to production
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: rabt-letsencrypt-key
# Enable the HTTP-01 challenge provider
# you prove ownership of a domain by ensuring that a particular
# file is present at the domain
solvers:
- dns01:
cloudflare:
email: <email>
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-key
Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rabt-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-protocol: https
appgw.ingress.kubernetes.io/ssl-redirect: "true"
cert-manager.io/cluster-issuer: rabt-letsencrypt
cert-manager.io/acme-challenge-type: dns01
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
tls:
- hosts:
- "*.rabt.<domain>"
secretName: rabt-cert-tls
rules:
- host: "mq.rabt.<domain>"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rabt-mq
port:
number: 15672
- host: es.rabt.<domain>
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rabt-db-es-http
port:
number: 9200
- host: "kibana.rabt.<domain>"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: rabt-kb-http
port:
number: 5601
As Harsh Manvar guessed, it was an issue with the secret. I wasn't running the kubectl apply command through envsubst so it was encoding the literal string "${CLOUDFLARE_API_TOKEN}"

connection reset by peer Error with nginx Ingress in Kind cluster

I installed vault helm chart 0.8.0 in dev mode and nginx ingress controller on Kind cluster and applied the following Ingress resource:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: vault-ingress
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
rules:
- host: vault.com
http:
paths:
- path: /v1/*
backend:
serviceName: vault
servicePort: 8200
This is the patch for Kind cluster as noted in https://kind.sigs.k8s.io/docs/user/ingress/:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "ingress-ready=true"
extraPortMappings:
- containerPort: 8200 # Vault port in K8s
hostPort: 80 # Vault port on host
- containerPort: 8200 # Vault port in K8s
hostPort: 443 # Vault port on host
when doing the following curl command I get error:
VAULT_ADDR="http://localhost"
vault login root
Error authenticating: error looking up token: Get "http://localhost/v1/auth/token/lookup-self": read tcp 127.0.0.1:34596->127.0.0.1:80: read: connection reset by peer
Name: vault-ingress
Namespace: vault
Address: localhost
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
vault.com
/v1/* vault:8200 (10.244.1.7:8200)
Annotations: nginx.ingress.kubernetes.io/use-regex: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 2m47s (x15 over 10h) nginx-ingress-controller Scheduled for sync

Access .NET Core app on Kubernetes on both http and https

Being new to Kubernetes, I am trying to make a simple .NET Core 3 MVC app run on Kubernetes and reply on port 443 as well as port 80. I have a working Docker-Compose setup which I am trying to port to Kubernetes.
Running Docker Desktop CE with nginx-ingress on Win 10 Pro.
So far it is working on port 80. (http://mymvc.local on host Win 10 - hosts file redirects mymvc.local to 127.0.0.1)
My MVC app is running behind service mvc on port 5000.
I've made a self-signed certificate for the domain 'mymvc.local', which is working in the Docker-Compose setup.
This is my ingress file
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: mvc-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- mymvc.local
secretName: mvcsecret-tls
rules:
- host: mymvc.local
http:
paths:
- path: /
backend:
serviceName: mvc
servicePort: 5000
This is my secrets file (keys abbreviated):
apiVersion: v1
kind: Secret
metadata:
name: mvcsecret-tls
data:
tls.crt: MIIDdzCCAl+gAwIBAgIUIok60uPHId5kve+/bZAw/ZGftIcwDQYJKoZIhvcNAQELBQAwKTELMAkGBxGjAYBgN...
tls.key: MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQDPGN6yq9yzxvDL8fEUJChqlnaTQW6bQX+H0...
type: kubernetes.io/tls
kubectl describes the ingress as follows:
Name: mvc-ingress
Namespace: default
Address: localhost
Default backend: default-http-backend:80 (<none>)
TLS:
mvcsecret-tls terminates mymvc.local
Rules:
Host Path Backends
---- ---- --------
mymvc.local
/ mvc:5000 (10.1.0.27:5000)
Annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: true
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 11m nginx-ingress-controller Ingress default/mvc-ingress
Normal UPDATE 11m nginx-ingress-controller Ingress default/mvc-ingress
In my Docker-Compose setup, I have an Nginx reverse proxy redirecting 80 and 443 to my MVC service, but I figured that is the role of ingress on Kubernetes?
My service YAML:
apiVersion: v1
kind: Service
metadata:
name: mvc
labels:
app: mymvc
spec:
ports:
- name: "mvc"
port: 5000
targetPort: 5000
selector:
app: mymvc
type: ClusterIP
EDIT:
Adding 'nginx.ingress.kubernetes.io/rewrite-target: /' to ingress annotations males the https forward work, but the certificate presented is the 'Kubernetes Ingress Controller Fake Certificate' - not my self-signed one.
The solution turned out to be the addition of a second kind of certificate.
Instead of using the secrets file above (where I pasted the contents of my certificates files), I issued kubectl to use my certificate files directly:
kubectl create secret tls mvcsecret-tls --key MyCert.key --cert MyCert.crt
kubectl create secret generic tls-rootca --from-file=RootCA.pem