Deploy ingress nginx with a domain in kubernetes - kubernetes

I have deployed airflow in kubernetes as is descrived in this link: https://github.com/apache/airflow/tree/master/chart
To access the airflow UI I can do:
kubectl port-forward svc/airflow2-webserver 8080:8080 --namespace default
But I would want to expose it in a url. I found this guide:
https://godatadriven.com/blog/deploying-apache-airflow-on-azure-kubernetes-service/
In the bottom part: FQDN with Ingress controller, he installs an nginx-ingress-controller and a cert manager.
Then I create a Cluster-issuer:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: my-email#gmail.com
privateKeySecretRef:
name: letsencrypt-staging
solvers:
- http01:
ingress:
class: nginx
podTemplate:
spec:
nodeSelector:
"kubernetes.io/os": linux
I install the ingress-routes.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: airflow-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
tls:
- hosts:
- mydomain.westeurope.cloudapp.azure.com
secretName: tls-secret-test2
rules:
- host: mydomain.westeurope.cloudapp.azure.com
http:
paths:
- path: /
backend:
serviceName: airflow2-webserver
servicePort: 8080
But when I try to get my certificate:
kubectl describe certificate
No resources found in default namespace.
I have deployed everything but I donĀ“t know what is missing but when I go to my domain is still not trusted

It looks like your DNS A record is not pointing to your Ingress LoadBalancer public IP. Thus your cert-manager issuer is not able to validate your domain ownership from outside using http challange.
To verify you can use tool called MXToolBox. In your configuration (https://mxtoolbox.com/SuperTool.aspx?action=a%3amydomain.westeurope.cloudapp.azure.com&run=toolpage) it's pointing to private IP.
Solution
To resolve this issue, you should Add an A record to your DNS zone
az network dns record-set a add-record \
--resource-group myResourceGroup \
--zone-name MY_CUSTOM_DOMAIN \
--record-set-name '*' \
--ipv4-address MY_EXTERNAL_IP

Related

ALB Ingress Controller, Terminating TLS via AWS ACM. How to go End-To-End?

We currently use the AWS ALB Ingress Controller to front out ingress, and terminate SSL using a certificate from AWS ACM. This work fine.
Is there a way, to also encrypt the traffic from the Load balancer to the cluster?
Here is what I attempted
Install/Configure Cert Manager
Add a TLS Secret to the Ingress
Change the ingress annotation to set the backend protocol to HTTPS alb.ingress.kubernetes.io/backend-protocol: HTTPS
This... results in a 502 gateway error
here is my current, working ingress, with only the relevant parts still shown.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/backend-protocol: HTTP
alb.ingress.kubernetes.io/certificate-arn: real-cert-arn
alb.ingress.kubernetes.io/group.name: public.monitor
alb.ingress.kubernetes.io/group.order: "40"
alb.ingress.kubernetes.io/healthcheck-path: /
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
alb.ingress.kubernetes.io/load-balancer-name: monitoring-public-qa
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-FS-1-2-Res-2020-10
cert-manager.io/cluster-issuer: cert-manager-r53-qa
kubernetes.io/ingress.class: alb
spec:
rules:
- host: goldilocks.qa.realdomain.com
http:
paths:
- backend:
service:
name: goldilocks-dashboard
port:
name: http
path: /*
pathType: ImplementationSpecific
tls:
- hosts:
- goldilocks.qa.realdomain.com
secretName: goldilocks-qa-cert
status:
loadBalancer:
ingress:
- hostname: real-lb-address.us-gov-west-1.elb.amazonaws.com
my (staging) cert exists, and appears fine.
[ec2-user#ip-10-17-2-102 ~]$ kubectl get cert -n goldilocks goldilocks-qa-cert -o yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: goldilocks-qa-cert
namespace: goldilocks
ownerReferences:
- apiVersion: networking.k8s.io/v1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: goldilocks-dashboard
uid: 2494e5be-e624-471b-afd3-c8d56f5dc853
resourceVersion: "81003934"
uid: d494a1ec-7509-474e-b154-d5db8ceb86c0
spec:
dnsNames:
- goldilocks.qa.realdomain.com
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: cert-manager-r53-qa
secretName: goldilocks-qa-cert
usages:
- digital signature
- key encipherment
status:
conditions:
- lastTransitionTime: "2023-02-08T16:06:56Z"
message: Certificate is up to date and has not expired
The fact I can not figure out what to google to find this answer, leads me to think that I'm attempting to do something weird? I understand I could just terminate the TLS on the pod, but I didn't want to rely on the lets encrypt to provide me good/valid certs, I just want the traffic encrypted.

How to setup a domain in GKE ingress nginx

I have a cluster in GKE and it is working, everything seems to be working. If I forward the ports I am able to see that the containers are working.
I am not able to setup a domain I own from namecheap.
These are the steps I followed
In Namecheap I setup a custom dns for the domain
ns-cloud-c1.googledomains.com.
ns-cloud-c2.googledomains.com.
ns-cloud-c3.googledomains.com.
ns-cloud-c3.googledomains.com.
I used the letter c because the cluster is in a c zone (I am not sure if this is right)
Because I am trying to setup as secure website I installed nginx ingress controller
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user $(gcloud config get-value account)
and
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/cloud/deploy.yaml
I applied the issuer.yml
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: example#email.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
I applied ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: staging
name: ingress
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- www.stagingmyappsrl.com
- api.stagingmyappsrl.com
secretName: stagingmyappsrl-tls
rules:
- host: wwwstaging.myappsrl.com
http:
paths:
- backend:
serviceName: myappcatalogo-svc
servicePort: 80
- host: apistaging.stagingmyappsrl.com
http:
paths:
- backend:
serviceName: myappnodeapi-svc
servicePort: 80
It seems that everything is created and working if I check in GKE website, but when I try to access I get DNS_PROBE_FINISHED_NXDOMAIN
I am not sure if I am missing an step or if I am setting up something wrong
GKE should have created a cloud load balancer for your ingress service. Depending on your config, the LB can be internal or external. You can get your LB information by looking at the services:
kubectl get svc -n ingress-nginx
Create a CNAME record in your DNS (namecheap) with the LB address and that should do it. Alternatively, if you have an IP address of the LB, create an A record in your DNS.
Cert-manager will create an ingress resource to resolve HTTPS01 challenges. Make sure your ingresses are reachable over the Internet for the HTTPS01 challenges to work. Alternatively, you could explore other solvers.

How to deploy a letsencryp with cert-manager and HAProxy-ingress

Digging into www, didn't found an answer:
I want to know how I can use cert-manager with haproxy-ingress and lets encrypt.
Any documentation / guidelines?
Deploy Certmanager with:
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.yaml
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.4.0/cert-manager.crds.yaml
Deploy a ClusterIssuer (cluster issuers are namespace agnostic)
cat > prod-issuer.yaml <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: email#example.com
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
ingress:
class: haproxy
EOF
Apply the cluster issuer with kubectl apply -f prod-issuer.yaml
Create an Ingress Resource (namespace gnostic, this example is using Nginx backend service)
cat > nginx-ingress.yaml <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-router
namespace: production
annotations:
kubernetes.io/ingress.class: "haproxy"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- "example.com"
secretName: nginx-tls-secret
rules:
- host: example.com
http:
paths:
- pathType: ImplementationSpecific
path: "/"
backend:
service:
name: nginx
port:
number: 80
- host: www.example.com
http:
paths:
- pathType: ImplementationSpecific
path: "/"
backend:
service:
name: nginx
port:
number: 80
EOF
Apply the ingress recourse with kubectl apply -f nginx-ingress.yaml
The important piece of information here, is that the Haproxy controller does NOT need the annotation acme.cert-manager.io/http01-edit-in-place: "true"
that nginx-ingress controller does. It works as expected without any extra annotations.
When you apply the Ingress Resourse to the cluster, the certificate will be issued in 1-2 minutes tops.
Use kubectl describe certificate nginx-tls-secret -n production to check the status of the certificate, and look at the event to get the certificate.
For more debugging info incase something went wrong, refer here https://cert-manager.io/docs/faq/acme/
you can try installing cert manager provided by jetstack, can be found here and then you need to follow the steps mentioned in this stackoverflow post and this will get things sorted for you.
An internal acme-challenge will be done by cert manager and once you patch the secret name, mentioned in the certificate to the TLS of ingress then certificate status will get ready state, Note that the secret will get created automatically, you need not create it

What is correct way to configure https to my services (kubernetes, nginx-ingress, letsencrypt, cert-manager)?

I just will describe how it configured on my side. I've installed cert-manger on my Kubernetes by using this tutorial :
https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html
I've checked is it installed and it is :
Also I have ingress-resource with the next config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/acme-http01-edit-in-place: "true"
certmanager.k8s.io/cluster-issuer: letsencrypt-issuer
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
name: boonotes-ingress-resource
namespace: default
spec:
rules:
- host: www.bnsfun.com
http:
paths:
- backend:
serviceName: booknotes-front-end-service
servicePort: 80
path: /?(.*)
- host: www.bnsfun.com
http:
paths:
- backend:
serviceName: booknotes-back-end-service
servicePort: 3000
path: /api/?(.*)
tls:
- hosts:
- www.bnsfun.com
secretName: letsencrypt-certs
status:
loadBalancer:
ingress:
- ip: some ip
Also, I've configured the certificate :
kubectl describe certificate booknotes-certificate
Name: booknotes-certificate
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Certificate","metadata":{"annotations":{},"name":"booknotes-certificate","namespace":"...
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-11-17T04:51:57Z
Generation: 2
Resource Version: 7257970
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/booknotes-certificate
UID: fbe1d9c0-08f5-11ea-82b3-42010a80017a
Spec:
Acme:
Config:
Domains:
www.bnsfun.com
http01:
Ingress: boonotes-ingress-resource
Common Name: www.bnsfun.com
Dns Names:
www.bnsfun.com
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-issuer
Secret Name: letsencrypt-certs
Events: <none>
I've also created a secret:
Here is my sevice & ingress section:
I've used this tutorial to configure it :
https://medium.com/#betandr/kubernetes-ingress-with-tls-on-gke-744efd37e49e
and official documentation of cert-manager to install cert managed. What do I wrong? How can I check why this doesn't work? I've tried a lot of stuff, but all doesn't work for me. For sure I do something wrong. But what? I've understood that I need cert-manager for updating my lets-encrypt certificate, also I need to create secret to store it, then I need configure my ingress in tls and annotaions. Pls could you help me to find out more what should happen there and what are the main steps to complete it? If you need more info , pls let me know
Here is my issuer:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-issuer
spec:
acme:
email: email
http01: {}
privateKeySecretRef:
name: letsencrypt
server: "https://acme-v02.api.letsencrypt.org/directory"
let's take an another path, Letsencrypt official docs say that they won't be supporting any longer for below 0.8 versions, so I recommend you to install cert-manager provided by Jetstack, that you can find here, to install the helm chart for it.
The follow this stackoverflow post, for configurations, note that if the api version mentioned in that post doesn't support in case of cluster issuer, then rather use
apiVersion: cert-manager.io/v1alpha2
Note that , the tls secret name mentioned in the certificate will be auto-generated by cert-manager, and it automatically starts an acme-challenge to validate the domain, once you patch that secret name to the TLS in your ingress rule.
It shall solve the issue and the certificate's status will change to ready after the domain verification

Kubernetes's Ingress annotations for x509 certificate authentificate

I'm trying to use kubernetes ingress annotation rules in order to enable X509 authentication.
My ingress yaml file is defined below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: bdf-opengie-test
name: keycloak-opengie-test-ssl
labels:
app: keycloak-opengie
annotations:
nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
nginx.ingress.kubernetes.io/auth-tls-secret: "opengie-tls-secret"
nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
spec:
rules:
- host: keycloak-opengie-test-ssl.bdf-clu4.paas.eclair.local
http:
paths:
- path: /
backend:
serviceName: keycloak-opengie
servicePort: http
tls:
- hosts:
- keycloak-opengie-test-ssl.bdf-clu4.paas.eclair.local
When I invoke my application url, I'm expecting to see a popup requesting for a certificate, but nothing happens.
It seems like the annotations has no effect in the ingress definition.
Can someone tell me what's going wrong in my ingress definition.
I'm using Nginx Ingress: 0.15.0 and Kubernetes 1.10.5
First of all you are missing the secret with SSL files issued for your domain. (if we are talking about a native k8s secret management) You secret should be created by:
kubectl --namespace bdf-opengie-test create secret tls <secret_name> --key <key_path> --cert <cert_path>
Then your Ingress .yml file should contain this secret:
...
tls:
- hosts:
- keycloak-opengie-test-ssl.<domain>
secretName: <secret_name>
Only after this you can think about any annotations for auth or something else which is not working
Note: the secret is a namespaced object.
ingress:
hostname: id.login.example.com
annotations:
nginx.ingress.kubernetes.io/configuration-snippet: "more_clear_input_headers \"x-forwarded-client-cert\";\n more_set_input_headers \"x-forwarded-client-cert: $ssl_client_cert\";\n"
nginx.ingress.kubernetes.io/server-snippet: |
ssl_verify_client on;
ssl_client_certificate /etc/nginx/truststore-development.crt;
ssl_verify_depth 2;
ssl_session_cache off;