Access https backend with ingress and traefik - kubernetes

I am trying to use 'Kubernetes Ingress with Traefik, CertManager, LetsEncrypt and HAProxy' for certificates management.
What I want to do is use certificates in my application which deployed on kubernetes.
My application contains following services:
my-app1 NodePort 10.43.204.206 16686:31149/TCP
my-app2 NodePort 10.43.204.208 2746:30972/TCP
So for my-app1 without certificates I was accessing it as http://{IP}:31149/app1. And with certificates I am now accessing it as https://my-dns.com/app1. For this I am using this link. I created following ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: prod-ingress
namespace: my-ns
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: traefik
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
- hosts:
- "my-dns.com"
secretName: prod-cert
rules:
- host: "my-dns.com"
http:
paths:
- path: /app1
pathType: Prefix
backend:
service:
name: my-app1
port:
number: 16686
But for my-app2 without certificates I was accessing it as https://{IP}:30972/app2. So as I am already using https for my-app2 but I want to also use certificates for this service.
Any idea how to do this?

So the issue was, I was deploying my application with self signed certificates. So because of this I am getting issue while accessing my dashboard.
So now I just disabled self signed certificates in my application. And now I am able
to access dashboard with domain name https://my-dns.com.

Related

Enable App Gateway ingress for Azure Kubernetes on 443 (https)

I am new to AKS and trying to set up the cluster and expose it via an app gateway ingress controller. While I was able to set up the cluster using az commands and was able to deploy and hit it using HTTP. I am having some challenges in enabling HTTPS over 443 in-app gateway ingress and looking to get some help.
Below is our workflow and I am trying to setup app gateway listener on port 443
Below is the k8 we used for enabling the ingress. If I apply is without ssl cert it woks but if I give ssl cert I get a 502 bad gateway.
Cert is uploaded to KV and Cluster has KV add-on installed. But I am not sure how to attach this specific kv to cluster and whether the cert should be uploaded to gateway or Kubernetes.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-web-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: workspace-dev-cluster-cert
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
appgw.ingress.kubernetes.io/request-timeout: "90"
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-svc
port:
number: 80
This link can help you with KV add-on certificate on App GW: https://azure.github.io/application-gateway-kubernetes-ingress/features/appgw-ssl-certificate/
I use different configuration to set certs on Appgw.
I'm getting certificates via the akv2k8s tool. This creates secrets on k8s cluster.
Then I use those certs in the ingress configuration. Please check tls definition under spec.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: frontend-web-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: workspace-dev-cluster-cert
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
appgw.ingress.kubernetes.io/request-timeout: "90"
appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
tls:
- hosts:
- yourdomain.com
secretName: your-tls-secret-name
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend-svc
port:
number: 80

Nginx ingress results in redirect of login and 'invalid token specified error'

I have the following setup in Kubernetes:
Front-end Angular application (connected to Keycloak)
Keycloak for authorisation/authentication
Back-end service in Spring Webflux (connected to Keycloak)
The problem I am facing is: when I login to the Angular application (while running everything local, except Keycloak), everything works fine. However when I connect from the Angular app (weather local or deployed on kubernetes) to the Back-end Spring webflux service behind the Nginx ingress (depoloyed on kubernetes) I get redirected to the keycloak login page with the error: Invalid token specificied and token successfully revoked.
Here is my Nginx ingress config:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: main-ingress-prd
namespace: default
annotations:
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "https://auth.xxxxx.com, https://001.xxxxxx.com, http://localhost:4200"
nginx.ingress.kubernetes.io/use-regex: "true"
kubernetes.io/ingress.class: nginx
spec:
tls:
- hosts:
- mainservice.xxxxxxx.com
secretName: mainservice-com-06-01-2021
rules:
- host: mainservice.xxxxxxx.com
http:
paths:
- path: /*
pathType: Prefix
backend:
service:
name: main-service-prd
port:
number: 80
I have been struggling on this for days and playing around with settings for the Nginx ingress, but have not been successfully in getting this fixed. What am I missing here? Thanks in advance.

Application webhook endpoints are blocked on GKE (Kubernetes)

We are implementing kubernetes on GKE.
I got stuck with the following.
The staging application on is hosted on GKE 1.19.7-gke.1500.
We have an application where we have to set up webhooks with applications. When setting up the webhook I get the same error. Namely that the url is not available or does not exist.
The application on kubernetes has a valid certificate. The application works locally (minikube and skaffold) when we use ngrok.
In addition, it is possible to perform an oauth with a third party software. But we cannot activate the webhook because our webhook endpoint is not found.
I looked at the cert-manager documentation, which states that there are known issues between Cert-manager and GKE. They say that this has to do with the settings in GKE, I opened some firewall myself, but without result. https://cert-manager.io/docs/installation/compatibility/
In addition, I have looked on the internet to find out if this problem can be fixed. But I find a lot of information about webhook to be set up for Kubernetes itself but not the application that runs on it.
I think it has to do with a port incorrectly GKE configured?
See our Pod Service setting:
---
apiVersion: v1
kind: Service
metadata:
name: backend
spec:
selector:
app: back-end
ports:
- name: back-end
protocol: TCP
port: 5000
targetPort: 5000
Our Ingress setting:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.global-static-ip-name: "**.**.***.***"
kubernetes.io/ingress.class: 'nginx'
nginx.ingress.kubernetes.io/use-regex: 'true'
cert-manager.io/issuer: '****'
kubernetes.io/tls-acme: "true"
spec:
tls:
- hosts:
- test.xxxxxx.com
secretName: xxxxx
rules:
- host: test.xxxxx.com
http:
paths:
- path: /api/?(.*)
backend:
serviceName: backend
servicePort: 5000
- path: /?(.*)
backend:
serviceName: client
servicePort: 3000
Updated 03/15/2021
For example we want to set up a webhook for the following endpoint:
"https://test.xxxx.nl/api/v1/webhook/person".
We make a request to PipeDrive to set up a webhook, but then PipeDrive gives the following error: "errors: { subscription_url: [ 'invalid or non-reachable URL' ] } }"
response: [cmd]: kubectl get ep backend
NAME: backend
ENDPOINTS: 10.0.0.21:5000, 10.0.0.22:5000
AGE: 81d

Kubernetes Google OAuth2 Sign In

I'm trying to figure this out for a while and can't seem to find an answer anywhere. Maybe someone here will have an idea.
I have deployed an application on Kubernetes Cluster (in GCP). The Application is pretty much a micro-service type of app under a single domain. What I'm trying to achieve at the moment is to have a domain-wide-authentication via Google and OAuth2 protocol.
So I have a main app under the main domain https://example.com and pretty much every single path under that domain will be a separate K8s Service + Deployment. So it's a different set of containers for https://example.com/foo and different for https://example.com/bar and obviously different set of containers for the main https://example.com. I don't want to "connect" each individual App to Google due to two reasons:
Would require for me that each app has the OAuth2 Protocol implemented, and
A token granted by one app would be invalid for the other one, so it would require the user to log in each time he/she changes the URL, kind bad UX if you ask me
So, I'm trying to set up in the Kubernetes Cluster an Nginx Ingress Controller that would facilitate authentication and validate all incoming requests before they would reach any of the backend apps. This is where the problem lies...
I've managed to set up OAuth2 Proxy in my Nginx Ingress Controller and the user can log in. The whole OAuth2 Flow works. HOWEVER, the user is not forced to log in. The user can navigate to the https://example.com but he/she won't be required to have an Authentication cookie granted after successful login via Google.
Here is my YAML config for Ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/force-ssl-redirect: "true"
name: example-com-ingress
namespace: default
spec:
tls:
- hosts:
- example.com
secretName: example-com-tls
rules:
- host: example.com
http:
paths:
- backend:
serviceName: oauth2-proxy
servicePort: http-proxy
path: /oauth2
- backend:
serviceName: example-com-nginx-service
servicePort: 80
path: /
Any ideas anyone?
Make sure that while configuring oauth2 proxy you set your cookie domain (set --cookie-domain=example.com to allow the cookie to be read).
You have to also add the whitelist-domain feature while configuring oauth2 proxy:
--whitelist-domain=example.com
Then have to create ingress' obects:
1. Expose endpoint /oauth2, look at the example below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: oauth2-proxy
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: oauth2-proxy
servicePort: 4180
path: /oauth2
tls:
- hosts:
- example.com
secretName: example-com-tls
2. Then create ingress for your application:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-com-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
tls:
- hosts:
- example.com
secretName: example-com-tls
rules:
- host: example.com
http:
paths:
- path: /
backend:
serviceName: example-com-nginx-service
servicePort: 80
It is clearly to create two separate ingresses, but you can also create just one with two defined backends as you wanted.
Take a look here: nginx-ingress-controller-oauth2, oauth2-dynamic-callbacks, oauth2-proxy.
Once again make sure that you have follow this instruction: external-OAUTH-authentication.

Wildcard SSL certificate with subdomain redirect in Kubernetes

I've configured my Kubernetes to use one wildcard SSL certificate to all my apps using cert-manager and letsencrypt, now the problem is that I can't configure subdomain redirects cause Ingress is kinda "stiff". Here's how I'm trying to achieve this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-wildcard-ingress
namespace: mynamespace
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
certmanager.k8s.io/acme-challenge-type: dns01
certmanager.k8s.io/acme-dns01-provider: azuredns
ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: "domain.com"
http:
paths:
- path: /
backend:
serviceName: some-service
servicePort: 3000
- host: somesub.domain.com
http:
paths:
- path: /
backend:
serviceName: some-other-service
servicePort: 80
- host: othersub.domain.com
http:
paths:
- path: /
backend:
serviceName: one-more-service
servicePort: 8080
- host: "*.domain.com"
http:
paths:
- path: /
backend:
serviceName: default-service-to-all-other-non-mapped-subdomains
servicePort: 8000
tls:
- secretName: domain-com-tls
hosts:
- "*.domain.com.br"
The problem is that Ingress ignores the declared subdomain redirects just because they're not listed in the "tls:hosts" section. And if I do put them there, it tries to issue the SSL certificate using the wildcard and the other subdomains as well in the same cert, which causes the issuer to refuse the order, saying the obvious: "subdomain.domain.com and *.domain.com are redundant"
Is there any other way that I can declare those redirects and force them to use my SSL wildcard certificate?
Well, for anyone who's having this kind of trouble, I've managed to solve it (not the best solution, but it's a start). For this, I'll be using cert-manager and letsencrypt.
First, I've created a ClusterIssuer to issue for my certs with letsencrypt:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod-dns
spec:
acme:
dns01:
providers:
- azuredns:
clientID: MY_AZURE_CLIENT_ID
clientSecretSecretRef:
key: client-secret
name: azure-secret
hostedZoneName: mydomain.com
resourceGroupName: MY_AZURE_RESOURCE_GROUP_NAME
subscriptionID: MY_AZURE_SUBSCRIPTION_ID
tenantID: MY_AZURE_TENANT_ID
name: azuredns
email: somemail#mydomain.com
privateKeySecretRef:
key: ""
name: letsencrypt-prod-dns
server: https://acme-v02.api.letsencrypt.org/directory
Then I've created a fallback ingress to all my subdomains (this one will be the cert generator):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
certmanager.k8s.io/acme-challenge-type: dns01
certmanager.k8s.io/acme-dns01-provider: azuredns
certmanager.k8s.io/cluster-issuer: letsencrypt-prod-dns
ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/ingress.class: nginx
name: wildcard-ingress
namespace: some-namespace
spec:
rules:
- host: '*.mydomain.com'
http:
paths:
- backend:
serviceName: some-default-service
servicePort: 80
path: /
tls:
- hosts:
- '*.mydomain.com'
- mydomain.com
secretName: wildcard-mydomain-com-tls
Notice that I've declared at the TLS section the wildcard AND the absolute paths, so the cert will be valid for the URLs without subdomains too.
At this point, any requests to your domain, will be redirected to "some-default-service" with SSL(cert-manager will issue for a new cert as soon as you create the fallback ingress. This can take a while once cert-manager dns01 issuer is not mature yet), great!!!
But, what if you need to redirect some specific subdomain to another service? No problem (since they're running on the same namespace), all you have to do is to create a new ingress to your subdomain, pointing it to your existing wildcard-mydomain-com-tls cert secret:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/force-ssl-redirect: "false"
ingress.kubernetes.io/ssl-redirect: "true"
kubernetes.io/ingress.class: nginx
name: somesubdomain-ingress
namespace: some-namespace
spec:
rules:
- host: somesubdomain.mydomain.com
http:
paths:
- backend:
serviceName: some-other-service
servicePort: 8080
path: /
tls:
- hosts:
- somesubdomain.mydomain.com
secretName: wildcard-mydomain-com-tls
Easy peasy lemon squeezy!!! Now your somesubdomain.mydomain.com overrides your fallback rule and sends the user to another app. The only thing you should notice here is that the secret is valid only for "some-namespace" namespace, if you need to use this cert in another namespace, you could:
Copy the secret from namespace "some-namespace" to "other-namespace". If you do this, remember that cert-manager will NOT renew this cert automatically for "other-namespace", so, you'd have to copy the secret again, every time your cert expires.
Recreate the fallback ingress to every namespace you have, so you'd have a new cert for each of them. This approach is more ingress verbose, but, it's fully automatic.
I guess that's it. Hope someone out there can benefit from this info.
Cheers
So the best course of action here is probably to just not use ingress-shim to manage your Certificate resource.
Instead, you can manually create a Certificate resource and then reference the secret it produces in all of your ingresses.
We are exploring options to workaround this limitation in ingresses at the moment, however there has so far not been any progress!