Path Based Routing with Nginx Controller not working - kubernetes

I have been trying my Nginx Path-Based routing to work, however, after spending almost 4 hours, I am failed to understand, why is it not working. I have gone through almost every possible answer on StackOverflow before anyone downgrades my question, but none worked for me.
So here what I did:
I installed nginx-ingress using Helm 3 (https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/) in a separate namespace - nginx-test:
helm install my-release nginx-stable/nginx-ingress
A version of the ingress controller (https://hub.helm.sh/charts/nginx-edge/nginx-ingress):
$ POD_NAME=$(kubectl get pods -l app=nginx-controller-nginx-ingress -o jsonpath='{.items[0].metadata.name}')
$
$ kubectl exec -it $POD_NAME -- /nginx-ingress --version
Version=edge GitCommit=50e908aa
$
There are 2 basic nginx deployments, 2 services already configured in the same namespace, and working fine when I configure host-based routing for them.
Below one works fine for me (when I define host-based routing and get the required page index.html when I run both individual URLs):
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: nginx1.example.com
http:
paths:
- path: /
backend:
serviceName: nginx1
servicePort: 80
- host: nginx2.example.com
http:
paths:
- path: /
backend:
serviceName: nginx2
servicePort: 80
Now I wanted to achieve the same result using Path-Based routing, where there will be 1 URL and 2 Paths /nginx1 (pointing to nginx1 service) and /nginx2 (pointing to nginx2 service). So I configured the below ingress resource (and many permutations and combinations I applied based on different examples on internet), none of them worked for me.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-path-based
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: nginx.example.com
http:
paths:
- path: /nginx1
backend:
serviceName: nginx1
servicePort: 80
- path: /nginx2
backend:
serviceName: nginx2
servicePort: 80
When I access services directly, it works fine, however when I try to access - curl http://nginx.example.com/nginx1 or curl http://nginx.example.com/nginx2 - I get 404 Not Found error.
I was expecting to receive the same response which I was getting for Host-Based routing. But it does not seem to work.

So finally I had to install the controller using manifests, instead of helm charts (edge version).
I installed it from here (https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal), changed NodePort to LoadBalancer to get a LoadBalancer IP. I am using MetalLB on BareMetal.
$ POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
$ kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.32.0
Build: git-446845114
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.17.10
-------------------------------------------------------------------------------
$
My Ingress resource looks like the same which I posted while asking the question.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-path-based
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: nginx.gofork8s.com
http:
paths:
- path: /nginx1
backend:
serviceName: nginx1
servicePort: 80
- path: /nginx2
backend:
serviceName: nginx2
servicePort: 80
Modified the new LoadBalancer IP in /etc/hosts file to get the domain work.
192.168.0.1 nginx.example.com
Now I am able to access - http://nginx.example.com/nginx1 and http://nginx.example.com/nginx2.
I hope it will help someone. I still need to figure out settings with Helm Charts.

Related

Website responding with "default backend - 404" on GKE - Kubernetes

On one of our domains https://www.secretwish.in - all the 404 traffic is going to the default GKE Ingress whereas it has to route to my application.
All other pages on my application are working fine, the problem is just with 404 page, all traffic is going on gke default ingress. Sample URL - https://www.secretwish.in/hshshs
Need to find a solution for this so that all traffic starts routing to my app
Cluster Version - 1.14.10-gke.27
The ingress file looks like:-
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ans-commerce-ingress
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/tls-minimum-version: "1.0"
spec:
tls:
- hosts:
- www.secretwish.in
secretName: www-secretwish-in-tls
- host: www.secretwish.in
http:
paths:
- path: /
backend:
serviceName: ans-commerce
servicePort: 80
In GKE docs you can find information regarding GKE Ingress, that for specific path you should specify backend, otherwise you will received issue 404 default backend.
You can specify a default backend by providing a backend field in your
Ingress manifest. Any requests that don't match the paths in the rules
field are sent to the Service and port specified in the backend field.
... If you don't specify a default backend, GKE provides a default
backend that returns 404.
Default backend will redirect all request which could not be found
in any spec.rules.http.paths.path
For little test I've used 2 deployments and 2 services form this gke example.
Option 1 without configured default end
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
rules:
- http:
paths:
- path: /world
backend:
serviceName: hello-world
servicePort: 60000
- path: /kube
backend:
serviceName: hello-kubernetes
servicePort: 80
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176
default backend - 404
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176/kube
Hello Kubernetes!
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176/world
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-vqzxg
user#cloudshell:~ (prjoect-name)$ curl 35.244.197.176/yvgbhuij
default backend - 404
Option 2 With defailt backend
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
backend:
serviceName: hello-world
servicePort: 60000
rules:
- http:
paths:
- path: /world
backend:
serviceName: hello-world
servicePort: 60000
- path: /kube
backend:
serviceName: hello-kubernetes
servicePort: 80
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-vqzxg
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95/hello
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-kd6fg
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95/kube
Hello Kubernetes!
user#cloudshell:~ (prjoect-name)$ curl 35.244.186.95/fyvgbhujnkl
Hello, world!
Version: 2.0.0
Hostname: hello-world-deployment-7f67f479f5-vqzxg
Please keep in mind that Ingress on GKE needs about 5-6 minutes before it starts working properly
Not sure our solution applies in you case. We also faced similar issue(not exact), we didn't want to change the deployed ingress(i.e. can't add server-alias) So we some how made sure that all host address typed in browser/request translates to what is configured in ingress.
In this example, we would have created entry in local /etc/hosts
<actual ingress IP> www.secretwish.in

Minikube with nginx ingress path rewrite issue

I'm using minikube with nginx ingress.
I'm trying to use url rewriting like this
Here is my ingress definition:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: config-reader-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
backend:
serviceName: default-http-backend
servicePort: 80
rules:
- host: microservices.info
http:
paths:
- path: /reload(/|$)(.*)
backend:
serviceName: spring-cloud-reload
servicePort: 8080
- path: /upload(/|$)(.*)
backend:
serviceName: spring-cloud-upload
servicePort: 8080
The following urls rewrites like this:
/reload to /
/reload/xyx to /xyz
which is fine. But
/reload/x/y rewrites to /x// instead of /x/y
also
/reload/x/y/ rewrites to /x/y// instead of /x/y/
minikube version: v0.35.0
How can be fixed the multiple path elements case?
UPDATE
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.21.0
Build: git-b65b85cd9
Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------
Prior nginx-ingress v0.22.0 there was a known issue with trailing slashes.
To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command.
kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.24.1
Build: git-ce418168f
Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
Rewrite target annotation is very sensitive for trailing slashes. If it's not present, the request will not be rewritten.
Thus, you should always supply url as /reload/x/y/
Another problem with duplicated trailing slashes (fixed in v 0.22.0 and higher).
In order to understand, what's is going on there, you can exec to ingress-controller pod, find /etc/nginx/nginx.conf and search for
set $location_path "/reload(/|${literal_dollar})(.*)";
or
rewrite "(?i)/reload(/|$)(.*)" /$2 break;
blocks
To fix it, either update ingress-controller version to the latest or you can also use configuration-snippet annotation:
nginx.ingress.kubernetes.io/configuration-snippet: |
rewrite (?i)/reload/x/y/ /x/y break;

How to make Traefik compatible with Microk8s

I have a working setup on Minikube with Traefik as ingress controller. I tried to use that setup on Microk8s but Traefik is not able to work and although I can see the Traefik dashboard and it says that everything is working but every time I try to use the ingress urls I face timeout but if I use the endpoint IP of that service (which I can see in the traefik dashboard) I am able to access to that service but not fully. I can have access to IP/service1 but I can't have access to any of its sub urls, IP/service1/sub-service1 not working.
I also tried microk8s.enable ingress but it created an nginx ingress for me and then I disabled it because I want to use traefik.
Do I need to change my configuration so it becomes compatible with Microk8s? If yes how?
I have to mention that I have two ingress files:
traefik-ui.yaml: which contains both the service and ingress for my traefik. I use this service+ingress to access the traefik dashboard and as I mentioned it works
wws-ingress.yaml: is my main ingress which enables the communication with my components inside kubernetes and this is the part that doesn't work.
My yaml files:
traefik-ui.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- name: web
port: 80
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
rules:
- host: traefik-ui.minikube
http:
paths:
- path: /
backend:
serviceName: traefik-web-ui
servicePort: web
wws-ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wws
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.frontend.rule.type: PathPrefixStrip
traefik.frontend.passHostHeader: "true"
traefik.backend.loadbalancer.sticky: "true"
#traefik.ingress.kubernetes.io/rule-type: ReplacePathRegex
traefik.wss.protocol: http
traefik.wss.protocol: https
spec:
rules:
- host: streambridge.local
http:
paths:
- path: /streambridge
backend:
serviceName: streambridge
servicePort: 9999
- path: /dashboard
backend:
serviceName: dashboard
servicePort: 9009
- path: /gateway
backend:
serviceName: gateway
servicePort: 8080
- path: /rdb
backend:
serviceName: rethinkdb
servicePort: 8085
Minikube commands (this works without a problem):
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
kubectl apply -f traefik-ui.yaml
kubectl apply -f wws-ingress.yaml
And in Microk8s I tried:
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
microk8s.kubectl apply -f traefik-ui.yaml
microk8s.kubectl apply -f wws-ingress.yaml
After testing my setup on another machine and seeing that it is working there I found out that something is wrong with my machine and after spending a good amount of time on this with the help of two of my colleagues and trying everything we found out that the problem is related to iptable in my machine and we solved it as described here: https://github.com/ubuntu/microk8s/issues/72

Kubernetes ingress edit: HTTP 400 Bad request - The plain HTTP request was sent to HTTPS

Could there be any reason why a webapp which perfectly loads up fine gives a *HTTP 400 Bad request - The plain HTTP request was sent to HTTPS* port after the webapp's ingress has been edited manually or edited through an automated job which updates the ingress modifying the Whitelisted IPs
Apparently, this issue gets fixed when we redeploy the webapp after purging the webapp deployment...
Any pointers to this would be great as this happens on our PROD env and not reproducible on any lower envs.
Points to note:-
- Nginx Ingress controller setup is the same across lower envs and Prod env.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/force-ssl-redirect: "true"
ingress.kubernetes.io/ingress.allow-http: "false"
ingress.kubernetes.io/proxy-body-size: 20M
ingress.kubernetes.io/secure-backends: "true"
ingress.kubernetes.io/whitelist-source-range: xxx.yy.zz.pp/32, yyy.ss.dd.kkl/32
ingress.kubernetes.io/whitelist-source-range-status: unlocked
creationTimestamp: 2018-11-29T15:34:05Z
generation: 5
labels:
app: xyz-abc-pqr
name: xxxx-webapp-yyyy-zzzz
namespace: nspsace-name
resourceVersion: "158237270"
selfLink: /apis/extensions/v1beta1/namespaces/nspsace-name/ingresses/xxxx-webapp-yyyy-zzzz
uid: 348f892e-f3ec-11e8-aa6f-0a0340041348
spec:
rules:
- host: ssssssss.wwwwweewwerwerw.co.uk
http:
paths:
- backend:
serviceName: xxxx-webapp-yyyy-zzzz
servicePort: 443
path: /
- host: xxxx-webapp-yyyy-zzzz.bbbbv.lasdfkla.ajksdfh.ioohhsaf.pp
http:
paths:
- backend:
serviceName: xxxx-webapp-yyyy-zzzz
servicePort: 443
path: /
tls:
- hosts:
- ssssssss.wwwwweewwerwerw.co.uk
- xxxx-webapp-yyyy-zzzz.bbbbv.lasdfkla.ajksdfh.ioohhsaf.pp
secretName: xxxx-webapp-yyyy-zzzz-server-tls
status:
loadBalancer:
ingress:
- {}
There may be something wrong with the ingress controller and how it updates its configuration. I'm assuming you are using a nginx ingress controller so you can inspect the configs before an after:
$ kubectl cp <nginx-ingress-controller-pod>:nginx.conf nginx.conf.before
$ kubectl edit ingress <your-ingress>
$ kubectl cp <nginx-ingress-controller-pod>:nginx.conf nginx.conf.after
$ diff nginx.conf.before nginx.conf.after
You can see the this may happen with nginx because of something like this: Dealing with nginx 400 "The plain HTTP request was sent to HTTPS port" error.

Setting up a Kuberentes cluster with HTTP Load balancing ingress for RStudio and Shiny results in error pages

I'm attempting to create a cluster on Google Kubernetes Engine that runs nginx, RStudio server and two Shiny apps, following and adapting this guide.
I have 4 workloads that are all green in the UI, deployed via:
kubectl run nginx --image=nginx --port=80
kubectl run rstudio --image gcr.io/gcer-public/persistent-rstudio:latest --port 8787
kubectl run shiny1 --image gcr.io/gcer-public/shiny-googleauthrdemo:latest --port 3838
kubectl run shiny5 --image=flaviobarros/shiny-wordcloud --port=80
They were then all exposed as node ports via:
kubectl expose deployment nginx --target-port=80 --type=NodePort
kubectl expose deployment rstudio --target-port=8787 --type=NodePort
kubectl expose deployment shiny1 --target-port=3838 --type=NodePort
kubectl expose deployment shiny5 --target-port=80 --type=NodePort
..that are all green in the UI.
I then deployed this Ingress backend
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: r-ingress
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80
- path: /rstudio/
backend:
serviceName: rstudio
servicePort: 8787
- path: /shiny1/
backend:
serviceName: shiny1
servicePort: 3838
- path: /shiny5/
backend:
serviceName: shiny5
servicePort: 80
The result is that the nginx routing works great, I can see "Welcome to nginx" webpage from home, but the three other paths I get:
/rstudio/ - Input/output error
/shiny1/ - Page not found (the Shiny 404 page)
/shiny5/ - Page not found (the Shiny 404 page)
The RStudio and Shiny workloads both work when exposing via the single load balancer, mapped to 8787 and 3838 respectively.
Can anyone point to where I'm going wrong?
Qs:
Do the Dockerfiles need to be adapted so they all give a 200 status on port 80 when requesting "/"? Do I need to change the health checker? I tried changing the RStudio login page (that 302 to /auth-sign-in if you are not logged in) but no luck
Both RStudio and Shiny need websockets - does this affect this?
Does session affinity need to be on? I tried adding that with IP but same errors.
As Radek suggested, ingress.kubernetes.io/rewrite-target: / is required to re-write your requests. However, this is not currently supported by the GKE ingress controller and is the reason that you're receiving 404 responses.
Instead, on GKE, you must use an nginx ingress controller.
You will then be able to configure ingress for your rstudio and shiny images that obeys the rewrite rule:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: r-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- backend:
serviceName: rstudio
servicePort: 8787
path: /rstudio/*
- backend:
serviceName: shiny1
servicePort: 3838
path: /shiny1/*
- backend:
serviceName: shiny5
servicePort: 80
path: /shiny5/*
the most likely problem you have is that when you go with this ingress you attached your URI is different then with direct accesc ( /shiny1/ vs / ) so your app is lost and has no content for that uri.
With Nginx Ingress Controller you can use ingress.kubernetes.io/rewrite-target: / annotation to mitigate this and make sure that / is accessed even when there is a subfolder in the ingress path.