Minikube with nginx ingress path rewrite issue - kubernetes

I'm using minikube with nginx ingress.
I'm trying to use url rewriting like this
Here is my ingress definition:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: config-reader-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
backend:
serviceName: default-http-backend
servicePort: 80
rules:
- host: microservices.info
http:
paths:
- path: /reload(/|$)(.*)
backend:
serviceName: spring-cloud-reload
servicePort: 8080
- path: /upload(/|$)(.*)
backend:
serviceName: spring-cloud-upload
servicePort: 8080
The following urls rewrites like this:
/reload to /
/reload/xyx to /xyz
which is fine. But
/reload/x/y rewrites to /x// instead of /x/y
also
/reload/x/y/ rewrites to /x/y// instead of /x/y/
minikube version: v0.35.0
How can be fixed the multiple path elements case?
UPDATE
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.21.0
Build: git-b65b85cd9
Repository: https://github.com/aledbf/ingress-nginx
-------------------------------------------------------------------------------

Prior nginx-ingress v0.22.0 there was a known issue with trailing slashes.
To detect which version of the ingress controller is running, exec into the pod and run nginx-ingress-controller version command.
kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.24.1
Build: git-ce418168f
Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------
Rewrite target annotation is very sensitive for trailing slashes. If it's not present, the request will not be rewritten.
Thus, you should always supply url as /reload/x/y/
Another problem with duplicated trailing slashes (fixed in v 0.22.0 and higher).
In order to understand, what's is going on there, you can exec to ingress-controller pod, find /etc/nginx/nginx.conf and search for
set $location_path "/reload(/|${literal_dollar})(.*)";
or
rewrite "(?i)/reload(/|$)(.*)" /$2 break;
blocks
To fix it, either update ingress-controller version to the latest or you can also use configuration-snippet annotation:
nginx.ingress.kubernetes.io/configuration-snippet: |
rewrite (?i)/reload/x/y/ /x/y break;

Related

error: resource mapping not found || make sure CRDs are installed first

error: resource mapping not found for name: "ingress-srv" namespace: "" from "ingress-srv.yaml": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
ensure CRDs are installed first
I am new to Kubernetes, I was setting up ingress nginx on minikube and it installed successfully but when I try to run using kubectl apply -f filename it gives above error
here is the code
filename: ingress-srv.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: posts.com
http:
paths:
- path: /posts
pathType: Prefix
backend:
serviceName: posts-clusterip-srv
servicePort: 4000
The resource type specified in your manifest, networking.k8s.io/v1beta1 Ingress, was removed in Kubernetes v1.22 and replaced by networking.k8s.io/v1 Ingress (see the deprecation guide for details). If your cluster's Kubernetes server version is 1.22 or higher (which I suspect it is) trying to create an Ingress resource from your manifest will result in exactly the error you're getting.
You can check your cluster's Kubernetes server version (as Kamol Hasan points out) using the command kubectl version --short.
If the version is indeed 1.22 or higher, you'll need to modify your YAML file so that its format is valid in the new version of the API. This pull request summarises the format differences. In your case, ingress-srv.yaml needs to be changed to:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: posts.com
http:
paths:
- path: /posts
pathType: Prefix
backend:
service:
name: posts-clusterip-srv
port:
number: 4000

kubernetes nginx ingress server-snippet annotation not taking effect

I have following ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-configuration-snippet
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/server-snippet: |
location /base/path/v1/api/update {
deny all;
return 404;
}
spec:
rules:
- http:
paths:
- path: /base/path(/|$)(.*)
backend:
serviceName: myApi
servicePort: 8080
But when I send PUT request to /base/path/v1/api/update instead of getting 404 I am getting 500 which means that the path is reachable from ingress controller. Can anybody help me identify why ?
I changed to configuration-snippet and the error I got is :
Error: exit status 1
2020/08/06 18:35:07 [emerg] 1734#1734: location "/base/path/v1/api/update" is outside location "^/base/path(/|$)(.*)" in /tmp/nginx-cfg008325631:2445
nginx: [emerg] location "/base/path/v1/api/update" is outside location "^/base/path(/|$)(.*)" in /tmp/nginx-cfg008325631:2445
nginx: configuration file /tmp/nginx-cfg008325631 test failed
Adding my own answer :
The final config that worked for me was :
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: nginx-configuration-snippet
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/server-snippet: |
location ~* "^/base/path/v1/api/update" {
deny all;
return 403;
}
spec:
rules:
- http:
paths:
- path: /base/path(/|$)(.*)
backend:
serviceName: myApi
servicePort: 8080
Just for other coming along here looking for a solution, worth knowing that snippets are ignored unless the nginx controller is started with the -enable-snippet option on the command line. It should be noted that this applies to nginxinc/kubernetes-ingress project.
The docs say:
Before creating a Deployment or Daemonset resource, make sure to
update the command-line arguments of the Ingress Controller container
in the corresponding manifest file according to your requirements.
For instance if you are running the Daemonset, you will have to edit the sudo nano kubernetes-ingress/deployments/daemon-set/nginx-ingress.yaml manifest and add your argument as shown below:
args:
- -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
- -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
- -enable-snippets
#- -v=3 # Enables extensive logging. Useful for troubleshooting.
#- -report-ingress-status
#- -external-service=nginx-ingress
#- -enable-prometheus-metrics
#- -global-configuration=$(POD_NAMESPACE)/nginx-configuration
Also check other Command-line Arguments in the docs here
If you are looking for helm follow this guide in the docs

Path Based Routing with Nginx Controller not working

I have been trying my Nginx Path-Based routing to work, however, after spending almost 4 hours, I am failed to understand, why is it not working. I have gone through almost every possible answer on StackOverflow before anyone downgrades my question, but none worked for me.
So here what I did:
I installed nginx-ingress using Helm 3 (https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/) in a separate namespace - nginx-test:
helm install my-release nginx-stable/nginx-ingress
A version of the ingress controller (https://hub.helm.sh/charts/nginx-edge/nginx-ingress):
$ POD_NAME=$(kubectl get pods -l app=nginx-controller-nginx-ingress -o jsonpath='{.items[0].metadata.name}')
$
$ kubectl exec -it $POD_NAME -- /nginx-ingress --version
Version=edge GitCommit=50e908aa
$
There are 2 basic nginx deployments, 2 services already configured in the same namespace, and working fine when I configure host-based routing for them.
Below one works fine for me (when I define host-based routing and get the required page index.html when I run both individual URLs):
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: nginx1.example.com
http:
paths:
- path: /
backend:
serviceName: nginx1
servicePort: 80
- host: nginx2.example.com
http:
paths:
- path: /
backend:
serviceName: nginx2
servicePort: 80
Now I wanted to achieve the same result using Path-Based routing, where there will be 1 URL and 2 Paths /nginx1 (pointing to nginx1 service) and /nginx2 (pointing to nginx2 service). So I configured the below ingress resource (and many permutations and combinations I applied based on different examples on internet), none of them worked for me.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-path-based
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: nginx.example.com
http:
paths:
- path: /nginx1
backend:
serviceName: nginx1
servicePort: 80
- path: /nginx2
backend:
serviceName: nginx2
servicePort: 80
When I access services directly, it works fine, however when I try to access - curl http://nginx.example.com/nginx1 or curl http://nginx.example.com/nginx2 - I get 404 Not Found error.
I was expecting to receive the same response which I was getting for Host-Based routing. But it does not seem to work.
So finally I had to install the controller using manifests, instead of helm charts (edge version).
I installed it from here (https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal), changed NodePort to LoadBalancer to get a LoadBalancer IP. I am using MetalLB on BareMetal.
$ POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
$ kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.32.0
Build: git-446845114
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.17.10
-------------------------------------------------------------------------------
$
My Ingress resource looks like the same which I posted while asking the question.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-path-based
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: nginx.gofork8s.com
http:
paths:
- path: /nginx1
backend:
serviceName: nginx1
servicePort: 80
- path: /nginx2
backend:
serviceName: nginx2
servicePort: 80
Modified the new LoadBalancer IP in /etc/hosts file to get the domain work.
192.168.0.1 nginx.example.com
Now I am able to access - http://nginx.example.com/nginx1 and http://nginx.example.com/nginx2.
I hope it will help someone. I still need to figure out settings with Helm Charts.

How to make Traefik compatible with Microk8s

I have a working setup on Minikube with Traefik as ingress controller. I tried to use that setup on Microk8s but Traefik is not able to work and although I can see the Traefik dashboard and it says that everything is working but every time I try to use the ingress urls I face timeout but if I use the endpoint IP of that service (which I can see in the traefik dashboard) I am able to access to that service but not fully. I can have access to IP/service1 but I can't have access to any of its sub urls, IP/service1/sub-service1 not working.
I also tried microk8s.enable ingress but it created an nginx ingress for me and then I disabled it because I want to use traefik.
Do I need to change my configuration so it becomes compatible with Microk8s? If yes how?
I have to mention that I have two ingress files:
traefik-ui.yaml: which contains both the service and ingress for my traefik. I use this service+ingress to access the traefik dashboard and as I mentioned it works
wws-ingress.yaml: is my main ingress which enables the communication with my components inside kubernetes and this is the part that doesn't work.
My yaml files:
traefik-ui.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- name: web
port: 80
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
rules:
- host: traefik-ui.minikube
http:
paths:
- path: /
backend:
serviceName: traefik-web-ui
servicePort: web
wws-ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wws
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.frontend.rule.type: PathPrefixStrip
traefik.frontend.passHostHeader: "true"
traefik.backend.loadbalancer.sticky: "true"
#traefik.ingress.kubernetes.io/rule-type: ReplacePathRegex
traefik.wss.protocol: http
traefik.wss.protocol: https
spec:
rules:
- host: streambridge.local
http:
paths:
- path: /streambridge
backend:
serviceName: streambridge
servicePort: 9999
- path: /dashboard
backend:
serviceName: dashboard
servicePort: 9009
- path: /gateway
backend:
serviceName: gateway
servicePort: 8080
- path: /rdb
backend:
serviceName: rethinkdb
servicePort: 8085
Minikube commands (this works without a problem):
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
kubectl apply -f traefik-ui.yaml
kubectl apply -f wws-ingress.yaml
And in Microk8s I tried:
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
microk8s.kubectl apply -f traefik-ui.yaml
microk8s.kubectl apply -f wws-ingress.yaml
After testing my setup on another machine and seeing that it is working there I found out that something is wrong with my machine and after spending a good amount of time on this with the help of two of my colleagues and trying everything we found out that the problem is related to iptable in my machine and we solved it as described here: https://github.com/ubuntu/microk8s/issues/72

Setting up a Kuberentes cluster with HTTP Load balancing ingress for RStudio and Shiny results in error pages

I'm attempting to create a cluster on Google Kubernetes Engine that runs nginx, RStudio server and two Shiny apps, following and adapting this guide.
I have 4 workloads that are all green in the UI, deployed via:
kubectl run nginx --image=nginx --port=80
kubectl run rstudio --image gcr.io/gcer-public/persistent-rstudio:latest --port 8787
kubectl run shiny1 --image gcr.io/gcer-public/shiny-googleauthrdemo:latest --port 3838
kubectl run shiny5 --image=flaviobarros/shiny-wordcloud --port=80
They were then all exposed as node ports via:
kubectl expose deployment nginx --target-port=80 --type=NodePort
kubectl expose deployment rstudio --target-port=8787 --type=NodePort
kubectl expose deployment shiny1 --target-port=3838 --type=NodePort
kubectl expose deployment shiny5 --target-port=80 --type=NodePort
..that are all green in the UI.
I then deployed this Ingress backend
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: r-ingress
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80
- path: /rstudio/
backend:
serviceName: rstudio
servicePort: 8787
- path: /shiny1/
backend:
serviceName: shiny1
servicePort: 3838
- path: /shiny5/
backend:
serviceName: shiny5
servicePort: 80
The result is that the nginx routing works great, I can see "Welcome to nginx" webpage from home, but the three other paths I get:
/rstudio/ - Input/output error
/shiny1/ - Page not found (the Shiny 404 page)
/shiny5/ - Page not found (the Shiny 404 page)
The RStudio and Shiny workloads both work when exposing via the single load balancer, mapped to 8787 and 3838 respectively.
Can anyone point to where I'm going wrong?
Qs:
Do the Dockerfiles need to be adapted so they all give a 200 status on port 80 when requesting "/"? Do I need to change the health checker? I tried changing the RStudio login page (that 302 to /auth-sign-in if you are not logged in) but no luck
Both RStudio and Shiny need websockets - does this affect this?
Does session affinity need to be on? I tried adding that with IP but same errors.
As Radek suggested, ingress.kubernetes.io/rewrite-target: / is required to re-write your requests. However, this is not currently supported by the GKE ingress controller and is the reason that you're receiving 404 responses.
Instead, on GKE, you must use an nginx ingress controller.
You will then be able to configure ingress for your rstudio and shiny images that obeys the rewrite rule:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: r-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- backend:
serviceName: rstudio
servicePort: 8787
path: /rstudio/*
- backend:
serviceName: shiny1
servicePort: 3838
path: /shiny1/*
- backend:
serviceName: shiny5
servicePort: 80
path: /shiny5/*
the most likely problem you have is that when you go with this ingress you attached your URI is different then with direct accesc ( /shiny1/ vs / ) so your app is lost and has no content for that uri.
With Nginx Ingress Controller you can use ingress.kubernetes.io/rewrite-target: / annotation to mitigate this and make sure that / is accessed even when there is a subfolder in the ingress path.