Setting up a Kuberentes cluster with HTTP Load balancing ingress for RStudio and Shiny results in error pages - kubernetes

I'm attempting to create a cluster on Google Kubernetes Engine that runs nginx, RStudio server and two Shiny apps, following and adapting this guide.
I have 4 workloads that are all green in the UI, deployed via:
kubectl run nginx --image=nginx --port=80
kubectl run rstudio --image gcr.io/gcer-public/persistent-rstudio:latest --port 8787
kubectl run shiny1 --image gcr.io/gcer-public/shiny-googleauthrdemo:latest --port 3838
kubectl run shiny5 --image=flaviobarros/shiny-wordcloud --port=80
They were then all exposed as node ports via:
kubectl expose deployment nginx --target-port=80 --type=NodePort
kubectl expose deployment rstudio --target-port=8787 --type=NodePort
kubectl expose deployment shiny1 --target-port=3838 --type=NodePort
kubectl expose deployment shiny5 --target-port=80 --type=NodePort
..that are all green in the UI.
I then deployed this Ingress backend
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: r-ingress
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: nginx
servicePort: 80
- path: /rstudio/
backend:
serviceName: rstudio
servicePort: 8787
- path: /shiny1/
backend:
serviceName: shiny1
servicePort: 3838
- path: /shiny5/
backend:
serviceName: shiny5
servicePort: 80
The result is that the nginx routing works great, I can see "Welcome to nginx" webpage from home, but the three other paths I get:
/rstudio/ - Input/output error
/shiny1/ - Page not found (the Shiny 404 page)
/shiny5/ - Page not found (the Shiny 404 page)
The RStudio and Shiny workloads both work when exposing via the single load balancer, mapped to 8787 and 3838 respectively.
Can anyone point to where I'm going wrong?
Qs:
Do the Dockerfiles need to be adapted so they all give a 200 status on port 80 when requesting "/"? Do I need to change the health checker? I tried changing the RStudio login page (that 302 to /auth-sign-in if you are not logged in) but no luck
Both RStudio and Shiny need websockets - does this affect this?
Does session affinity need to be on? I tried adding that with IP but same errors.

As Radek suggested, ingress.kubernetes.io/rewrite-target: / is required to re-write your requests. However, this is not currently supported by the GKE ingress controller and is the reason that you're receiving 404 responses.
Instead, on GKE, you must use an nginx ingress controller.
You will then be able to configure ingress for your rstudio and shiny images that obeys the rewrite rule:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: r-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- backend:
serviceName: rstudio
servicePort: 8787
path: /rstudio/*
- backend:
serviceName: shiny1
servicePort: 3838
path: /shiny1/*
- backend:
serviceName: shiny5
servicePort: 80
path: /shiny5/*

the most likely problem you have is that when you go with this ingress you attached your URI is different then with direct accesc ( /shiny1/ vs / ) so your app is lost and has no content for that uri.
With Nginx Ingress Controller you can use ingress.kubernetes.io/rewrite-target: / annotation to mitigate this and make sure that / is accessed even when there is a subfolder in the ingress path.

Related

Path Based Routing with Nginx Controller not working

I have been trying my Nginx Path-Based routing to work, however, after spending almost 4 hours, I am failed to understand, why is it not working. I have gone through almost every possible answer on StackOverflow before anyone downgrades my question, but none worked for me.
So here what I did:
I installed nginx-ingress using Helm 3 (https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/) in a separate namespace - nginx-test:
helm install my-release nginx-stable/nginx-ingress
A version of the ingress controller (https://hub.helm.sh/charts/nginx-edge/nginx-ingress):
$ POD_NAME=$(kubectl get pods -l app=nginx-controller-nginx-ingress -o jsonpath='{.items[0].metadata.name}')
$
$ kubectl exec -it $POD_NAME -- /nginx-ingress --version
Version=edge GitCommit=50e908aa
$
There are 2 basic nginx deployments, 2 services already configured in the same namespace, and working fine when I configure host-based routing for them.
Below one works fine for me (when I define host-based routing and get the required page index.html when I run both individual URLs):
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: nginx1.example.com
http:
paths:
- path: /
backend:
serviceName: nginx1
servicePort: 80
- host: nginx2.example.com
http:
paths:
- path: /
backend:
serviceName: nginx2
servicePort: 80
Now I wanted to achieve the same result using Path-Based routing, where there will be 1 URL and 2 Paths /nginx1 (pointing to nginx1 service) and /nginx2 (pointing to nginx2 service). So I configured the below ingress resource (and many permutations and combinations I applied based on different examples on internet), none of them worked for me.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-path-based
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: nginx.example.com
http:
paths:
- path: /nginx1
backend:
serviceName: nginx1
servicePort: 80
- path: /nginx2
backend:
serviceName: nginx2
servicePort: 80
When I access services directly, it works fine, however when I try to access - curl http://nginx.example.com/nginx1 or curl http://nginx.example.com/nginx2 - I get 404 Not Found error.
I was expecting to receive the same response which I was getting for Host-Based routing. But it does not seem to work.
So finally I had to install the controller using manifests, instead of helm charts (edge version).
I installed it from here (https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal), changed NodePort to LoadBalancer to get a LoadBalancer IP. I am using MetalLB on BareMetal.
$ POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
$ kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: 0.32.0
Build: git-446845114
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.17.10
-------------------------------------------------------------------------------
$
My Ingress resource looks like the same which I posted while asking the question.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-path-based
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- host: nginx.gofork8s.com
http:
paths:
- path: /nginx1
backend:
serviceName: nginx1
servicePort: 80
- path: /nginx2
backend:
serviceName: nginx2
servicePort: 80
Modified the new LoadBalancer IP in /etc/hosts file to get the domain work.
192.168.0.1 nginx.example.com
Now I am able to access - http://nginx.example.com/nginx1 and http://nginx.example.com/nginx2.
I hope it will help someone. I still need to figure out settings with Helm Charts.

Microk8s ingress or loadbalancer not working in local single node cluster

I was Trying out kuberenets in my local with a tutorial and ran into some issues.
I used Docker to build images of few snippets in node js and then used it to deploy in microk8s. Everything is working perfectly without the ingress controller.
The containers are
client - port 3000 (react app)
posts - port 4000
comments - port 4001
moderation ...
I applied this configuration after enabling ingress in microk8s
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: posts.com
http:
paths:
- path: /posts/create
backend:
serviceName: posts-clusterip-srv
servicePort: 4000
- path: /posts
backend:
serviceName: query-clusterip-srv
servicePort: 4002
- path: /posts/?(.*)/comments
backend:
serviceName: comments-clusterip-srv
servicePort: 4001
- path: /?(.*)
backend:
serviceName: client-clusterip-srv
servicePort: 3000
I've ensured to edit the hosts file to set
127.0.0.1 posts.com
When i check the k8s for ingress This is my result
Please guide me on how to troubleshoot this or what's going wrong in my setup.
It worked once and when i tried it again with skaffold,(it deploys pods without any issue, i can still access via NodePort service) Its not working when i goto posts.com in browser. please tell me what went wrong.
PS. When i ping posts.com its pointing to 127.0.0.1 as well
Actually I resolved it by Restarting my PC.
I'm not sure how, as I've also reinstalled microk8s and tried minikube as well.
It was something associated with the OS I suppose, somehow its working after a restart.

Prometheus dashboard exposed over ingress controller

I am trying to setup Prometheus in k8 cluster, able to run using helm. Accessing dashboard when i expose prometheus-server as LoadBalancer service using external ip.
Same does not work when I try to configure this service as ClusterIP and making it as backend using ingress controller. Receiving 404 error, any thoughts on how to troubleshoot this?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ops-ingress
annotations:
#nginx.org/server-snippet: "proxy_ssl_verify off;"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- path: /prometheus(/|$)(.*)
backend:
serviceName: prometheus-server
servicePort: 80
with above ingress definition in place, url “http://<>/prometheus/ getting redirected to http://<>/graph/ and then 404 error page getting rendered. When url adjusted to http://<>/prometheus/graph some of webcontrols gets rendered with lots of errors on browser console.
Prometheus might be expecting to have control over the root path (/).
Please change the Ingress to prometheus.example.com and it should work fine. (Changing it to a subdomain)
Please change your Ingress configuration file, add host field:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ops-ingress
annotations:
#nginx.org/server-snippet: "proxy_ssl_verify off;"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: prometheus.example.com
http:
paths:
- path: /prometheus(/|$)(.*)
backend:
serviceName: prometheus-server
servicePort: 80
then apply changes executing command:
$ kubectl aply -f your_ingress_congifguration_file.yaml
The host header field in a request provides the host and port
information from the target URI, enabling the origin server to
distinguish among resources while servicing requests for multiple
host names on a single IP address.
Please take a look here: hosts-header.
Ingress definition: ingress.
Useful information: helm-prometheus.
Useful documentation: ingress-path-matching.

How to make Traefik compatible with Microk8s

I have a working setup on Minikube with Traefik as ingress controller. I tried to use that setup on Microk8s but Traefik is not able to work and although I can see the Traefik dashboard and it says that everything is working but every time I try to use the ingress urls I face timeout but if I use the endpoint IP of that service (which I can see in the traefik dashboard) I am able to access to that service but not fully. I can have access to IP/service1 but I can't have access to any of its sub urls, IP/service1/sub-service1 not working.
I also tried microk8s.enable ingress but it created an nginx ingress for me and then I disabled it because I want to use traefik.
Do I need to change my configuration so it becomes compatible with Microk8s? If yes how?
I have to mention that I have two ingress files:
traefik-ui.yaml: which contains both the service and ingress for my traefik. I use this service+ingress to access the traefik dashboard and as I mentioned it works
wws-ingress.yaml: is my main ingress which enables the communication with my components inside kubernetes and this is the part that doesn't work.
My yaml files:
traefik-ui.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
selector:
k8s-app: traefik-ingress-lb
ports:
- name: web
port: 80
targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: traefik-web-ui
namespace: kube-system
spec:
rules:
- host: traefik-ui.minikube
http:
paths:
- path: /
backend:
serviceName: traefik-web-ui
servicePort: web
wws-ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wws
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.frontend.rule.type: PathPrefixStrip
traefik.frontend.passHostHeader: "true"
traefik.backend.loadbalancer.sticky: "true"
#traefik.ingress.kubernetes.io/rule-type: ReplacePathRegex
traefik.wss.protocol: http
traefik.wss.protocol: https
spec:
rules:
- host: streambridge.local
http:
paths:
- path: /streambridge
backend:
serviceName: streambridge
servicePort: 9999
- path: /dashboard
backend:
serviceName: dashboard
servicePort: 9009
- path: /gateway
backend:
serviceName: gateway
servicePort: 8080
- path: /rdb
backend:
serviceName: rethinkdb
servicePort: 8085
Minikube commands (this works without a problem):
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
kubectl apply -f traefik-ui.yaml
kubectl apply -f wws-ingress.yaml
And in Microk8s I tried:
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-rbac.yaml
microk8s.kubectl apply -f https://raw.githubusercontent.com/containous/traefik/v1.7/examples/k8s/traefik-ds.yaml
microk8s.kubectl apply -f traefik-ui.yaml
microk8s.kubectl apply -f wws-ingress.yaml
After testing my setup on another machine and seeing that it is working there I found out that something is wrong with my machine and after spending a good amount of time on this with the help of two of my colleagues and trying everything we found out that the problem is related to iptable in my machine and we solved it as described here: https://github.com/ubuntu/microk8s/issues/72

kubernetes: access Ingress within a Pod

I have an Ingress object set up to route traffic to the appropriate Service based on the Url path. I would like to access/expose this Ingress object within another Pod. I'm wondering if this is possible?
I tried to set up a Service on the Ingress but that didn't seem to work.
So, for whatever reason (ssr, lots of microservices, etc) you want to access k8s resources using their ingress path mapping, instead of calling each service by its internal name.
For example, you have an ingress config like that:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: example.com
http:
paths:
- path: /api/users/?(.*)
backend:
serviceName: auth-service
servicePort: 80
- path: /api/cart/?(.*)
backend:
serviceName: cart-service
servicePort: 80
and you want to access auth-service using http://example.com/api/auth instead of http://auth-service.
All you have to do is replace domain part (example.com in our case) with ingress service url. It depends on your configuration and environment, but usually it looks like http://[SERVICE_NAME].[NAMESPACE], for example:
GCP - http://ingress-nginx-controller.ingress-nginx
Helm ingress nginx - http://my-release-ingress-nginx-controller (here we are
using only service name part, because helm installs ingress in
default namespace)
Minikube - if you are using minikube ingress
addon, then you might run into problem where you cannot access
ingress, then just use helm version. (dont disable ingress addon - just install helm version alongside of it)
Get namespaces: kubectl get namespaces
Get service names inside namespace kubectl get services -n [NAMESPACE].
If you have assigned a host name, you also have to provide the domain name and IP address of the cluster to the /etc/hosts file. When you access a service via Ingress from outside the cluster, this is the file that is consulted for host name resolution.
However, a pod running inside a cluster does not have access to this /etc/hosts file. It has its own /etc/hosts file. To use ingress, the pod needs to have the same domain name and IP address entry in it's own /etc/hosts file.
To achieve this, you have to use hostAliases. Here's a sample of how that works:
apiVersion: v1
kind: Pod
metadata:
...
spec:
hostAliases:
- ip:<IP address>
hostnames:
- <host name>
For more detail on hostAliases, go to this link
I have spend so much time on this issue. I found very simple solution. I am using Mac Docker Desktop 3.3.1.
My Kubernetes Version: 1.19.7
I am trying to access UI URL from another pod running in the cluster.
My UI Service
apiVersion: v1
kind: Service
metadata:
name: my-ui-service
spec:
type: LoadBalancer
selector:
app: my-ui
ports:
- protocol: TCP
port: 8080
targetPort: 8080
Ingress for the service
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
spec:
rules:
- host: my-site.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: my-ui-service
port:
number: 8080
I have used NGINX Ingress Controller.
Command to run the Ingress Controller:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.48.1/deploy/static/provider/cloud/deploy.yaml
Once the controller is ready run the command to see the status of ingress.
kubectl get ingress
Now see the description of the ingress:
kubectl describe ingress my-ingress
Here you will find
Rules:
Host Path Backends
---- ---- --------
my-site.com
/ my-ui-service:8080 (10.1.2.198:8080)
In any pod in the cluster you can access the domain my-site.com by using my-ui-service:8080.
Inside your cluster your pods use services to reach other pods.
From outside the cluster a client may use ingress to reach services.
Ingress resource allows connection to services.
So your pod need to be reachable by a service (my-svc-N in the following example), which you're going to use in your ingress definition.
Take a look at this example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ing
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
backend:
serviceName: default-http-backend
servicePort: 80
rules:
- host: my-kube.info
http:
paths:
- path: /
backend:
serviceName: my-svc-1
servicePort: 80
- host: cheeses.all
http:
paths:
- path: /aaa
backend:
serviceName: my-svc-2
servicePort: 80
- path: /bbb
backend:
serviceName: my-svc-3
servicePort: 80