Kubernetes nginx controller: avoid loading assets (css, js, etc.) via 302 redirect to https in browser - kubernetes

I have a local kubernetes cluster (k3s) with an nginx controller (installed via helm). I have two services (Spring-Boot myapp and an auth-server (OAuth2)).
I'm trying to make my application work with http only. Therefore, I have defined an ingress resource in the following way:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |-
if ($uri = /){
return 302 http://$http_host/myapp/;
}
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
ingressClassName: nginx
rules:
- host: myapp.cloud
http:
paths:
- path: /myapp
pathType: Prefix
backend:
service:
name: myapp
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: auth-server
port:
number: 8080
I have also added the following parameters to the nginx-controller config-map:
hsts: "false"
ssl-redirect: "false"
hsts-max-age: "0"
I have also cleared HSTS in my browsers (Safari & Firefox). SSL (server.ssl.enabled=false) is disabled for both of my backend services.
When loading http://myapp.cloud, I get redirected correctly to the login page of the auth-server (http://myapp.cloud/login). However, the page doesn't get loaded correctly, because the static assets (js, css) are not loaded. Instead the requests to load them are redirected with 302 to the same resources with https. Due to the fact that the default fake certificate of nginx is invalid, these don't get loaded.
If I access these assets directly in my browser (e.g. http://myapp.cloud/assets/style.css), I also get redirected 302 to http://myapp.cloud/assets/style.css and this doesn't load because the nginx certificate is invalid.
If I port-forward to the k8s service directly via http, they are loaded correctly.
Is there a possibility to make this work with http only or do I absolutely need to use a certificate manager etc. and make this work via https? What is missing in my configuration/settings?

I have decided to go with enabling HTTPS with a self-signed certificate, I think there's currently no way around it.

Related

Kubernetes Ingress not forwarding routes

I am fairly new to Kubernetes and have just deployed my first cluster to IBM Cloud. When I created the cluster, I get a dedicated ingress subdomain, which I will be referring to as <long-k8subdomain>.cloud for the scope of this post. Now, this subdomain works for my app. For example: <long-k8subdomain>.cloud/ping works from my browser/curl just fine- I get the expected JSON response back. But, if I add this subdomain to a CNAME record on my domain provider's DNS settings (I have used Bluehost and IBM Cloud's Internet Services), I get a 404 response back from all routes. However this response is the default nginx 404 response (it says "nginx" under "404 Not Found"). I believe this means that this means the ingress load balancer is being reached, but the request does not get routed right. I am using Kubernetes version 1.20.12_1561 on VPC gen 2 and this is my ingress-config.yaml file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress-resource
annotations:
kubernetes.io/ingress.class: "public-iks-k8s-nginx"
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Host: <long-k8subdomain>.cloud";
spec:
rules:
- host: <long-k8subdomain>.cloud
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service-name
port:
number: 80
I am pretty sure this problem is due to the annotations. Maybe I am using the wrong ones or I do not have enough. Ideally, I would like something like this: api..com/ to route correctly. I have also read a little bit about default backends, but I have not dove too much into that just yet. Any help would be greatly appreciated, as I have spent multiple hours trying to fix this.
Some sources I have used:
https://cloud.ibm.com/docs/containers?topic=containers-cs_network_planning
https://cloud.ibm.com/docs/containers?topic=containers-ingress-types
https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotations#annotations
Note: The reason why I have the second annotation is because for some reason, requests without that header were not being routed directly. So that was part of my debugging process and I just ended up leaving it as I am not sure if that annotation solves that, so I left it for now.
For the NGINX ingress controller to route requests for your own domain's CNAME record to the service instead of the IBM Cloud one, you need a rule in the ingress where the host identifies your domain.
For instance, if your domain's DNS entry is api.example.com, then change the resource YAML to:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress-resource
annotations:
kubernetes.io/ingress.class: "public-iks-k8s-nginx"
spec:
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-service-name
port:
number: 80
You should not need the second annotation for this to work.
If you want both of the hosts to work, then you could add a second rule instead of replacing host in the existing one.

Ingress controller cannot find assets

I am trying to set up an Ingress controller on Microk8s to host a react (NextJS) app.
The pod and service are up and running, as well as reachable internally via machine.domain.eu:31111
My goal is to make this service available via machine.domain.eu/dev on the default port 80 using an nginx ingress controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: machine.domain.eu
http:
paths:
- path: /dev(/|$)(.*)
pathType: Prefix
backend:
service:
name: web-svc
port:
number: 3000
If I issue a curl command via curl machine.domain.eu/dev from the machine, where the node is running, I am getting back the desired html, although I cannot see, if assets are being loaded.
When switching back to my local machine opening my browser of choice and hit that http endpoint I do get a blank screen and all static assets 404. The index.html comes through fine, yet all the other resources are not found.
I've tried any number of combinations on the annotation to rewrite to /$1 /$2, while changing the path to - path: /dev/?(.*) but to no effect.
In other words: The document itself is loaded via
http://machine.domain.eu/dev
Yet the assets, which should be loaded via
http://machine.domain.eu/dev/_next/{....}
are actually being requested via:
http://machine.domain.eu/_next/{....}
What excactly am I doing wrong here?

NGINX Ingress to Microk8s Bare metal cluster not working as expected

First a little background:
We currently have several websites and services hosted on a Plesk server and I am setting up a bare-metal development server to provide an area where we can test updates, etc. before going to production. I am using a 3 node kubernetes cluster running microk8s on Ubunutu 20.04.01. The services we host are pretty diverse: we have a couple Moodle sites, a few Wordpress sites, a site running limesurvery, an instance of Mantis bugtracker, and a few more. I have successfully gotten most of the sites containerized and running on k8s. I can also access each individual site either through a NodePort or a MetalLB load balancer.
However, I'd really like to use the NGINX Ingress Controller on top of the load balancer so that I can have a consistent way to access the sites without using a bunch of IP addresses (or in the NodePort case, ports that change). No matter what I've done, I cannot seem to get the Ingress to do what I want. I simply want to do the following:
http://<LB IP Address>/bugtracker to access the Mantis Bug Tracker site
http://<LB IP Address>/moodle1 to access one of the Moodle sites
http://<LB IP Address>/limesurvey to access the limesurvey,
etc. I seem to be able to get to the main page of the site (e.g. index.html, index.php, etc.), but any references from there on do not work - i.e. they give path not found errors or 404 errors.
Here's a sample of my Ingress file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- path: /limesurvey(/|$)(.*)
pathType: Prefix
backend:
service:
name: limesurvey-svc
port:
number: 80
- path: /moodle(/|$)(.*)
pathType: Prefix
backend:
service:
name: moodle-svc
port:
number: 8080
This ingress does not work (I get the 404s). However, if I only have one path in the file and just use '/' it works (but I can only use it for one service):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: limesurvey-svc
port:
number: 80
I think what I need is for the path (limesurvey) to continue to be pre-pended onto each URL request, but I cannot seem to make this happen. The reason I think this is because when I navigate to http://<LB IP Address>/limesurvey and click on a survey, I get a 404 error at http://<LB IP Address>/index.php/<survey number>. However, if I manually change the URL in my browser to http://<LB IP Address>/limesurvey/index.php/<survey number> it will access the survey (but still have 404s with supporting assets).
Am I trying to do something outside of what the Ingress controller was designed for? I feel like I should be able to use the rewrite-target to accomplish this but I'm missing something critical.
I appreciate any help.
Some applications relay on static content served from different URL webserver locations and moreover do internal path routing (e.g. you hit "/" path but get served "/admin" section immediately).
In such cases creation of right Ingress rules gets more tricky, and requires you to better understand behavior and constructs of your web application, to predict all possible URL path locations that user may visit (these forced by app internal redirects too), and these sourced by html code as well.
Seems like your case with limesurvey app falls into that category:
Why do I think that?
Just try to open limesurvey public demo (https://demo.limesurvey.org/) and inspect site content.
You will learn that main page is using a lot of static files (e.g. css, javascrpt files), referenced from absolute path starting with: /tmp/assets/...
<link rel="stylesheet" type="text/css" href="/tmp/assets/2d523ae6/survey.css" />
of course variants of different path locations can be matched with single smart reg-ex pattern, to avoid creation of dozen of individual ingress rules (what you tried).
What's the issue?
rules:
- http:
paths:
- path: /limesurvey(/|$)(.*) <---- it won't match "/tmp/assets/..." location
pathType: Prefix
backend:
service:
name: limesurvey-svc
port:
number: 80
Please try to create additional Ingress rule to support static file location (watch out, I'm using old syntax of Ingress resource, adjust it to your needs):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
name: ingress-limesurvey-static
spec:
rules:
- http:
paths:
- backend:
serviceName: limesurvey-svc
servicePort: 80
path: /?(.*)
Best Solution (in my opinion)
You should define custom public URL within your application directly. Detailed information can be found in Advanced path settings, see publicurl option.
This way you wouldn't need to define internal reference for static files, however it should be done during installation.

How to redirect from specified path in the ingress file to the root path of the service

I have an ingress defined as follows;
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: webserver-ingress
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/configuration-snippet: |
rewrite ^(/master)$ $1/ redirect;
spec:
rules:
- host: my.test.domain
http:
paths:
- path: /master
backend:
serviceName: airflow-webserver-svc
servicePort: 80
The service airflow-webserver-svc is pointing towards airflow webserver. the ingress now hits airflow webserver on GET /, but the site gets redirected from my.test.domain/master to my.test.domain/home which is the landing page of airflow webserver, but nginx returns 404 Page not found on that link :(. All works fine if I change the ingress path: /, going to my.test.domain open up airflow webserver /home page, which then gets redirected to /login page. But I need to open the same webpage from my.test.domain/master. How can I do that ?
It doesn't redirect from path: "/master" to "/" inside the service. Is
there any way to redirect from specified path in the ingress file to the root path of the service?
UPDATE:
Found out that airflow has its special configuration to do, when running behind proxy (here). I applied this solution but still no luck.
Made it work by keeping ingress same as mentioned above, and adding this in airflow config under webserver section;
[webserver]
enable_proxy_fix = True
base_url = http://my.test.domain/master/
and restart the webserver.

GKE: Identity-aware proxy > L7 load balancer > Custom host and path rules > Error code 11

I am using GKE Identity-aware proxy > L7 load balancer > Custom host and path rules. It works fine for the root-path. But it does not work for custom paths.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: basic-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: "my-static-ip"
spec:
rules:
- host: my.custom.org
http:
paths:
- path: /v1/*
backend:
serviceName: webv1
servicePort: 8080
- path: /v3/*
backend:
serviceName: webv3
servicePort: 8080
- path: /nginx/*
backend:
serviceName: nginx
servicePort: 80
- path: /*
backend:
serviceName: nginx
servicePort: 80
Path / and /nginx/ works fine. /v2/ and /v3/ throws There was a problem with your request. Error code 11.
If I disable the IAP, everything works fine.
Environment:
Kubernetes version (use kubectl version): 1.12.7-gke.17
Cloud provider or hardware configuration: GKE
Setting up HTTP Load Balancing with Ingress # https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer
Enabling Cloud IAP for GKE # https://cloud.google.com/iap/docs/enabling-kubernetes-howto
This error code 11 is related to a misconfiguration of the OAuth client ID1.
I see that you the failing one are using a different serviceport. If you are serving the following 2 URLs from two different backends both may have IAP enabled but configured with different clientIds.
As of GCP does not support different clientIds for different paths, and the effort for IAP to use a single clientId across b/e services is underway. For now, if you can verify the clientId, and change it to a single one if they are different, that would work.
The recommended approach would be setting up an NGINX proxy. The ingress + IAP only have one service to deal with instead of two, and it will work like a charm.