Specific requests being sent to general path on Nginx Ingress - kubernetes

I have a Flask Python APP that has many routes. This app (container) is served in many different deployments, one for each plan type. Example: we have one deployment called esg-enterprise to process the enterprise plan, another esg-professional for professional and so on. Finally, we have another pod just to serve our frontend application with authentication and the models it needs. All this is the same container.
As you can see in the Ingress file bellow, we have the backend rules to route the traffic to specific services. The problem is that, most of the specific requests like /task/connections/update/advanced/ or /task/connections/update/advanced/ are being sent to the root path / that should only serve the front-end (which works since they use the same container). The problem is that those specific requests are very massive causing the front end API to become unavailable. The specific services run on stronger nodes so they can handle this load, while the front-end api runs on weaker nodes. When I run kubectl get hpa I can see that most of the load (replicas) stays on the API. I even had to increase the max replicas to try to mitigate the issue. I've seen in the logs that some requests are being sent to the specific routes as they should, but the majority is not being sent.
All I wanted to do is to prevent / route to receive those specific requests.
Our ingress looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/proxy-body-size: 150m
nginx.ingress.kubernetes.io/proxy-connect-timeout: "1200"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1200"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1200"
nginx.ingress.kubernetes.io/upstream-fail-timeout: "1200"
name: ingress-nginx
namespace: default
spec:
tls:
- hosts:
- app.ourdomain.com
- api.ourdomain.com
secretName: ourdomain-certificate
rules:
- host: app.ourdomain.com
http:
paths:
- backend:
serviceName: frontend
servicePort: 80
path: /
- host: api.ourdomain.com
http:
paths:
- backend:
serviceName: api
servicePort: 8000
path: /
- backend:
serviceName: esg
servicePort: 8000
path: /cron/
- backend:
serviceName: esg-bigquery-migration
servicePort: 8000
path: /cron/big-query/
- backend:
serviceName: esg
servicePort: 8000
path: /task/
- backend:
serviceName: esg-trial
servicePort: 8000
path: /task/connections/update/trial/
- backend:
serviceName: esg-advanced
servicePort: 8000
path: /task/connections/update/advanced/
- backend:
serviceName: esg-professional
servicePort: 8000
path: /task/connections/update/professional/
- backend:
serviceName: esg-enterprise
servicePort: 8000
path: /task/connections/update/enterprise/

From documentation:
Starting in Version 0.22.0, ingress definitions using the annotation
nginx.ingress.kubernetes.io/rewrite-target are not backwards
compatible with previous versions. In Version 0.22.0 and beyond, any
substrings within the request URI that need to be passed to the
rewritten path must explicitly be defined in a capture group.
In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. To circumvent this you can set the annotation ingress.kubernetes.io/rewrite-target to the path expected by the service. Please, refer to documentation to learn more about this. Also, you should add following line to your annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1

Related

REST URI with NGINX Ingress Controller

I'm trying to configure NGINX Ingress controller as the correct entry point to my Kubernetes cluster. Inside the cluster, I've created two REST Web services as well as frontend application. I'm trying to achieve the following scenario.
When the ingress IP is hit without any parameters it should be routed
to the frontend app. Example: 192.168.1.20 should lead to frontend
service on port 80.
When parameters are given, the request should be
routed to correct REST service. Example:
192.168.1.20/first-rest/api/flower?id=1 should route the request to
the first-rest service so that it could return the flower with id =
1.
I can correctly access the frontend application but when trying to access any REST service I'm getting 404 error or no response at all. First-rest, Second-rest and frontend are running correctly and are accessible when configured as LoadBalancer services. With Nginx, they are configured as ClusterIp services.
My ingress configuration
------------------------
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: main-routes
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- http:
paths:
- path: /first-rest(/|$)(.*)
backend:
serviceName: first-rest
servicePort: 8090
- path: /second-rest(/|$)(.*)
backend:
serviceName: second-rest
servicePort: 9000
- path: /(.*)
backend:
serviceName: frontend
servicePort: 80
It seems like NGINX is cutting short my URL parameters that are required for my REST API. Is there any way to pass the right URL path so that `192.168.1.20/first-rest/api/flower?id=1` would be routed to `[first-rest add and port]/api/flower?id=1` ?
You need to specify the ingress path type otherwise , depending on the ingress class specific implementation it will default to either exact or prefix (I assume in your case it is defaulting to exact)
So, you need to do something like
spec:
rules:
- http:
paths:
- path: /first-rest
pathType: Prefix
backend:
serviceName: first-rest
servicePort: 8090
see docs on ingress path here

Nginx Ingress Controller: What is the purpose of the host variable?

I have this nginx ingress controller:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
kubernetes.io/ingress.allow-http: "true"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
# Limit uploads to 8TB
nginx.ingress.kubernetes.io/proxy-body-size: 800000m
spec:
rules:
- host: myhost.info
http:
paths:
# NOTE: this one should come after all other routes. To avoid hijacking requests.
- path: /api/walrus(/|$)(.*)
backend:
serviceName: service-a
servicePort: 8080
- path: /api(/|$)(.*)
backend:
serviceName: service-b
servicePort: 8080
- path: /(.*)
backend:
serviceName: frontend
servicePort: 8080
- http:
paths:
# NOTE: this one should come after all other routes. To avoid hijacking requests.
- path: /api/walrus(/|$)(.*)
backend:
serviceName: service-a
servicePort: 8080
- path: /api(/|$)(.*)
backend:
serviceName: service-b
servicePort: 8080
- path: /(.*)
backend:
serviceName: frontend
servicePort: 8080
I duplicated the paths just make it clear. My question is, what is the difference in the end result when I add the host key vs when I don't?
Until now I've used it because if I don't have it I'm getting my POST request redirected into get request as in this question: Kubernetes NGINX Ingress changes HTTP request from a POST to a GET
But I also noticed that on EKS, if I DO add a host, the ingress just returns 404 for everything until I remove it and leavit only with http. So I'm a bit confused on this and wanted someone to clarify the correct way to do things here.
Also, for a production enviroment, how do I set the host correctly to a public domain and how do I set the tls certificates?
Regarding HTTPS: https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/#bGA9CAkdlMh has a section "Defining the Ingress resource (with SSL termination) to route traffic to the services created above" that shows how to terminate TLS at nginx-ingress. Even if you're not using an AWS Network Load Balancer (NLB), that may be helpful. In the case of AWS with an NLB, you have another option, terminating at the NLB: https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/
There are two nginx ingress controllers, and it's unclear which one you're using. The nginxinc controller requires a 'host'. The other, https://github.com/kubernetes/ingress-nginx, I'm not sure about. When you use TLS, nginx uses SNI for HTTPS, which seems like it would require a 'host': http://nginx.org/en/docs/http/configuring_https_servers.html#sni

Traefik 'PathPrefix' not working as expected

I have a kube service that has a /customers resource that will return all customers. It can also return a specific customer at /customers/1. I've configured Traefik ingress as follows:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
traefik.frontend.rule.type: PathPrefix
name: customerd
namespace: video
spec:
rules:
- host: custd.kube
http:
paths:
- backend:
serviceName: customerd
servicePort: http
path: /customers
- backend:
serviceName: customerd
servicePort: http
path: /custdhealth
- backend:
serviceName: customerd
servicePort: http
path: /metrics
- backend:
serviceName: customerd
servicePort: http
path: /sleeper
Note that the following annotation is present: traefik.frontend.rule.type: PathPrefix. From the Traefik documentation:
Use a * Prefix * matcher if your backend listens on a particular base
path but also serves requests on sub-paths. For instance, PathPrefix:
/products would match /products but also /products/shoes and
/products/shirts. Since the path is forwarded as-is, your backend is
expected to listen on /products.
The issue is that when I submit a request to /customers/1 the response is a 404. I've confirmed that the request doesn't reach the service. If I change PathPrefix to PathPrefixStrip requests to /customers return a 404, as expected, since the service isn't listening on /. So it seems like I'm using the annotation correctly.
Any ideas what I'm doing wrong or further troubleshooting steps?
After more debugging I figured out the problem. It wasn’t with how I was using Traefik, it was a misunderstanding on my part about how Golang HTTP routing works. My route was coded as “/customers”. Turns out this route will never satisfy “/customers/{id}”. “/customers/“ however will route both “/customers” and “/customers/{id}”. So after a simple code change it all worked.
One factor complicating my debugging efforts was this behavior isn’t visible by inspecting the (non-Golang library) code or putting in debug log messages.

Kubenetes Ingress API Routing

I have a react web application listening on the default path and I'm looking to include my API backend on the same URL.
A snippet of my ingress is below:
http:
paths:
- backend:
serviceName: atsweb
servicePort: 80
path: /(.*)
- backend:
serviceName: atsapi
servicePort: 80
path: /api(/|$)(.*)
My API has a bunch of functions which are routed after /api/ and I have a test page at mydomain.io/api/values which I cannot get to. My frontend service works fine.
Is it just the pathing which is incorrect?
I've deployed a standalone API just to check the container port/service ports are correct.
Looks like you copied the example from . What are your ingress annotations? Check the rewrite as it looks like is making a redirect. Nonetheless, the ingress that would work for looks like this:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: your-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: atsweb
servicePort: 80
- path: /api/
backend:
serviceName: atsapi
servicePort: 80
Check there is no rewrite annotation. This makes your uri be appended fully to the next proxy. Thus, making mydomain.io/api/values go to atsapi:80/api/values

Issue with Kubernetes ingress routing to Nextjs applications

So I have an interesting use case. I am running multiple micro-services on my Kubernetes cluster. My applications use NextJS which make internal calls to _next routes.
My issue came from the fact that I needed a way to differentiate between services and their requests to the _next files. So I implemented NextJS's assetPrefix feature which works perfectly in development, appending my prefix in front of _next so the requests look like .../${PREFIX}/_next/.... That way I could set up an ingress and route files base on the prefix to the appropriate service on my cluster. I set up a Kubernetes Ingress controller following this guide: https://akomljen.com/kubernetes-nginx-ingress-controller/
My ingress config is:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: dev-ingress
spec:
rules:
- host: baseurl.com
http:
paths:
- path: /auth
backend:
serviceName: auth-svc
servicePort: 80
- path: /static/auth
backend:
serviceName: auth-svc
servicePort: 80
- path: /login
backend:
serviceName: auth-svc
servicePort: 80
- path: /settings
backend:
serviceName: auth-svc
servicePort: 80
- path: /artwork
backend:
serviceName: artwork-svc
servicePort: 80
- path: /static/artwork
backend:
serviceName: artwork-svc
servicePort: 80
So here is the problem. Now that everything is set up, properly deployed, and the ingress is running following the above guide and using the above rules, my services are trying to make requests to .../_next/... instead of .../${PREFIX}/_next/... so they can't find the proper files and nothing is working. I cannot seem to figure out what is going on. Anyone have any ideas? Thanks in advance!
You are using built-in NGINX Ingress Controller that, unfortunately, has no such functionality.
My advice is to use NGINX Plus Ingress Controller annotation functionality if you can afford it.
You can find official example here.
Example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
annotations:
nginx.org/rewrites: "serviceName=tea-svc rewrite=/;serviceName=coffee-svc rewrite=/beans/"
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea/
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee/
backend:
serviceName: coffee-svc
servicePort: 80
Below are the examples of how the URI of requests to the tea-svc are rewritten (Note that the /tea requests are redirected to /tea/).
/tea/ -> /
/tea/abc -> /abc
Below are the examples of how the URI of requests to the coffee-svc are rewritten (Note that the /coffee requests are redirected to /coffee/).
/coffee/ -> /beans/
/coffee/abc -> /beans/abc