Traefik 'PathPrefix' not working as expected - kubernetes

I have a kube service that has a /customers resource that will return all customers. It can also return a specific customer at /customers/1. I've configured Traefik ingress as follows:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
traefik.frontend.rule.type: PathPrefix
name: customerd
namespace: video
spec:
rules:
- host: custd.kube
http:
paths:
- backend:
serviceName: customerd
servicePort: http
path: /customers
- backend:
serviceName: customerd
servicePort: http
path: /custdhealth
- backend:
serviceName: customerd
servicePort: http
path: /metrics
- backend:
serviceName: customerd
servicePort: http
path: /sleeper
Note that the following annotation is present: traefik.frontend.rule.type: PathPrefix. From the Traefik documentation:
Use a * Prefix * matcher if your backend listens on a particular base
path but also serves requests on sub-paths. For instance, PathPrefix:
/products would match /products but also /products/shoes and
/products/shirts. Since the path is forwarded as-is, your backend is
expected to listen on /products.
The issue is that when I submit a request to /customers/1 the response is a 404. I've confirmed that the request doesn't reach the service. If I change PathPrefix to PathPrefixStrip requests to /customers return a 404, as expected, since the service isn't listening on /. So it seems like I'm using the annotation correctly.
Any ideas what I'm doing wrong or further troubleshooting steps?

After more debugging I figured out the problem. It wasn’t with how I was using Traefik, it was a misunderstanding on my part about how Golang HTTP routing works. My route was coded as “/customers”. Turns out this route will never satisfy “/customers/{id}”. “/customers/“ however will route both “/customers” and “/customers/{id}”. So after a simple code change it all worked.
One factor complicating my debugging efforts was this behavior isn’t visible by inspecting the (non-Golang library) code or putting in debug log messages.

Related

REST URI with NGINX Ingress Controller

I'm trying to configure NGINX Ingress controller as the correct entry point to my Kubernetes cluster. Inside the cluster, I've created two REST Web services as well as frontend application. I'm trying to achieve the following scenario.
When the ingress IP is hit without any parameters it should be routed
to the frontend app. Example: 192.168.1.20 should lead to frontend
service on port 80.
When parameters are given, the request should be
routed to correct REST service. Example:
192.168.1.20/first-rest/api/flower?id=1 should route the request to
the first-rest service so that it could return the flower with id =
1.
I can correctly access the frontend application but when trying to access any REST service I'm getting 404 error or no response at all. First-rest, Second-rest and frontend are running correctly and are accessible when configured as LoadBalancer services. With Nginx, they are configured as ClusterIp services.
My ingress configuration
------------------------
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: main-routes
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- http:
paths:
- path: /first-rest(/|$)(.*)
backend:
serviceName: first-rest
servicePort: 8090
- path: /second-rest(/|$)(.*)
backend:
serviceName: second-rest
servicePort: 9000
- path: /(.*)
backend:
serviceName: frontend
servicePort: 80
It seems like NGINX is cutting short my URL parameters that are required for my REST API. Is there any way to pass the right URL path so that `192.168.1.20/first-rest/api/flower?id=1` would be routed to `[first-rest add and port]/api/flower?id=1` ?
You need to specify the ingress path type otherwise , depending on the ingress class specific implementation it will default to either exact or prefix (I assume in your case it is defaulting to exact)
So, you need to do something like
spec:
rules:
- http:
paths:
- path: /first-rest
pathType: Prefix
backend:
serviceName: first-rest
servicePort: 8090
see docs on ingress path here

Nginx Ingress Controller: What is the purpose of the host variable?

I have this nginx ingress controller:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
kubernetes.io/ingress.allow-http: "true"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
# Limit uploads to 8TB
nginx.ingress.kubernetes.io/proxy-body-size: 800000m
spec:
rules:
- host: myhost.info
http:
paths:
# NOTE: this one should come after all other routes. To avoid hijacking requests.
- path: /api/walrus(/|$)(.*)
backend:
serviceName: service-a
servicePort: 8080
- path: /api(/|$)(.*)
backend:
serviceName: service-b
servicePort: 8080
- path: /(.*)
backend:
serviceName: frontend
servicePort: 8080
- http:
paths:
# NOTE: this one should come after all other routes. To avoid hijacking requests.
- path: /api/walrus(/|$)(.*)
backend:
serviceName: service-a
servicePort: 8080
- path: /api(/|$)(.*)
backend:
serviceName: service-b
servicePort: 8080
- path: /(.*)
backend:
serviceName: frontend
servicePort: 8080
I duplicated the paths just make it clear. My question is, what is the difference in the end result when I add the host key vs when I don't?
Until now I've used it because if I don't have it I'm getting my POST request redirected into get request as in this question: Kubernetes NGINX Ingress changes HTTP request from a POST to a GET
But I also noticed that on EKS, if I DO add a host, the ingress just returns 404 for everything until I remove it and leavit only with http. So I'm a bit confused on this and wanted someone to clarify the correct way to do things here.
Also, for a production enviroment, how do I set the host correctly to a public domain and how do I set the tls certificates?
Regarding HTTPS: https://aws.amazon.com/blogs/opensource/network-load-balancer-nginx-ingress-controller-eks/#bGA9CAkdlMh has a section "Defining the Ingress resource (with SSL termination) to route traffic to the services created above" that shows how to terminate TLS at nginx-ingress. Even if you're not using an AWS Network Load Balancer (NLB), that may be helpful. In the case of AWS with an NLB, you have another option, terminating at the NLB: https://aws.amazon.com/blogs/aws/new-tls-termination-for-network-load-balancers/
There are two nginx ingress controllers, and it's unclear which one you're using. The nginxinc controller requires a 'host'. The other, https://github.com/kubernetes/ingress-nginx, I'm not sure about. When you use TLS, nginx uses SNI for HTTPS, which seems like it would require a 'host': http://nginx.org/en/docs/http/configuring_https_servers.html#sni

ingress routing api prefix issue

paths:
- backend:
serviceName: booknotes-front-end-service
servicePort: 80
path: /
- backend:
serviceName: booknotes-back-end-service
servicePort: 3000
path: /api
Here is a rules in my ingres-nginx resource . I try to direct all traffic which starts from /api to my back end service, which works properly, but if some route in my back end will be like /api/users it doesn't work , my back end send response not found , when I run it locally this route working properly. Also I've tried delete /api prefix from my koa routing and change it to /users and then I've also changed path: /api to path: /users and this stuff are working properly. What should I do for fixing it ? If you need additional info , pls let me know !
Which version of nginx-ingress are you using?
They changed the way for defining a path.
https://kubernetes.github.io/ingress-nginx/examples/rewrite/
Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.
For example you can use a definition like this.
kind: Ingress
metadata:
name: some-ingress-name
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
rules:
- http:
paths:
- path: /?(.*)
backend:
serviceName: booknotes-front-end-service
servicePort: 80
- path: /api/?(.*)
backend:
serviceName: booknotes-back-end-service
servicePort: 3000
That's because it is searching for the file /api/users, which probably doesn't exist.
Put a file in /api/users/, in the backends of the service booknotes-back-end-service, say user1, and make the requeat explicitly to /api/users/user1.
You should get 200 there.
https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer , section step6 try replacing path /api with /api/* and / with /*

Specific requests being sent to general path on Nginx Ingress

I have a Flask Python APP that has many routes. This app (container) is served in many different deployments, one for each plan type. Example: we have one deployment called esg-enterprise to process the enterprise plan, another esg-professional for professional and so on. Finally, we have another pod just to serve our frontend application with authentication and the models it needs. All this is the same container.
As you can see in the Ingress file bellow, we have the backend rules to route the traffic to specific services. The problem is that, most of the specific requests like /task/connections/update/advanced/ or /task/connections/update/advanced/ are being sent to the root path / that should only serve the front-end (which works since they use the same container). The problem is that those specific requests are very massive causing the front end API to become unavailable. The specific services run on stronger nodes so they can handle this load, while the front-end api runs on weaker nodes. When I run kubectl get hpa I can see that most of the load (replicas) stays on the API. I even had to increase the max replicas to try to mitigate the issue. I've seen in the logs that some requests are being sent to the specific routes as they should, but the majority is not being sent.
All I wanted to do is to prevent / route to receive those specific requests.
Our ingress looks like this:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/proxy-body-size: 150m
nginx.ingress.kubernetes.io/proxy-connect-timeout: "1200"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1200"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1200"
nginx.ingress.kubernetes.io/upstream-fail-timeout: "1200"
name: ingress-nginx
namespace: default
spec:
tls:
- hosts:
- app.ourdomain.com
- api.ourdomain.com
secretName: ourdomain-certificate
rules:
- host: app.ourdomain.com
http:
paths:
- backend:
serviceName: frontend
servicePort: 80
path: /
- host: api.ourdomain.com
http:
paths:
- backend:
serviceName: api
servicePort: 8000
path: /
- backend:
serviceName: esg
servicePort: 8000
path: /cron/
- backend:
serviceName: esg-bigquery-migration
servicePort: 8000
path: /cron/big-query/
- backend:
serviceName: esg
servicePort: 8000
path: /task/
- backend:
serviceName: esg-trial
servicePort: 8000
path: /task/connections/update/trial/
- backend:
serviceName: esg-advanced
servicePort: 8000
path: /task/connections/update/advanced/
- backend:
serviceName: esg-professional
servicePort: 8000
path: /task/connections/update/professional/
- backend:
serviceName: esg-enterprise
servicePort: 8000
path: /task/connections/update/enterprise/
From documentation:
Starting in Version 0.22.0, ingress definitions using the annotation
nginx.ingress.kubernetes.io/rewrite-target are not backwards
compatible with previous versions. In Version 0.22.0 and beyond, any
substrings within the request URI that need to be passed to the
rewritten path must explicitly be defined in a capture group.
In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. To circumvent this you can set the annotation ingress.kubernetes.io/rewrite-target to the path expected by the service. Please, refer to documentation to learn more about this. Also, you should add following line to your annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1

Ingress paths not matching with Minikube Nginx ingress

I'm attempting to set up an Nginx Ingress on my local Minikube, but am having problems with the paths actually matching. I have two services set up, that I want to each serve different paths at the same domain. One is a Django-based API backend, the other is a Node-based frontend. My Ingress configuration is as follows:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test
spec:
rules:
- host: test.local
http:
paths:
- path: /
backend:
serviceName: frontend
servicePort: 3000
- path: /api
backend:
serviceName: backend
servicePort: 8000
- path: /admin
backend:
serviceName: backend
servicePort: 8000
- path: /static
backend:
serviceName: backend
servicePort: 8000
If I navigate to http://test.local/ in my browser, the Node frontend successfully serves that route. If I navigate to http://test.local/admin/, the Django backend successfully serves that route, and corrects redirects to http://pingpong.local/admin/login/?next=/admin/ since I am not logged in (which is also served correctly from the Django backend). However, none of the CSS loads, because http://test.local/static/ is being served by the Node frontend for some reason. Everything under the /api route also gets served by the Node frontend.
None of the documentation, examples, or other resources that I've been able to find seem to indicate that I'm doing anything incorrectly here, so I'm at a bit of a loss to figure out why it's sort of working.
Well, I'm still not sure exactly what the problem was, but after a restart of my computer, the Ingress is now working as I'd expect it to… Current best guess is some sort of caching happening somewhere.