k8s - ingress-nginx - rewrite-target annotation causing 504 - kubernetes

I've deployed a default nginx ingress controller v1.5.1 via helm (kubernetes.github.io/ingress-nginx v4.4.0) to my AKS cluster. Running kubernetes v1.24.6.
I have created the following ingress to reach my app service/pod. The idea is to remove the prefix path (/api/v1) entirely in the rewrite.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: my-ingress
namespace: my-namespace
labels:
app: my-app
spec:
ingressClassName: nginx
rules:
- http:
paths:
- path: /api/v1(/|$)(.*)
pathType: Prefix
backend:
service:
name: my-service
port:
number: 8080
This results in a 504 (150s timeout). The app pod is healthy, and when I remove the rewrite-target annotation the controller becomes responsive again.
What is unusual is that I see no request log entries in the controller pod whatsoever when attempting to use the rewrite-target annotation. I've added nginx.ingress.kubernetes.io/enable-rewrite-log: "true" but it makes no difference.
Following documentation here: https://kubernetes.github.io/ingress-nginx/examples/rewrite/
What am I missing?

The problem was the version of Kubernetes on the cluster. I created a new cluster running v1.23.12 using the same ingress and had no issues with the path rewrite working as expected. To confirm, I tried v1.24.3 and v1.24.6 again on new clusters and in both instances reproduced the timeout.

Related

Deploying Jaeger in a url different than root

I am trying to deploy Jaeger all-in-one image in a kubernetes cluster.
Jaeger is not in the root of the url, meaning it's accessible through https://somedomain.com/xyz/jaeger
I have an ingress rule which seems to be pointing correctly to a Service which is also referencing fine the pod in a deployment (I can see all this in Rancher UI).
But somehow when I try to access, nginx is throwing a 502 Bad Gateway error.
This is how the ingress rule looks like
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
namespace: my-namespace
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: somedomain.com
http:
# Jaeger
- path: /xyz/jaeger(/|$)(.*)
pathType: Prefix
backend:
service:
name: jaeger
port:
number: 16868
Then in the pod definition I tried using the QUERY_BASE_PATH env var setting it to /xyz/jaeger but that made no difference at all.
The problem was an incorrect port being specified.
16868 instead of 16686

HAProxy Ingress Rewrite rules seemed not applied

I'm trying to setup a simple ingress with path rewriting to pass requests to my backend services.
Ref.: https://haproxy-ingress.github.io/v0.10/docs/configuration/keys/#rewrite-target
The ingress controller uses this image: quay.io/jcmoraisjr/haproxy-ingress:v0.10-beta.1.
Here is sample YAML:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: "myapp-apis-ingress"
namespace: "my-namespace"
labels:
app: myapp
tier: ingress
annotations:
kubernetes.io/ingress.class: "haproxy"
haproxy.org/rewrite-target: "/"
spec:
rules:
- host: "myapp.mydomain"
http:
paths:
- path: /api/v1/hello
pathType: Prefix
backend:
service:
name: "myapp-hello-svc"
port:
number: 8080
Expected behaviour:
It should route requests from https://myapp.mydomain/api/v1/hello/* to the GKE service at myapp-hello-svc:8080/*
Actual behaviour:
It routes everything to myapp-hello-svc:8080/api/v1/hello/* (myapp-hello-svc pod receives GET /api/v1/hello/*
I've tried some other combinations and rules, but none seemed working neither.
Any ideas what I may have missed here?
Thanks!
UPDATE: WORKAROUND
As currently I still can't find solution to this, I decided to use workaround by adding NGINX ingress controller to K8S cluster and routing the traffic for new APIs through it instead.

Nginx Controller Returns 404 Not Found On Path and Default Backend

I have created a kubernetes cluster using Vagrant. I created a Nginx pod and a Cluster IP service for it. I can curl both the pod and the service getting a successful result. I have now installed an Nginx Ingress Controller from: https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal-clusters and ran the following command:
kubectl create ingress nginxingress --rule="/nginx=nginx-service:80" --annotation nginx.ingress.kubernetes.io/rewrite-target=/ --default-backend=nginx-service:80 and they both have been setup correctly as far as I see as there are no errors. But whenever I try to curl the path then it fails, the controller keeps throwing a 404 Not found.
Some more information that might help:
services:
ingresses:
any help will be greatly appreciated
Try adding the ingress class annotation to the ingress configuration. kubernetes.io/ingress.class: "nginx"
use below YAML as reference and try to update the configuration.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-myserviceb
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: myserviceb.foo.org
http:
paths:
- path: /nginx
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
Had the same problem but was already using kubernetes.io/ingress.class: "nginx" as in Harsh Manvar answer.
If you need to match multiple URLs and not only the basic host URL like me, on your ingress use:
spec.rules.[your-host].path: "/(.*)"
spec.rules.[your-host].path: Prefix
metadata.annotations.nginx.ingress.kubernetes.io/use-regex: "true"
More detail about this config here
OBS: some times update a ingress dosent work, you may need to delete and recreate the ingress.

Prometheus dashboard exposed over ingress controller

I am trying to setup Prometheus in k8 cluster, able to run using helm. Accessing dashboard when i expose prometheus-server as LoadBalancer service using external ip.
Same does not work when I try to configure this service as ClusterIP and making it as backend using ingress controller. Receiving 404 error, any thoughts on how to troubleshoot this?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ops-ingress
annotations:
#nginx.org/server-snippet: "proxy_ssl_verify off;"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- http:
paths:
- path: /prometheus(/|$)(.*)
backend:
serviceName: prometheus-server
servicePort: 80
with above ingress definition in place, url “http://<>/prometheus/ getting redirected to http://<>/graph/ and then 404 error page getting rendered. When url adjusted to http://<>/prometheus/graph some of webcontrols gets rendered with lots of errors on browser console.
Prometheus might be expecting to have control over the root path (/).
Please change the Ingress to prometheus.example.com and it should work fine. (Changing it to a subdomain)
Please change your Ingress configuration file, add host field:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ops-ingress
annotations:
#nginx.org/server-snippet: "proxy_ssl_verify off;"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: prometheus.example.com
http:
paths:
- path: /prometheus(/|$)(.*)
backend:
serviceName: prometheus-server
servicePort: 80
then apply changes executing command:
$ kubectl aply -f your_ingress_congifguration_file.yaml
The host header field in a request provides the host and port
information from the target URI, enabling the origin server to
distinguish among resources while servicing requests for multiple
host names on a single IP address.
Please take a look here: hosts-header.
Ingress definition: ingress.
Useful information: helm-prometheus.
Useful documentation: ingress-path-matching.

Creating a path based Ingress on a GKE cluster

So I am in the process of migrating my bare metal cluster onto GKE and ran into an issue with the ingress. On my bare metal cluster, I used the ingress controller from nginxinc which worked fine. Below is a sample of an Ingress file of a particular deployment:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
namespace: mynamespace
name: app-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- host: myhost
http:
paths:
- path: /dev/appname(/|$)(.*)
backend:
serviceName: app
servicePort: 80
Basically, when a user types in myhost/dev/appname, myhost is resolved to a HAProxy server. The proxy then routes the request to the NodePort that the ingress service is running on.
I tried to do the same thing on my GKE cluster with the only exception being that the Ingress controller on the GKE cluster is exposed using a LoadBalancer as per the documentation
However I get a 502 error if I try to access the URL.
TL;DR: Looking for the best way to access various applications (deployments) deployed on a GKE cluster using URL's like: myhost/dev/firstapp and myhost/dev/secondapp etc.
You can use Kong Ingress as your ingress controller on GKE for your path based ingresses. You can install Kong Ingress from GCP Marketplace. It is easy to integrate and also supports various plugins for authenticating, monitoring etc.
You'll get detailed information and installation instructions from https://github.com/Kong/google-marketplace-kong-app#basic-usage
I would follow this guide on setting up Nginx on GKE. The ingress looks like below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-resource
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /hello
backend:
serviceName: hello-app
servicePort: 8080
You should be able to access your app externally http://external-ip-of-ingress-controller/hello
Now to debug 502 issue verify that the health-checks from the Loadbalancer to your app is passing or not.