access forbidden by rule, on ingress nginx log - kubernetes

which ingress rule is blocking the request coming in.
app : based on drupal.
any suggestions would help.
ingress log :
2022/08/11 10:00:59 [error] 20516#20516: *159406145 access forbidden by rule, client: 2a02:9b0:3d:54a2:40bf:a951:e203:79a5, server: example.com, request: "GET / HTTP/1.1", host: "example.com"
{"time": "2022-08-11T10:00:59+00:00", "remote_addr": "-", "x_forward_for": "2a02:9b0:3d:54a2:40bf:a951:e203:79a5, 172.70.156.137, 2a02:9b0:3d:54a2:40bf:a951:e203:79a5", "request_id": "d8d07bd09c84b802af91a60adbf46a73", "remote_user": "-", "bytes_sent": 583, "request_time": 0.000, "status": 403, "vhost": "example.com", "request_proto": "HTTP/1.1", "path": "/", "request_query": "-", "request_length": 647, "duration": 0.000,"method": "GET", "http_referrer": "-", "http_user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 15_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Mobile/15E148 Safari/604.1", "auth_apikey": "-", "auth_authorization": "-", "auth_X-Api-caps": "-", "auth_uid": "-"}
ingress.yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
meta.helm.sh/release-name: example-production
meta.helm.sh/release-namespace: example
nginx.ingress.kubernetes.io/configuration-snippet: |
more_clear_headers "Server";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-XSS-Protection: 1";
more_set_headers "X-Request-Id: $request_id";
more_set_headers "X-Router-Id: example-portal-anonymous";
more_set_headers "Strict-Transport-Security: max-age=15768000";
more_set_headers "Content-Security-Policy: frame-ancestors 'self'";
nginx.ingress.kubernetes.io/cors-allow-headers: X-Forwarded-For, Authorization
nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, PUT, PATCH, DELETE,
OPTIONS
nginx.ingress.kubernetes.io/cors-allow-origin: example.com
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/enable-modsecurity: "true"
nginx.ingress.kubernetes.io/enable-owasp-core-rules: "false"
nginx.ingress.kubernetes.io/limit-rpm: "0"
nginx.ingress.kubernetes.io/modsecurity-transaction-id: $request_id
nginx.ingress.kubernetes.io/proxy-buffer-size: 20k
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
creationTimestamp: "2022-08-10T11:19:21Z"
generation: 1
labels:
app.kubernetes.io/managed-by: example
name: portal-anonymous
namespace: example
resourceVersion: "523173759"
selfLink: /apis/extensions/v1beta1/namespaces/example/ingresses/portal-anonymous
uid: 9cdb29b2-a463-4851-b4c4-b5a58be28580
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: portal
servicePort: 80
path: /
pathType: Prefix
status:
loadBalancer: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""

I suspect the problem is actually revealed in your config:
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
You are allowing all IPv4 IPs with this range, but in your log, you have this:
"x_forward_for": "2a02:9b0:3d:54a2:40bf:a951:e203:79a5, 172.70.156.137, 2a02:9b0:3d:54a2:40bf:a951:e203:79a5"
So this means you're using IPv6 as well, so you might want to add ::/0 to the whitelist-source-range to allow all IPv6 as well as all IPv4 ranges. OR just omit the annotation entirely if you want to allow anything in.

Related

change in manifest file for ingress while upgrading k8s from 1.18 to 1.22 (v1beta1 to v1)using helm

I tried to make changes in menifest to run from k8s version 1.18 to 1.22 and below is how my manifest now look like
Older file (v1beta1)
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-api
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleRemoveById 933160
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Content-Type-Options: nosniff"
more_set_headers "Cache-Control: no-cache, no-store"
more_set_headers "X-XSS-Protection: 1; mode=block"
more_set_headers "Content-Security-Policy: default-src 'self';script-src 'self' 'unsafe-inline'; font-src 'self' fonts.gstatic.com fonts.gstatic.com data:;style-src 'self' fonts.googleapis.com fonts.gstatic.com 'unsafe-inline'; img-src 'self' data:; connect-src 'self' login.microsoftonline.com graph.microsoft.com https://{{.Values.instanceApiHost}} https://{{.Values.configurationApiHost}}; frame-src 'self' https://login.microsoftonline.com/;"
more_set_headers "X-RateLimit-Limit: 2500"
more_set_headers "X-Frame-Options: sameorigin"
more_clear_headers "server"
more_clear_headers "Server"
more_clear_headers "X-Powered-By"
more_clear_headers "x-powered-by"
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
tls:
- hosts:
- {{.Values.configurationApiHost}}
secretName: managedservices-configurationapi-tls
- hosts:
- {{.Values.instanceApiHost}}
secretName: managedservices-instanceapi-tls
- hosts:
- {{.Values.host}}
rules:
- host: {{.Values.configurationApiHost}}
http:
paths:
- path: /
backend:
serviceName: configuration-api
servicePort: 8080
- host: {{.Values.instanceApiHost}}
http:
paths:
- path: /
backend:
serviceName: instance-api
servicePort: 8080
...
New file (v1)
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
app.kubernetes.io/component: controller
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-api
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleRemoveById 933160
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Content-Type-Options: nosniff"
more_set_headers "Cache-Control: no-cache, no-store"
more_set_headers "X-XSS-Protection: 1; mode=block"
more_set_headers "Content-Security-Policy: default-src 'self';script-src 'self' 'unsafe-inline'; font-src 'self' fonts.gstatic.com fonts.gstatic.com data:;style-src 'self' fonts.googleapis.com fonts.gstatic.com 'unsafe-inline'; img-src 'self' data:; connect-src 'self' login.microsoftonline.com graph.microsoft.com https://{{.Values.instanceApiHost}} https://{{.Values.configurationApiHost}}; frame-src 'self' https://login.microsoftonline.com/;"
more_set_headers "X-RateLimit-Limit: 2500"
more_set_headers "X-Frame-Options: sameorigin"
more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains"
more_clear_headers "server"
more_clear_headers "Server"
more_clear_headers "X-Powered-By"
more_clear_headers "x-powered-by"
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
kubernetes.io/ingress.class: nginx
tls:
- hosts:
- {{.Values.configurationApiHost}}
secretName: managedservices-configurationapi-tls
- hosts:
- {{.Values.instanceApiHost}}
secretName: managedservices-instanceapi-tls
- hosts:
- {{.Values.host}}
rules:
- host: {{.Values.configurationApiHost}}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: configuration-api
port:
number: 8080
- host: {{.Values.instanceApiHost}}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: instance-api
port:
number: 8080
Earlier older file was working with k8s 1.18 but it is now updated to 1.22 and we need to change the ingress manifest accordingly.
Attached above is the newer one having v1 version.
I am doing it via Helm Upgrade and below is the snippet of Azure pipeline task:
- task: HelmDeploy#0
displayName: Helm deploy instance-api
inputs:
connectionType: Kubernetes Service Connection
kubernetesServiceConnection: ${{ parameters.kubernetesServiceEndpointName }}
namespace: ${{ parameters.kubernetesNamespace }}
command: upgrade
chartType: FilePath
chartPath: $(Pipeline.Workspace)/HelmCharts/instance-api
releaseName: instance-api
force: false
Getting below error
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "ingress-api" namespace: "" from "": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
Still it is picking up the older v1beta1 call and not working. please help in case there are more changes required ?
No clues....

Kubernetes rewrite-target paths for appending path to match

I'm using the OSS ingress-nginx Ingress controller and trying to create a rewrite-target rule such that I can append a path string before my string match.
If I wanted to create a rewrite rule with regex that matches /matched/path and rewrites that to /prefix/matched/path, how might I be able to do that?
I've tried something like the following but it's no good, and I'm just confused about the syntax of this ingress definition:
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
rules:
- path: /(/prefix/)(/|$)(/matched/path)(.*)
backend:
serviceName: webapp1
If I wanted to create a rewrite rule with regex that matches
/matched/path and rewrites that to /prefix/matched/path, how might
I be able to do that?
In order to achieve this you have add /prefix into your rewrite-target.
Here's a working example with ingress syntax from k8s v1.18:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: example-ingress-v118
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /prefix/$1
spec:
rules:
http:
paths:
- path: /(matched/path/?.*)
backend:
serviceName: test
servicePort: 80
Since the syntax for the new ingress changed in 1.19 (see release notes and some small info at the end) I`m placing also an example with it:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress-v119
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /prefix/$1
spec:
rules:
- http:
paths:
- path: /(matched/path/?.*)
pathType: Prefix
backend:
service:
name: test
port:
number: 80
Here is a test with http echo server:
➜ ~ curl 172.17.0.4/matched/path
{
"path": "/prefix/matched/path",
"headers": {
"host": "172.17.0.4",
"x-request-id": "011585443ebc6adcf913db1c506abbe6",
"x-real-ip": "172.17.0.1",
"x-forwarded-for": "172.17.0.1",
"x-forwarded-host": "172.17.0.4",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-scheme": "http",
"user-agent": "curl/7.52.1",
"accept": "*/*"
},
This rule will also ignore the / at the end of the request:
➜ ~ curl 172.17.0.4/matched/path/
{
"path": "/prefix/matched/path/",
"headers": {
"host": "172.17.0.4",
"x-request-id": "0575e9022d814ba07457395f78dbe0fb",
"x-real-ip": "172.17.0.1",
"x-forwarded-for": "172.17.0.1",
"x-forwarded-host": "172.17.0.4",
"x-forwarded-port": "80",
"x-forwarded-proto": "http",
"x-scheme": "http",
"user-agent": "curl/7.52.1",
"accept": "*/*"
},
Worth to mention some notable differences/changes in the new ingress syntax:
spec.backend -> spec.defaultBackend
serviceName -> service.name
servicePort -> service.port.name (for string values)
servicePort -> service.port.number (for numeric values) pathType no longer has a default value in v1; "Exact", "Prefix", or
"ImplementationSpecific" must be specified Other Ingress API updates
backends can now be resource or service backends
path is no longer required to be a valid regular expression (#89778,
#cmluciano) [SIG API Machinery, Apps, CLI, Network and Testing]

access kibana with traefik ingress

I would like to configure access to kibana with ingress.
I'm using Traefik reverse proxy.
I have this configuration, and can't get it to work.
What is the right configuration to make it work?
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: kube-system
name: rule-001
annotations:
kubernetes.io/ingress.class: traefik
#traefik.ingress.kubernetes.io/rewrite-target: "/app/kibana#/"
traefik.ingress.kubernetes.io/rewrite-target: "/"
#ingress.kubernetes.io/referrer-policy: /
#traefik.ingress.kubernetes.io/request-modifier: "AddPrefix: /app/kibana"
traefik.ingress.kubernetes.io/rule-type: PathPrefixStrip
#traefik.ingress.kubernetes.io/rule-type: PathStrip
#traefik.ingress.kubernetes.io/rule-type: PathPrefix
#traefik.ingress.kubernetes.io/rule-type: Path
traefik.ingress.kubernetes.io/app-root: "/"
traefik.ingress.kubernetes.io/preserve-host: "true"
spec:
rules:
- host:
http:
paths:
- backend:
serviceName: kibana-logging
servicePort: 5601
path: /kibana
#path: /app/kibana
Here the are the access via nodeport logs (works):
{
"type":"response",
"#timestamp":"2020-06-04T11:12:26Z",
"tags":[
],
"pid":6,
"method":"get",
"statusCode":200,
"req":{
"url":"/ui/fonts/inter_ui/Inter-UI-Bold.woff2",
"method":"get",
"headers":{
"host":"3.221.49.216:30346",
"user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0",
"accept":"application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8",
"accept-language":"en-US,en;q=0.5",
"accept-encoding":"identity",
"dnt":"1",
"connection":"keep-alive",
"referer":"http://3.221.49.216:30346/app/kibana"
},
"remoteAddress":"172.31.55.73",
"userAgent":"172.31.55.73",
"referer":"http://3.221.49.216:30346/app/kibana"
},
"res":{
"statusCode":200,
"responseTime":240,
"contentLength":9
},
"message":"GET /ui/fonts/inter_ui/Inter-UI-Bold.woff2 200 240ms - 9.0B"
}
Here are the access via ingress logs (wont work):
{
"type":"response",
"#timestamp":"2020-06-04T11:15:29Z",
"tags":[
],
"pid":6,
"method":"get",
"statusCode":200,
"req":{
"url":"/app/kibana",
"method":"get",
"headers":{
"host":"3.221.49.216",
"user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.125",
"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
"accept-encoding":"gzip, deflate",
"accept-language":"en-GB,en-US;q=0.9,en;q=0.8",
"cache-control":"max-age=0",
"upgrade-insecure-requests":"1",
"x-forwarded-for":"148.71.30.119",
"x-forwarded-host":"3.221.49.216",
"x-forwarded-port":"80",
"x-forwarded-prefix":"/kibana",
"x-forwarded-proto":"http",
"x-forwarded-server":"node1",
"x-real-ip":"148.71.30.119"
},
"remoteAddress":"172.31.55.73",
"userAgent":"172.31.55.73"
},
"res":{
"statusCode":200,
"responseTime":10,
"contentLength":9
},
"message":"GET /app/kibana 200 10ms - 9.0B"
}
Using host configuration in ingress works, but I would like to use the path.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
namespace: kube-system
name: rule-001
annotations:
kubernetes.io/ingress.class: traefik
spec:
rules:
- host: kibana.host.com
http:
paths:
- backend:
serviceName: kibana-logging
servicePort: 5601
path: /

Setting "Cache-Control" header with Kubernetes ingress

I have a kubernetes cluster running in AWS and am trying to modify the Cache Controller headers via the kubernetes ingress as such:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress-lab-static
namespace: lab
annotations:
ingress.kubernetes.io/rewrite-target: /$1
ingress.kubernetes.io/enable-cors: "true"
ingress.kubernetes.io/cors-allow-methods: GET, POST,PUT, OPTIONS, DELETE, HEAD, PATCH
ingress.kubernetes.io/cors-allow-headers: >-
Accept-Charset, Accept-Encoding, Access-Control-Request-Headers, Access-Control-Request-Method, Authorization,
Cache-Control, Connection, Content-Encoding, Content-Type, Content-Length, DNT, Date, Host, If-Modified-Since,
Keep-Alive, Origin, Referer, Server, TokenIssueTime, Transfer-Encoding, User-Agent, Vary, X-CustomHeader, X-Requested-With,
password, username, x-request-id, x-ratelimit-app, x-auth-id, x-auth-key, x-guest-token, X-HTTP-Method-Override,
x-oesp-username, x-oesp-token, x-cus, x-dev, X-Client-Id, X-Device-Code, X-Language-Code, UserRole, x-session-id, x-entitlements-token
ingress.kubernetes.io/configuration-snippet: |
more_set_headers 'Access-Control-Allow-Origin:$origin';
ingress.kubernetes.io/proxy-buffering: "on"
ingress.kubernetes.io/proxy-buffer-size: "2048k"
ingress.kubernetes.io/server-snippet: |
chunked_transfer_encoding off;
location ((https|http):\/\/.*\/test-service\/images\/.*\/imageName.*) {
more_set_headers 'Cache-Control: public, max-age=14400';
}
spec:
rules:
- host: static-url-lab.lab.cdn.example.com
http:
paths:
- path: /test-service/(.*)
backend:
serviceName: test-service
servicePort: 80
However this does not seem to be working. When I curl a resource matching that pattern I get the default values back:
# Example curl - not exact
curl -v "https://static-url-lab.lab.cdn.example.com/test-service/intent/test/image_name" -o /dev/null 2>&1 grep -E "(Cache-Control: max|X-Cache)"
< Cache-Control: max-age=172800, public
As far as I can tell the regex should be matching, but no change is taking place, what am I missing?
Try something like this :
working for me
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($request_uri ~* \.(js|css|gif|jpe?g|png)) {
expires 1M;
add_header Cache-Control "public";
}
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"

How to pass the part of request uri as custom header in Kubernetes ingress controller

I've the below configuration in ingress.yaml which forwards the requests with uris like /default/demoservice/health or /custom/demoservice/health to backend demoservice. I would want to retrieve the first part of uri (i.e default or custom in the example above)from the uri and pass as custom header to upstream.
I've deployed the ingress configmap with custom header
X-MyVariable-Path: ${request_uri}
but this sends the full request uri. How can I split?
- path: "/(.*?)/(demoservice.*)$"
backend:
serviceName: demoservice
servicePort: 80
I have found a solution, tested it and it works.
All you need is to add following annotations to your ingress object :
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-MyVariable-Path $1;
Where $1 is referencing whatever is captured in first group of regexp expression in path: field.
I've reproduced your scenario using the following yaml:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header X-MyVariable-Path $1;
nginx.ingress.kubernetes.io/use-regex: "true"
name: foo-bar-ingress
spec:
rules:
- http:
paths:
- backend:
serviceName: echo
servicePort: 80
path: /(.*?)/(demoservice.*)$
---
apiVersion: v1
kind: Service
metadata:
labels:
run: echo
name: echo
spec:
ports:
- port: 80
targetPort: 80
selector:
run: echo
---
apiVersion: v1
kind: Pod
metadata:
labels:
run: echo
name: echo
spec:
containers:
- image: mendhak/http-https-echo
imagePullPolicy: Always
name: echo
You can test using curl:
curl -k https://<your_ip>/default/demoservice/healthz
Output:
{
"path": "/default/demoservice/healthz",
"headers": {
"host": "192.168.39.129",
"x-request-id": "dfcc67a80f5b02e6fe6c647c8bf8cdf0",
"x-real-ip": "192.168.39.1",
"x-forwarded-for": "192.168.39.1",
"x-forwarded-host": "192.168.39.129",
"x-forwarded-port": "443",
"x-forwarded-proto": "https",
"x-scheme": "https",
"x-myvariable-path": "default", # your variable here
"user-agent": "curl/7.52.1",
"accept": "*/*"
},
"method": "GET",
"body": "",
"fresh": false,
"hostname": "192.168.39.129",
"ip": "::ffff:172.17.0.4",
"ips": [],
"protocol": "http",
"query": {},
"subdomains": [],
"xhr": false,
"os": {
"hostname": "echo"
}
}
I hope it helps =)
I found two ways to achieve this.
One is by using regular expression in your configmap to parse the first part of request_uri.
To do the above, you need to add nginx.ingress.kubernetes.io/use-regex: "true" annotation to your ingress, before you do so. As this is set to false by default.
Another approach is defining that particular header in the annotation itself and add something like $1 below is the example.
ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-MyVariable-Path: "$1;
Not entirely sure, that would work for you. But I got this logic from Rewrite example here.
And add the path as /customheader(/|$)(.*) something like this will create a capture group.
Hope it works and helpful.