SignalR behind k8s nginx ingress disconnects immediately upon connection - kubernetes

Here's what I have for the nginx ingress:
nginx.ingress.kubernetes.io/proxy-set-headers: xyz/proxy-headers
nginx.ingress.kubernetes.io/read-timeout: 3600
nginx.ingress.kubernetes.io/session-cookie-expires: 14400
nginx.ingress.kubernetes.io/ssl-protocols: TLSv1.3 TLSv1.2
cert-manager.io/cluster-issuer: letsencrypt
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/configuration-snippet: if ($host = 'example.com' ) {
rewrite ^ https://www.example
.com$request_uri permanent;
}
nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
nginx.ingress.kubernetes.io/force-ssl-redirect: true
nginx.ingress.kubernetes.io/proxy-body-size: 8m
nginx.ingress.kubernetes.io/proxy-send-timeout: 3600
nginx.ingress.kubernetes.io/server-alias: example.com
nginx.ingress.kubernetes.io/session-cookie-max-age: 14400
nginx.ingress.kubernetes.io/session-cookie-name: affinity
...
}
Here's the proxy headers:
apiVersion: v1
kind: ConfigMap
metadata:
name: proxy-headers
namespace: xyz
data:
X-Content-Type-Options: "nosniff"
X-XSS-Protection: "1; mode=block"
Referrer-Policy: "no-referrer-when-downgrade"
Feature-Policy: "notifications 'self'; usemedia *;gyroscope: 'none'"
When connected directly, the websocket is fine. When it's behind nginx I get the following:
main.5f9ab0903b48ce06734b.js:6 [2020-04-20T18:54:39.190Z] Information: Connection disconnected.
main.5f9ab0903b48ce06734b.js:6 Could not connect Error: Cannot send data if the connection is not in the 'Connected' State.
main.5f9ab0903b48ce06734b.js:6 ERROR Error: Uncaught (in promise): [object Undefined]
at _ (polyfills.d1de8a43b27b04443379.js:1)
at t.handshakeRejecter (polyfills.d1de8a43b27b04443379.js:1)
at t.connectionClosed (main.5f9ab0903b48ce06734b.js:6)
at t.connection.onclose (main.5f9ab0903b48ce06734b.js:6)
at t.stopConnection (main.5f9ab0903b48ce06734b.js:6)
at t.Y.transport.onclose (main.5f9ab0903b48ce06734b.js:6)
at t.close (main.5f9ab0903b48ce06734b.js:6)
at t.stop (main.5f9ab0903b48ce06734b.js:6)
at t.<anonymous> (main.5f9ab0903b48ce06734b.js:6)
at main.5f9ab0903b48ce06734b.js:6
I can't find anything that would cause it to do what it's doing and I can't get any deeper insight into what's causing it. Ideas?

Here's what I ultimately came up with:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/x-forwarded-proto: https
nginx.ingress.kubernetes.io/use-forwarded-headers: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
nginx.ingress.kubernetes.io/read-timeout: "3600"
nginx.ingress.kubernetes.io/send-timeout: "3600"
nginx.ingress.kubernetes.io/ssl-protocols: "TLSv1.3 TLSv1.2"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/session-cookie-name: "affinity"
nginx.ingress.kubernetes.io/session-cookie-expires: "14400"
nginx.ingress.kubernetes.io/session-cookie-max-age: "14400"
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: 16m
nginx.ingress.kubernetes.io/add-headers: "yournamespace/add-headers"
And then the following in the add-headers config map:
apiVersion: v1
kind: ConfigMap
metadata:
name: add-headers
namespace: yournamespace
data:
X-Content-Type-Options: "nosniff"
X-XSS-Protection: "1; mode=block"
Referrer-Policy: "no-referrer-when-downgrade"
Feature-Policy: "notifications 'self'; usemedia *;gyroscope: 'none'"
kind: ConfigMap
This enables SignalR to work properly behind the proxy, handles affinity (which apparently was the original issue) and gives you full HSTS encryption and an A+ rating on crypto tests.

Related

change in manifest file for ingress while upgrading k8s from 1.18 to 1.22 (v1beta1 to v1)using helm

I tried to make changes in menifest to run from k8s version 1.18 to 1.22 and below is how my manifest now look like
Older file (v1beta1)
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-api
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleRemoveById 933160
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Content-Type-Options: nosniff"
more_set_headers "Cache-Control: no-cache, no-store"
more_set_headers "X-XSS-Protection: 1; mode=block"
more_set_headers "Content-Security-Policy: default-src 'self';script-src 'self' 'unsafe-inline'; font-src 'self' fonts.gstatic.com fonts.gstatic.com data:;style-src 'self' fonts.googleapis.com fonts.gstatic.com 'unsafe-inline'; img-src 'self' data:; connect-src 'self' login.microsoftonline.com graph.microsoft.com https://{{.Values.instanceApiHost}} https://{{.Values.configurationApiHost}}; frame-src 'self' https://login.microsoftonline.com/;"
more_set_headers "X-RateLimit-Limit: 2500"
more_set_headers "X-Frame-Options: sameorigin"
more_clear_headers "server"
more_clear_headers "Server"
more_clear_headers "X-Powered-By"
more_clear_headers "x-powered-by"
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
tls:
- hosts:
- {{.Values.configurationApiHost}}
secretName: managedservices-configurationapi-tls
- hosts:
- {{.Values.instanceApiHost}}
secretName: managedservices-instanceapi-tls
- hosts:
- {{.Values.host}}
rules:
- host: {{.Values.configurationApiHost}}
http:
paths:
- path: /
backend:
serviceName: configuration-api
servicePort: 8080
- host: {{.Values.instanceApiHost}}
http:
paths:
- path: /
backend:
serviceName: instance-api
servicePort: 8080
...
New file (v1)
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
app.kubernetes.io/component: controller
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-api
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleRemoveById 933160
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Content-Type-Options: nosniff"
more_set_headers "Cache-Control: no-cache, no-store"
more_set_headers "X-XSS-Protection: 1; mode=block"
more_set_headers "Content-Security-Policy: default-src 'self';script-src 'self' 'unsafe-inline'; font-src 'self' fonts.gstatic.com fonts.gstatic.com data:;style-src 'self' fonts.googleapis.com fonts.gstatic.com 'unsafe-inline'; img-src 'self' data:; connect-src 'self' login.microsoftonline.com graph.microsoft.com https://{{.Values.instanceApiHost}} https://{{.Values.configurationApiHost}}; frame-src 'self' https://login.microsoftonline.com/;"
more_set_headers "X-RateLimit-Limit: 2500"
more_set_headers "X-Frame-Options: sameorigin"
more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains"
more_clear_headers "server"
more_clear_headers "Server"
more_clear_headers "X-Powered-By"
more_clear_headers "x-powered-by"
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
kubernetes.io/ingress.class: nginx
tls:
- hosts:
- {{.Values.configurationApiHost}}
secretName: managedservices-configurationapi-tls
- hosts:
- {{.Values.instanceApiHost}}
secretName: managedservices-instanceapi-tls
- hosts:
- {{.Values.host}}
rules:
- host: {{.Values.configurationApiHost}}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: configuration-api
port:
number: 8080
- host: {{.Values.instanceApiHost}}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: instance-api
port:
number: 8080
Earlier older file was working with k8s 1.18 but it is now updated to 1.22 and we need to change the ingress manifest accordingly.
Attached above is the newer one having v1 version.
I am doing it via Helm Upgrade and below is the snippet of Azure pipeline task:
- task: HelmDeploy#0
displayName: Helm deploy instance-api
inputs:
connectionType: Kubernetes Service Connection
kubernetesServiceConnection: ${{ parameters.kubernetesServiceEndpointName }}
namespace: ${{ parameters.kubernetesNamespace }}
command: upgrade
chartType: FilePath
chartPath: $(Pipeline.Workspace)/HelmCharts/instance-api
releaseName: instance-api
force: false
Getting below error
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "ingress-api" namespace: "" from "": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
Still it is picking up the older v1beta1 call and not working. please help in case there are more changes required ?
No clues....

access forbidden by rule, on ingress nginx log

which ingress rule is blocking the request coming in.
app : based on drupal.
any suggestions would help.
ingress log :
2022/08/11 10:00:59 [error] 20516#20516: *159406145 access forbidden by rule, client: 2a02:9b0:3d:54a2:40bf:a951:e203:79a5, server: example.com, request: "GET / HTTP/1.1", host: "example.com"
{"time": "2022-08-11T10:00:59+00:00", "remote_addr": "-", "x_forward_for": "2a02:9b0:3d:54a2:40bf:a951:e203:79a5, 172.70.156.137, 2a02:9b0:3d:54a2:40bf:a951:e203:79a5", "request_id": "d8d07bd09c84b802af91a60adbf46a73", "remote_user": "-", "bytes_sent": 583, "request_time": 0.000, "status": 403, "vhost": "example.com", "request_proto": "HTTP/1.1", "path": "/", "request_query": "-", "request_length": 647, "duration": 0.000,"method": "GET", "http_referrer": "-", "http_user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 15_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Mobile/15E148 Safari/604.1", "auth_apikey": "-", "auth_authorization": "-", "auth_X-Api-caps": "-", "auth_uid": "-"}
ingress.yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
meta.helm.sh/release-name: example-production
meta.helm.sh/release-namespace: example
nginx.ingress.kubernetes.io/configuration-snippet: |
more_clear_headers "Server";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-XSS-Protection: 1";
more_set_headers "X-Request-Id: $request_id";
more_set_headers "X-Router-Id: example-portal-anonymous";
more_set_headers "Strict-Transport-Security: max-age=15768000";
more_set_headers "Content-Security-Policy: frame-ancestors 'self'";
nginx.ingress.kubernetes.io/cors-allow-headers: X-Forwarded-For, Authorization
nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, PUT, PATCH, DELETE,
OPTIONS
nginx.ingress.kubernetes.io/cors-allow-origin: example.com
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/enable-modsecurity: "true"
nginx.ingress.kubernetes.io/enable-owasp-core-rules: "false"
nginx.ingress.kubernetes.io/limit-rpm: "0"
nginx.ingress.kubernetes.io/modsecurity-transaction-id: $request_id
nginx.ingress.kubernetes.io/proxy-buffer-size: 20k
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
creationTimestamp: "2022-08-10T11:19:21Z"
generation: 1
labels:
app.kubernetes.io/managed-by: example
name: portal-anonymous
namespace: example
resourceVersion: "523173759"
selfLink: /apis/extensions/v1beta1/namespaces/example/ingresses/portal-anonymous
uid: 9cdb29b2-a463-4851-b4c4-b5a58be28580
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: portal
servicePort: 80
path: /
pathType: Prefix
status:
loadBalancer: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
I suspect the problem is actually revealed in your config:
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
You are allowing all IPv4 IPs with this range, but in your log, you have this:
"x_forward_for": "2a02:9b0:3d:54a2:40bf:a951:e203:79a5, 172.70.156.137, 2a02:9b0:3d:54a2:40bf:a951:e203:79a5"
So this means you're using IPv6 as well, so you might want to add ::/0 to the whitelist-source-range to allow all IPv6 as well as all IPv4 ranges. OR just omit the annotation entirely if you want to allow anything in.

Setting up kubernets Ingress proxy-body-size based on request method

I've been trying to set up a max body size in the Ingress controller based on the HTTP method of a given path.
Basically the POST method should allow 3m as max size and all the other methods should allow 1m.
Right now my main idea was to do something like:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-custom-service
namespace: development
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
kubernetes.io/ingress.class: "nginx-dev"
nginx.ingress.kubernetes.io/configuration-snippet: |
internal;
rewrite ^ $original_uri break;
nginx.ingress.kubernetes.io/server-snippet: |
location /api/v1/my-endpoint {
if ( $request_method = POST) {
set $target_destination '/_post';
client_max_body_size 3M;
}
if ( $request_method != POST) {
set $target_destination '/_not_post';
client_max_body_size 1M;
}
set $original_uri $uri;
rewrite ^ $target_destination last;
}
spec:
tls:
rules:
- host: my-host.com
http:
paths:
- path: /_post
backend:
serviceName: my-service
servicePort: 8080
- path: /_not_post
backend:
serviceName: my-service
servicePort: 8080
But then I'm getting the following error in the pod:
Is there any way I can correctly set-up the max body size via the ingress controller?
Try changing your annotations with the configuration-snippet
nginx.ingress.kubernetes.io/configuration-snippet: |
location /upload-path {
client_max_body_size 8M;
}
Read more at : https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#configuration-snippet

GKE ingress resource returns 404 although it is connected to a service

I added a default nginx-ingress deployment with a regional IP that I got from GCP.
helm install nginx-ingress \
nginx-stable/nginx-ingress \
--set rbac.create=true \
--set controller.service.loadBalancerIP="<GCP Regional IP>"
I have a dockerized node app with a single .js file. Which I deployed with a basic helm chart. The service is called node-app-blue-helm-chart
const http = require('http');
const hostname = '0.0.0.0';
const port = 80;
const server = http.createServer((req, res) => {
if (req.url == '/another-page'){
res.statusCode = 200;
res.setHeader('Content-Type', 'text/html');
res.end('<h1>another page</h1>');
} else {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/html');
res.end('<h1>Hello World</h1>');
}
});
server.listen(port, hostname, () => {
console.log('Server running at http://%s:%s/', hostname, port);
});
process.on('SIGINT', function() {
console.log('Caught interrupt signal and will exit');
process.exit();
});
I deployed following ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-resource
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- host: "*.example.com"
http:
paths:
- path: /*
pathType: Prefix
backend:
service:
name: node-app-blue-helm-chart
port:
number: 80
Although ingress resource acquires IP and endpoint. It still returns 404 error. What can be wrong? Can host: "*.example.com" section be a problem?
More info:
kubectl describe ing ingress-resource
Name: ingress-resource
Namespace: default
Address: <GCP Regional IP>
Default backend: default-http-backend:80 (10.0.0.2:8080)
Rules:
Host Path Backends
---- ---- --------
*.example.com
/* node-app-blue-helm-chart:80 (10.0.0.15:80)
Annotations: kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: false
Events: <none>
kubectl describe svc node-app-blue-helm-chart
Name: node-app-blue-helm-chart
Namespace: default
Labels: app.kubernetes.io/instance=node-app-blue
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=helm-chart
app.kubernetes.io/version=1.16.0
helm.sh/chart=helm-chart-0.1.0
Annotations: meta.helm.sh/release-name: node-app-blue
meta.helm.sh/release-namespace: default
Selector: app.kubernetes.io/instance=node-app-blue,app.kubernetes.io/name=helm-chart
Type: ClusterIP
IP Families: <none>
IP: 10.3.248.13
IPs: 10.3.248.13
Port: http 80/TCP
TargetPort: 80/TCP
Endpoints: 10.0.0.15:80
Session Affinity: None
Events: <none>
What I tried:
Removing * from /* in ingress resource. Didn't fix the issue.
kubectl describe ing ingress-resource
Name: ingress-resource
Namespace: default
Address: W.X.Y.Z
Default backend: default-http-backend:80 (10.0.0.2:8080)
Rules:
Host Path Backends
---- ---- --------
*.example.com
/ node-app-blue-helm-chart:80 (10.0.0.15:80)
Annotations: kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated <invalid> nginx-ingress-controller Configuration for default/ingress-resource was added or updated
Try to edit your Ingress. You have set a path=/*, which may not be what you meant to do. A / should do:
[...]
spec:
rules:
- host: "*.example.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: node-app-blue-helm-chart
port:
number: 80

Setting "Cache-Control" header with Kubernetes ingress

I have a kubernetes cluster running in AWS and am trying to modify the Cache Controller headers via the kubernetes ingress as such:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress-lab-static
namespace: lab
annotations:
ingress.kubernetes.io/rewrite-target: /$1
ingress.kubernetes.io/enable-cors: "true"
ingress.kubernetes.io/cors-allow-methods: GET, POST,PUT, OPTIONS, DELETE, HEAD, PATCH
ingress.kubernetes.io/cors-allow-headers: >-
Accept-Charset, Accept-Encoding, Access-Control-Request-Headers, Access-Control-Request-Method, Authorization,
Cache-Control, Connection, Content-Encoding, Content-Type, Content-Length, DNT, Date, Host, If-Modified-Since,
Keep-Alive, Origin, Referer, Server, TokenIssueTime, Transfer-Encoding, User-Agent, Vary, X-CustomHeader, X-Requested-With,
password, username, x-request-id, x-ratelimit-app, x-auth-id, x-auth-key, x-guest-token, X-HTTP-Method-Override,
x-oesp-username, x-oesp-token, x-cus, x-dev, X-Client-Id, X-Device-Code, X-Language-Code, UserRole, x-session-id, x-entitlements-token
ingress.kubernetes.io/configuration-snippet: |
more_set_headers 'Access-Control-Allow-Origin:$origin';
ingress.kubernetes.io/proxy-buffering: "on"
ingress.kubernetes.io/proxy-buffer-size: "2048k"
ingress.kubernetes.io/server-snippet: |
chunked_transfer_encoding off;
location ((https|http):\/\/.*\/test-service\/images\/.*\/imageName.*) {
more_set_headers 'Cache-Control: public, max-age=14400';
}
spec:
rules:
- host: static-url-lab.lab.cdn.example.com
http:
paths:
- path: /test-service/(.*)
backend:
serviceName: test-service
servicePort: 80
However this does not seem to be working. When I curl a resource matching that pattern I get the default values back:
# Example curl - not exact
curl -v "https://static-url-lab.lab.cdn.example.com/test-service/intent/test/image_name" -o /dev/null 2>&1 grep -E "(Cache-Control: max|X-Cache)"
< Cache-Control: max-age=172800, public
As far as I can tell the regex should be matching, but no change is taking place, what am I missing?
Try something like this :
working for me
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($request_uri ~* \.(js|css|gif|jpe?g|png)) {
expires 1M;
add_header Cache-Control "public";
}
nginx.ingress.kubernetes.io/proxy-body-size: 50m
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"