change in manifest file for ingress while upgrading k8s from 1.18 to 1.22 (v1beta1 to v1)using helm - kubernetes

I tried to make changes in menifest to run from k8s version 1.18 to 1.22 and below is how my manifest now look like
Older file (v1beta1)
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-api
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleRemoveById 933160
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Content-Type-Options: nosniff"
more_set_headers "Cache-Control: no-cache, no-store"
more_set_headers "X-XSS-Protection: 1; mode=block"
more_set_headers "Content-Security-Policy: default-src 'self';script-src 'self' 'unsafe-inline'; font-src 'self' fonts.gstatic.com fonts.gstatic.com data:;style-src 'self' fonts.googleapis.com fonts.gstatic.com 'unsafe-inline'; img-src 'self' data:; connect-src 'self' login.microsoftonline.com graph.microsoft.com https://{{.Values.instanceApiHost}} https://{{.Values.configurationApiHost}}; frame-src 'self' https://login.microsoftonline.com/;"
more_set_headers "X-RateLimit-Limit: 2500"
more_set_headers "X-Frame-Options: sameorigin"
more_clear_headers "server"
more_clear_headers "Server"
more_clear_headers "X-Powered-By"
more_clear_headers "x-powered-by"
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
tls:
- hosts:
- {{.Values.configurationApiHost}}
secretName: managedservices-configurationapi-tls
- hosts:
- {{.Values.instanceApiHost}}
secretName: managedservices-instanceapi-tls
- hosts:
- {{.Values.host}}
rules:
- host: {{.Values.configurationApiHost}}
http:
paths:
- path: /
backend:
serviceName: configuration-api
servicePort: 8080
- host: {{.Values.instanceApiHost}}
http:
paths:
- path: /
backend:
serviceName: instance-api
servicePort: 8080
...
New file (v1)
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
app.kubernetes.io/component: controller
name: nginx
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-api
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleRemoveById 933160
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Content-Type-Options: nosniff"
more_set_headers "Cache-Control: no-cache, no-store"
more_set_headers "X-XSS-Protection: 1; mode=block"
more_set_headers "Content-Security-Policy: default-src 'self';script-src 'self' 'unsafe-inline'; font-src 'self' fonts.gstatic.com fonts.gstatic.com data:;style-src 'self' fonts.googleapis.com fonts.gstatic.com 'unsafe-inline'; img-src 'self' data:; connect-src 'self' login.microsoftonline.com graph.microsoft.com https://{{.Values.instanceApiHost}} https://{{.Values.configurationApiHost}}; frame-src 'self' https://login.microsoftonline.com/;"
more_set_headers "X-RateLimit-Limit: 2500"
more_set_headers "X-Frame-Options: sameorigin"
more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains"
more_clear_headers "server"
more_clear_headers "Server"
more_clear_headers "X-Powered-By"
more_clear_headers "x-powered-by"
nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
spec:
kubernetes.io/ingress.class: nginx
tls:
- hosts:
- {{.Values.configurationApiHost}}
secretName: managedservices-configurationapi-tls
- hosts:
- {{.Values.instanceApiHost}}
secretName: managedservices-instanceapi-tls
- hosts:
- {{.Values.host}}
rules:
- host: {{.Values.configurationApiHost}}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: configuration-api
port:
number: 8080
- host: {{.Values.instanceApiHost}}
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: instance-api
port:
number: 8080
Earlier older file was working with k8s 1.18 but it is now updated to 1.22 and we need to change the ingress manifest accordingly.
Attached above is the newer one having v1 version.
I am doing it via Helm Upgrade and below is the snippet of Azure pipeline task:
- task: HelmDeploy#0
displayName: Helm deploy instance-api
inputs:
connectionType: Kubernetes Service Connection
kubernetesServiceConnection: ${{ parameters.kubernetesServiceEndpointName }}
namespace: ${{ parameters.kubernetesNamespace }}
command: upgrade
chartType: FilePath
chartPath: $(Pipeline.Workspace)/HelmCharts/instance-api
releaseName: instance-api
force: false
Getting below error
Error: UPGRADE FAILED: unable to build kubernetes objects from current release manifest: resource mapping not found for name: "ingress-api" namespace: "" from "": no matches for kind "Ingress" in version "networking.k8s.io/v1beta1"
Still it is picking up the older v1beta1 call and not working. please help in case there are more changes required ?
No clues....

Related

Example Nginx plus ingress for sticky sessions during canary deployment

I’m deploying 2 services to kubernetes pods which simply echo a version number; echo-v1 & echo-v2
Where echo-v2 is considered the canary deployment, I can demonstrate sticky sessions as canary weight is reconfigured from 0 to 100 using canary & canary-weight annotations.
2 ingresses are used:
The first routes to echo-v1 with a session cookie annotation.
The second routes to echo-v2 with canary true,canary weight and session cookie annotations.
The second ingress I can apply without impacting those sessions started on the first ingress and new sessions follow the canary weighting as expected.
However I’ve since learned that those annotations are for nginx community and won’t work with nginx plus.
How can I achieve the same using ingress(es) with nginx plus?
This is the ingress configuration that works for me using Nginx community vs Nginx plus.
Nginx community:
(coffee-v1 service)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/affinity: "cookie"
name: ingress-coffee
spec:
rules:
- http:
paths:
- path: /coffee
pathType: Exact
backend:
service:
name: coffee-v1
port:
number: 80
(coffee-v2 'canary' service)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "100"
name: ingress-coffee-canary
spec:
rules:
- http:
paths:
- path: /coffee
pathType: Exact
backend:
service:
name: coffee-v2
port:
number: 80
Nginx plus:
(coffee-v1 & coffee-v2 as type 'virtualserver' not 'ingress')
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: cafe
spec:
host: cloudbees-training.group.net
tls:
secret: cloudbees-trn.aks.group.net-tls
upstreams:
- name: coffee-v1
service: coffee-v1-svc
port: 80
sessionCookie:
enable: true
name: srv_id_v1
path: /coffee
expires: 2h
- name: coffee-v2
service: coffee-v2-svc
port: 80
sessionCookie:
enable: true
name: srv_id_v2
path: /coffee
expires: 2h
routes:
- path: /coffee
matches:
- conditions:
- cookie: srv_id_v1
value: ~*
action:
pass: coffee-v1
- conditions:
- cookie: srv_id_v2
value: ~*
action:
pass: coffee-v2
# 3 options to handle new session below:
#
# 1) All new sessions to v1:
# action:
# pass: coffee-v1
#
# 2) All new sessions to v2:
# action:
# pass: coffee-v2
#
# 3) Split new sessions by weight
# Note: 0,100 / 100,0 weightings causes sessions
# to drop for the 0 weighted service:
# splits:
# - weight: 50
# action:
# pass: coffee-v1
# - weight: 50
# action:
# pass: coffee-v2

access forbidden by rule, on ingress nginx log

which ingress rule is blocking the request coming in.
app : based on drupal.
any suggestions would help.
ingress log :
2022/08/11 10:00:59 [error] 20516#20516: *159406145 access forbidden by rule, client: 2a02:9b0:3d:54a2:40bf:a951:e203:79a5, server: example.com, request: "GET / HTTP/1.1", host: "example.com"
{"time": "2022-08-11T10:00:59+00:00", "remote_addr": "-", "x_forward_for": "2a02:9b0:3d:54a2:40bf:a951:e203:79a5, 172.70.156.137, 2a02:9b0:3d:54a2:40bf:a951:e203:79a5", "request_id": "d8d07bd09c84b802af91a60adbf46a73", "remote_user": "-", "bytes_sent": 583, "request_time": 0.000, "status": 403, "vhost": "example.com", "request_proto": "HTTP/1.1", "path": "/", "request_query": "-", "request_length": 647, "duration": 0.000,"method": "GET", "http_referrer": "-", "http_user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 15_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Mobile/15E148 Safari/604.1", "auth_apikey": "-", "auth_authorization": "-", "auth_X-Api-caps": "-", "auth_uid": "-"}
ingress.yaml
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
meta.helm.sh/release-name: example-production
meta.helm.sh/release-namespace: example
nginx.ingress.kubernetes.io/configuration-snippet: |
more_clear_headers "Server";
more_set_headers "X-Content-Type-Options: nosniff";
more_set_headers "X-XSS-Protection: 1";
more_set_headers "X-Request-Id: $request_id";
more_set_headers "X-Router-Id: example-portal-anonymous";
more_set_headers "Strict-Transport-Security: max-age=15768000";
more_set_headers "Content-Security-Policy: frame-ancestors 'self'";
nginx.ingress.kubernetes.io/cors-allow-headers: X-Forwarded-For, Authorization
nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, PUT, PATCH, DELETE,
OPTIONS
nginx.ingress.kubernetes.io/cors-allow-origin: example.com
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/enable-modsecurity: "true"
nginx.ingress.kubernetes.io/enable-owasp-core-rules: "false"
nginx.ingress.kubernetes.io/limit-rpm: "0"
nginx.ingress.kubernetes.io/modsecurity-transaction-id: $request_id
nginx.ingress.kubernetes.io/proxy-buffer-size: 20k
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
creationTimestamp: "2022-08-10T11:19:21Z"
generation: 1
labels:
app.kubernetes.io/managed-by: example
name: portal-anonymous
namespace: example
resourceVersion: "523173759"
selfLink: /apis/extensions/v1beta1/namespaces/example/ingresses/portal-anonymous
uid: 9cdb29b2-a463-4851-b4c4-b5a58be28580
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: portal
servicePort: 80
path: /
pathType: Prefix
status:
loadBalancer: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""
I suspect the problem is actually revealed in your config:
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
You are allowing all IPv4 IPs with this range, but in your log, you have this:
"x_forward_for": "2a02:9b0:3d:54a2:40bf:a951:e203:79a5, 172.70.156.137, 2a02:9b0:3d:54a2:40bf:a951:e203:79a5"
So this means you're using IPv6 as well, so you might want to add ::/0 to the whitelist-source-range to allow all IPv6 as well as all IPv4 ranges. OR just omit the annotation entirely if you want to allow anything in.

error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): eks v1.22

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-west-1:0987654322:certificate/ee21bffb-ba4a-45a6-95a6-00551c1cfa32"
alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
alb.ingress.kubernetes.io/healthy-threshold-count: "2"
alb.ingress.kubernetes.io/listen-ports: "[{"HTTPS":443}, {"HTTP":80}]"
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-1-2017-01
alb.ingress.kubernetes.io/ssl-redirect: "443"
alb.ingress.kubernetes.io/success-codes: "200"
alb.ingress.kubernetes.io/target-group-attributes: "stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=60"
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
kubernetes.io/ingress.class: alb
name: alb-alb
namespace: deploy
spec:
rules:
-
http:
paths:
-
backend:
serviceName: service-svc
servicePort: 80
path: /*
Try:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
...
name: alb-alb
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service-svc
port: # <-- Correction to your spec starts here
number: 80

Kubernetes ingres.yml unknown field "service.port.number" in io.k8s.api.networking.v1

I'm running into issues with my ingress.yml. Getting this error:
error: error validating "ingress.yml": error validating data: [ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "service.name" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[0].backend): unknown field "service.port.number" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[1].backend): unknown field "service.name" in io.k8s.api.networking.v1.IngressBackend, ValidationError(Ingress.spec.rules[0].http.paths[1].backend): unknown field "service.port.number" in io.k8s.api.networking.v1.IngressBackend];
Based on this documentation serviceName changed to service.name and servicePort changed to service.port and so on. Here is the ingress file:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: bitwarden
namespace: bitwarden
labels:
app: bitwarden
annotations:
# the name of the nginx-ingress-controller class
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "Request-Id: $req_id";
nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/limit-connections: "25"
nginx.ingress.kubernetes.io/limit-rps: "15"
nginx.ingress.kubernetes.io/proxy-body-size: 1024m
nginx.ingress.kubernetes.io/proxy-connect-timeout: "10"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
tls:
- hosts:
- kryskanbit.ddns.net
secretName: bitwarden-tls
rules:
- host: kryskanbit.ddns.net
http:
paths:
- path: /
pathType: Prefix
backend:
service.name: bitwarden
service.port.number: 80
- path: /notifications/hub
pathType: Exact
backend:
service.name: bitwarden
service.port.number: 3012
Kubernetes has a special treatment for YAML format, so basically, you just need to extend the properties individually the service.name and service.port.number:
...
paths:
- path: /
pathType: Prefix
backend:
service:
name: bitwarden
port:
number: 80
- path: /notifications/hub
pathType: Exact
backend:
service:
name: bitwarden
port:
number: 3012
It should be as mentioned below:
backend:
serviceName: bitwarden
servicePort: 80

k8s ingress setup - invalid type for io.k8s.api.networking.v1.ServiceBackendPort.number: got "string", expected

I'm trying to setup an ingress in kubernetes for my frontend, the code is as below.
datahub-frontend:
enabled: true
image:
repository: linkedin/datahub-frontend-react
tag: "v0.8.31"
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:601628467906:certificate/xxxxxxxxxxxxx
alb.ingress.kubernetes.io/inbound-cidrs: 0.0.0.0/0
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
hosts:
- host: xxxxx.com
redirectPaths:
- path: /*
name: ssl-redirect
port: use-annotation
paths:
- /*
This is giving me the following error.
Error: UPGRADE FAILED: error validating "": error validating data: ValidationError(Ingress.spec.rules[0].http.paths[0].backend.service.port.number): invalid type for io.k8s.api.networking.v1.ServiceBackendPort.number: got "string", expected "integer"
seems like the ingress definition you posted is incorrect. It is missing the required spec and rules according to kubernetes doc.
This contains an example ingress for ssl-redirect
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: default
name: ingress
annotations:
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:xxxx:certificate/xxxxxx
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
spec:
ingressClassName: alb
rules:
- http:
paths:
- path: /users/*
pathType: ImplementationSpecific
backend:
service:
name: user-service
port:
number: 80
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: default-service
port:
number: 80
I see that you are using actions annotation, you need to specify Ingress.spec.rules[0].http.paths[0].backend.service.port.name with use-annotation. Make sure Ingress.spec.rules[0].http.paths[0].backend.service.port.number is not used.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ingress
annotations:
...
...
alb.ingress.kubernetes.io/actions.my-rule: '{"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"200","messageBody":"hello world"}}'
spec:
rules:
- http:
paths:
- path: /*
pathType: ImplementationSpecific
backend:
service:
name: my-rule
port:
name: use-annotation