GKE BackendConfig not working with customRequestHeaders - kubernetes

Question

I have a nodejs application running on Google Kubernetes Engine (v1.20.8-gke.900)
I want to add custom header to get client's Region and lat long so I refer to this article and this one also and created below kubernetes config file, but when I am printing the header I am not getting any custom header.
#k8s.yaml
apiVersion: v1
kind: Namespace
metadata:
name: my-app-ns-prod
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: npm-app-deployment
namespace: my-app-ns-prod
labels:
app: npm-app-deployment
tier: backend
spec:
template:
metadata:
name: npm-app-pod
namespace: my-app-ns-prod
labels:
app: npm-app-pod
tier: backend
spec:
containers:
- name: my-app-container
image: us.gcr.io/img/my-app:latest
ports:
- containerPort: 3000
protocol: TCP
envFrom:
- secretRef:
name: npm-app-secret
- configMapRef:
name: npm-app-configmap
imagePullPolicy: Always
imagePullSecrets:
- name: gcr-regcred
replicas: 3
minReadySeconds: 30
selector:
matchLabels:
app: npm-app-pod
tier: backend
---
apiVersion: v1
kind: Service
metadata:
name: npm-app-service
namespace: my-app-ns-prod
annotations:
cloud.google.com/backend-config: '{"ports": {"80":"npm-app-backendconfig"}}'
cloud.google.com/neg: '{"ingress": true}'
spec:
selector:
app: npm-app-pod
tier: backend
ports:
- name: http
protocol: TCP
port: 80
targetPort: 3000
- name: https
protocol: TCP
port: 443
targetPort: 3000
type: LoadBalancer
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: npm-app-backendconfig
namespace: my-app-ns-prod
spec:
customRequestHeaders:
headers:
- "HTTP-X-Client-CityLatLong:{client_city_lat_long}"
- "HTTP-X-Client-Region:{client_region}"
- "HTTP-X-Client-Region-SubDivision:{client_region_subdivision}"
- "HTTP-X-Client-City:{client_city}"
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
rules:
- host: app.example.com
http:
paths:
- path: /api/v1
pathType: Prefix
backend:
service:
name: npm-app-service
port:
number: 80
---
apiVersion: v1
kind: ConfigMap
metadata:
name: npm-app-configmap
namespace: my-app-ns-prod
data:
APP_ID: "My App"
PORT: "3000"
---
apiVersion: v1
kind: Secret
metadata:
name: npm-app-secret
namespace: my-app-ns-prod
type: Opaque
data:
MONGO_CONNECTION_URI: ""
SESSION_SECRET: ""

Answer 1

Actually the issue was with Ingress Controller, I missed to defined "cloud.google.com/backend-config". Once I had defined that I was able to get the custom header. Also I switched to GKE Ingress controller (gce) from nginx. But the same things works with nginx also
This is how my final ingress looks like.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
annotations:
cloud.google.com/backend-config: '{"default": "npm-app-backendconfig"}'
kubernetes.io/ingress.class: "gce"
spec:
...
...
Reference: User-defined request headers :