403-Forbidden at ingress controller in AKS cluster - kubernetes

Question

I am trying to configure an AKS cluster and assign a DNS(say abc.com) that is bought from GoDaddy. I have configured following components:
Azure DNS zone for abc.com
Update Name-Servers generated by Azure DNS zone at GoDaddy DNS settings
Add 'DNS zone contributor' role to K8s cluster agentpool.
Setup Ingress controller, External DNS, Cert-manager, ClusterIssuer
Add Ingress route for my application
Deploy my application to setup it's Pod and service on cluster.
When I try to hit the API(https://dev.abc.com/backend/usermanager/api/user) from postman I am getting a 403-Forbidden error.
When I open a shell from ingress-controller container within the cluster and curl above API it gives same 403-Forbidden error with following details:
bash-5.1$ curl -v -d '{"name": "John Doe", "mobileNumber": "1234554321"}' -H 'Content-Type:
application/json' https://dev.abc.com/backend/usermanager/api/user
* Trying 21.81.66.166:443...
* Connected to dev.abc.com (21.81.66.166) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=dev.abc.com
* start date: Jul 20 03:29:46 2022 GMT
* expire date: Oct 18 03:29:45 2022 GMT
* subjectAltName: host "dev.abc.com" matched cert's "dev.abc.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: POST]
* h2h3 [:path: /backend/usermanager/api/user]
* h2h3 [:scheme: https]
* h2h3 [:authority: dev.abc.com]
* h2h3 [user-agent: curl/7.83.1]
* h2h3 [accept: */*]
* h2h3 [content-type: application/json]
* h2h3 [content-length: 50]
* Using Stream ID: 1 (easy handle 0x64161d71800)
> POST /backend/usermanager/api/user HTTP/2
> Host: dev.abc.com
> user-agent: curl/7.83.1
> accept: */*
> content-type: application/json
> content-length: 50
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
< HTTP/2 403
< date: Fri, 22 Jul 2022 04:40:59 GMT
< content-length: 0
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< strict-transport-security: max-age=15724800; includeSubDomains
< x-frame-options: DENY
<
* Connection #0 to host dev.abc.com left intact
bash-5.1$ * We are completely uploaded and fine
< HTTP/2 403
< date: Fri, 22 Jul 2022 04:40:59 GMT
< content-length: 0
< vary: Origin
< vary: Access-Control-Request-Method
< vary: Access-Control-Request-Headers
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< cache-control: no-cache, no-store, max-age=0, must-revalidate
< pragma: no-cache
< expires: 0
< strict-transport-security: max-age=15724800; includeSubDomains
< x-frame-options: DENY
When I curl the above endpoint directly with the service name as host within a shell from ingress-controller container, it works properly:
bash-5.1$ curl -v -d '{"name": "John Doe", "mobileNumber": "1234554321"}' -H 'Content-Type: application/json' http://usermanager:8082/api/user
* Trying 10.0.236.14:8082...
* Connected to usermanager (10.0.236.14) port 8082 (#0)
> POST /api/user HTTP/1.1
> Host: usermanager:8082
> User-Agent: curl/7.83.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 50
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Vary: Origin
< Vary: Access-Control-Request-Method
< Vary: Access-Control-Request-Headers
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Content-Type: application/json
< Transfer-Encoding: chunked
< Date: Fri, 22 Jul 2022 03:58:33 GMT
Following is my ingress route config:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: abc-ingress
namespace: k8sns-abc-dev
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt
spec:
tls:
- hosts:
- dev.abc.com
secretName: tls-secret
rules:
- host: dev.abc.com
http:
paths:
- path: /backend/usermanager
pathType: Prefix
backend:
service:
name: usermanager
port:
number: 8082
Can you please help me identify and fix why am I getting 403-Forbidden error when making an API request through DNS host?

Answer 1

When you are hitting the path https://dev.abc.com/backend/usermanager/api/user
and when you are trying the API inside of container, the path is api/user
Inside your ingress, there is no path re-write config.
So i think your service will be getting the path backend/usermanager/api/user
However, 403 is for unauth so make sure there is no another path logic check or so.