Address null for ingress in EKS with classic LB - kubernetes

Question

when I create ingress it is created with no address and when I describe my ingress I see message
Failed build model due to WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
status code: 403, request id: 5423ee08-9a72-47fe-8389-3f50ce78b0e5
and when I check pod logs for aws loadbalancer controller, see the similar error
{"level":"error","ts":1674658664.611337,"logger":"controller-runtime.manager.controller.ingress","msg":"Reconciler error","name":"catch","namespace":"sa-backup","error":"WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: b4a791b1-f56b-4d4b-84b4-a7b6bc5ff8b9"}
I can confirm that classic load balancer is created fine and ingressRoutes are working. Just problem with ingress controller

Answer 1

Your AWS Load Balancer Controller needs access to the AWS API. The standard way to give API access to a pod in EKS is using IRSA (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) which allows the Pod to assume roles through OIDC in the cluster and a trust relationship with the AWS API. This trust relationship needs setup before IRSA will work (https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-enable-IAM.html).
The AWS IAM Role that your AWS LB Controller is trying to assume will need to have a Trust policy that allows the OIDC endpoint of your EKS cluster that references the Namespace and ServiceAccount name used by the pod.
Assuming you are in us-east-2 and the service account is in the aws-lbc namespace and named aws-load-balancer-controller, the ServiceAccount that the pod is using will need an annotation specifying which AWS IAM role to use:
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:<account number>:role/<iam role name>
name: aws-load-balancer-controller
namespace: aws-lbc
The trust policy on the AWS IAM role should look something like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account number>:oidc-provider/oidc.eks.us-east-2.amazonaws.com/id/<OIDC endpoint ID>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-2.amazonaws.com/id/<OIDC endpoint ID>:sub": "system:serviceaccount:aws-lbc:aws-load-balancer-controller"
}
}
}
]
}
The <OIDC endpoint ID> can be retrieved from the AWS console under your EKS cluster Overview -> Details -> OpenID Connect provider URL (it's the 32 character string after /id).