Keda Operator Access Denied for SQS - kubernetes

Question

I am using SQS scaler for Keda to scale a deployment. At present I am using operator based identityOwner for scaledObject. I have build an IAM role with following permissions and trust relationship :-
permission
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "KedaSQSReadPermission",
"Effect": "Allow",
"Action": "sqs:GetQueueAttributes",
"Resource": "arn:aws:sqs:<region-name>:<account-id>:<test-queue>"
},
{
"Sid": "KedaSQSListPermission",
"Effect": "Allow",
"Action": "sqs:ListQueues",
"Resource": ["*"]
}
]
}
trust relationship
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.<region-name>.amazonaws.com/id/<oidc-id>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.<region-name>.amazonaws.com/id/<oidc-id>:sub": "system:serviceaccount:keda:keda-operator"
}
}
}
]
}
This role is annotated in keda-operator service account in keda namespace as detailed below :-
{
"apiVersion": "v1",
"kind": "ServiceAccount",
"metadata": {
"annotations": {
"eks.amazonaws.com/role-arn": "arn:aws:iam::<account-id>:role/<role-name>",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/name\":\"keda-operator\",\"app.kubernetes.io/part-of\":\"keda-operator\",\"app.kubernetes.io/version\":\"2.7.0\"},\"name\":\"keda-operator\",\"namespace\":\"keda\"}}\n"
},
"creationTimestamp": "2022-07-14T13:49:40Z",
"labels": {
"app.kubernetes.io/name": "keda-operator",
"app.kubernetes.io/part-of": "keda-operator",
"app.kubernetes.io/version": "2.7.0"
},
"name": "keda-operator",
"namespace": "keda",
"resourceVersion": "174401872",
"uid": "a5862afb-fcd1-49e3-8ebb-6c81f4fd5014"
},
"secrets": [
{
"name": "keda-operator-token-wzqsz"
}
]
}
I have a deployment running in different namespace - test-keda. For scaling against queue length, I using following spec :-
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: scaled-object-test
namespace: test-keda
labels:
name: test-app # Required Name of the deployment we want to scale.
spec:
scaleTargetRef:
kind: Deployment
name: test-app
pollingInterval: 5
cooldownPeriod: 10
maxReplicaCount: 8
minReplicaCount: 0
triggers:
- type: aws-sqs-queue
metadata:
queueURL: https://sqs.<region-name>.amazonaws.com/<account-id>/<queue-name>
queueLength: "10"
awsRegion: "<region-name>"
identityOwner: operator
I have used official docs for SQS scaler for setting up above configurations. However, upon checking keda-opeator pod log, I find following error :-
error": "AccessDenied: Access to the resource https://sqs.<region-name>.amazonaws.com/ is denied.\n\tstatu │
│ s code: 403, request id: 87fd482d-89cf-54ef-af3e-13a25f4d9e23"
I tried using pod as identityOwner for my scaledobject, but seems to throw Access Denied, due to node-role trying to assume deployment IRSA role.
Am I missing something here ? Would appreciate any hint on where am I going wrong.
Thanks

Answer 1

Restarting keda deployments solved the issue. Run following commands :-
kubectl rollout restart deployment keda-metrics-apiserver -n keda
kubectl rollout restart deployment keda-operator -n keda