I have setup EKS cluster and have configured the OIDC. I have then created the IAM role with the below trust relationship
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/55A76A4197643C67E88FD47738722195"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.ap-south-1.amazonaws.com/id/55A76A4197643C67E88FD47738722195:sub": "system:serviceaccount:default:aws-test",
"oidc.eks.ap-south-1.amazonaws.com/id/55A76A4197643C67E88FD47738722195:aud": "sts.amazonaws.com"
}
}
}
]
}
When I use the service account in the pod, and from within the Pod, when I run the aws s3 ls, I am getting the below error:
Could not connect to the endpoint URL: "https://sts.ap-south-1.amazonaws.com/"
How to troubleshoot this error
I have installed external-dns using bitnami helm chart with version 6.10.2.
I have created IAM policy to give permissions to list Route53 hosted zones.
IAM policy
{
"Statement": [
{
"Action": [
"route53:ListHostedZones",
"route53:ListResourceRecordSets"
],
"Effect": "Allow",
"Resource": [
"*"
]
},
{
"Action": [
"route53:ChangeResourceRecordSets"
],
"Effect": "Allow",
"Resource": [
"*"
]
}
],
"Version": "2012-10-17"
}
IAM policy is also mapped to serviceaccount
eksctl get iamserviceaccount --cluster=testcluster --namespace add-ons
NAMESPACE NAME ROLE ARN
add-ons external-dns arn:aws:iam::442355839237:role/eksctl-testcluster-addon-iamserviceaccount-ad-Role1-6AV1JQ2NPCMO
OIDC is already enabled
eksctl utils associate-iam-oidc-provider --cluster=testcluster
2022-10-17 14:31:13 [ℹ] IAM Open ID Connect provider is already associated with cluster "testcluster" in "us-west-2"
However I still see the error in external-dns pod that it is unable to list the hosted zones.
7T21:24:03Z" level=error msg="records retrieval failed: failed to list hosted zones: AccessDenied: User: arn:aws:sts::442355839237:assumed-role/workers-eks-node-group-20221013233723853600000005/i-0a1b8a914bcff1436 is not authorized to perform: route53:ListHostedZones because no identity-based policy allows the route53:ListHostedZones action\n\tstatus code: 403, request id: a8f86d66-d7af-4bd0-975c-6f99d1134d50"
Strangely..it is also showing "User: arn:aws:sts::442355839237:assumed-role/workers-eks-node-group-20221013233723853600000005/i-0a1b8a914bcff1436" workers-eks-node-group in the role name instead of the role arn:aws:iam::442355839237:role/eksctl-testcluster-addon-iamserviceaccount-ad-Role1-6AV1JQ2NPCMO which maps the service account with Route53 permissions.
Any pointers on why it is failing?arn:aws:iam::442355839237:role/eksctl-testcluster-addon-iamserviceaccount-ad-Role1-6AV1JQ2NPCMO
I have fixed this..
changeresourcerecordsets need to be specific with arns of hosted zones instead of "*"
https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md#iam-policy
data "aws_iam_policy_document" "external_dns" {
count = var.create_role && var.attach_external_dns_policy ? 1 : 0
statement {
actions = ["route53:ChangeResourceRecordSets"]
resources = var.external_dns_hosted_zone_arns
}
statement {
actions = [
"route53:ListHostedZones",
"route53:ListResourceRecordSets",
]
resources = ["*"]
}
}
I am trying to set up cross account Postgres RDS IAM authentication. My use case is a python code that is containerized and executed by AWS Batch on the top of the ECS engine connects to the Postgres RDS in another AWS account. I tried to follow the route (single role in the account where DB connection is originated) that is described here but the connection fails with:
2020-06-12 19:41:10,363 - root - ERROR - Error reading data from data DB: FATAL: PAM authentication failed for user "db_user"
I also found this one and tried to set up something similar (a role per respective account but no EC2 instance as a connection source). Unfortunately it failed with the same error. Does anyone know any other AWS documentation that might match my use case?
I managed to sort it out with help of AWS support folks. These are the actions that I had to do:
Add the following policy to the IAM role applied to AWS Batch job (AWS account A):
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::ACCOUNT_B_ID:role/ecsTaskExecutionRole"
}
}
With a following trust policy:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Add the following IAM role within the AWS account that is used for RDS hosting (AWS account B):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rds-db:connect"
],
"Resource": [
"arn:aws:rds-db:<region>:ACCOUNT_B_ID:dbuser:{rds-resource-id}/{batch-user}"
]
}
]
}
With a following trust policy:
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::ACCOUNT_A_ID:root",
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Update the code that is executed within the AWS Batch container:
sts_client = boto3.client('sts')
assumed_role_object=sts_client.assume_role(
RoleArn="arn:aws:iam::ACCOUNT_B_ID:role/ROLE_TO_BE_ASSUMED",
RoleSessionName="AssumeRoleSession1"
)
credentials=assumed_role_object['Credentials']
client = boto3.client(
'rds',
aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'],
region_name=REGION )
#client = boto3.client('rds')
token = client.generate_db_auth_token(DBHostname=ENDPOINT, Port=PORT, DBUsername=USR, Region=REGION)
I have an app I am trying to move to a new k8s cluster, having a permissions issue when trying to ConfirmSubscription:
"sns confirmation failed. Reason: AuthorizationError: User: arn:aws:sts::-:assumed-role/-/- is not authorized to perform:
SNS:ConfirmSubscription on resource: arn:aws:sns:-:-:topicname
status code: 403, request id: 000d2844-3a3d-5544-922a-7d9e3db07a16"
The app was able to execute a confirm subscription in the old cluster, so I assume it's an IAM issue, but the role policy it's assuming is:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sns:ConfirmSubscription",
"sns:Subscribe"
],
"Effect": "Allow",
"Resource": [
"arn:aws:sns:::*"
]
}
]
}
I haven't been able to diagnose where the IAM issue is.
I was able to get this working by fully qualifying the sns topic to which I wanted to confirm subscription:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"sns:ConfirmSubscription",
"sns:Subscribe"
],
"Effect": "Allow",
"Resource": [
"arn:aws:sns:us-east-1:000000000:full-topic-name-no-wildcard"
]
}
]
}
My team has a pipeline which runs under an execution IAM role. We want to deploy code to AWS through CloudFormation or the CDK.
In the past, we would upload some artifacts to S3 buckets before creating/updating our CloudFormation stack, using the execution IAM role.
We recently switched to the CDK, and are trying to get as much automated with using CDK Deploy as possible, but are running into a lot of permission items we need to add which we didn't have prior (for instance, cloudformation:GetTemplate).
We don't want to just grant * (we want to follow least privilege) but I can't find any clear documented list.
Is there a standard list of permissions that CDK Deploy relies on? Are there any "nice to have's" beyond a standard list?
The CDK v2 now brings and assumes its own roles. No more manual permission management required. You only need to grant permission to assume the cdk roles:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::*:role/cdk-*"
]
}
]
}
These roles are created via cdk bootstrap, which then of course requires the permission to create the roles and policies. After the bootstrapping though, this no longer is required. So you could run this manually with a privileged role.
Apparently CDK proceeds if any of the cdk roles cannot be assumed. So it's still possible to manually manage a CDK policy as below, but it might now requires additional permissions.
Be aware, the CFN role has the Administrator policy attached.
Previous answer for CDK v1:
I'm using below policy to deploy CDK apps. Besides CFN full access and S3 full access to the CDK staging bucket, it grants permission to do everything through CloudFormation.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"cloudformation:*"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Condition": {
"ForAnyValue:StringEquals": {
"aws:CalledVia": [
"cloudformation.amazonaws.com"
]
}
},
"Action": "*",
"Resource": "*",
"Effect": "Allow"
},
{
"Action": "s3:*",
"Resource": "arn:aws:s3:::cdktoolkit-stagingbucket-*",
"Effect": "Allow"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm::*:parameter/cdk-bootstrap/*"
}
]
}
You might want to add some explicit denies for things you don't want to allow.
Also, be aware that above condition does not mean the principal is limited to things possible with CloudFormation. A potential attack vector would be to create a custom CFN resource, backed by a Lambda function. When creating resources through that custom resource you then could do anything in the Lambda, because it is triggered via CloudFormation.
When you use lookups (those are the .fromXxx(...) methods), the CDK will make read/list requests to the related service at runtime - while the CDK synth is running, not the CloudFormation deploy. Which permissions you need, of course, depends on the lookups you have in your code. For example, if you would have a Vpc.fromLookup() you should allow the action ec2:DescribeVpcs. Of course you could attach the ReadOnlyAccess policy, if you have no concerns about accessing sensitive content.
Since I couldn't find any documentation anywhere I had to do some trial and error to get this to work.
Apart from the permissions you need to create the actual resources you define in your stack, you need to give the following:
cloudformation:DescribeStacks
cloudformation:CreateChangeSet
cloudformation:DescribeChangeSet
cloudformation:ExecuteChangeSet
cloudformation:DescribeStackEvents
cloudformation:DeleteChangeSet
cloudformation:GetTemplate
To the stack ARN you are creating, as well as the bootstrap stack:
arn:aws:cloudformation:${AWS::Region}:${AWS::AccountId}:stack/CDKToolkit/*
You also need s3 permissions to the bucket that the boostrap added (otherwise you get that dreaded Forbidden: null error):
s3:*Object
s3:ListBucket
s3:GetBucketLocation
to
arn:aws:s3:::cdktoolkit-stagingbucket-*
CDK has two phases the bootstrap and the synth/deploy.
In the case of bootstrap the IAM role or profile used must have the following policy permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "StsAccess",
"Effect": "Allow",
"Action": [
"sts:AssumeRole",
"iam:*Role*"
],
"Resource": [
"arn:aws:iam::${AWS_ACCOUNT_ID}:role/cdk-*"
]
},
{
"Action": [
"cloudformation:*"
],
"Resource": [
"arn:aws:cloudformation:${AWS_REGION}:${AWS_ACCOUNT_ID}:stack/CDKToolkit/*"
],
"Effect": "Allow"
},
{
"Sid": "S3Access",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"*"
]
},
{
"Sid": "ECRAccess",
"Effect": "Allow",
"Action": [
"ecr:SetRepositoryPolicy",
"ecr:GetLifecyclePolicy",
"ecr:PutImageScanningConfiguration",
"ecr:DescribeRepositories",
"ecr:CreateRepository",
"ecr:DeleteRepository"
],
"Resource": [
"arn:aws:ecr:${AWS_REGION}:${AWS_ACCOUNT_ID}:repository/cdk-*"
]
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter*",
"ssm:PutParameter*",
"ssm:DeleteParameter*"
],
"Resource": "arn:aws:ssm:${AWS_REGION}:${AWS_ACCOUNT_ID}:parameter/cdk-bootstrap/*"
}
]
}
While in the case of deployment, the role or profile must have the following as mandatory permission:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"sts:AssumeRole"
],
"Resource": [
"arn:aws:iam::*:role/cdk-*"
]
}
]
}
Plus with all the permissions needed for the infrastructure you're deploying.
The thing I can recommend is to use two different roles so that you have more security, and in case you're using GitHub workflow to take advantage of the OpenIdConnect.
The bootstrap policy could be improved by restricting permissions but the documentation is lacking so I do not delve into specific aspects (example s3)
We also needed to add below permissions as well.
ssm:PutParameter
ecr:SetRepositoryPolicy
ecr:GetLifecyclePolicy
ecr:PutImageScanningConfiguration
ssm:GetParameters
ecr:DescribeRepositories