Access denied to S3 when using COPY command with IAM role - amazon-redshift

Question

I have the following copy command:
copy ink.contact_trace_records
from 's3://mybucket/mykey'
iam_role 'arn:aws:iam::accountnum:role/rolename'
json 'auto';
Where the role has the full access to S3 and full access to RS policies (I know this is not a good idea, but I'm just losing it here :) ). The cluster has the role attached. The cluster has VPC enhanced routing. I get the following:
[2019-05-28 14:07:34] [XX000][500310] [Amazon](500310) Invalid operation: S3ServiceException:Access Denied,Status 403,Error AccessDenied,Rid 370F75618922AFC0,ExtRid dp2mcnlofFzt4dz00Lcm188/+ta3OEKVpuFjnZSYsC0pJPMiULk7I6spOpiZwXc04VVRdxizIj4=,CanRetry 1
[2019-05-28 14:07:34] Details:
[2019-05-28 14:07:34] -----------------------------------------------
[2019-05-28 14:07:34] error: S3ServiceException:Access Denied,Status 403,Error AccessDenied,Rid 370F75618922AFC0,ExtRid dp2mcnlofFzt4dz00Lcm188/+ta3OEKVpuFjnZSYsC0pJPMiULk7I6spOpiZwXc04VVRdxizIj4=,CanRetry 1
[2019-05-28 14:07:34] code: 8001
[2019-05-28 14:07:34] context: S3 key being read : s3://redacted
[2019-05-28 14:07:34] query: 7794423
[2019-05-28 14:07:34] location: table_s3_scanner.cpp:372
[2019-05-28 14:07:34] process: query3_986_7794423 [pid=13695]
[2019-05-28 14:07:34] -----------------------------------------------;
What am I missing here? The cluster has full access to S3, it is not even in a custom VPC, it is in the default one. Thoughts?

Answer 1

Check the object owner. The object is likely owned by another account. This is the reason for S3 403s in many situations where objects cannot be accessed by a role or account that has full permissions. The typical indicator is that you can list the object(s) but cannot get or head them.
In the following example I'm trying to access a Redshift audit log from a 2nd account. The account 012345600000 is granted access to the bucket owned by 999999999999 using the following policy:
{"Version": "2012-10-17",
"Statement": [
{"Action": [
"s3:Get*",
"s3:ListBucket*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-audit-logs",
"arn:aws:s3:::my-audit-logs/*"
],
"Principal": {
"AWS": [
"arn:aws:iam::012345600000:root"
]}}]
}
Now I try to list (s3 ls) then copy (s3 cp) a single log object:
aws --profile 012345600000 s3 ls s3://my-audit-logs/AWSLogs/999999999999/redshift/us-west-2/2019/05/25/
# 2019-05-28 14:49:46 376 …connectionlog_2019-05-25T19:03.gz
aws --profile 012345600000 s3 cp s3://my-audit-logs/AWSLogs/999999999999/redshift/us-west-2/2019/05/25/…connectionlog_2019-05-25T19:03.gz ~/Downloads/…connectionlog_2019-05-25T19:03.gz
# fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
Then I check the object ownership from the account that owns the bucket.
aws --profile 999999999999 s3api get-object-acl --bucket my-audit-logs --key AWSLogs/999999999999/redshift/us-west-2/2019/05/25/…connectionlog_2019-05-25T19:03.gz
# "Owner": {
# "DisplayName": "aws-cm-prod-auditlogs-uswest2", ## Not my account!
# "ID": "b2b456ce30a967fb1877b3c9594773ae0275fee248e3ebdbff43d66907b89144"
# },
I then copy the object in the same bucket with --acl bucket-owner-full-control. This makes me the owner of the new object. Note the changed SharedLogs/ prefix.
aws --profile 999999999999 s3 cp \
s3://my-audit-logs/AWSLogs/999999999999/redshift/us-west-2/2019/05/25/…connectionlog_2019-05-25T19:03.gz \
s3://my-audit-logs/SharedLogs/999999999999/redshift/us-west-2/2019/05/25/…connectionlog_2019-05-25T19:03.gz \
--acl bucket-owner-full-control
Now I can download (or load to Redshift!) the new object that is shared from the same bucket.
aws --profile 012345600000 s3 cp s3://my-audit-logs/SharedLogs/999999999999/redshift/us-west-2/2019/05/25/…connectionlog_2019-05-25T19:03.gz ~/Downloads/…connectionlog_2019-05-25T19:03.gz
# download: …