I have successfully created a role with policy attached to that role which allows required actions on the bucket. Policy document is:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "s3import",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/*"
]
}
]
}
And then i attached this role to my RDS instance with feature s3Import.
This is the command i ran.
SELECT aws_s3.table_import_from_s3(
'table name',
'',
'DELIMITER ''|''',
aws_commons.create_s3_uri(
'bucket-name',
'file.csv',
'region')
);
I am getting this error:
SQL Error [XX000]: ERROR: HTTP 404. Requested file does not exist.
Is anything missing here ?
Based on the comments.
Based on the error message provided, the issue was not due to access deny to S3, but rather due to wrong file name used in create_s3_uri.
The solution was to use the correct file name.
Related
I have the problem with running index managment policy for new indices. I get following error on "set number_of_replicas" step:
{
"cause": "no permissions for [indices:admin/settings/update] and associated roles [index_management_full_access, own_index, security_rest_api_access]",
"message": "Failed to set number_of_replicas to 2 [index=sample.name-2022.10.22]"
}
The indices are created by logstash with "sample.name-YYYY.MM.DD" name template, so in the index policy I have "sample.name-*" index pattern.
My policy:
{
"policy_id": "sample.name-*",
"description": "sample.name-* policy ",
"schema_version": 16,
"error_notification": null,
"default_state": "set replicas",
"states": [
{
"name": "set replicas",
"actions": [
{
"replica_count": {
"number_of_replicas": 2
}
}
]
],
"ism_template": [
{
"index_patterns": [
"sample.name-*"
],
"priority": 1
}
]
}
I don't understand the reason of this error.
Am I doing something wrong?
Retry of the policy doesn't work.
The policy works only if I manually reassign it to index by Dashboards or API.
Opensearch version: 2.3.0
First time I created the policy using API under custom internal user with mapped “security_rest_api_access” security role only.
So I added all_access rights to my internal user and re-created policy and it works!
Seems that the policy runs under my internal user, which created it
I am trying to upload a file to a cross account s3 bucket which has enabled kms key.
below is the code:
InstanceProfileCredentialsProvider iamCredentials = new InstanceProfileCredentialsProvider();
AmazonS3 s3client = new AmazonS3Client(iamCredentials);
FileInputStream stream = new FileInputStream("/home/tomcat/Test.txt");
ObjectMetadata objectMetadata = new ObjectMetadata();
int bytesAvailable = stream.available();
byte[] fileBytes = new byte[bytesAvailable];
Long contentLength = Long.valueOf(fileBytes.length);
objectMetadata.setContentLength(contentLength);
objectMetadata.setSSEAlgorithm(SSEAlgorithm.KMS.getAlgorithm());
PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName,"/Test/Test.txt",stream, objectMetadata).withCannedAcl(CannedAccessControlList.BucketOwnerFullControl);
putObjectRequest.withSSEAwsKeyManagementParams(new SSEAwsKeyManagementParams(kmsKeyId));
s3client.putObject(putObjectRequest);
Here I am using iam role to connect to external s3 bucket.
iam policy:
arn:aws:iam::xxxxxxxxxxxxxx:role/role1 (xxxxxxxxxxxxxx is the local account id)
----------------------------------------
role1_kms_access_policy
-----------------------
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"kms:EnableKeyRotation",
"kms:EnableKey",
"kms:Decrypt",
"kms:ListKeyPolicies",
"kms:UntagResource",
"kms:ListRetirableGrants",
"kms:GetKeyPolicy",
"kms:GenerateDataKeyWithoutPlaintext",
"kms:ListResourceTags",
"kms:ReEncryptFrom",
"kms:ListGrants",
"kms:GetParametersForImport",
"kms:DescribeCustomKeyStores",
"kms:ListKeys",
"kms:TagResource",
"kms:GetKeyRotationStatus",
"kms:Encrypt",
"kms:ScheduleKeyDeletion",
"kms:ListAliases",
"kms:GenerateDataKey",
"kms:ReEncryptTo",
"kms:DescribeKey",
"kms:ConnectCustomKeyStore"
],
"Resource": "*"
}
]
}
external bucket kms policy:
{
"Effect": "Allow",
"Principal": {
enter code here
"AWS": "arn:aws:iam::xxxxxxxxxxxxxx:role/role1"
},
"Action": [
"kms:Decrypt",
"kms:DescribeKey",
"kms:Encrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*"
],
"Resource": "*"
}
When running the code if i pass kms key of external bucket to variable kmsKeyId then exception is occurring:
com.amazonaws.services.s3.model.AmazonS3Exception: Key 'arn:aws:kms:us-east-1:xxxxxxxxxxxxxx:key/{kmsKeyId}' (here kmskey id is the value of passed km key) does not exist (Service: Amazon S3; Status Code: 400; Error Code: KMS.NotFoundException; Request ID: BQ3VWZK5VC88GXV1; S3 Extended Request ID: csDK/X8MjPHNuV4NrziYoUPBbqZG+Jp269IgBXFTnPQYxjJBgLailWtY7F0JjDLsNyHNO60xeVE=), S3 Extended Request ID: csDK/X8MjPHNuV4NrziYoUPBbqZG+Jp269IgBXFTnPQYxjJBgLailWtY7F0JjDLsNyHNO60xeVE=
And if we pass whole arn to kmskeyId then it shows access denied exception.
Right now, it's impossible to have assigned user assigned identities on arm templates (and terraform) on cluster creation. I already tried a lot of things, and updates works great, after inserting manually with:
az aks pod-identity add --cluster-name my-aks-cn --resource-group myrg --namespace myns --name example-pod-identity --identity-resource-id /subscriptions/......
But, I want to have this done at once, with the deployment, so I need to insert the pod user identities to the cluster automatically. I also tried to run the command using the DeploymentScripts but the deployment scripts are not ready to use preview aks extersion.
My config looks like this:
{
"type": "Microsoft.ContainerService/managedClusters",
"apiVersion": "2021-02-01",
"name": "[variables('cluster_name')]",
"location": "[variables('location')]",
"dependsOn": [
"[resourceId('Microsoft.Network/virtualNetworks', variables('vnet_name'))]"
],
"properties": {
....
"podIdentityProfile": {
"allowNetworkPluginKubenet": null,
"enabled": true,
"userAssignedIdentities": [
{
"identity": {
"clientId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'managed-indentity'), '2018-11-30').clientId]",
"objectId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'managed-indentity'), '2018-11-30').principalId]",
"resourceId": "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'managed-indentity')]"
},
"name": "managed-indentity",
"namespace": "myns"
}
],
"userAssignedIdentityExceptions": null
},
....
},
"identity": {
"type": "SystemAssigned"
}
},
I'm always getting the same issue:
"statusMessage": "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'deployment_test' is not valid according to the validation procedure. The tracking id is '.....'. See inner errors for details.\",\"details\":[{\"code\":\"PodIdentityAddonUserAssignedIdentitiesNotAllowedInCreation\",\"message\":\"Provisioning of resource(s) for container service cluster-12344 in resource group myrc failed. Message: {\\n \\\"code\\\": \\\"PodIdentityAddonUserAssignedIdentitiesNotAllowedInCreation\\\",\\n \\\"message\\\": \\\"PodIdentity addon does not support assigning pod identities on creation.\\\"\\n }. Details: \"}]}}",
The Product team has shared the answer here: https://github.com/Azure/aad-pod-identity/issues/1123
which says:
This is a known limitation in the existing configuration. We will fix
this in the V2 implementation.
For others who are facing the same issue, please refer to the GitHub issue above.
Objective
My objective is to export data from a Postgres RDS Instance to an s3 Bucket. I just want to prove that the concept works on my VPC, so I am using dummy data.
What I have tried so far
I followed the docs here using the console and cli.
Created an s3 bucket (I chose to block all public access)
Created an RDS Instance with the following settings:
Created on 2 public subnets
Public accessibility: No
Security group rules for outbound: CIDR/IP - Inbound 0.0.0.0/0
Security group rules for inbound: CIDR/IP - Inbound 0.0.0.0/0
Created a policy as shown in the example:
aws iam create-policy --policy-name rds-s3-export-policy --policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "s3export",
"Action": [
"S3:PutObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::your-s3-bucket/*"
]
}
]
}'
Created an IAM Role like:
aws iam create-role --role-name rds-s3-export-role --assume-role-policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "rds.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}'
Attached the policy to the role like:
aws iam attach-role-policy --policy-arn your-policy-arn --role-name rds-s3-export-role
Added the IAM Role to the DB like:
aws rds add-role-to-db-instance \
--db-instance-identifier my-db-instance \
--feature-name s3Export \
--role-arn your-role-arn \
--region your-region
Did all the requirements within PSQL like:
CREATE EXTENSION IF NOT EXISTS aws_s3 CASCADE;
CREATE TABLE sample_table (bid bigint PRIMARY KEY, name varchar(80));
INSERT INTO sample_table (bid,name) VALUES (1, 'Monday'), (2,'Tuesday'), (3, 'Wednesday');
SELECT aws_commons.create_s3_uri(
'dummy-data-bucket-path',
'',
'us-west-2'
) AS s3_uri_1 \gset
What does not work
When I try to make the actual export by:
SELECT * FROM aws_s3.query_export_to_s3('SELECT * FROM sample_table', :'s3_uri_1');
I get the error:
ERROR: could not upload to Amazon S3
DETAIL: Amazon S3 client returned 'Unable to connect to endpoint'.
CONTEXT: SQL function "query_export_to_s3" statement 1
Other things I have tried:
I have tried using Access analyzer for S3 but my bucket does not seem to appear in the list. I believe as the bucket itself does not have a policy attached to it.
How can I debug this issue? What am I doing wrong? I am happy to share further details if needed.
What I see from the documentation you are following does not assume that you are running this whole setup inside a VPC.
So for connecting from within the VPC(as you have blocked all the public access) , you need to have an endpoint policies for Amazon S3 attached.
for example from documentation sample policy :
The following is an example of an S3 bucket policy that allows access to a specific bucket, my_secure_bucket, from endpoint vpce-1a2b3c4d only.
{
"Version": "2012-10-17",
"Id": "Policy1415115909152",
"Statement": [
{
"Sid": "Access-to-specific-VPCE-only",
"Principal": "*",
"Action": "s3:*",
"Effect": "Deny",
"Resource": ["arn:aws:s3:::my_secure_bucket",
"arn:aws:s3:::my_secure_bucket/*"],
"Condition": {
"StringNotEquals": {
"aws:sourceVpce": "vpce-1a2b3c4d"
}
}
}
]
}
I have tried to fetch my RedShift cluster details. I'm able to see many details about the cluster but few details got missed.
For Ex:- Details like Storageand Memory
The below is the code:-
redshiftClient = boto3.client('redshift', aws_access_key_id = role.credentials.access_key,
aws_secret_access_key = role.credentials.secret_key, aws_session_token = role.credentials.session_token, region_name='us-west-2')
#Getting all the clusters
clusters = redshiftClient.describe_clusters()
can you please check provide the way to get it.
Thanks.
The describe-clusters command does not return that type of information. The output of that command is:
{
"Clusters": [
{
"NodeType": "dw.hs1.xlarge",
"Endpoint": {
"Port": 5439,
"Address": "mycluster.coqoarplqhsn.us-east-1.redshift.amazonaws.com"
},
"ClusterVersion": "1.0",
"PubliclyAccessible": "true",
"MasterUsername": "adminuser",
"ClusterParameterGroups": [
{
"ParameterApplyStatus": "in-sync",
"ParameterGroupName": "default.redshift-1.0"
} ],
"ClusterSecurityGroups": [
{
"Status": "active",
"ClusterSecurityGroupName": "default"
} ],
"AllowVersionUpgrade": true,
"VpcSecurityGroups": \[],
"AvailabilityZone": "us-east-1a",
"ClusterCreateTime": "2013-01-22T21:59:29.559Z",
"PreferredMaintenanceWindow": "sat:03:30-sat:04:00",
"AutomatedSnapshotRetentionPeriod": 1,
"ClusterStatus": "available",
"ClusterIdentifier": "mycluster",
"DBName": "dev",
"NumberOfNodes": 2,
"PendingModifiedValues": {}
} ],
"ResponseMetadata": {
"RequestId": "65b71cac-64df-11e2-8f5b-e90bd6c77476"
}
}
You will need to retrieve Memory and Storage statistics from Amazon CloudWatch.
See your other question: Amazon CloudWatch is not returning Redshift metrics
If you actually want to retrieve information about a standard cluster (that is, the amount of storage and memory assigned to each node, rather than current memory and storage usage), that is not available from an API call. Instead see: Amazon Redshift Clusters