Regarding using the COPY command for populating a Redshift table with data from S3; I'm wondering if there is a reason for why you have to specify a role via its ARN which provides the permissions even though the Redshift cluster is already associated with that rule. This seems redundant to me but probably there is a reason for this. Hence my question.
This question arose upon reading the Redshift getting started guide; specifically regarding steps 2, 3 and 6.
It's not mandatory to reference an IAM Role when using the COPY command. This is one of several authorization methods available for the cluster to access external resources (e.g. files stored on S3). The reason for specifying the IAM_ROLE clause is to tell Redshift that this is the authorization method to use, you could alternatively specify ACCESS_KEY_ID/SECRET_ACCESS_KEY or CREDENTIALS.
https://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html
The reason you need to add the ARN for a specific IAM role is that it's possible to add more than one role to a cluster.
Related
I am using Hashicorp Vault to generate dynamic creds in multiple database clusters. We have one database cluster that is somewhat ephemeral so on occasion it will be refreshed from another database cluster. This database cluster will be connected to via Vault dynamic creds just like the other database clusters.
I have the process to clean up the database users brought over by the backup from the source system when this cluster is refreshed but I don't know how I should handle the Vault cleanup. The database config will be the same (same host/user) but all the existing database user accounts recently created by Vault will be gone after the refresh so I don't know what I need to do to reset/clean up Vault for that database. The database system I'm using (Redshift) doesn't seem to have DROP USER ... IF EXISTS type of syntax otherwise I would simply use that in the dynamic role's revocation_statements and let it cycle out naturally that way.
So my main question is how do I reset or delete all the dynamic creds that were created for a specific database cluster in Vault if the database cluster is refreshed or no longer exists?
I figured out the answer to this and I wanted to share here in case anyone else encounters this.
The "lease revoke" documentation explains that you can use the -prefix switch to revoke leases based on a partial or full prefix match.
Using this information you can run a command similar to the following in order to force revoke existing leases for a specific role:
vault lease revoke -force -prefix database/creds/ROLE_NAME
Using the -force switch will remove the lease even if the revocation_statements fails to process (case when the database user no longer exists).
As an aside, the following command can be used to list leases and is useful to check before and after that all the leases are, in fact, revoked:
vault list sys/leases/lookup/database/creds/ROLE_NAME
This solves my problem of "how to I remove leases for orphaned Vault dynamic credentials" in cases where the target database is refreshed from a backup which is the case I am using this for.
I'm putting together a role/policy for running cloudformation/sam to limit access as much as I can. Is there a general set of policy actions that should be used to run create-stack?
This is for a codebuild which I'm using to create infrastructure using a cloudformation template during runtime of my application.
At the moment I've got a policy which allows full access, because it needs to create the infrastructure within the stack.
But there are only a subset of actions which cloudformation can actually perform and it doesn't need full access. For example, CF can't put items into a dynamodb table.
So this led me to think that maybe there's a basic role/policy that is limited to only the actions which cloudformation is able to perform.
If you're having to assign a role to a service (such as CodePipeline or CodeBuild) to deploy a stack, you do not only need to assign the necessary CloudFormation permissions (such as cloudformation:CreateStack or cloudformation:ExecuteChangeSet) but also permissions necessary for the deployment of the CloudFormation stack itself.
When you are deploying a stack manually, CloudFormation will use your user permissions to verify access to the services you are deploying/updating. When you're initiating the action from another AWS service, the same thing happens, but with the services from the service role. (Unless you are specifically assigning a role to the CloudFormation stack, documentation).
Keep in mind if you're constructing such a role, that CloudFormation might need more permissions than you think, such as extra read permissions, permissions to add tags to resources, permissions to delete and/or update those resources when you're deleting/updating the resources etc.
Is it possible to have multiple databse in a cluster with Crunchydata (postgres)?
When I create a cluster with "pgo create cluster" command I can specify only one database.
-d, --database string If specified, sets the name of the initial database that is created for the user. Defaults to the value set in the PostgreSQL Operator configuration, or if that is not present, the name of the cluster
But I need multiple database per cluster, and I can't find any official way to create them.
Another question: How can I find the "superuser" username and password to login to PGOAmin Web?
Thanks a lot.
This could be useful, however it stated:
"It may make more sense to have each of your databases in its own
cluster if you want to have them spread out over your Kubernetes
topology."
https://github.com/CrunchyData/postgres-operator/issues/2655
It seems that by default a lambda function created by Pulumi has an AWSLambdaFullAccess permissions. This type of access is too wide and I'd like to replace it with fine-grained ACLs.
For instance, assuming I am creating a cloud.Table in my index.js file, I would like to specify that the lambda endpoint I am creating (in the same file) only has read access to that specific table.
Is there a way to do it without coding the IAM policy myself?
The #pulumi/cloud library currently runs all compute (lambdas and containerized services) with a single uniform set of IAM policies on AWS.
You can set the policies to use by running:
pulumi config set cloud-aws:computeIAMRolePolicyARNs "arn:aws:iam::aws:policy/AWSLambdaFullAccess,arn:aws:iam::aws:policy/AmazonEC2ContainerServiceFullAccess"
The values above are the defaults. See https://github.com/pulumi/pulumi-cloud/blob/master/aws/config/index.ts#L52-L56.
There are plans to support more fine-grained control over permissions and computing permissions directly from resources being used in #pulumi/cloud - see e.g. https://github.com/pulumi/pulumi-cloud/issues/145 and https://github.com/pulumi/pulumi-cloud/issues/168.
Lower level libraries (like #pulumi/aws and #pulumi/aws-serverless) provide complete control over the Role and/or Policies applied to Function objects.
I have read that you can replicate a Cloud SQL database to MySQL. Instead, I want to replicate from a MySQL database (that the business uses to keep inventory) to Cloud SQL so it can have up-to-date inventory levels for use on a web site.
Is it possible to replicate MySQL to Cloud SQL. If so, how do I configure that?
This is something that is not yet possible in CloudSQL.
I'm using DBSync to do it, and working fine.
http://dbconvert.com/mysql.php
The Sync version do the service that you want.
It work well with App Engine and Cloud SQL. You must authorize external conections first.
This is a rather old question, but it might be worth noting that this seems now possible by Configuring External Masters.
The high level steps are:
Create a dump of the data from the master and upload the file to a storage bucket
Create a master instance in CloudSQL
Setup a replica of that instance, using the external master IP, username and password. Also provide the dump file location
Setup additional replicas if needed
Voilà!