It seems that by default a lambda function created by Pulumi has an AWSLambdaFullAccess permissions. This type of access is too wide and I'd like to replace it with fine-grained ACLs.
For instance, assuming I am creating a cloud.Table in my index.js file, I would like to specify that the lambda endpoint I am creating (in the same file) only has read access to that specific table.
Is there a way to do it without coding the IAM policy myself?
The #pulumi/cloud library currently runs all compute (lambdas and containerized services) with a single uniform set of IAM policies on AWS.
You can set the policies to use by running:
pulumi config set cloud-aws:computeIAMRolePolicyARNs "arn:aws:iam::aws:policy/AWSLambdaFullAccess,arn:aws:iam::aws:policy/AmazonEC2ContainerServiceFullAccess"
The values above are the defaults. See https://github.com/pulumi/pulumi-cloud/blob/master/aws/config/index.ts#L52-L56.
There are plans to support more fine-grained control over permissions and computing permissions directly from resources being used in #pulumi/cloud - see e.g. https://github.com/pulumi/pulumi-cloud/issues/145 and https://github.com/pulumi/pulumi-cloud/issues/168.
Lower level libraries (like #pulumi/aws and #pulumi/aws-serverless) provide complete control over the Role and/or Policies applied to Function objects.
Related
I'm putting together a role/policy for running cloudformation/sam to limit access as much as I can. Is there a general set of policy actions that should be used to run create-stack?
This is for a codebuild which I'm using to create infrastructure using a cloudformation template during runtime of my application.
At the moment I've got a policy which allows full access, because it needs to create the infrastructure within the stack.
But there are only a subset of actions which cloudformation can actually perform and it doesn't need full access. For example, CF can't put items into a dynamodb table.
So this led me to think that maybe there's a basic role/policy that is limited to only the actions which cloudformation is able to perform.
If you're having to assign a role to a service (such as CodePipeline or CodeBuild) to deploy a stack, you do not only need to assign the necessary CloudFormation permissions (such as cloudformation:CreateStack or cloudformation:ExecuteChangeSet) but also permissions necessary for the deployment of the CloudFormation stack itself.
When you are deploying a stack manually, CloudFormation will use your user permissions to verify access to the services you are deploying/updating. When you're initiating the action from another AWS service, the same thing happens, but with the services from the service role. (Unless you are specifically assigning a role to the CloudFormation stack, documentation).
Keep in mind if you're constructing such a role, that CloudFormation might need more permissions than you think, such as extra read permissions, permissions to add tags to resources, permissions to delete and/or update those resources when you're deleting/updating the resources etc.
I use Pulumi to bring up my infrastructures in GCP . Pulumi has the stack features that helps you to build multiple replications of the same type of Pulumi's code.
So I have dev/stage/prod stack that corresponds to each of the environment we have.
I want to know if there is a way that I can protect the production stack so that no one can delete any resources in there.
I am aware that about the protect bit flag, but that would apply to all the stacks which I don't want to.
there are a couple options to achieve this:
Option 1
One option would be to restrict access to the Pulumi state file such that only a privileged user or entity (e.g. a continuous delivery pipeline) is able to read and write the prod state and therefore able to perform operations that might destroy resources. The Pulumi Console backend supports this with stack permissions at a granular level and access can be restricted with the other state backends via the IAM capabilities of the specific provider (e.g. AWS IAM).
Option 2
Another option (that could be used in conjunction with the first) would be to programmatically set the protect flag based on the stack name. Below is an example in Python, but the same concept works in all languages:
import pulumi
from pulumi_aws import s3
# only set `protect=True` for "prod" stacks
prod_protected = False
if "prod" == pulumi.get_stack():
prod_protected = True
bucket = s3.Bucket("my-bucket",
opts=pulumi.ResourceOptions(
protect=prod_protected, # use `prod_protected` flag
),
)
You would be required to set protect=... on each resource in your stack to protect all resources in the prod stack. The Pulumi SDK provides a way to set this on all resources at once with a stack transformation. There's an example of doing a stack transformation to set tags on resources here.
What factors do folk take into account when deciding to write 1 large CF template, or nest many smaller ones? The use case I have in mind is RDS based where I'll need to define RDS instances, VPC Security groups, parameter and option groups as well as execute some custom lambda resources.
My gut feel is that this should be split, perhaps by resource type, but I was wondering if there was generally accepted practice on this.
My current rule of thumb is to split resources by deployment units - what deploys together, goes together.
I want to have the smallest deployable stack, because it's fast to deploy or fail if there's an issue. I don't follow this rule religiously. For example, I often group Lambdas together (even unrelated ones, depends on the size of the project), as they update only if the code/config changed and I tend to push small updates where only one Lambda changed.
I also often have a stack of shared resources that are used (Fn::Import-ed) throughout the other stacks like a KMS key, a shared S3 Bucket, etc.
Note that I have a CD process set up for every stack, hence the rule.
My current setup requires deployment of a VPC (with endpoints), RDS & application (API gateway, Lambdas). I have broken them down as
VPC stack: a shared resource with 1 VPC per region with public & private subnets, VPC endpoints, S3 bucket, NAT gateways, ACLs, security groups.
RDS stack: I can have multiple RDS clusters inside a VPC so made sense to keep it separate. Also, this is created after VPC as it needs VPC resources such as private subnets, security groups. This cluster is shared by multiple application stacks.
Application stack: This deploys API gateway & Lambdas (basically a serverless application) with the above RDS cluster as the DB.
So, in general, I pretty much follow what #Milan Cermak described. But in my case, these deployments are done when needed (not part of CD) so exported parameters are stored in parameter store of AWS Systems Manager.
Regarding using the COPY command for populating a Redshift table with data from S3; I'm wondering if there is a reason for why you have to specify a role via its ARN which provides the permissions even though the Redshift cluster is already associated with that rule. This seems redundant to me but probably there is a reason for this. Hence my question.
This question arose upon reading the Redshift getting started guide; specifically regarding steps 2, 3 and 6.
It's not mandatory to reference an IAM Role when using the COPY command. This is one of several authorization methods available for the cluster to access external resources (e.g. files stored on S3). The reason for specifying the IAM_ROLE clause is to tell Redshift that this is the authorization method to use, you could alternatively specify ACCESS_KEY_ID/SECRET_ACCESS_KEY or CREDENTIALS.
https://docs.aws.amazon.com/redshift/latest/dg/copy-usage_notes-access-permissions.html
The reason you need to add the ARN for a specific IAM role is that it's possible to add more than one role to a cluster.
The One Binary principle explained here:
http://programmer.97things.oreilly.com/wiki/index.php/One_Binary states that one should...
"Build a single binary that you can identify and promote through all the stages in the release pipeline. Hold environment-specific details in the environment. This could mean, for example, keeping them in the component container, in a known file, or in the path."
I see many dev-ops engineers arguably violate this principle by creating one docker image per environment (ie, my-app-qa, my-app-prod and so on). I know that Docker favours immutable infrastructure which implies not changing an image after deployment, therefore not uploading or downloading configuration post deployment. Is there a trade-off between immutable infrastructure and the one binary principle or can they complement each-other? When it comes to separating configuration from code what is the best practice in a Docker world??? Which one of the following approaches should one take...
1) Creating a base binary image and then having a configuration Dockerfile that augments this image by adding environment specific configuration. (i.e my-app -> my-app-prod)
2) Deploying a binary-only docker image to the container and passing in the configuration through environment variables and so on at deploy time.
3) Uploading the configuration after deploying the Docker file to a container
4) Downloading configuration from a configuration management server from the running docker image inside the container.
5) Keeping the configuration in the host environment and making it available to the running Docker instance through a bind mount.
Is there another better approach not mentioned above?
How can one enforce the one binary principle using immutable infrastructure? Can it be done or is there a trade-off? What is the best practice??
I've about 2 years of experience deploying Docker containers now, so I'm going to talk about what I've done and/or know to work.
So, let me first begin by saying that containers should definitely be immutable (I even mark mine as read-only).
Main approaches:
use configuration files by setting a static entrypoint and overriding the configuration file location by overriding the container startup command - that's less flexible, since one would have to commit the change and redeploy in order to enable it; not fit for passwords, secure tokens, etc
use configuration files by overriding their location with an environment variable - again, depends on having the configuration files prepped in advance; ; not fit for passwords, secure tokens, etc
use environment variables - that might need a change in the deployment code, thus lessening the time to get the config change live, since it doesn't need to go through the application build phase (in most cases), deploying such a change might be pretty easy. Here's an example - if deploying a containerised application to Marathon, changing an environment variable could potentially just start a new container from the last used container image (potentially on the same host even), which means that this could be done in mere seconds; not fit for passwords, secure tokens, etc, and especially so in Docker
store the configuration in a k/v store like Consul, make the application aware of that and let it be even dynamically reconfigurable. Great approach for launching features simultaneously - possibly even accross multiple services; if implemented with a solution such as HashiCorp Vault provides secure storage for sensitive information, you could even have ephemeral secrets (an example would be the PostgreSQL secret backend for Vault - https://www.vaultproject.io/docs/secrets/postgresql/index.html)
have an application or script create the configuration files before starting the main application - store the configuration in a k/v store like Consul, use something like consul-template in order to populate the app config; a bit more secure - since you're not carrying everything over through the whole pipeline as code
have an application or script populate the environment variables before starting the main application - an example for that would be envconsul; not fit for sensitive information - someone with access to the Docker API (either through the TCP or UNIX socket) would be able to read those
I've even had a situation in which we were populating variables into AWS' instance user_data and injecting them into container on startup (with a script that modifies containers' json config on startup)
The main things that I'd take into consideration:
what are the variables that I'm exposing and when and where am I getting their values from (could be the CD software, or something else) - for example you could publish the AWS RDS endpoint and credentials to instance's user_data, potentially even EC2 tags with some IAM instance profile magic
how many variables do we have to manage and how often do we change some of them - if we have a handful, we could probably just go with environment variables, or use environment variables for the most commonly changed ones and variables stored in a file for those that we change less often
and how fast do we want to see them changed - if it's a file, it typically takes more time to deploy it to production; if we're using environment variable
s, we can usually deploy those changes much faster
how do we protect some of them - where do we inject them and how - for example Ansible Vault, HashiCorp Vault, keeping them in a separate repo, etc
how do we deploy - that could be a JSON config file sent to an deployment framework endpoint, Ansible, etc
what's the environment that we're having - is it realistic to have something like Consul as a config data store (Consul has 2 different kinds of agents - client and server)
I tend to prefer the most complex case of having them stored in a central place (k/v store, database) and have them changed dynamically, because I've encountered the following cases:
slow deployment pipelines - which makes it really slow to change a config file and have it deployed
having too many environment variables - this could really grow out of hand
having to turn on a feature flag across the whole fleet (consisting of tens of services) at once
an environment in which there is real strive to increase security by better handling sensitive config data
I've probably missed something, but I guess that should be enough of a trigger to think about what would be best for your environment
How I've done it in the past is to incorporate tokenization into the packaging process after a build is executed. These tokens can be managed in an orchestration layer that sits on top to manage your platform tools. So for a given token, there is a matching regex or xpath expression. That token is linked to one or many config files, depending on the relationship that is chosen. Then, when this build is deployed to a container, a platform service (i.e. config mgmt) will poke these tokens with the correct value with respect to its environment. These poke values most likely would be pulled from a vault.