Cloud formation Nested Stacks best way to version control templates using TemplateURL - version-control

We're creating a micro-services project to be deployed in multiple environments (dev,qa, stg, prd), we plan on making use of cloud formation templates using nested stacks for the shared resources between multiple services.
The thing is that when using nested stacks you need to specify the TemplateURL of the nested resource, and this is a static URL pointing an S3 Bucket that changes every time you do update the template (Upload a new template with some changes).
So the question is, what is the best way to use a version control tool like GIT to keep track of the changes made in a nested template that once it is upload to S3 would give you a new URL?

The cloudformation package command in the AWS Command Line Interface will upload local artifacts (including the TemplateURL property for the AWS::CloudFormation::Stack resource) to an S3 bucket, and output a transformed CloudFormation template referencing the uploaded artifact.
Using this command, the best way to track changes would be to commit both the base template and nested-stack templates to Git, then use cloudformation package as an intermediate processing step in your deploy script, e.g., with cloudformation deploy:
S3_BUCKET=my_s3_bucket
TEMPLATE=base_template.yml
OUTPUT_TEMPLATE=$(mktemp)
aws cloudformation package \
--template-file $TEMPLATE \
--s3-bucket $S3_BUCKET \
--output-template-file $OUTPUT_TEMPLATE
aws cloudformation deploy \
--template-file $OUTPUT_TEMPLATE \
--stack-name $STACK

Related

Who has done gitops with aws ssm and secrets parameters in codepipeline?

Just wondering who has implemented the use of github and updating ssm parameters and secrets within a codepipeline and what's the best practices around that.
I was thinking of putting some cli commands in a shell script in github and use codepipeline to migrate it. Any thoughts?
Whenever a command or API execution is required in CodePipeline, think about these 3 options (i.e. Actions):
CodeBuild (best to execute a command)
Lambda (More suited for API execution)
Custom Action (for advanced usage, you control the action completely)
So based on this, I would say CodeBuild will be the easiest option for your use case to run the command. Use a "latest" "curated image" so the execution is fastest for your CodeBuild job.

Mounting Azure Blob Storage to Azure Databricks without using cluster

We have a requirement that while provisioning the Databricks service thru CI/CD pipeline in Azure DevOps we should able to mount a blob storage to DBFS without connecting to a cluster. Is it possible to mount object storage to DBFS cluster by using a bash script from Azure DevOps ?
I looked thru various forums but they all mention about doing this using dbutils.fs.mount but the problem is we cannot run this command in Azure DevOps CI/CD pipeline.
Will appreciate any help on this.
Thanks
What you're asking is possible but it requires a bit of extra work. In our organisation we've tried various approaches and I've been working with Databricks for a while. The solution that works best for us is to write a bash script that makes use of the databricks-cli in your Azure Devops pipeline. The approach we have is as follows:
Retrieve a Databricks token using the token API
Configure the Databricks CLI in the CI/CD pipeline
Use Databricks CLI to upload a mount script
Create a Databricks job using the Jobs API and set the mount script as file to execute
The steps above are all contained in a bash script that is part of our Azure Devops pipeline.
Setting up the CLI
Setting up the Databricks CLI without any manual steps is now possible since you can generate a temporary access token using the Token API. We use a Service Principal for authentication.
https://learn.microsoft.com/en-US/azure/databricks/dev-tools/api/latest/tokens
Create a mount script
We have a scala script that follows the mount instructions. This can be Python as well. See the following link for more information:
https://docs.databricks.com/data/data-sources/azure/azure-datalake-gen2.html#mount-azure-data-lake-storage-gen2-filesystem.
Upload the mount script
In the Azure Devops pipeline the databricks-cli is configured by creating a temporary token using the token API. Once this step is done, we're free to use the CLI to upload our mount script to DBFS or import it as a notebook using the Workspace API.
https://learn.microsoft.com/en-US/azure/databricks/dev-tools/api/latest/workspace#--import
Configure the job that actually mounts your storage
We have a JSON file that defines the job that executes the "mount storage" script. You can define a job to use the script/notebook that you've uploaded in the previous step. You can easily define a job using JSON, check out how it's done in the Jobs API documentation:
https://learn.microsoft.com/en-US/azure/databricks/dev-tools/api/latest/jobs#--
At this point, triggering the job should create a temporary cluster that mounts the storage for you. You should not need to use the web interface, or perform any manual steps.
You can apply this approach to different environments and resource groups, as do we. For this we make use of Jinja templating to fill out variables that are environment or project specific.
I hope this helps you out. Let me know if you have any questions!

How to write to dags directory from Cloud Composer webserver?

I'm writing Cloud Composer plugin and I need to create DAG in runtime. How can I create DAG file from webserver or how can I access bucket ID from plugin code(so I can use gcs client and just upload DAG)? I tried code below and it doesn't work, I don't get any exceptions but also I don't see any results:
dag_path = os.path.join(settings.DAGS_FOLDER, dag_id + '.py')
with open(dag_path, 'w') as dag:
dag.write(result)
Possible solution is to read bucket ID from Cloud Composer env variable
You may either use the Environment Variables, or you may make use of the API's provided by GCloud SDK.
gcloud composer environments describe --format=json --project=<project-name> --location=<region> <cluster-name>
This would return the details of the cloud composer cluster.
It would have the dag location under the key dagGcsPrefix
The format of dagGcsPrefix would be gs://<GCSBucket>/dags

Convert Terraform Templates to Cloudformation Templates

I want to convert the existing terraform templates(hcl) to aws cloudformation templates(json/yaml).
I basically want to find security issues with these templates through CFN_NAG.
An approach that I have already tried was converting HCL to JSON and then passing the template to CFN_NAG but I received a failure since both the templates have different structure.
Can anyone please provide any suggestions here?
A rather convoluted way of achieving this is to use Terraform to stand-up actual AWS environments, and then to use AWS’s CloudFormer to extract CloudFormation templates (JSON or YAML) from what Terraform has built. At which point you can use cfn-nag.
CloudFormer has some limitations, in that not all AWS resources are currently supported (RDS Security Groups for example) , but it will get you all the basic AWS resources.
Don't forget to remove all the environments, including CloudFormer's, to minimise the cost.
You want to use static code analysis to find security issues in your Terraform setup.
Trying to converting Terraform to CloudFormation to later use cfn-nag is one way. However, there exist tools now that directly operate on the Terraform setup.
I would recommend to take a look at terrascan. It is built on terraform_validate.
https://github.com/bridgecrewio/checkov/ runs security scanning for both terraform and cloudformation

How to convert existing AWS environment into infra as code?

When we were building our AWS account, we did not think about using cloud formation or terraform. Now we have our environmemt all setup but don't want to tear down everything and build using cloud formation or terraform. So is there a way we can get our infrastructure to be imported and managed through one of them?
Thanks,
Terraform supports import, but that only supports the present state into state file. You still need to write the code. Cloudformation does not support import.
Something like https://github.com/dtan4/terraforming can be of help but YMMV.
A pretty complete answer could be found at AWS Export configuration as cloudformation template, which also covers Terraform for this purpose.
TL;DR
AWS Import/Export configuration as code (CloudFormationn | Terraform).
Based on our Infrastructure as Code (IaC) experience we found several ways to translate existing manually deployed (from Web Console UI) AWS infra to Cloudformation (CF) and / or Terraform (TF) code. Posible solutions are listed below:
AWS Cloudformation Templates
CF-1 | AWS CloudFormation native import feature
CF-2 | aws cli & manually translate to CF
CF-3 | Former2
CF-4 | AWS CloudFormer
Terraform Code / Modules
TF-1 | Terraforming
TF-2 | CloudCraft + Modules.tf
Related Article: https://medium.com/#exequiel.barrirero/aws-export-configuration-as-code-cloudformation-terraform-b1bca8949bca
As per October 2019, AWS supports importing legacy resources into CloudFormation. See https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html for examples.