I'm looking for an option to pick all the templates from the repository without hardcode the yml template files and in future if new templates are added, the pipeline should automatically pick all of them and do the deploy and create a single stack in aws environment, without making any modification to gitlab-ci.yml/pipeline file.
I tried using deploy CLI command, it deploy all the templates but then it goes for update and start deleting one by one and only the last template resource will be available after the pipeline execution is complete.
Let me know if there is an option to do this?
Related
We have an Azure DevOps Pipeline defined as a template in a .yml file, and would like to use this to build 100+ projects.
There is zero parameterization to do, and the template essentially just runs against the project repo as-is.
Normally, we would just go into each of those project repos and create an azure-pipelines.yml file and reference the template pipeline, easy.
However, I would like to know if we can avoid creating these azure-pipelines.yml in each of the 100+ project repos. Is there some way to just run the template against 100+ repos, but as separate pipelines?
No.
YAML pipelines are based upon the YAML being stored in the application's repo, as code, alongside the application code. You can pull in templates from other repos, but you can't have one pipeline that is automatically valid for any repo.
Trying to use one template to build 100+ applications without the possibility of pinning the template to a specific version on an application by application basis is setting yourself up for a nightmare at some point in the future, when the template changes in such a way that it's no longer valid for a subset of your 100 applications.
I have a repository in CodeCommit, and in this repository, there are 3 branches dev, stage, and prod, in this repository there are multi stacks versioned, for example:
root/
--task-1
----template.yml
------src
--------index.js
--------package.json
--task-2
--task-3
--task-....
--buildspec.yml
Where every folder contains a different template yml and its src folder for the specific Lamba code, the buildspec.yml contains the commands to enter in every task folder and execute the required commands to install the node packages required and the sam or cloudformation commands to create or update the stack.
When a new commit is pushed to origins this trigger the pipeline and executes all the commands of buildspec.yml and create/update all the stacks even when only one stack has been changed in the code, here the question if there are better solutions to handle multi stacks in one repository and one pipeline.
One idea is to create one repository and pipeline for each stack in this way every stack will be updated independently of the other stacks, but in this way, if there are 20 stacks will be required 20 repositories and 20 pipelines.
I would like to know what is the best practice to handle multi stacks in the same repository and one pipeline and avoid deploying all the stacks when just one stack has been updated, or update only stacks that were updated in codecommit.
Create an Amazon EventBridge (Amazon CloudWatch Events) rule that invokes an AWS Lambda function to evaluate changes to the repository and run the appropriate pipeline.
it could be fixed using a lambda and event bridge when a commit happens, more details https://aws.amazon.com/es/blogs/devops/adding-custom-logic-to-aws-codepipeline-with-aws-lambda-and-amazon-cloudwatch-events/
Reusing same environment rule within same workflow
Running our workflow in Github, we split our tasks up into 2 jobs; Building docker image & attach tags and deploying to AWS using CodeDeploy. The reason for splitting the tasks up is to avoid creating new tags whenever our deployment fails.
However... using environment protection rules creates a roadblock as every job needs to be approved(even though we already ran the same environment previously)
The deployment job is a conditional job, meaning it depends on the success of the Build job.
Is there any way to get around this?
Github workflow
I'm attempting to write a CloudFormation template to fully to define all resources required for an ECS service, including...
CodeCommit repository for the nodejs code
CodePipeline to manage builds
ECR Repository
ECS Task Definition
ECS Service
ALB Target Group
ALB Listener Rule
I've managed to get all of this working. The stack builds fine. However I'm not sure how to correctly handle updates.
The Container in the Task Defition in the template required an image to be defined. However the actual application image won't exist until after the code is first built by the pipeline.
I had an idea that I might be able to work around this issue, by defining some kind of placeholder image "amazon/amazon-ecs-sample" for example, just to allow the stack to build. This image would be replaced by CodeBuild when the pipeline first runs.
This part also works fine.
The issues occur when I attempt to update the task definition, for example adding environment variables, in the CloudFormation template. When I re-run the stack, it replaces my application image in the container definition, with the original placeholder image from the template.
This is logical enough, as CloudFormation obviously assumes the image in the template is the correct one to use.
I'm just trying to figure out the best way to handle this.
Essentially I'd like to find some way to tell CloudFormation to just use whatever image is defined in the most recent revision of the task definition when creating new revisions, rather than replacing it with the original template property.
Is what I'm trying to do actually possible with pure CloudFormation, or will I need to use a custom resource or something similar?
Ideally I'd like to keep extra stack dependencies to a minimum.
One possibility I had thought of, would be to use a fixed tag for the container definition image, which won't actually exist when the cloudformation stack first builds, but which will exist after the first code-pipeline build.
For example
image: [my_ecr_base_uri]/[my_app_name]:latest
I can then have my pipeline push a new revision with this tag. However, I prefer to define task defition revisions with specific verion tags, like so ...
image: [my_ecr_base_uri]/[my_app_name]:v1.0.1-[git-sha]
... as this makes it very easy to see exactly what version of the application is currently running, and to revert revisions easily if needed.
Your problem is that you're putting too many things into this CloudFormation template. Your template could include the CodeCommit repository and the CodePipeline. But the other things should be outputs from your pipeline. Remember: Your pipeline will have a build and a deploy stage. The build stage can "build" another cloudformation template that is executed in the deploy stage. During this deploy stage, your pipeline will construct the ECS services, tasks, ALB etc...
I'm using Bitbucket Pipelines to do CD for a Serverless app. I want to use as few "build minutes" as possible for each deployment. The lifecycle of the serverless deploy command, when using AWS as the backing, seems to be:
Push the package to CloudFormation. (~60 seconds)
Sit around watching the logs from CloudFormation until the deployment finishes. (~20-30 minutes)
Because of the huge time difference, I don't want to do step two. So my question is simple: how do I deploy a serverless app such that it only does step one and returns success or failure based on whether or now CloudFormation successfully accepted the new package?
I've looked at the docs for serverless deploy and I can't see any options to enable that. Also, there seem to be AWS specific options in the serverless deploy command already, so maybe this is an option that the serverless team will consider if there is no other way to do this.
N.B. As for, "how will you know if CloudFormation fails?", for that, I would rather set up notifications to come from CloudFormation directly. The build can just have the responsibility of pushing to CloudFormation.
I don't think you can do it with serverless deploy. You can try serverless package command that will store the package in .serverless folder or you can specify the path using --package. Package will create a CloudFormation template file e.g. cloudformation-template-update-stack.json. You can then call Create Stack API action to create the stack. It will return the stack ID without waiting for all the resources to be created.