How can I set a cloudformation field with function shorthand (ie. !Sub) using yq? - aws-cloudformation

I need to add a managed policy to a large number of lambda and ecs task roles across many cloudformation yaml files. Some of them already have this array field with items in it while some do not. I need to add an item to the ManagedPolicyArns array field where that item contains cloudformation shorthand such as !Sub. I'm using yq, which is a fantastic tool, but I can't figure out handling this shorthand using the docs.
cloudformation.yaml:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
MyPrefix:
Description: MyPrefix
Type: String
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- Fn::ImportValue: !Sub "${MyPrefix}-my-policy-arn"
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
- lambda.amazonaws.com
Action:
- sts:AssumeRole
- sts:TagSession
Path: /
Expected output .yaml:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
MyPrefix:
Description: MyPrefix
Type: String
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- Fn::ImportValue: !Sub "${MyPrefix}-my-policy-arn"
- Fn::ImportValue: !Sub "${MyPrefix}-my-new-policy-arn"
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
- lambda.amazonaws.com
Action:
- sts:AssumeRole
- sts:TagSession
Path: /
My failed attempt:
add_policy.yq:
( .Resources[] |=
select(.Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "ecs-tasks.amazonaws.com" or .Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "lambda.amazonaws.com")
.Properties.ManagedPolicyArns += {"Fn::ImportValue": {"!Sub": "${MyPrefix}-my-new-policy-arn"} }
)
command:
yq --from-file add_policy.yq cloudformation.yaml
which outputs:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
MyPrefix:
Description: MyPrefix
Type: String
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
ManagedPolicyArns:
- Fn::ImportValue: !Sub "${MyPrefix}-my-policy-arn"
- Fn::ImportValue:
'!Sub': ${MyPrefix}-my-new-policy-arn
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
- lambda.amazonaws.com
Action:
- sts:AssumeRole
- sts:TagSession
Path: /
How do I get
- Fn::ImportValue: !Sub "${MyPrefix}-my-policy-arn"
- Fn::ImportValue: !Sub "${MyPrefix}-my-new-policy-arn"
instead of
- Fn::ImportValue: !Sub "${MyPrefix}-my-policy-arn"
- Fn::ImportValue:
'!Sub': ${MyPrefix}-my-new-policy-arn
?

Use the tag operator
add_policy.yq:
( .Resources[] |=
select(.Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "ecs-tasks.amazonaws.com" or .Properties.AssumeRolePolicyDocument.Statement[].Principal.Service.[] == "lambda.amazonaws.com")
.Properties.ManagedPolicyArns += {"Fn::ImportValue": "${AuthAuditBucketStackName}-logging-policy-arn" | . tag = "!Sub" }
)
results in:
- Fn::ImportValue: !Sub "${MyPrefix}-my-policy-arn"
- Fn::ImportValue: !Sub ${MyPrefix}-my-new-policy-arn
Now I'm not sure why the string value isn't quoted.

Related

AWS Cloudformation - security group ids list export and import - SecurityGroupIds not valid

Im working with 2 nested stacks. I need to use security group ids exported from NestedA in NestedB. The exported security group ids are to be used in a SecurityGroupIds property in NestedB based on conditions.
However cloudformation returns error: Property validation failure: [Value of property {/LaunchTemplateData/SecurityGroupIds/0} does not match type {String}]
The following are snippets of what I have tried:
NestedA export
Outputs:
SG1
Value: !Join
- ','
- - !Ref securitygroup1
- !Ref securitygroup2
Export:
Name: !Sub ${ExportVpcStackName}-SG1
SG2
Value: !Join
- ','
- - !Ref securitygroup3
- !Ref securitygroup4
Export:
Name: !Sub ${ExportVpcStackName}-SG2
ParentStack
Resources:
...
launchtemplate:
Type: AWS::Cloudformation::Stack
Properties:
TemplateURL: https://s3/nestedB.yaml
...
SG1:
Fn::ImportValue: !Sub ${ExportVpcStackName}-SG1
SG2:
Fn::ImportValue: !Sub ${ExportVpcStackName}-SG2
NestedB import
Parameters:
SG1
Type: List<AWS::EC2::SecurityGroup::Id>
SG2
Type: List<AWS::EC2::SecurityGroup::Id>
Resources:
launchtemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateData:
...
SecurityGroupIds:
!If
- Condition1
-
- !Ref SG1
- !Ref SG2
- !If
- Condition2
-
- !Ref SG1
- !Ref AWS::NoValue
Ive also tried importing each of the security groups directly/individually into NestedB with no success ie:
NestedA export
Outputs:
securitygroup1:
Value: !Ref securitygroup1
Export:
Name: !Sub ${ExportVpcStackName}-securitygroup1
securitygroup2:
Value: !Ref securitygroup2
Export:
Name: !Sub ${ExportVpcStackName}-securitygroup2
securitygroup3:
Value: !Ref securitygroup3
Export:
Name: !Sub ${ExportVpcStackName}-securitygroup3
securitygroup4:
Value: !Ref securitygroup4
Export:
Name: !Sub ${ExportVpcStackName}-securitygroup4
NestedB import
Resources:
launchtemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateData:
...
SecurityGroupIds:
!If
- Condition1
-
- Fn::ImportValue: !Sub ${ExportVpcStackName}-securitygroup1
- Fn::ImportValue: !Sub ${ExportVpcStackName}-securitygroup2
- Fn::ImportValue: !Sub ${ExportVpcStackName}-securitygroup3
- Fn::ImportValue: !Sub ${ExportVpcStackName}-securitygroup4
- !If
- Condition2
-
- Fn::ImportValue: !Sub ${ExportVpcStackName}-securitygroup1
- Fn::ImportValue: !Sub ${ExportVpcStackName}-securitygroup2
- !Ref AWS::NoValue
Whats the error Im making?
Edit: I have tried #marcin suggestion but still get the error:
Property validation failure: [Value of property {/LaunchTemplateData/SecurityGroupIds/0} does not match type {String}]
Instead of Type: List<AWS::EC2::SecurityGroup::Id>, please use CommaDelimitedList.
Also your SG1 is a list of SGs. You have to use Fn::Select to get individual SG values from the list.

How to create an ECS task in CloudFormation before the CodePipeline is created

I'm trying to define my ECS stack in Cloudformation, including the CI/CD pipeline and ECR repository. However you run into a bit of a conundrum in that:
To create an ECS task definition (AWS::ECS::TaskDefinition) you have to first create a populated ECR repository (AWS::ECR::Repository) so that you can specify the Image property.
To populate this repository you have to first create the CodePipeline (AWS::CodePipeline::Pipeline) which will run automatically on creation.
To create the pipeline you have to first create the ECS task definition / cluster as the pipeline needs to deploy onto it (back to step 1).
The solutions to this I can see are:
Don't create the ECR repository in Cloudformation & pass it as a parameter to the stacks.
Define a dummy image in the task definition to deploy the first time and then create the pipeline which will create the real ECR repository and deploy the real image.
Create the CodeBuild project and ECR repository in a separate stack, trigger the CodeBuild project with a lambda function (I don't think it runs automatically on creation like the pipeline does), create the ECS cluster and then create the pipeline. This seems more complicated than it should be.
Are there any better ways of approaching this problem?
The way I do it is to ECR repository first, but still using CloudFormation. So I have two templates. One for ECR repo. And the second one for the rest. The ECR repo is passed as a parameter to the second template. But you can also export its Uri to be ImportValue in the second step. The Uri is created as follows:
Uri:
Value: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${MyECR}"
You will also need some initial image in the repo for the task definition. This you can automate by having separated CodeBuild project (no need for CodePipeline) for this initial build.
Another way to create this by using a single stack is to trigger the Fargate deployment after pushing the image by making an initial commit to the CodeCommit repository and setting the DesiredCount property of the ECS service to zero:
Repo:
Type: AWS::CodeCommit::Repository
Properties:
Code:
BranchName: main
S3:
Bucket: some-bucket
Key: code.zip
RepositoryName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
RepositoryDescription: Repository
Triggers:
- Name: Trigger
CustomData: The Code Repository
DestinationArn: !Ref Topic
Branches:
- main
Events: [all]
Service:
Type: AWS::ECS::Service
Properties:
Cluster: !Ref Cluster
DesiredCount: 0
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
SecurityGroups:
- !Ref SecG
Subnets: !Ref Subs
ServiceName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
TaskDefinition: !Ref TaskDefinition
Build:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: CODEPIPELINE
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
phases:
pre_build:
commands:
- echo "[`date`] PRE_BUILD"
- echo "Logging in to Amazon ECR..."
- aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.$REGION.amazonaws.com
- IMAGE_URI="$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:$TAG"
build:
commands:
- echo "[`date`] BUILD"
- echo "Building Docker Image..."
- docker build -t $REPO:$TAG .
- docker tag $REPO:$TAG $IMAGE_URI
post_build:
commands:
- echo "[`date`] POST_BUILD"
- echo "Pushing Docker Image..."
- docker push $IMAGE_URI
- echo Writing image definitions file...
- printf '[{"name":"svc","imageUri":"%s"}]' $IMAGE_URI > $FILE
artifacts:
files: $FILE
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/standard:6.0
Type: LINUX_CONTAINER
EnvironmentVariables:
- Name: REGION
Type: PLAINTEXT
Value: !Ref AWS::Region
- Name: ACCOUNT
Type: PLAINTEXT
Value: !Ref AWS::AccountId
- Name: TAG
Type: PLAINTEXT
Value: latest
- Name: REPO
Type: PLAINTEXT
Value: !Ref Registry
- Name: FILE
Type: PLAINTEXT
Value: !Ref ImagesFile
PrivilegedMode: true
Name: !Ref AWS::StackName
ServiceRole: !GetAtt CodeBuildServiceRole.Arn
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineServiceRole.Arn
ArtifactStore:
Type: S3
Location: !Ref ArtifactBucket
Stages:
- Name: Source
Actions:
- Name: Site
ActionTypeId:
Category: Source
Owner: AWS
Version: '1'
Provider: CodeCommit
Configuration:
RepositoryName: !GetAtt Repo.Name
BranchName: main
PollForSourceChanges: 'false'
InputArtifacts: []
OutputArtifacts:
- Name: SourceArtifact
RunOrder: 1
- Name: Build
Actions:
- Name: Docker
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
Configuration:
ProjectName: !Ref Build
InputArtifacts:
- Name: SourceArtifact
OutputArtifacts:
- Name: BuildArtifact
RunOrder: 1
- Name: Deploy
Actions:
- Name: Fargate
ActionTypeId:
Category: Deploy
Owner: AWS
Version: '1'
Provider: ECS
Configuration:
ClusterName: !Ref Cluster
FileName: !Ref ImagesFile
ServiceName: !GetAtt Service.Name
InputArtifacts:
- Name: BuildArtifact
RunOrder: 1
Note that the some-bucket S3 bucket needs to contain the zipped .Dockerfile and any source code without any .git directory included.
If you use another service for your repo, like GitHub for instance, or your already have a repo, simple remove the section and configure the pipeline as required.
The entire CloudFormation stack is listed below for reference:
AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFormation Stack to Trigger CodeBuild via CodePipeline
Parameters:
SecG:
Description: Single security group
Type: AWS::EC2::SecurityGroup::Id
Subs:
Description: Comma separated subnet IDs
Type: List<AWS::EC2::Subnet::Id>
ImagesFile:
Type: String
Default: images.json
Resources:
ArtifactBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
Tags:
- Key: UseWithCodeDeploy
Value: true
CodeBuildServiceRole:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: codebuild.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: !Sub 'ssm-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action:
- ssm:GetParameters
- secretsmanager:GetSecretValue
Resource: '*'
- PolicyName: !Sub 'logs-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: '*'
- PolicyName: !Sub 'ecr-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: Allow
Action:
- ecr:BatchCheckLayerAvailability
- ecr:CompleteLayerUpload
- ecr:GetAuthorizationToken
- ecr:InitiateLayerUpload
- ecr:PutImage
- ecr:UploadLayerPart
- lightsail:*
Resource: '*'
- PolicyName: !Sub bkt-${ArtifactBucket}-${AWS::Region}
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action:
- s3:ListBucket
- s3:GetBucketLocation
- s3:ListBucketVersions
- s3:GetBucketVersioning
Resource:
- !Sub arn:aws:s3:::${ArtifactBucket}
- arn:aws:s3:::some-bucket
- PolicyName: !Sub obj-${ArtifactBucket}-${AWS::Region}
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
- s3:GetObjectAcl
- s3:PutObjectAcl
- s3:GetObjectTagging
- s3:PutObjectTagging
- s3:GetObjectVersion
- s3:GetObjectVersionAcl
- s3:PutObjectVersionAcl
Resource:
- !Sub arn:aws:s3:::${ArtifactBucket}/*
- arn:aws:s3:::some-bucket/*
CodeDeployServiceRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Sid: '1'
Effect: Allow
Principal:
Service:
- codedeploy.us-east-1.amazonaws.com
- codedeploy.eu-west-1.amazonaws.com
Action: sts:AssumeRole
Path: /
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSCodeDeployRoleForECS
- arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole
- arn:aws:iam::aws:policy/service-role/AWSCodeDeployRoleForLambda
CodeDeployRolePolicies:
Type: AWS::IAM::Policy
Properties:
PolicyName: !Sub 'CDPolicy-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Statement:
- Effect: Allow
Resource:
- '*'
Action:
- ec2:Describe*
- Effect: Allow
Resource:
- '*'
Action:
- autoscaling:CompleteLifecycleAction
- autoscaling:DeleteLifecycleHook
- autoscaling:DescribeLifecycleHooks
- autoscaling:DescribeAutoScalingGroups
- autoscaling:PutLifecycleHook
- autoscaling:RecordLifecycleActionHeartbeat
Roles:
- !Ref CodeDeployServiceRole
CodePipelineServiceRole:
Type: AWS::IAM::Role
Properties:
Path: /
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: codepipeline.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: !Sub 'root-${AWS::Region}-${AWS::StackName}'
PolicyDocument:
Version: '2012-10-17'
Statement:
- Resource:
- !Sub 'arn:aws:s3:::${ArtifactBucket}/*'
- !Sub 'arn:aws:s3:::${ArtifactBucket}'
Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketAcl
- s3:GetBucketLocation
- Resource: "*"
Effect: Allow
Action:
- ecs:*
- Resource: "*"
Effect: Allow
Action:
- iam:PassRole
Condition:
StringLike:
iam:PassedToService:
- ecs-tasks.amazonaws.com
- Resource: !GetAtt Build.Arn
Effect: Allow
Action:
- codebuild:BatchGetBuilds
- codebuild:StartBuild
- codebuild:BatchGetBuildBatches
- codebuild:StartBuildBatch
- Resource: !GetAtt Repo.Arn
Effect: Allow
Action:
- codecommit:CancelUploadArchive
- codecommit:GetBranch
- codecommit:GetCommit
- codecommit:GetRepository
- codecommit:GetUploadArchiveStatus
- codecommit:UploadArchive
AmazonCloudWatchEventRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Principal:
Service:
- events.amazonaws.com
Action: sts:AssumeRole
Path: /
Policies:
-
PolicyName: cwe-pipeline-execution
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Effect: Allow
Action: codepipeline:StartPipelineExecution
Resource: !Sub arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}
AmazonCloudWatchEventRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.codecommit
detail-type:
- CodeCommit Repository State Change
resources:
- !GetAtt Repo.Arn
detail:
event:
- referenceCreated
- referenceUpdated
referenceType:
- branch
referenceName:
- main
Targets:
-
Arn: !Sub arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}
RoleArn: !GetAtt AmazonCloudWatchEventRole.Arn
Id: codepipeline-Pipeline
Topic:
Type: AWS::SNS::Topic
Properties:
Subscription:
- Endpoint: user#example.com
Protocol: email
TopicPolicy:
Type: AWS::SNS::TopicPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
-
Sid: AllowPublish
Effect: Allow
Principal:
Service:
- 'codestar-notifications.amazonaws.com'
Action:
- 'SNS:Publish'
Resource:
- !Ref Topic
Topics:
- !Ref Topic
Repo:
Type: AWS::CodeCommit::Repository
Properties:
Code:
BranchName: main
S3:
Bucket: some-bucket
Key: code.zip
RepositoryName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
RepositoryDescription: Repository
Triggers:
- Name: Trigger
CustomData: The Code Repository
DestinationArn: !Ref Topic
Branches:
- main
Events: [all]
RepoUser:
Type: AWS::IAM::User
Properties:
Path: '/'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AWSCodeCommitPowerUser
RepoUserKey:
Type: AWS::IAM::AccessKey
Properties:
UserName:
!Ref RepoUser
Registry:
Type: AWS::ECR::Repository
Properties:
RepositoryName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
RepositoryPolicyText:
Version: '2012-10-17'
Statement:
- Sid: AllowPushPull
Effect: Allow
Principal:
AWS:
- !GetAtt CodeDeployServiceRole.Arn
Action:
- ecr:GetDownloadUrlForLayer
- ecr:BatchGetImage
- ecr:BatchCheckLayerAvailability
- ecr:PutImage
- ecr:InitiateLayerUpload
- ecr:UploadLayerPart
- ecr:CompleteLayerUpload
Build:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: CODEPIPELINE
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
phases:
pre_build:
commands:
- echo "[`date`] PRE_BUILD"
- echo "Logging in to Amazon ECR..."
- aws ecr get-login-password --region $REGION | docker login --username AWS --password-stdin $ACCOUNT.dkr.ecr.$REGION.amazonaws.com
- IMAGE_URI="$ACCOUNT.dkr.ecr.$REGION.amazonaws.com/$REPO:$TAG"
build:
commands:
- echo "[`date`] BUILD"
- echo "Building Docker Image..."
- docker build -t $REPO:$TAG .
- docker tag $REPO:$TAG $IMAGE_URI
post_build:
commands:
- echo "[`date`] POST_BUILD"
- echo "Pushing Docker Image..."
- docker push $IMAGE_URI
- echo Writing image definitions file...
- printf '[{"name":"svc","imageUri":"%s"}]' $IMAGE_URI > $FILE
artifacts:
files: $FILE
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/standard:6.0
Type: LINUX_CONTAINER
EnvironmentVariables:
- Name: REGION
Type: PLAINTEXT
Value: !Ref AWS::Region
- Name: ACCOUNT
Type: PLAINTEXT
Value: !Ref AWS::AccountId
- Name: TAG
Type: PLAINTEXT
Value: latest
- Name: REPO
Type: PLAINTEXT
Value: !Ref Registry
- Name: FILE
Type: PLAINTEXT
Value: !Ref ImagesFile
PrivilegedMode: true
Name: !Ref AWS::StackName
ServiceRole: !GetAtt CodeBuildServiceRole.Arn
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
RoleArn: !GetAtt CodePipelineServiceRole.Arn
ArtifactStore:
Type: S3
Location: !Ref ArtifactBucket
Stages:
- Name: Source
Actions:
- Name: Site
ActionTypeId:
Category: Source
Owner: AWS
Version: '1'
Provider: CodeCommit
Configuration:
RepositoryName: !GetAtt Repo.Name
BranchName: main
PollForSourceChanges: 'false'
InputArtifacts: []
OutputArtifacts:
- Name: SourceArtifact
RunOrder: 1
- Name: Build
Actions:
- Name: Docker
ActionTypeId:
Category: Build
Owner: AWS
Version: '1'
Provider: CodeBuild
Configuration:
ProjectName: !Ref Build
InputArtifacts:
- Name: SourceArtifact
OutputArtifacts:
- Name: BuildArtifact
RunOrder: 1
- Name: Deploy
Actions:
- Name: Fargate
ActionTypeId:
Category: Deploy
Owner: AWS
Version: '1'
Provider: ECS
Configuration:
ClusterName: !Ref Cluster
FileName: !Ref ImagesFile
ServiceName: !GetAtt Service.Name
InputArtifacts:
- Name: BuildArtifact
RunOrder: 1
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
FargateTaskExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
TaskRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
-
Name: svc
Image: !Sub ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${Registry}:latest
PortMappings:
- ContainerPort: 8080
Cpu: 256
ExecutionRoleArn: !Ref FargateTaskExecutionRole
Memory: 512
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
RuntimePlatform:
CpuArchitecture: ARM64
OperatingSystemFamily: LINUX
TaskRoleArn: !Ref TaskRole
Service:
Type: AWS::ECS::Service
Properties:
Cluster: !Ref Cluster
DesiredCount: 0
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
SecurityGroups:
- !Ref SecG
Subnets: !Ref Subs
ServiceName: !Select [4, !Split ['-', !Select [2, !Split ['/', !Ref AWS::StackId]]]]
TaskDefinition: !Ref TaskDefinition
Outputs:
ArtifactBucketName:
Description: ArtifactBucket S3 Bucket Name
Value: !Ref ArtifactBucket
ArtifactBucketSecureUrl:
Description: ArtifactBucket S3 Bucket Domain Name
Value: !Sub 'https://${ArtifactBucket.DomainName}'
ClusterName:
Value: !Ref Cluster
ServiceName:
Value: !GetAtt Service.Name
RepoUserAccessKey:
Description: S3 User Access Key
Value: !Ref RepoUserKey
RepoUserSecretKey:
Description: S3 User Secret Key
Value: !GetAtt RepoUserKey.SecretAccessKey
BuildArn:
Description: CodeBuild URL
Value: !GetAtt Build.Arn
RepoArn:
Description: CodeCommit Repository ARN
Value: !GetAtt Repo.Arn
RepoName:
Description: CodeCommit Repository NAme
Value: !GetAtt Repo.Name
RepoCloneUrlHttp:
Description: CodeCommit HTTP Clone URL
Value: !GetAtt Repo.CloneUrlHttp
RepoCloneUrlSsh:
Description: CodeCommit SSH Clone URL
Value: !GetAtt Repo.CloneUrlSsh
PipelineUrl:
Description: CodePipeline URL
Value: !Sub https://console.aws.amazon.com/codepipeline/home?region=${AWS::Region}#/view/${Pipeline}
RegistryUri:
Description: ECR Repository URI
Value: !GetAtt Registry.RepositoryUri
TopicArn:
Description: CodeCommit Notification SNS Topic ARN
Value: !Ref Topic
Hope this helps!

CloudFormation Combine Sub - Import - Sub

I am trying to create a parameter and would like to combine !Sub and !Import several times.
Parameters:
Environment:
Description: Stackname of Environment
Type: String
Resources:
IAMRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: '*'
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: S3Files
PolicyDocument:
Statement:
- Sid: 'S3Files'
Effect: Allow
Action:
- 's3:DeleteObjectTagging'
- 's3:GetObjectRetention'
- 's3:ListMultipartUploadParts'
- 's3:PutObject'
- 's3:GetObjectAcl'
- 's3:GetObject'
- 's3:AbortMultipartUpload'
- 's3:PutObjectRetention'
- 's3:GetObjectVersionAcl'
- 's3:GetObjectTagging'
- 's3:PutObjectTagging'
- 's3:DeleteObject'
- 's3:PutObjectAcl'
- 's3:GetObjectVersion'
Resource: !Sub
- '${ARN}/*'
- ARN:
Fn::ImportValue: !Sub ${Environment}:S3:Arn
According to the documentation it should be possible, but unfortunately I always get an error message
Template contains errors.: [/Resources/IAMRole/Type/Policies/0/PolicyDocument/Statement/0/Resource/Fn::Sub/1/ARN] 'null' values are not allowed in templates
How could the UseCase work?
There is a space issue in the Resource section.
Resource: !Sub
- '${ARN}/*'
- ARN:
Fn::ImportValue: !Sub ${Environment}:S3:Arn
It should be
Resource: !Sub
- '${ARN}/*'
- ARN:
Fn::ImportValue: !Sub ${Environment}:S3:Arn
Note: Fn starts under N of ARN instead of A.
Explanation: With the first indentation the line with Fn::ImportValue is considered as an input for !Sub, with the second indentation it becomes the value for ARN: defined the line above it.
Side note: Use 2 spaces or 4 spaces or tabs uniformly throughout the template.

is it possible to create Kubernetes pods, services, replica controllers etc on AWS cloudfromation?

does AWS cloudformation supports creation of Kubernetes pods, services, replica controllers etc or setting up the EKS clusters and worker nodes and using Kubectl to create the resources are the only way?
Not out of the box, but you can if you use a custom resource type backed by a lambda function in CloudFormation.
The AWS EKS quickstart has an example:
AWSTemplateFormatVersion: "2010-09-09"
Description: "deploy an example workload into an existing kubernetes cluster (qs-1p817r5f9)"
Parameters:
KubeConfigPath:
Type: String
KubeConfigKmsContext:
Type: String
Default: "EKSQuickStart"
KubeClusterName:
Type: String
NodeInstanceProfile:
Type: String
QSS3BucketName:
AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$
ConstraintDescription: Quick Start bucket name can include numbers, lowercase
letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen
(-).
Default: aws-quickstart
Description: S3 bucket name for the Quick Start assets. This string can include
numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start
or end with a hyphen (-).
Type: String
QSS3KeyPrefix:
AllowedPattern: ^[0-9a-zA-Z-/.]*$
ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters,
uppercase letters, hyphens (-), dots(.) and forward slash (/).
Default: quickstart-amazon-eks/
Description: S3 key prefix for the Quick Start assets. Quick Start key prefix
can include numbers, lowercase letters, uppercase letters, hyphens (-), dots(.) and
forward slash (/).
Type: String
QSS3BucketRegion:
Default: 'us-east-1'
Description: The AWS Region where the Quick Start S3 bucket (QSS3BucketName) is
hosted. When using your own bucket, you must specify this value.
Type: String
LambdaZipsBucketName:
Description: 'OPTIONAL: Bucket Name where the lambda zip files should be placed,
if left blank a bucket will be created.'
Type: String
Default: ''
K8sSubnetIds:
Type: List<AWS::EC2::Subnet::Id>
VPCID:
Type: AWS::EC2::VPC::Id
ControlPlaneSecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
Conditions:
CreateLambdaZipsBucket: !Equals
- !Ref 'LambdaZipsBucketName'
- ''
UsingDefaultBucket: !Equals [!Ref QSS3BucketName, 'aws-quickstart']
Resources:
WorkloadStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub
- 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}templates/example-workload.template.yaml'
- S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion]
S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
Parameters:
KubeManifestLambdaArn: !GetAtt KubeManifestLambda.Arn
HelmLambdaArn: !GetAtt HelmLambda.Arn
KubeConfigPath: !Ref KubeConfigPath
KubeConfigKmsContext: !Ref KubeConfigKmsContext
KubeClusterName: !Ref KubeClusterName
NodeInstanceProfile: !Ref NodeInstanceProfile
CopyZips:
Type: Custom::CopyZips
Properties:
ServiceToken: !GetAtt 'CopyZipsFunction.Arn'
DestBucket: !Ref LambdaZipsBucketName
SourceBucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
Prefix: !Ref 'QSS3KeyPrefix'
Objects:
- functions/packages/Helm/lambda.zip
- functions/packages/DeleteBucketContents/lambda.zip
- functions/packages/KubeManifest/lambda.zip
- functions/packages/LambdaEniCleanup/lambda.zip
VPCLambdaCleanup:
Type: Custom::LambdaCleanup
Properties:
ServiceToken: !GetAtt VPCLambdaCleanupLambdaFunction.Arn
Region: !Ref "AWS::Region"
LambdaFunctionNames:
- !Ref KubeManifestLambda
VPCLambdaCleanupLambdaFunction:
DependsOn: CopyZips
Type: "AWS::Lambda::Function"
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt LambdaCleanUpFunctionRole.Arn
Runtime: python3.7
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/LambdaEniCleanup/lambda.zip'
HelmLambda:
DependsOn: CopyZips
Type: AWS::Lambda::Function
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt ManifestRole.Arn
Runtime: python3.6
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/Helm/lambda.zip'
VpcConfig:
SecurityGroupIds: [ !Ref EKSLambdaSecurityGroup ]
SubnetIds: !Ref K8sSubnetIds
KubeManifestLambda:
DependsOn: CopyZips
Type: AWS::Lambda::Function
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt ManifestRole.Arn
Runtime: python3.6
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/KubeManifest/lambda.zip'
VpcConfig:
SecurityGroupIds: [ !Ref EKSLambdaSecurityGroup ]
SubnetIds: !Ref K8sSubnetIds
DeleteBucketContentsLambda:
DependsOn: CopyZips
Type: AWS::Lambda::Function
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt DeleteBucketContentsRole.Arn
Runtime: python3.7
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/DeleteBucketContents/lambda.zip'
CopyZipsFunction:
Type: AWS::Lambda::Function
Properties:
Description: Copies objects from a source S3 bucket to a destination
Handler: index.handler
Runtime: python3.7
Role: !GetAtt CopyZipsRole.Arn
Timeout: 900
Code:
ZipFile: |
import json
import logging
import threading
import boto3
import cfnresponse
def copy_objects(source_bucket, dest_bucket, prefix, objects):
s3 = boto3.client('s3')
for o in objects:
key = prefix + o
copy_source = {
'Bucket': source_bucket,
'Key': key
}
print('copy_source: %s' % copy_source)
print('dest_bucket = %s'%dest_bucket)
print('key = %s' %key)
s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,
Key=key)
def delete_objects(bucket, prefix, objects):
s3 = boto3.client('s3')
objects = {'Objects': [{'Key': prefix + o} for o in objects]}
s3.delete_objects(Bucket=bucket, Delete=objects)
def timeout(event, context):
logging.error('Execution is about to time out, sending failure response to CloudFormation')
cfnresponse.send(event, context, cfnresponse.FAILED, {}, physical_resource_id)
def handler(event, context):
physical_resource_id = None
if "PhysicalResourceId" in event.keys():
physical_resource_id = event["PhysicalResourceId"]
# make sure we send a failure to CloudFormation if the function is going to timeout
timer = threading.Timer((context.get_remaining_time_in_millis()
/ 1000.00) - 0.5, timeout, args=[event, context])
timer.start()
print('Received event: %s' % json.dumps(event))
status = cfnresponse.SUCCESS
try:
source_bucket = event['ResourceProperties']['SourceBucket']
dest_bucket = event['ResourceProperties']['DestBucket']
prefix = event['ResourceProperties']['Prefix']
objects = event['ResourceProperties']['Objects']
if event['RequestType'] == 'Delete':
delete_objects(dest_bucket, prefix, objects)
else:
copy_objects(source_bucket, dest_bucket, prefix, objects)
except Exception as e:
logging.error('Exception: %s' % e, exc_info=True)
status = cfnresponse.FAILED
finally:
timer.cancel()
cfnresponse.send(event, context, status, {}, physical_resource_id)
LambdaZipsBucket:
Type: AWS::S3::Bucket
Condition: CreateLambdaZipsBucket
LambdaCleanUpFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [lambda.amazonaws.com]
Version: '2012-10-17'
Path: /
Policies:
- PolicyName: LambdaRole
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Effect: Allow
Resource: !Sub "arn:${AWS::Partition}:logs:*:*:*"
- Action:
- 'ec2:*'
Effect: Allow
Resource: "*"
DeleteBucketContentsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- !Sub 'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
Policies:
- PolicyName: deletebucketcontents
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: s3:*
Resource:
- !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}/*'
- !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}'
CopyZipsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- !Su 'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
Policies:
- PolicyName: lambda-copier
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: s3:GetObject
Resource: !Sub
- 'arn:${AWS::Partition}:s3:::${S3Bucket}/${QSS3KeyPrefix}*'
- S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
- Effect: Allow
Action:
- s3:PutObject
- s3:DeleteObject
Resource: !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}/${QSS3KeyPrefix}*'
ManifestRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: eksStackPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: s3:GetObject
Resource: !Sub
- "arn:${AWS::Partition}:s3:::${BucketName}/*"
- S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
Resource:
- "*"
- Action: "kms:decrypt"
Effect: Allow
Resource: "*"
- Action: "s3:GetObject"
Effect: Allow
Resource: "*"
EKSLambdaSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for lambda to communicate with cluster API
VpcId: !Ref VPCID
ClusterControlPlaneSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
Description: Allow lambda to communicate with the cluster API Server
GroupId: !Ref ControlPlaneSecurityGroup
SourceSecurityGroupId: !Ref EKSLambdaSecurityGroup
IpProtocol: tcp
ToPort: 443
FromPort: 443
It works by creating a lambda function customer resource KubeManifestLambda and HelmLambda that has kubectl and helm installed respectively, both configured with a role that allows them to access the EKS k8s cluster.
Then these custom resources can be used to deploy k8s manifests and helm charts with custom values, like in this example.
KubeManifestExample:
Type: "Custom::KubeManifest"
Version: '1.0'
Properties:
# The lambda function that executes the manifest against the cluster. This is created in one of the parent stacks
ServiceToken: !Ref KubeManifestLambdaArn
# S3 path to the encrypted config file eg. s3://my-bucket/kube/config.encrypted
KubeConfigPath: !Ref KubeConfigPath
# context for KMS to use when decrypting the file
KubeConfigKmsContext: !Ref KubeConfigKmsContext
# Kubernetes manifest
Manifest:
apiVersion: v1
kind: ConfigMap
metadata:
# If name is not specified it will be automatically generated,
# and can be retrieved with !GetAtt LogicalID.name
#
# name: test
#
# if namespace is not specified, "default" namespace will be used
namespace: kube-system
data:
# examples of consuming outputs of the HelmExample resource below's output. Creates an implicit dependency, so
# this resource will only launch once the HelmExample resource has completed successfully
ServiceCatalogReleaseName: !Ref HelmExample
ServiceCatalogKubernetesServiceName: !GetAtt HelmExample.Service0
This even lets you reference other Cloud formation resources such as RDS instances that are created as part of a workload.
You can use CloudFormation to create EKS cluster and worker nodes but you have to use kubectl for any operation on cluster like creating service, pods, deployments etc.....you can’t use CloudFormation for that
If you use CDK, you can use cluster.add_helm_chart() or HelmChart class. It will create a lambda behind the scenes.
Or you can create a lambda directly with https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.lambda_layer_kubectl/README.html

cloudtrail log using cloudformation template

In cloud-trail, I can select the existing log group CloudTrail/DefaultLogGroup under CloudWatch Logs section. Is it possible to complete this step using cloudformation Template?
Assuming you are creating the log group with CloudFormation as well:
LogGroup: # A new log group
Type: AWS::Logs::LogGroup
Properties:
RetentionInDays: 365 # optional
CloudTrailLogsRole: # A role for your trail
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: cloudtrail.amazonaws.com
Version: '2012-10-17'
CloudTrailLogsPolicy: # The policy for your role
Type: AWS::IAM::Policy
Properties:
PolicyDocument:
Statement:
- Action:
- logs:PutLogEvents
- logs:CreateLogStream
Effect: Allow
Resource:
Fn::GetAtt:
- LogGroup
- Arn
Version: '2012-10-17'
PolicyName: DefaultPolicy
Roles:
- Ref: CloudTrailLogsRole
CloudTrail: # The trail
Type: AWS::CloudTrail::Trail
Properties:
IsLogging: true
CloudWatchLogsLogGroupArn:
Fn::GetAtt:
- LogGroup
- Arn
CloudWatchLogsRoleArn:
Fn::GetAtt:
- CloudTrailLogsRole
- Arn
DependsOn:
- CloudTrailLogsPolicy
- CloudTrailLogsRole
If using an existing log group:
CloudTrailLogsRole: # A role for your trail
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: cloudtrail.amazonaws.com
Version: '2012-10-17'
CloudTrailLogsPolicy: # The policy for your role
Type: AWS::IAM::Policy
Properties:
PolicyDocument:
Statement:
- Action:
- logs:PutLogEvents
- logs:CreateLogStream
Effect: Allow
Resource: <your existing log group arn here>
Version: '2012-10-17'
PolicyName: DefaultPolicy
Roles:
- Ref: CloudTrailLogsRole
CloudTrail: # The trail
Type: AWS::CloudTrail::Trail
Properties:
IsLogging: true
CloudWatchLogsLogGroupArn: <your existing log group arn here>
CloudWatchLogsRoleArn:
Fn::GetAtt:
- CloudTrailLogsRole
- Arn
DependsOn:
- CloudTrailLogsPolicy
- CloudTrailLogsRole