create service using cloudformation in aws ecs fargate "did not stabilize" - aws-cloudformation

AWSTemplateFormatVersion: "2010-09-09"
Description: Not Have Any Idea
Parameters:
Service:
Description: Service Name
Type: String
Cluster:
Description: Cluster Name
Type: String
TaskDefinition:
Description: TaskDefinition Name
Type: String
securitygroup:
Description: securitygroup
Type: AWS::EC2::SecurityGroup::Id
SubnetId:
Type: List<AWS::EC2::Subnet::Id>
Description: Select at two subnets in your selected VPC.
Resources:
sernginx:
Type: "AWS::ECS::Service"
Properties:
ServiceName:
Ref: Service
LaunchType: "FARGATE"
DesiredCount: 1
Cluster:
Ref: Cluster
TaskDefinition:
Ref: TaskDefinition
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 70
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- Ref: securitygroup
Subnets:
Ref: SubnetId
I am using this code to create aws ecs service fargate
I am getting ROLLBACK_COMPLETE status with an error message saying
Service arn:aws:ecs:us-east-1:439138162442:service/yuiyiuyiu did not stabilize

I've used this kind of template for creating fargate task.
AWSTemplateFormatVersion: '2010-09-09'
Description: Docker Core App service
Parameters:
EnvironmentName:
Type: String
Default: coreapp
Description: A name for the environment that this cloudformation will be part of.
ServiceName:
Type: String
Default: coreapi
Description: A name for the service
ImageUrl:
Type: String
Default: your_image_URI
Description: The url of a docker image
ContainerPort:
Type: Number
Default: 5000
Description: What port number the application inside the docker container
ContainerCpu:
Type: Number
Default: 256
Description: How much CPU to give the container.
ContainerMemory:
Type: Number
Default: 512
Description: How much memory in megabytes to give the container
DesiredCount:
Type: Number
Default: 2
Description: How many copies of the service task to run
Role:
Type: String
Default: ""
Description: (Optional) An IAM role to give the service's containers
Resources:
Service:
Type: AWS::ECS::Service
Properties:
ServiceName: !Ref 'ServiceName'
Cluster:
- coreapp
LaunchType: FARGATE
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 75
DesiredCount: !Ref 'DesiredCount'
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
SecurityGroups:
- sg-483aa03e
Subnets:
- subnet-2ce41866
- subnet-ceefe5aa
TaskDefinition: !Ref 'TaskDefinition'

Related

how to pass bucket/key name to fargate job via a cloudwatch event trigger on s3 object creation event?

I have create a fargate task and trying to trigger it via s3 object creation event ( see sample below) via cloudformation.as it cannot trigger it directly, i have created a cloudwatchevent. I am trying to pass the bucket and obj name to my fargate task code . doing some research, i came across -> InputTransformer, but i'm not sure how to pass the value of my bucket and key name and how to read it in my python code. any help will be appreciated.
AWSTemplateFormatVersion: 2010-09-09
Description: An example CloudFormation template for Fargate.
Parameters:
VPC:
Type: AWS::EC2::VPC::Id
SubnetA:
Type: AWS::EC2::Subnet::Id
SubnetB:
Type: AWS::EC2::Subnet::Id
Image:
Type: String
Default: 123456789012.dkr.ecr.region.amazonaws.com/image:tag
Resources:
mybucket:
Properties:
BucketName: 'mytestbucket-us'
cloudwatchEvent:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.s3
detail:
eventSource:
- s3.amazonaws.com
eventName:
- PutObject
- CompleteMultipartUpload
requestParameters:
bucketName:
- !Ref mybucket
Targets:
- Id: my-fargate-task
Arn: myclusterArn
RoleArn: myinvocationrolearn
Input:
'Fn::Sub':
- >-
{"containerOverrides": [{"name":"somecontainer"]}
EcsParameters:
TaskDefinition:
LaunchType: 'FARGATE'
...
NetworkConfiguration:
...
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Join ['', [!Ref ServiceName, Cluster]]
TaskDefinition:
Type: AWS::ECS::TaskDefinition
DependsOn: LogGroup
Properties:
Family: !Join ['', [!Ref ServiceName, TaskDefinition]]
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 256
Memory: 2GB
ExecutionRoleArn: !Ref ExecutionRole
TaskRoleArn: !Ref TaskRole
ContainerDefinitions:
- Name: !Ref ServiceName
Image: !Ref Image
# A role needed by ECS
ExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, ExecutionRole]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
# A role for the containers
TaskRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, TaskRole]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
You would use a CloudWatch Event Input Transformer to extract the data you need from the event, and pass that data to the ECS task as environment variable(s) in the target's ContainerOverrides. I don't use CloudFormation, but here's an example using Terraform.
You can't. CloudWatch events do not pass data to ECS jobs. You need to develop your own mechanism for that. For example, trigger lambda first, store event in S3 Parameter Store or DynamoDB, and then invoke your ECS job which will get stored data.

AWS batch cloudformation - “CannotPullContainerError”

I have a cloud Formation template for a AWS Batch POC with 6 resources.
3 AWS::IAM::Role
1 AWS::Batch::ComputeEnvironment
1 AWS::Batch::JobQueue
1 AWS::Batch::JobDefinition
The AWS::IAM::Role have the policy "arn:aws:iam::aws:policy/AdministratorAccess" (In order to avoid issues.)
The roles are used:
1 into the AWS::Batch::ComputeEnvironment
2 into the AWS::Batch::JobDefinition
But even with the policy "arn:aws:iam::aws:policy/AdministratorAccess" I get "CannotPullContainerError: Error response from daemon: Get https://********.dkr.ecr.eu-west-1.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" when I rin a job.
Disclainer: All is FARGATE (Compute enviroment and Job), not EC2
AWSTemplateFormatVersion: '2010-09-09'
Description: Creates a POC AWS Batch environment.
Parameters:
Environment:
Type: String
Description: 'Environment Name'
Default: TEST
Subnets:
Type: List<AWS::EC2::Subnet::Id>
Description: 'List of Subnets to boot into'
ImageName:
Type: String
Description: 'Name and tag of Process Container Image'
Default: 'upload:6.0.0'
Resources:
BatchServiceRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Join ['', ['Demo', BatchServiceRole]]
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: 'Allow'
Principal:
Service: 'batch.amazonaws.com'
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
BatchContainerRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Join ['', ['Demo', BatchContainerRole]]
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
-
Effect: 'Allow'
Principal:
Service:
- 'ecs-tasks.amazonaws.com'
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
BatchJobRole:
Type: 'AWS::IAM::Role'
Properties:
RoleName: !Join ['', ['Demo', BatchJobRole]]
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: 'Allow'
Principal:
Service: 'ecs-tasks.amazonaws.com'
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AdministratorAccess'
BatchCompute:
Type: "AWS::Batch::ComputeEnvironment"
Properties:
ComputeEnvironmentName: DemoContentInput
ComputeResources:
MaxvCpus: 256
SecurityGroupIds:
- sg-0b33333333333333
Subnets: !Ref Subnets
Type: FARGATE
ServiceRole: !Ref BatchServiceRole
State: ENABLED
Type: Managed
Queue:
Type: "AWS::Batch::JobQueue"
DependsOn: BatchCompute
Properties:
ComputeEnvironmentOrder:
- ComputeEnvironment: DemoContentInput
Order: 1
Priority: 1
State: "ENABLED"
JobQueueName: DemoContentInput
ContentInputJob:
Type: "AWS::Batch::JobDefinition"
Properties:
Type: Container
ContainerProperties:
Command:
- -v
- process
- new-file
- -o
- s3://contents/{content_id}/{content_id}.mp4
Environment:
- Name: SECRETS
Value: !Join [ ':', [ '{{resolve:secretsmanager:common.secrets:SecretString:aws_access_key_id}}', '{{resolve:secretsmanager:common.secrets:SecretString:aws_secret_access_key}}' ] ]
- Name: APPLICATION
Value: upload
- Name: API_KEY
Value: '{{resolve:secretsmanager:common.secrets:SecretString:fluzo.api_key}}'
- Name: CLIENT
Value: upload-container
- Name: ENVIRONMENT
Value: !Ref Environment
- Name: SETTINGS
Value: !Join [ ':', [ '{{resolve:secretsmanager:common.secrets:SecretString:aws_access_key_id}}', '{{resolve:secretsmanager:common.secrets:SecretString:aws_secret_access_key}}', 'upload-container' ] ]
ExecutionRoleArn: 'arn:aws:iam::**********:role/DemoBatchJobRole'
Image: !Join ['', [!Ref 'AWS::AccountId','.dkr.ecr.', !Ref 'AWS::Region', '.amazonaws.com/', !Ref ImageName ] ]
JobRoleArn: !Ref BatchContainerRole
ResourceRequirements:
- Type: VCPU
Value: 1
- Type: MEMORY
Value: 2048
JobDefinitionName: DemoContentInput
PlatformCapabilities:
- FARGATE
RetryStrategy:
Attempts: 1
Timeout:
AttemptDurationSeconds: 600
Into AWS::Batch::JobQueue:ContainerProperties:ExecutionRoleArn I harcoded the arn because if write !Ref BatchJobRole I get an error. But it's no my goal with this question.
The question is how to avoid "CannotPullContainerError: Error response from daemon: Get https://********.dkr.ecr.eu-west-1.amazonaws.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" when I run a Job.
It sounds like you can't reach the internet from inside your subnet.
Make sure:
There is an internet gateway device associated with your VPC (create one if there isn't -- even if you are just using nat-gateway for egress)
The route table that is associated with your subnet has a default route (0.0.0./0) to an internet gateway or nat-gateway with an attached elastic-ip.
An attached security group has rules allowing outbound internet traffic (0.0.0.0/0) for your ports and protocols. (e.g. 80/http, 443/https)
The network access control list (network ACL) that is associated with the subnet has rules allowing both outbound and inbound traffic to the internet.
References:
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-connect-internet-gateway/

The target group with targetGroupArn does not have associated load balancer error

I'm running two scripts, one that creates a cluster with the required resources, and then another script to create the tasks and service. The first script creates everything correctly, but when I run the second script to create the service and task, it gives the following error:
The target group with targetGroupArn arn:aws:elasticloadbalancing:ca-central-1:XXXXXXXXX:targetgroup/XXXXXXXXXXXXXXXXXXXXX does not have an associated load balancer. (Service: AmazonECS; Status Code: 400; Error Code: InvalidParameterException; Request ID: 3899fd23-3eee-473f-9914-453a8d669f14)
What am I doing wrong?
Script 1 to create cluster:
AWSTemplateFormatVersion: '2010-09-09'
Description: A stack for deploying containerized applications onto a cluster of EC2
hosts using Elastic Container Service. This stack runs containers on
hosts that are in a public VPC subnet, and includes a public facing load
balancer to register the services in.
Parameters:
DesiredCapacity:
Type: Number
Default: '3'
Description: Number of EC2 instances to launch in your ECS cluster.
MaxSize:
Type: Number
Default: '6'
Description: Maximum number of EC2 instances that can be launched in your ECS cluster.
ECSAMI:
Description: AMI ID
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id
InstanceType:
Description: EC2 instance type
Type: String
Default: t2.micro
AllowedValues: [t2.micro, t2.small, t2.medium, t2.large, m3.medium, m3.large,
m3.xlarge, m3.2xlarge, m4.large, m4.xlarge, m4.2xlarge, m4.4xlarge, m4.10xlarge,
c4.large, c4.xlarge, c4.2xlarge, c4.4xlarge, c4.8xlarge, c3.large, c3.xlarge,
c3.2xlarge, c3.4xlarge, c3.8xlarge, r3.large, r3.xlarge, r3.2xlarge, r3.4xlarge,
r3.8xlarge, i2.xlarge, i2.2xlarge, i2.4xlarge, i2.8xlarge]
ConstraintDescription: Please choose a valid instance type.
Mappings:
# Hard values for the subnet masks. These masks define
# the range of internal IP addresses that can be assigned.
# The VPC can have all IP's from 10.0.0.0 to 10.0.255.255
# There are two subnets which cover the ranges:
#
# 10.0.0.0 - 10.0.0.255
# 10.0.1.0 - 10.0.1.255
#
# If you need more IP addresses (perhaps you have so many
# instances that you run out) then you can customize these
# ranges to add more
SubnetConfig:
VPC:
CIDR: '10.0.0.0/16'
PublicOne:
CIDR: '10.0.0.0/24'
PublicTwo:
CIDR: '10.0.1.0/24'
Resources:
# VPC in which containers will be networked.
# It has two public subnets
# We distribute the subnets across the first two available subnets
# for the region, for high availability.
VPC:
Type: AWS::EC2::VPC
Properties:
EnableDnsSupport: true
EnableDnsHostnames: true
CidrBlock: !FindInMap ['SubnetConfig', 'VPC', 'CIDR']
# Two public subnets, where containers can have public IP addresses
PublicSubnetOne:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 0
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !FindInMap ['SubnetConfig', 'PublicOne', 'CIDR']
MapPublicIpOnLaunch: true
PublicSubnetTwo:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone:
Fn::Select:
- 1
- Fn::GetAZs: {Ref: 'AWS::Region'}
VpcId: !Ref 'VPC'
CidrBlock: !FindInMap ['SubnetConfig', 'PublicTwo', 'CIDR']
MapPublicIpOnLaunch: true
# Setup networking resources for the public subnets. Containers
# in the public subnets have public IP addresses and the routing table
# sends network traffic via the internet gateway.
InternetGateway:
Type: AWS::EC2::InternetGateway
GatewayAttachement:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref 'VPC'
InternetGatewayId: !Ref 'InternetGateway'
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref 'VPC'
PublicRoute:
Type: AWS::EC2::Route
DependsOn: GatewayAttachement
Properties:
RouteTableId: !Ref 'PublicRouteTable'
DestinationCidrBlock: '0.0.0.0/0'
GatewayId: !Ref 'InternetGateway'
PublicSubnetOneRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetOne
RouteTableId: !Ref PublicRouteTable
PublicSubnetTwoRouteTableAssociation:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnetTwo
RouteTableId: !Ref PublicRouteTable
# ECS Resources
ECSCluster:
Type: AWS::ECS::Cluster
# A security group for the EC2 hosts that will run the containers.
# Two rules, allowing network traffic from a public facing load
# balancer and from other hosts in the security group.
#
# Remove any of the following ingress rules that are not needed.
# If you want to make direct requests to a container using its
# public IP address you'll need to add a security group rule
# to allow traffic from all IP addresses.
EcsHostSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Access to the ECS hosts that run containers
VpcId: !Ref 'VPC'
EcsSecurityGroupIngressFromPublicALB:
Type: AWS::EC2::SecurityGroupIngress
Properties:
Description: Ingress from the public ALB
GroupId: !Ref 'EcsHostSecurityGroup'
IpProtocol: -1
SourceSecurityGroupId: !Ref 'PublicLoadBalancerSG'
EcsSecurityGroupIngressFromSelf:
Type: AWS::EC2::SecurityGroupIngress
Properties:
Description: Ingress from other hosts in the same security group
GroupId: !Ref 'EcsHostSecurityGroup'
IpProtocol: -1
SourceSecurityGroupId: !Ref 'EcsHostSecurityGroup'
# Autoscaling group. This launches the actual EC2 instances that will register
# themselves as members of the cluster, and run the docker containers.
ECSAutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
VPCZoneIdentifier:
- !Ref PublicSubnetOne
- !Ref PublicSubnetTwo
LaunchConfigurationName: !Ref 'ContainerInstances'
MinSize: '1'
MaxSize: !Ref 'MaxSize'
DesiredCapacity: !Ref 'DesiredCapacity'
CreationPolicy:
ResourceSignal:
Timeout: PT15M
UpdatePolicy:
AutoScalingReplacingUpdate:
WillReplace: 'true'
ContainerInstances:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
ImageId: !Ref 'ECSAMI'
SecurityGroups: [!Ref 'EcsHostSecurityGroup']
InstanceType: !Ref 'InstanceType'
IamInstanceProfile: !Ref 'EC2InstanceProfile'
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
yum install -y aws-cfn-bootstrap
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource ECSAutoScalingGroup --region ${AWS::Region}
AutoscalingRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [application-autoscaling.amazonaws.com]
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: service-autoscaling
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'application-autoscaling:*'
- 'cloudwatch:DescribeAlarms'
- 'cloudwatch:PutMetricAlarm'
- 'ecs:DescribeServices'
- 'ecs:UpdateService'
Resource: '*'
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles: [!Ref 'EC2Role']
# Role for the EC2 hosts. This allows the ECS agent on the EC2 hosts
# to communciate with the ECS control plane, as well as download the docker
# images from ECR to run on your host.
EC2Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [ec2.amazonaws.com]
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: ecs-service
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'ecs:CreateCluster'
- 'ecs:DeregisterContainerInstance'
- 'ecs:DiscoverPollEndpoint'
- 'ecs:Poll'
- 'ecs:RegisterContainerInstance'
- 'ecs:StartTelemetrySession'
- 'ecs:Submit*'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
- 'ecr:GetAuthorizationToken'
- 'ecr:BatchGetImage'
- 'ecr:GetDownloadUrlForLayer'
Resource: '*'
# Load balancers for getting traffic to containers.
# This sample template creates one load balancer:
#
# - One public load balancer, hosted in public subnets that is accessible
# to the public, and is intended to route traffic to one or more public
# facing services.
# A public facing load balancer, this is used for accepting traffic from the public
# internet and directing it to public facing microservices
PublicLoadBalancerSG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Access to the public facing load balancer
VpcId: !Ref 'VPC'
SecurityGroupIngress:
# Allow access to ALB from anywhere on the internet
- CidrIp: 0.0.0.0/0
IpProtocol: -1
PublicLoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Scheme: internet-facing
#LoadBalancerAttributes:
# - Key: idle_timeout.timeout_seconds
# Value: '30'
Subnets:
# The load balancer is placed into the public subnets, so that traffic
# from the internet can reach the load balancer directly via the internet gateway
- !Ref PublicSubnetOne
- !Ref PublicSubnetTwo
#SecurityGroups: [!Ref 'PublicLoadBalancerSG']
Type: network
# A dummy target group is used to setup the ALB to just drop traffic
# initially, before any real service target groups have been added.
DummyTargetGroupPublic:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckIntervalSeconds: 10
#HealthCheckPath: /
HealthCheckProtocol: TCP
#HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 2
Name: !Join ['-', [!Ref 'AWS::StackName', 'drop-1']]
Port: 80
Protocol: TCP
UnhealthyThresholdCount: 2
VpcId: !Ref 'VPC'
PublicLoadBalancerListener:
Type: AWS::ElasticLoadBalancingV2::Listener
DependsOn:
- PublicLoadBalancer
Properties:
DefaultActions:
- TargetGroupArn: !Ref 'DummyTargetGroupPublic'
Type: 'forward'
LoadBalancerArn: !Ref 'PublicLoadBalancer'
Port: 80
Protocol: TCP
# This is an IAM role which authorizes ECS to manage resources on your
# account on your behalf, such as updating your load balancer with the
# details of where your containers are, so that traffic can reach your
# containers.
ECSRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: [ecs.amazonaws.com]
Action: ['sts:AssumeRole']
Path: /
Policies:
- PolicyName: ecs-service
PolicyDocument:
Statement:
- Effect: Allow
Action:
# Rules which allow ECS to attach network interfaces to instances
# on your behalf in order for awsvpc networking mode to work right
- 'ec2:AttachNetworkInterface'
- 'ec2:CreateNetworkInterface'
- 'ec2:CreateNetworkInterfacePermission'
- 'ec2:DeleteNetworkInterface'
- 'ec2:DeleteNetworkInterfacePermission'
- 'ec2:Describe*'
- 'ec2:DetachNetworkInterface'
# Rules which allow ECS to update load balancers on your behalf
# with the information sabout how to send traffic to your containers
- 'elasticloadbalancing:DeregisterInstancesFromLoadBalancer'
- 'elasticloadbalancing:DeregisterTargets'
- 'elasticloadbalancing:Describe*'
- 'elasticloadbalancing:RegisterInstancesWithLoadBalancer'
- 'elasticloadbalancing:RegisterTargets'
Resource: '*'
These are the values output by the CloudFormation template. Be careful
about changing any of them, because of them are exported with specific
names so that the other task related CF templates can use them.
Outputs:
ClusterName:
Description: The name of the ECS cluster
Value: !Ref 'ECSCluster'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ClusterName' ] ]
ExternalUrl:
Description: The url of the external load balancer
Value: !Join ['', ['http://', !GetAtt 'PublicLoadBalancer.DNSName']]
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ExternalUrl' ] ]
ECSRole:
Description: The ARN of the ECS role
Value: !GetAtt 'ECSRole.Arn'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'ECSRole' ] ]
PublicListener:
Description: The ARN of the public load balancer's Listener
Value: !Ref PublicLoadBalancerListener
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicListener' ] ]
VPCId:
Description: The ID of the VPC that this stack is deployed in
Value: !Ref 'VPC'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'VPCId' ] ]
PublicSubnetOne:
Description: Public subnet one
Value: !Ref 'PublicSubnetOne'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicSubnetOne' ] ]
PublicSubnetTwo:
Description: Public subnet two
Value: !Ref 'PublicSubnetTwo'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'PublicSubnetTwo' ] ]
EcsHostSecurityGroup:
Description: A security group used to allow containers to receive traffic
Value: !Ref 'EcsHostSecurityGroup'
Export:
Name: !Join [ ':', [ !Ref 'AWS::StackName', 'EcsHostSecurityGroup' ] ]
Script 2 to create service and task:
AWSTemplateFormatVersion: '2010-09-09'
Description: Deploy a service into an ECS cluster behind a public load balancer.
Parameters:
StackName:
Type: String
Default: PublicNLBCluster2
Description: The name of the parent cluster stack that you created. Necessary
to locate and reference resources created by that stack.
ServiceName:
Type: String
Default: nginx
Description: A name for the service
ImageUrl:
Type: String
Default: nginx
Description: The url of a docker image that contains the application process that
will handle the traffic for this service
ContainerPort:
Type: Number
Default: 80
Description: What port number the application inside the docker container is binding to
ContainerCpu:
Type: Number
Default: 256
Description: How much CPU to give the container. 1024 is 1 CPU
ContainerMemory:
Type: Number
Default: 512
Description: How much memory in megabytes to give the container
Path:
Type: String
Default: "*"
Description: A path on the public load balancer that this service
should be connected to. Use * to send all load balancer
traffic to this service.
Priority:
Type: Number
Default: 1
Description: The priority for the routing rule added to the load balancer.
This only applies if your have multiple services which have been
assigned to different paths on the load balancer.
DesiredCount:
Type: Number
Default: 2
Description: How many copies of the service task to run
Role:
Type: String
Default: ""
Description: (Optional) An IAM role to give the service's containers if the code within needs to
access other AWS resources like S3 buckets, DynamoDB tables, etc
Conditions:
HasCustomRole: !Not [ !Equals [!Ref 'Role', ''] ]
Resources:
# The task definition. This is a simple metadata description of what
# container to run, and what resource requirements it has.
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Ref 'ServiceName'
Cpu: !Ref 'ContainerCpu'
Memory: !Ref 'ContainerMemory'
TaskRoleArn:
Fn::If:
- 'HasCustomRole'
- !Ref 'Role'
- !Ref "AWS::NoValue"
ContainerDefinitions:
- Name: !Ref 'ServiceName'
Cpu: !Ref 'ContainerCpu'
Memory: !Ref 'ContainerMemory'
Image: !Ref 'ImageUrl'
PortMappings:
- ContainerPort: !Ref 'ContainerPort'
# The service. The service is a resource which allows you to run multiple
# copies of a type of task, and gather up their logs and metrics, as well
# as monitor the number of running tasks and replace any that have crashed
Service:
Type: AWS::ECS::Service
DependsOn: PublicLoadBalancerListener
Properties:
ServiceName: !Ref 'ServiceName'
Cluster:
Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'ClusterName']]
DeploymentConfiguration:
MaximumPercent: 200
MinimumHealthyPercent: 75
DesiredCount: !Ref 'DesiredCount'
TaskDefinition: !Ref 'TaskDefinition'
LoadBalancers:
- ContainerName: !Ref 'ServiceName'
ContainerPort: !Ref 'ContainerPort'
TargetGroupArn: !Ref 'TargetGroup'
# A target group. This is used for keeping track of all the tasks, and
# what IP addresses / port numbers they have. You can query it yourself,
# to use the addresses yourself, but most often this target group is just
# connected to an application load balancer, or network load balancer, so
# it can automatically distribute traffic across all the targets.
TargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Name: !Ref 'ServiceName'
Port: 80
Protocol: TCP
VpcId:
Fn::ImportValue:
!Join [':', [!Ref 'StackName', 'VPCId']]

SAM API Gateway with Cloudformation WAFRegional

To secure our API, I'm trying to deploy a WAFRegional with a RateBasedRule. The API Gateway is located in a SAM template wherein I have also a nested stack for the child template holding the WAFRegional configurations. The child template for the WAFRegional configuration is provided below. What happens during the ExecuteChangeSet phase is the following:
CamerasIpSet is created
CamerasRateRule is created
WAFCamerasWebACL CREATE_FAILED: The referenced item does not exist. (Service: AWSWAFRegional; Status Code: 400; Error Code: WAFNonexistentItemException
I found the following post from about 2 months ago where someone has the same issue when using Serverless: https://forum.serverless.com/t/dependon-api-gateway-deployment/7792
What am I missing out on here?
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Template for WAF Configuration'
Parameters:
CamerasApi:
Description: "Arn of the Cameras Api"
Type: String
Default: cameras-api-dev
StageName:
Description: "Stage name of the Cameras Api"
Type: String
Default: v
Blocking:
Description: "Number of calls per 5 minutes for WAF IP blocking."
Type: Number
Default: 2000
EnvironmentType:
Type: String
Default: "dev"
Description: "Type of environment: dev, staging or prod."
Resources:
WAFCamerasWebACL:
Type: AWS::WAFRegional::WebACL
DependsOn: CamerasRateRule
Properties:
DefaultAction:
Type: ALLOW
MetricName: !Join ['', ['IPBlockingMetric', !Ref EnvironmentType]]
Name: !Join ['', ['IPBlockingACL', !Ref EnvironmentType]]
Rules:
-
Action:
Type: "BLOCK"
Priority: 1
RuleId: !Ref CamerasRateRule
CamerasRateRule:
Type: AWS::WAFRegional::RateBasedRule
Properties:
MetricName: UnallowedAccessCount
Name: FiveMinuteRule
RateKey: IP
RateLimit: !Ref Blocking
MatchPredicates:
-
DataId: !Ref CamerasIpSet
Negated: false
Type: "IPMatch"
CamerasIpSet:
Type: AWS::WAFRegional::IPSet
Properties:
Name: !Join ['-', ['IpBlacklist', !Ref EnvironmentType]]
MyWebACLAssociation:
Type: AWS::WAFRegional::WebACLAssociation
Properties:
ResourceArn: !Sub arn:aws:apigateway:${AWS::Region}::/restapis/${CamerasApi}/stages/${StageName}
WebACLId: !Ref WAFCamerasWebACL
Outputs:
WebACL:
Description: Name of the web ACL
Value: !Ref WAFCamerasWebACL
I finally resolved the issue with the help of the AWS customer service. This is a limitation they have with CloudFormation when dealing with AWS::WAFRegional::RateBasedRule.
Despite the fact that CloudFormation supports creating WAF regional rate-based rules, the association of them with a Web ACL is not currently supported. If you observe link [1] below, you will realize that:
"To add the rate-based rules created through CloudFormation to a web ACL, use the AWS WAF console, API, or command line interface (CLI)."
[1] AWS::WAFRegional::RateBasedRule:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-wafregional-ratebasedrule.html
I used the Cloudformation template to generate the WebACL, the RateBasedRule, and the association of the WebACL with my APIGW. Using CodeBuild in our CI/CD pipeline, I'm now adding the RateBasedRule to the WebACL by using the CLI command aws waf-regional update-web-acl.
I ran in the same issue and I solve the problem with WAFv2
AWSTemplateFormatVersion: '2010-09-09'
Description: 'Template for WAF Configuration'
Parameters:
CamerasApi:
Description: "Arn of the Cameras Api"
Type: String
Default: YOUR-API-ID
StageName:
Description: "Stage name of the Cameras Api"
Type: String
Default: YOUR-Stage
Blocking:
Description: "Number of calls per 5 minutes for WAF IP blocking."
Type: Number
Default: 2000
EnvironmentType:
Type: String
Default: Prod
Description: "Type of environment: dev, staging or prod."
Resources:
WAFCamerasWebACL:
Type: AWS::WAFv2::WebACL
Properties:
Name: ExampleWebACL
Description: This is an example WebACL
Scope: REGIONAL
DefaultAction:
Allow: {}
VisibilityConfig:
SampledRequestsEnabled: true
CloudWatchMetricsEnabled: true
MetricName: ExampleWebACLMetric
Rules:
- Name: RulesTest
Priority: 0
Action:
Block: {}
VisibilityConfig:
SampledRequestsEnabled: true
CloudWatchMetricsEnabled: true
MetricName: test
Statement:
RateBasedStatement:
Limit: 100
AggregateKeyType: IP
MyWebACLAssociation:
Type: AWS::WAFv2::WebACLAssociation
Properties:
ResourceArn: !Sub arn:aws:apigateway:${AWS::Region}::/restapis/${CamerasApi}/stages/${StageName}
WebACLArn: !GetAtt WAFCamerasWebACL.Arn
Outputs:
WebACL:
Description: Name of the web ACL
Value: !Ref WAFCamerasWebACL
Assuming a AWS::WAFRegional::WebACL and AWS::WAFRegional::RateBasedRule are defined in a Cloudformation stack, they can be attached using the following bash script:
CHANGE_TOKEN=$(aws waf-regional get-change-token --output text)
WEBACL_ID=$(aws waf-regional list-web-acls --query WebACLs[0].WebACLId --output text)
RULE_ID=$(aws waf-regional list-rate-based-rules --query Rules[0].RuleId --output text)
aws waf-regional update-web-acl --web-acl-id $WEBACL_ID --change-token $CHANGE_TOKEN \
--updates Action="INSERT",ActivatedRule='{Priority=1,RuleId="'$RULE_ID'",Action={Type="BLOCK"},Type="RATE_BASED"}'
However unfortunately this leads to issues when deleting the Cloudformation stack
The following resource(s) failed to delete: [RateBasedRuleName].
Any ideas how to enable the stack to remove the rule when issueing aws cloudformation delete-stack?
Resources:
BlueWafAlbAssociation:
Type: "AWS::WAFv2::WebACLAssociation"
Properties:
WebACLArn: arn:aws:wafv2:us-east-1:1234567890:regional/webacl/name-of-webacl/id-of-webacl
ResourceArn: arn:aws:elasticloadbalancing:us-east-1:1234567890:loadbalancer/app/load-balancer-name/xxxxxxxxxxx

Fargate error: cannot pull container hosted in ECR from a private subnet

I am trying to create a following architecture: a vpc with two subnets (one is public containing a NatGateway and an InternetGateway, and another one is private.
I start a fargate service in a private subnet and it fails with this error:
CannotPullContainerError: API error (500): Get
https://XYZ.dkr.ecr.us-east-1.amazonaws.com/v2/: net/http:
request cancelled while waiting for connection (Client.Timeout exceeded
while awaiting headers)
Here's my CloudFormation template (the service is intentionally commented out, and the ECR image url is scrambled):
Resources:
#Network resources: VPC
WorkflowVpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: "10.0.0.0/16"
EnableDnsSupport: false
Tags:
- Key: Project
Value: Workflow
#PublicSubnet
WorkflowPublicSubnet:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: "10.0.0.0/24"
VpcId:
Ref: WorkflowVpc
WorkflowInternetGateway:
Type: AWS::EC2::InternetGateway
WorkflowVCPGatewayAttachment:
DependsOn:
- WorkflowInternetGateway
- WorkflowVpc
Type: AWS::EC2::VPCGatewayAttachment
Properties:
InternetGatewayId:
Ref: WorkflowInternetGateway
VpcId:
Ref: WorkflowVpc
WorkflowElasticIp:
Type: AWS::EC2::EIP
Properties:
Domain: vpc
WorkflowPublicSubnetRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId:
Ref: WorkflowVpc
PublicSubnetToRouteTable:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId:
Ref: WorkflowPublicSubnetRouteTable
SubnetId:
Ref: WorkflowPublicSubnet
WorkflowInternetRoute:
Type: AWS::EC2::Route
Properties:
RouteTableId:
Ref: WorkflowPublicSubnetRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId:
Ref: WorkflowInternetGateway
WorkflowNat:
DependsOn:
- WorkflowVCPGatewayAttachment
- WorkflowElasticIp
Type: AWS::EC2::NatGateway
Properties:
AllocationId:
Fn::GetAtt:
- WorkflowElasticIp
- AllocationId
SubnetId:
Ref: WorkflowPublicSubnet
#Private subnet
WorkflowPrivateSubnet:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: "10.0.1.0/24"
VpcId:
Ref: WorkflowVpc
WorkflowPrivateSubnetRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId:
Ref: WorkflowVpc
PrivateSubnetToRouteTable:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId:
Ref: WorkflowPrivateSubnetRouteTable
SubnetId:
Ref: WorkflowPrivateSubnet
WorkflowNatRoute:
Type: AWS::EC2::Route
Properties:
RouteTableId:
Ref: WorkflowPrivateSubnetRouteTable
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId:
Ref: WorkflowNat
#Fargate:
WorkflowFargateTask:
Type: AWS::ECS::TaskDefinition
Properties:
RequiresCompatibilities:
- "FARGATE"
Cpu: "256"
Memory: "0.5GB"
ContainerDefinitions:
- Name: WorkflowFargateContainer
Image: "XYZ.dkr.ecr.us-east-1.amazonaws.com/workflow:latest"
NetworkMode: awsvpc
ExecutionRoleArn: "arn:aws:iam::XXX:role/ecsTaskExecutionRole"
WorkflowCluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: WorkflowServiceCluster
# WorkflowService:
# DependsOn:
# - WorkflowNatRoute
# Type: AWS::ECS::Service
# Properties:
# Cluster:
# Ref: WorkflowCluster
# DesiredCount: 1
# TaskDefinition:
# Ref: WorkflowFargateTask
# NetworkConfiguration:
# AwsvpcConfiguration:
# AssignPublicIp: DISABLED
# Subnets:
# - Ref: WorkflowPrivateSubnet
# LaunchType: FARGATE
I also tried to set AssignPublicIp: ENABLED within the public subnet, and it works just fine, but it is not what I'm aiming for.
So, the questions that I have: is my template ok and is it the problem of Fargate/ECR?
Also, what would be the best way to debug such a behaviour? It seems that CloudWatch has no logs concerning this error...
Following Steve E's hints I've figured out that the internet access is present, the only problem is in this parameter for the VPC:
EnableDnsSupport: false
Naturally, when I tried to update linux packages, or ping google.com, it couldn't resolve the host names. Switching it to "true" resolved the problem.