I have a very simple Cloudformation script which just tries to create a CloudWatch LogGroup and associated LogStream -
AWSTemplateFormatVersion: '2010-09-09'
Description: Hello World
Parameters:
AppName:
Type: String
EnvName:
Type: String
Resources:
AppLogGroup:
Properties:
LogGroupName:
Fn::Join:
- '-'
- - Ref: AppName
- Ref: EnvName
RetentionInDays: 7
Type: AWS::Logs::LogGroup
AppLogStream:
Properties:
LogGroupName:
Fn::Join:
- '-'
- - Ref: AppName
- Ref: EnvName
LogStreamName:
Fn::Join:
- '-'
- - Ref: AppName
- Ref: EnvName
- info
Type: AWS::Logs::LogStream
Now this deploys successfully about 50% of the time - but for the other 50%, whilst the LogGroup creates fine, the creation of the LogStream fails with The specified log group does not exist; and overall the stack rolls back.
I assume this is because of some kind of failure in the "eventual consistency" within AWS, but am a bit surpised that creation of the LogStream can't wait until the LogGroup has completed.
Am I doing something wrong ? Should I be creating LogStreams in my app rather than the stack ? Does a LogStream get automatically created when you send a first message (doubt it) ? Is there a workaround here ?
TIA.
Instead of manually joining strings together to reference the log group in the log stream, you can use !Ref for it. With that you also get an implicit dependency of the log stream to the log group, which ensures that CloudFormation will create the log stream after it created the log group. While you could manually specify DependsOn to achieve the same, using !Ref is the way AWS suggests as best practice. That'd look like:
AWSTemplateFormatVersion: '2010-09-09'
Description: Hello World
Parameters:
AppName:
Type: String
EnvName:
Type: String
Resources:
AppLogGroup:
Properties:
LogGroupName:
Fn::Join:
- '-'
- - Ref: AppName
- Ref: EnvName
RetentionInDays: 7
Type: AWS::Logs::LogGroup
AppLogStream:
Properties:
LogGroupName: !Ref AppLogGroup
LogStreamName:
Fn::Join:
- '-'
- - Ref: AppName
- Ref: EnvName
- info
Type: AWS::Logs::LogStream
Try to add a DependsOn: AppLogGroup attribute to the AppLogStream resource. It will wait for the dependent resource to complete.
See
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-dependson.html
Related
I want to create a cloudformation stackset with resources like IAM and lambda in different regions. when I tried to deploy these resources, it failed because IAM roles are global and it is trying to create again in second region and whole stackset is failed.
Is there anyway I can mention the stackset to deploy GLobal Resources in one region and resources like lambda in all other regions?
Is there anyway I can mention the stackset to deploy GLobal Resources in one region and resources like lambda in all other regions?
Sadly there is not. You have to split your template, so that global resource are created as normal regional stacks.
I went through many resources and finally found a solution. If we split the template in stacksets then my dependent resources will break because creation is parallel in cloudformation. i.e. before global role gets created, lambda will try to get deployed and it will fail because the role is not available(required by lambda).
Hence we can add a condition to each of the global resources like below
Conditions:
RegionCheck: !Equals
- !Ref "AWS::Region"
- us-east-1
And, add the condition in the resources section as below,
Resources:
GlobalRolelambda:
Type: 'AWS::IAM::Role'
Condition: RegionCheck
Properties:
RoleName: !Ref LambdaExecutionRole
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/ReadOnlyAccess'
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
Path: /
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
Policies:
- PolicyName: lambda-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'glue:GetConnections'
- 'mediastore:ListContainers'
- 'mediastore:GetContainerPolicy'
Resource: '*'
But, after doing this, the problem would still exist because, if you add lambda resource with depends on attribute, role would get created in one region but not in the second region, lambda will fail to create in second region. We need to add a wait condition in the template to handle this as below Conditions:
CreateLambdaRole: !Equals [ !Ref LambdaRoleName, 'false' ]
CreateLamdaRoleRegion: !And
- !Condition RegionCheck
- !Condition CreateLambdaRole
and, add below resources after Role Resource,
CreateRoleWaitHandle:
Condition: CreateLamdaRoleRegion
DependsOn: GlobalRolelambda
Type: "AWS::CloudFormation::WaitConditionHandle"
#added, since DependsOn: !If is not possible, trigger by WaitCondition if CreateLamdaRoleRegion is false
WaitHandle:
Type: "AWS::CloudFormation::WaitConditionHandle"
#added, since DependsOn: !If is not possible
WaitCondition:
Type: "AWS::CloudFormation::WaitCondition"
Properties:
Handle: !If [CreateLamdaRoleRegion, !Ref CreateRoleWaitHandle, !Ref WaitHandle]
Timeout: "1"
Count: 0
and now, refer this in lambda resource,
lambdaProcessorFunction:
Type: AWS::Lambda::Function
Properties:
FunctionName: Lambda-processor
Description: ''
Handler: index.handler
Role:
Fn::Sub: 'arn:aws:iam::${AWS::AccountId}:role/LambdaExecutionRole'
Runtime: python3.6
Timeout: 600
MemorySize: 1024
Code:
S3Bucket: !Ref SourceBucketName
S3Key: !Ref SourceBucketKey
DependsOn: WaitCondition
Refer to the below source links, which might help
https://garbe.io/blog/2017/07/17/cloudformation-hacks/
CloudFormation, apply Condition on DependsOn
I'm trying to create or update a stack with the following CloudFormation Template:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ApiGatewayId:
Type: String
ApiLayerArn:
Type: String
JarLocation:
Type: String
Resources:
Function:
Type: 'AWS::Lambda::Function'
Properties:
Handler: net.bitsandpaper.api.kiosk.PlatformChecker
Runtime: java11
Code:
S3Bucket: bnp-build-artifacts
S3Key: !Ref JarLocation
Description: ''
MemorySize: 128
Timeout: 5
Role: arn:aws:iam::479832603967:role/bnp-api-lambda-execution-role
Layers:
- !Ref ApiLayerArn
ApiIntegration:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref ApiGatewayId
IntegrationType: AWS_PROXY
IntegrationUri: !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !Ref Function
- /invocations
TimeoutInMillis: 6000
ApiRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref ApiGatewayId
RouteKey: 'GET /kiosk/platform-check'
Target: !Join
- /
- - integrations
- !Ref ApiIntegration
The parameters are correctly passed by an external file, they look good in the Web Console, notably parameter ApiGatewayId has value 8548rqrsm5. Yet during deployment I have a CREATE_FAILED for ApiIntegration, with the message:
Invalid API identifier specified 479832603967:8548rqrsm5 (Service:
AmazonApiGatewayV2; Status Code: 404; Error Code: NotFoundException;
Request ID: 84918a83-cf9d-48d2-acf7-18d9d2e4d330; Proxy: null)
The API is an EDGE Rest API, in the same region than the CloudFormation stack. The ID is retrieved by the CLI with aws apigateway get-rest-apis.
Am I missing something in the ApiId format? The litterature is very scarce when not referencing an API in the same stack...
AWS::ApiGatewayV2 is only for WEBSOCKTES and HTTP types. From docs:
The API protocol. Valid values are WEBSOCKET or HTTP.
But since you are writing about Edge-optimized (not supported by HTTP api) it seems to that you are using REST API, rather then HTTP API. So you should be using AWS::ApiGateway resources, not AWS::ApiGatewayV2.
It seem's like the AWS::ApiGatewayV2::Route is created before the AWS::ApiGatewayV2::Integration. So When it trying to refer ApiIntegration it is not yet created.
So you should try to use DependsOn attribute.
With the DependsOn attribute you can specify that the creation of a
specific resource follows another. When you add a DependsOn attribute
to a resource, that resource is created only after the creation of the
resource specified in the DependsOn attribute.
Try this below CloudFormation code:
ApiRoute:
Type: AWS::ApiGatewayV2::Route
DependsOn: ApiIntegration
Properties:
ApiId: !Ref ApiGatewayId
RouteKey: 'GET /kiosk/platform-check'
Target: !Join
- /
- - integrations
- !Ref ApiIntegration
I hope this will help you out to resolve your problem.
Link: DependsOn Attribute UserGuide
This CloudFormation template works as expected and creates all the resources required by this article:
Data visualization and anomaly detection using Amazon Athena and Pandas from Amazon SageMaker | AWS Machine Learning Blog
But the WorkflowStartTrigger resource does not actually run the crawler. How do I run a crawler using the CloudFormation template?
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "glue.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "*"
Resource: "*"
MyDatabase:
Type: AWS::Glue::Database
Properties:
CatalogId: !Ref AWS::AccountId
DatabaseInput:
Name: "dbcrawler123"
Description: "TestDatabaseDescription"
LocationUri: "TestLocationUri"
Parameters:
key1 : "value1"
key2 : "value2"
MyCrawler2:
Type: AWS::Glue::Crawler
Properties:
Description: example classifier
Name: "testcrawler123"
Role: !GetAtt MyRole.Arn
DatabaseName: !Ref MyDatabase
Targets:
S3Targets:
- Path: 's3://nytaxi162/'
SchemaChangePolicy:
UpdateBehavior: "UPDATE_IN_DATABASE"
DeleteBehavior: "LOG"
TablePrefix: test-
Configuration: "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}}}"
WorkflowStartTrigger:
Type: AWS::Glue::Trigger
Properties:
Description: Trigger for starting the Crawler
Name: StartTrigger
Type: ON_DEMAND
Actions:
- CrawlerName: "testcrawler123"
You should be able to do that by creating a custom resource attached to a lambda whereby the lambda actually does the action of starting the crawler. You should be able to even make it wait for the crawler to complete its execution
CloudFormation directly doesn't run crawlers, it just create them.
But you can create a schedule in order to run a crawler while defining trigger:
ScheduledJobTrigger:
Type: 'AWS::Glue::Trigger'
Properties:
Type: SCHEDULED
StartOnCreation: true
Description: DESCRIPTION_SCHEDULED
Schedule: cron(5 * * * ? *)
Actions:
- CrawlerName: "testcrawler123"
Name: ETLGlueTrigger
If needed to run crawler as part of CloudFormation stack creation, Lambda could be used.
So the last item I added to this template was the attempt to have it use a particular SecurityGroup. I did not want it to create a new one. When I do the validate check that comes back ok but apparently my code is still not correct. Other that the template was working ok.
I have tried all I can think of. there is no error when i finally times out other than "internal error" so I am at a loss here.
Parameters:
VPC:
Description: Testing using this VPC
Type: String
Default: vpc-02765
SecGroup:
Description: Name of security group
Type: AWS::EC2::SecurityGroup
KeyName:
Description: Name of an existing EC2 key pair for SSH access to the EC2 instance.
Type: AWS::EC2::KeyPair::KeyName
InstanceType:
Description: EC2 instance type.
Type: String
Default: t2.micro
...
...
...
Resources:
EC2Instance:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref 'InstanceType'
SubnetId: subnet-08b
KeyName: !Ref 'KeyName'
SecurityGroupIds:
- !Ref SecGroup
ImageId: !FindInMap
- AWSRegionArch2AMI
- !Ref 'AWS::Region'
- HVM64
'''
all I am trying to do is use the items I listed in the template. the vpc,securitygroup. The last time this worked was when I had the code in the template that builds a new SG. I than changed my mind and want to use an existing SG. so somewhere I messed up
This works in my templates:
Parameters:
SecGroup:
Type: AWS::EC2::SecurityGroup::Id
...
Resources:
MyInstance:
Properties:
SecurityGroupIds:
- !Ref SecGroup
Since I had an issue of 200 resource error, I found a way of using cross stack reference by dividing into different services. I managed to do that by using the cross-stack reference. The issue is I cannot give the restApiId & restApiRootResourceId dynamically. Right now, am statically setting ids into the service-2.
Basically the service-1 looks like,
provider:
name: aws
runtime: nodejs8.10
apiGateway:
restApiId:
Ref: ApiGatewayRestApi
restApiResources:
Fn::GetAtt:
- ApiGatewayRestApi
- RootResourceId
custom:
stage: "${opt:stage, self:provider.stage}"
resources:
Resources:
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Name: ${self:service}-${self:custom.stage}-1
Outputs:
ApiGatewayRestApiId:
Value:
Ref: ApiGatewayRestApi
Export:
Name: ApiGatewayRestApi-restApiId
ApiGatewayRestApiRootResourceId:
Value:
Fn::GetAtt:
- ApiGatewayRestApi
- RootResourceId
Export:
Name: ApiGatewayRestApi-rootResourceId
And the service-2 looks like this,
provider:
name: aws
runtime: nodejs8.10
apiGateway-shared:
restApiId:
'Fn::ImportValue': ApiGatewayRestApi-restApiId
restApiRootResourceId:
'Fn::ImportValue': ApiGatewayRestApi-rootResourceId
As the above service-2 config, I cannot reference the Ids.
FYI: Both services are in different files.
So How what's wrong with this approach?
Serverless has special syntax on how to access stack output variables: {cf:stackName.outputKey}.
Note that using the Fn::ImportValue would work inside the resources section.