This part of cloudformation template creates the event, but do not actually execute it.
MyCWEventRule:
Type: AWS::Events::Rule
Properties:
Name: my-batch-event-rule
ScheduleExpression: cron(01 07 ? * * *)
State: ENABLED
Targets:
- Id: my-batch-job
Arn: !Ref Queue
BatchParameters:
JobDefinition: !Ref StartJob
JobName: tds-job
RoleArn: !Join [ "", [!Sub "arn:aws:iam::${AWS::AccountId}:role/service-role/", !Ref ruleIAMRole]]
There are no logs to check what the problem is. If I submit the job myself using console, it works as expected.
The Job Definition parameter mentioned in cron is correct as seen in this image...
How do I troubleshoot failed cloudwatch events?
Update:
When I click on Edit and use "Create a new role for this specific resource" it works.
It means there is some problem with the line...
RoleArn: !Join [ "", [!Sub
"arn:aws:iam::${AWS::AccountId}:role/service-role/", !Ref
ruleIAMRole]]
You guessed it right! There is no need of "service-role" in the ARN role path. It should look like this...
RoleArn: !Join [ "", [!Sub "arn:aws:iam::${AWS::AccountId}:role/", !Ref ruleIAMRole]]
Or even better:
RoleArn: !GetAtt ruleIAMRole.Arn
Related
I'm trying to create or update a stack with the following CloudFormation Template:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ApiGatewayId:
Type: String
ApiLayerArn:
Type: String
JarLocation:
Type: String
Resources:
Function:
Type: 'AWS::Lambda::Function'
Properties:
Handler: net.bitsandpaper.api.kiosk.PlatformChecker
Runtime: java11
Code:
S3Bucket: bnp-build-artifacts
S3Key: !Ref JarLocation
Description: ''
MemorySize: 128
Timeout: 5
Role: arn:aws:iam::479832603967:role/bnp-api-lambda-execution-role
Layers:
- !Ref ApiLayerArn
ApiIntegration:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref ApiGatewayId
IntegrationType: AWS_PROXY
IntegrationUri: !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !Ref Function
- /invocations
TimeoutInMillis: 6000
ApiRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref ApiGatewayId
RouteKey: 'GET /kiosk/platform-check'
Target: !Join
- /
- - integrations
- !Ref ApiIntegration
The parameters are correctly passed by an external file, they look good in the Web Console, notably parameter ApiGatewayId has value 8548rqrsm5. Yet during deployment I have a CREATE_FAILED for ApiIntegration, with the message:
Invalid API identifier specified 479832603967:8548rqrsm5 (Service:
AmazonApiGatewayV2; Status Code: 404; Error Code: NotFoundException;
Request ID: 84918a83-cf9d-48d2-acf7-18d9d2e4d330; Proxy: null)
The API is an EDGE Rest API, in the same region than the CloudFormation stack. The ID is retrieved by the CLI with aws apigateway get-rest-apis.
Am I missing something in the ApiId format? The litterature is very scarce when not referencing an API in the same stack...
AWS::ApiGatewayV2 is only for WEBSOCKTES and HTTP types. From docs:
The API protocol. Valid values are WEBSOCKET or HTTP.
But since you are writing about Edge-optimized (not supported by HTTP api) it seems to that you are using REST API, rather then HTTP API. So you should be using AWS::ApiGateway resources, not AWS::ApiGatewayV2.
It seem's like the AWS::ApiGatewayV2::Route is created before the AWS::ApiGatewayV2::Integration. So When it trying to refer ApiIntegration it is not yet created.
So you should try to use DependsOn attribute.
With the DependsOn attribute you can specify that the creation of a
specific resource follows another. When you add a DependsOn attribute
to a resource, that resource is created only after the creation of the
resource specified in the DependsOn attribute.
Try this below CloudFormation code:
ApiRoute:
Type: AWS::ApiGatewayV2::Route
DependsOn: ApiIntegration
Properties:
ApiId: !Ref ApiGatewayId
RouteKey: 'GET /kiosk/platform-check'
Target: !Join
- /
- - integrations
- !Ref ApiIntegration
I hope this will help you out to resolve your problem.
Link: DependsOn Attribute UserGuide
This CloudFormation template works as expected and creates all the resources required by this article:
Data visualization and anomaly detection using Amazon Athena and Pandas from Amazon SageMaker | AWS Machine Learning Blog
But the WorkflowStartTrigger resource does not actually run the crawler. How do I run a crawler using the CloudFormation template?
Resources:
MyRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "glue.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: "root"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action: "*"
Resource: "*"
MyDatabase:
Type: AWS::Glue::Database
Properties:
CatalogId: !Ref AWS::AccountId
DatabaseInput:
Name: "dbcrawler123"
Description: "TestDatabaseDescription"
LocationUri: "TestLocationUri"
Parameters:
key1 : "value1"
key2 : "value2"
MyCrawler2:
Type: AWS::Glue::Crawler
Properties:
Description: example classifier
Name: "testcrawler123"
Role: !GetAtt MyRole.Arn
DatabaseName: !Ref MyDatabase
Targets:
S3Targets:
- Path: 's3://nytaxi162/'
SchemaChangePolicy:
UpdateBehavior: "UPDATE_IN_DATABASE"
DeleteBehavior: "LOG"
TablePrefix: test-
Configuration: "{\"Version\":1.0,\"CrawlerOutput\":{\"Partitions\":{\"AddOrUpdateBehavior\":\"InheritFromTable\"},\"Tables\":{\"AddOrUpdateBehavior\":\"MergeNewColumns\"}}}"
WorkflowStartTrigger:
Type: AWS::Glue::Trigger
Properties:
Description: Trigger for starting the Crawler
Name: StartTrigger
Type: ON_DEMAND
Actions:
- CrawlerName: "testcrawler123"
You should be able to do that by creating a custom resource attached to a lambda whereby the lambda actually does the action of starting the crawler. You should be able to even make it wait for the crawler to complete its execution
CloudFormation directly doesn't run crawlers, it just create them.
But you can create a schedule in order to run a crawler while defining trigger:
ScheduledJobTrigger:
Type: 'AWS::Glue::Trigger'
Properties:
Type: SCHEDULED
StartOnCreation: true
Description: DESCRIPTION_SCHEDULED
Schedule: cron(5 * * * ? *)
Actions:
- CrawlerName: "testcrawler123"
Name: ETLGlueTrigger
If needed to run crawler as part of CloudFormation stack creation, Lambda could be used.
I've created the following CloudFormation template:
AWSTemplateFormatVersion: 2010-09-09
Description: Creates all resources necessary to send SES emails & track bounces/complaints through AWS
Resources:
IAMUser:
Type: 'AWS::IAM::User'
Properties:
UserName: iam-ses-sqs
SQSQueue:
Type: 'AWS::SQS::Queue'
Properties:
QueueName: ses-queue
SNSTopic:
Type: 'AWS::SNS::Topic'
Properties:
TopicName: sns-notifications
IAMUserPolicy:
Type: 'AWS::IAM::Policy'
Properties:
PolicyName: IAM_Send_SES_Email
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'SES:SendEmail'
- 'SES:SendRawEmail'
Resource: 'arn:aws:ses:*:*:identity/*'
Users:
- !Ref IAMUser
SQSQueuePolicy:
Type: 'AWS::SQS::QueuePolicy'
Properties:
Queues:
- !Ref SQSQueue
PolicyDocument:
Statement:
- Action:
- 'SQS:ReceiveMessage'
- 'SQS:DeleteMessage'
- 'SQS:GetQueueAttributes'
Effect: Allow
Resource: !Ref SQSQueue
Principal:
AWS:
- !Ref IAMUser
SNSTopicSubscription:
Type: 'AWS::SNS::Subscription'
Properties:
Protocol: SQS
Endpoint: !GetAtt
- SQSQueue
- Arn
TopicArn: !Ref SNSTopic
I'd like to allow IAMUser to perform the SQS ReceiveMessage, DeleteMessage, and GetQueueAttributes actions on the SQSQueue resource. SQSQueue should also be subscribed to the SNSTopic.
When creating a stack using this template in CloudFormation, the SQSQueue, SNSTopic, SNSTopicSubscription, IAMUser, and IAMUserPolicy all create with no problem, in that order. However, the SQSQueuePolicy fails to create and generates the error message:
Invalid value for the parameter Policy. (Service: AmazonSQS; Status Code: 400; Error Code: InvalidAttributeValue; Request ID: {request id})
Why is this failing, and how should I modify the template to ensure that all resources and their associated policies/subscriptions are created successfully?
I found two problems in your CloudFormation template.
The first one, like Marcin said, the resource reference must be the Queue ARN and not the Queue URL.
Resource: !GetAtt SQSQueue.Arn
The second one is that your AWS reference is with your IAM user but it must be the Account ID.
Principal:
AWS:
- !Ref 'AWS::AccountId'
That said, I was able to create successfully the CloudFormation Stack in my account with this CloudFormation Template:
AWSTemplateFormatVersion: 2010-09-09
Description: Creates all resources necessary to send SES emails & track bounces/complaints through AWS
Resources:
IAMUser:
Type: 'AWS::IAM::User'
Properties:
UserName: iam-ses-sqs
SQSQueue:
Type: 'AWS::SQS::Queue'
Properties:
QueueName: ses-queue
SQSQueuePolicy:
Type: 'AWS::SQS::QueuePolicy'
Properties:
Queues:
- !Ref SQSQueue
PolicyDocument:
Statement:
- Action:
- 'SQS:ReceiveMessage'
- 'SQS:DeleteMessage'
- 'SQS:GetQueueAttributes'
Effect: Allow
Resource: !GetAtt SQSQueue.Arn
Principal:
AWS:
- !Ref 'AWS::AccountId'
The following will return queue URL, not ARN:
Resource: !Ref SQSQueue
But you need to use queue ARN in the policy:
Resource: !GetAtt SQSQueue.Arn
So the last item I added to this template was the attempt to have it use a particular SecurityGroup. I did not want it to create a new one. When I do the validate check that comes back ok but apparently my code is still not correct. Other that the template was working ok.
I have tried all I can think of. there is no error when i finally times out other than "internal error" so I am at a loss here.
Parameters:
VPC:
Description: Testing using this VPC
Type: String
Default: vpc-02765
SecGroup:
Description: Name of security group
Type: AWS::EC2::SecurityGroup
KeyName:
Description: Name of an existing EC2 key pair for SSH access to the EC2 instance.
Type: AWS::EC2::KeyPair::KeyName
InstanceType:
Description: EC2 instance type.
Type: String
Default: t2.micro
...
...
...
Resources:
EC2Instance:
Type: AWS::EC2::Instance
Properties:
InstanceType: !Ref 'InstanceType'
SubnetId: subnet-08b
KeyName: !Ref 'KeyName'
SecurityGroupIds:
- !Ref SecGroup
ImageId: !FindInMap
- AWSRegionArch2AMI
- !Ref 'AWS::Region'
- HVM64
'''
all I am trying to do is use the items I listed in the template. the vpc,securitygroup. The last time this worked was when I had the code in the template that builds a new SG. I than changed my mind and want to use an existing SG. so somewhere I messed up
This works in my templates:
Parameters:
SecGroup:
Type: AWS::EC2::SecurityGroup::Id
...
Resources:
MyInstance:
Properties:
SecurityGroupIds:
- !Ref SecGroup
The following cloudformation script sets up an Api Gateway method that proxies to an S3 bucket.
The S3BucketPolicy opens up the bucket to public read access but the AWS UI warns that this should never be done.
I tried setting the S3BucketPolicy Principal to service apigateway.amazonaws.com but this results in Access Denied.
1) What is the right way to limit bucket access to the API gateway function? (Sample YAML would be great)
2) How could I debug this Access Denied failure to get more information on why it failed?
3) Where should I be looking for sample code on what should be a very standard template snippet?
ATTEMPT #1 - Works but only by making the S3 Bucket Public otherwise Access Denied
AWSTemplateFormatVersion: 2010-09-09
Parameters:
S3BucketName:
Type: String
Description: >
Name for the S3 bucket that contains the nested templates.
Resources:
RestAPI:
Type: 'AWS::ApiGateway::RestApi'
Properties:
BinaryMediaTypes:
- '*/*'
Name: !Ref 'AWS::StackName'
RestAPIRootGET:
Type: 'AWS::ApiGateway::Method'
Properties:
AuthorizationType: NONE
HttpMethod: GET
Integration:
IntegrationHttpMethod: GET
PassthroughBehavior: WHEN_NO_MATCH
Type: HTTP_PROXY
Uri: !Sub https://${S3BucketName}.s3.amazonaws.com/static-assets/index.html
ResourceId: !GetAtt RestAPI.RootResourceId
RestApiId: !Ref RestAPI
DependsOn:
- RestAPI
BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref S3BucketName
PolicyDocument:
Version: 2012-10-17
Statement:
-
Sid: GetObject
Effect: Allow
Principal: "*"
Action:
- s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketName}/static-assets/*'
I think perhaps the right away is to create a role with access to the bucket and then have the ApiGateway assume this role but I'm having a hard time finding documentation that explains how to do this in a cloudformation template. (see also Michael - sqlbot comment suggesting using the credentials property of the method)
Here is my attempt which still fails with Access Denied
ATTEMPT #2 - Access Denied
AWSTemplateFormatVersion: 2010-09-09
Parameters:
S3BucketName:
Type: String
Description: >
Name for the S3 bucket that contains the nested templates.
Resources:
RestAPI:
Type: 'AWS::ApiGateway::RestApi'
Properties:
BinaryMediaTypes:
- '*/*'
Name: !Ref 'AWS::StackName'
RestAPIRootGET:
Type: 'AWS::ApiGateway::Method'
Properties:
AuthorizationType: NONE
HttpMethod: GET
Integration:
IntegrationHttpMethod: GET
PassthroughBehavior: WHEN_NO_MATCH
Type: HTTP_PROXY
Uri: !Sub https://${S3BucketName}.s3.amazonaws.com/static-assets/index.html
Credentials: !GetAtt AllowStaticAccessRole.Arn
ResourceId: !GetAtt RestAPI.RootResourceId
RestApiId: !Ref RestAPI
DependsOn:
- RestAPI
- AllowStaticAccessRole
AllowStaticAccessRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "apigateway.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: "AllowStaticAccessPolicy"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketName}/static-assets/*'
As already mentioned in the comments, the Access Denied error could be coming from KMS rather than from the S3 itself.
To solve the issue, you need to add at minimum these permissions to the role that the Api Gateway assumes: "kms:Decrypt", "kms:ReEncryptFrom", preferably also stating which Resource this should apply too for proper least privilege implementation.