I'm trying to get API Gateway to do the above. See template at the bottom - there is a single message parameter, which I've tried to restrict to either foo or bar values, via specification of an AWS::ApiGateway::Model resource, bound to Content-Type application/x-www-form-urlencoded.
Now APIGW will validate the presence of the message parameter -
curl -H "Content-Type: application/x-www-form-urlencoded" "https://xxxxxxxxxx.execute-api.eu-west-1.amazonaws.com/1-0-0/hello?messag=foo"
{"message": "Missing required request parameters: [message]"}
But it doesn't seem to validate or restrict the value sent -
curl -H "Content-Type: application/x-www-form-urlencoded" "https://xxxxxxxxxx.execute-api.eu-west-1.amazonaws.com/1-0-0/hello?message=whatever"
you sent 'whatever'
Any idea what I am doing wrong here?
AWSTemplateFormatVersion: '2010-09-09'
Outputs:
PublicApiEndpoint:
Value:
Fn::Sub: https://${PublicApiRestApi}.execute-api.${AWS::Region}.${AWS::URLSuffix}/${PublicApiStage}
Parameters:
MemorySizeDefault:
Default: '512'
Type: String
RuntimeVersion:
Default: '3.8'
Type: String
TimeoutDefault:
Default: '5'
Type: String
Resources:
HelloFunction:
Properties:
Code:
ZipFile: |
def handler(event, context):
message=event["queryStringParameters"]["message"]
response="you sent '%s'" % message
return {'statusCode': 200,
'headers': {"Content-Type": "text/plain"},
'body': response}
Handler: index.handler
MemorySize:
Ref: MemorySizeDefault
Role:
Fn::GetAtt:
- HelloFunctionRole
- Arn
Runtime:
Fn::Sub: python${RuntimeVersion}
Timeout:
Ref: TimeoutDefault
Type: AWS::Lambda::Function
HelloFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: sts:AssumeRole
Effect: Allow
Principal:
Service: lambda.amazonaws.com
Version: '2012-10-17'
Policies:
- PolicyDocument:
Statement:
- Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Effect: Allow
Resource: '*'
Version: '2012-10-17'
PolicyName:
Fn::Sub: hello-function-role-policy-${AWS::StackName}
Type: AWS::IAM::Role
HelloEndpointMethod:
Properties:
AuthorizationType: NONE
HttpMethod: GET
Integration:
IntegrationHttpMethod: POST
Type: AWS_PROXY
Uri:
Fn::Sub:
- arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${arn}/invocations
- arn:
Fn::GetAtt:
- HelloFunction
- Arn
RequestParameters:
"method.request.querystring.message": true
RequestValidatorId:
Ref: HelloEndpointValidator
RequestModels:
"application/x-www-form-urlencoded": HelloEndpointModel
ResourceId:
Ref: HelloEndpointResource
RestApiId:
Ref: PublicApiRestApi
Type: AWS::ApiGateway::Method
HelloEndpointPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: HelloFunction
Principal: apigateway.amazonaws.com
SourceArn:
Fn::Sub: arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${PublicApiRestApi}/${PublicApiStage}/GET/hello
Type: AWS::Lambda::Permission
HelloEndpointResource:
Properties:
ParentId:
Fn::GetAtt:
- PublicApiRestApi
- RootResourceId
PathPart: hello
RestApiId:
Ref: PublicApiRestApi
Type: AWS::ApiGateway::Resource
HelloEndpointValidator:
Properties:
RestApiId:
Ref: PublicApiRestApi
ValidateRequestParameters: true
Type: AWS::ApiGateway::RequestValidator
HelloEndpointModel:
Properties:
RestApiId:
Ref: PublicApiRestApi
ContentType: 'application/x-www-form-urlencoded'
Name: HelloEndpointModel
Schema:
"$schema": "http://json-schema.org/draft-04/schema#"
type: object
properties:
message:
type: string
pattern: "^((foo)|(bar))$"
required:
- message
Type: AWS::ApiGateway::Model
PublicApiDeployment:
DependsOn:
- HelloEndpointMethod
Properties:
RestApiId:
Ref: PublicApiRestApi
Type: AWS::ApiGateway::Deployment
PublicApiRestApi:
Properties:
Name:
Fn::Sub: public-api-rest-api-${AWS::StackName}
Type: AWS::ApiGateway::RestApi
PublicApiStage:
Properties:
DeploymentId:
Ref: PublicApiDeployment
RestApiId:
Ref: PublicApiRestApi
StageName: 1-0-0
Type: AWS::ApiGateway::Stage
Apparently this is not possible - https://twitter.com/alexbdebrie/status/1603059764489252864?s=20&t=LI7OUEw9b5qxCIvkeP4AYQ
I have create a fargate task and trying to trigger it via s3 object creation event ( see sample below) via cloudformation.as it cannot trigger it directly, i have created a cloudwatchevent. I am trying to pass the bucket and obj name to my fargate task code . doing some research, i came across -> InputTransformer, but i'm not sure how to pass the value of my bucket and key name and how to read it in my python code. any help will be appreciated.
AWSTemplateFormatVersion: 2010-09-09
Description: An example CloudFormation template for Fargate.
Parameters:
VPC:
Type: AWS::EC2::VPC::Id
SubnetA:
Type: AWS::EC2::Subnet::Id
SubnetB:
Type: AWS::EC2::Subnet::Id
Image:
Type: String
Default: 123456789012.dkr.ecr.region.amazonaws.com/image:tag
Resources:
mybucket:
Properties:
BucketName: 'mytestbucket-us'
cloudwatchEvent:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.s3
detail:
eventSource:
- s3.amazonaws.com
eventName:
- PutObject
- CompleteMultipartUpload
requestParameters:
bucketName:
- !Ref mybucket
Targets:
- Id: my-fargate-task
Arn: myclusterArn
RoleArn: myinvocationrolearn
Input:
'Fn::Sub':
- >-
{"containerOverrides": [{"name":"somecontainer"]}
EcsParameters:
TaskDefinition:
LaunchType: 'FARGATE'
...
NetworkConfiguration:
...
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Join ['', [!Ref ServiceName, Cluster]]
TaskDefinition:
Type: AWS::ECS::TaskDefinition
DependsOn: LogGroup
Properties:
Family: !Join ['', [!Ref ServiceName, TaskDefinition]]
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 256
Memory: 2GB
ExecutionRoleArn: !Ref ExecutionRole
TaskRoleArn: !Ref TaskRole
ContainerDefinitions:
- Name: !Ref ServiceName
Image: !Ref Image
# A role needed by ECS
ExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, ExecutionRole]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
# A role for the containers
TaskRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Join ['', [!Ref ServiceName, TaskRole]]
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: 'sts:AssumeRole'
You would use a CloudWatch Event Input Transformer to extract the data you need from the event, and pass that data to the ECS task as environment variable(s) in the target's ContainerOverrides. I don't use CloudFormation, but here's an example using Terraform.
You can't. CloudWatch events do not pass data to ECS jobs. You need to develop your own mechanism for that. For example, trigger lambda first, store event in S3 Parameter Store or DynamoDB, and then invoke your ECS job which will get stored data.
I am trying to create a new role and a new policy which would be attached
to the same role created in the template in the same template and getting
this error:
Error:
Missing required field Principal(Service:AmazonIdentityManagement;
Status Code: 400;
Error Code: MalformedPolicyDocument;
Proxy: null)
Resources:
lambdaFullPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: "*"
Resource: "*"
LambdaFullRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version : '2012-10-17'
Statement :
- Effect : Allow
- Principal :
service :
- lambda.amazonaws.com
- Action :
- sts: AssumeRole
ManagedPolicyArns:
- !Ref lambdaFullPolicy
DependsOn:
- lambdaFullPolicy
#------------------------------output -----------------------#
Outputs:
PolicyFullLambda:
Description: table
Value: !Ref lambdaFullPolicy
Export:
Name:
"Fn::Sub": "${AWS::StackName}-PolicyFullLambda"
RollFullLambda:
Value: !Ref LambdaFullRole
Export:
Name:
"Fn::Sub": "${AWS::StackName}-RollFullLambda"
There is an extra space in sts: AssumeRole and it should read sts:AssumeRole. This is because this is not a YAML component, but a string literal that AWS uses for the Action section of the Role creation/update.
Edit: This started working as expected by itself the day after. I'm very sure I didn't do anything different. Don't know what to do with the question. Close, Delete?, Let it be?
I am creating a servless web api using AWS SAM and the new HTTP API gateway.
This is my current template file:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
fotffleet-api
Sample SAM Template for fotffleet-api
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
CodeUri: fotffleet/
Runtime: python3.7
Resources:
ServerlessHttpApi:
Type: AWS::Serverless::HttpApi
Properties:
CorsConfiguration:
AllowOrigins:
- "http://localhost:3000"
AllowMethods:
- GET
DefinitionBody:
openapi: "3.0.1"
info:
title: "sam-app"
version: "1.0"
tags:
- name: "httpapi:createdBy"
x-amazon-apigateway-tag-value: "SAM"
paths:
/cells:
get:
responses:
default:
description: "Default response for GET /cells"
x-amazon-apigateway-integration:
payloadFormatVersion: "2.0"
type: "aws_proxy"
httpMethod: "POST"
uri:
Fn::GetAtt: [CellsFunction, Arn]
connectionType: "INTERNET"
/cells/{cellName}:
get:
responses:
default:
description: "Default response for GET /cells/{cellName}"
x-amazon-apigateway-integration:
payloadFormatVersion: "2.0"
type: "aws_proxy"
httpMethod: "POST"
uri:
Fn::GetAtt: [CellInfoFunction, Arn]
connectionType: "INTERNET"
/hello:
get:
responses:
default:
description: "Default response for GET /hello"
x-amazon-apigateway-integration:
payloadFormatVersion: "2.0"
type: "aws_proxy"
httpMethod: "POST"
uri:
Fn::GetAtt: [HelloFunction, Arn]
connectionType: "INTERNET"
/jobs/{cellName}:
get:
responses:
default:
description: "Default response for GET /jobs/{cellName}"
x-amazon-apigateway-integration:
payloadFormatVersion: "2.0"
type: "aws_proxy"
httpMethod: "POST"
uri:
Fn::GetAtt: [JobsFunction, Arn]
connectionType: "INTERNET"
x-amazon-apigateway-cors:
maxAge: 0
allowCredentials: false
allowOrigins:
- "http://localhost:3000"
x-amazon-apigateway-importexport-version: "1.0"
HelloFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.hello
Events:
Hello:
Type: HttpApi
Properties:
Path: /hello
Method: get
ApiId: !Ref ServerlessHttpApi
CellInfoFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.cell_info
Policies:
- AWSIoTFullAccess
Events:
Http:
Type: HttpApi
Properties:
Path: /cells/{cellName}
Method: get
ApiId: !Ref ServerlessHttpApi
CellsFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.cells
Policies:
- AWSIoTFullAccess
Events:
Http:
Type: HttpApi
Properties:
Path: /cells
Method: get
ApiId: !Ref ServerlessHttpApi
JobsFunction:
Type: AWS::Serverless::Function
Properties:
Handler: app.jobs
Policies:
- AmazonDynamoDBReadOnlyAccess
Events:
Http:
Type: HttpApi
Properties:
Path: /jobs/{cellName}
Method: get
ApiId: !Ref ServerlessHttpApi
Outputs:
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage"
Value: !Sub "https://${ServerlessHttpApi}.execute-api.${AWS::Region}.amazonaws.com"
As it can be seen, I have added some CORS configuration. Both in the AWS::Serverless::HttpApi and also in the DefinitionBody i.e. the OpenAPI spec.
My problem is that as far as I can tell, all CORS configurations are completely ignored. When I deploy after changing cors configuration, it says:
Waiting for changeset to be created..
Error: No changes to deploy. Stack sam-app is up to date
When I run sam validate --debug --profile [my-profile] it is my understandign that the CloudFormation file which it tries to deploy is output. It looks like this:
AWSTemplateFormatVersion: '2010-09-09'
Description: 'fotffleet-api
Sample SAM Template for fotffleet-api
'
Resources:
HelloFunction:
Properties:
Code:
S3Bucket: bucket
S3Key: value
Handler: app.hello
Role:
Fn::GetAtt:
- HelloFunctionRole
- Arn
Runtime: python3.7
Tags:
- Key: lambda:createdBy
Value: SAM
Timeout: 3
Type: AWS::Lambda::Function
HelloFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Tags:
- Key: lambda:createdBy
Value: SAM
Type: AWS::IAM::Role
HelloFunctionHelloPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: HelloFunction
Principal: apigateway.amazonaws.com
SourceArn:
Fn::Sub:
- arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/hello
- __ApiId__:
Ref: ServerlessHttpApi
__Stage__: '*'
Type: AWS::Lambda::Permission
CellInfoFunction:
Properties:
Code:
S3Bucket: bucket
S3Key: value
Handler: app.cell_info
Role:
Fn::GetAtt:
- CellInfoFunctionRole
- Arn
Runtime: python3.7
Tags:
- Key: lambda:createdBy
Value: SAM
Timeout: 3
Type: AWS::Lambda::Function
CellInfoFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/AWSIoTFullAccess
Tags:
- Key: lambda:createdBy
Value: SAM
Type: AWS::IAM::Role
CellInfoFunctionHttpPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: CellInfoFunction
Principal: apigateway.amazonaws.com
SourceArn:
Fn::Sub:
- arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/cells/*
- __ApiId__:
Ref: ServerlessHttpApi
__Stage__: '*'
Type: AWS::Lambda::Permission
CellsFunction:
Properties:
Code:
S3Bucket: bucket
S3Key: value
Handler: app.cells
Role:
Fn::GetAtt:
- CellsFunctionRole
- Arn
Runtime: python3.7
Tags:
- Key: lambda:createdBy
Value: SAM
Timeout: 3
Type: AWS::Lambda::Function
CellsFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/AWSIoTFullAccess
Tags:
- Key: lambda:createdBy
Value: SAM
Type: AWS::IAM::Role
CellsFunctionHttpPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: CellsFunction
Principal: apigateway.amazonaws.com
SourceArn:
Fn::Sub:
- arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/cells
- __ApiId__:
Ref: ServerlessHttpApi
__Stage__: '*'
Type: AWS::Lambda::Permission
JobsFunction:
Properties:
Code:
S3Bucket: bucket
S3Key: value
Handler: app.jobs
Role:
Fn::GetAtt:
- JobsFunctionRole
- Arn
Runtime: python3.7
Tags:
- Key: lambda:createdBy
Value: SAM
Timeout: 3
Type: AWS::Lambda::Function
JobsFunctionRole:
Properties:
AssumeRolePolicyDocument:
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Version: '2012-10-17'
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
- arn:aws:iam::aws:policy/AmazonDynamoDBReadOnlyAccess
Tags:
- Key: lambda:createdBy
Value: SAM
Type: AWS::IAM::Role
JobsFunctionHttpPermission:
Properties:
Action: lambda:InvokeFunction
FunctionName:
Ref: JobsFunction
Principal: apigateway.amazonaws.com
SourceArn:
Fn::Sub:
- arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/GET/jobs/*
- __ApiId__:
Ref: ServerlessHttpApi
__Stage__: '*'
Type: AWS::Lambda::Permission
ServerlessHttpApi:
Properties:
Body:
info:
title:
Ref: AWS::StackName
version: '1.0'
openapi: 3.0.1
paths:
/cells:
get:
responses: {}
x-amazon-apigateway-integration:
httpMethod: POST
payloadFormatVersion: '2.0'
type: aws_proxy
uri:
Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CellsFunction.Arn}/invocations
/cells/{cellName}:
get:
parameters:
- in: path
name: cellName
required: true
responses: {}
x-amazon-apigateway-integration:
httpMethod: POST
payloadFormatVersion: '2.0'
type: aws_proxy
uri:
Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${CellInfoFunction.Arn}/invocations
/hello:
get:
responses: {}
x-amazon-apigateway-integration:
httpMethod: POST
payloadFormatVersion: '2.0'
type: aws_proxy
uri:
Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${HelloFunction.Arn}/invocations
/jobs/{cellName}:
get:
parameters:
- in: path
name: cellName
required: true
responses: {}
x-amazon-apigateway-integration:
httpMethod: POST
payloadFormatVersion: '2.0'
type: aws_proxy
uri:
Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${JobsFunction.Arn}/invocations
tags:
- name: httpapi:createdBy
x-amazon-apigateway-tag-value: SAM
Type: AWS::ApiGatewayV2::Api
ServerlessHttpApiApiGatewayDefaultStage:
Properties:
ApiId:
Ref: ServerlessHttpApi
AutoDeploy: true
StageName: $default
Tags:
httpapi:createdBy: SAM
Type: AWS::ApiGatewayV2::Stage
Outputs:
HelloWorldApi:
Description: API Gateway endpoint URL for Prod stage
Value:
Fn::Sub: https://${ServerlessHttpApi}.execute-api.${AWS::Region}.amazonaws.com
I don't know what this should look like, but I find it strange that there is no mention of anything CORS related in there at all.
I have tried many variations such as: CORS settings only in the properites, only in the openAPI definition, not having a definition body and letting SAM generate it, having it in a seperate file.
No matter what I do, it is as if SAM completely and silently ignores any cors settings.
If it is not obvious, I would like to know what to do in order to get SAM to apply these CORS settings on deploy.
does AWS cloudformation supports creation of Kubernetes pods, services, replica controllers etc or setting up the EKS clusters and worker nodes and using Kubectl to create the resources are the only way?
Not out of the box, but you can if you use a custom resource type backed by a lambda function in CloudFormation.
The AWS EKS quickstart has an example:
AWSTemplateFormatVersion: "2010-09-09"
Description: "deploy an example workload into an existing kubernetes cluster (qs-1p817r5f9)"
Parameters:
KubeConfigPath:
Type: String
KubeConfigKmsContext:
Type: String
Default: "EKSQuickStart"
KubeClusterName:
Type: String
NodeInstanceProfile:
Type: String
QSS3BucketName:
AllowedPattern: ^[0-9a-zA-Z]+([0-9a-zA-Z-]*[0-9a-zA-Z])*$
ConstraintDescription: Quick Start bucket name can include numbers, lowercase
letters, uppercase letters, and hyphens (-). It cannot start or end with a hyphen
(-).
Default: aws-quickstart
Description: S3 bucket name for the Quick Start assets. This string can include
numbers, lowercase letters, uppercase letters, and hyphens (-). It cannot start
or end with a hyphen (-).
Type: String
QSS3KeyPrefix:
AllowedPattern: ^[0-9a-zA-Z-/.]*$
ConstraintDescription: Quick Start key prefix can include numbers, lowercase letters,
uppercase letters, hyphens (-), dots(.) and forward slash (/).
Default: quickstart-amazon-eks/
Description: S3 key prefix for the Quick Start assets. Quick Start key prefix
can include numbers, lowercase letters, uppercase letters, hyphens (-), dots(.) and
forward slash (/).
Type: String
QSS3BucketRegion:
Default: 'us-east-1'
Description: The AWS Region where the Quick Start S3 bucket (QSS3BucketName) is
hosted. When using your own bucket, you must specify this value.
Type: String
LambdaZipsBucketName:
Description: 'OPTIONAL: Bucket Name where the lambda zip files should be placed,
if left blank a bucket will be created.'
Type: String
Default: ''
K8sSubnetIds:
Type: List<AWS::EC2::Subnet::Id>
VPCID:
Type: AWS::EC2::VPC::Id
ControlPlaneSecurityGroup:
Type: AWS::EC2::SecurityGroup::Id
Conditions:
CreateLambdaZipsBucket: !Equals
- !Ref 'LambdaZipsBucketName'
- ''
UsingDefaultBucket: !Equals [!Ref QSS3BucketName, 'aws-quickstart']
Resources:
WorkloadStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: !Sub
- 'https://${S3Bucket}.s3.${S3Region}.${AWS::URLSuffix}/${QSS3KeyPrefix}templates/example-workload.template.yaml'
- S3Region: !If [UsingDefaultBucket, !Ref 'AWS::Region', !Ref QSS3BucketRegion]
S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
Parameters:
KubeManifestLambdaArn: !GetAtt KubeManifestLambda.Arn
HelmLambdaArn: !GetAtt HelmLambda.Arn
KubeConfigPath: !Ref KubeConfigPath
KubeConfigKmsContext: !Ref KubeConfigKmsContext
KubeClusterName: !Ref KubeClusterName
NodeInstanceProfile: !Ref NodeInstanceProfile
CopyZips:
Type: Custom::CopyZips
Properties:
ServiceToken: !GetAtt 'CopyZipsFunction.Arn'
DestBucket: !Ref LambdaZipsBucketName
SourceBucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
Prefix: !Ref 'QSS3KeyPrefix'
Objects:
- functions/packages/Helm/lambda.zip
- functions/packages/DeleteBucketContents/lambda.zip
- functions/packages/KubeManifest/lambda.zip
- functions/packages/LambdaEniCleanup/lambda.zip
VPCLambdaCleanup:
Type: Custom::LambdaCleanup
Properties:
ServiceToken: !GetAtt VPCLambdaCleanupLambdaFunction.Arn
Region: !Ref "AWS::Region"
LambdaFunctionNames:
- !Ref KubeManifestLambda
VPCLambdaCleanupLambdaFunction:
DependsOn: CopyZips
Type: "AWS::Lambda::Function"
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt LambdaCleanUpFunctionRole.Arn
Runtime: python3.7
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/LambdaEniCleanup/lambda.zip'
HelmLambda:
DependsOn: CopyZips
Type: AWS::Lambda::Function
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt ManifestRole.Arn
Runtime: python3.6
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/Helm/lambda.zip'
VpcConfig:
SecurityGroupIds: [ !Ref EKSLambdaSecurityGroup ]
SubnetIds: !Ref K8sSubnetIds
KubeManifestLambda:
DependsOn: CopyZips
Type: AWS::Lambda::Function
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt ManifestRole.Arn
Runtime: python3.6
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/KubeManifest/lambda.zip'
VpcConfig:
SecurityGroupIds: [ !Ref EKSLambdaSecurityGroup ]
SubnetIds: !Ref K8sSubnetIds
DeleteBucketContentsLambda:
DependsOn: CopyZips
Type: AWS::Lambda::Function
Properties:
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt DeleteBucketContentsRole.Arn
Runtime: python3.7
Timeout: 900
Code:
S3Bucket: !Ref LambdaZipsBucketName
S3Key: !Sub '${QSS3KeyPrefix}functions/packages/DeleteBucketContents/lambda.zip'
CopyZipsFunction:
Type: AWS::Lambda::Function
Properties:
Description: Copies objects from a source S3 bucket to a destination
Handler: index.handler
Runtime: python3.7
Role: !GetAtt CopyZipsRole.Arn
Timeout: 900
Code:
ZipFile: |
import json
import logging
import threading
import boto3
import cfnresponse
def copy_objects(source_bucket, dest_bucket, prefix, objects):
s3 = boto3.client('s3')
for o in objects:
key = prefix + o
copy_source = {
'Bucket': source_bucket,
'Key': key
}
print('copy_source: %s' % copy_source)
print('dest_bucket = %s'%dest_bucket)
print('key = %s' %key)
s3.copy_object(CopySource=copy_source, Bucket=dest_bucket,
Key=key)
def delete_objects(bucket, prefix, objects):
s3 = boto3.client('s3')
objects = {'Objects': [{'Key': prefix + o} for o in objects]}
s3.delete_objects(Bucket=bucket, Delete=objects)
def timeout(event, context):
logging.error('Execution is about to time out, sending failure response to CloudFormation')
cfnresponse.send(event, context, cfnresponse.FAILED, {}, physical_resource_id)
def handler(event, context):
physical_resource_id = None
if "PhysicalResourceId" in event.keys():
physical_resource_id = event["PhysicalResourceId"]
# make sure we send a failure to CloudFormation if the function is going to timeout
timer = threading.Timer((context.get_remaining_time_in_millis()
/ 1000.00) - 0.5, timeout, args=[event, context])
timer.start()
print('Received event: %s' % json.dumps(event))
status = cfnresponse.SUCCESS
try:
source_bucket = event['ResourceProperties']['SourceBucket']
dest_bucket = event['ResourceProperties']['DestBucket']
prefix = event['ResourceProperties']['Prefix']
objects = event['ResourceProperties']['Objects']
if event['RequestType'] == 'Delete':
delete_objects(dest_bucket, prefix, objects)
else:
copy_objects(source_bucket, dest_bucket, prefix, objects)
except Exception as e:
logging.error('Exception: %s' % e, exc_info=True)
status = cfnresponse.FAILED
finally:
timer.cancel()
cfnresponse.send(event, context, status, {}, physical_resource_id)
LambdaZipsBucket:
Type: AWS::S3::Bucket
Condition: CreateLambdaZipsBucket
LambdaCleanUpFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Statement:
- Action: ['sts:AssumeRole']
Effect: Allow
Principal:
Service: [lambda.amazonaws.com]
Version: '2012-10-17'
Path: /
Policies:
- PolicyName: LambdaRole
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Effect: Allow
Resource: !Sub "arn:${AWS::Partition}:logs:*:*:*"
- Action:
- 'ec2:*'
Effect: Allow
Resource: "*"
DeleteBucketContentsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- !Sub 'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
Policies:
- PolicyName: deletebucketcontents
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: s3:*
Resource:
- !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}/*'
- !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}'
CopyZipsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- !Su 'arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
Policies:
- PolicyName: lambda-copier
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: s3:GetObject
Resource: !Sub
- 'arn:${AWS::Partition}:s3:::${S3Bucket}/${QSS3KeyPrefix}*'
- S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
- Effect: Allow
Action:
- s3:PutObject
- s3:DeleteObject
Resource: !Sub 'arn:${AWS::Partition}:s3:::${LambdaZipsBucketName}/${QSS3KeyPrefix}*'
ManifestRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: eksStackPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action: s3:GetObject
Resource: !Sub
- "arn:${AWS::Partition}:s3:::${BucketName}/*"
- S3Bucket: !If [UsingDefaultBucket, !Sub '${QSS3BucketName}-${AWS::Region}', !Ref QSS3BucketName]
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
Resource:
- "*"
- Action: "kms:decrypt"
Effect: Allow
Resource: "*"
- Action: "s3:GetObject"
Effect: Allow
Resource: "*"
EKSLambdaSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for lambda to communicate with cluster API
VpcId: !Ref VPCID
ClusterControlPlaneSecurityGroupIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
Description: Allow lambda to communicate with the cluster API Server
GroupId: !Ref ControlPlaneSecurityGroup
SourceSecurityGroupId: !Ref EKSLambdaSecurityGroup
IpProtocol: tcp
ToPort: 443
FromPort: 443
It works by creating a lambda function customer resource KubeManifestLambda and HelmLambda that has kubectl and helm installed respectively, both configured with a role that allows them to access the EKS k8s cluster.
Then these custom resources can be used to deploy k8s manifests and helm charts with custom values, like in this example.
KubeManifestExample:
Type: "Custom::KubeManifest"
Version: '1.0'
Properties:
# The lambda function that executes the manifest against the cluster. This is created in one of the parent stacks
ServiceToken: !Ref KubeManifestLambdaArn
# S3 path to the encrypted config file eg. s3://my-bucket/kube/config.encrypted
KubeConfigPath: !Ref KubeConfigPath
# context for KMS to use when decrypting the file
KubeConfigKmsContext: !Ref KubeConfigKmsContext
# Kubernetes manifest
Manifest:
apiVersion: v1
kind: ConfigMap
metadata:
# If name is not specified it will be automatically generated,
# and can be retrieved with !GetAtt LogicalID.name
#
# name: test
#
# if namespace is not specified, "default" namespace will be used
namespace: kube-system
data:
# examples of consuming outputs of the HelmExample resource below's output. Creates an implicit dependency, so
# this resource will only launch once the HelmExample resource has completed successfully
ServiceCatalogReleaseName: !Ref HelmExample
ServiceCatalogKubernetesServiceName: !GetAtt HelmExample.Service0
This even lets you reference other Cloud formation resources such as RDS instances that are created as part of a workload.
You can use CloudFormation to create EKS cluster and worker nodes but you have to use kubectl for any operation on cluster like creating service, pods, deployments etc.....you can’t use CloudFormation for that
If you use CDK, you can use cluster.add_helm_chart() or HelmChart class. It will create a lambda behind the scenes.
Or you can create a lambda directly with https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.lambda_layer_kubectl/README.html