Unable to transform request to binary - aws-api-gateway

To implement binary support in api-gateway for file upload, i have used serverless-apigw-binary plugin and added necessary content types which should be converted by api-gateway.
This is my serverless.ymlfile
service: aws-java-gradle
provider:
name: aws
runtime: java8
stage: dev
region: us-east-1
custom:
apigwBinary:
types:
- 'application/pdf'
plugins:
- serverless-apigw-binary
package:
artifact: build/distributions/hello.zip
functions:
uploadLoadFiles:
handler: com.serverless.UploadFileHandler
role: UploadFileRole
timeout: 180
events:
- http:
integration: lambda
path: upload
method: put
cors:
origin: '*'
headers:
- Content-Type
- X-Amz-Date
- Authorization
- X-Api-Key
- X-Amz-Security-Token
- X-Amz-User-Agent
- X-Requested-With
allowCredentials: false
request:
passThrough: WHEN_NO_TEMPLATES
template:
application/pdf: '{ "operation":"dev-aws-java-gradle", "base64Image": "$input.body", "query": "$input.params("fileName")" }'
response:
template: $input.body
statusCodes:
400:
pattern: '.*"httpStatus":400,.*'
template: ${file(response-template.txt)}
401:
pattern: '.*"httpStatus":401,.*'
template: ${file(response-template.txt)}
403:
pattern: '.*"httpStatus":403,.*'
template: ${file(response-template.txt)}
500:
pattern: '.*"httpStatus":500,.*'
template: ${file(response-template.txt)}
501:
pattern: '.*[a-zA-Z].*'
template: ${file(unknown-error-response-template.txt)}
environment:
S3BUCKET: ${env:S3_BUCKET}
APS_ENV: ${env:APS_ENV}
# you can add CloudFormation resource templates here
resources:
Resources:
UploadFileRole:
Type: AWS::IAM::Role
Properties:
RoleName: "UploadFileRole"
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: loggingPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- 'Fn::Join':
- ':'
-
- 'arn:aws:logs'
- Ref: 'AWS::Region'
- Ref: 'AWS::AccountId'
- 'log-group:/aws/lambda/*:*:*'
all the necessary settings got implemented in api-gateway after doing sls deploy
(checked based on this article https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/)
but when i hit my end point api gateway is giving out an error like this
Verifying Usage Plan for request: 45bac722-f039-11e7-bcc6-f9a1aa509052. API Key: API Stage: fd4ue8lpia/int
API Key authorized because method 'PUT /upload' does not require API Key. Request will not contribute to throttle or quota limits
Usage Plan check succeeded for API Key and API Stage fd4ue8lpia/int
Starting execution for request: 45bac722-f039-11e7-bcc6-f9a1aa509052
HTTP Method: PUT, Resource Path: /upload
Method request path:
{}
Method request query string:
{}
Method request headers: {Accept=*/*, CloudFront-Viewer-Country=IN, postman-token=6f1a23ba-36c2-104d-2e12-dc2c76a985ee, CloudFront-Forwarded-Proto=https, CloudFront-Is-Tablet-Viewer=false, origin=chrome-extension://aicmkgpgakddgnaphhhpliifpcfhicfo, CloudFront-Is-Mobile-Viewer=false, User-Agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36, X-Forwarded-Proto=https, CloudFront-Is-SmartTV-Viewer=false, Host=fd4ue8lpia.execute-api.us-east-1.amazonaws.com, Accept-Encoding=gzip, deflate, br, X-Forwarded-Port=443, X-Amzn-Trace-Id=Root=1-5a4c530e-2c1927f85440b71b66cf0c15, Via=2.0 fc18daf68173838631b562fe2efaf8f8.cloudfront.net (CloudFront), x-postman-interceptor-id=8fa8440f-0541-fdac-c60c-6019c2269a66, X-Amz-Cf-Id=P4PYoi5Wwb-NCYRSdTQ_5TdtpbQLBaEATXoyJGC7cS8g6LsoCRPbkg==, X-Forwarded-For=157.50.20.12, 52.46.37.156, content-type=application/pdf, Accept-Language=en-US,en;q=0.9, cache-control=no-cache, CloudFront-Is-Desktop-Viewer=true}
Method request body before transformations: [Binary Data]
Execution failed due to configuration error: Unable to transform request
Method completed with status: 500
and the request is not crossing api-gateway to lambda function.
But i followed a solution mentioned here
to edit lambda function name in integration request section of resource in api-gateway to the same name and click ok. Then error's were gone and working fine.
I checked for changes in roles after doing that but none were found.
Can any one suggest what could have happened there, and any better solutions for the above mentioned problem.
Thanks in advance.

But i followed a solution mentioned here to edit lambda function name
in integration request section of resource in api-gateway to the same
name and click ok. Then error's were gone and working fine.
When you edit lambda function on the console, it sets up the permissions to call the lambda function automatically. If you want to do that manually, you can do that using the CLI

Related

AWS Private API gateway deployment error when using serverless deploy

I am hitting the below error when I try to create a PRIVATE endpoint type using serverless in AWS.
Below is the serverless.yml file for reference
provider:
name: aws
endpointType: PRIVATE
vpcEndpointIds:
- vpce-xxxx
region: ap-southeast-2
apiKeys:
- ${self:custom.actualStage}-xxxxxx2
resourcePolicy:
- Effect: Allow
Principal: '*'
Action: execute-api:Invoke
Resource:
- execute-api:/*/*/*
Condition:
StringEquals:
aws:SourceVpc:
- "vpc-xxxxx"
plugins:
- serverless-plugin-warmup
iamRoleStatements:
- Effect: 'Allow'
Action:
- 'lambda:InvokeFunction'
Resource: "*"
Error from logs:
CREATE_FAILED: ApiGatewayDeployment1655171823778 (AWS::ApiGateway::Deployment)
Resource handler returned message: "Private REST API doesn't have a resource policy attached to it (Service: ApiGateway, Status Code: 400, Request ID: e321f00e-42b6-4ef6-b984-46500ca40492)" (RequestToken: 475924b8-998d-58fc-89bd-51fc0b80f2d4, HandlerErrorCode: InvalidRequest)
Not sure if AWS released new changes in the serverless configuration for API GW. Your policy version should have worked. However, I have tested this on my end. We now have to mention the resourcePolicy under the apiGateway attribute. Please use the following this will resolve your issue:
apiGateway:
resourcePolicy:
- Effect: Allow
Principal: '*'
Action: execute-api:Invoke
Resource:
- execute-api:/*
- Effect: Deny
Principal: '*'
Action: execute-api:Invoke
Resource:
- execute-api:/*
Condition:
StringNotEquals:
aws:SourceVpc:
- 'vpc-*******'

How do I add backend rule to the API service config file?

Trying to create an API for the first time in GCP and getting the following error
ERROR: (gcloud.api-gateway.api-configs.create) API Config's backend has no rules. If using gRPC, be sure to specify the 'rules[]' under the 'Backend' field. See https://cloud.google.com/endpoints/docs/grpc-service-config/reference/rpc/google.api#backendrule for more details.
However, if I try to add a backend to the yaml file, I am getting the following error.
ERROR: (gcloud.api-gateway.api-configs.create) INVALID_ARGUMENT: Cannot convert to service config.
'location: "unknown location"
kind: ERROR
message: "Invalid OpenAPI file. Please fix the schema errors:\nerror: object instance has properties which are not allowed by the schema: ["backend"]\n level: "error"\n schema: {"loadingURI":"http://swagger.io/v2/schema.json#","pointer":""}\n instance: {"pointer":""}\n domain: "validation"\n keyword: "additionalProperties"\n unwanted: ["backend"]"
This is the bare bones yaml file I am using.
swagger: '2.0'
info:
title: sayHello
version: '1.0'
description: sayHello
schemes:
- https
produces:
- application/json
consumes:
- application/json
backend:
rules:
- selector: "*"
address: https://abc.cloudfunctions.net/sayHello
paths:
'/getPoints/{clientId}':
parameters:
- type: string
name: clientId
in: path
required: true
definitions: {}
responses:
'200':
description: Success
schema:
type: string
examples: {}
headers: {}
What am I missing? Somehow, not able to find much resources around adding backend rules to the service config file. Not even able to get the spec to validate against.
Any help would be appreciated.

What is the format to specify an external EDGE Gateway ApiId in CloudFormation templates?

I'm trying to create or update a stack with the following CloudFormation Template:
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ApiGatewayId:
Type: String
ApiLayerArn:
Type: String
JarLocation:
Type: String
Resources:
Function:
Type: 'AWS::Lambda::Function'
Properties:
Handler: net.bitsandpaper.api.kiosk.PlatformChecker
Runtime: java11
Code:
S3Bucket: bnp-build-artifacts
S3Key: !Ref JarLocation
Description: ''
MemorySize: 128
Timeout: 5
Role: arn:aws:iam::479832603967:role/bnp-api-lambda-execution-role
Layers:
- !Ref ApiLayerArn
ApiIntegration:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref ApiGatewayId
IntegrationType: AWS_PROXY
IntegrationUri: !Join
- ''
- - 'arn:'
- !Ref 'AWS::Partition'
- ':apigateway:'
- !Ref 'AWS::Region'
- ':lambda:path/2015-03-31/functions/'
- !Ref Function
- /invocations
TimeoutInMillis: 6000
ApiRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref ApiGatewayId
RouteKey: 'GET /kiosk/platform-check'
Target: !Join
- /
- - integrations
- !Ref ApiIntegration
The parameters are correctly passed by an external file, they look good in the Web Console, notably parameter ApiGatewayId has value 8548rqrsm5. Yet during deployment I have a CREATE_FAILED for ApiIntegration, with the message:
Invalid API identifier specified 479832603967:8548rqrsm5 (Service:
AmazonApiGatewayV2; Status Code: 404; Error Code: NotFoundException;
Request ID: 84918a83-cf9d-48d2-acf7-18d9d2e4d330; Proxy: null)
The API is an EDGE Rest API, in the same region than the CloudFormation stack. The ID is retrieved by the CLI with aws apigateway get-rest-apis.
Am I missing something in the ApiId format? The litterature is very scarce when not referencing an API in the same stack...
AWS::ApiGatewayV2 is only for WEBSOCKTES and HTTP types. From docs:
The API protocol. Valid values are WEBSOCKET or HTTP.
But since you are writing about Edge-optimized (not supported by HTTP api) it seems to that you are using REST API, rather then HTTP API. So you should be using AWS::ApiGateway resources, not AWS::ApiGatewayV2.
It seem's like the AWS::ApiGatewayV2::Route is created before the AWS::ApiGatewayV2::Integration. So When it trying to refer ApiIntegration it is not yet created.
So you should try to use DependsOn attribute.
With the DependsOn attribute you can specify that the creation of a
specific resource follows another. When you add a DependsOn attribute
to a resource, that resource is created only after the creation of the
resource specified in the DependsOn attribute.
Try this below CloudFormation code:
ApiRoute:
Type: AWS::ApiGatewayV2::Route
DependsOn: ApiIntegration
Properties:
ApiId: !Ref ApiGatewayId
RouteKey: 'GET /kiosk/platform-check'
Target: !Join
- /
- - integrations
- !Ref ApiIntegration
I hope this will help you out to resolve your problem.
Link: DependsOn Attribute UserGuide

API Gateway HTTP Proxy integration with serverless-offline (NOT Lambda Proxy)

I am trying to use serverless-offline to develop / simulate my API Gateway locally. My API gateway makes liberal use of the HTTP proxy integrations. The production Resource looks like this:
I have created a serverless-offline configuration based on a few documents and discussion which say that it is possible to define an HTTP Proxy integration using Cloud Formation configuration:
httpProxyWithApiGateway.md - Setting an HTTP Proxy on API Gateway by using Serverless framework.
Setting an HTTP Proxy on API Gateway (official Serverless docs: API Gateway)
I have adapted the above two configuration examples for my purposes, see below.
Have any tips, for what I might be doing wrong here?
plugins:
- serverless-offline
service: company-apig
provider:
name: aws
stage: dev
runtime: python2.7
resources:
Resources:
# Parent APIG RestApi
ApiGatewayRestApi:
Type: AWS::ApiGateway::RestApi
Properties:
Name: company-apig
Description: 'The main entry point of the APIG'
# Resource /endpoint
EndpointResource:
Type: AWS::ApiGateway::Resource
Properties:
ParentId:
Fn::GetAtt:
- ApiGatewayRestApi
- RootResourceId
PathPart: 'endpoint'
RestApiId:
Ref: ApiGatewayRestApi
# Resource /endpoint/{proxy+}
EndpointProxyPath:
Type: AWS::ApiGateway::Resource
Properties:
ParentId:
Ref: EndpointResource
PathPart: '{proxy+}'
RestApiId:
Ref: ApiGatewayRestApi
# Method ANY /endpoint/{proxy+}
EndpointProxyAnyMethod:
Type: AWS::ApiGateway::Method
Properties:
AuthorizationType: NONE
HttpMethod: ANY
Integration:
IntegrationHttpMethod: ANY
Type: HTTP_PROXY
Uri: http://endpoint.company.cool/{proxy}
PassthroughBehavior: WHEN_NO_MATCH
MethodResponses:
- StatusCode: 200
ResourceId:
Ref: EndpointProxyPath
RestApiId:
Ref: ApiGatewayRestApi
For the above configuration, I get this output. Apparently, the configuration registers no routes at all.
{
"statusCode":404,
"error":"Serverless-offline: route not found.",
"currentRoute":"get - /endpoint/ping",
"existingRoutes":[]
}
Related: I am also attempting to solve the same problem using aws-sam, at the following post - API Gateway HTTP Proxy integration with aws-sam (NOT Lambda Proxy)
By default serverless-offline doesn't parse your resources for endpoints, enable it via custom config.
custom:
serverless-offline:
resourceRoutes: true
Ends up serving:
Serverless: Routes defined in resources:
Serverless: ANY /endpoint/{proxy*} -> http://endpoint.company.cool/{proxy}
Serverless: Offline listening on http://localhost:3000
Documentation

AWS ApiGateway Proxy to non-Public S3 Bucket

The following cloudformation script sets up an Api Gateway method that proxies to an S3 bucket.
The S3BucketPolicy opens up the bucket to public read access but the AWS UI warns that this should never be done.
I tried setting the S3BucketPolicy Principal to service apigateway.amazonaws.com but this results in Access Denied.
1) What is the right way to limit bucket access to the API gateway function? (Sample YAML would be great)
2) How could I debug this Access Denied failure to get more information on why it failed?
3) Where should I be looking for sample code on what should be a very standard template snippet?
ATTEMPT #1 - Works but only by making the S3 Bucket Public otherwise Access Denied
AWSTemplateFormatVersion: 2010-09-09
Parameters:
S3BucketName:
Type: String
Description: >
Name for the S3 bucket that contains the nested templates.
Resources:
RestAPI:
Type: 'AWS::ApiGateway::RestApi'
Properties:
BinaryMediaTypes:
- '*/*'
Name: !Ref 'AWS::StackName'
RestAPIRootGET:
Type: 'AWS::ApiGateway::Method'
Properties:
AuthorizationType: NONE
HttpMethod: GET
Integration:
IntegrationHttpMethod: GET
PassthroughBehavior: WHEN_NO_MATCH
Type: HTTP_PROXY
Uri: !Sub https://${S3BucketName}.s3.amazonaws.com/static-assets/index.html
ResourceId: !GetAtt RestAPI.RootResourceId
RestApiId: !Ref RestAPI
DependsOn:
- RestAPI
BucketPolicy:
Type: AWS::S3::BucketPolicy
Properties:
Bucket: !Ref S3BucketName
PolicyDocument:
Version: 2012-10-17
Statement:
-
Sid: GetObject
Effect: Allow
Principal: "*"
Action:
- s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketName}/static-assets/*'
I think perhaps the right away is to create a role with access to the bucket and then have the ApiGateway assume this role but I'm having a hard time finding documentation that explains how to do this in a cloudformation template. (see also Michael - sqlbot comment suggesting using the credentials property of the method)
Here is my attempt which still fails with Access Denied
ATTEMPT #2 - Access Denied
AWSTemplateFormatVersion: 2010-09-09
Parameters:
S3BucketName:
Type: String
Description: >
Name for the S3 bucket that contains the nested templates.
Resources:
RestAPI:
Type: 'AWS::ApiGateway::RestApi'
Properties:
BinaryMediaTypes:
- '*/*'
Name: !Ref 'AWS::StackName'
RestAPIRootGET:
Type: 'AWS::ApiGateway::Method'
Properties:
AuthorizationType: NONE
HttpMethod: GET
Integration:
IntegrationHttpMethod: GET
PassthroughBehavior: WHEN_NO_MATCH
Type: HTTP_PROXY
Uri: !Sub https://${S3BucketName}.s3.amazonaws.com/static-assets/index.html
Credentials: !GetAtt AllowStaticAccessRole.Arn
ResourceId: !GetAtt RestAPI.RootResourceId
RestApiId: !Ref RestAPI
DependsOn:
- RestAPI
- AllowStaticAccessRole
AllowStaticAccessRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Principal:
Service:
- "apigateway.amazonaws.com"
Action:
- "sts:AssumeRole"
Path: "/"
Policies:
-
PolicyName: "AllowStaticAccessPolicy"
PolicyDocument:
Version: "2012-10-17"
Statement:
-
Effect: "Allow"
Action:
- s3:*
Resource:
- !Sub 'arn:aws:s3:::${S3BucketName}/static-assets/*'
As already mentioned in the comments, the Access Denied error could be coming from KMS rather than from the S3 itself.
To solve the issue, you need to add at minimum these permissions to the role that the Api Gateway assumes: "kms:Decrypt", "kms:ReEncryptFrom", preferably also stating which Resource this should apply too for proper least privilege implementation.