Kafka monitoring via JMX - apache-kafka

I'm using Prometheus JMX Exporter to monitor Kafka. I've defined the following pattern rules in the JMX config file:
- pattern : kafka.server<type=(.+), name=(.+)PerSec\w*, topic=(.+)><>Count
name: kafka_server_$1_$2_total
labels:
topic: "$3"
- pattern: kafka.server<type=(.+), name=(.+)PerSec\w*><>Count
name: kafka_server_$1_$2_total
type: COUNTER
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>(Count|Value)
name: kafka_server_$1_$2
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern: kafka.server<type=(.+), name=(.+), topic=(.+), partition=(.*)><>(Count|Value)
name: kafka_server_$1_$2
labels:
topic: "$3"
partition: "$4"
- pattern: kafka.server<type=(.+), name=(.+), topic=(.+)><>(Count|Value)
name: kafka_server_$1_$2
labels:
topic: "$3"
type: COUNTER
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>(Count|Value)
name: kafka_server_$1_$2
labels:
clientId: "$3"
broker: "$4:$5"
- pattern: kafka.server<type=(.+), name=(.+), clientId=(.+)><>(Count|Value)
name: kafka_server_$1_$2
labels:
clientId: "$3"
- pattern: kafka.server<type=(.+), name=(.+)><>(Count|Value)
name: kafka_server_$1_$2
Now I'm having the following issue. When I send data to the topic in this way:
/bin/kafka-console-producer.sh --broker-list kafka-hostname:9092 --topic test1
The counter of the metric kafka_server_brokertopicmetrics_bytesin_total increases correctly.
When I try to send data by using the following code:
"use strict";
const envs = process.env;
const options = {
"metadata.broker.list": "kafka-hostname:9092",
"group.id": "kafka1",
topic: "test1",
key: "testKey"
};
const kafkesque = require("untubo")(options);
let count = 0;
const interval = setInterval(function() {
kafkesque.push({ hello: "world", count });
console.log("sent", count);
count++;
}, 500);
process.once("SIGINT", function() {
clearInterval(interval);
console.log("closing");
kafkesque.stop(() => {
console.log("closed");
});
});
In this case the metric doesn't change at all but I can receive the message in the consumer. I think there is something not configured properly in the pattern. Do you have any idea?

Related

Get part of yaml file with terraform

I have a yaml file as given below
apiVersion: helm.toolkit.gitops.io/v2beta1
kind: HelmRelease
metadata:
name: "test"
namespace: "test-system"
spec:
chart:
spec:
chart: "test-environment"
version: "0.1.10"
values:
key1: "value1"
key2: "value2"
key3: "value3"
key4: "value4"
gitRepository:
url: https://github.com/test-eng/test.git
helmRepositories:
- name: testplatform
url: https://test-platform/charts
How I can read only part of it using terraform. Read all the contents below ".spec.values".
key1: "value1"
key2: "value2"
key3: "value3"
key4: "value4"
gitRepository:
url: https://github.com/test-eng/test.git
helmRepositories:
- name: testplatform
url: https://test-platform/charts
I tried with yamldecode function as given below
flux_config = yamldecode((data.github_repository_file.my_file.content)[".spec.values"])
but it failed with below error
This value does not have any indices.

MongoDB credentials are not working with StatefulSet

I have this sts:
apiVersion: "apps/v1"
kind: "StatefulSet"
metadata:
name: "mongo-benchmark"
spec:
serviceName: mongo-benchmark-headless
replicas: 1
selector:
matchLabels:
app: "mongo-benchmark"
template:
metadata:
labels:
app: "mongo-benchmark"
spec:
containers:
- name: "mongo-benchmark"
image: "mongo:5"
imagePullPolicy: "IfNotPresent"
env:
- name: "MONGO_INITDB_ROOT_USERNAME"
value: "admin"
- name: "MONGO_INITDB_ROOT_PASSWORD"
value: "admin"
ports:
- containerPort: 27017
name: "mongo-port"
volumeMounts:
- name: "mongo-benchmark-data"
mountPath: "/data/db"
volumes:
- name: "mongo-benchmark-data"
persistentVolumeClaim:
claimName: "mongo-benchmark-pvc"
Everything is deployed.
The root user's username and password is admin
But when I go to the pod terminal and execute these commands I get:
$ mongo
$ use admin
$ db.auth("admin", "admin")
Error: Authentication failed.
0
I can't even read/write from/to other databases.
For example:
$ mongo
$ use test
$ db.col.findOne({})
uncaught exception: Error: error: {
"ok" : 0,
"errmsg" : "not authorized on test to execute command { find: \"col\", filter: {}, limit: 1.0, singleBatch: true, lsid: { id: UUID(\"30788b3e-48f0-4ff0-aaec-f17e20c67bde\") }, $db: \"test\" }",
"code" : 13,
"codeName" : "Unauthorized"
}
I don't know where I'm doing wrong. Anyone knows how to authenticate?

Sidekiq failing to connect to postgresql database

I am attempting to deploy sidekiq as a sidecar container alongside Discourse and I am receiving the following error
2022-05-31T02:57:01.242Z pid=1 tid=cd1 WARN:
ActiveRecord::ConnectionNotEstablished: could not connect to server:
No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Both Sidekiq and Discourse uses the same bitnami docker image with the only difference is the Sidekiq container has a run file thats ran to start sidekiq. The postgreql server I am connecting to is an existing server and Discourse itself doesn't seem to have any issues connecting to it. I have looked at the run file for sidekiq and I don't think it's pulling the env variables properly. I have tried various different variable notations thinking it was a syntax issue. Below is the deployment I am using, Any insight would be greatly appreciated
containers:
- name: discourse
image: bitnami/discourse
livenessProbe:
tcpSocket:
port: 3000
initialDelaySeconds: 90
periodSeconds: 90
env:
- name: DISCOURSE_HOST
value: "xxx"
- name: DISCOURSE_DATABASE_HOST
value: "my-release-postgresql.default"
- name: DISCOURSE_DATABASE_PORT_NUMBER
value: "5432"
- name: DISCOURSE_DATABASE_USER
value: "postgres"
- name: DISCOURSE_DATABASE_PASSWORD
value: "xxx"
- name: DISCOURSE_DATABASE_NAME
value: "bitnami_discourse"
- name: DISCOURSE_REDIS_HOST
value: "redis.redis"
- name: DISCOURSE_REDIS_PORT_NUMER
value: "6379"
- name: POSTGRESQL_CLIENT_DATABASE_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_CLIENT_DATABASE_PORT_NUMBER
value: "5432"
- name: POSTGRESQL_CLIENT_POSTGRES_USER
value: "postgres"
- name: POSTGRESQL_CLIENT_POSTGRES_PASSWORD
value: "xxx"
- name: POSTGRESQL_CLIENT_CREATE_DATABASE_NAME
value: "bitnami_discourse"
- name: POSTGRESQL_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_PORT_NUMBER
value: "5432"
- name: DISCOURSE_POSTGRESQL_USERNAME
value: "postgres"
- name: DISCOURSE_POSTGRESQL_PASSWORD
value: "xxx"
- name: DISCOURSE_POSTGRESQL_NAME
value: "bitnami_discourse"
- name: DISCOURSE_SMTP_HOST
value: "smtp.mailgun.com"
- name: DISCOURSE_SMTP_PORT
value: "587"
- name: DISCOURSE_SMTP_USER
value: "xxx"
- name: DISCOURSE_SMTP_PASSWORD
value: "xxx"
- name: DISCOURSE_SMTP_PROTOCOL
value: "tls"
ports:
- name: portone
containerPort: 3000
- name: porttwo
containerPort: 5432
- name: portthree
containerPort: 6379
volumeMounts:
- mountPath: "/bitnami/discourse"
name: discourse
- name: sidekiq
image: docker.io/bitnami/discourse
command: ["/opt/bitnami/scripts/discourse-sidekiq/run.sh"]
env:
- name: DISCOURSE_HOST
value: "xxx"
- name: DISCOURSE_DATABASE_HOST
value: "my-release-postgresql.default"
- name: DISCOURSE_DATABASE_PORT_NUMBER
value: "5432"
- name: DISCOURSE_DATABASE_USER
value: "postgres"
- name: DISCOURSE_DATABASE_PASSWORD
value: "xxx"
- name: DISCOURSE_DATABASE_NAME
value: "bitnami_discourse"
- name: DISCOURSE_REDIS_HOST
value: "redis.redis"
- name: DISCOURSE_REDIS_PORT_NUMER
value: "6379"
- name: DISCOURSE_SMTP_HOST
value: "smtp.mailgun.com"
- name: DISCOURSE_SMTP_PORT
value: "587"
- name: DISCOURSE_SMTP_USER
value: "xxx"
- name: DISCOURSE_SMTP_PASSWORD
value: "xxx"
- name: DISCOURSE_SMTP_PROTOCOL
value: "tls"
- name: POSTGRESQL_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_PORT_NUMBER
value: "5432"
- name: DISCOURSE_POSTGRESQL_USERNAME
value: "postgres"
- name: DISCOURSE_POSTGRESQL_PASSWORD
value: "xxx"
- name: DISCOURSE_POSTGRESQL_NAME
value: "bitnami_discourse"
- name: POSTGRESQL_CLIENT_DATABASE_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_CLIENT_DATABASE_PORT_NUMBER
value: "5432"
- name: POSTGRESQL_CLIENT_POSTGRES_USER
value: "postgres"
- name: POSTGRESQL_CLIENT_POSTGRES_PASSWORD
value: "xxx"
Hello you need to add one more command ./opt/bitnami/scripts/discourse-sidekiq/setup.sh in sidekiq container command.
e.g
containers:
- name: discourse
image: bitnami/discourse
livenessProbe:
tcpSocket:
port: 3000
initialDelaySeconds: 90
periodSeconds: 90
env:
- name: DISCOURSE_HOST
value: "xxx"
- name: DISCOURSE_DATABASE_HOST
value: "my-release-postgresql.default"
- name: DISCOURSE_DATABASE_PORT_NUMBER
value: "5432"
- name: DISCOURSE_DATABASE_USER
value: "postgres"
- name: DISCOURSE_DATABASE_PASSWORD
value: "xxx"
- name: DISCOURSE_DATABASE_NAME
value: "bitnami_discourse"
- name: DISCOURSE_REDIS_HOST
value: "redis.redis"
- name: DISCOURSE_REDIS_PORT_NUMER
value: "6379"
- name: POSTGRESQL_CLIENT_DATABASE_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_CLIENT_DATABASE_PORT_NUMBER
value: "5432"
- name: POSTGRESQL_CLIENT_POSTGRES_USER
value: "postgres"
- name: POSTGRESQL_CLIENT_POSTGRES_PASSWORD
value: "xxx"
- name: POSTGRESQL_CLIENT_CREATE_DATABASE_NAME
value: "bitnami_discourse"
- name: POSTGRESQL_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_PORT_NUMBER
value: "5432"
- name: DISCOURSE_POSTGRESQL_USERNAME
value: "postgres"
- name: DISCOURSE_POSTGRESQL_PASSWORD
value: "xxx"
- name: DISCOURSE_POSTGRESQL_NAME
value: "bitnami_discourse"
- name: DISCOURSE_SMTP_HOST
value: "smtp.mailgun.com"
- name: DISCOURSE_SMTP_PORT
value: "587"
- name: DISCOURSE_SMTP_USER
value: "xxx"
- name: DISCOURSE_SMTP_PASSWORD
value: "xxx"
- name: DISCOURSE_SMTP_PROTOCOL
value: "tls"
ports:
- name: portone
containerPort: 3000
- name: porttwo
containerPort: 5432
- name: portthree
containerPort: 6379
volumeMounts:
- mountPath: "/bitnami/discourse"
name: discourse
- name: sidekiq
image: docker.io/bitnami/discourse
command:
- bash
- -c
- |
./opt/bitnami/scripts/discourse-sidekiq/setup.sh
./opt/bitnami/scripts/discourse-sidekiq/run.sh
env:
- name: DISCOURSE_HOST
value: "xxx"
- name: DISCOURSE_DATABASE_HOST
value: "my-release-postgresql.default"
- name: DISCOURSE_DATABASE_PORT_NUMBER
value: "5432"
- name: DISCOURSE_DATABASE_USER
value: "postgres"
- name: DISCOURSE_DATABASE_PASSWORD
value: "xxx"
- name: DISCOURSE_DATABASE_NAME
value: "bitnami_discourse"
- name: DISCOURSE_REDIS_HOST
value: "redis.redis"
- name: DISCOURSE_REDIS_PORT_NUMER
value: "6379"
- name: DISCOURSE_SMTP_HOST
value: "smtp.mailgun.com"
- name: DISCOURSE_SMTP_PORT
value: "587"
- name: DISCOURSE_SMTP_USER
value: "xxx"
- name: DISCOURSE_SMTP_PASSWORD
value: "xxx"
- name: DISCOURSE_SMTP_PROTOCOL
value: "tls"
- name: POSTGRESQL_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_PORT_NUMBER
value: "5432"
- name: DISCOURSE_POSTGRESQL_USERNAME
value: "postgres"
- name: DISCOURSE_POSTGRESQL_PASSWORD
value: "xxx"
- name: DISCOURSE_POSTGRESQL_NAME
value: "bitnami_discourse"
- name: POSTGRESQL_CLIENT_DATABASE_HOST
value: "my-release-postgresql.default"
- name: POSTGRESQL_CLIENT_DATABASE_PORT_NUMBER
value: "5432"
- name: POSTGRESQL_CLIENT_POSTGRES_USER
value: "postgres"
- name: POSTGRESQL_CLIENT_POSTGRES_PASSWORD
value: "xxx"

Using Pulumi and Azure, is there any API to create a SecretProviderClass without using yaml?

I'm trying to find a better way to solve this scenario than resorting to a yaml inside a pulumi.apply call (which has problems with preview apparently).
The idea here is (using Azure Kubernetes) to create a secret and then make it available inside a pod (nginx pod here just for test purposes).
The current code works, but is there an API that I'm missing?
Started to mess around with:
const foobar = new k8s.storage.v1beta1.CSIDriver("testCSI", { ...
but not really sure if it is the right path and if it is, what to put where to get the same effect.
Sidenote, no, I do not want to put secrets into environment variables. Although convenient they leak in the gui and logs and possibly more places.
const provider = new k8s.Provider("provider", {
kubeconfig: config.kubeconfig,
namespace: "default",
});
const secret = new keyvault.Secret("mysecret", {
resourceGroupName: environmentResourceGroupName,
vaultName: keyVaultName,
secretName: "just-some-secret",
properties: {
value: administratorLogin,
},
});
pulumi.all([environmentTenantId, keyVaultName, clusterManagedIdentityClientId])
.apply(([environmentTenantId, keyVaultName, clusterManagedIdentityClientId]) => {
let yammie = `apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kvname-system-msi
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "${clusterManagedIdentityClientId}"
keyvaultName: ${keyVaultName}
cloudName: ""
objects: |
array:
- |
objectName: just-some-secret
objectType: secret
tenantId: ${environmentTenantId}`;
const yamlConfigGroup = new k8s.yaml.ConfigGroup("test-secret",
{
yaml: yammie,
},
{
provider: provider,
dependsOn: [secret],
}
);
});
const deployment = new k8s.apps.v1.Deployment(
name,
{
metadata: {
labels: appLabels,
},
spec: {
replicas: 1,
selector: { matchLabels: appLabels },
template: {
metadata: {
labels: appLabels,
},
spec: {
containers: [
{
name: name,
image: "nginx:latest",
ports: [{ name: "http", containerPort: 80 }],
volumeMounts: [
{
name: "secrets-store01-inline",
mountPath: "/mnt/secrets-store",
readOnly: true,
},
],
},
],
volumes: [
{
name: "secrets-store01-inline",
csi: {
driver: "secrets-store.csi.k8s.io",
readOnly: true,
volumeAttributes: { secretProviderClass: "azure-kvname-system-msi" },
},
},
],
},
},
},
},
{
provider: provider,
}
);
SecretsProviderClass is a CustomResource which isn't typed because the fields can be anything you want.
const secret = new k8s.apiextensions.CustomResource("cert", {
apiVersion: "secrets-store.csi.x-k8s.io/v1",
kind: "SecretProviderClass",
metadata: {
namespace: "kube-system",
},
spec: {
provider: "azure",
secretObjects: [{
data: [{
objectName: cert.certificate.name,
key: "tls.key",
}, {
objectName: cert.certificate.name,
key: "tls.crt"
}],
secretName: "ingress-tls-csi",
type: "kubernetes.io/tls",
}],
parameters: {
usePodIdentity: "true",
keyvaultName: cert.keyvault.name,
objects: pulumi.interpolate`array:\n - |\n objectName: ${cert.certificate.name}\n objectType: secret\n`,
tenantId: current.then(config => config.tenantId),
}
}
}, { provider: k8sCluster.k8sProvider })
Note: the objects array might work with JSON.stringify, but I haven't yet tried that.
If you want to get strong typing for a card, you can use crd2pulumi

cloudformation - apigateway stages to multiple lambda alias and version

How can I deploy APIGateway stages for already lambda alias and version to specific stage, means I dont want to update lambda again, but I want to do mapping to the new stage or update the existing stage with specific lambda alias
i.e I have myFunction version 2,3 and alias dev, test and stage.
want to map stage of /dev to $LATEST, /test to alias test with version 2, /stage to alias with version 3.
How to achieve this.
I have tried with ${!stageVariables.lambdaAlias} on Method Integration but I get internal server with log says Invalid permission
apiGateway:
Type: "AWS::ApiGateway::RestApi"
Properties:
Name: "StacksampleapidevNewPOC"
Description: "SAMPLE New Template API"
apiGatewayResource:
Type: "AWS::ApiGateway::Resource"
Properties:
ParentId: !GetAtt
- apiGateway
- RootResourceId
PathPart: "MyFunction"
RestApiId: !Ref "apiGateway"
ApiAuthorizer:
Type: "AWS::ApiGateway::Authorizer"
Properties:
AuthorizerResultTtlInSeconds: 300
IdentitySource: method.request.header.Authorization
Name: CognitoDefaultUserPoolAuthorizer
ProviderARNs:
- arn:aws:cognito-idp:ap-south-1:accountid:userpool/poolid
RestApiId: !Ref apiGateway
Type: "COGNITO_USER_POOLS"
apiGatewayStage:
Type: AWS::ApiGateway::Stage
Properties:
RestApiId: !Ref "apiGateway"
StageName: sampledev
TracingEnabled: Yes
DeploymentId: !Ref "apiGatewayDeployment"
Variables:
ClientMaster: ClientMaster_dev
UserMaster: UserMaster_dev
RedisCacheEndpoint: !Ref RedisCacheEndpoint
UserClientMapping: UserClientMapping_dev
lambdaAlias: dev
apiGatewayStage1:
Type: AWS::ApiGateway::Stage
Properties:
RestApiId: !Ref "apiGateway"
StageName: sampletest
TracingEnabled: Yes
DeploymentId: !Ref "apiGatewayDeployment"
Variables:
ClientMaster: ClientMaster_dev
UserMaster: UserMaster_dev
RedisCacheEndpoint: !Ref RedisCacheEndpoint
UserClientMapping: UserClientMapping_dev
lambdaAlias: test
apiGatewayStage2:
Type: AWS::ApiGateway::Stage
Properties:
RestApiId: !Ref "apiGateway"
StageName: samplestage
TracingEnabled: Yes
DeploymentId: !Ref "apiGatewayDeployment"
Variables:
ClientMaster: ClientMaster_dev
UserMaster: UserMaster_dev
RedisCacheEndpoint: !Ref RedisCacheEndpoint
UserClientMapping: UserClientMapping_dev
lambdaAlias: stage
apiGatewayRootMethod:
Type: 'AWS::ApiGateway::Method'
Properties:
AuthorizationType: "COGNITO_USER_POOLS"
AuthorizerId: !Ref ApiAuthorizer
HttpMethod: POST
Integration:
Type: "AWS_PROXY"
IntegrationHttpMethod: POST
Uri: !Sub
- "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}:${!stageVariables.lambdaAlias}/invocations"
- lambdaArn: !GetAtt "MyFunction.Arn"
IntegrationResponses:
- StatusCode: 200
ResponseTemplates:
application/json: ''
ResponseParameters:
method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
method.response.header.Access-Control-Allow-Methods: "'POST,OPTIONS'"
method.response.header.Access-Control-Allow-Origin: "'*'"
RequestTemplates:
application/json: $input.json('$')
RequestParameters:
method.request.querystring.name: false
ResourceId: !Ref "apiGatewayResource"
RestApiId: !Ref apiGateway
MethodResponses:
- ResponseModels:
application/json: Empty
ResponseParameters:
method.response.header.Access-Control-Allow-Headers: true
method.response.header.Access-Control-Allow-Methods: true
method.response.header.Access-Control-Allow-Origin: true
StatusCode: '200'
apiGatewayCORSOptionMethod:
Type: "AWS::ApiGateway::Method"
Properties:
ResourceId: !Ref apiGatewayResource
RestApiId: !Ref apiGateway
AuthorizationType: NONE
HttpMethod: OPTIONS
Integration:
Type: MOCK
IntegrationResponses:
- ResponseParameters:
method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
method.response.header.Access-Control-Allow-Methods: "'POST,OPTIONS'"
method.response.header.Access-Control-Allow-Origin: "'*'"
ResponseTemplates:
application/json: ''
StatusCode: '200'
PassthroughBehavior: WHEN_NO_MATCH
RequestTemplates:
application/json: '{"statusCode": 200}'
MethodResponses:
- ResponseModels:
application/json: Empty
ResponseParameters:
method.response.header.Access-Control-Allow-Headers: true
method.response.header.Access-Control-Allow-Methods: true
method.response.header.Access-Control-Allow-Origin: true
StatusCode: '200'
apiGatewayDeployment:
Type: "AWS::ApiGateway::Deployment"
DependsOn: apiGatewayRootMethod
# DependsOn: [
# apiGatewayRootMethod,
# GetRightMenuapiGatewayRootMethod,
# GetAreaapiGatewayRootMethod,
# ResetRedisCacheapiGatewayRootMethod,
# # GetChartsByUseCaseIDapiGatewayRootMethod,
# ShowUserClientMappingsapiGatewayRootMethod,
# GetChartKPIValuesapiGatewayRootMethod,
# GetChartUseCaseMappingsapiGatewayRootMethod]
Properties:
RestApiId: !Ref "apiGateway"
# StageName: !Ref "apiGatewayStageName"
MyFunction:
Type: "AWS::Lambda::Function"
Properties:
Handler: PwC.SAMPLE.Lambda::PwC.SAMPLE.Lambda.Functions.Common.MyFunction::Run
FunctionName: MyFunction_LambdaName
Runtime: dotnetcore2.1
Code:
S3Bucket: "s3-sample-api-dev"
S3Key: !Ref "CodeZip"
MemorySize: 512
Timeout: 30
Role:
Ref: Role
VpcConfig:
SecurityGroupIds:
Ref: SecurityGroupIds
SubnetIds:
Ref: SubnetIds
MyFunctionVersion:
DeletionPolicy: Retain
Type: AWS::Lambda::Version
Properties:
FunctionName:
Ref: MyFunction
MyFunctionAliasDev:
Type: AWS::Lambda::Alias
Properties:
FunctionName:
Ref: MyFunction
FunctionVersion: devversion
Name: dev
# MyFunctionAliasDev:
# Type: AWS::Lambda::Alias
# Properties:
# FunctionName:
# Ref: MyFunction
# FunctionVersion:
# Fn::GetAtt:
# - MyFunctionVersion
# - Version
# Name: dev
MyFunctionAliasTest:
Type: AWS::Lambda::Alias
Properties:
FunctionName:
Ref: MyFunction
FunctionVersion: testversion
Name: test
MyFunctionAliasStage:
Type: AWS::Lambda::Alias
Properties:
FunctionName:
Ref: MyFunction
FunctionVersion: stageversion
Name: stage
MyFunctionlambdaApiGatewayInvoke:
Type: "AWS::Lambda::Permission"
Properties:
Action: "lambda:InvokeFunction"
FunctionName: !GetAtt "MyFunction.Arn"
Principal: "apigateway.amazonaws.com"
SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${apiGateway}/*/POST/MyFunction"```
I have achieved this with following changes on cloudformation template by calling Lambda permission by each alias created.
Now I can see each lambda alias and version has permission to APIGateway call Lambda function
Here is the sample yaml code I used to fix this issue.
AWSTemplateFormatVersion: "2010-09-09"
Description: "My API Gateway and Lambda function"
Parameters:
apiGatewayStageName:
Type: "String"
AllowedPattern: "^[a-z0-9]+$"
Default: "samplesample"
Role:
Type: String
Default: arn:aws:iam::accountid:role/Fincockpit_AuroraServerless
Description: ''
SecurityGroupIds:
Default: "sgid"
Description: ""
Type: CommaDelimitedList
SubnetIds:
Default: "subnet"
Description: ""
Type: CommaDelimitedList
Policies:
Type: CommaDelimitedList
Default: AWSLambdaFullAccess,AmazonRDSFullAccess,AmazonEC2FullAccess,AmazonDynamoDBFullAccess,AmazonVPCFullAccess
Description: ''
CodeZip:
Type: String
Description: SAMPLE API Build Package
RedisCacheEndpoint:
Type: String
Default: 'redisendpoint'
Environment:
Type: String
Default: sample
S3Bucket:
Type: String
Default: s3-changeme-api-sample
# AliasName:
# Type: String
# Default: stagename
FunctionVersion:
Type: String
Default: commitid
Resources:
apiGateway:
Type: "AWS::ApiGateway::RestApi"
Properties:
Name: "StackchangemeapisampleNewPOC"
Description: "SAMPLE New Template API"
apiGatewayResource:
Type: "AWS::ApiGateway::Resource"
Properties:
ParentId: !GetAtt
- apiGateway
- RootResourceId
PathPart: "MyFunction"
RestApiId: !Ref "apiGateway"
ApiAuthorizer:
Type: "AWS::ApiGateway::Authorizer"
Properties:
AuthorizerResultTtlInSeconds: 300
IdentitySource: method.request.header.Authorization
Name: CognitoDefaultUserPoolAuthorizer
ProviderARNs:
- arn:aws:cognito-idp:ap-south-1:accountid:userpool/poolid
RestApiId: !Ref apiGateway
Type: "COGNITO_USER_POOLS"
apiGatewayStage:
Type: AWS::ApiGateway::Stage
Properties:
RestApiId: !Ref "apiGateway"
StageName: changemesample
TracingEnabled: Yes
DeploymentId: !Ref "apiGatewayDeployment"
Variables:
ClientMaster: ClientMaster_sample
UserMaster: UserMaster_sample
RedisCacheEndpoint: !Ref RedisCacheEndpoint
UserClientMapping: UserClientMapping_sample
lambdaAlias: sample
apiGatewayStage1:
Type: AWS::ApiGateway::Stage
Properties:
RestApiId: !Ref "apiGateway"
StageName: changemetest
TracingEnabled: Yes
DeploymentId: !Ref "apiGatewayDeployment"
Variables:
ClientMaster: ClientMaster_test
UserMaster: UserMaster_ctest
RedisCacheEndpoint: "sample-redis-test.hreh1d.ng.0001.aps1.cache.amazonaws.com:6379"
UserClientMapping: UserClientMapping_test
lambdaAlias: test
apiGatewayStage2:
Type: AWS::ApiGateway::Stage
Properties:
RestApiId: !Ref "apiGateway"
StageName: samplestage
TracingEnabled: Yes
DeploymentId: !Ref "apiGatewayDeployment"
Variables:
ClientMaster: ClientMaster_stage
UserMaster: UserMaster_stage
RedisCacheEndpoint: "sample-redis-stage.hreh1d.ng.0001.aps1.cache.amazonaws.com:6379"
UserClientMapping: UserClientMapping_stage
lambdaAlias: stage
apiGatewayRootMethod:
DependsOn: [
MyFunctionlambdaApiGatewayInvokeDev,
MyFunctionlambdaApiGatewayInvokeTest,
MyFunctionlambdaApiGatewayInvokeStage]
Type: 'AWS::ApiGateway::Method'
Properties:
AuthorizationType: "COGNITO_USER_POOLS"
AuthorizerId: !Ref ApiAuthorizer
HttpMethod: POST
Integration:
Type: "AWS_PROXY"
IntegrationHttpMethod: POST
Uri: !Sub
- "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}:${!stageVariables.lambdaAlias}/invocations"
- lambdaArn: !GetAtt "MyFunction.Arn"
IntegrationResponses:
- StatusCode: 200
ResponseTemplates:
application/json: ''
ResponseParameters:
method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
method.response.header.Access-Control-Allow-Methods: "'POST,OPTIONS'"
method.response.header.Access-Control-Allow-Origin: "'*'"
RequestTemplates:
application/json: $input.json('$')
RequestParameters:
method.request.querystring.name: false
ResourceId: !Ref "apiGatewayResource"
RestApiId: !Ref apiGateway
MethodResponses:
- ResponseModels:
application/json: Empty
ResponseParameters:
method.response.header.Access-Control-Allow-Headers: true
method.response.header.Access-Control-Allow-Methods: true
method.response.header.Access-Control-Allow-Origin: true
StatusCode: '200'
apiGatewayCORSOptionMethod:
Type: "AWS::ApiGateway::Method"
Properties:
ResourceId: !Ref apiGatewayResource
RestApiId: !Ref apiGateway
AuthorizationType: NONE
HttpMethod: OPTIONS
Integration:
Type: MOCK
IntegrationResponses:
- ResponseParameters:
method.response.header.Access-Control-Allow-Headers: "'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token'"
method.response.header.Access-Control-Allow-Methods: "'POST,OPTIONS'"
method.response.header.Access-Control-Allow-Origin: "'*'"
ResponseTemplates:
application/json: ''
StatusCode: '200'
PassthroughBehavior: WHEN_NO_MATCH
RequestTemplates:
application/json: '{"statusCode": 200}'
MethodResponses:
- ResponseModels:
application/json: Empty
ResponseParameters:
method.response.header.Access-Control-Allow-Headers: true
method.response.header.Access-Control-Allow-Methods: true
method.response.header.Access-Control-Allow-Origin: true
StatusCode: '200'
apiGatewayDeployment:
Type: "AWS::ApiGateway::Deployment"
DependsOn: apiGatewayRootMethod
# DependsOn: [
# apiGatewayRootMethod,
# GetRightMenuapiGatewayRootMethod,
# GetAreaapiGatewayRootMethod,
# ResetRedisCacheapiGatewayRootMethod,
# # GetChartsByUseCaseIDapiGatewayRootMethod,
# ShowUserClientMappingsapiGatewayRootMethod,
# GetChartKPIValuesapiGatewayRootMethod,
# GetChartUseCaseMappingsapiGatewayRootMethod]
Properties:
RestApiId: !Ref "apiGateway"
# StageName: !Ref "apiGatewayStageName"
MyFunction:
Type: "AWS::Lambda::Function"
Properties:
Handler: PwC.SAMPLE.Lambda::PwC.SAMPLE.Lambda.Functions.Common.MyFunction::Run
FunctionName: MyFunction_LambdaName
Runtime: dotnetcore2.1
Code:
S3Bucket: "s3-sample-api-sample"
S3Key: !Ref "CodeZip"
MemorySize: 512
Timeout: 30
Role:
Ref: Role
VpcConfig:
SecurityGroupIds:
Ref: SecurityGroupIds
SubnetIds:
Ref: SubnetIds
MyFunctionVersion:
DeletionPolicy: Retain
Type: AWS::Lambda::Version
Properties:
FunctionName:
Ref: MyFunction
MyFunctionAliasDev:
Type: AWS::Lambda::Alias
Properties:
FunctionName:
Ref: MyFunction
FunctionVersion:
Fn::GetAtt:
- MyFunctionVersion
- Version
Name: dev
MyFunctionAliasTest:
Type: AWS::Lambda::Alias
Properties:
FunctionName:
Ref: MyFunction
FunctionVersion: testversion
Name: test
MyFunctionAliasStage:
Type: AWS::Lambda::Alias
Properties:
FunctionName:
Ref: MyFunction
FunctionVersion: stageversion
Name: stage
MyFunctionlambdaApiGatewayInvokeDev:
Type: "AWS::Lambda::Permission"
Properties:
Action: "lambda:InvokeFunction"
FunctionName: !Ref "MyFunctionAliasDev"
Principal: "apigateway.amazonaws.com"
SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${apiGateway}/*/POST/MyFunction"
MyFunctionlambdaApiGatewayInvokeTest:
Type: "AWS::Lambda::Permission"
Properties:
Action: "lambda:InvokeFunction"
FunctionName: !Ref "MyFunctionAliasTest"
Principal: "apigateway.amazonaws.com"
SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${apiGateway}/*/POST/MyFunction"
MyFunctionlambdaApiGatewayInvokeStage:
Type: "AWS::Lambda::Permission"
Properties:
Action: "lambda:InvokeFunction"
FunctionName: !Ref "MyFunctionAliasStage"
Principal: "apigateway.amazonaws.com"
SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${apiGateway}/*/POST/MyFunction"