Question
What is the correct way to get the output of a cloudformation stack in a serverless.yml file without hardcoding the stack name?
Steps
I have a serverless.yml file where I import a cloudformation template to create an ElastiCache cluster.
When I try to do so, I get this error:
Serverless Error ---------------------------------------
Invalid variable reference syntax for variable AWS::StackName. You can only reference env vars, options, & files. You can check our docs for more info.
In my file I'd like to expose as an environment variable the ElastiCacheAddress output from the cloudformation stack. I am using the serverless pseudo-parameters plugin to get the output:
# Here is where I try to reference the CF output value
service: hello-world
provider:
name: aws
# ...
environment:
cacheUrl: ${cf:#{AWS::StackName}.ElastiCacheAddress}
# Reference to the CF template
resources:
- '${file(./cf/cf-elasticache.yml)}'
The CF template is the one from the AWS Samples GitHub repository.
The snippet with the output is here:
ElastiCacheAddress:
Description: ElastiCache endpoint address
Value: !If [ IsRedis, !GetAtt ElastiCacheCluster.RedisEndpoint.Address, !GetAtt ElastiCacheCluster.ConfigurationEndpoint.Address]
Export:
Name: !Sub ${AWS::StackName}-ElastiCacheAddress
You can use a workaround to get your way through these syntax caveats.
In this case, I would suggest you to create a custom node to set variables you would want to reuse. You can then reference these variables using Serverless Framework syntax only, to avoid that error, like so:
# Here is where I try to reference the CF output value
service: hello-world
custom:
stackName:'#{AWS::StackName}'
provider:
name: aws
# ...
environment:
cacheUrl: ${cf:${self:custom.stackName}.ElastiCacheAddress}
# Reference to the CF template
resources:
- '${file(./cf/cf-elasticache.yml)}'
Related
I'm having some issues when trying to use Hashicorp vault template (kubernetes with Google Kubernetes Engine) with to.be.continuous.
Actually when I use it with Google Docker Kaniko layer I got an error message: ... wget: bad address 'vault-secrets-provider'.
It seems that Kaniko doesn't recognize the vault-secrets-provider layer. Would you please help me with this? Or perhaps, where I can ask for some help?
This is a summary of .gitlab-ci.yml
# Kubernetes template
- project: 'to-be-continuous/kubernetes'
ref: '2.0.4'
file: '/templates/gitlab-ci-k8s.yml'
- project: "to-be-continuous/kubernetes"
ref: "2.0.4"
file: "templates/gitlab-ci-k8s-vault.yml"
...
K8S_DEFAULT_KUBE_CONFIG: "#url#http://vault-secrets-provider/api/secrets/noprod?field=kube_config"
VAULT_BASE_URL: "http://myvault.myserver.com/v1"
Error Message:
[ERROR] Failed getting secret K8S_DEFAULT_KUBE_CONFIG:
... wget: bad address 'vault-secrets-provider'
I tried many times directly without Vault layer and Kaniko works ok, I mean without Vault secrets.
How I can accomplish this? I tried modifying the kaniko template but without success.
I will appreciate any help with this.
To fix your issue, first upgrade the docker template to its latest version (2.3.0 at the time this response was written).
Then depending on your case you have 2 options:
Docker needs to handle some of your secrets managed by Vault: then you shall also activate the Vault variant for Docker,
Docker doesn't needs to handle any secret managed by Vault: don't use the Vault variant for Docker, you'll have a warning message from Docker not being able to decode the secret (basically the same as the one you had, but not failing the build),
You shall simply use it in your .gitlab-ci.yml file:
include:
# Docker template
- project: 'to-be-continuous/docker'
ref: '2.3.0'
file: '/templates/gitlab-ci-docker.yml'
# Vault variant for Docker (depending on your above case)
- project: 'to-be-continuous/docker'
ref: '2.3.0'
file: '/templates/gitlab-ci-docker-vault.yml'
# Kubernetes template
- project: 'to-be-continuous/kubernetes'
ref: '2.0.4'
file: '/templates/gitlab-ci-k8s.yml'
- project: "to-be-continuous/kubernetes"
ref: "2.0.4"
file: "/templates/gitlab-ci-k8s-vault.yml"
K8S_DEFAULT_KUBE_CONFIG: "#url#http://vault-secrets-provider/api/secrets/noprod?field=kube_config"
VAULT_BASE_URL: "http://myvault.myserver.com/v1"
I'm creating a managed policy in a cloud formation template which locks down access to an s3 container and key path. I've followed the docs in aws for using !Join but I am getting a malformed template error.
Resource:
- !GetAtt ACHSFTPProxyBucket.Arn
-
- !Join
- - ''
- !GetAtt SomeBucketICreated.Arn
- /supersecret/upload/* #note I've also wrapped this in quotes and no dice
I've restricted access in the past using conditionals on the actions but was wondering if this could be done on a resource line and a !Join.
The output should look like this once deployed in the json editor of the console
Resource:[
"arn:aws:s3:::bucketbname",
"arn:aws:s3:::bucketbname/supersecret/upload/*"
]
I've manually modified the json in the console to test if the policy works and it does just trying to figure out how to translate this to the template.
What is the correct way to construct the arn combinded with a string
You need Fn::Sub. Something along the lines:
Resource:
- !GetAtt ACHSFTPProxyBucket.Arn
- !Sub '${SomeBucketICreated.Arn}/supersecret/upload/*'
I've just moved to cloud formation and I am starting with creating ECR repositories for docker,
I need all repositories to have the same properties except the repository name.
Since this is micro-services I will need at least 40 repo's so I want to create a stack that will create the repo's for me in a loop, and just change the name.
I started looking at nested stacks and this is what I got so far:
ecr-root.yaml:
---
AWSTemplateFormatVersion: '2010-09-09'
Description: ECR docekr repository
Parameters:
ECRRepositoryName:
Description: ECR repository name
Type: AWS::ECR::Repository::RepositoryName
Resources:
ECRStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://cloudformation.s3.amazonaws.com/ecr-stack.yaml
TimeoutInMinutes: '20'
Parameters:
ECRRepositoryName: !GetAtt 'ECRStack.Outputs.ECRRepositoryName'
And ecr-stack.yaml:
---
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
ECRRepositoryName:
Description: ECR repository name
Default: panpwr-mysql-base
Type: String
Resources:
MyRepository:
Type: AWS::ECR::Repository
Properties:
RepositoryName:
ref: ECRRepositoryName
RepositoryPolicyText:
Version: "2012-10-17"
Statement:
-
Sid: AllowPushPull
Effect: Allow
Principal:
AWS:
- "arn:aws:iam::123456789012:user/Bob"
- "arn:aws:iam::123456789012:user/Alice"
Action:
- "ecr:GetDownloadUrlForLayer"
- "ecr:BatchGetImage"
- "ecr:BatchCheckLayerAvailability"
- "ecr:PutImage"
- "ecr:InitiateLayerUpload"
- "ecr:UploadLayerPart"
- "ecr:CompleteLayerUpload"
RepositoryNameExport:
Description: RepositoryName for export
Value:
Ref: ECRRepositoryName
Export:
Name:
Fn::Sub: "ECRRepositoryName"
Everything is working fine,
But when I'm running the stack it asks me for the repository name I want to give it, and it creates one repository.
And then I can have as many stacks that I want with a different name but that is not my purpose.
How do I get it all in one stack that creates as many repositories that I want?
Sounds like you want to loop through a given list of parameters. Looping is not possible in a CloudFormation template. Few things you can try
You could programmatically generate a template. The troposphere Python library provides a nice abstraction to generate templates.
Write custom resource backed by AWS lambda. You can handle your custom logic in the AWS lambda function .
The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation. Use AWS CDK to write custom script for your usecase.
The below is the part of CloudForamtion file loaded by Serverless.
# resource.yml
.
.
.
{"Fn::Sub": "arn:aws:sqs:*:${AWS::AccountId}:sqs-spoon-*-${env:SERVICE}"}
# serverless.yml
.
.
resources:
- ${file:resource.yml}
${AWS::AccountId} is CloudFormation Pseudo Parameter and ${env:SERVICE} is Serverless variable.
When I run sls deploy, it returns the error.
Invalid variable reference syntax for variable AWS::AccountId. You can only reference env vars, options, & files. You can check our docs for more info.
It seems to say that Serverless recognize ${AWS::AccountId} as Serverless variable, not as CloudFormation Pseudo Parameter.
Right?
If so, how to have Serverless not to parse Pseudo Parameter so that it will be parsed by CloudFormation later?
I can solve it with the plugin.
With the plugin, It cloud be solved by replacing ${AWS::AccountId} with #{AWS::AccountId}.
{"Fn::Sub": "arn:aws:sqs:*:#{AWS::AccountId}:sqs-spoon-*-${env:SERVICE}"}
You can accomplish support for the native AWS syntax with a single config line in serverless.yml to define the variableSyntax. Details can be found here https://github.com/serverless/serverless/pull/3694.
provider:
name: aws
runtime: nodejs8.10
variableSyntax: "\${((env|self|opt|file|cf|s3)[:\(][ :a-zA-Z0-9._,\-\/\(\)]*?)}"
We currently use Codeship Pro to push Docker images to a private registry on AWS, as well as to deploy those images to an ECS cluster.
However, the codeship-steps.yml file includes a hard-coded region name for which AWS region I'm pushing to. For example:
- name: push_production
service: app
type: push
image_name: 123456789012.dkr.ecr.us-east-1.amazonaws.com/project/app-name
image_tag: "{{.Timestamp}}"
tag: master
registry: https://123456789012.dkr.ecr.us-east-1.amazonaws.com
dockercfg_service: aws_generator
I would like to be able to fairly easily switch this to deploy to a different AWS region. Thus the question:
Is it possible to use variables in a codeship-steps.yml file?
I know some of the properties can use a handful of built-in variables provided by Codeship (such as the {{.Timestamp}} value used for the image_tag property), but I don't know if, for example, values from an env_file can be used in the image_name, registry, and/or command properties of a step.
I'm imagining something like this...
codeship-steps.yml:
- name: push_production
service: app
type: push
image_name: "123456789012.dkr.ecr.{{.AWS_REGION}}.amazonaws.com/project/app-name"
image_tag: "{{.Timestamp}}"
tag: master
registry: "https://123456789012.dkr.ecr.{{.AWS_REGION}}.amazonaws.com"
dockercfg_service: aws_generator
... but that results in an "error parsing image name during push step: invalid reference format" on the push step.
I've tried simply not specifying the registry in the image_name...
image_name: project/app-name
... but I get a "Build Error: no basic auth credentials" on the push step. At this point, I'm running out of ideas.
Is it possible to use [environment] variables in a codeship-steps.yml file?
While the image_tag can take advantage of Go templates, the same is not the case for image_name, registry, or anything else. This is a separate set of templating variables that are accessible only to the image_tag generation.
As for environment variables in general (CI environment variables or those defined in the service configs), these values can be used in codeship-steps.yml on the command step when passed through a shell command. For example:
- service: app
command: echo The branch name is: $CI_BRANCH
Results in:
The branch name is: $CI_BRANCH
- service: app
command: /bin/sh -c 'echo The branch name is: $CI_BRANCH'
Results in:
The branch name is: master
As for your 'no basic auth credentials' error message, it's possible that there's an issue with how you are retrieving the basic auth credentials for access to your image registry. If you are on a MacOS device, I would recommend that you review our documentation on how to generate Docker credentials.