I can't figure out how do I convert the below section of my yaml cloudformation template to a json template. Can anyone please guide me here.
Tags:
- Key: Name
Value: !Join
- ''
- - !FindInMap
- BusinessUnitCode
- !Ref BusinessUnit
- Code
- A
- !FindInMap
- AWSRegionCode
- !Ref 'AWS::Region'
- Code
- '-'
- !FindInMap
- AccountsCode
- !Ref Account
- Code
- '-'
- !FindInMap
- ProdLevelCode
- !Ref ProductionLevel
- Code
- '-'
- !FindInMap
- ServerFunctionCode
- !Ref ServerFunction
- Code
- '-'
- !FindInMap
- SGFunctionCode
- !Ref SecurityGroupFunction
- Code
- '-'
- !Ref IncrementCode
Though I know there is no need for this conversion as both json and yaml works the same in cloudformation, have to do this as per a mandate.
In case you wanted a command-line solution for converting from JSON to YAML and vice versa, consider the aws-cfn-template-flip project.
Install the library:
▶ pip install cfn_flip
Usage:
▶ cfn-flip
Usage: cfn-flip [OPTIONS] [INPUT] [OUTPUT]
AWS CloudFormation Template Flip is a tool that converts AWS
CloudFormation templates between JSON and YAML formats, making use of the
YAML format's short function syntax where possible.
Options:
-i, --input [json|yaml] Specify the input format. Overrides -j and -y
flags.
-o, --output [json|yaml] Specify the output format. Overrides -j, -y, and
-n flags.
-j, --json Convert to JSON. Assume the input is YAML.
-y, --yaml Convert to YAML. Assume the input is JSON.
-c, --clean Performs some opinionated cleanup on your
template.
-l, --long Use long-form syntax for functions when converting
to YAML.
-n, --no-flip Perform other operations but do not flip the
output format.
--version Show the version and exit.
--help Show this message and exit.
And if you have your CloudFormation template in a YAML then you can flip it to JSON using:
▶ cfn-flip -j cloudformation.yml | jq . > cloudformation.json
Use of the JQ there is optional, just to clean the formatting.
AWS Console can help you with that.
Open your CloudFormation template in template designer.
CloudFormation > Select your stack > Template > View in designer
You can then easily convert your template by choosing your template language as Json.
I use this and it works like a charm: https://cfnflip.com/
You can convert JSON to YAML and YAML to JSON.
Copy/Paste either, will detect and flip. Opensource and Source Code available if you want to set this up as your own internal too. (no, this is NOT mine nor sponsored).
This is not just a normal Flipping tool, this works specifically with CloudFormation in mind.
Related
I have a simple CloudFomration template to create a VPC:
vpc.yaml
---
Parameters:
CidrBlock:
Type: String
Description: The primary IPv4 CIDR block for the VPC
Resources:
VPC:
Type: "AWS::EC2::VPC"
Properties:
CidrBlock: !Ref "CidrBlock"
Tags:
- Key: "Name"
Value: "This is the main VPC"
In my Azure Devops pipleline I want to use AWS Stack create to create a VPC by using this CloudFormation template.
Azure-pipeline.yaml
---
jobs:
- job: Job1
steps:
- task: CloudFormationCreateOrUpdateStack#1
inputs:
awsCredentials: 'My-cred'
regionName: 'ap-southeast-2'
stackName: 'Stack1'
templateSource: 'file'
templateFile: 'vpc.yaml'
templateParametersSource: inline
templateParameters: |
- ParameterKey: CidrBlock
ParameterValue: '10.10.10.0/24'
Obviousely, it'll work fine when I manually provide the value for the CidrBlock parameter when calling vpc.yaml template (like above ^). But what I want to do is using another file to store the parameters and values and pass that when running the vpc template.
In other words, I have a params.yaml file in which I stored the required parameters to run vpc.yaml. How can I use that when running the vpc.yaml template as the parameter files?
params.yaml
---
parameters:
- name: CidrBlock
value: '10.10.10.0/24'
Like how I refer to the template file in azure-pipeline file above, is there a similar way with which I can refer to the params file instead of using templateParametersSource: inline ?
It looks like there is an issue with YAML parameters file with aws-cli:
https://github.com/aws/aws-cli/issues/2275
You can use JSON file:
[
{
"CidrBlock": "10.10.10.0/24",
}
]
Sounds like unlike Azure Bicep or ARM where you can use keys of your own when creating the params files, AWS expect a specific structure where you must always use ParameterKey and ParameterValue as the key.
Here is how the params file should look like in my case:
---
- ParameterKey: CidrBlock
ParameterValue: '10.10.10.0/24'
Given a yaml file (helmfile) like the following
releases:
- chart: ../charts/foo
name: foo
namespace: '{{ .Values.stack }}'
values:
- ../config/templates/foo-values.yaml.gotmpl
set:
- name: image.tag
value: 22
- name: replicas
value: 1
- chart: ../charts/bar
name: bar
namespace: '{{ .Values.stack }}'
values:
- ../config/templates/bar-values.yaml.gotmpl
set:
- name: image.bar_proxy.tag
value: 46
- name: image.bar.tag
value: 29
- name: replicas
value: 1
I'm trying to figure out a clean way to update a specific image tag. For example, I'd like to update image.bar_proxy.tag from 46 to 51.
I have the following, which does the job, but it requires that you know the exact index of the array item:
yq -y '.releases[] |= if .name=="bar" then .set[0].value |= 51 else . end' helmfile-example.yaml
So if the array order were to change at some point this would break.
A preferred solution would be: "update image.bar_proxy.tag value from 46 to 51 where set[].name==image.bar_proxy.tag". Any ideas on how to achieve a more specific conditional selection like this?
FYI our yq version:
$ yq --version
yq 2.10.0
You can use the following filter to make it work. It works by dynamically selecting the index of the object where your tag exists. On the selected object .value=51 will update the value as you wanted. You can also use the -i flag to do in-place modification of the original file.
yq -y '.releases[].set |= map(select(.name == "image.bar_proxy.tag").value=51)' yaml
See the underlying jq filter acting on the JSON object at jq-playground
Given the context of using Helmfile, there are a couple of ways you can approach this without necessarily editing the helmfile.yaml. Helmfile allows using the Go text/template language in many places, similarly to the underlying Helm tool, and has some other features that can help.
One of the easiest things you can do is take advantage of values: being a list, and of unknown values generally being ignored. You (or your CI/CD system) can write a separate YAML file that contains just the tags (JSON may be easier to write and is valid YAML)
# tags.yaml
image:
tag: 22
bar: {tag: 29}
bar_proxy: {tag: 46}
and then include this file as an additional file in the helmfile.yaml. (This would be equivalent to using helm install -f with multiple local values files, rather than helm install --set individual values.)
releases:
- name: foo
values:
- ../config/templates/foo-values.yaml.gotmpl
- tags.yaml
# no `set:`
- name: bar
values:
- ../config/templates/bar-values.yaml.gotmpl
- tags.yaml
- replicas: 1
# no `set:`
Helmfile's templating extensions also include env and requiredEnv to read ordinary environment variables from the host system. Helm proper does not have these to try to minimize the number of implicit inputs to a chart, but for Helmfile it's a possible way to provide values at deploy time.
releases:
- name: bar
set:
- name: image.bar_proxy.tag
value: {{ env "BAR_PROXY_TAG" | default "46" }}
- name: image.bar.tag
value: {{ requiredEnv "BAR_TAG" }}
How do you override values in a Helm list with --set param in Azure DevOps?
Simple use case in values.yaml:
environment:
- name: foo
value: override_me
- name: bar
value: override_me
- name: baz
value: override_me
In the deployment.yaml file I use it like so:
env:
{{ toYaml .Values.environment | indent 10}}
One thing that kind of works, but not really, is:
environment[0].name=foo,environment[0].value=hello,{...}
The problem with this override is that it will override the entire list, even if I only want to replace value [0], not [1] and [2].
Also I get parsing errors when I pass url:s or int's (not on localhost, only AZ DevOps) - to overcome that paring error, you can escape it with \" - but then the chart is messed up - even though it passes the validation.
So, is it possible to override the env list in my case in Azure DevOps helm deployment? Or do I need to restructure the list to individual key=value pairs?
I've got weird experience when doing this, in 2 similar cases in one case it replaces them, in one overrides the whole array. so in the second case what I had to do is this:
environment:
- name: v1
value: keep_me
- name: v2
value: keep_me
- name: v3
value: keep_me
- name: foo
value: override_me
- name: bar
value: override_me
and I was doing this in the Azure Devops:
--set environment[3].name=foo,environment[03.value=xxx
for the other one I didnt have to do that, it would gladly overwrite only the values I've input. no idea why it did that.
Get yourself some variables defined in the task:
Use a standard set:
I was using a bash task in a release pipeline pointed at a deploy.sh file which existed as a published artifact. You need to chmod +x the file for this to work properly.
What's the best way to pass parameters between concourse tasks and jobs? For example; if my first task generates a unique ID, what would be the best way to pass that ID to the next job or task?
If you are just passing between tasks within the same job, you can use artifacts (https://concourse-ci.org/running-tasks.html#outputs) and if you are passing between jobs, you can use resources (like putting it in git or s3). For example, if you are passing between tasks, you can have a task file
---
platform: linux
image_resource: # ...
outputs:
- name: unique-id
run:
path: project-src/ci/fill-in-output.sh
And the script fill-in-output.sh will put the file that contains the unique ID into path unique-id/. With that, you can have another task that takes the unique-id output as an input (https://concourse-ci.org/running-tasks.html#inputs) and use that unique id file.
Additionally to tasks resources will place files automagically for you in their working directory.
For example I have a pipeline job as follows
jobs:
- name: build
plan:
- get: git-some-repo
- put: push-some-image
params:
build: git-some-repo/the-image
- task: Use-the-image-details
config:
platform: linux
image_resource:
type: docker-image
source:
repository: alpine
inputs:
- name: push-some-image
run:
path: sh
args:
- -exc
- |
ls -lrt push-some-image
cat push-some-image/repository
cat push-some-image/digest
Well see the details of the image push from push-some-image
+ cat push-some-image/repository
xxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/path/image
+ cat push-some-image/digest
sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Passing data within a job's tasks could easily be done with input/output artifacts (files), As Clara Fu noted.
For the case between jobs, when simple e.g. 'string' data has to be passed , and using a git is an overkill, the 'keyval' resource[1] seems to be a good solution.
The readme describes that the data is stored and managed as a standard properties file.
https://github.com/SWCE/keyval-resource
When I try to use template variable, e.g. {{hostname}} as a part of value, it gets wrapped with double quotes.
How to add a variable w/o quotes?
Example:
---
resource_types:
- name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
resources:
- name: maven-snapshot
type: maven
source:
url: http://{{hostname}}:8081/repository/maven-snapshots/
- name: repo
type: git
source:
uri: "git#bitbucket.org:foo/bar.git"
branch: master{{hostname}}
And the result for the command fly -t ci set-pipeline --pipeline test --config test.yml --var="hostname=localhost" is as follows (look at "localhost"):
resources:
resource maven-snapshot has been added:
name: maven-snapshot
type: maven
source:
url: http://"localhost":8081/repository/maven-snapshots/
resource repo has been added:
name: repo
type: git
source:
branch: master"localhost"
uri: git#bitbucket.org:foo/bar.git
resource types:
resource type maven has been added:
name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
The reason I've included a 3rd-party maven resource is that git resource does not allow {{}} in the uri, leading to the error:
failed to unmarshal configStructure: yaml: line 17: did not find expected key
UPDATE
As of concourse v3.2.0 {{someValue}} syntax is deprecated in favor of ((someValue)). New syntax will understand you are trying to interpolate the string and place the value accordingly.
Replacing {{hostname}} with ((hostname)) will solve your issue:
resources:
- name: maven-snapshot
type: maven
source:
url: http://((hostname)):8081/repository/maven-snapshots/
Concourse does not support this.
The Concourse yaml templating is very primitive and you can not insert variables in the middle of strings.
You will need to set your url parameter as http://localhost:8081/repository/maven-snapshots/ and your branch parameter as localmaster or whatever it should be.
We know this is a problem and we are working on it, but for now you can not set variables in the way you want.
While waiting for this feature from concourse team, I've written this small executable to work around the issue in this GitHub repo:
https://github.com/sercant/inline-yaml
I prepare my config.yml like this:
ftp-username: username
ftp-password: password
ftp-uri: 192.168.1.2
ftp-dir: home/ftp/
ftp-uri-combined: ftp://{{ftp-username}}:{{ftp-password}}#{{ftp-uri}}/{{ftp-dir}}
ftp-uri-combined-html5: {{ftp-uri-combined}}html5
ftp-uri-combined-android: {{ftp-uri-combined}}android
And prepared a create-pipeline.sh:
#!/usr/bin/env sh
TEMP=$(mktemp)
java -jar inline-yaml.jar $3 ${TEMP};
fly -t lite set-pipeline -p $2 -c $1 --load-vars-from ${TEMP};
rm ${TEMP};
Whenever I need to create a pipeline, I ran:
./create-pipeline.sh build-plan.yml build-plan-name config.yml