Concourse CI, git tag with constant value - concourse

I would like to tag my git commits as they are deployed to the various environments in my concourse pipeline with the name of the environment. For example, in my UAT deployment job, I would like to do something like:
- put: master-resource <-- a git resource
params:
repository: master <-- the resource local directory
tag: 'uat'
force: true <-- replace the tag, if it already exists
tag_only: true
This would seem like a common -or at least simple, thing to do however the value of the 'tag' parameter can only be the path to a file -there is no option to pass a constant/literal value.
I see two possible solutions but none of them seems 'simple' enough:
Create a file myself, but to do that (ideally?) I wish there were some kind of file resource that I could use to create the file.
The last alternative would be to create a custom task, and even there I was struggling to find a way to pass the name of the tag as a parameter.
Any suggestions on what would be the best way to accomplish my goal in the simplest way, or alternatively how to implement options 1 or 2?
Thanks!

The reason that tag takes in a file is so that you can dynamically set the tag of the commit based on information you imply during the course of the pipeline.
So, the best way I can see to do something like this would be workflow #2 that you described above.
So you would want something like this:
- task: generate-git-tag
params:
TAG: {{some-passed-in-tag}}
config:
platform: linux
image_resource:
type: docker-image
source:
repository: ruby
outputs:
- name: tag-file
params:
TAG:
run:
path: /bin/bash
args:
- -c
- |
echo "${TAG}" >> tag-file/tag.txt
- put: master-resource <-- a git resource
params:
repository: master <-- the resource local directory
tag: tag-file/tag.txt
force: true <-- replace the tag, if it already exists
tag_only: true

Related

reuse nested variables declared in another repository

With Auzre pipeline, I would like to how to reuse the variables declared in another repository.
Just a background, if all the three files organization_vars.yml, project_vars.yml, and cicd.yml are in the same repo, without using the resources like shown in project_vars.yml, it works.
Now I need to put the file organization_vars.yml in another repository in order to share it to every projects in the same organization, I tried with resources as explained here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#use-other-repositories , but it doesn't work.
In the pipeline, I got the error message Unexpected value 'resources' from the very beginning.
Hereunder a simplified example for the 3 files:
# file organization_vars.yml in the calling repository
variables:
orgName: xxx
# file project_vars.yml in the caller repository
resources:
repositories:
- repository: calling_repository
type: git
name: another_project/calling_repository
variables:
- template: organization_vars.yml#caller_repository
- name: callerVar
value: $(orgName)_xxx
# file cicd.yml in the caller repository, this file will be runned by the pipeline
trigger: xxx
variables:
- template: project_vars.yml
- name: otherVar
value: xxx
pool:
vmImage: xxx
steps:
xxx
I found it by myself, just need to move the resources part from the file project_vars.yml to the file cicd.yml

How can I run a pipeline with a yaml template that refers to a variables file in a different repo with Azure DevOps?

I have repo A that holds pipeline templates. Repo A has the following azure-template.yml:
# Repo A / azure-template.yml
stages:
- stage: ${{ variables.stageName }}
jobs:
- job:
steps:
- task:
A lot of the code in repo A that has the templates refers to variables in the following format:
${{ variables.variableName }}. The variable file is in a different folder in repo A. (e.g. variables/variables.yaml)
Now let's move to repo B. Repo B has my azure-pipeline.yml that needs to build from repo A:
# Repo B / azure-pipeline.yml
resources:
repositories:
- repository: templates
type: git
name: repoA
ref: refs/heads/develop
variables:
- template: variables/variables.yml#templates
stages:
- template: azure-template.yml#templates # Template reference
When I run azure-pipeline.yml, I get the following error:
An error occurred while loading the YAML build pipeline. The string must have at least one character. Parameter name: environmentName
That parameter is not one of mine. I don't have it declared or set anywhere. This tells me it is Azure specific but I have no idea where/why it's there or where it is even set.
How can I run a pipeline with a yaml template that refers to a
variables file in a different repo with Azure DevOps?
You're in correct direction, at least a working direction. It's supported to do that like what you've done above. About the error you got, I assume there could be something wrong with your yaml syntax. You can try following steps to locate the issue:
Copy the content of azure-template.yml and variables.yaml directly into azure-pipeline.yml file, and run the pipeline again to check if the issue persists.
In your Azure-pipeline.yml, specify the trigger and pool.
In azure-template.yml, try replacing the ${{ variables.stageName }} with hard-code value.
This is my first time to see this error message, but according to Parameter name: environmentName.You can also check if Release.EnvironmentName has valid value in one PS task. Hope it helps :)

Concourse call job from another job with parameters

I have a job with many tasks like this:
- name: main-job
serial: true
plan:
- aggregate:
- get: <git-resource>
passed: [previous-job]
trigger: true
- get: <git-resource-3>
- task: <task-1>
file: <git-resource>/<path>/<task-1-no-db>.yml
- task: <task-2>
tags: ['<specific-tag>']
file: <git-resource>/<path>/<task-1>.yml
params:
DATABASE_HOST: <file>
DATABASE: <my-db-1>
- task: <task-2>
tags: ['<specific-tag>']
file: <git-resource>/<path>/<task-1>.yml
params:
DATABASE_HOST: <file>
DATABASE: <my-db-1>
The problem for me is, I have to literally call the same job but instead of DATABASE params being my-db-1, I want it to be my-db-2.
The only way I am able to do this is by having new job and pass the params, literally copy the entire set of lines. My job is too fat, as in has too many tasks in them, so copying it though is the obvious solution, I am wondering if there's a way to re-use by having multiple pipelines and one main pipeline that essentially calls these pipelines with the param for DATABASE passed or have two small jobs that calls this main job with different params something like this:
- name: <call-main-job-with-db-1>
serial: true
plan:
- aggregate:
- get: <git-resource>
passed: [previous-job]
trigger: true
- task: <call-main-job-task>
params:
DATABASE_HOST: <file>
DATABASE: <my-db-1>
- name: <call-main-job-with-db-2>
serial: true
plan:
- aggregate:
- get: <git-resource>
passed: [previous-job]
trigger: true
- task: <call-main-job-task>
params:
DATABASE: <my-db-2>
I am not sure if this is even possible since I didn't find any example of this.
Remember you are using YAML, so you can use YAML features like "Anchors"
You will find some additional information about "Anchors" in this link. Look for "EXTRA YAML FEATURES"
YAML also has a handy feature called 'anchors', which let you easily duplicate
content across your document. Both of these keys will have the same value: anchored_content: &anchor_name This string will appear as the
value of two keys. other_anchor: *anchor_name
# Anchors can be used to duplicate/inherit properties
base: &base
name: Everyone has same name
foo: &foo
<<: *base
age: 10
bar: &bar
<<: *base
age: 20
Try this for your Concourse Pipeline:
common:
db_common: &db_common
serial: true
plan:
- aggregate:
- get: <git-resource>
passed: [previous-job]
trigger: true
- task: <call-main-job-task>
params:
jobs:
- name: <call-main-job-with-db-1>
<<: *db_common
DATABASE_HOST: <file>
DATABASE: <my-db-1>
- name: <call-main-job-with-db-2>
<<: *db_common
DATABASE: <my-db-2>
NOTE: Remember that you can have as many Anchors as you want, you can define two or more anchors for the same Job/Task/Resource, etc.
You need to just copy and paste the task as you do in the question description. Concourse expects an expressive yaml, there is no branching or logic allowed. If you don't want to copy and paste so much yaml, then you can do some yaml generation magic to simplify what you look at and work with, but concourse will want the full yaml with each job defined separately.
Concourse has this fan in fan out paradigm, where you want to keep the jobs simple and short. Use a scripting language e.g. like python or ruby to make your pipeline creation more flexible.
Personally i use one pipeline.yml.erb file where i render different job templates inside. I try to keep my job.yml.erb as generic as possible so i can reuse them for different pipelines.
To bring it to the next level you could specify a meta config.yml and use this config inside your templates to generate your pipeline depending on what you specified in the config.

Passing parameters between concourse jobs / tasks

What's the best way to pass parameters between concourse tasks and jobs? For example; if my first task generates a unique ID, what would be the best way to pass that ID to the next job or task?
If you are just passing between tasks within the same job, you can use artifacts (https://concourse-ci.org/running-tasks.html#outputs) and if you are passing between jobs, you can use resources (like putting it in git or s3). For example, if you are passing between tasks, you can have a task file
---
platform: linux
image_resource: # ...
outputs:
- name: unique-id
run:
path: project-src/ci/fill-in-output.sh
And the script fill-in-output.sh will put the file that contains the unique ID into path unique-id/. With that, you can have another task that takes the unique-id output as an input (https://concourse-ci.org/running-tasks.html#inputs) and use that unique id file.
Additionally to tasks resources will place files automagically for you in their working directory.
For example I have a pipeline job as follows
jobs:
- name: build
plan:
- get: git-some-repo
- put: push-some-image
params:
build: git-some-repo/the-image
- task: Use-the-image-details
config:
platform: linux
image_resource:
type: docker-image
source:
repository: alpine
inputs:
- name: push-some-image
run:
path: sh
args:
- -exc
- |
ls -lrt push-some-image
cat push-some-image/repository
cat push-some-image/digest
Well see the details of the image push from push-some-image
+ cat push-some-image/repository
xxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/path/image
+ cat push-some-image/digest
sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Passing data within a job's tasks could easily be done with input/output artifacts (files), As Clara Fu noted.
For the case between jobs, when simple e.g. 'string' data has to be passed , and using a git is an overkill, the 'keyval' resource[1] seems to be a good solution.
The readme describes that the data is stored and managed as a standard properties file.
https://github.com/SWCE/keyval-resource

How to use yaml template variable as a part of value in Concourse CI

When I try to use template variable, e.g. {{hostname}} as a part of value, it gets wrapped with double quotes.
How to add a variable w/o quotes?
Example:
---
resource_types:
- name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
resources:
- name: maven-snapshot
type: maven
source:
url: http://{{hostname}}:8081/repository/maven-snapshots/
- name: repo
type: git
source:
uri: "git#bitbucket.org:foo/bar.git"
branch: master{{hostname}}
And the result for the command fly -t ci set-pipeline --pipeline test --config test.yml --var="hostname=localhost" is as follows (look at "localhost"):
resources:
resource maven-snapshot has been added:
name: maven-snapshot
type: maven
source:
url: http://"localhost":8081/repository/maven-snapshots/
resource repo has been added:
name: repo
type: git
source:
branch: master"localhost"
uri: git#bitbucket.org:foo/bar.git
resource types:
resource type maven has been added:
name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
The reason I've included a 3rd-party maven resource is that git resource does not allow {{}} in the uri, leading to the error:
failed to unmarshal configStructure: yaml: line 17: did not find expected key
UPDATE
As of concourse v3.2.0 {{someValue}} syntax is deprecated in favor of ((someValue)). New syntax will understand you are trying to interpolate the string and place the value accordingly.
Replacing {{hostname}} with ((hostname)) will solve your issue:
resources:
- name: maven-snapshot
type: maven
source:
url: http://((hostname)):8081/repository/maven-snapshots/
Concourse does not support this.
The Concourse yaml templating is very primitive and you can not insert variables in the middle of strings.
You will need to set your url parameter as http://localhost:8081/repository/maven-snapshots/ and your branch parameter as localmaster or whatever it should be.
We know this is a problem and we are working on it, but for now you can not set variables in the way you want.
While waiting for this feature from concourse team, I've written this small executable to work around the issue in this GitHub repo:
https://github.com/sercant/inline-yaml
I prepare my config.yml like this:
ftp-username: username
ftp-password: password
ftp-uri: 192.168.1.2
ftp-dir: home/ftp/
ftp-uri-combined: ftp://{{ftp-username}}:{{ftp-password}}#{{ftp-uri}}/{{ftp-dir}}
ftp-uri-combined-html5: {{ftp-uri-combined}}html5
ftp-uri-combined-android: {{ftp-uri-combined}}android
And prepared a create-pipeline.sh:
#!/usr/bin/env sh
TEMP=$(mktemp)
java -jar inline-yaml.jar $3 ${TEMP};
fly -t lite set-pipeline -p $2 -c $1 --load-vars-from ${TEMP};
rm ${TEMP};
Whenever I need to create a pipeline, I ran:
./create-pipeline.sh build-plan.yml build-plan-name config.yml