How to use yaml template variable as a part of value in Concourse CI - concourse

When I try to use template variable, e.g. {{hostname}} as a part of value, it gets wrapped with double quotes.
How to add a variable w/o quotes?
Example:
---
resource_types:
- name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
resources:
- name: maven-snapshot
type: maven
source:
url: http://{{hostname}}:8081/repository/maven-snapshots/
- name: repo
type: git
source:
uri: "git#bitbucket.org:foo/bar.git"
branch: master{{hostname}}
And the result for the command fly -t ci set-pipeline --pipeline test --config test.yml --var="hostname=localhost" is as follows (look at "localhost"):
resources:
resource maven-snapshot has been added:
name: maven-snapshot
type: maven
source:
url: http://"localhost":8081/repository/maven-snapshots/
resource repo has been added:
name: repo
type: git
source:
branch: master"localhost"
uri: git#bitbucket.org:foo/bar.git
resource types:
resource type maven has been added:
name: maven
type: docker-image
source:
repository: patrickcrocker/maven-resource
tag: latest
The reason I've included a 3rd-party maven resource is that git resource does not allow {{}} in the uri, leading to the error:
failed to unmarshal configStructure: yaml: line 17: did not find expected key

UPDATE
As of concourse v3.2.0 {{someValue}} syntax is deprecated in favor of ((someValue)). New syntax will understand you are trying to interpolate the string and place the value accordingly.
Replacing {{hostname}} with ((hostname)) will solve your issue:
resources:
- name: maven-snapshot
type: maven
source:
url: http://((hostname)):8081/repository/maven-snapshots/
Concourse does not support this.
The Concourse yaml templating is very primitive and you can not insert variables in the middle of strings.
You will need to set your url parameter as http://localhost:8081/repository/maven-snapshots/ and your branch parameter as localmaster or whatever it should be.
We know this is a problem and we are working on it, but for now you can not set variables in the way you want.

While waiting for this feature from concourse team, I've written this small executable to work around the issue in this GitHub repo:
https://github.com/sercant/inline-yaml
I prepare my config.yml like this:
ftp-username: username
ftp-password: password
ftp-uri: 192.168.1.2
ftp-dir: home/ftp/
ftp-uri-combined: ftp://{{ftp-username}}:{{ftp-password}}#{{ftp-uri}}/{{ftp-dir}}
ftp-uri-combined-html5: {{ftp-uri-combined}}html5
ftp-uri-combined-android: {{ftp-uri-combined}}android
And prepared a create-pipeline.sh:
#!/usr/bin/env sh
TEMP=$(mktemp)
java -jar inline-yaml.jar $3 ${TEMP};
fly -t lite set-pipeline -p $2 -c $1 --load-vars-from ${TEMP};
rm ${TEMP};
Whenever I need to create a pipeline, I ran:
./create-pipeline.sh build-plan.yml build-plan-name config.yml

Related

reuse nested variables declared in another repository

With Auzre pipeline, I would like to how to reuse the variables declared in another repository.
Just a background, if all the three files organization_vars.yml, project_vars.yml, and cicd.yml are in the same repo, without using the resources like shown in project_vars.yml, it works.
Now I need to put the file organization_vars.yml in another repository in order to share it to every projects in the same organization, I tried with resources as explained here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#use-other-repositories , but it doesn't work.
In the pipeline, I got the error message Unexpected value 'resources' from the very beginning.
Hereunder a simplified example for the 3 files:
# file organization_vars.yml in the calling repository
variables:
orgName: xxx
# file project_vars.yml in the caller repository
resources:
repositories:
- repository: calling_repository
type: git
name: another_project/calling_repository
variables:
- template: organization_vars.yml#caller_repository
- name: callerVar
value: $(orgName)_xxx
# file cicd.yml in the caller repository, this file will be runned by the pipeline
trigger: xxx
variables:
- template: project_vars.yml
- name: otherVar
value: xxx
pool:
vmImage: xxx
steps:
xxx
I found it by myself, just need to move the resources part from the file project_vars.yml to the file cicd.yml

Is there a way I can define the azure pipeline template to be imported from a git tag instead of master?

Consider a yaml based pipeline
resources:
repositories:
- repository: templates
type: git
name: My-Proj/azure-build-templates
...
stages:
- template: test_pipeline/include-build-java-sonarqube.yml#templates
parameters:
agent_pool_name: $(agentPoolName)
maven_goal: 'mvn clean package'
...
This will refer to repository azure-build-templates in branch master. Is there a way I can declare to refer a tag instead of master? like name: My-Proj/azure-build-templates#release-20211215-better-tag
Context of its utility: Recently we pushed something in master which consequently broke all the pipelines referring. As a fix we had to rollback with git operations on master branch. I wonder if there would be a way to refer from tag like jenkins and rollback would be just point from one git tag to another. (Downside on every release you have update all the pipelines, so I am up for any suggestions for templates release management). Thanks.
ref: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=schema#define-a-repositories-resource
You can link to any ref.
Tags are represented in git as refs/tags/myTag
resources:
repositories:
- repository: string # identifier (A-Z, a-z, 0-9, and underscore)
type: enum # see the following "Type" topic
name: string # repository name (format depends on `type`)
ref: string # ref name to use; defaults to 'refs/heads/main'
You can list your tags with: git show-ref --tags

Concourse: Read git resource meta-data

I have a concourse pipeline:
groups: []
resources:
- name: source-code
type: git
resource_types: []
jobs:
- name: build
public: true
plan:
- get: source-code
- task: build
privileged: true
config:
platform: linux
image_resource:
type: docker-image
source:
repository: java
tag: openjdk-8-alpine
run:
path: sh
args:
- -exc
- |
set -e -u -x
MVN_VERSION=$(cat pom.xml | grep "^ <version>.*</version>$" | awk -F'[><]' '{print $3}')
**CUSTOM_VERSION = [POM_FILE_VERSION]-build.[NO_OF_COMMITS_IN_BRANCH]**
inputs:
- name: source-code
Using Unix commands, I want to create the custom version:
CUSTOM_VERSION = [POM_FILE_VERSION]-build.[NO_OF_COMMITS_IN_BRANCH]
For [POM_FILE_VERSION], I have found the code:
MVN_VERSION=$(cat pom.xml | grep "^ <version>.*</version>$" | awk -F'[><]' '{print $3}')
But for [NO_OF_COMMITS_IN_BRANCH], i don't know how get that data from git resource. Can anyone help please ?
The Concourse git resource doesn't provide the number of commits in the branch.
I suggest to use something more meaningful like the commit hash, which the Concourse git resource makes available as the .git/short_ref file.
If you really need to use the number of commits (but why?), then you need to run the git command directly in the Concourse task, for example git rev-list --count HEAD.
In the pipeline above, the task is using the openjdk-8-alpine Docker image from the java repository. If that image contains git, then you are all set. If not, I suggest first to see if the java repository offers a variation of openjdk-8-alpine with git installed. If not, you have to create your own Docker image, based off openjdk-8-alpine.
Security note: you have privileged: true specified. Probably you don't need it.

trigger concourse job via CLI: "resource not found"

I am attempting to trigger a concourse job from the command line. My pipeline has one resource (a git repo) and one job, which uses that repo. I am seeing:
$ fly -t tutorial trigger-job -j my-pipeline/my-job -w
error: resource not found
However, when I go the web UI and manually trigger the job by pressing the "+" button in the top right, it works fine.
Here is the full pipeline:
resources:
- name: cruise-source
type: git
source:
uri: git#github.com:my-org/cruise.git
branch: develop
jobs:
- name: build-image
public: true
plan:
- get: cruise-source
- task: list-files
config:
platform: linux
image_resource:
type: docker-image
source: {repository: alpine}
inputs:
- name: cruise-source
run:
path: ls
args: [cruise-source]
How can I trigger this job from the CLI?
The "resource not found" you get has nothing to do with the git resource :-) it actually means that the pipeline or job name is wrong. Looking at your pipeline configuration, you should issue
fly -t tutorial trigger-job -j my-pipeline/build-image -w
or if your configuration is different from what you have posted, maybe you have a typo in the name of the pipeline or job.

Concourse CI, git tag with constant value

I would like to tag my git commits as they are deployed to the various environments in my concourse pipeline with the name of the environment. For example, in my UAT deployment job, I would like to do something like:
- put: master-resource <-- a git resource
params:
repository: master <-- the resource local directory
tag: 'uat'
force: true <-- replace the tag, if it already exists
tag_only: true
This would seem like a common -or at least simple, thing to do however the value of the 'tag' parameter can only be the path to a file -there is no option to pass a constant/literal value.
I see two possible solutions but none of them seems 'simple' enough:
Create a file myself, but to do that (ideally?) I wish there were some kind of file resource that I could use to create the file.
The last alternative would be to create a custom task, and even there I was struggling to find a way to pass the name of the tag as a parameter.
Any suggestions on what would be the best way to accomplish my goal in the simplest way, or alternatively how to implement options 1 or 2?
Thanks!
The reason that tag takes in a file is so that you can dynamically set the tag of the commit based on information you imply during the course of the pipeline.
So, the best way I can see to do something like this would be workflow #2 that you described above.
So you would want something like this:
- task: generate-git-tag
params:
TAG: {{some-passed-in-tag}}
config:
platform: linux
image_resource:
type: docker-image
source:
repository: ruby
outputs:
- name: tag-file
params:
TAG:
run:
path: /bin/bash
args:
- -c
- |
echo "${TAG}" >> tag-file/tag.txt
- put: master-resource <-- a git resource
params:
repository: master <-- the resource local directory
tag: tag-file/tag.txt
force: true <-- replace the tag, if it already exists
tag_only: true