reuse nested variables declared in another repository - azure-devops

With Auzre pipeline, I would like to how to reuse the variables declared in another repository.
Just a background, if all the three files organization_vars.yml, project_vars.yml, and cicd.yml are in the same repo, without using the resources like shown in project_vars.yml, it works.
Now I need to put the file organization_vars.yml in another repository in order to share it to every projects in the same organization, I tried with resources as explained here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#use-other-repositories , but it doesn't work.
In the pipeline, I got the error message Unexpected value 'resources' from the very beginning.
Hereunder a simplified example for the 3 files:
# file organization_vars.yml in the calling repository
variables:
orgName: xxx
# file project_vars.yml in the caller repository
resources:
repositories:
- repository: calling_repository
type: git
name: another_project/calling_repository
variables:
- template: organization_vars.yml#caller_repository
- name: callerVar
value: $(orgName)_xxx
# file cicd.yml in the caller repository, this file will be runned by the pipeline
trigger: xxx
variables:
- template: project_vars.yml
- name: otherVar
value: xxx
pool:
vmImage: xxx
steps:
xxx

I found it by myself, just need to move the resources part from the file project_vars.yml to the file cicd.yml

Related

Azure Devops pipelines - clone multiple repos by same tag or branch

I have three Azure Repos, all in the same ADO Project - repo-0001, repo-0002, repo-0003
repo-0001 has my yaml pipeline which has a manual trigger, and the pipeline needs to be run on demand by either branch or tag (eg release/branch001, release/tag001)
The Pipeline checkouts out repo-0001, repo-0002 & repo-0003 at whatever ref (branch or tag) is used to start the pipeline in repo-0001 (these branches & tags are created by a different process and are always present in all repos)
I've been using $(Build.SourceBranch) to extract:
branch - refs/heads/release/branch001
tag - refs/tags/release/tag001
This yaml works for a pipeline started using a branch, but not for a tag, and it results in Could not get the latest source version for repository repo-0001 hosted on Azure Repos using refs/heads/refs/tags/release/tag001 so it appears to append the whole ref path of the tag to refs/heads
variables:
REPOSITORY_SOURCE_BRANCH: $(Build.SourceBranch)
resources:
repositories:
- repository: repo-0001
type: git
name: aks/repo-0001
ref: $(REPOSITORY_SOURCE_BRANCH)
- repository: repo-0002
type: git
name: aks/repo-0002
ref: $(REPOSITORY_SOURCE_BRANCH)
- repository: repo-0003
type: git
name: aks/repo-0003
ref: $(REPOSITORY_SOURCE_BRANCH)
I can make it work with a tag by doing something like this, or passing in as a parameter
variables:
tag: "release/tag001"
resources:
repositories:
- repository: repo-0001
type: git
name: aks/repo-0001
ref: 'refs/tags/$(tag)'
etc
But that's not what I need...
Can a tag ref be passed in from pipeline variables to make this work, and is there a way to make it flexible so that I can use the same pipeline code for tags and branches?
Thanks!
The ref field in repositories Resources doesn't support defining Pipeline variable. It only supports hardcode the ref value.
Can a tag ref be passed in from pipeline variables to make this work, and is there a way to make it flexible so that I can use the same pipeline code for tags and branches?
To meet your requirement, you can change to use the following format to checkout the repo.
- checkout: git://MyProject/MyRepo#refs/tags/$(tag)
- checkout: git://MyProject/MyRepo#$(REPOSITORY_SOURCE_BRANCH)
Here is an example:
variables:
REPOSITORY_SOURCE_BRANCH: test
steps:
- checkout: git://aks/repo-0001#$(REPOSITORY_SOURCE_BRANCH)
For more detailed info, you can refer to this doc: Checking out a specific ref

DevOps Build Pipeline - Templates - Run from the same branch

I am testing a POC build pipeline and I am trying to use templates as a sort of reference library, so I can reuse the same code in multiple pipelines, while also allowing for simpler editing/updating, if changes are required.
I want to be able to run the pipeline from a DevOps branch and refer to the same branch (or tag) when validating/running all the template pipelines in this build pipeline.
The ultimate goal is to pass multiple variables to the template (via a loop/for each) to create multiple build artifacts referencing public repositories (in this test a public terraform github repo). We could then maybe only need to maintain the variable data and not the source/reference code.
I thought DevOps did this 'OOTB', but realised that this was probably not the case. Something like this:
Repo: InfraAsData (all pipelines)
Branch: feature/sparse-clone-repo
Updated azure-pipeline.yaml and added template templates/sparse-clone.yaml (FYI - this template is untested as well)
azure-pipeline.yaml:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'IAC/InfraAsData'
refs: 'refs/heads/$(branch_name)'
parameters:
- name: branch
displayName: branch
type: string
default: $(Build.SourceBranch)
variables:
- name: branch_name
value: ${{ parameters.branch }}
- template: ./templates/variables/resource-groups.yaml#InfraAsData #variables template
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template-Sparse-Clone-To-Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: $(Build.SourceBranch)/templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: '${{ variables[template.repoUrl] }}'
repoPath: '${{ variables[template.repoPath] }}'
artifactPath: '${{ variables[template.artifactPath] }}'
sparse-clone.yaml
parameters:
repoUrl: ''
repoPath: ''
artifactPath: ''
#testing
steps:
- script: |
md $(Build.BinariesDirectory)/$(parameters.artifactPath)
git clone --filter=blob:none --sparse $(parameters.repoUrl) $(parameters.artifactPath)
cd $(Build.BinariesDirectory)/$(parameters.artifactPath)
git sparse-checkout init --cone
git sparse-checkout set $(parameters.repoPath)
git checkout main
dir $(Build.BinariesDirectory)/$(parameters.artifactPath)
displayName: 'Clone Github Repo Subdirectory - filter blob none'
When I run this from the feature/sparse-clone-repo branch in DevOps I get an error that the new template does not exist in the main branch (it doesn't of course).
/build/test/azure-pipeline.yaml: File /templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/main version
I have also tried using no resources.repository, using $(Build.SourceBranch) $(Build.SourceBranchName) as refs value and as the template path reference (like in the 2nd template example).
EDIT:
If i remove the resource.repositories reference, then the correct branch is identified, but the azure-pipeline.yaml relative path is appended to the template path:
/build/test/azure-pipeline.yaml: File /build/test/$(Build.SourceBranch)/templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/feature/pipelines
So correct branch, but not the correct relative path.
EDIT 2:
I can remove the resources section and use a relative path
../../templates/variables/resource-groups.yaml
but this is not dynamic, to a 'root' reference, so would not work if the folder structure were to change (EG I moved the azure-pipeline.yaml up a level)
I'm sure I am missing something obvious or misunderstanding the docs (or pipeline templates). Any pointers gratefully received!
When I run this from the feature/sparse-clone-repo branch in DevOps I
get an error that the new template does not exist in the main branch
(it doesn't of course).
From your description, seems you thought the 'refs' of the resources determine the branch of the alias of the template part.
But it is not, the 'resources.repositories.repository' doesn't have a section named 'refs', only 'ref' is allowed.
In your situation, you used a section named 'refs' which doesn't exist. Pipeline will use the default branch of the repository to looking for the template YAML. So change the default branch of the repository or change the 'refs' to 'ref' will solve the first issue.
And I notice you were using variable in resources section, you have two mistakes.
One is '$()' can't use in compile time part. Another is even compile time variables are also not allowed in 'ref' section.
ref
string
ref name to checkout; defaults to 'refs/heads/main'. The branch
checked out by default whenever the resource trigger fires. Does not
accept variables.
You can use your pipeline like this:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'BowmanCP/template_branch'
ref: 'refs/heads/${{parameters.branch}}'
parameters:
- name: branch
displayName: branch
type: string
default: main2
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template_Sparse_Clone_To_Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: ./templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: 'xxx'
repoPath: 'xxx'
artifactPath: 'xxx'
Basically, variables can't be used in your situation and there doesn't have a parameter reuse feature. That's everything.

Gitlab pipeline: use variable in trigger path

I'm trying to develop a common gitlab pipeline that uses a matrix to trigger another pipeline but I want to add a variable in the path. What I wrote till now is:
variables:
PROJECT_NAME: name
PROJECT_HELM: helmpipe
DEV_CUSTOMERS: cust1 cust2
deploy:dev:
stage: deploy
variables:
UPSTREAM_CI_COMMIT_SHORT_SHA: $CI_COMMIT_SHORT_SHA
UPSTREAM_CI_COMMIT_BRANCH: $CI_COMMIT_BRANCH
UPSTREAM_CI_COMMIT_TAG: $CI_COMMIT_TAG
IMAGE_NAME: ${PROJECT_NAME}
CUSTOMER: $CUSTOMER
trigger: my/project/$PROJECT_HELM
parallel:
matrix:
- CUSTOMER: $DEV_CUSTOMERS
only:
- DEV
But gitlab returns error saying that the triggered project can not be found. It seems that the variable $PROJECT_HELM is not converted to its real value in trigger path. How can I do it?
Thanks
Below is the working example where the variable ENV is used to trigger the sit pipeline. My pipeline file is in pipeline folder at root level. And the file name is sit-pipeline.yaml
stages:
- build
variables:
ENV: sit
sit-pipeline:
stage: build
trigger:
include:
- local: pipeline/${ENV}-pipeline.yaml
There could be 2 issues
Either you need to use ${var} syntax instead of $var syntax
trigger: my/project/${PROJECT_HELM}
Or PROJECT_HELM variable should resolve to a yaml file defining the pipeline. Your variable is assigned to helmpipe so I think you are missing .yaml extension

How can I run a pipeline with a yaml template that refers to a variables file in a different repo with Azure DevOps?

I have repo A that holds pipeline templates. Repo A has the following azure-template.yml:
# Repo A / azure-template.yml
stages:
- stage: ${{ variables.stageName }}
jobs:
- job:
steps:
- task:
A lot of the code in repo A that has the templates refers to variables in the following format:
${{ variables.variableName }}. The variable file is in a different folder in repo A. (e.g. variables/variables.yaml)
Now let's move to repo B. Repo B has my azure-pipeline.yml that needs to build from repo A:
# Repo B / azure-pipeline.yml
resources:
repositories:
- repository: templates
type: git
name: repoA
ref: refs/heads/develop
variables:
- template: variables/variables.yml#templates
stages:
- template: azure-template.yml#templates # Template reference
When I run azure-pipeline.yml, I get the following error:
An error occurred while loading the YAML build pipeline. The string must have at least one character. Parameter name: environmentName
That parameter is not one of mine. I don't have it declared or set anywhere. This tells me it is Azure specific but I have no idea where/why it's there or where it is even set.
How can I run a pipeline with a yaml template that refers to a
variables file in a different repo with Azure DevOps?
You're in correct direction, at least a working direction. It's supported to do that like what you've done above. About the error you got, I assume there could be something wrong with your yaml syntax. You can try following steps to locate the issue:
Copy the content of azure-template.yml and variables.yaml directly into azure-pipeline.yml file, and run the pipeline again to check if the issue persists.
In your Azure-pipeline.yml, specify the trigger and pool.
In azure-template.yml, try replacing the ${{ variables.stageName }} with hard-code value.
This is my first time to see this error message, but according to Parameter name: environmentName.You can also check if Release.EnvironmentName has valid value in one PS task. Hope it helps :)

Concourse CI, git tag with constant value

I would like to tag my git commits as they are deployed to the various environments in my concourse pipeline with the name of the environment. For example, in my UAT deployment job, I would like to do something like:
- put: master-resource <-- a git resource
params:
repository: master <-- the resource local directory
tag: 'uat'
force: true <-- replace the tag, if it already exists
tag_only: true
This would seem like a common -or at least simple, thing to do however the value of the 'tag' parameter can only be the path to a file -there is no option to pass a constant/literal value.
I see two possible solutions but none of them seems 'simple' enough:
Create a file myself, but to do that (ideally?) I wish there were some kind of file resource that I could use to create the file.
The last alternative would be to create a custom task, and even there I was struggling to find a way to pass the name of the tag as a parameter.
Any suggestions on what would be the best way to accomplish my goal in the simplest way, or alternatively how to implement options 1 or 2?
Thanks!
The reason that tag takes in a file is so that you can dynamically set the tag of the commit based on information you imply during the course of the pipeline.
So, the best way I can see to do something like this would be workflow #2 that you described above.
So you would want something like this:
- task: generate-git-tag
params:
TAG: {{some-passed-in-tag}}
config:
platform: linux
image_resource:
type: docker-image
source:
repository: ruby
outputs:
- name: tag-file
params:
TAG:
run:
path: /bin/bash
args:
- -c
- |
echo "${TAG}" >> tag-file/tag.txt
- put: master-resource <-- a git resource
params:
repository: master <-- the resource local directory
tag: tag-file/tag.txt
force: true <-- replace the tag, if it already exists
tag_only: true