How can I run a pipeline with a yaml template that refers to a variables file in a different repo with Azure DevOps? - azure-devops

I have repo A that holds pipeline templates. Repo A has the following azure-template.yml:
# Repo A / azure-template.yml
stages:
- stage: ${{ variables.stageName }}
jobs:
- job:
steps:
- task:
A lot of the code in repo A that has the templates refers to variables in the following format:
${{ variables.variableName }}. The variable file is in a different folder in repo A. (e.g. variables/variables.yaml)
Now let's move to repo B. Repo B has my azure-pipeline.yml that needs to build from repo A:
# Repo B / azure-pipeline.yml
resources:
repositories:
- repository: templates
type: git
name: repoA
ref: refs/heads/develop
variables:
- template: variables/variables.yml#templates
stages:
- template: azure-template.yml#templates # Template reference
When I run azure-pipeline.yml, I get the following error:
An error occurred while loading the YAML build pipeline. The string must have at least one character. Parameter name: environmentName
That parameter is not one of mine. I don't have it declared or set anywhere. This tells me it is Azure specific but I have no idea where/why it's there or where it is even set.

How can I run a pipeline with a yaml template that refers to a
variables file in a different repo with Azure DevOps?
You're in correct direction, at least a working direction. It's supported to do that like what you've done above. About the error you got, I assume there could be something wrong with your yaml syntax. You can try following steps to locate the issue:
Copy the content of azure-template.yml and variables.yaml directly into azure-pipeline.yml file, and run the pipeline again to check if the issue persists.
In your Azure-pipeline.yml, specify the trigger and pool.
In azure-template.yml, try replacing the ${{ variables.stageName }} with hard-code value.
This is my first time to see this error message, but according to Parameter name: environmentName.You can also check if Release.EnvironmentName has valid value in one PS task. Hope it helps :)

Related

Azure Devops Pipeline - Variable file based on branch

Is it possible to have different variable template files that would be used based on the branch being built? I was trying to do something like this:
jobs:
- job: BuildandPublish
variables:
- template: /env/$(Build.SourceBranchName).vars.yml
But that doesn't work, I'm guessing do to the order that those variables are replaced.
Is it possible to have different variable template files that would be used based on the branch being built?
From your YAML sample, you are using the format: $(Build.SourceBranchName). The variable value will be expanded at runtime.
But the template will read the variable at Compile time.
To solve this issue, you need to change the format : ${{ variables['Build.SourceBranchName'] }}
Here is an example:
jobs:
- job: BuildandPublish
variables:
- template: /env/${{ variables['Build.SourceBranchName'] }}.vars.yml

DevOps Build Pipeline - Templates - Run from the same branch

I am testing a POC build pipeline and I am trying to use templates as a sort of reference library, so I can reuse the same code in multiple pipelines, while also allowing for simpler editing/updating, if changes are required.
I want to be able to run the pipeline from a DevOps branch and refer to the same branch (or tag) when validating/running all the template pipelines in this build pipeline.
The ultimate goal is to pass multiple variables to the template (via a loop/for each) to create multiple build artifacts referencing public repositories (in this test a public terraform github repo). We could then maybe only need to maintain the variable data and not the source/reference code.
I thought DevOps did this 'OOTB', but realised that this was probably not the case. Something like this:
Repo: InfraAsData (all pipelines)
Branch: feature/sparse-clone-repo
Updated azure-pipeline.yaml and added template templates/sparse-clone.yaml (FYI - this template is untested as well)
azure-pipeline.yaml:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'IAC/InfraAsData'
refs: 'refs/heads/$(branch_name)'
parameters:
- name: branch
displayName: branch
type: string
default: $(Build.SourceBranch)
variables:
- name: branch_name
value: ${{ parameters.branch }}
- template: ./templates/variables/resource-groups.yaml#InfraAsData #variables template
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template-Sparse-Clone-To-Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: $(Build.SourceBranch)/templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: '${{ variables[template.repoUrl] }}'
repoPath: '${{ variables[template.repoPath] }}'
artifactPath: '${{ variables[template.artifactPath] }}'
sparse-clone.yaml
parameters:
repoUrl: ''
repoPath: ''
artifactPath: ''
#testing
steps:
- script: |
md $(Build.BinariesDirectory)/$(parameters.artifactPath)
git clone --filter=blob:none --sparse $(parameters.repoUrl) $(parameters.artifactPath)
cd $(Build.BinariesDirectory)/$(parameters.artifactPath)
git sparse-checkout init --cone
git sparse-checkout set $(parameters.repoPath)
git checkout main
dir $(Build.BinariesDirectory)/$(parameters.artifactPath)
displayName: 'Clone Github Repo Subdirectory - filter blob none'
When I run this from the feature/sparse-clone-repo branch in DevOps I get an error that the new template does not exist in the main branch (it doesn't of course).
/build/test/azure-pipeline.yaml: File /templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/main version
I have also tried using no resources.repository, using $(Build.SourceBranch) $(Build.SourceBranchName) as refs value and as the template path reference (like in the 2nd template example).
EDIT:
If i remove the resource.repositories reference, then the correct branch is identified, but the azure-pipeline.yaml relative path is appended to the template path:
/build/test/azure-pipeline.yaml: File /build/test/$(Build.SourceBranch)/templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/feature/pipelines
So correct branch, but not the correct relative path.
EDIT 2:
I can remove the resources section and use a relative path
../../templates/variables/resource-groups.yaml
but this is not dynamic, to a 'root' reference, so would not work if the folder structure were to change (EG I moved the azure-pipeline.yaml up a level)
I'm sure I am missing something obvious or misunderstanding the docs (or pipeline templates). Any pointers gratefully received!
When I run this from the feature/sparse-clone-repo branch in DevOps I
get an error that the new template does not exist in the main branch
(it doesn't of course).
From your description, seems you thought the 'refs' of the resources determine the branch of the alias of the template part.
But it is not, the 'resources.repositories.repository' doesn't have a section named 'refs', only 'ref' is allowed.
In your situation, you used a section named 'refs' which doesn't exist. Pipeline will use the default branch of the repository to looking for the template YAML. So change the default branch of the repository or change the 'refs' to 'ref' will solve the first issue.
And I notice you were using variable in resources section, you have two mistakes.
One is '$()' can't use in compile time part. Another is even compile time variables are also not allowed in 'ref' section.
ref
string
ref name to checkout; defaults to 'refs/heads/main'. The branch
checked out by default whenever the resource trigger fires. Does not
accept variables.
You can use your pipeline like this:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'BowmanCP/template_branch'
ref: 'refs/heads/${{parameters.branch}}'
parameters:
- name: branch
displayName: branch
type: string
default: main2
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template_Sparse_Clone_To_Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: ./templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: 'xxx'
repoPath: 'xxx'
artifactPath: 'xxx'
Basically, variables can't be used in your situation and there doesn't have a parameter reuse feature. That's everything.

Gitlab pipeline: use variable in trigger path

I'm trying to develop a common gitlab pipeline that uses a matrix to trigger another pipeline but I want to add a variable in the path. What I wrote till now is:
variables:
PROJECT_NAME: name
PROJECT_HELM: helmpipe
DEV_CUSTOMERS: cust1 cust2
deploy:dev:
stage: deploy
variables:
UPSTREAM_CI_COMMIT_SHORT_SHA: $CI_COMMIT_SHORT_SHA
UPSTREAM_CI_COMMIT_BRANCH: $CI_COMMIT_BRANCH
UPSTREAM_CI_COMMIT_TAG: $CI_COMMIT_TAG
IMAGE_NAME: ${PROJECT_NAME}
CUSTOMER: $CUSTOMER
trigger: my/project/$PROJECT_HELM
parallel:
matrix:
- CUSTOMER: $DEV_CUSTOMERS
only:
- DEV
But gitlab returns error saying that the triggered project can not be found. It seems that the variable $PROJECT_HELM is not converted to its real value in trigger path. How can I do it?
Thanks
Below is the working example where the variable ENV is used to trigger the sit pipeline. My pipeline file is in pipeline folder at root level. And the file name is sit-pipeline.yaml
stages:
- build
variables:
ENV: sit
sit-pipeline:
stage: build
trigger:
include:
- local: pipeline/${ENV}-pipeline.yaml
There could be 2 issues
Either you need to use ${var} syntax instead of $var syntax
trigger: my/project/${PROJECT_HELM}
Or PROJECT_HELM variable should resolve to a yaml file defining the pipeline. Your variable is assigned to helmpipe so I think you are missing .yaml extension

With yaml pipelines, is there a way to select an environment parameter from a dynamic list of all environments?

We've been migrating some of our manual deployment processes from Octopus to Azure DevOps Yaml pipelines. One of the QoL changes we're sorely missing is to be able to select the environment from a drop-down list/ auto-complete field as we could in Octopus.
Is there a way to achieve this? Currently, the only way I can think of doing it is to have a repo with a .yaml template file updated with a list of new environments as part of our provisioning process... Which seems less than ideal.
If you are going to trigger the pipeline manually then you can make use of Runtime parameters in the Azure DevOps pipeline.
For Example:
In order to make OS image name selectable from a list of choices, you can use the following snippet.
parameters:
- name: EnvName
displayName: EnvName
type: string
default: A
values:
- A
- B
- C
- D
- E
- F
trigger: none # trigger is explicitly set to none
jobs:
- job: build
displayName: build
steps:
- script: echo building $(Build.BuildNumber) with ${{ parameters.EnvName }}
Documentation about runtime parameters are here.
The downside to this is that the trigger: None limits you that the pipeline can only be manually triggered. Not sure how this works with other trigger options.

Azure DevOps - Use parameter to set path trigger in yaml pipeline definition

I'd like to set up an Azure pipeline using yaml template. What I struggle with is the path triggers definition. I'd like the individual pipelines to start up, only if they actually differ from their earlier versions.
core.yaml is a generic template, which is used across the different projects.
parameters:
- name: projectPath
type: string
trigger:
branches:
include:
- master
- feature/*
paths:
include:
- ${{ parameters.projectPath }} # this is the problematic part
steps:
... # restore, build, test, etc tasks are defined here
build.yaml files are defined for every project. They only extend on core.yaml and supply the
required parameters:
extends:
template: core.yaml
parameters:
projectPath: src/project1
Using the value ${{ parameters.projectPath }} fails as "Template expression is not allowed in the given context". Same occurs if I surround it with quotes.
Using the value $(parameters.projectPath) works, but it triggers even if I change a file outside the defined path.
I also tried using the actual values instead of a parameter (e.g. "src/project1"), but it also triggers for every change, even if I only touch the file "src/project2/foo.md".
The examples I saw did not use triggers within a template. But normally I got a syntax error when the pipeline is triggered (e.g. unexpected value) when I misplaced a node. So it's just my expectation that it should work.
Any suggestions?
This is not possible, because this part
trigger:
branches:
include:
- master
- feature/*
paths:
include:
- ${{ parameters.projectPath }} # this is the problematic part
needs to be known on compilation time, but you provide parameter which is evaluated at runtime.
And this syntax $(parameters.projectPath) doesn't make any sense, and probably is evaluated to empy value and then everything is included into trigger.