Using the Azure Devops gui, it is very easy to scope variable groups to pipeline stages. I need to replicate this functionality in a yaml build/release pipeline but I cannot find a way to do it. Anyone found a way to do this yet?
Scope variable groups to pipeline stages functionality in a yaml
build/release pipeline
With YAML, the way to achieve that is just specify the Variable group at the stage level to let it available only to this specific stage.
For example, I have a variable group names 1122. And 2 stages: one and two. Now, I want this variable group only available for stage one, which means the stage two should not access its content.
See below simple sample:
stages:
- stage: one
displayName: one
variables:
- group: 1122
jobs:
- job: A
steps:
- bash: echo $(a)
- stage: two
displayName: two
jobs:
- job: A1
steps:
- bash: echo $(a)
parameters:
- name: stage
displayName: Stage
type: string
default: Development
values:
- Development
- Staging
- NonProd
variables:
serviceConnection: 'Your-Favourite-SC'
stages:
- stage: Deploy_Resources
displayName: Deploy Resources
variables:
- group: "Somevargroupname - Release"
- group: "Somevargroupname - ${{ parameters.stage }}"
Quick and simple!
Related
My team is responsible for 10 microservices and it would be useful to have a single pipeline that triggers their individual CI/CD pipelines.
I know it is possible to use pipeline triggers, like
resources:
pipelines:
- pipeline: MasterPipeline
source: DeployAllMicroservices
trigger: true
and I can add this to the pipelines and create a very simple DeployAllMicroservices pipeline. This works, but the pipelines will be triggered in a random order.
The thing is, two services need to be rolled out first before the other 8 can be deployed. Is there a way to first trigger pipeline A & B, where pipelines C-J are triggered after their completion?
Something else I've tried is to load the pipeline files A.yml, B.yml as templates from the master pipeline.
steps:
- template: /CmcBs/Pipelines/A.yml
- template: /CmcBs/Pipelines/B.yml
but that doesn't work with full-fledged pipelines (starting with trigger, pool, parameters, et cetera).
Currently DevOps does not support multiple pipelines as the triggering pipeline of one pipeline at the same time.
There is a workaround you can refer to:
Set pipelineA as the triggering pipeline of the pipelineB.
Set pipelineB as the triggering pipeline of the other pipelines(pipelines c-j).
For more info about the triggering pipeline, please see Trigger one pipeline after another.
Another approach is to use stages in order to execute first pipeline A,B and then C to J.
An example .yml for this approach would be the below.
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- stage: FirstBatch
displayName: Build First Batch
jobs:
- job: pipelineA
displayName: Build pipelineA
steps:
- script: echo pipelineA
displayName: pipelineA
- job: pipelineB
displayName: Build pipelineB
steps:
- script: echo pipelineB
displayName: pipelineB
- stage: SecondBatch
displayName: Build Second Batch
jobs:
- job: pipelineC
displayName: Build pipelineC
steps:
- checkout: none
- script: echo Build pipelineC
displayName: Build pipelineC
- job: pipelineD
displayName: Build pipelineD
steps:
- checkout: none
- script: echo Build pipelineD
displayName: Build pipelineD
- job: pipelineE
displayName: Build pipelineE
steps:
- checkout: none
- script: echo Build pipelineE
displayName: Build pipelineE
The drawback of this approach would be to have a single pipeline and not separated pipelines for your microservices. In order to decouple more this solution you could use templates.
I know that it is possible to conditionally set a variable in azure pipelines yml.
See https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#conditional-insertion
Is it also somehow possible to conditionally use a variable group?
Let's say if a pipeline run variable is set or has a certain value then the pipeline shall use a variable group. If not then the group shall not be used.
Thank you
Is it possible like here for instance:
trigger: none
pr: none
variables:
isProd: true
stages:
- stage: Test
displayName: Build and restore
variables:
- ${{ if eq(variables['isProd'], 'false') }}:
- group: QA
- ${{ if eq(variables['isProd'], 'true') }}:
- group: PROD
jobs:
- job: A
steps:
- bash: echo $(name)
I have an azure-pipelines.yml controlling my build etc... works ok.
However I seem to be restricted, I can not use jobs: or stages: as documented in their allowable syntax for describing pipelines when editing the pipeline within the online editor.
In the above image no matter what or where I type, I do not get access to jobs, stages...
documentation: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
how can I get access to these keywords as documented?
According to the syntax in the screenshot, steps is defined in your yaml. Remove the steps and the stages should be displayed normally. steps should be under job level and jobs need to be under the stage level.
Please refer to the Pipeline structure in this official document:
Please refer to below structure to configure your azure-pipelines.yml.
pool:
vmImage: windows-2019
stages:
- stage: Build
jobs:
- job: BuildJob
steps:
- script: echo Building!
- stage: Test
jobs:
- job: TestOnWindows
steps:
- script: echo Testing on Windows!
- job: TestOnLinux
steps:
- script: echo Testing on Linux!
- stage: Deploy
jobs:
- job: Deploy
steps:
- script: echo Deploying the code!
Let's suppose I have 3 environments on Azure: Dev, Test and Prod. I have the same pipeline for building and deploying the resources and the code for each one of the environments except for two differences:
different trigger branch
different variable values
What is the correct approach for this scenario? Because at least 3 come to my mind, none of which is perfect:
Option 1: I guess I could create a single pipeline on Azure DevOps (triggered by any of 3 branches) with 3 stages for each environment and for each stage add a condition to run depending on the source branch, like this:
condition: eq(variables['Build.SourceBranch'], 'refs/heads/a-branch-name')
and in each stage reference different variables. But this would introduce code duplication in each stage - when adding or modifying a step I would have to remember to edit 3 stages - not desirable.
Option 2: Create 3 separate YAML files in my repository, each one of them with specified trigger branch and referencing the same variable names, then create 3 different pipeline on Azure DevOps, each one of them with different variable values. But this would also introduce code duplication.
Option 3: Create 1 build-and-deploy.yaml file as a template with the steps defined in it and then create another 3 YAML files referring to that template, each with different trigger branch and with different variable values in each Azure Pipeline, like this:
trigger:
branches:
include:
- a-branch-name
steps:
- template: build-and-deploy.yaml
parameters:
parameterName1: $(parameterValue1)
parameterName2: $(parameterValue2)
This seems to be the best option but I haven't seen it used anywhere in the examples so maybe I'm just unaware of downsides of it, if there are any.
Here's how to do it with a shared pipeline config that gets included into env-specific pipelines.
To support 2 environments (dev and prod) you'd need:
1 shared pipeline yaml
2 env-specific yamls, one for each env
2 pipelines created in Azure DevOps, one for each env; each pipeline referencing corresponding yaml
pipeline-shared.yml:
variables:
ARTIFACT_NAME: ApiBuild
NPM_CACHE_FOLDER: $(Pipeline.Workspace)/.npm
stages:
- stage: Build
displayName: Build
pool:
vmImage: 'ubuntu-latest'
demands: npm
jobs:
...
- stage: Release
displayName: Release
dependsOn: Build
pool:
vmImage: 'ubuntu-latest'
jobs:
...
pipeline-dev.yml:
# Trigger builds on commits to branches
trigger:
- dev
# Do not trigger builds on PRs
pr: none
extends:
template: pipeline-shared.yml
pipeline-prod.yml
trigger:
- master
pr: none
extends:
template: pipeline-shared.yml
According to your description, if you want different stages to share the same repo resource, but their trigger branch and variable values are different.
Regarding trigger branch, you can use expression {{if ......}} to determine the trigger branch condition.
Regarding variable values, you can define templates and variable groups to specify them through parameters.
Here is an example, you can refer to it：
First go to Library under Pipelines, click on the Variable group to add a variable group. You can add multiple variables to this variable group.
Repo structure:
azure-pipelines.yml：
sample：
stages:
- template: stage/test.yml
parameters:
${{if contains(variables['Build.SourceBranch'], 'master')}}:
variableGroup: devGroup
stageName: Dev
test: a
${{if contains(variables['Build.SourceBranch'], 'test')}}:
stageName: test
test: b
stage/test. yml：
parameters:
- name: stageName
displayName: Test
type: string
default: test
- name: test
displayName: Test
type: string
default: test
- name: variableGroup
displayName: Test
type: string
default: test
stages:
- stage: Test_${{ parameters.stageName }}
variables:
- group: ${{parameters.variableGroup}}
jobs:
- job: Test1
pool:
vmImage: vs2017-win2016
steps:
- script: echo "Hello Test1"
- script: echo ${{ parameters.test }}
- script: echo $(dev1)
Of course, if you want to use a single variable, you can define the variable directly in yaml without adding a variable group.
In the UI (classic) Release pipelines, there are build-in variables like $(Release.EnvironmentName), which changes depending on which stage you are at.
Is there an equivalent when using a YAML pipeline?
Also in the classis mode variable groups are automatically scoped to the stage if there are linked, is this also possible in YAML somehow?
The answer is yes. The usage of the predefined variable in Yaml pipeline is the same with classic pipeline. $(Release.EnvironmentName) is for classic release pipeline. You can refer to variables $(System.StageName) or $(Environment.Name) in the yaml pipeline.
Click the links to find more predefined build variables. Check here to learn more about variables in yaml pipeline.
For variable groups scoped to stage in classic release pipeline
It is possible in Yaml pipeline. In yaml you can scope your variable groups in job or stage by define variables under job or stage in yaml. For below example.
1,Variables scoped to stage Build:
stages:
- stage: Build
variables:
- group: SourceMessage
jobs:
- job: A
- stage: release
jobs:
- job: B
2, Variables scoped to all stages:
stages:
variables:
- group: SourceMessage
- stage: Build
jobs:
- job: A
- stage: release
jobs:
- job: B
3, Variables scoped to job A:
stages:
- stage: Build
jobs:
- job: A
variables:
- group: SourceMessage
- stage: release
jobs:
- job: B