Different agent pool or demands for the same Azure Pipeline based on trigger - azure-devops

I have an Azure Pipeline written in YAML which runs from a CI trigger whenever changes are made to the master branch. It can also be manually triggered from Pull Requests or by users against any branch.
Due to the use of a number of licensed components, the build from master needs to run on a specific agent. The other builds do not, and in fact I would rather they run on other agent(s).
So my question is, is there any way to specify a different agent/pool within the YAML pipeline based on what triggered the build, or what branch the build is building? I'd like this to be behaviour which is configured in the pipeline permanently, rather than requiring users to update the YAML on each branch they wish to build elsewhere.
I can't see anything obvious in the sections of the documentation on the pool/demands/condition keywords.

I solved this by putting the steps for the job into a template, and then creating a set of jobs in the pipeline with different condition entries, so that we can set demands based on those conditions.
A skeleton version looks like this:
- stage: Build
jobs:
- job: TopicBranchAndPullRequestBuild
condition: or(startsWith(variables['Build.SourceBranch'], 'refs/heads/topic'), startsWith(variables['Build.SourceBranch'], 'refs/pull'))
displayName: 'Build topic Branch or Pull Request'
pool:
name: the-one-and-only-pool
demands:
- HasLicensedComponents -equals false
steps:
- template: build-template.yml
- job: MasterAndReleaseBranchBuild
condition: or(eq(variables['Build.SourceBranch'], 'refs/heads/master'), startsWith(variables['Build.SourceBranch'], 'refs/heads/release'))
displayName: 'Build master or release Branch'
pool:
name: the-one-and-only-pool
demands:
- HasLicensedComponents -equals true
steps:
- template: build-template.yml
Obviously the values given here are for example only, but otherwise this is what I have working.

Could you try expressions? I've used this with success on variable groups so it might work on agent pools.
- ${{ if eq(variables['build.SourceBranchName'], 'prod') }}:
- pool: Host1
- ${{ if eq(variables['build.SourceBranchName'], 'staging') }}:
- pool: Host2
- ${{ if not(and(eq(variables['build.SourceBranchName'], 'staging'), eq(variables['build.SourceBranchName'], 'prod'))) }}:
- pool: Host3
Credit for original workaround for dynamically pulling variable groups here: https://github.com/MicrosoftDocs/vsts-docs/issues/3702#issuecomment-574278829

The pool keyword specifies which pool to use for a job of the pipeline. A pool specification also holds information about the job's strategy for running. You can specify a pool at the pipeline, stage, or job level. The pool specified at the lowest level of the hierarchy is used to run the job.
is there any way to specify a different agent/pool within the YAML
pipeline based on what triggered the build, or what branch the build
is building?
For this issue , you can create different yaml files for different branches, specify different agent pools in the corresponding yaml file.
You can switch to the yaml file of different branches in pipeline with the option shown below:

parameters:
- name: self_hosted_agent
displayName: Choose your pool agent
type: string
default: windows-latest
values:
- windows-latest
- self-hosted
- name: agent_name
displayName: Choose your agent
type: string
default: 'Laptop Clint'
values:
- 'Laptop Clint'
- 'Laptop Eastwood'
variables:
- name: pool_agent_key
${{ if eq(parameters.self_hosted_agent, 'windows-latest') }}:
value: vmImage
${{ if eq(parameters.self_hosted_agent, 'self-hosted') }}:
value: name
- name: pool_agent_value
${{ if eq(parameters.self_hosted_agent, 'windows-latest') }}:
value: windows-latest
${{ if eq(parameters.self_hosted_agent, 'self-hosted') }}:
value: Iot-Windows
- name: agent_name
value: ${{ parameters.agent_name }}
stages:
- stage: build
displayName: Build
pool:
${{ variables.pool_agent_key }}: ${{ variables.pool_agent_value }}
${{ if eq(parameters.self_hosted_agent, 'self-hosted') }}:
demands:
- agent.name -equals ${{ variables.agent_name }}

Related

How to create Azure Pipeline Template to run a jobList all on the same agent?

I am trying to make a pipeline template that takes a JobList a parameter and runs all the jobs, while ensuring that they run on the same agent every time. Basically the approach I've been taking is to try to adapt this answer into a genericized template format.
This is what I have so far, and I've tried a lot of slight tweaks of this with nothing passing the Validate test on the pipeline that calls it.
parameters:
- name: jobsToRun
type: jobList
- name: pool
type: string
default: Default
- name: demands
type: object
default: []
jobs:
- job:
steps:
- script: echo "##vso[task.setvariable variable=agentName;isOutput=true;]$(Agent.Name)"
pool:
name: ${{ parameters.pool }}
demands:
- ${{ each demand in parameters.demands }}:
${{ demand }}
- ${{ each j in parameters.jobsToRun }}:
${{ each pair in j }}:
${{ pair.key }} : ${{ pair.value }}
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
What am I doing wrong here? It seems like it should be possible if that answer I reference is correct, but it seems like I'm just a bit off.
Name missing on the job.., example below.
- job: 'test-Name'
Steps need a associated job and pool to run is declared inside
jobsToRun:
- job: sample_job1
displayName: "sample_job1"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"
On this bottom pool declaration...
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
i am not sure but i have tried this many times but i think this can't be included separate to the job since each individual job is passed as parameter. pool definition needs to be inside the job or inside the job template if you are using templates..
example:
jobsToRun:
- job: output_message_job1
displayName: "in pipe Output Message Job"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"

How to use variable from variable group in conditional insertion expression in Azure DevOps pipeline

My aim is to use a variable group to hold global configuration settings which apply to all pipelines. Specifically, I want the ability to flip a switch in a variable value to switch from using hosted build agents to using private build agents instead.
I have a variable group named my-variable-group which contains a variable named UseHostedAgents. I can toggle its value between true and false.
The pipeline:
variables:
- group: my-variable-group
stages:
- stage: deploy
pool:
${{ if eq(variables['UseHostedAgents'], 'true') }}:
vmImage: ubuntu-latest
${{ else }}:
name: private-pool
jobs:
...
I can't figure out how to get this to work. It seems as though the variable group variable values aren't available in the conditional insertion expression. I've tried everything I can think of to no avail. Any ideas?
You could use a variable/parameter to store whether you want to run on custom agent pool or shared and use it.
Example with parameters:
parameters:
- name: environment
type: string
default: private-pool
variables:
${{ if eq(parameters.environment, 'private-pool') }}:
buildPoolName: private-pool
${{ else }}:
buildPoolName: ubuntu-latest
And then on your job you should use the buildPoolName
- job: 'Job 1'
displayName: 'Build'
pool:
name: $(buildPoolName)
steps:
I think it's just the syntax error, can you try this?
variables:
- group: my-variable-group
stages:
- stage: deploy
pool:
${{ if eq(variables.UseHostedAgents, true) }}:
vmImage: ubuntu-latest
${{ else }}:
name: private-pool
jobs:
...

Conditional Default-Value in Azure Pipeline DevOps

I use Azure Pipelines to Build my solution. When building manually the user can decide which Build Configuration to use. Now I want the default value (and so the value when triggered automatically) to be different according to the branch. This is my (non-working) approach:
name: PR-$(Build.SourceBranchName)-$(Date:yyyyMMdd)$(Rev:.r)
trigger:
- develop
- test
- master
pool:
vmImage: 'windows-latest'
parameters:
- name: BuildConfiguration
displayName: Build Configuration
${{ if eq(variables['Build.SourceBranch'], 'refs/heads/develop') }}:
default: Debug
${{ if eq(variables['Build.SourceBranch'], 'refs/heads/test') }}:
default: Release
${{ if eq(variables['Build.SourceBranch'], 'refs/heads/master') }}:
default: Release
values:
- Debug
- Release
The conditional part (${{...}) simply does not work. (*"A template expression is not allowed in this context")
Is there any other approach to make this happen?
This is not posssible to use expressins to select default value for paramataers.
Paramaters alco can't be optionla:
Parameters must contain a name and data type. Parameters cannot be optional. A default value needs to be assigned in your YAML file or when you run your pipeline. If you do not assign a default value or set default to false, the first available value will be used.
What you can do is use parameters and variables:
name: PR-$(Build.SourceBranchName)-$(Date:yyyyMMdd)$(Rev:.r)
trigger:
- develop
- test
- master
pool:
vmImage: 'windows-latest'
parameters:
- name: BuildConfiguration
displayName: Build Configuration
default: Default
values:
- Debug
- Release
- Default
variables:
- name: buildConfiguration
${{ if and(eq(variables['Build.SourceBranch'], 'refs/heads/develop'), eq(parameters.BuildConfiguration, 'Default')) }}:
value: Debug
${{ elseif and(eq(variables['Build.SourceBranch'], 'refs/heads/test'), eq(parameters.BuildConfiguration, 'Default')) }}:
value: Release
${{ elseif and(eq(variables['Build.SourceBranch'], 'refs/heads/master'), eq(parameters.BuildConfiguration, 'Default')) }}:
value: Release
${{ else }}:
value: ${{parameters.BuildConfiguration}}
steps:
- script: |
echo $(buildConfiguration)
echo $(Build.SourceBranch)

Azure Pipeline deploy job with dynamic environment

We have two environments defined, "stage" and "prod". Prod requires approval for deploying, where as stage does not.
What I'm trying to achieve is that if the pipeline runs against the master branch, it should use the prod environment, and all other branches should use the stage environment.
Problem is that I can't seem to be able to change the environment in runtime. I've also tried using templates, which sort of works, except that I can't pass runtime information as parameters to the template.
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
I will start with first last one:
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
Resource restriction are evaluated on stage level so if you put those deployment jobs on the same stage (or you didn't define a stage - then you have implicit stage) - this is the reason why it asks for approval. If you move deployment jobs to seprate stages you won't be asked for approval. Please keep in mind to move condition to a stage level.
Another apporach it would be conditional for envrionment as follows:
jobs:
- deployment: DeployWeb
displayName: deploy Web App
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
environment: 'Dev'
${{ if ne(variables['Build.SourceBranchName'], 'master') }}:
environment: 'campo-dev'
strategy:
# Default deployment strategy, more coming...
runOnce:
deploy:
steps:
- checkout: self
- script: echo my first deployment
I do this with an extends yaml. You can define your stages via the environment's parameter (defaulted) and then your calling yaml extends it and passes tasks for build and deploy. The build section is ran once. And the deploy section will loop for each environment.
The approvals comes from the Environment approvals setup in Devops
#Extends Yaml
parameters:
- name: buildSteps
type: stepList
default: []
- name: deploySteps
type: stepList
default: []
- name: environments
type: object
default:
- name: Stage
branch: any
dependsOn:
- Builds
- name: Prod
branch: any
dependsOn:
- Stage
stages:
- stage: Builds
jobs:
- job: Building
steps:
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
- ${{ each env in parameters.environments }}:
- stage: ${{ env.name }}
dependsOn: ${{ env.dependsOn }}
condition: and(succeeded(), or(startsWith(variables['Build.SourceBranch'], 'refs/heads/${{ env.branch }}'), eq('${{ env.branch }}', 'any')))
jobs:
- deployment: Deploy${{ env.name }}
displayName: Deploy to ${{ env.name }}
variables:
- group: ENV-${{ env.name }}
environment: ${{ env.name }}
strategy:
runOnce:
deploy:
steps:
- ${{ each step in parameters.deploySteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
#Calling Yaml
#*Trimmed* header stuff
resources:
repositories:
- repository: cicd
type: git
name: SharedRepo
ref: main
extends:
template: extends-deploy.yaml#cicd
parameters:
buildSteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 2'
deploySteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 2'

How to use matrix variable within pool demands?

For an Azure Pipelines yaml file I want to run a set of tasks once on every agent in a certain pool. When I looked at jobs strategy matrix, it looked as a good solution for this, but it currently cannot pick-up the variable I use for this.
The pipeline yaml file, relevant for this problem is this part:
resources:
- repo: self
trigger: none
jobs:
- job: RunOnEveryAgent
strategy:
maxParallel: 3
matrix:
agent_1:
agentName: Hosted Agent
agent_2:
agentName: Hosted VS2017 2
agent_3:
agentName: Hosted VS2017 3
pool:
name: Hosted VS2017
demands:
- msbuild
- visualstudio
- Agent.Name -equals $(agentName)
steps:
- (etc.)
With this script I tried to setup a matrix to run once on each of the three agents in the pool. However when I try to reference the agent on the list of demands it doesn't pick it up. The actual error message is as follows:
[Error 1] No agent found in pool Hosted VS2017 which satisfies the specified demands:
msbuild
visualstudio
Agent.Name -equals $(agentName)
Agent.Version -gtVersion 2.141.1
If I hardcode the agent name it does work:
demands:
- msbuild
- visualstudio
- Agent.Name Hosted VS2017 3
Is it supported to use these variables in the pool demands? Or should I use a different variable or expression?
parameters:
- name: agentNames
type: object
default: []
jobs:
- job: RunOnEveryAgent
strategy:
matrix:
${{ each agentName in parameters.agentNames }}:
${{ agentName }}:
agentName: ${{ agentName }}
pool:
name: Hosted VS2017
demands:
- msbuild
- visualstudio
- Agent.Name -equals $(agentName)
This would be a better solution in case you want to add more agents in future
Variables are not supported in some of these jobs, due to the order in which they get expanded.
What you can do however, is to use template include syntax (https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops) for your job strategy and pass the agentname in as a parameter.
So your build job in its own YAML file might look like:
parameters:
agentName1: ''
agentName2: ''
agentName3: ''
jobs:
- job: RunOnEveryAgent
strategy:
maxParallel: 3
matrix:
agent_1:
agentName: ${{ parameters.agentName1 }}
agent_2:
agentName: ${{ parameters.agentName2 }}
agent_3:
agentName: ${{ parameters.agentName3 }}
pool:
name: Hosted VS2017
demands:
- msbuild
- visualstudio
- Agent.Name -equals ${{ parameters.agentName3 }}
steps:
Your main azure-pipelines.yml then changes to look like this:
resources:
- repo: self
trigger: none
jobs:
- template: buildjob.yml
parameters:
agentName1: 'Hosted Agent'
agentName2: 'Hosted VS2017 2'
agentName3: 'Hosted VS2017 3'