How do I write my ADO YAML in such a way that there might be demands such as Agent.Name = 'abc' that might be needed occasionally while queuing the build?
An example is if we want to investigate a build failure that is happening on a specific build agent
Another use case is if there is a software upgrade such as .NET core and we want to test our build on the upgraded software on one agent before we upgrade the software on all the build agents
Classic pipelines had the ability to add demands without updating the pipeline.
You can create a YAML Pipeline in Azure DevOps and use parameter to tweak the demands:
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- main
parameters:
# Define a parameter
- name: demand
displayName: Demand
type: string
default: DefaultPool
values:
- DefaultPool
- SpecialSoftware
variables:
demand: '${{ parameters.demand }}'
pool:
vmImage: ubuntu-latest
# Use the variable
${{ if ne(variables['demand'], '') }}:
demands: ${{variables.demand}}
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
When you want to manually run the pipeline, you can choose the value for the demand parameter:
Documentation:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/runtime-parameters
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/demands
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents
Related
I have an Azure DevOps Deployment YAML Pipeline that creates an Azure App Service and deploys code to it. The actual pipeline is more complex, but I am simplifying it for this question.
Currently my pipeline can successfully deploy to a specific Azure Subscription (Service Connector) with resource names defined in variables.
I need to parametrize the pipeline so that it can deploy to several different environments (means Azure Subscriptions) using multiple Service Connectors. Each environment has a different Azure Resource naming convention.
Is there any way to read the value of pipeline variables from an XML or JSON file? This way I can have multiple config files for each environment and store them as part of my repository.
Is this a right approach for multi-environment deployment pipeline configuration?
You can use variable templates. There is another interesting link: Learn more about variable reuse with templates.
Here I have this flat folder structure (for the clarity of the sample):
.
| deploy-app.job.yaml
| deploy-app.pipeline.yaml
| variables.dev.yaml
| variables.prod.yaml
So here we're trying to run the reusable job deploy-app.job.yaml with different variable sets.
I've defined some variables in each variable.{env}.yaml files
# variables.dev.yaml
variables:
vmImage: ubuntu-20.04
serviceConnection: dev-service-connection
# variables.prod.yaml
variables:
vmImage: ubuntu-20.04
serviceConnection: prod-service-connection
The deploy-app.job.yaml file accepts a parameter that allow to inject a variable template:
# deploy-app.job.yaml
parameters:
- name: envVariablesTemplate
type: string
jobs:
- deployment: deploy
variables:
# Inject the verianle template here
- template: ${{ parameters.envVariablesTemplate }}
pool:
# Use the variable from the template
vmImage: ${{ variables.vmImage }}
strategy:
runOnce:
deploy:
steps:
- task: AzureCLI#2
displayName: Hello from azure cli
inputs:
# Use the variable from the template
azureSubscription: ${{ variables.serviceConnection }}
scriptType: pscore
scriptLocation: inlineScript
inlineScript: echo 'Hello from azure cli'
In the main pipeline, I can create different stages and inject the desired vairables:
# deploy-app.pipeline..yaml
stages:
- stage: dev
condition: succeeded()
jobs:
- template: ./deploy-app.job.yaml
parameters:
envVariablesTemplate: ./variables.dev.yaml
- stage: prod
dependsOn: dev
condition: succeeded()
jobs:
- template: ./deploy-app.job.yaml
parameters:
envVariablesTemplate: ./variables.prod.yaml
Based on your needs, you can add multiple variable templates, having a naming convention etc. Really up to you and depends on the complexity of your pipelines.
By using XML transformation, we can perform the operation. Check the below link to get the complete steps.
https://www.dragonspears.com/blog/how-to-handle-continuous-deployment-across-multiple-environments
Create different stages.
Create build transformation using XML transformer
Another option is to utilize ‘XML variable substitution.’ Sensitive information can be stored and secured within Azure Pipelines vs. in plain text transformation files.
Continuous Deployment
The same steps mentioned here are available in the link mentioned above.
I have a solution where a git branch is directly related to an environment (this has to be this way, so please do not discuss whether this is good or bad, I know it is not best practice).
We have the option to run a verification deployment (including automatic tests) towards an environment, without actually deploying the solution to the environment. Because of this, I would like to set up a pipeline that runs this verification for an environment, whenever a pull request is opened towards that environment's branch. Moreover, I am using a template for the majority of the pipeline. The actual pipeline in the main repository is just a tiny solution that points towards the template pipeline in another repository. This template, in turn, has stages for each respective environment.
I have, in the main pipeline, successfully added a solution that identifies the current branch, which for pull requests should be the target branch:
variables:
- name: currentBranch
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
value: $(System.PullRequest.TargetBranch)
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
value: $(Build.SourceBranch)
I would like to send this variable currentBranch down to the template through a parameter, as my template pipeline has different stages depending on the branch. My solution was to use the pipeline like this:
extends:
template: <template-reference>
parameters:
branch: $(currentBranch)
...and then for a stage in my pipeline do this:
- stage: TestAndDeployBranchName
condition: eq('${{ parameters.branch }}', 'refs/heads/branchName')
jobs:
- job1... etc.
Basically, the stage should run if the current branch is either "branchName", or (for pull requests) when the target branch is "branchName", which comes from the "branch" parameters that is sent to the template.
However, I see here that System.PullRequest.TargetBranch is not available for templates and further here that the parameters are not available for templates (the variable is empty) when the template is expanded. Thus my pipeline does not work as expected (the condition does not trigger when it should, ie. when there is a match on the branch name).
Is there any way that I can use System.PullRequest.TargetBranch in a condition within a template, or should I look for another solution?
After investigating this further I concluded that what I am trying to do is not possible.
In short, System.PullRequest.TargetBranch (and I assume at least some other variables within System.PullRequest are not available in compile time for template, which is when conditions are evaluated. Thus, using these variables in a condition in a template is not possible.
As my goal was to have certain steps run for pull requests only, based on the target branch of the pull request, I solved this by creating duplicate pipelines. Each pipeline is the same and references the same template, except for that the input parameter for the template is different. I then added each "PR pipelines" to run as part of the branch policy each respective branch this was applicable.
This works great, however it requires me to create a new pipeline if I have the same requirement for another branch. Moreover, I have to maintain each PR pipeline separately (which can be both good and bad).
Not an ideal solution, but it works.
Reference PR pipeline:
trigger: none # no trigger as PR triggers are set by branch policies
#This references the template repository to reuse the basic pipeline
resources:
repositories:
- repository: <template repo>
type: git # "git" means azure devops repository
name: <template name> # Syntax: <project>/<repo>
ref: refs/heads/master # Grab latest pipeline template from the master branch
stages:
- stage: VerifyPullRequest
condition: |
and(
not(failed()),
not(canceled()),
eq(variables['Build.Reason'], 'PullRequest')
)
displayName: 'Verify Pull Request'
jobs:
- template: <template reference> # Template reference
parameters:
image: <image>
targetBranch: <targetBranch> # Adjust this to match each respective relevant branch
The targetBranch parameter is the used in relevant places in the template to run PR verification.
Example of branch policy:
(Set this up for each relevant branch)
Picture of branch policy set up
After checking your script, we find we can not use the
variables:
- name: currentBranch
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
value: $(System.PullRequest.TargetBranch)
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
value: $(Build.SourceBranch)
in the variables.
The variables will duplicate the second value to first one.
This will cause your issue.
So, on my side, I create a work around and hope this will help you. Here is my main yaml:
parameters:
- name: custom_agent
displayName: Use Custom Agent
type: boolean
default: true
- name: image
type: string
default: default
resources:
repositories:
- repository: templates
type: git
name: Tech-Talk/template
trigger: none
pool:
vmImage: windows-latest
# vmImage: ubuntu-20.04
stages:
- stage: A
jobs:
- job: A1
steps:
- task: PowerShell#2
name: printvar
inputs:
targetType: 'inline'
script: |
If("$(Build.Reason)" -eq "PullRequest"){
Write-Host "##vso[task.setvariable variable=currentBranch;isOutput=true]$(System.PullRequest.TargetBranch)"
}
else{
Write-Host "##vso[task.setvariable variable=currentBranch;isOutput=true]$(Build.SourceBranch)"
}
- stage: B
condition: eq(dependencies.A.outputs['A1.printvar.currentBranch'], 'refs/heads/master')
dependsOn: A
jobs:
- job: B1
variables:
varFromA: $[ stageDependencies.A.A1.outputs['printvar.currentBranch'] ]
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "$(varFromA)"
- template: temp.yaml#templates
parameters:
branchName: $(varFromA)
agent_pool_name: ''
db_resource_path: $(System.DefaultWorkingDirectory)
Please Note:
If we use this, we need to modified your temp yaml.
We need to move the condition to the main yaml and make the temp yaml only steps is left.
Is it possible to create a wrapper pipeline in Azure DevOps that simply runs two or more independent pipelines (in parallel) and does nothing else?
I have a problem to solve. and the scenario looks like this "*
In my project, I have say 9 teams and each designing separate Sanity Test Script. All of them have their own existing Sanity Pipeline. i.e. 9 Sanity Pipelines*
There is a plan that there will be only One Master/ Wrapper pipeline and this in turn calls upon 9 child pipelines pertaining to Sanity
When master run by Release Engineer or IT Area lead to get report, the child pipelines run in Parallel
Also in master Pipeline, I do not want to be too much lengthy. Simply I want to mention the name of Child pipeline in my individual Job tag ( with params may be ) and it will run. easy configurable " So I was thinking to use following at my master pipeline: resources: pipelines:
pipeline: Sanity1 Source: P00xxx-Sanity1-Pipeline
pipeline: Sanity2 Source: P00xxx-Sanity2-Pipeline
This list should be easily expandable.......
Then How in Jobs--> Job --> Steps can I run the pipeline using alias, e.g. Sanity1 ?? Any example code snippet?
Another approach would be to take the pipeline and leverage templates. The wrapper pipeline can call a template which will leverage all the desired tasks and execution and can be setup to run in parallel as part of the pipeline.
Here's a blog post about this
You can use PowerShell and rest API (Builds - Queue). Add PowerShell step and compose any run sequence. Here you can find different examples to queue builds:
Build Pipeline using powershell
Trigger another build exist in project in Azure Devops
According to your description, you can try to use parameters and conditions to set up the pipeline.
You can try the following Yaml sample:
trigger:
- none
parameters:
- name: pipeline1
displayName: Gradle sample #PipelineName
type: boolean
default: false
- name: pipeline2
displayName: groovy-spring-boot-restdocs-example.git #PipelineName
type: boolean
default: false
- name: pipeline3
displayName: Gradle sample-CI #PipelineName
type: boolean
default: false
pool:
vmImage: 'windows-latest'
steps:
- ${{ if eq(parameters.pipeline1, true) }}:
- task: TriggerPipeline#1
inputs:
serviceConnection: 'TestBuild'
project: '966ef694-1a7d-4c35-91f3-41b8c5363c48'
Pipeline: 'Build'
buildDefinition: 'Gradle sample' #PipelineName
Branch: 'master'
- ${{ if eq(parameters.pipeline2, 'true') }}:
- task: TriggerPipeline#1
inputs:
serviceConnection: 'TestBuild'
project: '966ef694-1a7d-4c35-91f3-41b8c5363c48'
Pipeline: 'Build'
buildDefinition: 'groovy-spring-boot-restdocs-example.git' #PipelineName
Branch: 'master'
...
Explanation:
I use the Trigger Azure DevOps Pipeline task from the Trigger Azure DevOps Pipeline Extension to trigger the child pipelines.
The parameters is used to list the pipeline names and the if condition is used to determine whether the pipeline name is selected.
Result:
When you run the pipeline you could select the checkbox.
If the pipeline name has been selected, the corresponding task will run and trigger the corresponding pipeline.
This should make the selection interface clearer.
I want the jobs to go one after another and the first job should control the execution of the following one.
As there are no approvals currently available in the YAML pipeline for deployment outside Kubernetes, I'm using the Manual Intervention to stop the job from being run. But apparently, it doesn't stop the job before instead, it stops the upcoming stage. What do I do wrong? I would expect some notification on the intervention, but it fails immediately and doesn't stop the next job at all.
This is the part of the code for the Deploy STG stage, where the parameters.interventionEnabled is set to true
jobs:
- job: RunOnServer
displayName: 'Reject or resume'
pool: server
continueOnError: false
steps:
- task: ManualIntervention#8
displayName: 'Manual Intervention'
timeoutInMinutes: 0
inputs:
instructions: 'reject or resume'
enabled: ${{ parameters.interventionEnabled }}
- job: Deploy
displayName: ${{ parameters.name }}
pool:
name: ${{ parameters.agentPoolName }}
steps:
- checkout: none # skip checking out the default repository resource
- task: DownloadPipelineArtifact#2
displayName: Download NPM build artifact
inputs:
artifact: ${{ parameters.artifactName }}
buildType: 'current'
targetPath: ${{ parameters.artifactPath }}
Hey Andree ManualIntervention#8 is not supported in YAML. It is roadmapped for 2020\Q2.
I think the route you want to go down is to use approvals with generic environment types.
So you define a deployment job and environment in your yaml like so
- deployment: DeploymentHosted Context
displayName: Runs in Hosted Pool
pool:
vmImage: 'Ubuntu-16.04'
# creates an environment if it doesn't exist
environment: 'Dev'
strategy:
runOnce:
deploy:
steps:
- bash: |
echo This multiline script always runs in Bash.
echo Even on Windows machines!
And you use the GUI to protect the Environment.
Navigate to Pipelines -> Environments.
Select the Environment (you can pre-create them).
Then add and an Approval
There are some drawbacks when compared to classic release definitions and being able to manual trigger to stages. You may not want every artifact to be a candidate for each stage, and if you don't approve the environment it will eventually timeout and report failure. Other good discussion in the comments here.
This is now available:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/manual-validation?view=azure-devops&tabs=yaml
- task: ManualValidation#0
timeoutInMinutes: 1440 # task times out in 1 day
inputs:
notifyUsers: |
test#test.com
example#example.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'resume'
I am creating YAML pipeline in Azure DevOps that consists of two stages.
The first stage (Prerequisites) is responsible for reading the git commit and creates a comma separated variable containing the list of services that has been affected by the commit.
The second stage (Build) is responsible for building and unit testing the project. This Stage consists of many templates, one for each Service. In the template script, the job will check if the relevant Service in in the variable created in the previous stage. If the job finds the Service it will continue to build and test the service. However if it cannot find the service, it will skip that job.
Run.yml:
stages:
- stage: Prerequisites
jobs:
- job: SetBuildQueue
steps:
- task: powershell#2
name: SetBuildQueue
displayName: 'Set.Build.Queue'
inputs:
targetType: inline
script: |
## ... PowerShell script to get changes - working as expected
Write-Host "Build Queue Auto: $global:buildQueueVariable"
Write-Host "##vso[task.setvariable variable=buildQueue;isOutput=true]$global:buildQueueVariable"
- stage: Build
jobs:
- job: StageInitialization
- template: Build.yml
parameters:
projectName: Service001
projectLocation: src/Service001
- template: Build.yml
parameters:
projectName: Service002
projectLocation: src/Service002
Build.yml:
parameters:
projectName: ''
projectLocation: ''
jobs:
- job:
displayName: '${{ parameters.projectName }} - Build'
dependsOn: SetBuildQueue
continueOnError: true
condition: and(succeeded(), contains(dependencies.SetBuildQueue.outputs['SetBuildQueue.buildQueue'], '${{ parameters.projectName }}'))
steps:
- task: NuGetToolInstaller#1
displayName: 'Install Nuget'
Issue:
When the first stages runs it will create a variable called buildQueue which is populated as seen in the console output of the PowerShell script task:
Service001 Changed
Build Queue Auto: Service001;
However when it gets to stage two and it tries to run the build template, when it checks the conditions it returns the following output:
Started: Today at 12:05 PM
Duration: 16m 7s
Evaluating: and(succeeded(), contains(dependencies['SetBuildQueue']['outputs']['SetBuildQueue.buildQueue'], 'STARS.API.Customer.Assessment'))
Expanded: and(True, contains(Null, 'service001'))
Result: False
So my question is how do I set the dependsOn and condition to get the information from the previous stage?
It because you want to access the variable in a different stage from where you defined them. currently, it's impossible, each stage it's a new instance of a fresh agent.
In this blog you can find a workaround that involves writing the variable to disk and then passing it as a file, leveraging pipeline artifacts.
To pass the variable FOO from a job to another one in a different stage:
Create a folder that will contain all variables you want to pass; any folder could work, but something like mkdir -p $(Pipeline.Workspace)/variables might be a good idea.
Write the contents of the variable to a file, for example echo "$FOO" > $(Pipeline.Workspace)/variables/FOO. Even though the name could be anything you’d like, giving the file the same name as the variable might be a good idea.
Publish the $(Pipeline.Workspace)/variables folder as a pipeline artifact named variables
In the second stage, download the variables pipeline artifact
Read each file into a variable, for example FOO=$(cat $(Pipeline.Workspace)/variables/FOO)
Expose the variable in the current job, just like we did in the first example: echo "##vso[task.setvariable variable=FOO]$FOO"
You can then access the variable by expanding it within Azure Pipelines ($(FOO)) or use it as an environmental variable inside a bash script ($FOO).