I'm trying to set up a pipelines template to streamline releases. We have numerous repos, and multiple team environments that need to utilize this. The release pipeline should look like this:
Build -> Dev Deploy -> Test Deploy -> UAT Deploy -> Prod Deploy
When the user manually runs the pipeline they should be allowed to specify which environment they are running for (ie: Team 1, Team 2; then the Dev Deploy step knows to use the Team 1 Dev environment for example). I first attempted to use a separate variable template that contained the environments in one spot. This does not work according to Azure Pipelines parameter value from variable template, other than allowing users to type a freeform string which is not ideal. The workarounds as far as I know are:
Add the list of environments to every azure-pipelines.yml (not feasible as we have many repos and have environments changing frequently)
Provide a freeform text box that the user can type in their environment (not a good solution as it allows typos and is generally just a bad user experience)
This is my current workaround, which "works" but is not ideal. UAT/Prod is not tied to a successful Test stage completion, because it would require all Test stages to succeed, and we typically will only deploy to the team's environment that is currently working on the repo. And since not all stages will always run, an app deployed to prod will still show the pipeline is still waiting on stages to run with the blue clock.
- stage: DeployToTeam1Dev
displayName: Deploy to Team1 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team1 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam1Test
displayName: Deploy to Team1 Test (MIVA)
dependsOn:
- DeployToTeam1Dev
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team1 Test
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam2Dev
displayName: Deploy to Team2 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team2 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam2Test
displayName: Deploy to Team2 Test (UAT)
dependsOn:
- DeployToTeam2Dev
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team2 Test
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam3Dev
displayName: Deploy to Team3 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team3 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam3Test
displayName: (NYI) Deploy to Team3 Test
dependsOn:
- DeployToTeam3Dev
jobs:
- job: Team3TestDeploy
timeoutInMinutes: 10
pool:
vmImage: 'ubuntu-latest'
steps:
- bash: echo "Team3 Test does not currently exist. Skipping this stage"
- stage: DeployToTeam4Dev
displayName: Deploy to Team4 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team4 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam4Test
displayName: (NYI) Deploy to Team4 Test
dependsOn:
- DeployToTeam4Dev
jobs:
- job: Team4TestDeploy
timeoutInMinutes: 10
pool:
vmImage: 'ubuntu-latest'
steps:
- bash: echo "Team4 Test does not currently exist. Skipping this stage"
- stage: DeployToUAT
displayName: (NYI) Deploy to UAT
dependsOn:
- Build
jobs:
- job: UATDeploy
timeoutInMinutes: 10
pool:
vmImage: 'ubuntu-latest'
steps:
- bash: echo "UAT is not yet implement"
- stage: DeployToProd
displayName: Deploy to Prod
dependsOn:
- DeployToUAT
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Prod
testRegion: false
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
Has anyone else ran in to a similar problem and come up with a solution besides hardcoding a list of environments in every azure-pipelines.yml? Thanks for the help in advance.
Related
How can i loop over an array or through an object to create stages?
Below is a yml file that works. You can see the build stage loops over the parameters environments for jobs. IS it possible to achieve the same thing for the publishing stages?
The publishing stages require manual approval, must run in order and only when the previous stage is successfully complete?
parameters:
- name: 'environments'
type: object
default:
- environment: development
variableGroup: strata2-admin-spa-vg
dependsOn: 'build'
- environment: test
variableGroup: strata2-test-admin-spa-vg
dependsOn: 'development'
- environment: production
variableGroup: strata2-development-variables
dependsOn: 'development'
- name: 'buildTemplate'
type: string
default: buildTemplate.yml
- name: 'publishTemplate'
type: string
default: publishTemplate.yml
trigger:
- main
pool:
vmImage: ubuntu-latest
stages:
- stage: build
displayName: Build stage
jobs:
# Can I do this for stages?
- ${{each build in parameters.environments}}:
- template: ${{parameters.buildTemplate}}
parameters:
environment: ${{build.environment}}
variableGroup: ${{build.variableGroup}}
# How to loop over parameters.environments to dynamically create stages
- stage: Publish_Development
displayName: Publish development environment
dependsOn: build
jobs:
- template: ${{parameters.publishTemplate}}
parameters:
environment: Development_websites
variableGroup: strata2-admin-spa-vg
- stage: Publish_Test
displayName: Publish test environment
dependsOn: Publish_Dev
jobs:
- template: ${{parameters.publishTemplate}}
parameters:
environment: Test_websites
variableGroup: strata2-test-admin-spa-vg
- stage: Publish_Production
displayName: Publish production environment
dependsOn: Publish_Test
jobs:
- template: ${{parameters.publishTemplate}}
parameters:
environment: Production_websites
variableGroup: strata2-development-variables
You can create a stages object the same way you created the environments object.
stages:
Publish_Development:
- stage: Publish_Development
- displayName: Publish development environment
- dependsOn:
- ...
Publish_Test
- stage: Publish_Development
- ...
Then you can loop over the stages object like you did with environments.
- ${{each stage in parameters.stages}}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName}}
dependsOn: ${{ stage.dependsOn}}
...
Managed to get this working for myself. Stages automatically generated based on numerical batch numbers, that run in parallel. Hope it helps someone out there.
- name: batches
displayName: BATCH
type: object
default:
- 1
- 2
- 3
stages:
- ${{ each stage in parameters.batches }}:
- stage: BATCH_${{ stage }}
dependsOn: []
jobs:
- job: PREP
steps:
- template: install.yml
- job: RUN
dependsOn: PREP
timeoutInMinutes: 300
steps:
- template: run.yml
parameters:
batch: ${{ stage }}
Would be nice if the batch numbers weren't displayed as an editable box when running the pipeline from Azure DevOps. I tried setting them as fixed values, but couldn't get that to work, so this is what I went with in the end.
We have two environments defined, "stage" and "prod". Prod requires approval for deploying, where as stage does not.
What I'm trying to achieve is that if the pipeline runs against the master branch, it should use the prod environment, and all other branches should use the stage environment.
Problem is that I can't seem to be able to change the environment in runtime. I've also tried using templates, which sort of works, except that I can't pass runtime information as parameters to the template.
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
I will start with first last one:
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
Resource restriction are evaluated on stage level so if you put those deployment jobs on the same stage (or you didn't define a stage - then you have implicit stage) - this is the reason why it asks for approval. If you move deployment jobs to seprate stages you won't be asked for approval. Please keep in mind to move condition to a stage level.
Another apporach it would be conditional for envrionment as follows:
jobs:
- deployment: DeployWeb
displayName: deploy Web App
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
environment: 'Dev'
${{ if ne(variables['Build.SourceBranchName'], 'master') }}:
environment: 'campo-dev'
strategy:
# Default deployment strategy, more coming...
runOnce:
deploy:
steps:
- checkout: self
- script: echo my first deployment
I do this with an extends yaml. You can define your stages via the environment's parameter (defaulted) and then your calling yaml extends it and passes tasks for build and deploy. The build section is ran once. And the deploy section will loop for each environment.
The approvals comes from the Environment approvals setup in Devops
#Extends Yaml
parameters:
- name: buildSteps
type: stepList
default: []
- name: deploySteps
type: stepList
default: []
- name: environments
type: object
default:
- name: Stage
branch: any
dependsOn:
- Builds
- name: Prod
branch: any
dependsOn:
- Stage
stages:
- stage: Builds
jobs:
- job: Building
steps:
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
- ${{ each env in parameters.environments }}:
- stage: ${{ env.name }}
dependsOn: ${{ env.dependsOn }}
condition: and(succeeded(), or(startsWith(variables['Build.SourceBranch'], 'refs/heads/${{ env.branch }}'), eq('${{ env.branch }}', 'any')))
jobs:
- deployment: Deploy${{ env.name }}
displayName: Deploy to ${{ env.name }}
variables:
- group: ENV-${{ env.name }}
environment: ${{ env.name }}
strategy:
runOnce:
deploy:
steps:
- ${{ each step in parameters.deploySteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
#Calling Yaml
#*Trimmed* header stuff
resources:
repositories:
- repository: cicd
type: git
name: SharedRepo
ref: main
extends:
template: extends-deploy.yaml#cicd
parameters:
buildSteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 2'
deploySteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 2'
I'm trying to figure how to dynamically create a dependency and run the job based on the condition.
Here is the structure of my pipeline:
main.yaml:
stages:
- stage: build
jobs:
- template: build.yaml
- stage: deployDev
dependsOn: build
jobs:
- template: deployApp1.yaml
parameters:
environmentName: Dev
- template: deployApp2.yaml
parameters:
environmentName: Dev
- stage: deployQA
dependsOn: deployDev
jobs:
- template: promote.yaml
parameters:
environmentName: QA
- template: deployApp1.yaml
parameters:
environmentName: QA
- template: deployApp2.yaml
parameters:
environmentName: QA
promote.yaml
jobs:
- job: copy
steps:
- task:
deployApp1.yaml
jobs:
-job: deployApp1
steps:
- task:
deployApp2.yaml
jobs:
- job: deployApp2
steps:
- task:
In deployQA i have a separate job which copies the build artifacts and the next two jobs (deployApp1 and deployApp2) will fail without the copy step in deployQA.
I would like to create a conditional dependency on job: copy for job: deployApp1 so that it should be able to skip if i'm deploying to Dev which doesn't have this dependency. I already tried different solutions from different posts without any luck.
I know if i can add additional stage for the copy that would solve my problem but i would like to have the copy as part of the QA stage.
You want deployApp1 to depend on copy when running in stage deployQA and not depend on anything when running in stage deployDev?
You could add a dependsOn parameter to your template and use it to control job's dependencies:
stages:
- stage: build
jobs:
- template: build.yaml
- stage: deployDev
dependsOn: build
jobs:
- template: deployApp1.yaml
parameters:
environmentName: Dev
- template: deployApp2.yaml
parameters:
environmentName: Dev
- stage: deployQA
dependsOn: deployDev
jobs:
- template: promote.yaml
parameters:
environmentName: QA
- template: deployApp1.yaml
parameters:
environmentName: QA
dependsOn: copy
- template: deployApp2.yaml
parameters:
environmentName: QA
dependsOn: copy
# promote.yaml
jobs:
- job: copy
steps:
- task:
# deployApp1.yaml
parameters:
- name: environmentName
- name: dependsOn
default: []
jobs:
- job: deployApp1
dependsOn: ${{ parameters.dependsOn }}
steps:
- task:
# deployApp2.yaml
parameters:
- name: environmentName
- name: dependsOn
default: []
jobs:
- job: deployApp2
dependsOn: ${{ parameters.dependsOn }}
steps:
- task:
I was able to figure out the solution for my scenario from below post however to be able to achieve the answer to my specific use-case i had to set the makeExplicitDependency parameter to false.
${{ if eq(parameters.makeExplicitDependency, true) }}:
dependsOn: Test
thanks to below post:
How to dynamically reference previous jobs in Azure Pipelines if there are any in the current stage
I have scaleset agents configured and working fine. But i would also like to take advantage of Microsoft Hosted agents whenever possible. So i added a parameter to the pipeline to choose which agent pool to use for that particular pipeline run.
I manged to set the pool/image variables based on the chosen parameter but unable to make a container job use the selected pool due to a difference in syntax as shown below.
Is there a way to modify the schema of yaml template based on the user selected parameter.
I think we need to not display vmImage property conditionally. I tried below snippet but it doesn't work. vmImage property is always missing.
pool:
${{ if ne(parameters.pool, 'Azure Pipelines') }}:
name: ${{ parameters.pool }}
${{ if eq(parameters.pool, 'Azure Pipelines') }}:
vmImage: 'ubuntu-latest'
UPDATED:
parameters:
- name: pool
displayName: Agent Pool
type: string
default: Scalesets
values:
- Scalesets
- Hosted
jobs:
- deployment: '${{ parameters.jobName }}'
displayName: ${{ coalesce(parameters.jobDisplayName, 'Deploy Infrastructure') }}
${{ if parameters.dependsOn }}:
dependsOn: '${{ parameters.dependsOn }}'
${{ if parameters.condition }}:
condition: '${{ parameters.condition }}'
pool:
name: ${{ parameters.pool }}
vmImage: 'ubuntu-latest'
container:
image: '${{ parameters.image }}'
endpoint: 'Container Registry - Prd'
environment: ${{ parameters.environment }}
variables:
- name: 'Release-Tag'
value: '${{ parameters.release }}'
- name: 'Layer'
value: '${{ parameters.environmentLayer }}'
strategy:
runOnce:
deploy:
steps:
- script: |
echo $(Release-Tag) $(Layer)
displayName: 'Show release condidate'
You can set the data type of your parameters as object. The value of object type can be any YAML structure. Please find more details about data types of parameters in this document.
Here is my sample:
parameters:
- name: pool
type: object
default:
pool:
vmImage: ubuntu-latest
jobs:
- job: build
pool: ${{ parameters.pool }}
steps:
- script: echo Hello, world!
You can also choose the job to run based on the value of your parameters:
trigger: none
parameters:
- name: type
type: string
values:
- Scalesets
- Hosted
jobs:
- ${{ if eq(parameters.type, 'Scalesets') }}:
- job: Scalesets
pool: Default
steps:
- script: echo Scalesets
- ${{ if eq(parameters.type, 'Hosted') }}:
- job: Hosted
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hosted
In Azure Devops multistage YAML pipeline we got multiple environments.
In stages to run normally we do a build and deploy only in QA, so we need to deselect each stage manually. By default all stages are selected is it possible to have exact opposite, where all stages are deselected by default???
trigger: none
pr: none
stages:
- stage: 'Build'
jobs:
- deployment: 'Build'
pool:
name: Default
# testing
environment: INT
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
Start-Sleep -Seconds 10
- stage: 'Sandbox'
jobs:
- job: 'Sandbox'
pool:
name: Default
steps:
- checkout: none
# testing
- powershell: |
echo "Hello Testing"
- stage: 'Test'
jobs:
- job: 'DEV'
pool:
name: Default
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
- stage: 'QA'
dependsOn: ['Test','Test1','Test2']
jobs:
- job: 'QA'
pool:
name: Default
steps:
- checkout: none
# Testing
- powershell: |
echo "Hello Testing"
I am afraid that there is no UI (like stage to run) method that can meet your needs.
You could try to add parameters to your Yaml Sample.
Here is an example:
trigger: none
pr: none
parameters:
- name: stageTest
displayName: Run Stage test
type: boolean
default: false
- name: stageBuild
displayName: Run Stage build
type: boolean
default: false
stages:
- ${{ if eq(parameters.stageBuild, true) }}:
- stage: 'Build'
jobs:
- deployment: 'Build'
pool:
name: Default
environment: INT
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "Hello Testing"
Start-Sleep -Seconds 10
- ${{ if eq(parameters.stageTest, true) }}:
- stage: Test
dependsOn: []
jobs:
- job: B1
steps:
- script: echo "B1"
The parameters are used to determine whether to run these stages. You could add expressions before the stage to check if the parameter value could meet expression.
The default value is false. This means that the stage will not run by default.
Here is the result:
You can select the stage you need to run by clicking the selection box.
Update
Workaround has some limitations. When the select stage has depenencies, you need to select all dependent stages to run.
For example:
- stage: 'QA'
dependsOn: ['Test','Test1','Test2']
On the other hand, I have created a suggestion ticket to report this feature request. Here is the suggestion ticket link: Pipeline Deselect Stages By Default You could vote and add comment in it .
I've used this solution to build a nuget-package, and:
always push packages from master
conditionally push packages from other branches
Using GitVersion ensures that the packages from other branches get prerelease version numbers, e.g. 2.2.12-my-branch-name.3 or 2.2.12-PullRequest7803.4. The main branch simply gets 2.2.12, so the master branch is recognized as a "regular" version.
The reason I'm repeating the answer above, is that I chose to make the stage conditional instead of using an if:
trigger:
- master
parameters:
- name: pushPackage
displayName: Push the NuGet package
type: boolean
default: false
stages:
- stage: Build
jobs:
- job: DoBuild
steps:
- script: echo "I'm building a NuGet package (versioned with GitVersion)"
- stage: Push
condition: and(succeeded('build'), or(eq('${{ parameters.pushPackage }}', true), eq(variables['build.sourceBranch'], 'refs/heads/master')))
jobs:
- job: DoPush
steps:
- script: echo "I'm pushing the NuGet package"
Like the other answer, this results in a dialog:
But what's different from the (equally valid) solution with '${{ if }}', is that the stage is always shown (even if it's skipped):