Azure Pipeline deploy job with dynamic environment - azure-devops

We have two environments defined, "stage" and "prod". Prod requires approval for deploying, where as stage does not.
What I'm trying to achieve is that if the pipeline runs against the master branch, it should use the prod environment, and all other branches should use the stage environment.
Problem is that I can't seem to be able to change the environment in runtime. I've also tried using templates, which sort of works, except that I can't pass runtime information as parameters to the template.
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.

I will start with first last one:
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
Resource restriction are evaluated on stage level so if you put those deployment jobs on the same stage (or you didn't define a stage - then you have implicit stage) - this is the reason why it asks for approval. If you move deployment jobs to seprate stages you won't be asked for approval. Please keep in mind to move condition to a stage level.
Another apporach it would be conditional for envrionment as follows:
jobs:
- deployment: DeployWeb
displayName: deploy Web App
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
environment: 'Dev'
${{ if ne(variables['Build.SourceBranchName'], 'master') }}:
environment: 'campo-dev'
strategy:
# Default deployment strategy, more coming...
runOnce:
deploy:
steps:
- checkout: self
- script: echo my first deployment

I do this with an extends yaml. You can define your stages via the environment's parameter (defaulted) and then your calling yaml extends it and passes tasks for build and deploy. The build section is ran once. And the deploy section will loop for each environment.
The approvals comes from the Environment approvals setup in Devops
#Extends Yaml
parameters:
- name: buildSteps
type: stepList
default: []
- name: deploySteps
type: stepList
default: []
- name: environments
type: object
default:
- name: Stage
branch: any
dependsOn:
- Builds
- name: Prod
branch: any
dependsOn:
- Stage
stages:
- stage: Builds
jobs:
- job: Building
steps:
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
- ${{ each env in parameters.environments }}:
- stage: ${{ env.name }}
dependsOn: ${{ env.dependsOn }}
condition: and(succeeded(), or(startsWith(variables['Build.SourceBranch'], 'refs/heads/${{ env.branch }}'), eq('${{ env.branch }}', 'any')))
jobs:
- deployment: Deploy${{ env.name }}
displayName: Deploy to ${{ env.name }}
variables:
- group: ENV-${{ env.name }}
environment: ${{ env.name }}
strategy:
runOnce:
deploy:
steps:
- ${{ each step in parameters.deploySteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
#Calling Yaml
#*Trimmed* header stuff
resources:
repositories:
- repository: cicd
type: git
name: SharedRepo
ref: main
extends:
template: extends-deploy.yaml#cicd
parameters:
buildSteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 2'
deploySteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 2'

Related

YAML Extend Stage Template

Looking to try and get some help figuring out a solution to what seems like a simple task. I am trying to develop some extend YAML templates for security within our ADO pipelines but first I have to get passed this error I am experiencing.
The error being returned by ADO when I try to validate or run the pipeline is
/ADO_Stage_Restrictions_Dev.yml#AdoRestrictions (Line: 7, Col: 3): Unexpected value job
Working Extend Template YAML
This template validates and executes without issue, which to me means I am successfully passing the stages object into the extends template
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
${{ stage }}
Broken Extend Template YAML
This template does not validate and throws the 'Unexpected value job' exception. based on the stage schema I would assume that I would be able to loop the jobs property within the stage.
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- ${{ each job in stage.jobs }}:
${{ job }}
Build YAML
The main yaml file that extends stages
resources:
repositories:
- repository: self
type: git
ref: refs/heads/Development
- repository: AdoRestrictions
type: git
name: utl-yaml-templates
ref: refs/heads/main
trigger: none
pool:
name: PROD
extends:
template: ADO_Stage_Restrictions_Dev.yml#AdoRestrictions
parameters:
stageObjs:
- stage: 'BuildStage'
displayName: 'Build Test'
jobs:
- job: 'BuildJob'
displayName: 'Build'
steps:
- task: PowerShell#2
displayName: 'Hello World'
inputs:
targetType: inline
script: |
Write-Host "Hello World"
There's nothing in your yaml which defines the start of the stage, or declares the start of the list of jobs; that's why it wasn't expecting to find 'job' there. You can add those parts in, like this:
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName }}
jobs:
- ${{ each job in stage.jobs }}:
${{ job }}

Dealing with multiple Azure Pipelines with multiple Environment targets

I'm trying to set up a pipelines template to streamline releases. We have numerous repos, and multiple team environments that need to utilize this. The release pipeline should look like this:
Build -> Dev Deploy -> Test Deploy -> UAT Deploy -> Prod Deploy
When the user manually runs the pipeline they should be allowed to specify which environment they are running for (ie: Team 1, Team 2; then the Dev Deploy step knows to use the Team 1 Dev environment for example). I first attempted to use a separate variable template that contained the environments in one spot. This does not work according to Azure Pipelines parameter value from variable template, other than allowing users to type a freeform string which is not ideal. The workarounds as far as I know are:
Add the list of environments to every azure-pipelines.yml (not feasible as we have many repos and have environments changing frequently)
Provide a freeform text box that the user can type in their environment (not a good solution as it allows typos and is generally just a bad user experience)
This is my current workaround, which "works" but is not ideal. UAT/Prod is not tied to a successful Test stage completion, because it would require all Test stages to succeed, and we typically will only deploy to the team's environment that is currently working on the repo. And since not all stages will always run, an app deployed to prod will still show the pipeline is still waiting on stages to run with the blue clock.
- stage: DeployToTeam1Dev
displayName: Deploy to Team1 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team1 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam1Test
displayName: Deploy to Team1 Test (MIVA)
dependsOn:
- DeployToTeam1Dev
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team1 Test
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam2Dev
displayName: Deploy to Team2 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team2 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam2Test
displayName: Deploy to Team2 Test (UAT)
dependsOn:
- DeployToTeam2Dev
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team2 Test
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam3Dev
displayName: Deploy to Team3 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team3 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam3Test
displayName: (NYI) Deploy to Team3 Test
dependsOn:
- DeployToTeam3Dev
jobs:
- job: Team3TestDeploy
timeoutInMinutes: 10
pool:
vmImage: 'ubuntu-latest'
steps:
- bash: echo "Team3 Test does not currently exist. Skipping this stage"
- stage: DeployToTeam4Dev
displayName: Deploy to Team4 Dev
dependsOn:
- Build
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Team4 Dev
testRegion: true
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
- stage: DeployToTeam4Test
displayName: (NYI) Deploy to Team4 Test
dependsOn:
- DeployToTeam4Dev
jobs:
- job: Team4TestDeploy
timeoutInMinutes: 10
pool:
vmImage: 'ubuntu-latest'
steps:
- bash: echo "Team4 Test does not currently exist. Skipping this stage"
- stage: DeployToUAT
displayName: (NYI) Deploy to UAT
dependsOn:
- Build
jobs:
- job: UATDeploy
timeoutInMinutes: 10
pool:
vmImage: 'ubuntu-latest'
steps:
- bash: echo "UAT is not yet implement"
- stage: DeployToProd
displayName: Deploy to Prod
dependsOn:
- DeployToUAT
jobs:
- template: ../../TaskGroups/DotNetCore/webDeploy-v2.yml
parameters:
environment: Prod
testRegion: false
siteName: ${{ parameters.siteName }}
virtualAppName: ${{ parameters.virtualAppName }}
Has anyone else ran in to a similar problem and come up with a solution besides hardcoding a list of environments in every azure-pipelines.yml? Thanks for the help in advance.

How to use stages from 1 ymal pipeline to 2 pipeline Azure Devops

I'm working on a project where we have 2 pipelines. 1 pipeline is currently working as expected having 8 stages in it.
Now I want to write code for 2 pipelines where I want to utilize few stages (approx. 4 stages) from the 1 pipeline because code and functionality are similar.
Is there any way I can achieve this in the Azure DevOps YAML pipeline?
Sure, you can export similar stages to a template, then you can use it in other pipelines with extends:
# File: azure-pipelines.yml
trigger:
- master
extends:
template: start.yml
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- task: CmdLine#2
displayName: Test 3 - Will Fail
inputs:
script: echo "Script Test"
The template will be (for example):
# File: start.yml
parameters:
- name: buildSteps # the name of the parameter is buildSteps
type: stepList # data type is StepList
default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
pool: Hosted VS2017
jobs:
- job: secure_buildjob
steps:
- script: echo This happens before code
displayName: 'Base: Pre-build'
- script: echo Building
displayName: 'Base: Build'
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ if ne(pair.value, 'CmdLine#2') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'CmdLine#2') }}:
'${{ pair.value }}': error
- script: echo This happens after code
displayName: 'Base: Signing'

Switch between Hosted and Scaleset agents in Azure Devops during runtime

I have scaleset agents configured and working fine. But i would also like to take advantage of Microsoft Hosted agents whenever possible. So i added a parameter to the pipeline to choose which agent pool to use for that particular pipeline run.
I manged to set the pool/image variables based on the chosen parameter but unable to make a container job use the selected pool due to a difference in syntax as shown below.
Is there a way to modify the schema of yaml template based on the user selected parameter.
I think we need to not display vmImage property conditionally. I tried below snippet but it doesn't work. vmImage property is always missing.
pool:
${{ if ne(parameters.pool, 'Azure Pipelines') }}:
name: ${{ parameters.pool }}
${{ if eq(parameters.pool, 'Azure Pipelines') }}:
vmImage: 'ubuntu-latest'
UPDATED:
parameters:
- name: pool
displayName: Agent Pool
type: string
default: Scalesets
values:
- Scalesets
- Hosted
jobs:
- deployment: '${{ parameters.jobName }}'
displayName: ${{ coalesce(parameters.jobDisplayName, 'Deploy Infrastructure') }}
${{ if parameters.dependsOn }}:
dependsOn: '${{ parameters.dependsOn }}'
${{ if parameters.condition }}:
condition: '${{ parameters.condition }}'
pool:
name: ${{ parameters.pool }}
vmImage: 'ubuntu-latest'
container:
image: '${{ parameters.image }}'
endpoint: 'Container Registry - Prd'
environment: ${{ parameters.environment }}
variables:
- name: 'Release-Tag'
value: '${{ parameters.release }}'
- name: 'Layer'
value: '${{ parameters.environmentLayer }}'
strategy:
runOnce:
deploy:
steps:
- script: |
echo $(Release-Tag) $(Layer)
displayName: 'Show release condidate'
You can set the data type of your parameters as object. The value of object type can be any YAML structure. Please find more details about data types of parameters in this document.
Here is my sample:
parameters:
- name: pool
type: object
default:
pool:
vmImage: ubuntu-latest
jobs:
- job: build
pool: ${{ parameters.pool }}
steps:
- script: echo Hello, world!
You can also choose the job to run based on the value of your parameters:
trigger: none
parameters:
- name: type
type: string
values:
- Scalesets
- Hosted
jobs:
- ${{ if eq(parameters.type, 'Scalesets') }}:
- job: Scalesets
pool: Default
steps:
- script: echo Scalesets
- ${{ if eq(parameters.type, 'Hosted') }}:
- job: Hosted
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hosted

Can't conditionally add script to Azure Devops Yaml pipeline

I'm trying to make use of Container Job template in a pipeline which runs on both Microsoft Hosted and Self-Hosted (Containerized) build agents.
ContainerJob template works well when run in Microsoft Hosted but fails in Self-Hosted agent saying cannot run container inside a containerized build agent. Error makes sense though.
I figured that if below section can be added/removed conditionally then same Container Job template works in both agents.
${{ if not(parameters.isSelfHosted) }}:
container:
image: ${{ parameters.generalImage}}
endpoint: ${{ parameters.endpoint}}
environment: ${{ parameters.environment }}
But the condition is always true and container section is added always and failed in self-hosted agent always. I think expressions are not allowed randomly inside a template.
I can separate out this template into 2 templates and load them in respective build agent but that's the last resort. Any help in dynamically creating/altering a template is highly appreciated.
Are you looking for conditional container job? See my script:
parameters:
- name: isSelfHosted
displayName: isSelfHosted
type: boolean
default: true
values:
- false
- true
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
${{ if not(parameters.isSelfHosted) }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.2
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.2 SDK
${{ if eq(parameters.isSelfHosted, 'true') }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.1
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.1 SDK
I can choose which container (.net core 2.1 or .net core 2.2) via the value of isSelfHosted parameter.
It uses Runtime parameters, so I can manage the condition(check or uncheck the box) when running the pipeline:
Update:
See Job, stage, and step templates with parameters
Move the content below into a template file:
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
${{ if not(parameters.isSelfHosted) }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.2
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.2 SDK
${{ if eq(parameters.isSelfHosted, 'true') }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.1
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.1 SDK
Then the main azure-pipelines.yml should be
parameters:
- name: isSelfHosted
displayName: isSelfHosted
type: boolean
default: true
values:
- false
- true
stages:
- template: templates/xxx.yml # Template reference
parameters:
isSelfHosted: xxx (Value of isSelfHosted parameter)