Can't conditionally add script to Azure Devops Yaml pipeline - azure-devops

I'm trying to make use of Container Job template in a pipeline which runs on both Microsoft Hosted and Self-Hosted (Containerized) build agents.
ContainerJob template works well when run in Microsoft Hosted but fails in Self-Hosted agent saying cannot run container inside a containerized build agent. Error makes sense though.
I figured that if below section can be added/removed conditionally then same Container Job template works in both agents.
${{ if not(parameters.isSelfHosted) }}:
container:
image: ${{ parameters.generalImage}}
endpoint: ${{ parameters.endpoint}}
environment: ${{ parameters.environment }}
But the condition is always true and container section is added always and failed in self-hosted agent always. I think expressions are not allowed randomly inside a template.
I can separate out this template into 2 templates and load them in respective build agent but that's the last resort. Any help in dynamically creating/altering a template is highly appreciated.

Are you looking for conditional container job? See my script:
parameters:
- name: isSelfHosted
displayName: isSelfHosted
type: boolean
default: true
values:
- false
- true
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
${{ if not(parameters.isSelfHosted) }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.2
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.2 SDK
${{ if eq(parameters.isSelfHosted, 'true') }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.1
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.1 SDK
I can choose which container (.net core 2.1 or .net core 2.2) via the value of isSelfHosted parameter.
It uses Runtime parameters, so I can manage the condition(check or uncheck the box) when running the pipeline:
Update:
See Job, stage, and step templates with parameters
Move the content below into a template file:
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
${{ if not(parameters.isSelfHosted) }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.2
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.2 SDK
${{ if eq(parameters.isSelfHosted, 'true') }}:
container: mcr.microsoft.com/dotnet/core/sdk:2.1
steps:
- task: CmdLine#2
inputs:
script: |
echo 2.1 SDK
Then the main azure-pipelines.yml should be
parameters:
- name: isSelfHosted
displayName: isSelfHosted
type: boolean
default: true
values:
- false
- true
stages:
- template: templates/xxx.yml # Template reference
parameters:
isSelfHosted: xxx (Value of isSelfHosted parameter)

Related

Selecting Azure DevOps service connection based on if else logic for tenant selection

I am trying to select a service connection based on if else logic.
I have following two pipelines azure-caller.yml and template.yaml.
azure-caller.yml
parameters:
- name: directory
displayName: 'select appropriate environment directory'
values:
- Dev
- Stage
stages:
- template: job-templates/template.yml
parameters:
selectdirectory: $(select-directory)
template.yml
parameters:
selectdirectory: default
stages:
- stage: A
variables:
- name: service-connection
${{ if eq(parameters.selectdirectory, 'Dev') }}:
value: Dev-Service-connection
${{ elseif eq(parameters.selectdirectory, 'Stage') }}:
value: Stage-Service-connection
jobs:
- job: Install_terraform
displayName: Intialize Terraform
continueOnError: true
steps:
- checkout: self
- task: TerraformInstaller#0
inputs:
terraformVersion: '1.0.5'
- job: Validate_terraform_Plan
displayName: Validate Terraform
continueOnError: true
dependsOn: Install_terraform
steps:
- checkout: self
- task: TerraformTaskV2#2
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: ${{ variables.service-connection }}
backendAzureRmResourceGroupName: 'any-rg'
backendAzureRmStorageAccountName: 'any-storage'
backendAzureRmContainerName: 'statefile'
backendAzureRmKey: abcd.tfstate
- task: TerraformTaskV2#2
displayName: 'plan'
inputs:
provider: 'azurerm'
command: 'plan'
commandOptions: '-out="plan.out"'
environmentServiceNameAzureRM: ${{ variables.service-connection }}
My tasks are failing because it fetches the value of the service connection at compile time, which is empty.
Azure Caller is a wrapper/main pipeline, and the template is the only one that is allowed to use Dev or Stage service connections for deployment purposes to achieve security. Is there a way to accomplish this?
Your input will be helpful. thx
It is not supported to dynamically specify value of pipeline variable conditionally.
In the YMAL pipeline, you can use the if conditional to judge using or not using a pipeline variable. You can not use it to judge specifying a value or another one to a pipeline variable.
However, you can dynamically set the input conditionally on a task. See "Conditionally set a task input".
You also can dynamically specify value of parameter conditionally.
Below is an example to dynamically specify value of parameter conditionally. You can reference it to change your YAML files.
template.yml
parameters:
- name: ServiceConnection
type: string
stages:
- stage: A
displayName: 'Stage A'
jobs:
- job: A1
displayName: 'Job A1'
pool:
vmImage: ubuntu-latest
steps:
- task: Bash#3
displayName: 'Show parameter value'
inputs:
targetType: inline
script: echo "The value of ServiceConnection is ${{ parameters.ServiceConnection }}"
azure-pipelines.yml
parameters:
- name: directory
displayName: 'Select appropriate environment directory:'
type: string
values:
- Dev
- Stage
stages:
- template: template.yml
parameters:
${{ if eq(parameters.directory, 'Dev') }}:
ServiceConnection: Dev-Service-connection
${{ if eq(parameters.directory, 'Stage') }}:
ServiceConnection: Stage-Service-connection
Result
[UPDATE]
Dynamically set the input conditionally on the task.
template.yml
parameters:
- name: selectdirectory
type: string
stages:
- stage: A
displayName: 'Stage A'
jobs:
- job: Validate_terraform_Plan
displayName: 'Validate Terraform'
. . .
steps:
. . .
- task: TerraformTaskV2#2
inputs:
provider: 'azurerm'
command: 'init'
${{ if eq(parameters.selectdirectory, 'Dev') }}:
backendServiceArm: Dev-Service-connection
${{ if eq(parameters.selectdirectory, 'Stage') }}:
backendServiceArm: Stage-Service-connection
backendAzureRmResourceGroupName: 'any-rg'
backendAzureRmStorageAccountName: 'any-storage'
backendAzureRmContainerName: 'statefile'
backendAzureRmKey: abcd.tfstate
. . .
azure-pipelines.yml
parameters:
- name: directory
displayName: 'Select appropriate environment directory:'
type: string
values:
- Dev
- Stage
stages:
- template: template.yml
parameters:
selectdirectory: ${{ parameters.directory }}
Have you checked this answer out: AzureDevops: Can we pass dynamic value as service connection in yaml
Essentially rather then pass in your service connection name via an if statement have the service connection name stored in a variable.yml file that is determined off your selected directory and will load the appropriate service connection name.

Azure Pipeline deploy job with dynamic environment

We have two environments defined, "stage" and "prod". Prod requires approval for deploying, where as stage does not.
What I'm trying to achieve is that if the pipeline runs against the master branch, it should use the prod environment, and all other branches should use the stage environment.
Problem is that I can't seem to be able to change the environment in runtime. I've also tried using templates, which sort of works, except that I can't pass runtime information as parameters to the template.
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
I will start with first last one:
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
Resource restriction are evaluated on stage level so if you put those deployment jobs on the same stage (or you didn't define a stage - then you have implicit stage) - this is the reason why it asks for approval. If you move deployment jobs to seprate stages you won't be asked for approval. Please keep in mind to move condition to a stage level.
Another apporach it would be conditional for envrionment as follows:
jobs:
- deployment: DeployWeb
displayName: deploy Web App
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
environment: 'Dev'
${{ if ne(variables['Build.SourceBranchName'], 'master') }}:
environment: 'campo-dev'
strategy:
# Default deployment strategy, more coming...
runOnce:
deploy:
steps:
- checkout: self
- script: echo my first deployment
I do this with an extends yaml. You can define your stages via the environment's parameter (defaulted) and then your calling yaml extends it and passes tasks for build and deploy. The build section is ran once. And the deploy section will loop for each environment.
The approvals comes from the Environment approvals setup in Devops
#Extends Yaml
parameters:
- name: buildSteps
type: stepList
default: []
- name: deploySteps
type: stepList
default: []
- name: environments
type: object
default:
- name: Stage
branch: any
dependsOn:
- Builds
- name: Prod
branch: any
dependsOn:
- Stage
stages:
- stage: Builds
jobs:
- job: Building
steps:
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
- ${{ each env in parameters.environments }}:
- stage: ${{ env.name }}
dependsOn: ${{ env.dependsOn }}
condition: and(succeeded(), or(startsWith(variables['Build.SourceBranch'], 'refs/heads/${{ env.branch }}'), eq('${{ env.branch }}', 'any')))
jobs:
- deployment: Deploy${{ env.name }}
displayName: Deploy to ${{ env.name }}
variables:
- group: ENV-${{ env.name }}
environment: ${{ env.name }}
strategy:
runOnce:
deploy:
steps:
- ${{ each step in parameters.deploySteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
#Calling Yaml
#*Trimmed* header stuff
resources:
repositories:
- repository: cicd
type: git
name: SharedRepo
ref: main
extends:
template: extends-deploy.yaml#cicd
parameters:
buildSteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 2'
deploySteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 2'

Azure pipelines, question about variables, powershell and deployments

I am building Azure Pipelines for deployments of intranet, dotnet core projects from our Windows Self hosted agent.
I have a request from the team where if a PowerShell script initialize a variable to some value, the next step should include a specific environment.
The goal is, for some project, we compare a file from the agent to the DEV server and if different we request an approval.
I'm trying to make this work with global variables and get confused with different syntax.
Here's what I have:
YAML
variables:
- name: MYVAR
value: '0'
stages:
- stage: Init
jobs:
- job: DisplayBefore
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo Varname before :
echo $(MYVAR)
===> Displays: 0
- stage: RunScript
dependsOn: Init
jobs:
- job: CallScript
steps:
- checkout: none
- task: PowerShell#2
name: SetVarname
inputs:
targetType: filePath
filePath: '{agent}\compareFileContent.ps1'
- job: DisplayInSameStage
dependsOn: CallScript
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo Varname after :
echo ${{ variables.MYVAR }}
===> Displays: 0
echo $(MYVAR)
===> Displays: 0
- stage: DeploymentStage
dependsOn: RunScript
condition: eq(variables.MYVAR, '1')
jobs:
- deployment: DeploymentJob
environment: 'ATest'
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo Varname after :
echo ${{ variables.MYVAR }}
{agent}\compareFileContent.ps1
[CmdletBinding()]
param ()
echo $env:MYVAR
===> Displays: 0
$env:MYVAR = 2
echo $env:MYVAR
===> Displays: 2
echo "##vso[task.setvariable variable=env:MYVAR;]3"
exit
My plan is to have another stage for the opposite condition, maybe you would see a better approach or guidance on what I am missing ? Thanks !!!
*********** Update 1
I used environment variables as I understood it was the way to access them from script, if there is another way, I'd be glad to know !
I used Stages because I need to use an Environment and they seem to need their own stage and deployment block otherwise the environment is tested before the condition is evaluated.
Here is my latest test:
variables:
- name: MYVAR
value: '0'
stages:
- stage: SingleStage
jobs:
- job: DisplayBefore
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo $(MYVAR)
- job: CallScript
dependsOn: DisplayBefore
steps:
- checkout: none
- task: PowerShell#2
name: SetVarname
inputs:
targetType: filePath
filePath: '..\compareFileContent.ps1'
- deployment: DeploymentJob
dependsOn: DisplayInSameStage
condition: eq(variables.MYVAR, '1')
environment: 'ATest'
strategy:
runOnce:
deploy:
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
echo EnvName Finally: $(EnvName)
This YAML doesn't do much as it request the approval from the environment at the start.
My scenario ask that this approval is triggered only based on a value calculated in the PowerShell script.
Any ideas ? Thanks.
Jobs and stages usually run on different agents so you cannot expect environmental variables from different jobs to keep their values.
If you want to work with globals, you can have steps within one job. Otherwise, to keep some logical job/stage structure, you can just write and read values to/from an external source. This depends on your scenario.

How to use stages from 1 ymal pipeline to 2 pipeline Azure Devops

I'm working on a project where we have 2 pipelines. 1 pipeline is currently working as expected having 8 stages in it.
Now I want to write code for 2 pipelines where I want to utilize few stages (approx. 4 stages) from the 1 pipeline because code and functionality are similar.
Is there any way I can achieve this in the Azure DevOps YAML pipeline?
Sure, you can export similar stages to a template, then you can use it in other pipelines with extends:
# File: azure-pipelines.yml
trigger:
- master
extends:
template: start.yml
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- task: CmdLine#2
displayName: Test 3 - Will Fail
inputs:
script: echo "Script Test"
The template will be (for example):
# File: start.yml
parameters:
- name: buildSteps # the name of the parameter is buildSteps
type: stepList # data type is StepList
default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
pool: Hosted VS2017
jobs:
- job: secure_buildjob
steps:
- script: echo This happens before code
displayName: 'Base: Pre-build'
- script: echo Building
displayName: 'Base: Build'
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ if ne(pair.value, 'CmdLine#2') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'CmdLine#2') }}:
'${{ pair.value }}': error
- script: echo This happens after code
displayName: 'Base: Signing'

Extend YAML pipeline example validate step

the extend template example shared by MSFT in documentation suppose to fail the step but it's not failing it.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#extend-from-a-template
can anyone share a working example of this.
I want to to validate user yaml in extend template and fail the build in case of steps used that are not allowed as per firm policies.
The document here provides the confused direction. You can try to download your build log which does not action like you expect, then analyze the initializeLog.txt file contents. All of key evaluating expression are list there.
Try with below sample:
start.yml
# File: start.yml
parameters:
- name: buildSteps # the name of the parameter is buildSteps
type: stepList # data type is StepList
default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
pool: Hosted VS2017
jobs:
- job: secure_buildjob
steps:
- script: echo This happens before code
displayName: 'Base: Pre-build'
- script: echo Building
displayName: 'Base: Build'
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ if ne(pair.value, 'CmdLine#2') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'CmdLine#2') }}:
'${{ pair.value }}': error
- script: echo This happens after code
displayName: 'Base: Signing'
azure-pipelines.yml
trigger:
- master
extends:
template: start.yml
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- script: echo "Script Test"
displayName: Fail
Then you will see the pipeline failed because of the script step detected out:
Note: This validation is, once template detect out there has one CmdLine#2 step was passed from azure-pipeline.yml, it will fail the current pipeline instead of one specified step.