the extend template example shared by MSFT in documentation suppose to fail the step but it's not failing it.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#extend-from-a-template
can anyone share a working example of this.
I want to to validate user yaml in extend template and fail the build in case of steps used that are not allowed as per firm policies.
The document here provides the confused direction. You can try to download your build log which does not action like you expect, then analyze the initializeLog.txt file contents. All of key evaluating expression are list there.
Try with below sample:
start.yml
# File: start.yml
parameters:
- name: buildSteps # the name of the parameter is buildSteps
type: stepList # data type is StepList
default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
pool: Hosted VS2017
jobs:
- job: secure_buildjob
steps:
- script: echo This happens before code
displayName: 'Base: Pre-build'
- script: echo Building
displayName: 'Base: Build'
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ if ne(pair.value, 'CmdLine#2') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'CmdLine#2') }}:
'${{ pair.value }}': error
- script: echo This happens after code
displayName: 'Base: Signing'
azure-pipelines.yml
trigger:
- master
extends:
template: start.yml
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- script: echo "Script Test"
displayName: Fail
Then you will see the pipeline failed because of the script step detected out:
Note: This validation is, once template detect out there has one CmdLine#2 step was passed from azure-pipeline.yml, it will fail the current pipeline instead of one specified step.
Related
I am trying to select a service connection based on if else logic.
I have following two pipelines azure-caller.yml and template.yaml.
azure-caller.yml
parameters:
- name: directory
displayName: 'select appropriate environment directory'
values:
- Dev
- Stage
stages:
- template: job-templates/template.yml
parameters:
selectdirectory: $(select-directory)
template.yml
parameters:
selectdirectory: default
stages:
- stage: A
variables:
- name: service-connection
${{ if eq(parameters.selectdirectory, 'Dev') }}:
value: Dev-Service-connection
${{ elseif eq(parameters.selectdirectory, 'Stage') }}:
value: Stage-Service-connection
jobs:
- job: Install_terraform
displayName: Intialize Terraform
continueOnError: true
steps:
- checkout: self
- task: TerraformInstaller#0
inputs:
terraformVersion: '1.0.5'
- job: Validate_terraform_Plan
displayName: Validate Terraform
continueOnError: true
dependsOn: Install_terraform
steps:
- checkout: self
- task: TerraformTaskV2#2
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: ${{ variables.service-connection }}
backendAzureRmResourceGroupName: 'any-rg'
backendAzureRmStorageAccountName: 'any-storage'
backendAzureRmContainerName: 'statefile'
backendAzureRmKey: abcd.tfstate
- task: TerraformTaskV2#2
displayName: 'plan'
inputs:
provider: 'azurerm'
command: 'plan'
commandOptions: '-out="plan.out"'
environmentServiceNameAzureRM: ${{ variables.service-connection }}
My tasks are failing because it fetches the value of the service connection at compile time, which is empty.
Azure Caller is a wrapper/main pipeline, and the template is the only one that is allowed to use Dev or Stage service connections for deployment purposes to achieve security. Is there a way to accomplish this?
Your input will be helpful. thx
It is not supported to dynamically specify value of pipeline variable conditionally.
In the YMAL pipeline, you can use the if conditional to judge using or not using a pipeline variable. You can not use it to judge specifying a value or another one to a pipeline variable.
However, you can dynamically set the input conditionally on a task. See "Conditionally set a task input".
You also can dynamically specify value of parameter conditionally.
Below is an example to dynamically specify value of parameter conditionally. You can reference it to change your YAML files.
template.yml
parameters:
- name: ServiceConnection
type: string
stages:
- stage: A
displayName: 'Stage A'
jobs:
- job: A1
displayName: 'Job A1'
pool:
vmImage: ubuntu-latest
steps:
- task: Bash#3
displayName: 'Show parameter value'
inputs:
targetType: inline
script: echo "The value of ServiceConnection is ${{ parameters.ServiceConnection }}"
azure-pipelines.yml
parameters:
- name: directory
displayName: 'Select appropriate environment directory:'
type: string
values:
- Dev
- Stage
stages:
- template: template.yml
parameters:
${{ if eq(parameters.directory, 'Dev') }}:
ServiceConnection: Dev-Service-connection
${{ if eq(parameters.directory, 'Stage') }}:
ServiceConnection: Stage-Service-connection
Result
[UPDATE]
Dynamically set the input conditionally on the task.
template.yml
parameters:
- name: selectdirectory
type: string
stages:
- stage: A
displayName: 'Stage A'
jobs:
- job: Validate_terraform_Plan
displayName: 'Validate Terraform'
. . .
steps:
. . .
- task: TerraformTaskV2#2
inputs:
provider: 'azurerm'
command: 'init'
${{ if eq(parameters.selectdirectory, 'Dev') }}:
backendServiceArm: Dev-Service-connection
${{ if eq(parameters.selectdirectory, 'Stage') }}:
backendServiceArm: Stage-Service-connection
backendAzureRmResourceGroupName: 'any-rg'
backendAzureRmStorageAccountName: 'any-storage'
backendAzureRmContainerName: 'statefile'
backendAzureRmKey: abcd.tfstate
. . .
azure-pipelines.yml
parameters:
- name: directory
displayName: 'Select appropriate environment directory:'
type: string
values:
- Dev
- Stage
stages:
- template: template.yml
parameters:
selectdirectory: ${{ parameters.directory }}
Have you checked this answer out: AzureDevops: Can we pass dynamic value as service connection in yaml
Essentially rather then pass in your service connection name via an if statement have the service connection name stored in a variable.yml file that is determined off your selected directory and will load the appropriate service connection name.
We have two environments defined, "stage" and "prod". Prod requires approval for deploying, where as stage does not.
What I'm trying to achieve is that if the pipeline runs against the master branch, it should use the prod environment, and all other branches should use the stage environment.
Problem is that I can't seem to be able to change the environment in runtime. I've also tried using templates, which sort of works, except that I can't pass runtime information as parameters to the template.
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
I will start with first last one:
I also tried using two deployment jobs, one for each environment and then chose job based on conditions, however, as soon as the pipeline involves the prod environment, whether it's going to be used or not, it will ask for approval.
Resource restriction are evaluated on stage level so if you put those deployment jobs on the same stage (or you didn't define a stage - then you have implicit stage) - this is the reason why it asks for approval. If you move deployment jobs to seprate stages you won't be asked for approval. Please keep in mind to move condition to a stage level.
Another apporach it would be conditional for envrionment as follows:
jobs:
- deployment: DeployWeb
displayName: deploy Web App
${{ if eq(variables['Build.SourceBranchName'], 'master') }}:
environment: 'Dev'
${{ if ne(variables['Build.SourceBranchName'], 'master') }}:
environment: 'campo-dev'
strategy:
# Default deployment strategy, more coming...
runOnce:
deploy:
steps:
- checkout: self
- script: echo my first deployment
I do this with an extends yaml. You can define your stages via the environment's parameter (defaulted) and then your calling yaml extends it and passes tasks for build and deploy. The build section is ran once. And the deploy section will loop for each environment.
The approvals comes from the Environment approvals setup in Devops
#Extends Yaml
parameters:
- name: buildSteps
type: stepList
default: []
- name: deploySteps
type: stepList
default: []
- name: environments
type: object
default:
- name: Stage
branch: any
dependsOn:
- Builds
- name: Prod
branch: any
dependsOn:
- Stage
stages:
- stage: Builds
jobs:
- job: Building
steps:
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
- ${{ each env in parameters.environments }}:
- stage: ${{ env.name }}
dependsOn: ${{ env.dependsOn }}
condition: and(succeeded(), or(startsWith(variables['Build.SourceBranch'], 'refs/heads/${{ env.branch }}'), eq('${{ env.branch }}', 'any')))
jobs:
- deployment: Deploy${{ env.name }}
displayName: Deploy to ${{ env.name }}
variables:
- group: ENV-${{ env.name }}
environment: ${{ env.name }}
strategy:
runOnce:
deploy:
steps:
- ${{ each step in parameters.deploySteps }}:
- ${{ each pair in step }}:
${{ pair.key }}: ${{ pair.value }}
#Calling Yaml
#*Trimmed* header stuff
resources:
repositories:
- repository: cicd
type: git
name: SharedRepo
ref: main
extends:
template: extends-deploy.yaml#cicd
parameters:
buildSteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Build Stuff 2'
deploySteps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 1'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host 'Deploy Stuff 2'
I'm working on a project where we have 2 pipelines. 1 pipeline is currently working as expected having 8 stages in it.
Now I want to write code for 2 pipelines where I want to utilize few stages (approx. 4 stages) from the 1 pipeline because code and functionality are similar.
Is there any way I can achieve this in the Azure DevOps YAML pipeline?
Sure, you can export similar stages to a template, then you can use it in other pipelines with extends:
# File: azure-pipelines.yml
trigger:
- master
extends:
template: start.yml
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- task: CmdLine#2
displayName: Test 3 - Will Fail
inputs:
script: echo "Script Test"
The template will be (for example):
# File: start.yml
parameters:
- name: buildSteps # the name of the parameter is buildSteps
type: stepList # data type is StepList
default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
pool: Hosted VS2017
jobs:
- job: secure_buildjob
steps:
- script: echo This happens before code
displayName: 'Base: Pre-build'
- script: echo Building
displayName: 'Base: Build'
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ if ne(pair.value, 'CmdLine#2') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'CmdLine#2') }}:
'${{ pair.value }}': error
- script: echo This happens after code
displayName: 'Base: Signing'
this is developer repo pipeline
resources:
repositories:
- repository: extendCheck
type: git
name: PUL/extendCheck
trigger:
- none
extends:
template: base2.yml#extendCheck
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- script: echo "Script Test"
displayName: Fail
this is extend template from centralized policy team's repo
# File: start.yml
name: $(Date:yyyy)$(Date:.MM)$(Date:.dd)$(Rev:-r)
parameters:
- name: buildSteps # the name of the parameter is buildSteps
type: stepList # data type is StepList
default: [] # default value of buildSteps
stages:
- stage: secure_buildstage
pool:
vmImage: 'ubuntu-latest'
jobs:
- job: secure_buildjob
steps:
- script: echo This happens before code
displayName: 'Base: Pre-build'
- script: echo Building
displayName: 'Base: Build'
- ${{ each step in parameters.buildSteps }}:
- ${{ each pair in step }}:
${{ if ne(pair.key, 'script') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.key, 'script') }}: # checks for buildStep with script
'Rejecting Script: ${{ pair.value }}': error # rejects buildStep when script is found
- script: echo This happens after code
displayName: 'Base: Signing'
When i run this , i get following error
/base2.yml#extendCheck (Line: 2, Col: 1): Unexpected value 'name'
We want to control the build number format and so don't want to keep it in developer repo.
Any suggestions for this problem
Unfortunately, it is not supported using a template for name. To work around this issue, we can use a script to define the variable and use the UpdateBuildNumber command to update the build numbering format. For example, add the following task in the end of base2.yml file:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
[string] $dateTime = (Get-Date -Format 'yyyyMMddTHHmmss')
Write-Host "##vso[task.setvariable variable=name]$dateTime"
- script: echo "##vso[build.updatebuildnumber]$(name)"
I'm trying to pass stage list from azure pipeline shown below
# File: azure-pipelines.yml
trigger:
- master
extends:
template: start_stage.yml
parameters:
cdstages:
- stage: secure_buildstage
pool: Hosted VS2017
jobs:
- job: secure_buildjob
steps:
- bash: echo This happens before code
displayName: 'Base: Pre-build'
- bash: echo Building
displayName: 'Base: Build'
- bash: echo This happens after code
displayName: 'Base: Signing'
- stage: secure_deploystage
pool: Hosted VS2017
jobs:
- job: secure_deployjob
steps:
- bash: echo This happens before code
displayName: 'Base: Pre-build'
- bash: echo Building
displayName: 'Base: Build'
- script: echo This happens after code
displayName: 'Base: Signing'
to extend template shown below
parameters:
- name: cdstages # the name of the parameter is buildSteps
type: stageList # data type is StepList
default: [] # default value of buildSteps
stages:
- ${{ each stage in parameters.cdstages }}:
- ${{ each job in stage.jobs }}:
- ${{ each step in job.steps }}:
- ${{ each pair in step }}:
${{ if ne(pair.value, 'CmdLine#2') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'CmdLine#2') }}:
'${{ pair.value }}': error
The goal is to take stage list and validate if users are only running steps approved by firm's compliance team.
I'm getting error
[][1
not sure why getting "task" error, there are no task keywords used anywhere.
Any help?
Based on my test , it seems that the stagelist in start_stage.yml doesn't support to add eachdirective to get deeper content (e.g. job and steps).
When you use the stagelist, it could get the stage and use it for comparison.
For example:
parameters:
- name: cdstages # the name of the parameter is buildSteps
type: stageList # data type is StepList
default: [] # default value of buildSteps
stages:
- ${{ each stage in parameters.cdstages }}:
- ${{ each pair in stage }}:
${{ if ne(pair.value, 'abc') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'abc') }}:
'${{ pair.value }}': error
This Yaml template could work.
But when I add the each directive behind the stage to get the jobs. The jobs are is not available.
parameters:
- name: cdstages
type: stageList
default: []
stages:
- ${{ each stage in parameters.cdstages }}:
- ${{ each job in stage.jobs }}:
- ${{ each pair in job }}:
${{ if ne(pair.value, 'abc') }}:
${{ pair.key }}: ${{ pair.value }}
${{ if eq(pair.value, 'abc') }}:
'${{ pair.value }}': error
According to your requirements, you need to get the build step and use it for comparison.
You could try to directly use the steplist type.
Here is an example about steplist, you could refer to it.
Hope this helps.