I have a pipeline (Azure Devops yaml) with some jobs.
They run in parallel.
But when one fails, the build ends with a warning and not an error badge.
Why?
I expected the pipeline to end in error.
Edit:
pipelines.yml
pool:
name: Default
jobs:
- job: FirstJob
steps:
- script: echo First!
- template: template.yml
parameters:
name: Second
- template: template.yml
parameters:
name: Third
- template: template.yml
parameters:
name: Fourth
- template: template.yml
parameters:
name: Fifth
- template: template.yml
parameters:
name: Sixth
- template: template.yml
parameters:
name: Seventh
template.yml:
parameters:
- name: name
type: string
default: --
jobs:
- job: ${{ parameters.name }}
dependsOn: FirstJob
continueOnError: true
variables:
BuildTag: ${{ parameters.name }}
steps:
- task : AddTag#0
inputs:
tags: '$(BuildTag)'
- task: DotNetCoreCLI#2
inputs:
command: 'build'
projects: 'foo'
condition: and(succeeded(), eq(variables['BuildTag'], 'Sixth'))
I see my mistake with continueOnError: true.
Azure Devops pipeline finish with warning badge when a job failed
Thanks for your comments and sample, which make me found out the reason for this issue:
The property continueOnError. This behavior is by designed and is not a bug. There is no way to fix it at present.
If we set the continueOnError: true, which will make the future jobs should run even if this job fails. In order to achieve it, Azure Devops will use a "deceptive way" to treat error as warning, so that the error will not prevent the building. That the reason why the job failed, but the pipeline show it as warning.
We could even just reproduce this issue with Control Options Continue on error in non-YAML task:
Besides, it does not affect the completion of the PR.
To resolve this issue, you could comment it in the YAML. If it is necessary for you, you could set the condition: always() for the future jobs:
jobs:
- job: ${{ parameters.name }}
dependsOn: FirstJob
variables:
BuildTag: ${{ parameters.name }}
steps:
...
- job: Second
condition: always()
- job: Third
condition: always()
Hope this helps.
Maybe u need to uncheck "advanced" => "FAIL ON STDERR" (If this option is selected, the build will fail when the remote commands or script write to STDERR).
Related
I am trying to make a pipeline template that takes a JobList a parameter and runs all the jobs, while ensuring that they run on the same agent every time. Basically the approach I've been taking is to try to adapt this answer into a genericized template format.
This is what I have so far, and I've tried a lot of slight tweaks of this with nothing passing the Validate test on the pipeline that calls it.
parameters:
- name: jobsToRun
type: jobList
- name: pool
type: string
default: Default
- name: demands
type: object
default: []
jobs:
- job:
steps:
- script: echo "##vso[task.setvariable variable=agentName;isOutput=true;]$(Agent.Name)"
pool:
name: ${{ parameters.pool }}
demands:
- ${{ each demand in parameters.demands }}:
${{ demand }}
- ${{ each j in parameters.jobsToRun }}:
${{ each pair in j }}:
${{ pair.key }} : ${{ pair.value }}
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
What am I doing wrong here? It seems like it should be possible if that answer I reference is correct, but it seems like I'm just a bit off.
Name missing on the job.., example below.
- job: 'test-Name'
Steps need a associated job and pool to run is declared inside
jobsToRun:
- job: sample_job1
displayName: "sample_job1"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"
On this bottom pool declaration...
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
i am not sure but i have tried this many times but i think this can't be included separate to the job since each individual job is passed as parameter. pool definition needs to be inside the job or inside the job template if you are using templates..
example:
jobsToRun:
- job: output_message_job1
displayName: "in pipe Output Message Job"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"
Looking to try and get some help figuring out a solution to what seems like a simple task. I am trying to develop some extend YAML templates for security within our ADO pipelines but first I have to get passed this error I am experiencing.
The error being returned by ADO when I try to validate or run the pipeline is
/ADO_Stage_Restrictions_Dev.yml#AdoRestrictions (Line: 7, Col: 3): Unexpected value job
Working Extend Template YAML
This template validates and executes without issue, which to me means I am successfully passing the stages object into the extends template
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
${{ stage }}
Broken Extend Template YAML
This template does not validate and throws the 'Unexpected value job' exception. based on the stage schema I would assume that I would be able to loop the jobs property within the stage.
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- ${{ each job in stage.jobs }}:
${{ job }}
Build YAML
The main yaml file that extends stages
resources:
repositories:
- repository: self
type: git
ref: refs/heads/Development
- repository: AdoRestrictions
type: git
name: utl-yaml-templates
ref: refs/heads/main
trigger: none
pool:
name: PROD
extends:
template: ADO_Stage_Restrictions_Dev.yml#AdoRestrictions
parameters:
stageObjs:
- stage: 'BuildStage'
displayName: 'Build Test'
jobs:
- job: 'BuildJob'
displayName: 'Build'
steps:
- task: PowerShell#2
displayName: 'Hello World'
inputs:
targetType: inline
script: |
Write-Host "Hello World"
There's nothing in your yaml which defines the start of the stage, or declares the start of the list of jobs; that's why it wasn't expecting to find 'job' there. You can add those parts in, like this:
parameters:
- name: stageObjs
type: stageList
default: []
stages:
- ${{ each stage in parameters.stageObjs }}:
- stage: ${{ stage.stage }}
displayName: ${{ stage.displayName }}
jobs:
- ${{ each job in stage.jobs }}:
${{ job }}
Problem Description
I was having some problems trying to use variables created in one stage in another stage and managed to find various articles old and new describing how this can be done. The more recent articles/posts identifying the new syntax
$[stageDependencies.{stageName}.{jobName}.outputs['{stepName}.{variableName}']
Used like this:
variables:
myVariable: $[stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage']]
This works great until you needed to use job templates.
None of the samples I found online covered the situation of templates. They just demonstrated how multiple stages in the same yaml file could obtain the value.
The syntax depends on being able to put the expression into a variable. Unfortunately, when you use a template for a job, it's not possible to declare variables and passing it as a parameter results in it being unevaluated.
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
In this snippet, it comes through as-is. The value does not get substituted in.
Theoretically, you could set your job to have the expression present in the variables block but that sort of hard-coding undermines one of the main benefits of templates.
Related Articles
Share variables across stages in Azure DevOps Pipelines
Azure DevOps Release Notes - sprint 168
I know the question asks about template jobs, but for future reference I want to describe how it can be achieved with template stages as well.
It is done with a variable workaround as in the accepted answer, and with a reference to stagedependencies when no dependsOn exists. (Templates don't allow dependsOn). Somehow, this still works.
Example YAML using stage template (I have modified the code from the accepted answer):
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
# stage template cannot use dependsOn, but is still allowed to refer to stagedependencies...
- template: templates/stage-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
Stage template:
parameters:
- name: ValueToOutput
type: string
stages:
- stage: ShowStageVarStage
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
Solution
The answer isn't actually far away. The original expression just need to be passed through a variable in the template job. Basically, set a variable to be the value of the parameter and use the macro syntax to evaluate the variable.
parameters:
- name: ValueToOutput
type: string
...
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
Using the macro syntax of $(LocalVarOfValueToOutputParam) will result in the value making its way into the template job correctly.
Example
If we have a yaml file for the build definition:
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
That uses the job template templates/job-showstagevars.yml
parameters:
- name: ValueToOutput
type: string
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
What we get in our output of the second stage is:
ValueToOutput parameter=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (pre-processor syntax)=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (macro syntax)=This is the value from the first stage
I am attempting to use the if condition to execute a block of operations but it doesn't seem to be working for me. I'm not entirely sure if I understand the if condition correctly, but this is how I have been using it - or at least a simplified version of it.
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
name: MyTest-$(date:yyyyMMdd)$(rev:.r)
trigger: none
resources:
- repo: self
clean: false
fetchDepth: 2
variables:
- group: TestVG
- name: EnableSigning
value: true
- name: SignDebugBuilds
value: false
stages:
- stage: Stage_Build
displayName: Build Stage
dependsOn: []
jobs:
- job: Job_Build
displayName: Build Job
condition: succeeded()
dependsOn: []
pool:
vmImage: 'windows-latest'
variables:
BuildPlatform: AnyCPU
strategy:
matrix:
Debug:
BuildConfiguration: Debug
Retail:
BuildConfiguration: Retail
steps:
- template: param-test.yml
parameters:
Value: 'Build Solution 1'
#- script: echo 'Build Solution 1'
- ${{ if and(eq(variables['EnableSigning'], True), or(ne(variables['BuildConfiguration'], 'Debug'), eq(variables['SignDebugBuilds'], True))) }}:
#- script: echo 'Build Solution 2 IF check'
- template: param-test.yml
parameters:
Value: 'Build Solution 2 IF check'
- script: echo 'Build Solution 2 COND check'
condition: and(eq(variables['EnableSigning'], True), or(ne(variables['BuildConfiguration'], 'Debug'), eq(variables['SignDebugBuilds'], True)))
The param-test.yml file is a simple file to echo a line for logging/debugging purposes.
parameters:
Value: string
steps:
- script: echo '${{ parameters.Value }}'
My expectation was that for the debug build I would see the first log and the other 2 logs would get skipped, while all 3 logs would be printed for retail build.
However, what I see is that for the debug build, the first and second logs are printed, while only the 3rd log is skipped (the one with the explicit condition in the script step), while all 3 logs are printed for retail build.
To clarify a few items, I used the 3rd log as a control to check if the condition was correct or not. And this is a highly simplified version of the actual pipeline that I have in our repo. We have a primary pipeline YAML file where all the variables are declared including the EnableSigning and SignDebugBuilds. This calls a template file passing in the EnableSigning and SignDebugBuilds flags to the same. This happens a few times around with subtemplates, till one of the sub-templates, the one responsible for the build, uses these from the parameters in the if check. In the actual pipeline, I have the condition use parameters instead of variables, but the result is still the same.
I have looked at a few documentation but it wasn't very clear as to what we could expect from the if statement. And since template tags don't support an explicit condition that we can pass in, this seems to be the only option aside from maintaining 2 separate versions of the template files corresponding to the flags
In your current situation, the 'if' keyword cannot get the variables['BuildConfiguration'], because this variable is created when the job is running. And 'if' key word needs to use the runtime parameters.
So, every time the job is running, the result of the variables['BuildConfiguration'] under the 'if' keyword is NULL. Because when the job is init, this variable has not been created. This variable will be created when the job is running and then the pipeline will help you create other two jobs under your job.
At present the work around is to split the matrix to different two jobs and use the parameters to instead of the variables.
Here is the demo I create:
name: MyTest-$(date:yyyyMMdd)$(rev:.r)
trigger: none
resources:
- repo: self
clean: false
fetchDepth: 2
parameters:
- name: EnableSigning
type: boolean
displayName: 'EnableSigning'
default: true
- name: SignDebugBuilds
type: boolean
displayName: 'SignDebugBuilds'
default: false
variables:
# - group: TestVG
# - name: EnableSigning
# value: true
# - name: SignDebugBuilds
# value: false
- name: system.debug
value: true
stages:
- stage: Stage_Build
displayName: Build Stage
dependsOn: []
jobs:
- job: Build_Job_Debug
pool:
# vmImage: 'windows-latest'
name: default
variables:
BuildPlatform: AnyCPU
BuildConfiguration: Debug
steps:
- template: param-test.yml
parameters:
Value: 'Build Solution 1'
- ${{ if and(eq(parameters.EnableSigning, true), eq(parameters.SignDebugBuilds, true))}}:
- template: param-test.yml
parameters:
Value: 'Build Solution 2 IF check'
- script: echo 'Build Solution 2 COND check and BuildConfiguration is $(BuildConfiguration)'
name: 'CMD3_check'
condition: eq('${{ parameters.EnableSigning }}', true)
- job: Build_Job_Retail
pool:
# vmImage: 'windows-latest'
name: default
variables:
BuildPlatform: AnyCPU
BuildConfiguration: Retail
steps:
- template: param-test.yml
parameters:
Value: 'Build Solution 1'
- ${{ if or(eq(parameters.EnableSigning, true), eq(parameters.SignDebugBuilds, false))}}:
- template: param-test.yml
parameters:
Value: 'Build Solution 2 IF check'
- script: echo 'Build Solution 2 COND check and BuildConfiguration is $(BuildConfiguration)'
name: 'CMD3_check'
condition: eq('${{ parameters.EnableSigning }}', true)
Also, if you need to use the variable BuildConfiguration, you can also define it under the different job.
Is there a way to reference a job parameter's name in an Azure DevOps YAML template? I know that I could pass in the job name as its own string parameter, but I was hoping for something that's a little less clumsy.
template.yml
parameters:
- name: MyJob
type: job
jobs:
- ${{ parameters.MyJob }}
- job: Job2
dependsOn: # How to make this depend on MyJob?
azure-pipelines.yml
stages:
- stage: Stage1
jobs:
- template: template.yml
parameters:
MyJob:
job: SomeJobName
steps:
- script: echo Hello
I tried accessing ${{ parameters.MyJob.name }} but it doesn't appear to exist.
I've figured out, though it took a lot of trial and error to get the exact syntax and spacing right. The docs aren't super clear about spacing and expressions, and when to lead with a dash.
parameters:
- name: MyJob
type: job
jobs:
- ${{ parameters.MyJob }}
- job: Job2
${{ each pair in parameters.MyJob }}:
${{ if eq(pair.key, 'job') }}:
dependsOn: ${{ pair.value }}
steps:
...
Can't comment, but as you see in previous answer, the property is not "name", but "job". You don't need the foreach then:
parameters:
- name: MyJob
type: job
jobs:
- ${{ parameters.MyJob }}
- job: Job2
dependsOn: ${{ parameters.MyJob.job }}
steps:
...