How to create a multi-stage pipeline depending on a stage/job-name derived from a parameter whereas stages run firstly in parallel and eventually one stage that waits for all previous stages?
Here's what I've tried so far:
A multi-stage pipeline runs for several stages depending on a tool parameter in parallel, whereas dependsOn is passed as parameter. Running it in parallel for each tool waiting for the previous stage for the said tool works smoothly.
Main template: all wait for for all
- ${{ each tool in parameters.Tools }}:
- template: ../stages/all-wait-for-all.yml
parameters:
Tool: ${{ tool }}
stages/all-wait-for-all.yml
parameters:
- name: Tool
type: string
stages:
- stage: ALL_WAIT_${{ parameters.Tool}}
dependsOn:
- PREPARE_STAGE
- OTHER_TEMPLATE_EXECUTED_FOR_ALL_TOOLS_${{ parameters.Tool }}
Now there should be one stage that should only run once and not per tool, but it should only run after the individual tool stages are done. It can't be hardcoded as there are various tools. So I hoped defining the individual wait-stages in a prepare job would work out:
Main template: prepare-stage
- script: |
toolJson=$(echo '${{ convertToJson(parameters.Tools) }}')
tools=$(echo "$toolJson" | jq '.[]' | xargs)
stage="ALL_WAIT"
for tool in $tools; do
stageName="${stage}_${tool }"
stageWaitArray+=($stageName)
done
echo "##vso[task.setvariable variable=WAIT_ON_STAGES]${stageWaitArray}"
echo "##vso[task.setvariable variable=WAIT_ON_STAGES;isOutput=true]${stageWaitArray}"
displayName: "Define wait stages"
name: WaitStage
stages/one-waits-for-all.yml
stages:
- stage: ONE_WAITS
dependsOn:
- $[ stageDependencies.PREPARE_STAGE.PREPARE_JOB.outputs['waitStage.WAIT_ON_STAGES'] ]
whereas below error is shown:
Stage ONE_WAITS depends on unknown stage $[ stageDependencies.PREPARE_STAGE.PREPARE_JOB.outputs['WaitStage.WAIT_ON_STAGES'] ].
As I understand depends on can not have dynamic $[] or macro $() expressions evaluated at runtime. You can use template expressions ${{}} which are evaluated at queue time.
Guess I was overthinking the solution as eventually it was pretty obvious.
So first template can be called within a loop from the main template whereas it's executed as many times as tools we got. Second template shall be called once waiting on previous stages for all tools, whereas the job/stage prefix is known, only the tool name as postfix was unknown. So just add them in a loop directly in dependsOn..
Here you go:
stages:
- stage: ONE_WAITS
dependsOn:
- PREPARE_STAGE
- ${{ each tool in parameters.Tools }}:
- OTHER_TEMPLATE_EXECUTED_FOR_ALL_TOOLS_${{ tool}}
- ALL_WAIT_${{ tool }}
Related
I have a pretty complex setup of my Pipelines in Azure DevOps for various reasons but I'm kind of stuck in a special scenario now. Let me explain a bit.
There is a Stage_A with Job_A setting a Variable_A. Now there is a Stage_B with Job_B, need to use the Variable_A from Stage_A.Job_A.
The variable in Job_A is set by this:
echo ##vso[task.setvariable variable=Variable_A;isOutput=true]$value
Now, Job_B in Stage_B can access the variable in a condition with
variables:
Variable_A_FromStageA: $[stageDependencies.Stage_A.Job_A.outputs['task_A.Variable_A']]
I can also do an echo on the variable by using
echo $(Variable_A_FromStageA)
the Question is now, how can I use this in an if-statement? I tried different approaches:
- ${{ if eq($(Variable_A_FromStageA), 'True') }}:
- ${{ if eq(variables.Variable_A_FromStageA, 'True') }}:
- ${{ if eq(variables['Variable_A_FromStageA'], 'True') }}:
- ${{ if eq(stageDependencies.Stage_A.Job_A.outputs['task_A.Variable_A'], "True") }}:
Nothing actually works. Either the system complains about syntax issues or it doesn't evaluate it correctly. I don't really know how to use the information in my if statement in the yaml file. The documentation is not really clear about it. It only mentions the usage of a stage dependency in a condition and that's it.
Hope anyone can help me here!
Cheers,
Frank
use stageDependencies in if-statements in Azure DevOps yaml
If you mean you want to use conditional insertion to use the variables output from the logging command, then answer is NO.
The reason is the conditional insertion needs compile time value(you must provide them before pipeline run.), but the variable that the logging command output is runtime. Conditional Insertion will be unable to get it.
The right way is to use "condition" instead of "Conditional Insertion". Using condition can achieve your situation.
I write a demo for you as below:
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- stage: A
jobs:
- job: A1
steps:
- bash: echo "##vso[task.setvariable variable=shouldrun;isOutput=true]true"
# or on Windows:
# - script: echo ##vso[task.setvariable variable=shouldrun;isOutput=true]true
name: printvar
- stage: B
condition: and(succeeded(), eq(dependencies.A.outputs['A1.printvar.shouldrun'], 'true'))
variables:
myStageAVar: $[stageDependencies.A.A1.outputs['printvar.shouldrun']]
dependsOn: A
jobs:
- job: B1
steps:
- script: echo $(myStageAVar)
I have an Azure DevOps pipeline which runs npm run based on some runtime parameters.
Is there a possibility to trigger for example 3 jobs in the same pipeline, each of the job with different runtime parameters?
Thank you
Is there a possibility to trigger for example 3 jobs in the same pipeline, each of the job with different runtime parameters?
Yes. You can define the object type parameter and use each expression to iterate over the values.
Here is an example:
parameters:
- name: tests
type: object
default: [test1,test2,test3]
jobs:
- ${{ each test in parameters.tests }}:
- job:
steps:
- script: echo ${{ test }}
Result:
For more detailed info, you can refer to this doc: Loop through parameters
I have an Azure Pipeline azure-pipelines.yml:
parameters:
- name: "stages"
type: object
default:
- stage1
- stage2
# possibly more...
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: InitialStage
displayName: I must run first
jobs: ...
- ${{ each stage in parameters.stages }}:
- stage:
dependsOn: InitialStage
condition: succeeded()
jobs: ...
What I want to achieve is creating a pipeline where the first stage is InitialStage, then I want to generate the next stages to run after it sequentially. So the final pipeline should look like:
InitialStage ==> stage1 ==> stage2 ==> ...
One stage must run after the previous completed and the first stage in absolute must be InitialStage.
If I use the syntax above, I get the stages run in parallel like:
InitialStage ==> stage1
|
==> stage2
|
...
How can I achieve that?
Just need to remove the dependsOn keyword. Per Microsoft Docs:
When you define multiple stages in a pipeline, by default, they run sequentially in the order in which you define them in the YAML file. The exception to this is when you add dependencies. With dependencies, stages run in the order of the dependsOn requirements.
Can also remove the condition keyword:
You can specify the conditions under which each stage runs. By default, a stage runs if it does not depend on any other stage, or if all of the stages that it depends on have completed and succeeded. You can customize this behavior by forcing a stage to run even if a previous stage fails or by specifying a custom condition.
If you do need to map custom stage dependencies, I have done this before as well by creating a stage object. Something like:
parameters:
- name: environmentObjects
type: object
default:
- environmentName: 'dev'
dependsOn: ''
- environmentName: 'tst'
dependsOn: 'dev'
and then looped through and dynamically build the dependsOn. This would allow for scenarios with two test environmnts depending on dev but not each other.
I have below main YML pipeline
And below is the template that is being called.
When trying to run the main pipeline is showing the error per below
Agree with Krzysztof Madej.
When you add condition to the stages level, you need to make sure that there is at least one stage that meets the condition at all times.
In addition to adding an empty stage, you could also add the negative condition in the Yaml sample:
For example:
stages:
- ${{if eq(variables['envName'],'sbx')}}:
- template: test.yml
parameters:
buildSteps: test
- ${{if ne(variables['envName'],'sbx')}}:
- stage: Test
jobs:
- job: TestJob
steps:
- script: echo Test
displayName: 'Test Stage'
In this case, if the envName = sbx, it will run the template, or it will run another stage.
It looks that your condition is not met. So envname is not sbx and thus your template is skipped, and you can't run pipeline without any stage. Please make sure you have always at least one stage regardless of the result of the condition, or sth like empty stage for non sbx env.
I have a scenario where I need to have both:
runtime parameters, so that the pipeline can be triggered manually from the UI, where users triggering it can choose from a predefined set of options (defined in YAML)
variables, so that the pipeline can be invoked via REST APIs
Regarding runtime parameters, I was able to create the following sample pipeline:
parameters:
- name: image
displayName: Pool Image
type: string
default: ubuntu-latest
values:
- windows-latest
- ubuntu-latest
trigger: none
stages:
- stage: A
jobs:
- job: A
steps:
- pwsh: |
echo "This should be triggering against image: $MY_IMAGE_NAME"
env:
MY_IMAGE_NAME: ${{ parameters.image }}
When I run it, I can see the dropdown list where I can choose the image name and it is reflected in the output message of the PowerShell script.
Regarding variables, I have defined one called "image" here (notice the value is empty):
The idea now is to invoke the pipeline from REST APIs and have the image name replaced by the value coming from the variable:
{
"definition": {
"id": 1
},
"sourceBranch": "master",
"parameters": "{\"image\": \"windows-latest\" }"
}
In order to make the step print the value I'm passing here, I need to correct the environment variable in some way. I thought it would be sufficient to write something like:
env:
MY_IMAGE_NAME: ${{ coalesce(variables.image, parameters.image) }}
That's because I want to give the priority to the variables, then to parameters, so that in case none is specified, I always have a default value the pipeline can use.
However, this approach doesn't work, probably because we're dealing with different expansion times for variables, but I don't really know what I should be writing instead (if there is a viable option, of course).
What I also tried is:
env:
MY_IMAGE_NAME: ${{ coalesce($(image), parameters.image) }}
MY_IMAGE_NAME: ${{ coalesce('$(image)', parameters.image) }}
MY_IMAGE_NAME: $[ coalesce(variables.image, parameters.image) ]
MY_IMAGE_NAME: $[ coalesce($(image), parameters.image) ]
None of those are working, so I suspect this may not be feasible at all.
There is a workaround that I'm currently thinking of, which is to create two different pipelines so that those can be invoked independently, but while this is quite easy for me to accomplish, given I'm using a lot of templates, I don't find it the right way to proceed, so I'm open to any suggestion.
I tested and found you might need to define a variable and assign the parameter's value to it (eg. Mimage: ${{parameters.image}}). And define another variable(eg. Vimage) and assign $[coalesce(variables.image, variables.Vimage)] to it. Then refer to $(Vimage) in the env field of powershell task. Please check out below yaml.
parameters:
- name: image
displayName: Pool Image
type: string
default: ubuntu-latest
values:
- windows-latest
- ubuntu-latest
trigger: none
stages:
- stage: A
jobs:
- job: A
variables:
Mimage: ${{parameters.image}}
Vimage: $[coalesce(variables.image, variables.Mimage)]
steps:
- pwsh: |
echo "This should be triggering against image: $env:MY_IMAGE_NAME"
env:
MY_IMAGE_NAME: $(Vimage)
Env field of powershell task is usually for mapping secret variables. You can directly refer to $(Vimage) in the powershell script: echo "This should be triggering against image: $(Vimage).
Note: To queue a build via REST API with provided parameters, you need to check Let users override this value when running this pipeline to make the varilabe to be settable at queue time.
Update:
You can try passing the variables to the parameters of the template to make the parameters for template dynamic. Please check below simple yaml.
jobs:
- template: template.yaml
parameters:
MTimage: ${{parameters.image}}
VTimage: $(Vimage)
template.yaml:
parameters:
MTimage:
VTimage:
jobs:
- job: buildjob
steps:
- powershell: |
echo "${{parameters.VTimage}}"
echo "${{parameters.MTimage}}"