Lock resources between multiple builds that run parallel in Azure DevOps - azure-devops

Suppose I have a build:
Job1:
Task1: Build
Job2:
dependsOn: Job1
Task2: Test
And the test task uses some kind of database, or another unique resource.
I would like to know if it is possible, when multiple builds are running in parallel, to lock Job2 to run unique without other builds trying to access the same resource.
I am using cmake and ctest, so I know I can do something similar between separate unit tests with RESOURCE_LOCK, but I am certain that I will not be able to lock that resource between multiple ctest processes.

Agree with #MarTin's work around. Set one variable with powershell task in Job1, then get this variable and use it in job condition for Job2.
You don't need to use API to add global variable. There has another easier way can for you try: Output variables. We expand one feature that customer can configure one output variable and access this output variable in the next job which is depending on the first job.
The sample of set output variable in Job1:
##vso[task.setvariable variable=firstVar;isOutput=true]Job2 Need skip
Then get the output variable from Job1 in Job2:
Job2Var: $[dependencies.Job1.outputs['outputVars.firstVar']]
Then, in job2, you can use it into job condition:
condition: eq(dependencies.Job1.outputs['outputVars.firstVar'], 'Job2 Need skip')
The completed simple sample script look like this:
jobs:
- job: Job1
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: 'echo "##vso[task.setvariable variable=firstVar;isOutput=true]Need skip Job2"'
name: outputVars
- job: Job2
dependsOn: Job1
variables:
Job2Var: $[dependencies.Job1.outputs['outputVars.firstVar'], 'Need skip Job2']
steps:
...
...
The logic I want to express is dynamically assign values to output variable based on Job1 and the current pipeline execution state. One specific value represent that the Job2 should be locked, which means skip its execution. In Job2's condition expression, when the obtained value $[dependencies.Job1.outputs['outputVars.firstVar'] satisfies the predefined expected value, skip current Job2.

Related

Azure Devops - Run a task only if previous task has run successfully

I am currently writing a pipeline to deploy my infrastructure in Terraform.
I would like that some tasks do not run unless the first task has run successfully. I have found the use of conditions such as success() or failed().
However, upon reading closely, I realize that these conditions are based on Job Status, while in my case there is just one job containing all my tasks.
Is there any way to specify such condition ? And will it be able to refer to a specific task name ? (For example, Task C can run only if Task A ran with success).
You could add a PowerShell task just behind the first task to set a variable (ones for example) by using ##vso[task.setvariable variable=ones]true. Then, use this variable in condition for those tasks that you want them do not run unless the first task has run successfully.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "##vso[task.setvariable variable=ones]true"
Then, set condition: condition: eq(variables.ones, 'true') for each task that you do not want it run unless the first task has run successfully. for example:
- task: PowerShell#2
displayName: task4
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Hello World"
condition: eq(variables.ones, 'true')
The PowerShell task will run if the previous tasks run successfully, in my case, there only is task1. Thus, if the task1 succeed, PowerShell task will set the variable value as true. Next tasks will run when the variable equal true. When the task1 failed, neither PowerShell task will run nor the next tasks.

Conditions on Stage template not working in Azure yml pipeline

I have below main YML pipeline
And below is the template that is being called.
When trying to run the main pipeline is showing the error per below
Agree with Krzysztof Madej.
When you add condition to the stages level, you need to make sure that there is at least one stage that meets the condition at all times.
In addition to adding an empty stage, you could also add the negative condition in the Yaml sample:
For example:
stages:
- ${{if eq(variables['envName'],'sbx')}}:
- template: test.yml
parameters:
buildSteps: test
- ${{if ne(variables['envName'],'sbx')}}:
- stage: Test
jobs:
- job: TestJob
steps:
- script: echo Test
displayName: 'Test Stage'
In this case, if the envName = sbx, it will run the template, or it will run another stage.
It looks that your condition is not met. So envname is not sbx and thus your template is skipped, and you can't run pipeline without any stage. Please make sure you have always at least one stage regardless of the result of the condition, or sth like empty stage for non sbx env.

Can you use build tags in conditions statements in multi-stage devops pipelines

Is it possible to use the build tags set on a multi-stage pipeline build, in the condition section of a later stage?
##### task in build stage #####
- task: YodLabs.VariableTasks.AddTag.AddTag#0
displayName: Adding environment tag to build
inputs:
tags: |
deploy
$(DEPLOY_ENV)
#### some later stage ####
- stage: deploy
displayName: deploy
condition: |
and(
succeeded(),
#Is there something I can put here to condition on tags
)
jobs:
Thanks
From what I know this is not possible with YAML yet, because there is no easy way to get tags availabe in YAML. What you can try is output variable
jobs:
- job: Foo
steps:
- script: |
echo "This is job Foo."
echo "##vso[task.setvariable variable=doThing;isOutput=true]Yes" #The variable doThing is set to true
name: DetermineResult
- job: Bar
dependsOn: Foo
condition: eq(dependencies.Foo.outputs['DetermineResult.doThing'], 'Yes') #map doThing and check if true
steps:
- script: echo "Job Foo ran and doThing is true."
You can also try with workaraund which is in this case:
Fetch tags using REST API in powershell script PUT https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}/tags/{tag}?api-version=5.1
Then assign your tags to output variables using logging commands
And finally use output variable in condition
EDIT
So it looks that this is not possible at the moment but should be available soon. Please check this GitHub issue.
Output variables may now be used across stages in a YAML-based
pipeline. This helps you pass useful information, such as a go/no-go
decision or the ID of a generated output, from one stage to the next.
The result (status) of a previous stage and its jobs is also
available.
Output variables are still produced by steps inside of jobs. Instead
of referring to dependencies.jobName.outputs['stepName.variableName'],
stages refer to
stageDependencies.stageName.jobName.outputs['stepName.variableName'].
Note: by default, each stage in a pipeline depends on the one just
before it in the YAML file. Therefore, each stage can use output
variables from the prior stage. You can alter the dependency graph,
which will also alter which output variables are available. For
instance, if stage 3 needs a variable from stage 1, it will need to
declare an explicit dependency on stage 1.
I tried it now:
stages:
- stage: A
jobs:
- job: JA
steps:
- script: |
echo "This is job Foo."
echo "##vso[task.setvariable variable=doThing;isOutput=true]Yes" #The variable doThing is set to true
name: DetermineResult
# stage B runs if A fails
- stage: B
condition: eq(stageDependencies.A.JA.outputs['DetermineResult.doThing'], 'Yes') #map doThing and check if true
jobs:
- job: JB
steps:
- bash: echo "Hello world stage B first job"
but I got this error:
An error occurred while loading the YAML build pipeline. Unrecognized
value: 'stageDependencies'. Located at position 4 within expression:
eq(stageDependencies.A.JA.outputs['DetermineResult.doThing'], 'Yes').
For more help, refer to https://go.microsoft.com/fwlink/?linkid=842996
However, this feature can be soon with us!

Azure DevOps. Reference Pipeline Settings in PowerShell

I'd like to reference a user who ran pipeline and indicate that a previous specific task in multi-stage pipeline ran successfully or not in a PowerShell script. How can I do that?
Something like:
Write-Host $env:UserThatRanPipeline $env:Task:BuildApp1:SuccessOrFail
So I'd get output:
John Smith Success
I'd like to reference a user who ran pipeline and indicate that a
previous specific task in multi-stage pipeline ran successfully or not
in a PowerShell script.
1.Just as Shayki Abramczyk suggests above, you can use Build.RequestedFor to output the user who runs the pipeline. See predefined variables, you can use something like: Write-Host $(Build.RequestedFor)
2.To get status of your AzureRmWebAppDeployment#4 task, for now there's no predefined variable available to do that job. So you have to do that with some logic...
As a workaround:
You can set one variable SuccessOrFail: 'Succeed' like this in yml:
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
SuccessOrFail: 'Succeed'
And then add one powershell task right after your AzureRmWebAppDeployment#4 task:
- task: PowerShell#2
condition: failed()
inputs:
targetType: 'inline'
script: |
# This script will run only when any previous task failed
echo "##vso[task.setvariable variable=SuccessOrFail]Fail"
3.Make sure there's no custom condition set for your AzureRmWebAppDeployment#4 task. As I know this task is not a independent task, so it will run when all previous tasks succeed.
To sum up:
This ps script will run when AzureRmWebAppDeployment#4 task failed or skipped, and it will reset the value of SuccessOrFail to Fail. So if your AzureRmWebAppDeployment#4 succeeds, the value of $(SuccessOrFail) is Succeed, if it fails or skips, the value would be Fail.
Now the value Succeed represents the task should certainly succeed, and if the value is Fail, it actually represents your task is NotSucceed(Fail+Skip).
The order of your tasks should be: other tasks =>AzureRmWebAppDeployment#4=> PS task above=> other tasks => Your own Powershell Task.

Azure YAML Get variable from a job run in a previous stage

I am creating YAML pipeline in Azure DevOps that consists of two stages.
The first stage (Prerequisites) is responsible for reading the git commit and creates a comma separated variable containing the list of services that has been affected by the commit.
The second stage (Build) is responsible for building and unit testing the project. This Stage consists of many templates, one for each Service. In the template script, the job will check if the relevant Service in in the variable created in the previous stage. If the job finds the Service it will continue to build and test the service. However if it cannot find the service, it will skip that job.
Run.yml:
stages:
- stage: Prerequisites
jobs:
- job: SetBuildQueue
steps:
- task: powershell#2
name: SetBuildQueue
displayName: 'Set.Build.Queue'
inputs:
targetType: inline
script: |
## ... PowerShell script to get changes - working as expected
Write-Host "Build Queue Auto: $global:buildQueueVariable"
Write-Host "##vso[task.setvariable variable=buildQueue;isOutput=true]$global:buildQueueVariable"
- stage: Build
jobs:
- job: StageInitialization
- template: Build.yml
parameters:
projectName: Service001
projectLocation: src/Service001
- template: Build.yml
parameters:
projectName: Service002
projectLocation: src/Service002
Build.yml:
parameters:
projectName: ''
projectLocation: ''
jobs:
- job:
displayName: '${{ parameters.projectName }} - Build'
dependsOn: SetBuildQueue
continueOnError: true
condition: and(succeeded(), contains(dependencies.SetBuildQueue.outputs['SetBuildQueue.buildQueue'], '${{ parameters.projectName }}'))
steps:
- task: NuGetToolInstaller#1
displayName: 'Install Nuget'
Issue:
When the first stages runs it will create a variable called buildQueue which is populated as seen in the console output of the PowerShell script task:
Service001 Changed
Build Queue Auto: Service001;
However when it gets to stage two and it tries to run the build template, when it checks the conditions it returns the following output:
Started: Today at 12:05 PM
Duration: 16m 7s
Evaluating: and(succeeded(), contains(dependencies['SetBuildQueue']['outputs']['SetBuildQueue.buildQueue'], 'STARS.API.Customer.Assessment'))
Expanded: and(True, contains(Null, 'service001'))
Result: False
So my question is how do I set the dependsOn and condition to get the information from the previous stage?
It because you want to access the variable in a different stage from where you defined them. currently, it's impossible, each stage it's a new instance of a fresh agent.
In this blog you can find a workaround that involves writing the variable to disk and then passing it as a file, leveraging pipeline artifacts.
To pass the variable FOO from a job to another one in a different stage:
Create a folder that will contain all variables you want to pass; any folder could work, but something like mkdir -p $(Pipeline.Workspace)/variables might be a good idea.
Write the contents of the variable to a file, for example echo "$FOO" > $(Pipeline.Workspace)/variables/FOO. Even though the name could be anything you’d like, giving the file the same name as the variable might be a good idea.
Publish the $(Pipeline.Workspace)/variables folder as a pipeline artifact named variables
In the second stage, download the variables pipeline artifact
Read each file into a variable, for example FOO=$(cat $(Pipeline.Workspace)/variables/FOO)
Expose the variable in the current job, just like we did in the first example: echo "##vso[task.setvariable variable=FOO]$FOO"
You can then access the variable by expanding it within Azure Pipelines ($(FOO)) or use it as an environmental variable inside a bash script ($FOO).