I have a pipeline that executes every hour and sometimes the execution takes more than an hour and the other execution starts, is there a way to prevent this? and for example the new execution to get queued?
thank you for all the help
It seems you have multiple build agents. Assuming you are using self-hosted build agents, you could specify certain demands of the agent to use only one agent. In this way, if the agent is not free, the build will keep waiting. To use a particular agent, add a demand of Agent.Name equals agentname, check the screenshot below. Agent name can be found in capabilities of the agent.
pool:
name: MyPool
demands:
- myCustomCapability # check for existence of capability
- agent.name -equals agentname # check for specific string in capability
Another way is triggering the pipeline via REST api and through the PowerShell task. You could use the REST API Builds - List to get the detailed build info and check the latest build status:
https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitions}&api-version=6.0
In the YAML, we could add a powershell task to get the build status, like:
- task: PowerShell#2
inputs:
targetType : inline
script: |
$url = "https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitionID}&api-version=6.0"
$connectionToken="Your PAT Here"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$buildPipeline= Invoke-RestMethod -Uri $url -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
$BuildStatus= $buildPipeline.value.status | Select-Object -first 1
Write-Host This is Build Status: $BuildStatus
This list all the build status for the specify definitions, then use Select-Object -first 1 to get the latest build status. If the status is completed, then queue the build. If the status is not completed, do not queue the build.
Related
I would like to identify the build policy for a build that was run by clicking the Queue (or Re-queue) button against a required/optional check from within a pull request. I wish to identify the policy programmatically from within a pipeline; e.g. a script task. Open to any approach, been exploring the az CLI but no luck thus far.
I've setup two build policies against a branch that both target the same build definition - Policy A and Policy B. Both are setup to be run manually - A is required, B is optional. Both will surface in the UI for a pull request as checks - A being required, B being optional. When a build is run by clicking the Queue (or Re-queue) button against either check, I would like to be able to identify which of the two policies the run was initiated from, (which policy provided the Queue or Re-queue button that was clicked).
EDIT: A bit more background on what I'm doing ...
I've got a single pipeline for building an application.
I've recently got a request to update the pipeline to support publishing to Chromatic.
I've added a Publish to Chromatic parameter to the pipeline and a task to push to Chromatic when the parameter is set to true.
I received a subsequent request to make it easier to publish changes from a feature branch to Chromatic. One engineer threw out the idea of having an optional check available in pull requests to give a single button click experience.
While researching my options, I was wondering if it would be possible to enhance the existing pipeline to set the Publish to Chromatic parameter to true during a run. I found this comment on Reddit which ultimately led to me posting here ...
set a default for your parameter (I like to use 'auto') add a script >task near the beginning that reads the pull request comment and sets a variable for you to use in later logic if the parameter is auto . you can even condition this to only run on a PR.
I am aware that I could create a separate pipeline for publishing to Chromatic instead of updating the existing one; that's one of a few options I have. At this point, I'm more-so curious whether or not this particular approach is technically feasible even if I opt not to go forward with it.
Hope that adds some clarity!
The policy that queued the pipeline isn't something that is visible to the pipeline as a pipeline variable. In fact, there doesn't seem to be any indication if the PullRequest was queued manually or automatically.
There might be a few other ways to approach this...
I would start by putting a publishChromatic parameter in the pipeline and then building up conditions in the pipeline execution around this variable. By default, let's assume that the value is false so that if you're manually queueing a pipeline run you can opt-in.
triggers:
- include:
branches:
- develop
parameters:
- name: publishChromatic
displayName: 'Publish build to Chromatic'
type: boolean
default: false
jobs:
- job: Build
variables:
publishChromatic: ${{ parameters.publishChromatic }}
steps:
... pre-execution steps
- task: CmdLine#2
displayName: Publish to Chromatic
condition: and(succeeded(), eq(variables['publishChromatic'], 'true'))
inputs:
script: npx chromatic --project-token=$(CHROMATIC_PROJECT_TOKEN) --branch=$(Build.SourceBranch)
... post execution steps
Option 1: Pull Request Labels
One option might be to inspect the pull request for the presence of a label as outlined in this answer. As a pre-execution step, a simple script could flip the flag when the label is present:
- pwsh: |
$urlFormat = "{0}/{1}/_apis/git/repositories/{1}/pullRequests/{2}/labels?api-version=6.0-preview.1"
$url = $urlFormat -f `
$env:SYSTEM_TEAMFOUNDATIONSERVERURI, `
$env:SYSTEM_TEAMPROJECTID, `
$env:BUILD_REPOSITORY_NAME, `
$env:SYSTEM_PULLREQUEST_PULLREQUESTID
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$response = Invoke-RestMethod -Uri $url -Method Get -Headers $headers
$labels = $response.value.name
Write-Host "##vso[task.setvariable variable=PullRequestTag]$labels"
displayName: 'Fetch Pull Request Labels'
condition: and( succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
- pwsh: |
if ("$(PullRequestLabels)" -like "*chromatic*") {
Write-Host "##vso[task.setvariable variable=publishChromatic]true"
}
condition: and( succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
displayName: 'Check for Chromatic label'
I like this option in that it provides a bit of traceability for which Pull Requests were deployed. Unfortunately, there's no way to queue a build automatically when the PR labels are modified so you'd need to have the tag on the PR before triggering the pipeline.
You could also establish a different pattern such as triggering based on a convention like a value that appears in the name of the Pull Request, etc.
Option 2: Pipeline to Trigger Chromatic
If you'd rather have a Build Validation option labeled 'Deploy to Chromatic' to automate triggering your deployment to Chromatic, a simple option would be to create a pipeline that triggers your pipeline with the publishChromatic parameter.
trigger: none
steps:
- checkout: none
- pwsh: |
$pipelineId = 1234
$urlFormat = "{0}/{1}/_apis/pipelines/{2}/runs?api-version=6.0-preview.1
$url = $urlFormat -f `
$env:SYSTEM_TEAMFOUNDATIONSERVERURI, `
$env:SYSTEM_TEAMPROJECTID `
$pipelineId
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$body = #{
resources = #{ repositories = #{ self = #{ refName = "$(Build.SourceBranch)" } } }
variables = #{
originalPrId = #{
value = "$(System.PullRequest.PullRequestId)
}
}
templateParameters = #{
publishChromatic = $true
}
}
Invoke-RestMethod -Uri $url -Method Post -Body $body -Headers $headers
displayName: 'Trigger Chromatic Pipeline'
condition: eq(variables['Build.Reason'],'PullRequest')
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
This simple script performs a fire-and-forget approach to triggering your original pipeline.
If you need to have a successful deployment to Chromatic as part of your PR, you could adjust the original pipeline to report a pull-request status.
In your original pipeline, add the following as a post-execution step:
- pwsh: |
$urlFormat = "{0}/{1}/_apis/git/repositories/{2}/pullRequests/{3}/statuses?api-version=6.0-preview.1
$url = $urlFormat -f `
$env:SYSTEM_TEAMFOUNDATIONSERVERURI, `
$env:SYSTEM_TEAMPROJECTID, `
$env:BUILD_REPOSITORY_NAME, `
"$(originalPrId)"
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$body = #{
status = "succeeded"
description = "completed chromatic regression"
context = #{
name = "qualitygate/chromatic"
}
targetUrl = "http://chromatic.com/your/buildid"
}
Invoke-RestMethod -Uri $url -Method POST -Body $body -Headers $headers
displayName: Report status to PR
condition: and( succeeded(), ne(variables['originalPrId'],''))
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
To require a successful chromatic quality gate, add a Status Check to your Branch Policy with the same name mentioned above qualitygate/chromatic.
Option 3: Further down the Rabbit hole
You can establish even deeper integration with Chromatic by building a custom extension that allows you to add specialized menus to the Pull Request checks menu. The custom menu could include javascript-enabled buttons to trigger your pipeline without the need for a custom pipeline mentioned in Option 2.
While not necessarily dependent on writing a custom extension, you could also create an Azure Function App that listens for webhooks from Chromatic and posts status updates back to your PR with the a custom UI that links back to the Chromatic build. You'd simply need to query the Azure DevOps API to map the branch name in the Chromatic payload to the corresponding PR.
I have a pipeline with a number of stages, e.g.:
Canary -> Dev -> UAT -> Prod
For each stage we have approvals set so a team member has to allow a build to proceed to the next stage. Sometimes we have long periods between stages being approved (e.g. someone needs to do some testing and feedback outside of devops we can proceed), therefore it can be hard to remember we have an old run pending approval unless we specifically check - which is prone to human error.
But if I have one build say queued at UAT awaiting push to production how can I prevent a second run which includes the changes from the first run (as they are both running off master) being allowed to proceeded.
E.g. I need newer jobs to be aware of older runs still pending approval to proceed.
Thanks!
There is no default way to prevent newer runs if old one queued, but you can try the following workaround:
You could add a script in the pipeline to check state of the latest pipeline run. If the state is inProgress, cancel the build. If the state is completed, proceed the build.
You should use Runs - List api to get state of the latest pipeline run. The script looks like:
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
$uri = "https://dev.azure.com/$org/$projectName/_apis/pipelines/$pipelineid/runs?api-version=6.0-preview.1"
$result = Invoke-RestMethod -Uri $uri -Method Get -Body $json -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
$latestrun = $result.value.state[0]
Write-Host "$latestrun"
Then you could refer the following case to cancel the build if $latestrun is inProgress:
Is it possible to cancel a Azure DevOps pipeline Job programmatically?
I'm working on a release pipeline there are around 3 task in the 1 Agent. taskA,taskB,taskC
I want to run specific task based on specific task failed. I tried custom condition but it didn't satisfied my case.
I want task C to be execute when only task B is failed for that I'm using output variable as well. In this case its working only for taskB. When TaskA failed Task B will skip and output variable become null in that case my task C is execute which is not correct.
I'm trying to make a condition which fulfill both TaskA and TaskB condition.
if TaskA failed --> TaskC should not run
if TaskB failed --> TaskC should run.
Here is my condition:-> and(eq(Agent.JobStatus, 'failed'), in(variables['oneboxout.oneboxvar'],'False'))
Is there any way we can get task name only so my work would be easier.
Below are my task screenshot for your reference.
Is there any way we can get task name only so my work would be easier.
The answer is yes.
We could use the REST API Releases - Get Release to get the task name or status；
https://learn.microsoft.com/en-us/rest/api/azure/devops/release/releases/get%20release?view=azure-devops-rest-6.0
As the above REST API, we need provide the current release Id to that REST API. To resolve this, we could use the REST API Releases - List with Parameter definitionId={definitionId} and powershell parameter Select-Object -first 1 to get the current release Id.
To resolve this request, I would like use following method:
Summary:
Add powershell a task (Let's call it Get JobB task result)between JobB and JobC to invoke REST API
to get the result of the JobB with condition Even if a previous task has failed, unless the build was canceled.
Set a variable RunJobC with different value based on the result of the task JobB in above powershell task.
Set condition and(always(), eq(variables['RunJobC'], 'True')) for the JobC.
My test scripts (Check Allow scripts to access the OAuth token option in the Phase):
$url = "https://vsrm.dev.azure.com/M<YourOrganization>/<YourProject>/_apis/release/releases?definitionId=24&api-version=6.0"
$RealeasePipeline= Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
} -Method Get
$ReleaseId= $RealeasePipeline.value.id | Select-Object -first 1
Write-Host The current Release Id: $ReleaseId
$url2 = "https://vsrm.dev.azure.com/<YourOrganization>/<YourProject>/_apis/release/releases/$($ReleaseId)?api-version=6.0"
$ReleaseInfo= Invoke-RestMethod -Uri $url2 -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
} -Method Get
$TargetTask=$ReleaseInfo.environments.deploySteps.releaseDeployPhases.deploymentJobs.tasks| where { $_.Name -eq "JobB"}
Write-Host JobB task Result is: $TargetTask.status
if ($TargetTask.status -eq "succeeded"){
Write-Host ("##vso[task.setvariable variable=RunJobC]False")
}elseif($TargetTask.status -eq "Failed"){
Write-Host ("##vso[task.setvariable variable=RunJobC]True")
}
My test result:
Using Azure DevOps Server 2019.1 i am starting to work with Multi jobs, to allow me to split up work onto multiple agents.
The flow itself works fine. I have it setup like this
Begin Job - this basically tests a few variables and Updated the buildnumber
(Depends on Begin Job) RunTest Job - A job to run "multi-configuration", which splits a comma seporated list of task categories
(Depends on RunTest Job) End Job - A trigger build task for a new build in the chain
While the jobs depend on another job, this only seems to affect the time they will start, they will not get access to the information provided by the job that ran before.
Basically what i need is the value of a variable that has been set (buildNumber) in the Begin Job.
I need this version number in the RunTest and End Job.
How can i get this information ? I read articles that this is not possible, but have not seen a valid workaround yet. Does anyone have a decent workaround ?
Did you try multi job output variable:
jobs:
# Set an output variable from job A
- job: A
pool:
vmImage: 'vs2017-win2016'
steps:
- powershell: echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]this is the value"
name: setvarStep
- script: echo $(setvarStep.myOutputVar)
name: echovar
# Map the variable into job B
- job: B
dependsOn: A
pool:
vmImage: 'ubuntu-16.04'
variables:
myVarFromJobA: $[ dependencies.A.outputs['setvarStep.myOutputVar'] ] # map in the variable
# remember, expressions require single quotes
steps:
- script: echo $(myVarFromJobA)
name: echovar
Update2:
Using YAML should be the simplest solution. If you insist on Classic build view. You could try to accomplish this by storing the values in a file (json, xml, yaml, what have you), you can read the file in the Job either direct use or re-set the variable again.
When you queue next build, it will not effect the file in source control and the default value will also not change.
Passing variables between jobs in the same stage, it requires working with output variables.
However, according to this, using outputs in a different job is not supported in Classic UI Format.
As workarounds in this scenario, you can share variables via Pipeline Variables(share variables across jobs in same pipeline).
1.You can set a key variable in pipeline variables:
2.Add one Powershell Inline task with content below in your first job:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable to its new value
$pipeline.variables.key.value = "value"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value
write-host "=========================================================="
3.Run the pipeline we can find the value of key variable is successfully updated:
So you can run the ps script in first job to update the value of key variable, then all next jobs can access the updated variable easily.
Note:
For the script itself, you only need to change lines $pipeline.variables.key.value = "value"(necessary) and Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value(optional).
If I want to set the variable named MyTest to value MyValue, the lines should be $pipeline.variables.MyTest.value = "MyValue" and Write-host "The value of Varialbe MyTest is updated to" $updatedef.variables.MyTest.value.
To make sure the ps task in one job can access OAuth Token, we should Allow Scripts to Access OAuth Token. Click the agent job name and check the box:
To enable the pipeline has the permission to update pipeline variable (edit build pipeline), go pipeline security to set the Edit build pipeline allow for user xxx(ProjectName) build service.
How to configure azure-pipeline.yaml file in post build to get the build status? I have to get the current build status of the pipeline by using azure-devops-rest-api or azure-devops-node-api within the azure-pipeline.yaml file.
Please help!
How to configure azure-pipeline.yaml file in post build to get the build status?
We could use the REST API Builds - List to get the detailed build info:
https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitions}&api-version=5.1
In the YAML, we could add a powershell task to get the build result, like:
- task: PowerShell#2
inputs:
targetType : inline
script: |
$url = "https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitionID}&api-version=5.1"
$connectionToken="Your PAT Here"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$buildPipeline= Invoke-RestMethod -Uri $url -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
$BuildResult= $buildPipeline.value.result | Select-Object -first 1
Write-Host This is Build Result: $BuildResult
We list all the build result for the specify definitions, then use Select-Object -first 1 to get the latest build result.
As test, use the REST API, we could get the result of latest build for the current pipeline, but we could not get the results of the build we are performing this time.
Besides, there is a Predefined variables, which we could check the get the current build status of the pipeline by the Predefined variables Agent.JobStatus, so I add a command line task to output this value in YAML:
- script: |
echo $(Agent.JobStatus)
This variable could get the current build status.
Hope this helps.