Using Azure DevOps Server 2019.1 i am starting to work with Multi jobs, to allow me to split up work onto multiple agents.
The flow itself works fine. I have it setup like this
Begin Job - this basically tests a few variables and Updated the buildnumber
(Depends on Begin Job) RunTest Job - A job to run "multi-configuration", which splits a comma seporated list of task categories
(Depends on RunTest Job) End Job - A trigger build task for a new build in the chain
While the jobs depend on another job, this only seems to affect the time they will start, they will not get access to the information provided by the job that ran before.
Basically what i need is the value of a variable that has been set (buildNumber) in the Begin Job.
I need this version number in the RunTest and End Job.
How can i get this information ? I read articles that this is not possible, but have not seen a valid workaround yet. Does anyone have a decent workaround ?
Did you try multi job output variable:
jobs:
# Set an output variable from job A
- job: A
pool:
vmImage: 'vs2017-win2016'
steps:
- powershell: echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]this is the value"
name: setvarStep
- script: echo $(setvarStep.myOutputVar)
name: echovar
# Map the variable into job B
- job: B
dependsOn: A
pool:
vmImage: 'ubuntu-16.04'
variables:
myVarFromJobA: $[ dependencies.A.outputs['setvarStep.myOutputVar'] ] # map in the variable
# remember, expressions require single quotes
steps:
- script: echo $(myVarFromJobA)
name: echovar
Update2:
Using YAML should be the simplest solution. If you insist on Classic build view. You could try to accomplish this by storing the values in a file (json, xml, yaml, what have you), you can read the file in the Job either direct use or re-set the variable again.
When you queue next build, it will not effect the file in source control and the default value will also not change.
Passing variables between jobs in the same stage, it requires working with output variables.
However, according to this, using outputs in a different job is not supported in Classic UI Format.
As workarounds in this scenario, you can share variables via Pipeline Variables(share variables across jobs in same pipeline).
1.You can set a key variable in pipeline variables:
2.Add one Powershell Inline task with content below in your first job:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable to its new value
$pipeline.variables.key.value = "value"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value
write-host "=========================================================="
3.Run the pipeline we can find the value of key variable is successfully updated:
So you can run the ps script in first job to update the value of key variable, then all next jobs can access the updated variable easily.
Note:
For the script itself, you only need to change lines $pipeline.variables.key.value = "value"(necessary) and Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value(optional).
If I want to set the variable named MyTest to value MyValue, the lines should be $pipeline.variables.MyTest.value = "MyValue" and Write-host "The value of Varialbe MyTest is updated to" $updatedef.variables.MyTest.value.
To make sure the ps task in one job can access OAuth Token, we should Allow Scripts to Access OAuth Token. Click the agent job name and check the box:
To enable the pipeline has the permission to update pipeline variable (edit build pipeline), go pipeline security to set the Edit build pipeline allow for user xxx(ProjectName) build service.
Related
I would like to identify the build policy for a build that was run by clicking the Queue (or Re-queue) button against a required/optional check from within a pull request. I wish to identify the policy programmatically from within a pipeline; e.g. a script task. Open to any approach, been exploring the az CLI but no luck thus far.
I've setup two build policies against a branch that both target the same build definition - Policy A and Policy B. Both are setup to be run manually - A is required, B is optional. Both will surface in the UI for a pull request as checks - A being required, B being optional. When a build is run by clicking the Queue (or Re-queue) button against either check, I would like to be able to identify which of the two policies the run was initiated from, (which policy provided the Queue or Re-queue button that was clicked).
EDIT: A bit more background on what I'm doing ...
I've got a single pipeline for building an application.
I've recently got a request to update the pipeline to support publishing to Chromatic.
I've added a Publish to Chromatic parameter to the pipeline and a task to push to Chromatic when the parameter is set to true.
I received a subsequent request to make it easier to publish changes from a feature branch to Chromatic. One engineer threw out the idea of having an optional check available in pull requests to give a single button click experience.
While researching my options, I was wondering if it would be possible to enhance the existing pipeline to set the Publish to Chromatic parameter to true during a run. I found this comment on Reddit which ultimately led to me posting here ...
set a default for your parameter (I like to use 'auto') add a script >task near the beginning that reads the pull request comment and sets a variable for you to use in later logic if the parameter is auto . you can even condition this to only run on a PR.
I am aware that I could create a separate pipeline for publishing to Chromatic instead of updating the existing one; that's one of a few options I have. At this point, I'm more-so curious whether or not this particular approach is technically feasible even if I opt not to go forward with it.
Hope that adds some clarity!
The policy that queued the pipeline isn't something that is visible to the pipeline as a pipeline variable. In fact, there doesn't seem to be any indication if the PullRequest was queued manually or automatically.
There might be a few other ways to approach this...
I would start by putting a publishChromatic parameter in the pipeline and then building up conditions in the pipeline execution around this variable. By default, let's assume that the value is false so that if you're manually queueing a pipeline run you can opt-in.
triggers:
- include:
branches:
- develop
parameters:
- name: publishChromatic
displayName: 'Publish build to Chromatic'
type: boolean
default: false
jobs:
- job: Build
variables:
publishChromatic: ${{ parameters.publishChromatic }}
steps:
... pre-execution steps
- task: CmdLine#2
displayName: Publish to Chromatic
condition: and(succeeded(), eq(variables['publishChromatic'], 'true'))
inputs:
script: npx chromatic --project-token=$(CHROMATIC_PROJECT_TOKEN) --branch=$(Build.SourceBranch)
... post execution steps
Option 1: Pull Request Labels
One option might be to inspect the pull request for the presence of a label as outlined in this answer. As a pre-execution step, a simple script could flip the flag when the label is present:
- pwsh: |
$urlFormat = "{0}/{1}/_apis/git/repositories/{1}/pullRequests/{2}/labels?api-version=6.0-preview.1"
$url = $urlFormat -f `
$env:SYSTEM_TEAMFOUNDATIONSERVERURI, `
$env:SYSTEM_TEAMPROJECTID, `
$env:BUILD_REPOSITORY_NAME, `
$env:SYSTEM_PULLREQUEST_PULLREQUESTID
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$response = Invoke-RestMethod -Uri $url -Method Get -Headers $headers
$labels = $response.value.name
Write-Host "##vso[task.setvariable variable=PullRequestTag]$labels"
displayName: 'Fetch Pull Request Labels'
condition: and( succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
- pwsh: |
if ("$(PullRequestLabels)" -like "*chromatic*") {
Write-Host "##vso[task.setvariable variable=publishChromatic]true"
}
condition: and( succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
displayName: 'Check for Chromatic label'
I like this option in that it provides a bit of traceability for which Pull Requests were deployed. Unfortunately, there's no way to queue a build automatically when the PR labels are modified so you'd need to have the tag on the PR before triggering the pipeline.
You could also establish a different pattern such as triggering based on a convention like a value that appears in the name of the Pull Request, etc.
Option 2: Pipeline to Trigger Chromatic
If you'd rather have a Build Validation option labeled 'Deploy to Chromatic' to automate triggering your deployment to Chromatic, a simple option would be to create a pipeline that triggers your pipeline with the publishChromatic parameter.
trigger: none
steps:
- checkout: none
- pwsh: |
$pipelineId = 1234
$urlFormat = "{0}/{1}/_apis/pipelines/{2}/runs?api-version=6.0-preview.1
$url = $urlFormat -f `
$env:SYSTEM_TEAMFOUNDATIONSERVERURI, `
$env:SYSTEM_TEAMPROJECTID `
$pipelineId
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$body = #{
resources = #{ repositories = #{ self = #{ refName = "$(Build.SourceBranch)" } } }
variables = #{
originalPrId = #{
value = "$(System.PullRequest.PullRequestId)
}
}
templateParameters = #{
publishChromatic = $true
}
}
Invoke-RestMethod -Uri $url -Method Post -Body $body -Headers $headers
displayName: 'Trigger Chromatic Pipeline'
condition: eq(variables['Build.Reason'],'PullRequest')
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
This simple script performs a fire-and-forget approach to triggering your original pipeline.
If you need to have a successful deployment to Chromatic as part of your PR, you could adjust the original pipeline to report a pull-request status.
In your original pipeline, add the following as a post-execution step:
- pwsh: |
$urlFormat = "{0}/{1}/_apis/git/repositories/{2}/pullRequests/{3}/statuses?api-version=6.0-preview.1
$url = $urlFormat -f `
$env:SYSTEM_TEAMFOUNDATIONSERVERURI, `
$env:SYSTEM_TEAMPROJECTID, `
$env:BUILD_REPOSITORY_NAME, `
"$(originalPrId)"
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$body = #{
status = "succeeded"
description = "completed chromatic regression"
context = #{
name = "qualitygate/chromatic"
}
targetUrl = "http://chromatic.com/your/buildid"
}
Invoke-RestMethod -Uri $url -Method POST -Body $body -Headers $headers
displayName: Report status to PR
condition: and( succeeded(), ne(variables['originalPrId'],''))
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
To require a successful chromatic quality gate, add a Status Check to your Branch Policy with the same name mentioned above qualitygate/chromatic.
Option 3: Further down the Rabbit hole
You can establish even deeper integration with Chromatic by building a custom extension that allows you to add specialized menus to the Pull Request checks menu. The custom menu could include javascript-enabled buttons to trigger your pipeline without the need for a custom pipeline mentioned in Option 2.
While not necessarily dependent on writing a custom extension, you could also create an Azure Function App that listens for webhooks from Chromatic and posts status updates back to your PR with the a custom UI that links back to the Chromatic build. You'd simply need to query the Azure DevOps API to map the branch name in the Chromatic payload to the corresponding PR.
I have a pipeline that executes every hour and sometimes the execution takes more than an hour and the other execution starts, is there a way to prevent this? and for example the new execution to get queued?
thank you for all the help
It seems you have multiple build agents. Assuming you are using self-hosted build agents, you could specify certain demands of the agent to use only one agent. In this way, if the agent is not free, the build will keep waiting. To use a particular agent, add a demand of Agent.Name equals agentname, check the screenshot below. Agent name can be found in capabilities of the agent.
pool:
name: MyPool
demands:
- myCustomCapability # check for existence of capability
- agent.name -equals agentname # check for specific string in capability
Another way is triggering the pipeline via REST api and through the PowerShell task. You could use the REST API Builds - List to get the detailed build info and check the latest build status:
https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitions}&api-version=6.0
In the YAML, we could add a powershell task to get the build status, like:
- task: PowerShell#2
inputs:
targetType : inline
script: |
$url = "https://dev.azure.com/{organization}/{project}/_apis/build/builds?definitions={definitionID}&api-version=6.0"
$connectionToken="Your PAT Here"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$buildPipeline= Invoke-RestMethod -Uri $url -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
$BuildStatus= $buildPipeline.value.status | Select-Object -first 1
Write-Host This is Build Status: $BuildStatus
This list all the build status for the specify definitions, then use Select-Object -first 1 to get the latest build status. If the status is completed, then queue the build. If the status is not completed, do not queue the build.
I created a PowerShell job and used the below code to set the environment variable in the azure pipeline using Powershell.
[Environment]::SetEnvironmentVariable("key", "value")
I can print the value using the $env:key in the same job itself.
But when I tried to display the value using $env:key in the next job nothing is printed. How to use the above environment variable through out the azure pipeline. Is there any other way to set and read custom environment variables.
you pretty much have to either use library variable groups (or sets, dont remember the name) or you have to use a specific way to share variables across jobs:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#use-output-variables-from-tasks
According to this, using outputs in a different job is not supported in Classic UI Format.
As workarounds in this scenario, you can share variables via Pipeline Variables(share variables across jobs in same pipeline) or Variable Groups(share variables across pipelines that use same Variable Group, it also works across jobs).
Since you only want to share variables across jobs in same pipeline, pipeline variable is enough for you.
1.You can set a key variable in pipeline variables:
2.Add one Powershell Inline task with content below in your first job:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable to its new value
$pipeline.variables.key.value = "value"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value
write-host "=========================================================="
3.Run the pipeline we can find the value of key variable is successfully updated:
So you can run the ps script in first job to update the value of key variable, then all next jobs can access the updated variable easily.
Note:
For the script itself, you only need to change lines $pipeline.variables.key.value = "value"(necessary) and Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value(optional).
If I want to set the variable named MyTest to value MyValue, the lines should be $pipeline.variables.MyTest.value = "MyValue" and Write-host "The value of Varialbe MyTest is updated to" $updatedef.variables.MyTest.value.
To make sure the ps task in one job can access OAuth Token, we should Allow Scripts to Access OAuth Token. Click the agent job name and check the box:
To enable the pipeline has the permission to update pipeline variable (edit build pipeline), go pipeline security to set the Edit build pipeline allow for user xxx(ProjectName) build service.
Hope all above helps :)
Question:
Is there any way to define a custom-variable in a build pipeline in azure-devops, which then can be used in/exposed to the release pipeline in any way?
Scenario:
We are not using a variable group, because we need to dynamically set the variable in the build pipeline and then consume it in the release pipeline - it is not a static super global.
Checked the docs at release variables and build variables but could not find any helping information or a hint, that this is possible.
What I tried
Define a variable in variables ( in the build pipeline ) and try to access it in the release pipeline using $(name) or checking if it is in env.
Extras - Motivation
The motivation behind this is
read the latest git-tag used in a build-pipeline step and expose this into the pipeline variable VERSION ( actually, we bump patch during that )
Releasing builds is a manual step.
If a build is released, the azure-devops gui shows us all the variables of the release-pipeline, which are "settable during release - this includes the version we want to release this package with
we want this "VERSION" to be prefilled with the version of the build-pipeline as a suggestion for the next version
In the release pipeline we checkout the repo, add the VERSION as a tag and package/publish the artifact with this version
This is impossible by default, but you can use 2 extensions from the marketplace:
1) Shared variable updater - Create a variable group and during the build update there the variables dynamically with this task. you can also do it with your script, see the answers here.
2) Variable Kit for Azure DevOps Services - During a Build, save variables to a json file stored along with your build assets. During a Release, load the saved variables and use them within the release definition.
How to use a custom variable from build-pipeline in a release pipeline
You could try to use the REST API Release Definitions - Update to update the default variable in the release pipeline to use the value defined in a build pipeline.
PUT https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.1
Details:
Define a custom-variable in a build pipeline, like TestValue, the value is 123:
Also define the same custom-variable in a Release pipeline with default value 123:
Then add a inline powershell scripts to invoke the REST API Definitions - Update to update the default value in the release pipeline:
$url = "https://vsrm.dev.azure.com/<OrganizationName>/<ProjectName>/_apis/release/definitions/<DefinitionId>?api-version=5.1"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named TestValue to its new value 987
$pipeline.variables.TestValue.value = "$(TestValue)"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
Write-Host "URL: $json "
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'TestValue' is updated to" $updatedef.variables.TestValue.value
In this case, we could dynamically set the variable when we queue the build pipeline, and this value will overwrite the default value in the release pipeline, so that we could use it release pipeline.
Hope this helps.
I have a build pipeline who's only purpose is to take binaries generated by a legacy build process and package them up into a NuGet package. The version number of the NuGet package needs to match the version number of the binaries, and I'd really like if the run number (YAML "name") also matched, but I also need to tack on a revision number to the end of each build due to the uniqueness rules for NuGet feeds.
I know how to achieve the revision number if I'm using a version number that's known in advance:
versionNumber: 4.0.0
revision: $[counter(variables['versionNumber'], 0)]
buildVersion: $[ format('{0}.{1}', variables.versionNumber, variables.revision) ]
name: $(BuildDefinitionName)_$(versionNumber)-rc$(revision)
but in this case I don't know versionNumber until runtime, because I'm pulling it from the version information on another DLL. So, I have adjusted this to do the following:
- powershell: |
$version = gci -Path $env:BUILD_SOURCESDIRECTORY -Recurse -Filter my.dll | % { $_.versioninfo.ProductVersion }
Write-Host "##vso[task.setvariable variable=version]$version"
- powershell: ${{ format('Write-Host "##vso[task.setvariable variable=revision]{0}"', counter(variables['version'], 0) }}
- powershell: |
$version = "${env:VERSION}-rc${env:REVISION}"
Write-Host "##vso[build.updatebuildnumber]${env:BUILDDEFINITIONNAME}_${version}"
This doesn't work, I assume, because you can't use counter in this context, you can only use it when defining a variable. But if I try to do this earlier in the variables section, the value of the version variable isn't set properly, so I get the wrong counter. Is there any way to get this revision number generated dynamically after I have run the first few tasks to get the version number?
How can I calculate a “build revision” if I don't know the build number until the middle of a job?
It seems you are using the YAML, if yes, I am afraid there is no such way to get the revision number generated dynamically after few tasks to get the version number at this moment.
Just as you test, regardless of whether we put the version number of the task that gets the binary in front of or behind the variable section, we could not seem to get this revision number dynamically generated correctly. It should be the limitation for the YAML at this moment.
As workaround, we could use the classic editor to resolve this issue. Please check following steps:
Create a classic editor pipeline and set the versionNumber with default value in variables tab.
Add powershell scripts to the version number of the binaries.
Invoke REST API (Definitions - Update) to update the default value versionNumber in the variables tab with the new version get from previous powershell task.
Details steps for the latest point:
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Grant Project Collection Build Service (xxx) account the edit build pipeline permission. (Select the build pipeline --> ... --> Security --> Edit release definition set to Allow)
Add a PowerShell task in your build pipeline
The poswershell scripts looks like:
$url = "https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named TestVar to its new value 2.0.0
$pipeline.variables.versionNumber.value = "<The value you get from previous powershell task>"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'versionNumber' is updated to" $updatedef.variables.versionNumber.value
Then, we could to see the versionNumber updated to 2.0.0:
Hope this helps.