How can I calculate a "build revision" if I don't know the build number until the middle of a job? - azure-devops

I have a build pipeline who's only purpose is to take binaries generated by a legacy build process and package them up into a NuGet package. The version number of the NuGet package needs to match the version number of the binaries, and I'd really like if the run number (YAML "name") also matched, but I also need to tack on a revision number to the end of each build due to the uniqueness rules for NuGet feeds.
I know how to achieve the revision number if I'm using a version number that's known in advance:
versionNumber: 4.0.0
revision: $[counter(variables['versionNumber'], 0)]
buildVersion: $[ format('{0}.{1}', variables.versionNumber, variables.revision) ]
name: $(BuildDefinitionName)_$(versionNumber)-rc$(revision)
but in this case I don't know versionNumber until runtime, because I'm pulling it from the version information on another DLL. So, I have adjusted this to do the following:
- powershell: |
$version = gci -Path $env:BUILD_SOURCESDIRECTORY -Recurse -Filter my.dll | % { $_.versioninfo.ProductVersion }
Write-Host "##vso[task.setvariable variable=version]$version"
- powershell: ${{ format('Write-Host "##vso[task.setvariable variable=revision]{0}"', counter(variables['version'], 0) }}
- powershell: |
$version = "${env:VERSION}-rc${env:REVISION}"
Write-Host "##vso[build.updatebuildnumber]${env:BUILDDEFINITIONNAME}_${version}"
This doesn't work, I assume, because you can't use counter in this context, you can only use it when defining a variable. But if I try to do this earlier in the variables section, the value of the version variable isn't set properly, so I get the wrong counter. Is there any way to get this revision number generated dynamically after I have run the first few tasks to get the version number?

How can I calculate a “build revision” if I don't know the build number until the middle of a job?
It seems you are using the YAML, if yes, I am afraid there is no such way to get the revision number generated dynamically after few tasks to get the version number at this moment.
Just as you test, regardless of whether we put the version number of the task that gets the binary in front of or behind the variable section, we could not seem to get this revision number dynamically generated correctly. It should be the limitation for the YAML at this moment.
As workaround, we could use the classic editor to resolve this issue. Please check following steps:
Create a classic editor pipeline and set the versionNumber with default value in variables tab.
Add powershell scripts to the version number of the binaries.
Invoke REST API (Definitions - Update) to update the default value versionNumber in the variables tab with the new version get from previous powershell task.
Details steps for the latest point:
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Grant Project Collection Build Service (xxx) account the edit build pipeline permission. (Select the build pipeline --> ... --> Security --> Edit release definition set to Allow)
Add a PowerShell task in your build pipeline
The poswershell scripts looks like:
$url = "https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named TestVar to its new value 2.0.0
$pipeline.variables.versionNumber.value = "<The value you get from previous powershell task>"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'versionNumber' is updated to" $updatedef.variables.versionNumber.value
Then, we could to see the versionNumber updated to 2.0.0:
Hope this helps.

Related

ADS 2019 - How to pass variables between build jobs

Using Azure DevOps Server 2019.1 i am starting to work with Multi jobs, to allow me to split up work onto multiple agents.
The flow itself works fine. I have it setup like this
Begin Job - this basically tests a few variables and Updated the buildnumber
(Depends on Begin Job) RunTest Job - A job to run "multi-configuration", which splits a comma seporated list of task categories
(Depends on RunTest Job) End Job - A trigger build task for a new build in the chain
While the jobs depend on another job, this only seems to affect the time they will start, they will not get access to the information provided by the job that ran before.
Basically what i need is the value of a variable that has been set (buildNumber) in the Begin Job.
I need this version number in the RunTest and End Job.
How can i get this information ? I read articles that this is not possible, but have not seen a valid workaround yet. Does anyone have a decent workaround ?
Did you try multi job output variable:
jobs:
# Set an output variable from job A
- job: A
pool:
vmImage: 'vs2017-win2016'
steps:
- powershell: echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]this is the value"
name: setvarStep
- script: echo $(setvarStep.myOutputVar)
name: echovar
# Map the variable into job B
- job: B
dependsOn: A
pool:
vmImage: 'ubuntu-16.04'
variables:
myVarFromJobA: $[ dependencies.A.outputs['setvarStep.myOutputVar'] ] # map in the variable
# remember, expressions require single quotes
steps:
- script: echo $(myVarFromJobA)
name: echovar
Update2:
Using YAML should be the simplest solution. If you insist on Classic build view. You could try to accomplish this by storing the values in a file (json, xml, yaml, what have you), you can read the file in the Job either direct use or re-set the variable again.
When you queue next build, it will not effect the file in source control and the default value will also not change.
Passing variables between jobs in the same stage, it requires working with output variables.
However, according to this, using outputs in a different job is not supported in Classic UI Format.
As workarounds in this scenario, you can share variables via Pipeline Variables(share variables across jobs in same pipeline).
1.You can set a key variable in pipeline variables:
2.Add one Powershell Inline task with content below in your first job:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable to its new value
$pipeline.variables.key.value = "value"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value
write-host "=========================================================="
3.Run the pipeline we can find the value of key variable is successfully updated:
So you can run the ps script in first job to update the value of key variable, then all next jobs can access the updated variable easily.
Note:
For the script itself, you only need to change lines $pipeline.variables.key.value = "value"(necessary) and Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value(optional).
If I want to set the variable named MyTest to value MyValue, the lines should be $pipeline.variables.MyTest.value = "MyValue" and Write-host "The value of Varialbe MyTest is updated to" $updatedef.variables.MyTest.value.
To make sure the ps task in one job can access OAuth Token, we should Allow Scripts to Access OAuth Token. Click the agent job name and check the box:
To enable the pipeline has the permission to update pipeline variable (edit build pipeline), go pipeline security to set the Edit build pipeline allow for user xxx(ProjectName) build service.

How to read environment variables in azure devops pipeline using powershell?

I created a PowerShell job and used the below code to set the environment variable in the azure pipeline using Powershell.
[Environment]::SetEnvironmentVariable("key", "value")
I can print the value using the $env:key in the same job itself.
But when I tried to display the value using $env:key in the next job nothing is printed. How to use the above environment variable through out the azure pipeline. Is there any other way to set and read custom environment variables.
you pretty much have to either use library variable groups (or sets, dont remember the name) or you have to use a specific way to share variables across jobs:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#use-output-variables-from-tasks
According to this, using outputs in a different job is not supported in Classic UI Format.
As workarounds in this scenario, you can share variables via Pipeline Variables(share variables across jobs in same pipeline) or Variable Groups(share variables across pipelines that use same Variable Group, it also works across jobs).
Since you only want to share variables across jobs in same pipeline, pipeline variable is enough for you.
1.You can set a key variable in pipeline variables:
2.Add one Powershell Inline task with content below in your first job:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable to its new value
$pipeline.variables.key.value = "value"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value
write-host "=========================================================="
3.Run the pipeline we can find the value of key variable is successfully updated:
So you can run the ps script in first job to update the value of key variable, then all next jobs can access the updated variable easily.
Note:
For the script itself, you only need to change lines $pipeline.variables.key.value = "value"(necessary) and Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value(optional).
If I want to set the variable named MyTest to value MyValue, the lines should be $pipeline.variables.MyTest.value = "MyValue" and Write-host "The value of Varialbe MyTest is updated to" $updatedef.variables.MyTest.value.
To make sure the ps task in one job can access OAuth Token, we should Allow Scripts to Access OAuth Token. Click the agent job name and check the box:
To enable the pipeline has the permission to update pipeline variable (edit build pipeline), go pipeline security to set the Edit build pipeline allow for user xxx(ProjectName) build service.
Hope all above helps :)

How to use a custom variable from build-pipeline in a release pipeline

Question:
Is there any way to define a custom-variable in a build pipeline in azure-devops, which then can be used in/exposed to the release pipeline in any way?
Scenario:
We are not using a variable group, because we need to dynamically set the variable in the build pipeline and then consume it in the release pipeline - it is not a static super global.
Checked the docs at release variables and build variables but could not find any helping information or a hint, that this is possible.
What I tried
Define a variable in variables ( in the build pipeline ) and try to access it in the release pipeline using $(name) or checking if it is in env.
Extras - Motivation
The motivation behind this is
read the latest git-tag used in a build-pipeline step and expose this into the pipeline variable VERSION ( actually, we bump patch during that )
Releasing builds is a manual step.
If a build is released, the azure-devops gui shows us all the variables of the release-pipeline, which are "settable during release - this includes the version we want to release this package with
we want this "VERSION" to be prefilled with the version of the build-pipeline as a suggestion for the next version
In the release pipeline we checkout the repo, add the VERSION as a tag and package/publish the artifact with this version
This is impossible by default, but you can use 2 extensions from the marketplace:
1) Shared variable updater - Create a variable group and during the build update there the variables dynamically with this task. you can also do it with your script, see the answers here.
2) Variable Kit for Azure DevOps Services - During a Build, save variables to a json file stored along with your build assets. During a Release, load the saved variables and use them within the release definition.
How to use a custom variable from build-pipeline in a release pipeline
You could try to use the REST API Release Definitions - Update to update the default variable in the release pipeline to use the value defined in a build pipeline.
PUT https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.1
Details:
Define a custom-variable in a build pipeline, like TestValue, the value is 123:
Also define the same custom-variable in a Release pipeline with default value 123:
Then add a inline powershell scripts to invoke the REST API Definitions - Update to update the default value in the release pipeline:
$url = "https://vsrm.dev.azure.com/<OrganizationName>/<ProjectName>/_apis/release/definitions/<DefinitionId>?api-version=5.1"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named TestValue to its new value 987
$pipeline.variables.TestValue.value = "$(TestValue)"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
Write-Host "URL: $json "
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'TestValue' is updated to" $updatedef.variables.TestValue.value
In this case, we could dynamically set the variable when we queue the build pipeline, and this value will overwrite the default value in the release pipeline, so that we could use it release pipeline.
Hope this helps.

Setting Git Tag from Azure Devops Build Pipeline on Complete

I'm trying to set a tag with the current version number determined by GitVersion on the GIT commit at the end of a successful build. Feels like I can't be the first one to be doing this, but I'm struggling to find something that works.
Azure Devops Pipeline has a feature in Get Sources to "Tag sources" On Success. I've set this and set to a variable that is set by one of the Agent Tasks I have (GitVersion)
I can see in the debug logs that this variable is getting set by the GitVersion component that I've added to the pipeline.
2019-12-06T20:54:20.2390794Z ##[debug]Processed: ##vso[task.setvariable variable=GitVersion.MajorMinorPatch;]2.98.0
However if I leave it just as this, I get a tag created as "v$(GitVersion.MajorMinorPatch)" which means that at the time that the tag is being created that that variable no longer exists.
The Tag Format help tooltip says
"Tag format could be a combination of user-defined or pre-defined variables that have a scope of "All". For example: '$(Build.DefinitionName)$(Build.DefinitionVersion)$(Build.BuildId)$(Build.BuildNumber)$(My.Variable)'"
So I guess the problem is that this variable created during the pipeline does not have a scope of All.
I then tried adding a pipeline variable to the pipeline of "GitVersion.MajorMinorPatch" with the hope that this was at the right scope and hoping that when the "task.setvariable" command is run, that this will set the variable value of this higher scoped variable.
However in this case I just got a tag "v" created.
So I am a bit stuck. Somehow I need to be able to dynamically create or set a variable at scope ALL with the value I want to tag here.
I'd be really grateful for any ideas on this.
If you are doing a yaml pipeline, you can add the following steps
- checkout: self
persistCredentials: true
## Rest of pipeline ##
- script: |
git tag $(GitVersion.NugetVersionV2)
git push origin $(GitVersion.NugetVersionV2)
workingDirectory: $(Build.SourcesDirectory)
The persistCredentials allows the token to be automatically passed to other git commands. Note the assignment of workingDirectory, otherwise I had an error that the location was not a git repository.
For an annotated tag rather than lightweight tag, the syntax would look like this...
- script: |
git tag -a <tagname> -m <message>
git push origin <tagname>
To get a user/date against it you need to set the user name/email as well e.g.
- script: |
git config --global user.name "BuildService"
git config --global user.email "autobuild#fabrikam.com"
git tag -a <tagname> -m <message>
git push origin <tagname>
For this to work, the Project Collection Build Server account (not the Project Build Service Accounts group) needs to be allocated the Contribute permission for the Repositories
Expanding on the excellent answer from Paul Hatcher, I'd like to add that for me the account was called Project Collection Build Service in Azure DevOps Server 2019. This also seems to be in line with the current Microsoft documentation.
Sorry for the answer, my reputation does not yet suffice to comment.
I can see in the debug logs that this variable is getting set by the
GitVersion component that I've added to the pipeline.
The variable GitVersion.MajorMinorPatch you saw from the log is a step-level variable, which means its life cycle is only start from the current GitVersion task.
As the definition you are referring, it scope must to all. This means is must be a global variable. For example, the predefined variables that the system default have, and the customized variables which specified in the Variables tab.
Based on the GitVersion task compile and work logic, in fact, the GitVersion.MajorMinorPatch value is generated and stored as current build's build number:
So, the most convenient method for you to tag the GitVersion.MajorMinorPatch value to repos is using $(Build.BuildNumber):
v$(Build.BuildNumber)
And this is my result:
Update:
To add the GitVersion.MajorMinorPatch which generated by the GitVersion task into Variables, please apply below scripts into PowerShell task:
$connectionToken="{PAT Token}"
$urlget = "https://dev.azure.com/{org}/{project}/_apis/build/definitions/$(System.DefinitionId)?api-version=5.1"
$base64AuthInfo = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$getdef = Invoke-RestMethod -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -Method GET -ContentType application/json -Uri $urlget
Write-Host Pipeline = $($getdef | ConvertTo-Json -Depth 100)
$bvalue=#"
{
"value": "$(GitVersion.MajorMinorPatch)"
}
"#
$getdef.variables | add-member -Name "GitVersion.MajorMinorPatch" -value (Convertfrom-Json $bvalue) -MemberType NoteProperty -Force -PassThru
$getdef = $getdef | ConvertTo-Json -Depth 100
$getdef | clip
$urlput = "https://dev.azure.com/{org}/{project}/_apis/build/definitions/$(System.DefinitionId)?api-version=5.1"
$putdef = Invoke-RestMethod -Uri $urlput -Method PUT -Body $getdef -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
As I mentioned previously, I still don't think it is available to specify $(GitVersion.MajorMinorPatch) in Tag format.
Still strongly suggest you by calling $(Build.BuildNumber) to tag the $(GitVersion.MajorMinorPatch) value
- pwsh: |
# Construct PAT authentication header
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "user",$env:SYSTEM_ACCESSTOKEN)))
$headers = #{Authorization=("Basic {0}" -f $base64AuthInfo)}
$url="$(System.CollectionUri)/$(System.TeamProject)/_apis/git/repositories/$(Build.Repository.ID)/annotatedtags?api-version=5.0-preview.1"
$body = #{name = "$(GitVersion.MajorMinorPatch)"
message = "automatically added"
taggedObject = #{
objectId = "$(Build.SourceVersion)"
}
} | ConvertTo-Json
Invoke-RestMethod -Uri $url -Headers $headers -Method Post -ContentType "application/json" -Body ($body)
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
displayName: 'Add tag'

How to modify Azure DevOps release definition variable from a release task?

What is the easiest way to get key rotation to work for azure storage accounts from a AzureDevOps relase task? The current plan is to re-generate the old key after release to invalidate it, and have a fresh key that can be used on next deployment. But to get that to work it seems like I at least need to store the name of the key to use in a release variable.
I had a look at he logging tasks (https://github.com/Microsoft/azure-pipelines-tasks/blob/master/docs/authoring/commands.md), but that only changes the value in the current release and does not modify the release definition.
You can use the REST API (Definitions - Update) to update the value of the release definition variable from a release task.
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Grant Project Collection Build Service (xxx) account the edit
release pipeline permission. (Select the release pipeline --> ... --> Security --> Edit release definition set to Allow)
Add a PowerShell task in your release pipeline
Run inline script: (Update the value of variable v1030 in below sample)
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/Release/definitions/$($env:RELEASE_DEFINITIONID)?api-version=5.0-preview.3"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named v1030 to its new value 1035
$pipeline.variables.v1030.value = "1035"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'v1030' is updated to" $updatedef.variables.v1030.value
write-host "=========================================================="
Here is a much cleaner and better solution that also allows for multiple builds being triggered simultaneously.
https://tsuyoshiushio.medium.com/how-to-pass-variables-with-pipeline-trigger-in-azure-pipeline-5771c5f18f91
Essentially your triggering build produces artifacts that your triggered build reads and turns into variables.
Still not at all great, but better than nothing and better than REST calls setting static global variable groups.
The other answer above talks about how to update Release pipelines.
If you would like to update a Build Pipeline's variables, here is how you do that:
Edit build pipeline
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Go to Manage Security -> Users -> Select Project Collection Build Service (YOUR TEAM NAME HERE)
Change "Edit Build Definitions" to Allow
Now add a powershell stage - 2.x - inline script called Update variables.
Script inline contents:
$api_version='5.0-preview.6'
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$(System.DefinitionId)?api-version=${api_version}"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update variables as desired here:
$pipeline.variables.mavenBuildVersionPatch.value = "2401"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Variable mavenBuildVersionPatch is updated to" $updatedef.variables.mavenBuildVersionPatch.value
write-host "=========================================================="
Take note of the API version in this script is 5.0-preview.6.
If your version of Azure Devops is newer, you may need to update this in the future.
Save build pipeline.
Now when you run job, after the job completes this powershell stage, this variable will be set.
Important: If you want to update a variable then make the updated variable for other pipeline stages, then you do it with a powershell stage with the following inline script:
$mavenBuildVersionPatch = [int]"$(mavenBuildVersionPatch)"
$mavenBuildVersionPatch = $mavenBuildVersionPatch + 1
Write-Host "##vso[task.setvariable variable=mavenBuildVersionPatch;]$mavenBuildVersionPatch"
This example would take our existing patch number and increment it by 1.
This does not save the variable at the end of the job, you still need to do that with another powershell script if desired.