How to use a custom variable from build-pipeline in a release pipeline - azure-devops

Question:
Is there any way to define a custom-variable in a build pipeline in azure-devops, which then can be used in/exposed to the release pipeline in any way?
Scenario:
We are not using a variable group, because we need to dynamically set the variable in the build pipeline and then consume it in the release pipeline - it is not a static super global.
Checked the docs at release variables and build variables but could not find any helping information or a hint, that this is possible.
What I tried
Define a variable in variables ( in the build pipeline ) and try to access it in the release pipeline using $(name) or checking if it is in env.
Extras - Motivation
The motivation behind this is
read the latest git-tag used in a build-pipeline step and expose this into the pipeline variable VERSION ( actually, we bump patch during that )
Releasing builds is a manual step.
If a build is released, the azure-devops gui shows us all the variables of the release-pipeline, which are "settable during release - this includes the version we want to release this package with
we want this "VERSION" to be prefilled with the version of the build-pipeline as a suggestion for the next version
In the release pipeline we checkout the repo, add the VERSION as a tag and package/publish the artifact with this version

This is impossible by default, but you can use 2 extensions from the marketplace:
1) Shared variable updater - Create a variable group and during the build update there the variables dynamically with this task. you can also do it with your script, see the answers here.
2) Variable Kit for Azure DevOps Services - During a Build, save variables to a json file stored along with your build assets. During a Release, load the saved variables and use them within the release definition.

How to use a custom variable from build-pipeline in a release pipeline
You could try to use the REST API Release Definitions - Update to update the default variable in the release pipeline to use the value defined in a build pipeline.
PUT https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.1
Details:
Define a custom-variable in a build pipeline, like TestValue, the value is 123:
Also define the same custom-variable in a Release pipeline with default value 123:
Then add a inline powershell scripts to invoke the REST API Definitions - Update to update the default value in the release pipeline:
$url = "https://vsrm.dev.azure.com/<OrganizationName>/<ProjectName>/_apis/release/definitions/<DefinitionId>?api-version=5.1"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named TestValue to its new value 987
$pipeline.variables.TestValue.value = "$(TestValue)"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
Write-Host "URL: $json "
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'TestValue' is updated to" $updatedef.variables.TestValue.value
In this case, we could dynamically set the variable when we queue the build pipeline, and this value will overwrite the default value in the release pipeline, so that we could use it release pipeline.
Hope this helps.

Related

How to read environment variables in azure devops pipeline using powershell?

I created a PowerShell job and used the below code to set the environment variable in the azure pipeline using Powershell.
[Environment]::SetEnvironmentVariable("key", "value")
I can print the value using the $env:key in the same job itself.
But when I tried to display the value using $env:key in the next job nothing is printed. How to use the above environment variable through out the azure pipeline. Is there any other way to set and read custom environment variables.
you pretty much have to either use library variable groups (or sets, dont remember the name) or you have to use a specific way to share variables across jobs:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#use-output-variables-from-tasks
According to this, using outputs in a different job is not supported in Classic UI Format.
As workarounds in this scenario, you can share variables via Pipeline Variables(share variables across jobs in same pipeline) or Variable Groups(share variables across pipelines that use same Variable Group, it also works across jobs).
Since you only want to share variables across jobs in same pipeline, pipeline variable is enough for you.
1.You can set a key variable in pipeline variables:
2.Add one Powershell Inline task with content below in your first job:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable to its new value
$pipeline.variables.key.value = "value"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value
write-host "=========================================================="
3.Run the pipeline we can find the value of key variable is successfully updated:
So you can run the ps script in first job to update the value of key variable, then all next jobs can access the updated variable easily.
Note:
For the script itself, you only need to change lines $pipeline.variables.key.value = "value"(necessary) and Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value(optional).
If I want to set the variable named MyTest to value MyValue, the lines should be $pipeline.variables.MyTest.value = "MyValue" and Write-host "The value of Varialbe MyTest is updated to" $updatedef.variables.MyTest.value.
To make sure the ps task in one job can access OAuth Token, we should Allow Scripts to Access OAuth Token. Click the agent job name and check the box:
To enable the pipeline has the permission to update pipeline variable (edit build pipeline), go pipeline security to set the Edit build pipeline allow for user xxx(ProjectName) build service.
Hope all above helps :)

How can I calculate a "build revision" if I don't know the build number until the middle of a job?

I have a build pipeline who's only purpose is to take binaries generated by a legacy build process and package them up into a NuGet package. The version number of the NuGet package needs to match the version number of the binaries, and I'd really like if the run number (YAML "name") also matched, but I also need to tack on a revision number to the end of each build due to the uniqueness rules for NuGet feeds.
I know how to achieve the revision number if I'm using a version number that's known in advance:
versionNumber: 4.0.0
revision: $[counter(variables['versionNumber'], 0)]
buildVersion: $[ format('{0}.{1}', variables.versionNumber, variables.revision) ]
name: $(BuildDefinitionName)_$(versionNumber)-rc$(revision)
but in this case I don't know versionNumber until runtime, because I'm pulling it from the version information on another DLL. So, I have adjusted this to do the following:
- powershell: |
$version = gci -Path $env:BUILD_SOURCESDIRECTORY -Recurse -Filter my.dll | % { $_.versioninfo.ProductVersion }
Write-Host "##vso[task.setvariable variable=version]$version"
- powershell: ${{ format('Write-Host "##vso[task.setvariable variable=revision]{0}"', counter(variables['version'], 0) }}
- powershell: |
$version = "${env:VERSION}-rc${env:REVISION}"
Write-Host "##vso[build.updatebuildnumber]${env:BUILDDEFINITIONNAME}_${version}"
This doesn't work, I assume, because you can't use counter in this context, you can only use it when defining a variable. But if I try to do this earlier in the variables section, the value of the version variable isn't set properly, so I get the wrong counter. Is there any way to get this revision number generated dynamically after I have run the first few tasks to get the version number?
How can I calculate a “build revision” if I don't know the build number until the middle of a job?
It seems you are using the YAML, if yes, I am afraid there is no such way to get the revision number generated dynamically after few tasks to get the version number at this moment.
Just as you test, regardless of whether we put the version number of the task that gets the binary in front of or behind the variable section, we could not seem to get this revision number dynamically generated correctly. It should be the limitation for the YAML at this moment.
As workaround, we could use the classic editor to resolve this issue. Please check following steps:
Create a classic editor pipeline and set the versionNumber with default value in variables tab.
Add powershell scripts to the version number of the binaries.
Invoke REST API (Definitions - Update) to update the default value versionNumber in the variables tab with the new version get from previous powershell task.
Details steps for the latest point:
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Grant Project Collection Build Service (xxx) account the edit build pipeline permission. (Select the build pipeline --> ... --> Security --> Edit release definition set to Allow)
Add a PowerShell task in your build pipeline
The poswershell scripts looks like:
$url = "https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named TestVar to its new value 2.0.0
$pipeline.variables.versionNumber.value = "<The value you get from previous powershell task>"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'versionNumber' is updated to" $updatedef.variables.versionNumber.value
Then, we could to see the versionNumber updated to 2.0.0:
Hope this helps.

How to set the Description of an Azure DevOps Release via script?

In Azure DevOps I'm trying to set the release description via PowerShell / CMD in order to have a dynamic description of my releases based on the input of the artifacts from the build step.
I've tried setting the release variables via powershell like:
Write-Host "##vso[task.setvariable variable=release.releasedescription;]bar"
Write-Host "##vso[task.setvariable variable=RELEASE_RELEASEDESCRIPTION;]bar"
But that didn't work and the description field remains always empty.
Is there any tweak / setting that would help achieve this behavior?
What you tried to do is just to set the environment variable that contains the release description data and not set the "real" release description, so after the release finished the description not changed.
If you want to set the release description during the release you can try to do it with Azure DevOps Rest API - Update Release.
So add a PowerShell task that executes the Rest API with Invoke-RestMethod, get the current release with GET method and then update the release with PUT, in the body change the description to the new one.
You could do this too:
- bash: |
echo "Updating pipeline job Run description"
echo "##vso[build.updatebuildnumber]$(Build.BuildNumber) $(App_Name)"
displayName: "Set pipeline job Run description for Azure DevOps console"
For anyone who simply wants to set the name of an Azure Pipelines run, you can find the description here.
TLDR: Set the top-level name: attribute.
Here is the PowerShell script to set release description in DevOps. Before the stage that need approval, add a PowerShell task to invoke REST API to change the release description. Please use your own orgName, projectName, PAT and modify the release description.
$url = https://vsrm.dev.azure.com/<orgName>/<ProjectName>/_apis/release/releases/$(Release.ReleaseId)?api-version=6.0
$token="PAT"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$head = #{ Authorization =" Basic $base64AuthInfo" }
$pipeline = Invoke-RestMethod -Uri $url -Headers $head -Method Get
$Pipeline.description = "<Modify the release description>"
$body = $Pipeline | ConvertTo-Json -Depth 100
$url2=https://vsrm.dev.azure.com/<orgName>/<ProjectName>/_apis/release/releases/$(Release.ReleaseId)?api-version=6.0
$resp = Invoke-RestMethod -Uri $url2 -Method Put -Headers $head -Body $body -ContentType application/json

How to modify Azure DevOps release definition variable from a release task?

What is the easiest way to get key rotation to work for azure storage accounts from a AzureDevOps relase task? The current plan is to re-generate the old key after release to invalidate it, and have a fresh key that can be used on next deployment. But to get that to work it seems like I at least need to store the name of the key to use in a release variable.
I had a look at he logging tasks (https://github.com/Microsoft/azure-pipelines-tasks/blob/master/docs/authoring/commands.md), but that only changes the value in the current release and does not modify the release definition.
You can use the REST API (Definitions - Update) to update the value of the release definition variable from a release task.
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Grant Project Collection Build Service (xxx) account the edit
release pipeline permission. (Select the release pipeline --> ... --> Security --> Edit release definition set to Allow)
Add a PowerShell task in your release pipeline
Run inline script: (Update the value of variable v1030 in below sample)
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/Release/definitions/$($env:RELEASE_DEFINITIONID)?api-version=5.0-preview.3"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named v1030 to its new value 1035
$pipeline.variables.v1030.value = "1035"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'v1030' is updated to" $updatedef.variables.v1030.value
write-host "=========================================================="
Here is a much cleaner and better solution that also allows for multiple builds being triggered simultaneously.
https://tsuyoshiushio.medium.com/how-to-pass-variables-with-pipeline-trigger-in-azure-pipeline-5771c5f18f91
Essentially your triggering build produces artifacts that your triggered build reads and turns into variables.
Still not at all great, but better than nothing and better than REST calls setting static global variable groups.
The other answer above talks about how to update Release pipelines.
If you would like to update a Build Pipeline's variables, here is how you do that:
Edit build pipeline
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Go to Manage Security -> Users -> Select Project Collection Build Service (YOUR TEAM NAME HERE)
Change "Edit Build Definitions" to Allow
Now add a powershell stage - 2.x - inline script called Update variables.
Script inline contents:
$api_version='5.0-preview.6'
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$(System.DefinitionId)?api-version=${api_version}"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update variables as desired here:
$pipeline.variables.mavenBuildVersionPatch.value = "2401"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Variable mavenBuildVersionPatch is updated to" $updatedef.variables.mavenBuildVersionPatch.value
write-host "=========================================================="
Take note of the API version in this script is 5.0-preview.6.
If your version of Azure Devops is newer, you may need to update this in the future.
Save build pipeline.
Now when you run job, after the job completes this powershell stage, this variable will be set.
Important: If you want to update a variable then make the updated variable for other pipeline stages, then you do it with a powershell stage with the following inline script:
$mavenBuildVersionPatch = [int]"$(mavenBuildVersionPatch)"
$mavenBuildVersionPatch = $mavenBuildVersionPatch + 1
Write-Host "##vso[task.setvariable variable=mavenBuildVersionPatch;]$mavenBuildVersionPatch"
This example would take our existing patch number and increment it by 1.
This does not save the variable at the end of the job, you still need to do that with another powershell script if desired.

How to Increase/Update Variable Group value using Azure Devops Build Definition?

Am working on Azure Devops CI&CD. Here, my release name must be in Version number using tags. Am getting this with the help of Variable Groups, by adding tags and value to it. Here am getting the tags value as constant like a static for every release like 1.1,1.2,1.3 etc.
Now am trying to increase/update my tag value dynamically for every new release triggered after completion of my Build Definition successfully which looks like 1.1,1.2,2.1,2.2,3.1,3.2 etc.It is possible with the help of statically by variable group, but manually we need to update it.
Is it possible to Increase/Update the tags value in Variable Group with the Build Definition tasks or other process.If possible, please suggest me to "How to done this?"
You can overwrite/update the value of the variables by using the logging command to set the variables again in Azure Devops Build pipleline:
Write-Host "##vso[task.setvariable variable=testvar;]testvalue"
To increase the value dynamically, you need to use the token $(Rev:.r). You can custom the variables based on the $(Build.BuildNumber) or $(Release.ReleaseName)as they will increase the value dynamically...
Just reference this thread to custom the variables:https://github.com/MicrosoftDocs/vsts-docs/issues/666#issuecomment-386769445
UPDATE:
If you just want to update the value of the variables which defined in a specific Variable Group, then you can call REST API in build pipeline to achieve that:
PUT https://{account}.visualstudio.com/{ProjectName or ID}/_apis/distributedtask/variablegroups/{Variable Group ID}?api-version=5.0-preview.1
Content-Type: application/json
Request Body:
{"id":2,"type":"Vsts","name":"VG0926","variables":{"TEST0926":{"isSecret":false,"value":"0930"}}}
UPDATE2:
You can write a PowerShell script to call the REST API, then add a PowerShell task to run the script in your build pipeline: (Use the OAuth token to access the REST API)
Below sample for your reference:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/variablegroups/{Variable Group ID}?api-version=5.0-preview.1"
Write-Host $url
function CreateJsonBody
{
$value = #"
{"id":2,"type":"Vsts","name":"VG0926","variables":{"TEST0926":{"isSecret":false,"value":"0930"}}}
"#
return $value
}
$json = CreateJsonBody
$pipeline = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "New Variable Value:" $pipeline.variables.TEST0926.value
UPDATE3:
Well, tested again, below scripts works for me as well. You can try it, just replace the parameters accordingly:
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "","PAT here")))
$url = "https://dev.azure.com/xxx/Test0924/_apis/distributedtask/variablegroups/1?api-version=5.0-preview.1"
$json = '{"id":1,"type":"Vsts","name":"VG0928","variables":{"TEST0928":{"isSecret":false,"value":"0931"}}}'
$pipeline = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
Write-Host "New Variable Value:" $pipeline.variables.TEST0928.value
You can simply update any number of variables in an Azure Devops variable group using its built-in az pipelines variable-group variable update command. You can use this command in a script task in the pipeline definition as shown below.
ps: Replace all the upper case values with corresponding values except the SYSTEM_ACCESSTOKEN.
variables:
- group: VARIABLE_GROUP_NAME
jobs:
- job: UpdateVarGroup
steps:
- script: |
newValue="This is the updated value"
echo $SYSTEM_ACCESSTOKEN | az devops login
az pipelines variable-group variable update --group-id $(group_id) \
--name NAME_OF_THE_VARIABLE \
--value "${newValue}" \
--org https://dev.azure.com/YOUR_ORGANIZATION_NAME \
--project AZURE_DEVOPS_PROJECT
displayName: 'Update variable inside a variable group'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
In order for this example to work, you need to have a variable inside your variable group called group_id. The value of that needs to be set to the group id of the variable group, which can be obtained by simply looking at the url of your variable group. (The group id is the value for variableGroupId in the url of your browser when you are inside the variable group)
System.AccessToken is required for az devops login
I used this task to update the value of my variables inside my group.
Shared variable updater (preview)
Dont forget to set those settings :
Requires 'Allow scripts to access the OAuth token' in agent job additional options
Set administrator role to 'Project Collection Build Service' in the variable group.
In case of using a YAML pipeline
When using a YAML pipeline, the OAuth token is automatically added (no need for step 1 above), but requires a bit of work to make accessible for the powershell script. Use the guidance here to be able to use the token.
If you want to update a variable group's value, use the REST.API methods.
Rather than constructing the PUT request body manually, use a GET first to get the original JSON, update only what you need, then replay it as a PUT. I used to keep the variable group id also as a variable in that group to avoid hard-coding.
variables:
- group: MyVarGroup
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/variablegroups/$(VariableGroupId)?api-version=6.0-preview.1"
$header = #{
"Authorization" = "Bearer $(System.AccessToken)"
}
$def = Invoke-RestMethod -Uri $url -Headers $header
$def.variables.MyTestVar.value = "NewValue"
$body = #($def) | ConvertTo-Json -Depth 100 -Compress
$def = Invoke-RestMethod -Method 'Put' -Uri $url -ContentType 'application/json' -Headers $header -Body $body
You can overwrite variables using the REST API with a PowerShell task without needing to create a PAT.
You'll need to first going into the agent job and set "Allow scripts to access OAuth token".
You'll need too go to your variable group and add the 'Project Collection Build Service' as an administrator.
Now you can call the Rest API using the OAuth Bearer token. Code for reference:
$id = <variable group id>
# This is using some environment variables provided by the pipeline to build the URL
$url = ("$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/variablegroups/{0}?api-version=5.0-preview" -f $id)
# You might find it useful to us a GET method to grab the variable group, update it and then convert it to this json string rather than doing this here
$json = '{"id":$id,"type":"Vsts","name":"<Variable Group Name>","<Variable Name":{"ThisIsMyVariable":{"isSecret":false,"value":"20"}}}'
$pipeline = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
I made this task to manage variable groups from Pipelines:
ManageVariableGroupTask
Using this you can Create/Update/Delete variable groups and do the same operations on variables contained in them.
Hope that's helpfull!