How to add a new dynamically discoverable capability to an agent? - azure-devops

Updates
I am going to propose a Capabilities Provider here as an update to my post.
If you need more details please let me know.
We currently have a bunch of shipped Capabilities Providers in the agent source code:
https://github.com/microsoft/azure-pipelines-agent/tree/master/src/Microsoft.VisualStudio.Services.Agent/Capabilities
Agent
Environment
Nix
PowerShell
What is being proposed is one additional Provider named ExecutableCapabilitiesProvider.
This new ExecutableCapabilitiesProvider will probably have a config file which can be edited on the agent machine.
The format of this file could probably be:
#name,executable
pip,pip3 freeze
xyz,/usr/bin/xyz-runner
abc,sh -C "ls -l /blah/blah"
As the maintainer of the self-hosted pool, I would configure this file with entries suiting me and have the agent run it as it starts. This way I am not hard-coding any values for my capabilities but rather those be determined at the start up.
And I would go one step further and add a new API call to add capabilities which is more flexible than the current one asking for name/values. An example, would be to change the parameters to Name, Provider, Params:
efg, NixProvider, /path/to/file/efg
klm, ExecutableCapabilitiesProvider, /usr/bin/klm -a -b -c
Original Post
I'd like to make my agents report on new capabilities which are not static but rather result of a command or something similar? How can I do that?
Our agents run on linux boxes.
To be specific, I'd like to have a new capability called pip-packages and the value for that is the result of the command pip freeze executed on the shell.

If you mean to add User-defined capabilities, then you can write a script to call the REST API to update the agent capabilities.
PUT https://dev.azure.com/{organization}/_apis/distributedtask/pools/{poolid}/agents/{agentid}/usercapabilities?api-version=5.0
Request body:
{"pip-packages": "xxxx"}
For example, you can set a variable and run command pip freeze and export the response as the value of that variable, then update the agent capability by calling the REST API:
Below PowerShell sample for your reference :
Param(
[string]$collectionurl = "https://dev.azure.com/{organization}",
[string]$poolid = "14",
[string]$agentid = "16",
[string]$user = "user",
[string]$token = "PAT/Password"
)
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
# Run pip freeze command and get response as the value of the variable $pipfreeze (Just for your reference here, you need to extract the value with running the commands)
$pipfreeze = "response of pip freeze"
# Create json body with that value
$baseUri = "$collectionurl/_apis/distributedtask/pools/$poolid/agents/$agentid/usercapabilities?api-version=5.0"
function CreateJsonBody
{
$value = #"
{"pip-packages":"$pipfreeze"}
"#
return $value
}
$json = CreateJsonBody
# Update the Agent user capability
$agentcapability = Invoke-RestMethod -Uri $baseUri -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
write-host "=========================================================="
Write-host "userCapabilities :" $agentcapability.userCapabilities.'pip-packages'
write-host "=========================================================="

Related

Using semaphore-like conditions for pipeline job queueing

we use floating licenses for our more expensive compilers/tools, so that we can do local development as well as our production builds. The license manager (flexLM) has an api we can query, so we could block the license. However, I cannot find a mechanism by which I can cause my pipeline to queue based on the state of an auxiliary variable or return value of a script or something like that.
This means that I can launch the build on any machine on which the compiler is installed but it will then fail if the license is not available and I will have to relaunch the pipeline. If I did that automatically I would effectively just block that machine until the license becomes available.
Is there anything I have missed that could achieve a "queueing until license becomes available" kind of thing?
Thank you,
Manuel
We can add the first task power shell in the pipeline definition and define new variable in the variable tab such as Value:true, then add script to check the license status, if the license is available, set the variable Value to true, if the license is not available, set the variable Value to False. Then add condition eq(variables['{variable name}'], '{variable value}') in the Second task.
After configure, if your license is available, the pipeline will run successfully.
Or we could check the license first, then call the below script to queue build pipeline.
$token = "$(pat)"
$url = "https://dev.azure.com/{Org name}/{project name}/_apis/build/builds?api-version=6.1-preview.6"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$JSON = #"
{
"definition": {
"id": {Build definition ID}
}
}
"#
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method Post -ContentType application/json -body $JSON
write-host $response

How to read environment variables in azure devops pipeline using powershell?

I created a PowerShell job and used the below code to set the environment variable in the azure pipeline using Powershell.
[Environment]::SetEnvironmentVariable("key", "value")
I can print the value using the $env:key in the same job itself.
But when I tried to display the value using $env:key in the next job nothing is printed. How to use the above environment variable through out the azure pipeline. Is there any other way to set and read custom environment variables.
you pretty much have to either use library variable groups (or sets, dont remember the name) or you have to use a specific way to share variables across jobs:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#use-output-variables-from-tasks
According to this, using outputs in a different job is not supported in Classic UI Format.
As workarounds in this scenario, you can share variables via Pipeline Variables(share variables across jobs in same pipeline) or Variable Groups(share variables across pipelines that use same Variable Group, it also works across jobs).
Since you only want to share variables across jobs in same pipeline, pipeline variable is enough for you.
1.You can set a key variable in pipeline variables:
2.Add one Powershell Inline task with content below in your first job:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable to its new value
$pipeline.variables.key.value = "value"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value
write-host "=========================================================="
3.Run the pipeline we can find the value of key variable is successfully updated:
So you can run the ps script in first job to update the value of key variable, then all next jobs can access the updated variable easily.
Note:
For the script itself, you only need to change lines $pipeline.variables.key.value = "value"(necessary) and Write-host "The value of Varialbe key is updated to" $updatedef.variables.key.value(optional).
If I want to set the variable named MyTest to value MyValue, the lines should be $pipeline.variables.MyTest.value = "MyValue" and Write-host "The value of Varialbe MyTest is updated to" $updatedef.variables.MyTest.value.
To make sure the ps task in one job can access OAuth Token, we should Allow Scripts to Access OAuth Token. Click the agent job name and check the box:
To enable the pipeline has the permission to update pipeline variable (edit build pipeline), go pipeline security to set the Edit build pipeline allow for user xxx(ProjectName) build service.
Hope all above helps :)

Store result of Azure CLI task into release variable

I have a release pipeline with multiple jobs.
On the first agent job there is an Azure CLI task that retrieves the keys of a storage account on azure.
The command it executes is :
az storage account keys list --account-name "$(diagnosticsStorageAccountName)" --resource-group "$(resourceGroup)"
What I want to do is store the result of this command and utilize it in a task that is running under a deployment group job.
I've already looked into these resource :
Set Output Variable in Azure CLI task on VSTS
How to modify Azure DevOps release definition variable from a release task?
I've tried the first one but I didn't get it working.
I didn't bothered with the second because it seems way to hacky.
Is there any way do achieve this that isn't hacky ?
The output values didn't get stored properly.
The output of az storage account keys list --account-name "$(diagnosticsStorageAccountName)" --resource-group "$(resourceGroup)"is spread over multiple lines when you use the following syntax:
echo "##vso[task.setvariable variable=testvar;]%myvar%"
So one of the problems was that only the first line of the JSON array was being stored into the variable.
I solved this problem in the following way:
keys=`az storage account keys list --account-name "$(diagnosticsStorageAccountName)" --resource-group "$(resourceGroup)"`
taskvariable="##vso[task.setvariable variable=_[tempVariable];]"
echo $taskvariable$keys
According to documentation echo "##vso[task.setvariable variable=testvar output=true;]%myvar% should make the variable available for the whole release. Unfortunately I had no luck with this.
I overcame this using a Powershell task under the same Agent Job (Ubuntu 16.0) :
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/Release/definitions/$($env:RELEASE_DEFINITIONID)?api-version=5.1"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $(System.AccessToken)"
}
#Parsing JSON string from previous task
$keys = $(#"
$(_tempVariable)
"# | ConvertFrom-Json)
# Assignment of variable value
$pipeline.variables.[variableName].value = $keys[0].value
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $(System.AccessToken)"}
write-host "=========================================================="
Write-host "The value of Variable '[variableName]' is updated to" $updatedef.variables.[variableName].value
write-host "=========================================================="
Please note that in order to get this working there are a couple of things you need to do.
First you need to allow access to the OAuth token on the Agent Job.
On top of that you need to give the "Project Collection Build Service".
Click on security on the release and click on the "Project Collection Build Service" user.
Change the values for the "Edit release" and "Manage release" to allow and save the changes.

How to Increase/Update Variable Group value using Azure Devops Build Definition?

Am working on Azure Devops CI&CD. Here, my release name must be in Version number using tags. Am getting this with the help of Variable Groups, by adding tags and value to it. Here am getting the tags value as constant like a static for every release like 1.1,1.2,1.3 etc.
Now am trying to increase/update my tag value dynamically for every new release triggered after completion of my Build Definition successfully which looks like 1.1,1.2,2.1,2.2,3.1,3.2 etc.It is possible with the help of statically by variable group, but manually we need to update it.
Is it possible to Increase/Update the tags value in Variable Group with the Build Definition tasks or other process.If possible, please suggest me to "How to done this?"
You can overwrite/update the value of the variables by using the logging command to set the variables again in Azure Devops Build pipleline:
Write-Host "##vso[task.setvariable variable=testvar;]testvalue"
To increase the value dynamically, you need to use the token $(Rev:.r). You can custom the variables based on the $(Build.BuildNumber) or $(Release.ReleaseName)as they will increase the value dynamically...
Just reference this thread to custom the variables:https://github.com/MicrosoftDocs/vsts-docs/issues/666#issuecomment-386769445
UPDATE:
If you just want to update the value of the variables which defined in a specific Variable Group, then you can call REST API in build pipeline to achieve that:
PUT https://{account}.visualstudio.com/{ProjectName or ID}/_apis/distributedtask/variablegroups/{Variable Group ID}?api-version=5.0-preview.1
Content-Type: application/json
Request Body:
{"id":2,"type":"Vsts","name":"VG0926","variables":{"TEST0926":{"isSecret":false,"value":"0930"}}}
UPDATE2:
You can write a PowerShell script to call the REST API, then add a PowerShell task to run the script in your build pipeline: (Use the OAuth token to access the REST API)
Below sample for your reference:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/variablegroups/{Variable Group ID}?api-version=5.0-preview.1"
Write-Host $url
function CreateJsonBody
{
$value = #"
{"id":2,"type":"Vsts","name":"VG0926","variables":{"TEST0926":{"isSecret":false,"value":"0930"}}}
"#
return $value
}
$json = CreateJsonBody
$pipeline = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "New Variable Value:" $pipeline.variables.TEST0926.value
UPDATE3:
Well, tested again, below scripts works for me as well. You can try it, just replace the parameters accordingly:
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "","PAT here")))
$url = "https://dev.azure.com/xxx/Test0924/_apis/distributedtask/variablegroups/1?api-version=5.0-preview.1"
$json = '{"id":1,"type":"Vsts","name":"VG0928","variables":{"TEST0928":{"isSecret":false,"value":"0931"}}}'
$pipeline = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
Write-Host "New Variable Value:" $pipeline.variables.TEST0928.value
You can simply update any number of variables in an Azure Devops variable group using its built-in az pipelines variable-group variable update command. You can use this command in a script task in the pipeline definition as shown below.
ps: Replace all the upper case values with corresponding values except the SYSTEM_ACCESSTOKEN.
variables:
- group: VARIABLE_GROUP_NAME
jobs:
- job: UpdateVarGroup
steps:
- script: |
newValue="This is the updated value"
echo $SYSTEM_ACCESSTOKEN | az devops login
az pipelines variable-group variable update --group-id $(group_id) \
--name NAME_OF_THE_VARIABLE \
--value "${newValue}" \
--org https://dev.azure.com/YOUR_ORGANIZATION_NAME \
--project AZURE_DEVOPS_PROJECT
displayName: 'Update variable inside a variable group'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
In order for this example to work, you need to have a variable inside your variable group called group_id. The value of that needs to be set to the group id of the variable group, which can be obtained by simply looking at the url of your variable group. (The group id is the value for variableGroupId in the url of your browser when you are inside the variable group)
System.AccessToken is required for az devops login
I used this task to update the value of my variables inside my group.
Shared variable updater (preview)
Dont forget to set those settings :
Requires 'Allow scripts to access the OAuth token' in agent job additional options
Set administrator role to 'Project Collection Build Service' in the variable group.
In case of using a YAML pipeline
When using a YAML pipeline, the OAuth token is automatically added (no need for step 1 above), but requires a bit of work to make accessible for the powershell script. Use the guidance here to be able to use the token.
If you want to update a variable group's value, use the REST.API methods.
Rather than constructing the PUT request body manually, use a GET first to get the original JSON, update only what you need, then replay it as a PUT. I used to keep the variable group id also as a variable in that group to avoid hard-coding.
variables:
- group: MyVarGroup
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/variablegroups/$(VariableGroupId)?api-version=6.0-preview.1"
$header = #{
"Authorization" = "Bearer $(System.AccessToken)"
}
$def = Invoke-RestMethod -Uri $url -Headers $header
$def.variables.MyTestVar.value = "NewValue"
$body = #($def) | ConvertTo-Json -Depth 100 -Compress
$def = Invoke-RestMethod -Method 'Put' -Uri $url -ContentType 'application/json' -Headers $header -Body $body
You can overwrite variables using the REST API with a PowerShell task without needing to create a PAT.
You'll need to first going into the agent job and set "Allow scripts to access OAuth token".
You'll need too go to your variable group and add the 'Project Collection Build Service' as an administrator.
Now you can call the Rest API using the OAuth Bearer token. Code for reference:
$id = <variable group id>
# This is using some environment variables provided by the pipeline to build the URL
$url = ("$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/variablegroups/{0}?api-version=5.0-preview" -f $id)
# You might find it useful to us a GET method to grab the variable group, update it and then convert it to this json string rather than doing this here
$json = '{"id":$id,"type":"Vsts","name":"<Variable Group Name>","<Variable Name":{"ThisIsMyVariable":{"isSecret":false,"value":"20"}}}'
$pipeline = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
I made this task to manage variable groups from Pipelines:
ManageVariableGroupTask
Using this you can Create/Update/Delete variable groups and do the same operations on variables contained in them.
Hope that's helpfull!

TFS: Best way to trigger build on server restart, or on Windows Updates installation

In short, the requirement is to verify that our latest released software can be built and then installed after the latest Windows updates and/or other patches were applied. So the build server VM(s) will be configured just for this purpose and the build only needs to run after an update.
Since such updates usually are followed with a restart, I am thinking of a server restart event triggering a build and deployment. Does such option exist in TFS 2017?
If there is no way to do it through TFS then, I guess, a PowerShell script that runs on startup should work?
No such a build-in function to achieve that. However create a PowerShell script that runs on startup should work. Just as Jessehouwing said, you can create the script with the REST API to trigger builds.
Create a script to trigger the specific build definition. (Reference below sample)
Run the script on startup:
How to run a batch file each time the computer boots
How to schedule a Batch File to run automatically in Windows
10/8/7
Param(
[string]$collectionurl = "http://server:8080/tfs/DefaultCollection",
[string]$projectName = "ProjectName",
[string]$keepForever = "true",
[string]$BuildDefinitionId = "34",
[string]$user = "username",
[string]$token = "password"
)
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
function CreateJsonBody
{
$value = #"
{
"definition": {
"id": $BuildDefinitionId
},
"parameters": "{\"system.debug\":\"true\",\"BuildConfiguration\":\"debug\",\"BuildPlatform\":\"x64\"}"
}
"#
return $value
}
$json = CreateJsonBody
$uri = "$($collectionurl)/$($projectName)/_apis/build/builds?api-version=2.0"
$result = Invoke-RestMethod -Uri $uri -Method Post -Body $json -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
There is no existing trigger that handles this, but there is a simple REST API to query and trigger builds.
It would be easy to create an on startup job in the task scheduler, use the REST API to query a list of Build Definitions based on a certain name or tag and then queue it.
List build definitions
Queue a build