Azure DevOps get Deployment Agent status during release - azure-devops

I'm trying to get the status of agents in a deployment pool at release time.
The use case is I have 2 servers with shared disk, I want the release to run on one server only. I have two Deployment groups that run based on a custom condition:
eq(variables['DeployGroupSelector'], '1')
With a job that runs prior to those that will determine the value of the DeployGroupSelector var, essentially a case statement.
In the job that sets the var, I'm trying to reach out to the Azure DevOps REST API:
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$url = "https://dev.azure.com/$($organization)/_apis/distributedtask/pools/$($poolId)/agents?api-version=5.1"
$response = Invoke-RestMethod $url -Headers $headers -Verbose
write-host "Output: $response"
$status = ($response.value | where {$_.name -eq $($env:primaryPoolName)}).status
if($status -eq "online")
{
Write-Output("##vso[task.setvariable variable=DeployGroupSelector;]1")
}
else
{
Write-Output("##vso[task.setvariable variable=DeployGroupSelector;]2")
}
For the group containing the script above the "Allow scripts access to the OAuth token" box is checked.
When I run this powershell locally using a PAT it returns data. When I run the release in ADO, it hits the service, but returns an empty data set:
2019-10-07T14:16:18.8942915Z VERBOSE: GET https://dev.azure.com/xxxxxx/_apis/distributedtask/pools/13/agents?api-version=5.1 with 0-byte payload
2019-10-07T14:16:19.3235204Z VERBOSE: received 22-byte response of content type application/json
2019-10-07T14:16:19.9626359Z VERBOSE: Content encoding: utf-8
2019-10-07T14:16:19.9835101Z Output: #{count=0; value=System.Object[]}
I have tried giving the "Project Collection Build Service Accounts" group read access to pools and groups, I even tried bumping it up to user. I tried adding the build service accounts group to release administrators. I even tried using the old url format just in case.
Added picture of data returned from powershell:
UPDATE: Just to further rule out an issue with how I was using the token, I added a second powershell task to the task group in question. This script hits a different segment of the AzDO Rest API (below). This successfully gets a response. So the OAuth token is working, it just doesn't seem to have access to the entire API.
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$url = "https://dev.azure.com/$($organization)/$($project)/_apis/git/repositories?api-version=5.1"
$response = Invoke-RestMethod $url -Headers $headers -Verbose
write-host "Output: $($response)"
Response: Output: #{value=System.Object[]; count=10}

Since you're using the System_AccessToken variable, did you also enable the "Allow scripts to access OAuth token" checkbox in the agent job? https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=classic#systemaccesstoken This link shows where it is in build but you'll find it at the bottom of the Agent Job panel in release. If that's not checked, that may be why you're getting an empty response.

Had exactly the same issue. Were considered two options, the same as you have been trying.
System.AccessToken
PAT
Problem was solved by putting PAT into a KeyVault and using it as a basic auth token for REST API call in a pipeline.
My suggestion is that it seems expected and right behavior. Why do I think so? From Azure DevOps point of view there are two levels in our case Organization's level and Project's level. You can notice the difference by URI that were used:
$url = "https://dev.azure.com/$($organization)/_apis/distributedtask/pools/$($poolId)/agents?api-version=5.1"
$url = "https://dev.azure.com/$($organization)/$($project)/_apis/git/repositories?api-version=5.1
From security point of view it's a bad practice, let entities from lower layers, which is project in our case, access and manipulate on the higher layer, which is organization in our case.
As a conclusion I would say that SystemToken and PAT in essence have slightly different nature, one is devoted for agent, and another for one's personal profile.

Related

Azure DevOps Pipeline Not Displaying All Runs

I'm having issues when trying to display all runs for one of my Pipelines. I'm running on premise Azure Devops 2020 and I get a perpetual spinning Loading dialogue. I don't have this issue in any of my other pipelines. It seems that if I filter by run state in the problematic pipeline I am able to get past this and view some of the runs but for other states in my case "Succeeded" and "Succeeded with Issues" I continue to get the spinning loading symbol. Any advice?
Here are some suggestions:
Suggestion 1
Clean the cache or load the page from a different brower and restart the Azure DevOps Server and the SQL Server machine.
Suggestion 2
Create a new pipeline that has the same settings as the affected pipeline.
Suggestion 3
You can use the REST API Builds - List to get all runs of your pipeline.
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds?api-version=6.0
Because the cause of this question may be that the runs list is too long. You can use the REST API Builds-Delete to delete some of the runs you don't need to see whether the question can be solved.
DELETE https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}?api-version=6.0
Suggestion 4
You can refer to a similar question on the Developer Community. The Jan Selbach's comment offers a solution.
1.Run the following SQL script to find the LeaseId of duplicate rows. Please change collection Db name according to your’s.
SELECT LeaseId
FROM [AzureDevOps_DefaultCollection].[Build].[tbl_RetentionLease]
WHERE
partitionId > 0 AND
LeaseId NOT IN (
SELECT MIN(LeaseId) as LeaseId
FROM [AzureDevOps_DefaultCollection].[Build].[tbl_RetentionLease]
WHERE PartitionId > 0
GROUP BY OwnerId,RunId
)
2.Concatenate the LeaseIds into a comma separated list
3.Run the following PowerShell script by filling in the org, project, pat, and leaseId by passing in the comma separated list of LeaseIds
(remember if on-prem change the https://dev.azure.com in the $uri to
your on-prem server)
$org = "";
$project = "";
$pat = "";
$encoded = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes([string]::Format("{0}:{1}", "", $pat)));
$accessToken = "Basic $encoded";
$leaseId = "";
$uri = "https://dev.azure.com/$org/$project/_apis/build/retention/leases?ids=$leaseId&api-version=6.0-preview";
try {
$response = Invoke-RestMethod -uri $uri -method DELETE -Headers #{ Authorization = $accessToken } -ContentType "application/json"
$response
}
catch {
$errorDetails = ConvertFrom-Json $_.ErrorDetails
Write-Host "StatusCode: $($_.Exception.Response.StatusCode)`nExceptionType: $($errorDetails.typeKey)`nExceptionMessage: $($errorDetails.message)"
#StackTrace: $($errorDetails.stackTrace)"
}

Retrieving info from Azure Devops with Powershell

I'm trying to automate the deployment of releases in Azure Devops using the Azure Devops API with Powershell. I'm able to retrieve the last release definition, but I'm not able to extract a release for a specific environment.
I'm looking to retrieve the last 'prod' release in order to obtain the 'release ID', and then deploy the release to UAT or DEV using a HTTP 'PATCH' request.
I can hit a URL manually (example below), which gives me the last release for DEV (definitionEnvironmentId=1), and I know I could easily changed the environment ID to '12' for production, but I'd like to get this via an API call.
https://{redacted url}/{redacted}/{redacted}/_apis/release/deployments?api-version=6.0&definitionId=1&definitionEnvironmentId=1&$top=1
Any ideas if this is possible as I can't find any info on it?
You were using the correct Deployments - List Rest api. The reason you didnot get any info from the api call via powershell is because you didnot escape the special character $ for $top parameter in the url.
$ is a special character in powershell. You should escape it by adding a back tick '`' in the url. See below:
#add ` to $top parameter
$url = "https://{redacted url}/_apis/release/deployments?definitionId=3&definitionEnvironmentId=5&`$top=1&api-version=6.0"
$connectionToken="PAT"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$response= Invoke-RestMethod -Uri $url -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
$releaseId = $response.value[0].release.id

Permissions required for these TFS (on premise) REST urls?

I am trying to automate some jobs that require information about agent pools and agents and while the job script works just fine for regular users of my TFS collection it fails miserably for a service account.
My job script tries to access urls
http://<instance>/<collection>/_apis/distributedtask/pools
http://<instance>/<collection>/_apis/distributedtask/pools/<pool>/agents
Initially my service account got a response like
TF400813: The user '<service account>' is not authorized to access this resource
The service account was previously not member of any TFS related AD group but after creating a new group and adding it to 'Project Collection Valid Users' the call does not fail but the response does not include any pool information still.
If I modify the service account interactive logins the GUI for agents in the agent pool shows no information but the hint
no agents are registered or you do not have permission to view the agents
suggests that permission is missing.
I have tried to add the service account to various groups in TFS like Project Collection Administrators, Project Collection Build Administrators, etc. all to no avail.
So in short, what permissions does a service account need to retrieve information from the urls mentioned in the start?
I can reproduce this issue, it does not work even change the regular users as the service account.
So, as a workaround for now you can call the REST API using the regular users or PAT in your script.
Below PowerShell script works for me:
$user = "Domain\username"
$token = "password"
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
$uri = "http://SERVER:8080/tfs/_apis/distributedtask/pools/1/agents/5"
$result = Invoke-RestMethod -Uri $uri -Method Get -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
Write-Host "result = $($result | ConvertTo-Json -Depth 100)"

VSTS Personal Access Token (PAT) vs OAUTH: different results of query

I am using a PowerShell task to among others query a TFVC repository for changes after a certain point of time. To develop locally I created a PAT related to my user. In the release definition I enabled Allow scripts to access OAUTH token. If I now execute the release definition with the PAT respectively the OAUTH I get different results for existing changes (e.g. count is 1 for PAT and 0 for OAUTH). The queries are exactly the same:
https://xxx.visualstudio.com/xxx/_apis/tfvc/changesets?searchCriteria.itemPath=$projectPath&searchCriteria.fromDate=$cloudVersionTimestampUTC&api-version=4.1
Here the code to execute the query:
Write-Host "Get Request with the URI '$uri'"
if ($localDevelopment) {
$GetResponse = Invoke-RestMethod `
-Uri $uri `
-Headers #{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($env:SYSTEM_ACCESSTOKEN)")) }
Write-Host "Requested last changes for local development: $($GetResponse | ConvertTo-Json -Depth 100)"
}
else {
$GetResponse = Invoke-RestMethod `
-Uri $uri `
-Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN" }
Write-Host "Requested last changes: $($GetResponse | ConvertTo-Json -Depth 100)"
}
Does someone know why this is the case?
Thanks
Edits:
here the permissions of the Project Collection Build Service (xxx) account:
It is also important to mention that we don't use hosted agents but custom ones.
The problem is the different handling of timestamps when comparing PAT and OAUTH. As you can see I use the variable $cloudVersionTimestampUTC for fromDate. When I set the timestamp I did not add Z (like MM-dd-yyyy HH:mm:ss instead of MM-dd-yyyy HH:mm:ssZ). PAT uses the given timestamp as UTC, OAUTH interprets it as local time (but I am not sure of what - e.g. the default profile) when Z is not provided. This of course led to a different number of changesets (if the missing changeset was in the difference between UTC and local time).
In the end I had to be explicit about the timestamps. So what I did was to get the timestamp as string that was stored as tag in the Azure Resource and do the following:
$cloudVersionTimestampUTC = [DateTime]($cloudJobDefinition.Tags.'version-timestamp')
$cloudVersionTimestampUTC = (Get-Date -Date $cloudVersionTimestampUTC).ToString("MM-dd-yyyy HH:mm:ssZ")
Then I was able to make the REST Call and retrieve the same number of changesets.

VSTS - Where has 'Allow Scripts to Access OAuth Token' gone?

I have VSTS builds that use the Powershell script to access the VSTS API, similar to that detailed in Microsoft's Documentation.
The document states:
To enable your script to use the build process OAuth token, go to the Options tab of the build definition and select Allow Scripts to Access OAuth Token.
After you've done that, your script can use to SYSTEM_ACCESSTOKEN
environment variable to access the VSTS REST API. For example:
Example:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build-release/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=2.0"
Write-Host "URL: $url"
$definition = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Definition = $($definition | ConvertTo-Json -Depth 1000)"
As detailed in Microsofts Documentation, I'm sure you used to go to Build > Options and click 'Allow Scripts to Access OAuth Token', but it is no longer there (see pic below).
When I try a build I get the following (which doesn't happen for my old builds):
Invoke-RestMethod :
{"$id":"1","innerException":null,"message":"TF400813: The user '' is
not authorized to access this
In addition, when I clone a build (which worked nicely when I did it 3 months ago), the property is set to false (it's set to true on the original).
UPDATE: If I export from VSTS, change that property and import again, it works, but I really need a solution without this sort of manual intervention.
How should this be done now please?
It is available in the agent phase now