VSTS Personal Access Token (PAT) vs OAUTH: different results of query - powershell

I am using a PowerShell task to among others query a TFVC repository for changes after a certain point of time. To develop locally I created a PAT related to my user. In the release definition I enabled Allow scripts to access OAUTH token. If I now execute the release definition with the PAT respectively the OAUTH I get different results for existing changes (e.g. count is 1 for PAT and 0 for OAUTH). The queries are exactly the same:
https://xxx.visualstudio.com/xxx/_apis/tfvc/changesets?searchCriteria.itemPath=$projectPath&searchCriteria.fromDate=$cloudVersionTimestampUTC&api-version=4.1
Here the code to execute the query:
Write-Host "Get Request with the URI '$uri'"
if ($localDevelopment) {
$GetResponse = Invoke-RestMethod `
-Uri $uri `
-Headers #{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($env:SYSTEM_ACCESSTOKEN)")) }
Write-Host "Requested last changes for local development: $($GetResponse | ConvertTo-Json -Depth 100)"
}
else {
$GetResponse = Invoke-RestMethod `
-Uri $uri `
-Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN" }
Write-Host "Requested last changes: $($GetResponse | ConvertTo-Json -Depth 100)"
}
Does someone know why this is the case?
Thanks
Edits:
here the permissions of the Project Collection Build Service (xxx) account:
It is also important to mention that we don't use hosted agents but custom ones.

The problem is the different handling of timestamps when comparing PAT and OAUTH. As you can see I use the variable $cloudVersionTimestampUTC for fromDate. When I set the timestamp I did not add Z (like MM-dd-yyyy HH:mm:ss instead of MM-dd-yyyy HH:mm:ssZ). PAT uses the given timestamp as UTC, OAUTH interprets it as local time (but I am not sure of what - e.g. the default profile) when Z is not provided. This of course led to a different number of changesets (if the missing changeset was in the difference between UTC and local time).
In the end I had to be explicit about the timestamps. So what I did was to get the timestamp as string that was stored as tag in the Azure Resource and do the following:
$cloudVersionTimestampUTC = [DateTime]($cloudJobDefinition.Tags.'version-timestamp')
$cloudVersionTimestampUTC = (Get-Date -Date $cloudVersionTimestampUTC).ToString("MM-dd-yyyy HH:mm:ssZ")
Then I was able to make the REST Call and retrieve the same number of changesets.

Related

Azure DevOps get Deployment Agent status during release

I'm trying to get the status of agents in a deployment pool at release time.
The use case is I have 2 servers with shared disk, I want the release to run on one server only. I have two Deployment groups that run based on a custom condition:
eq(variables['DeployGroupSelector'], '1')
With a job that runs prior to those that will determine the value of the DeployGroupSelector var, essentially a case statement.
In the job that sets the var, I'm trying to reach out to the Azure DevOps REST API:
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$url = "https://dev.azure.com/$($organization)/_apis/distributedtask/pools/$($poolId)/agents?api-version=5.1"
$response = Invoke-RestMethod $url -Headers $headers -Verbose
write-host "Output: $response"
$status = ($response.value | where {$_.name -eq $($env:primaryPoolName)}).status
if($status -eq "online")
{
Write-Output("##vso[task.setvariable variable=DeployGroupSelector;]1")
}
else
{
Write-Output("##vso[task.setvariable variable=DeployGroupSelector;]2")
}
For the group containing the script above the "Allow scripts access to the OAuth token" box is checked.
When I run this powershell locally using a PAT it returns data. When I run the release in ADO, it hits the service, but returns an empty data set:
2019-10-07T14:16:18.8942915Z VERBOSE: GET https://dev.azure.com/xxxxxx/_apis/distributedtask/pools/13/agents?api-version=5.1 with 0-byte payload
2019-10-07T14:16:19.3235204Z VERBOSE: received 22-byte response of content type application/json
2019-10-07T14:16:19.9626359Z VERBOSE: Content encoding: utf-8
2019-10-07T14:16:19.9835101Z Output: #{count=0; value=System.Object[]}
I have tried giving the "Project Collection Build Service Accounts" group read access to pools and groups, I even tried bumping it up to user. I tried adding the build service accounts group to release administrators. I even tried using the old url format just in case.
Added picture of data returned from powershell:
UPDATE: Just to further rule out an issue with how I was using the token, I added a second powershell task to the task group in question. This script hits a different segment of the AzDO Rest API (below). This successfully gets a response. So the OAuth token is working, it just doesn't seem to have access to the entire API.
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$url = "https://dev.azure.com/$($organization)/$($project)/_apis/git/repositories?api-version=5.1"
$response = Invoke-RestMethod $url -Headers $headers -Verbose
write-host "Output: $($response)"
Response: Output: #{value=System.Object[]; count=10}
Since you're using the System_AccessToken variable, did you also enable the "Allow scripts to access OAuth token" checkbox in the agent job? https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=classic#systemaccesstoken This link shows where it is in build but you'll find it at the bottom of the Agent Job panel in release. If that's not checked, that may be why you're getting an empty response.
Had exactly the same issue. Were considered two options, the same as you have been trying.
System.AccessToken
PAT
Problem was solved by putting PAT into a KeyVault and using it as a basic auth token for REST API call in a pipeline.
My suggestion is that it seems expected and right behavior. Why do I think so? From Azure DevOps point of view there are two levels in our case Organization's level and Project's level. You can notice the difference by URI that were used:
$url = "https://dev.azure.com/$($organization)/_apis/distributedtask/pools/$($poolId)/agents?api-version=5.1"
$url = "https://dev.azure.com/$($organization)/$($project)/_apis/git/repositories?api-version=5.1
From security point of view it's a bad practice, let entities from lower layers, which is project in our case, access and manipulate on the higher layer, which is organization in our case.
As a conclusion I would say that SystemToken and PAT in essence have slightly different nature, one is devoted for agent, and another for one's personal profile.

Authenticate Azure Pipelines Powershell task against Azure DevOps Rest API

Requirements
My requirement is that for my Azure devops release pipeline I want to tag a specific commit with an annotated tag which contains the build number and the date (which is auto-set on an annotated tag).
Proposed solution
My solution to this is to use a the Azure Powershell pipeline task, the one shown here:
The task (ignore what's in the script box right now) will use the Azure Subscription that I have set in order to authenticate towards the Azure DevOps REST API. I have successfully been able to perform the task I want using a personal access token (PAT) but this is not stable long-term for a whole team and I want to use our Azure Subscription.
The problem
My problem is that I'm not sure how to use the authentication of the Azure Subscription correctly. I seem to get some data using Get-AzureRmContext (see current code below) and then I found a GitHub issue which seems to do sort of the same thing. The code gets some kind of OAuth token but using the code below, Azure still returns to me that I need to sign in, so I assume it's not the correct token. I don't understand how things hatch into each other.
Note that my subscription should have all the permissions it needs to do what I want.
Code so far:
Function Get-AccessToken($tenantId) {
$cache = [Microsoft.IdentityModel.Clients.ActiveDirectory.TokenCache]::DefaultShared
$cacheItem = $cache.ReadItems() | Where-Object { $_.TenantId -eq $tenantId } | Select-Object -First 1
return $cacheItem.AccessToken
}
$context = Get-AzureRmContext
$uri = "https://dev.azure.com/<my_org>/<my_area>/_apis/git/repositories/<project_sha>/annotatedtags?api-version=5.0-preview.1"
$token = Get-AccessToken $context.tenantID
$body = #"
{
"taggedObject": {
"objectId": "$(BUILD.SOURCEVERSION)"
},
"name": "D-$(Build.BuildNumber)",
"message": "dummy"
}
"#
$header = #{"Authorization" = "Bearer" + $token}
Invoke-RestMethod -Uri $uri -Method Post -ContentType "application/json" -Body $body -Headers $header
Any help is greatly appreciated!
There is the example for the build tasks: Use a PowerShell script to customize your build pipeline
You have to enable access to token (option Allow Scripts to Access OAuth Token)
Then use it in your script. Script from the example:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=5.0"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"

How to modify Azure DevOps release definition variable from a release task?

What is the easiest way to get key rotation to work for azure storage accounts from a AzureDevOps relase task? The current plan is to re-generate the old key after release to invalidate it, and have a fresh key that can be used on next deployment. But to get that to work it seems like I at least need to store the name of the key to use in a release variable.
I had a look at he logging tasks (https://github.com/Microsoft/azure-pipelines-tasks/blob/master/docs/authoring/commands.md), but that only changes the value in the current release and does not modify the release definition.
You can use the REST API (Definitions - Update) to update the value of the release definition variable from a release task.
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Grant Project Collection Build Service (xxx) account the edit
release pipeline permission. (Select the release pipeline --> ... --> Security --> Edit release definition set to Allow)
Add a PowerShell task in your release pipeline
Run inline script: (Update the value of variable v1030 in below sample)
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/Release/definitions/$($env:RELEASE_DEFINITIONID)?api-version=5.0-preview.3"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named v1030 to its new value 1035
$pipeline.variables.v1030.value = "1035"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'v1030' is updated to" $updatedef.variables.v1030.value
write-host "=========================================================="
Here is a much cleaner and better solution that also allows for multiple builds being triggered simultaneously.
https://tsuyoshiushio.medium.com/how-to-pass-variables-with-pipeline-trigger-in-azure-pipeline-5771c5f18f91
Essentially your triggering build produces artifacts that your triggered build reads and turns into variables.
Still not at all great, but better than nothing and better than REST calls setting static global variable groups.
The other answer above talks about how to update Release pipelines.
If you would like to update a Build Pipeline's variables, here is how you do that:
Edit build pipeline
Go to the Agent Phase and select Allow Scripts to Access OAuth Token. See Use the OAuth token to access the REST API
Go to Manage Security -> Users -> Select Project Collection Build Service (YOUR TEAM NAME HERE)
Change "Edit Build Definitions" to Allow
Now add a powershell stage - 2.x - inline script called Update variables.
Script inline contents:
$api_version='5.0-preview.6'
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/$(System.DefinitionId)?api-version=${api_version}"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update variables as desired here:
$pipeline.variables.mavenBuildVersionPatch.value = "2401"
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Variable mavenBuildVersionPatch is updated to" $updatedef.variables.mavenBuildVersionPatch.value
write-host "=========================================================="
Take note of the API version in this script is 5.0-preview.6.
If your version of Azure Devops is newer, you may need to update this in the future.
Save build pipeline.
Now when you run job, after the job completes this powershell stage, this variable will be set.
Important: If you want to update a variable then make the updated variable for other pipeline stages, then you do it with a powershell stage with the following inline script:
$mavenBuildVersionPatch = [int]"$(mavenBuildVersionPatch)"
$mavenBuildVersionPatch = $mavenBuildVersionPatch + 1
Write-Host "##vso[task.setvariable variable=mavenBuildVersionPatch;]$mavenBuildVersionPatch"
This example would take our existing patch number and increment it by 1.
This does not save the variable at the end of the job, you still need to do that with another powershell script if desired.

Search for a repository in our Azure DevOps organization

In our azure devops organization, we have ~100 projects. Each projects holds between 1 and 20 git repositories. This makes it hard to find a specific repository unless you also remember which project it is located in.
The search box at the project list only searches by project names. Is there any way to search and find repository names?
There is a code search, but that returns results from files already in a repository, which gives both redundant and wrong results.
I think the simplest is to call this REST API.
You can call it even from the browser and use the Find feature.
Note that the project element is not required.
To automate, you can use a few lines of Powershell to search in the result JSON document, e.g.
$organization = ...
$PAT = ...a personal access token
$repoNameStart = ...initial name of repo
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("dummy:${PAT}"))
$result = Invoke-RestMethod -Method Get -Uri "https://dev.azure.com/${organization}/_apis/git/repositories?api-version=4.1" -Headers #{Authorization = "Basic $base64AuthInfo" } -Credential $credential -ContentType "application/json"
$result.value | where {$_.name -like "${repoNameStart}*"}
(Untested code, use it as an example)

VSTS - Where has 'Allow Scripts to Access OAuth Token' gone?

I have VSTS builds that use the Powershell script to access the VSTS API, similar to that detailed in Microsoft's Documentation.
The document states:
To enable your script to use the build process OAuth token, go to the Options tab of the build definition and select Allow Scripts to Access OAuth Token.
After you've done that, your script can use to SYSTEM_ACCESSTOKEN
environment variable to access the VSTS REST API. For example:
Example:
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/build-release/definitions/$($env:SYSTEM_DEFINITIONID)?api-version=2.0"
Write-Host "URL: $url"
$definition = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Definition = $($definition | ConvertTo-Json -Depth 1000)"
As detailed in Microsofts Documentation, I'm sure you used to go to Build > Options and click 'Allow Scripts to Access OAuth Token', but it is no longer there (see pic below).
When I try a build I get the following (which doesn't happen for my old builds):
Invoke-RestMethod :
{"$id":"1","innerException":null,"message":"TF400813: The user '' is
not authorized to access this
In addition, when I clone a build (which worked nicely when I did it 3 months ago), the property is set to false (it's set to true on the original).
UPDATE: If I export from VSTS, change that property and import again, it works, but I really need a solution without this sort of manual intervention.
How should this be done now please?
It is available in the agent phase now