[https://learn.microsoft.com/en-us/rest/api/azure/devops/distributedtask/pools/get%20agent%20pools?view=azure-devops-rest-6.0]
The above document indicates that Deployment pools should be included. Unfortunately, they are not returned (although all of the "automation" pools are.
So how does one programmatically list, get (by id) and create/update these pools????
UPDATE....
Capturing network traces while viewing /_settings/deploymentpools?view=pool
has revealed that there is a ReST call to _apis/distributedtask/deploymentPools
which displays the proper information. HOWEVER, continued searching of learn.microsoft.com does not provide any information on this rest call.
UPDATE 2: Returned (sanitized) JSON for call Matt recommended:
JSON from _apis/distributedtask/deploymentPools/deploymentPoolsSummary?expands=2&api-version=6.1-preview.1
As can be seen, the two are not equivalent and so far only the call reverse engineered from the UI shows the same information set (including 2 needed fields related to online and offline counts)
Given this example Pool configuration:
When utilizing the Pool API without any filters using this script:
$AzureDevOpsAuthenicationHeader = #{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzureDevOpsPAT)")) }
$uri = "https://dev.azure.com/{organization}/_apis/distributedtask/pools?api-version=6.0"
Invoke-RestMethod -Method Get -Uri $uri -Headers $AzureDevOpsAuthenicationHeader |
Select-Object -ExpandProperty value |
Select-Object name, poolType
Returns:
name poolType
---- --------
Default automation
Hosted automation
Hosted VS2017 automation
Hosted Windows Container automation
Hosted macOS automation
Hosted Ubuntu 1604 automation
Hosted Windows 2019 with VS2019 automation
Hosted macOS High Sierra automation
Azure Pipelines automation
It seems like the default without filters adds an implied filter for automation poolType as you suggested. Adding a filter targeting deployment pools though I think gets the result you are expecting:
$AzureDevOpsAuthenicationHeader = #{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzureDevOpsPAT)")) }
$uri = "https://dev.azure.com/{organization}/_apis/distributedtask/pools?api-version=6.0&poolType=deployment"
Invoke-RestMethod -Method Get -Uri $uri -Headers $AzureDevOpsAuthenicationHeader |
Select-Object -ExpandProperty value |
Select-Object name, poolType
Returns:
name poolType
---- --------
ExamplePool deployment
including 2 needed fields related to online and offline counts
We can use the Rest API to List Deployment Pools, get the deployment status via field onlineAgentsCount and offlineAgentsCount and get pool ID.
API:
https://dev.azure.com/{Org name}/_apis/distributedtask/deploymentPools/deploymentPoolsSummary?expands=2
Result:
And get the deployment pool detail or update deployment pool via this API
API:
https://dev.azure.com/{Org name}/_apis/distributedtask/pools?poolIds={pool ID}&api-version=6.0
Update1
The Microsoft doc does not list all REST APIs, but we can capture these APIs via F12 or fiddler. In addition, you can also try this API, it will also show Deployment pool info.
In addition, you can raise a new suggestion ticket to the user voice, Microsoft Azure DevOps Teams will check it.
Related
I am working on creating qnamaker based Chatbot and for this I have prepared ARM template for creating and deploying resources.
PowerShell script is being used to create knowledge base once all the resources are created.
This process works just fine when I am executing PowerShell script from local PowerShell tool.
Now I want to do CI/CD for this deployment process so that deployment process can be automated
Creating resources and deploying to azure through Pipeline is quite possible through Azure ARM Template deployment task but I am not sure how to execute PowerShell script that is responsible for creating knowledge base once resources are created based on recently created qnamaker service
Any help greatly appreciated
You can refer to Create a CI/CD pipeline for .NET with the Azure DevOps Project for step by step lab to create the pipelines.
If you need help on ARM templates (if you are stuck somewhere), you can check out A quick guide on writing your Azure Resource Manager (ARM) Templates. and get started quickly.
Please clarify that your issue is creating and training the QnA Maker KB after the actual resources have been created by ARM? There are probably multiple ways to do this, but here is what I have done. First of all, I am using the Azure CLI Task version 2.* which simplifies some of the initial steps. You could probably use regular PowerShell for some of the subsequent steps. I have selected all as PowerShell script types and inline scripts. Each of these segments I do as a separate task but you could probably do all as one. I like being able to see which step fails if something happens, though.
First you need to get the key for your cognitive service
$QNAKEY= & az cognitiveservices account keys list -g "YOUR_RESOURCE_GROUP" --name "YOUR_QNA_RESOURCE_NAME" --query key1 -o tsv
Write-Output("##vso[task.setvariable variable=QNA_KEY;]$QNAKEY")
Next you need to create the KB. I have mine seeded with information via a json file in my project repository. You can do this from wherever, or just create a blank KB (I think). Note I'm checking here to see if the KB exists; KB name does not have to be unique, so if you don't do this you will end up creating a lot of duplicate KBs.
$header = #{
"Content-Type"="application/json"
"Ocp-Apim-Subscription-Key"="$(QNA_KEY)"
}
$res = Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases" -Method 'Get' -Headers $header
$kb = $res.knowledgebases | Where-Object name -eq "YOUR_KB_NAME"
Write-Host $res.knowledgebases
Write-Host $kb
if (!$kb) {
Write-Host "KB does not exist, so creating new KB"
$body = Get-Content '$(System.DefaultWorkingDirectory)/PATH_TO_MY_MODEL.json' | Out-String
Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases/create" -Method 'Post' -Body $body -Headers $header
}
Finally, you will likely want to publish your KB. With LUIS I think you need to train it first (via separate CLI task), but QnA Maker you should be able to publish directly. I do this as a dependent stage to ensure the KB is created before I try to publish.
$QNAKEY= & az cognitiveservices account keys list -g "YOUR_RESOURCE_GROUP" --name "YOUR_QNA_RESOURCE_NAME" --query key1 -o tsv
$header = #{
"Content-Type"="application/json"
"Ocp-Apim-Subscription-Key"="$QNAKEY"
}
$res = Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases" -Method 'Get' -Headers $header
$kb = $res.knowledgebases | Where-Object name -eq "YOUR_KB_NAME"
$qnaId = $kb.id
Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases/$qnaId" -Method 'Post' -Headers $header
And that's it! At this point your QnA Maker knowledgebase should be created, published, and ready to go.
I'm trying to automate the deployment of releases in Azure Devops using the Azure Devops API with Powershell. I'm able to retrieve the last release definition, but I'm not able to extract a release for a specific environment.
I'm looking to retrieve the last 'prod' release in order to obtain the 'release ID', and then deploy the release to UAT or DEV using a HTTP 'PATCH' request.
I can hit a URL manually (example below), which gives me the last release for DEV (definitionEnvironmentId=1), and I know I could easily changed the environment ID to '12' for production, but I'd like to get this via an API call.
https://{redacted url}/{redacted}/{redacted}/_apis/release/deployments?api-version=6.0&definitionId=1&definitionEnvironmentId=1&$top=1
Any ideas if this is possible as I can't find any info on it?
You were using the correct Deployments - List Rest api. The reason you didnot get any info from the api call via powershell is because you didnot escape the special character $ for $top parameter in the url.
$ is a special character in powershell. You should escape it by adding a back tick '`' in the url. See below:
#add ` to $top parameter
$url = "https://{redacted url}/_apis/release/deployments?definitionId=3&definitionEnvironmentId=5&`$top=1&api-version=6.0"
$connectionToken="PAT"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$response= Invoke-RestMethod -Uri $url -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
$releaseId = $response.value[0].release.id
I'm trying to get the status of agents in a deployment pool at release time.
The use case is I have 2 servers with shared disk, I want the release to run on one server only. I have two Deployment groups that run based on a custom condition:
eq(variables['DeployGroupSelector'], '1')
With a job that runs prior to those that will determine the value of the DeployGroupSelector var, essentially a case statement.
In the job that sets the var, I'm trying to reach out to the Azure DevOps REST API:
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$url = "https://dev.azure.com/$($organization)/_apis/distributedtask/pools/$($poolId)/agents?api-version=5.1"
$response = Invoke-RestMethod $url -Headers $headers -Verbose
write-host "Output: $response"
$status = ($response.value | where {$_.name -eq $($env:primaryPoolName)}).status
if($status -eq "online")
{
Write-Output("##vso[task.setvariable variable=DeployGroupSelector;]1")
}
else
{
Write-Output("##vso[task.setvariable variable=DeployGroupSelector;]2")
}
For the group containing the script above the "Allow scripts access to the OAuth token" box is checked.
When I run this powershell locally using a PAT it returns data. When I run the release in ADO, it hits the service, but returns an empty data set:
2019-10-07T14:16:18.8942915Z VERBOSE: GET https://dev.azure.com/xxxxxx/_apis/distributedtask/pools/13/agents?api-version=5.1 with 0-byte payload
2019-10-07T14:16:19.3235204Z VERBOSE: received 22-byte response of content type application/json
2019-10-07T14:16:19.9626359Z VERBOSE: Content encoding: utf-8
2019-10-07T14:16:19.9835101Z Output: #{count=0; value=System.Object[]}
I have tried giving the "Project Collection Build Service Accounts" group read access to pools and groups, I even tried bumping it up to user. I tried adding the build service accounts group to release administrators. I even tried using the old url format just in case.
Added picture of data returned from powershell:
UPDATE: Just to further rule out an issue with how I was using the token, I added a second powershell task to the task group in question. This script hits a different segment of the AzDO Rest API (below). This successfully gets a response. So the OAuth token is working, it just doesn't seem to have access to the entire API.
$headers = #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$url = "https://dev.azure.com/$($organization)/$($project)/_apis/git/repositories?api-version=5.1"
$response = Invoke-RestMethod $url -Headers $headers -Verbose
write-host "Output: $($response)"
Response: Output: #{value=System.Object[]; count=10}
Since you're using the System_AccessToken variable, did you also enable the "Allow scripts to access OAuth token" checkbox in the agent job? https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=classic#systemaccesstoken This link shows where it is in build but you'll find it at the bottom of the Agent Job panel in release. If that's not checked, that may be why you're getting an empty response.
Had exactly the same issue. Were considered two options, the same as you have been trying.
System.AccessToken
PAT
Problem was solved by putting PAT into a KeyVault and using it as a basic auth token for REST API call in a pipeline.
My suggestion is that it seems expected and right behavior. Why do I think so? From Azure DevOps point of view there are two levels in our case Organization's level and Project's level. You can notice the difference by URI that were used:
$url = "https://dev.azure.com/$($organization)/_apis/distributedtask/pools/$($poolId)/agents?api-version=5.1"
$url = "https://dev.azure.com/$($organization)/$($project)/_apis/git/repositories?api-version=5.1
From security point of view it's a bad practice, let entities from lower layers, which is project in our case, access and manipulate on the higher layer, which is organization in our case.
As a conclusion I would say that SystemToken and PAT in essence have slightly different nature, one is devoted for agent, and another for one's personal profile.
Our Azure DevOps build agents are setup on Kubernetes. Failed pods can easily be deleted from kube, but they appear as "offline" agents from the Azure DevOps Web UI.
Overtime the list of offline agents has grown really long. Is there a way to programmatically delete them ?
$agents = Invoke-RestMethod -uri 'http://dev.azure.com/{organization}/_apis/distributedtask/pools/29/agents' -Method Get -UseDefaultCredentials
$agents.value |
Where-Object { $_.status -eq 'offline' } |
ForEach-Object {
Invoke-RestMethod -uri "http://dev.azure.com/{organization}/_apis/distributedtask/pools/29/agents/$($_.id)?api-version=4.1" -Method Delete -UseDefaultCredentials
}
Some assumptions for this solution:
You are looking for build agents
You know the id of the pool you are looking for already. You can get to this programatically also, or just loop through all pools if you want
You don't have any issues deleting any offline agents
Note: I'm using Azure DevOps Server, so replace the -UseDefaultCredentials with your authorization.
I think you would need to use a combination of these 2 api calls:
Get deployment groups
Delete agent
Or you can use some naming matrix for agents, so they are using the same "name" even if it is not the same container.
https://github.com/julioct/azure-pipelines-kubernetes-agents
I want to connect to VSTS and get a list of agent pools. I want to enumerate the agent pools and then delete agents on the VSTS server. I can't find any documentation for how to do this in the VSTS API reference.
I'm connecting to VSTS fine like this to list projects for example, but how to list agent pools?
$User = 'mark.allison#domain.com'
$PersonalAccessToken = '{PAT_TOKEN}'
$base64authinfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $User, $PersonalAccessToken)))
$vstsAccount = "{VSTS_ACCOUNT}"
# projects
$resource = 'projects'
$uri = "https://sabinio.visualstudio.com/DefaultCollection/_apis/$($resource)?api-version=3.0"
$projects = Invoke-RestMethod -Method Get -ContentType application/json -Uri $Uri -Headers #{Authorization=("Basic {0}" -f $base64authinfo)}
I did as well not find anything about it in the official documents.
However by checking the network traffic I figured out the following:
https://xyz.visualstudio.com/_apis/distributedtask/pools/ is how to get the Agent Pools.
So with a GET request you will get a list of agent pools, that includes name, id and some more properties. If you send a DELETE request to that url with the ID of the pool you want to delete, it will be removed.
If at first you want to get the agents of an agent pool, you do a GET to:
https://xyz.visualstudio.com/_apis/distributedtask/pools/POOLID/agents
Then again you get the agents of that pool listed with name, id etc.
To delete that agent you send a DELETE request to https://xyz.visualstudio.com/_apis/distributedtask/pools/POOLID/agents/AGENTID
So in your PowerShell script you as well don't need to fetch anything "project-specific", as the AgentPools and Agents are available for all the projects of the collection.
Hope that helps