Our Azure DevOps build agents are setup on Kubernetes. Failed pods can easily be deleted from kube, but they appear as "offline" agents from the Azure DevOps Web UI.
Overtime the list of offline agents has grown really long. Is there a way to programmatically delete them ?
$agents = Invoke-RestMethod -uri 'http://dev.azure.com/{organization}/_apis/distributedtask/pools/29/agents' -Method Get -UseDefaultCredentials
$agents.value |
Where-Object { $_.status -eq 'offline' } |
ForEach-Object {
Invoke-RestMethod -uri "http://dev.azure.com/{organization}/_apis/distributedtask/pools/29/agents/$($_.id)?api-version=4.1" -Method Delete -UseDefaultCredentials
}
Some assumptions for this solution:
You are looking for build agents
You know the id of the pool you are looking for already. You can get to this programatically also, or just loop through all pools if you want
You don't have any issues deleting any offline agents
Note: I'm using Azure DevOps Server, so replace the -UseDefaultCredentials with your authorization.
I think you would need to use a combination of these 2 api calls:
Get deployment groups
Delete agent
Or you can use some naming matrix for agents, so they are using the same "name" even if it is not the same container.
https://github.com/julioct/azure-pipelines-kubernetes-agents
Related
I am working on creating qnamaker based Chatbot and for this I have prepared ARM template for creating and deploying resources.
PowerShell script is being used to create knowledge base once all the resources are created.
This process works just fine when I am executing PowerShell script from local PowerShell tool.
Now I want to do CI/CD for this deployment process so that deployment process can be automated
Creating resources and deploying to azure through Pipeline is quite possible through Azure ARM Template deployment task but I am not sure how to execute PowerShell script that is responsible for creating knowledge base once resources are created based on recently created qnamaker service
Any help greatly appreciated
You can refer to Create a CI/CD pipeline for .NET with the Azure DevOps Project for step by step lab to create the pipelines.
If you need help on ARM templates (if you are stuck somewhere), you can check out A quick guide on writing your Azure Resource Manager (ARM) Templates. and get started quickly.
Please clarify that your issue is creating and training the QnA Maker KB after the actual resources have been created by ARM? There are probably multiple ways to do this, but here is what I have done. First of all, I am using the Azure CLI Task version 2.* which simplifies some of the initial steps. You could probably use regular PowerShell for some of the subsequent steps. I have selected all as PowerShell script types and inline scripts. Each of these segments I do as a separate task but you could probably do all as one. I like being able to see which step fails if something happens, though.
First you need to get the key for your cognitive service
$QNAKEY= & az cognitiveservices account keys list -g "YOUR_RESOURCE_GROUP" --name "YOUR_QNA_RESOURCE_NAME" --query key1 -o tsv
Write-Output("##vso[task.setvariable variable=QNA_KEY;]$QNAKEY")
Next you need to create the KB. I have mine seeded with information via a json file in my project repository. You can do this from wherever, or just create a blank KB (I think). Note I'm checking here to see if the KB exists; KB name does not have to be unique, so if you don't do this you will end up creating a lot of duplicate KBs.
$header = #{
"Content-Type"="application/json"
"Ocp-Apim-Subscription-Key"="$(QNA_KEY)"
}
$res = Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases" -Method 'Get' -Headers $header
$kb = $res.knowledgebases | Where-Object name -eq "YOUR_KB_NAME"
Write-Host $res.knowledgebases
Write-Host $kb
if (!$kb) {
Write-Host "KB does not exist, so creating new KB"
$body = Get-Content '$(System.DefaultWorkingDirectory)/PATH_TO_MY_MODEL.json' | Out-String
Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases/create" -Method 'Post' -Body $body -Headers $header
}
Finally, you will likely want to publish your KB. With LUIS I think you need to train it first (via separate CLI task), but QnA Maker you should be able to publish directly. I do this as a dependent stage to ensure the KB is created before I try to publish.
$QNAKEY= & az cognitiveservices account keys list -g "YOUR_RESOURCE_GROUP" --name "YOUR_QNA_RESOURCE_NAME" --query key1 -o tsv
$header = #{
"Content-Type"="application/json"
"Ocp-Apim-Subscription-Key"="$QNAKEY"
}
$res = Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases" -Method 'Get' -Headers $header
$kb = $res.knowledgebases | Where-Object name -eq "YOUR_KB_NAME"
$qnaId = $kb.id
Invoke-RestMethod -Uri "https://westus.api.cognitive.microsoft.com/qnamaker/v4.0/knowledgebases/$qnaId" -Method 'Post' -Headers $header
And that's it! At this point your QnA Maker knowledgebase should be created, published, and ready to go.
[https://learn.microsoft.com/en-us/rest/api/azure/devops/distributedtask/pools/get%20agent%20pools?view=azure-devops-rest-6.0]
The above document indicates that Deployment pools should be included. Unfortunately, they are not returned (although all of the "automation" pools are.
So how does one programmatically list, get (by id) and create/update these pools????
UPDATE....
Capturing network traces while viewing /_settings/deploymentpools?view=pool
has revealed that there is a ReST call to _apis/distributedtask/deploymentPools
which displays the proper information. HOWEVER, continued searching of learn.microsoft.com does not provide any information on this rest call.
UPDATE 2: Returned (sanitized) JSON for call Matt recommended:
JSON from _apis/distributedtask/deploymentPools/deploymentPoolsSummary?expands=2&api-version=6.1-preview.1
As can be seen, the two are not equivalent and so far only the call reverse engineered from the UI shows the same information set (including 2 needed fields related to online and offline counts)
Given this example Pool configuration:
When utilizing the Pool API without any filters using this script:
$AzureDevOpsAuthenicationHeader = #{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzureDevOpsPAT)")) }
$uri = "https://dev.azure.com/{organization}/_apis/distributedtask/pools?api-version=6.0"
Invoke-RestMethod -Method Get -Uri $uri -Headers $AzureDevOpsAuthenicationHeader |
Select-Object -ExpandProperty value |
Select-Object name, poolType
Returns:
name poolType
---- --------
Default automation
Hosted automation
Hosted VS2017 automation
Hosted Windows Container automation
Hosted macOS automation
Hosted Ubuntu 1604 automation
Hosted Windows 2019 with VS2019 automation
Hosted macOS High Sierra automation
Azure Pipelines automation
It seems like the default without filters adds an implied filter for automation poolType as you suggested. Adding a filter targeting deployment pools though I think gets the result you are expecting:
$AzureDevOpsAuthenicationHeader = #{Authorization = 'Basic ' + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($AzureDevOpsPAT)")) }
$uri = "https://dev.azure.com/{organization}/_apis/distributedtask/pools?api-version=6.0&poolType=deployment"
Invoke-RestMethod -Method Get -Uri $uri -Headers $AzureDevOpsAuthenicationHeader |
Select-Object -ExpandProperty value |
Select-Object name, poolType
Returns:
name poolType
---- --------
ExamplePool deployment
including 2 needed fields related to online and offline counts
We can use the Rest API to List Deployment Pools, get the deployment status via field onlineAgentsCount and offlineAgentsCount and get pool ID.
API:
https://dev.azure.com/{Org name}/_apis/distributedtask/deploymentPools/deploymentPoolsSummary?expands=2
Result:
And get the deployment pool detail or update deployment pool via this API
API:
https://dev.azure.com/{Org name}/_apis/distributedtask/pools?poolIds={pool ID}&api-version=6.0
Update1
The Microsoft doc does not list all REST APIs, but we can capture these APIs via F12 or fiddler. In addition, you can also try this API, it will also show Deployment pool info.
In addition, you can raise a new suggestion ticket to the user voice, Microsoft Azure DevOps Teams will check it.
I'm trying to automate the deployment of releases in Azure Devops using the Azure Devops API with Powershell. I'm able to retrieve the last release definition, but I'm not able to extract a release for a specific environment.
I'm looking to retrieve the last 'prod' release in order to obtain the 'release ID', and then deploy the release to UAT or DEV using a HTTP 'PATCH' request.
I can hit a URL manually (example below), which gives me the last release for DEV (definitionEnvironmentId=1), and I know I could easily changed the environment ID to '12' for production, but I'd like to get this via an API call.
https://{redacted url}/{redacted}/{redacted}/_apis/release/deployments?api-version=6.0&definitionId=1&definitionEnvironmentId=1&$top=1
Any ideas if this is possible as I can't find any info on it?
You were using the correct Deployments - List Rest api. The reason you didnot get any info from the api call via powershell is because you didnot escape the special character $ for $top parameter in the url.
$ is a special character in powershell. You should escape it by adding a back tick '`' in the url. See below:
#add ` to $top parameter
$url = "https://{redacted url}/_apis/release/deployments?definitionId=3&definitionEnvironmentId=5&`$top=1&api-version=6.0"
$connectionToken="PAT"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$response= Invoke-RestMethod -Uri $url -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
$releaseId = $response.value[0].release.id
I have around 80 DLL's for which builds are created and pushed to Nuget using Azure Devops.
I am trying to create release pipeline by seggrigating different builds into different release pipelines.
But before creating I would like to identify if other release pipeline exist which using these builds as artifacts?
How could I identify if a release pipelines exits which already uses above builds.
You will want to use the API for Release Pipelines and pass the artifacts expand option. I'll leave you to figure out the authentication piece (there are other answers that cover it).
Here's an example in PowerShell:
$uri = 'https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/definitions?api-version=5.1&$expand=artifacts'
Invoke-RestMethod -Method get -uri $uri -UseDefaultCredentials | Select-Object -ExpandProperty value | Select-Object id, artifacts
I want to connect to VSTS and get a list of agent pools. I want to enumerate the agent pools and then delete agents on the VSTS server. I can't find any documentation for how to do this in the VSTS API reference.
I'm connecting to VSTS fine like this to list projects for example, but how to list agent pools?
$User = 'mark.allison#domain.com'
$PersonalAccessToken = '{PAT_TOKEN}'
$base64authinfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $User, $PersonalAccessToken)))
$vstsAccount = "{VSTS_ACCOUNT}"
# projects
$resource = 'projects'
$uri = "https://sabinio.visualstudio.com/DefaultCollection/_apis/$($resource)?api-version=3.0"
$projects = Invoke-RestMethod -Method Get -ContentType application/json -Uri $Uri -Headers #{Authorization=("Basic {0}" -f $base64authinfo)}
I did as well not find anything about it in the official documents.
However by checking the network traffic I figured out the following:
https://xyz.visualstudio.com/_apis/distributedtask/pools/ is how to get the Agent Pools.
So with a GET request you will get a list of agent pools, that includes name, id and some more properties. If you send a DELETE request to that url with the ID of the pool you want to delete, it will be removed.
If at first you want to get the agents of an agent pool, you do a GET to:
https://xyz.visualstudio.com/_apis/distributedtask/pools/POOLID/agents
Then again you get the agents of that pool listed with name, id etc.
To delete that agent you send a DELETE request to https://xyz.visualstudio.com/_apis/distributedtask/pools/POOLID/agents/AGENTID
So in your PowerShell script you as well don't need to fetch anything "project-specific", as the AgentPools and Agents are available for all the projects of the collection.
Hope that helps