Azure DevOps Server - pause/resume all pipelines during maintenance window - azure-devops

We have an Azure DevOps Server (on prem) and different build/release pipelines.
A build/release is depending to other systems. Now if we plan to do maintenance work on this other systems, no Azure build/release pipeline should be run during this time because of the dependency of this systems.
We can go to every pipeline and set the pipeline to "pause". This is working well for a small numbers of build/release pipelines, but if we have a lot of pipelines this would be time-consuming to enabled-disable all pipelines.
Is there any way to pause/resume all Azure Pipelines at the same time? (e.g. TeamCity has a simple flag to pause/resume the whole queue).
I checked the API, but there is also no way to disable the queue itself (change setting on the build/release pipeline). It this would be possible, we could loop through every pipeline definition and pause/resume the queue.

You can disable the agents to prevent the pipeline from running.
Go the Agent Pools under Pipelines in Project settings-->Select the agent pool -->Go to Agents tab-->Disable all the agents.
You can also use rest api to pause the build pipeline mentioned in the comments. See below example in powershell scripts: See here to get a personal access token.
$PAT="Personal Access Token"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($PAT)"))
#list build definiations api
$url = "https://dev.azure.com/org/proj/_apis/build/definitions?api-version=6.1-preview.7"
#list all build definitions
$AllDefinitons = Invoke-RestMethod -Uri $url1 -Headers #{Authorization = ("Bearer {0}" -f $base64AuthInfo)} -Method get
#get all the definition ids
$Ids = $AllDefinitons.value | select id
foreach($id in $Ids){
$definitionUrl="https://dev.azure.com/org/proj/_apis/build/definitions/$($id.id)?api-version=6.1-preview.7"
#get the definition of each build pipeline
$definiton = Invoke-RestMethod -Uri $definitionUrl-Headers #{Authorization = ("Bearer {0}" -f $base64AuthInfo)} -Method get
#set queueStatus to paused
$definiton.queueStatus= "paused"
#update the definition
Invoke-RestMethod -Uri $definitionUrl-Headers #{Authorization = ("Basic {0}" -f $base64AuthInfo)} -Method put -Body (ConvertTo-Json $definiton-Depth 100) -ContentType "application/json"
}

Related

Azure cli: clone pipeline

Looking at az pipelines documentation it seems it's not possible to clone a pipeline using cli.
I've looked at getting the yaml (az pipelines show -name=x > x_orig.yaml) and then trying to change json and create pipeline from modified yaml, but that feels like a lot of work that could break after next update.
Is there a way to clone a pipline without going the the Web UI?
Currently, there indeed is not available Azure CLI that can clone or export/import a pipeline to create a new pipeline.
I also searched and tried the Azure DevOps REST API for Pipelines, but did not find the available API.
Ideally, the Azure CLI "az pipelines create" can provide an input parameter that allows users specify an existing pipeline as a starting point for the new pipeline.
If your projects really need this feature, I recommend that you can directly report a feature request on the "Azure/azure-cli" repository to ask adding the parameter like as above mentioned. That will allow you directly interact with the appropriate engineering team, and make it more convenient for the engineering team to collect and categorize your suggestions.
As a workaround, we could clone the build definition via power shell script to call REST API.
Note: We need to change the original build definition name.
REST API
Get build definition:
GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=6.0
Create build definition
POST https://dev.azure.com/{organization}/{project}/_apis/build/definitions?api-version=6.0
Power shell script
$connectionToken="{pat}"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$BuildDefinitionInfoURL = "https://dev.azure.com/{org name}/{project name}/_apis/build/definitions/386"
$BuildDefinitionInfo = Invoke-RestMethod -Uri $BuildDefinitionInfoURL -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
Write-Host $BuildDefinitionInfo.name
$BuildDefinitionInfo.name = $BuildDefinitionInfo.name +" clone"
Write-Host $BuildDefinitionInfo.name
$body = $BuildDefinitionInfo | ConvertTo-Json -Depth 99
$createBuildDefinitionURL = "https://dev.azure.com/{org name}/{project name}/_apis/build/definitions?api-version=6.0"
$response = Invoke-RestMethod -Uri $createBuildDefinitionURL -ContentType "application/json" -Body $body -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -Method POST
Write-Host $response.id
Result:

Execute Mutliple release pipeline

I have 50+ release pipelines for the production environment and each time doing production has to manually approve each of these pipelines for deployment.
Is there any way to automate and in a single click, all the pipeline gets approved and gets deployed?
The only way I am aware of accomplishing this would be to combine the 50+ pipelines into one master release and have those dependent on a separate stage who is deploying to a gated environment. Assign an approval on that environment. Thus once that stage in the "approval environment" is approved all subsequent stages will run.
In Azure DevOps, we can configure approvals, It checks all the resources used in that stage, such as source code and follow the target server deployed, this should let managers to check and agree this deploy, batch approval is meaningless.
If you insist on approving multiple release pipeline stages at the same time. we recommend that you can remove these approvals.
Or we can use API to list all Approvals and get the approval id, then update status of an approval.
In addition, if there are multiple approvals in the pipeline, it will only approve the current stage.
Steps:
Create a PAT token->create a build pipeline->click the tab variable and add the variable pat = {Token}, then set it to secret variable->add task powershell and enter the following script to approve.
$connectionToken="$(pat)"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$PipelineUrl = "https://vsrm.dev.azure.com/{Org name}/{Project name}/_apis/release/approvals?api-version=5.1"
$Pipelines = (Invoke-RestMethod -Uri $PipelineUrl -Method Get -UseDefaultCredential -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)})
$ ApprovalId = $Pipelines.value.id
Write-Host $ ApprovalId
# List all Approvals and get approval id
ForEach ($Id in $ ApprovalId)
{
$baseUrl = "https://vsrm.dev.azure.com/{Org name}/{Project name}/_apis/release/approvals/$($Id)?api-version=5.1"
$body ="{
`"status`": `"approved`"
}"
$response = Invoke-RestMethod -Uri $baseUrl -ContentType "application/json" -Body $body -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -Method PATCH
}

Azure DevOps pipelines: Cancel multiple pending jobs in queue

In Azure DevOps pipelines, How do I cancel all pending jobs for a job pool. I've got lots queued and couldn't see where I can cancel all the jobs I have waiting.
Azure devops doesnot yet have this feature to cancel all the pending jobs in batch from the UI partal.
You can write scripts to call rest api to cancel all the pending jobs as walkaround. Check out below steps:
First, use list build rest api to get all the pending jobs.
https://dev.azure.com/{organization}/{project}/_apis/build/builds?statusFilter=notStarted&api-version=5.1
Then, use update build api to cancel the pending jobs:
PATCH https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}?api-version=5.1
See below powershell scripts for reference:
Check here to get a Personal access token that will be used in below scripts.
$url= "https://dev.azure.com/{organization}/{project}/_apis/build/builds?statusFilter=notStarted&api-version=5.1"
$pat="Personal Access Token"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($pat)"))
$pendingJobs=Invoke-RestMethod -Uri $url-Headers #{Authorization = ("Basic {0}" -f $base64AuthInfo)} -Method get -ContentType "application/json"
$jobsToCancel = $pendingJobs.value
#Pending jobs donot consume the job agents in the agent pool. To filter the definition name to cancel pending jobs for a particular pipeline, you can use below filter criteria.
#$jobsToCancel = $pendingJobs.value | where {$_.definition.Name -eq "{Name of your pipeline }"}
#call update api to cancel each job.
ForEach($build in $jobsToCancel)
{
$build.status = "Cancelling"
$body = $build | ConvertTo-Json -Depth 10
$urlToCancel = "https://dev.azure.com/{organization}/{project}/_apis/build/builds/$($build.id)?api-version=5.1"
Invoke-RestMethod -Uri $urlToCancel -Method Patch -ContentType application/json -Body $body -Header #{Authorization = ("Basic {0}" -f $base64AuthInfo)}
}
You can also submit a new feature request(Click Suggest a feature and choose azure devops) to Microsoft development team for supporting cancelling pending jobs in batch. Hopefully they will consider adding this feature in the future sprint.
I find that using v6 of the api works, but instead of PATCH use DELETE.
(Reusing some of the code from #Levi Lu-MSFT)
$url= "https://dev.azure.com/{organization}/{project}/_apis/build/builds?statusFilter=notStarted&api-version=6.0-preview"
$pendingJobs=Invoke-RestMethod -Method GET -UseDefaultCredentials -Uri $url -ContentType "application/json"
$jobsToCancel = $pendingJobs.value
#$jobsToCancel = $pendingJobs.value | Where {$_.definition.Name -eq "{Name of your pipeline }"}
ForEach($build in $jobsToCancel)
{
$urlToCancel = "https://dev.azure.com/{organization}/{project}/_apis/build/builds/$($build.id)?api-version=6.0-preview"
Invoke-RestMethod -Uri $urlToCancel -Method DELETE -UseDefaultCredentials -ContentType application/json -Body $body
}

How to get Releases Pipelines pre-deployment gates status through REST API in Azure DevOps

I am using Azure DevOps (ADO) REST APIs to build dashboards on Power BI. I wanted to query whether the releases pipelines of all projects in my organization have pre-deployment gates of "SonarCloud Quality Gate status" by using the REST APIs invocation.
I have tried this API (Releases - List) GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/releases?api-version=5.1. I am reading the MS docs and thought this particular GateStatuswill be useful for my needs, but this GateStatus does not show up when I am testing on my pipelines that have pre-deployment gates of "SonarCloud Quality Gate status" configured.
I found this API (Release - Get Release) GET https://vsrm.dev.azure.com/{organization}/{project}/_apis/release/releases/{releaseId}?api-version=5.1 with MS docs. That preDeploymentGates shows the information I want when I am testing in POSTMAN. But, the problem is it requires releaseId in each query which is troublesome for me since my ultimate goal is to have a list of ALL releases in multiple projects of multiple organizations.
Thanks.
since my ultimate goal is to have a list of ALL releases in multiple projects of multiple organizations.
Since you want to get the Pipelines pre-deployment gates status for ALL releases in multiple projects of multiple organizations, so we use the API (Release - Get Release) to get the gates status, then loop this API in the all Release pipelines in one project, next, loop the all projects in one ORG, latest, loop all ORGs.
First, We need to create a All accessible organization PAT:
Because we need access multiple organizations.
Second, set a array to store the multiple organization names:
$ORGNames=#("Organization1","Organization2")
Then using following powershell scripts to loop the API (Release - Get Release):
$connectionToken="PAT"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$ORGNames=#("Organization1","Organization2")
ForEach ($ORGName in $ORGNames[0,1])
{
$ProjectUrl = "https://dev.azure.com/$($ORGName)/_apis/projects?api-version=5.1"
Write-Host "URL: $ProjectUrl"
$Projects = (Invoke-RestMethod -Uri $ProjectUrl -Method Get -UseDefaultCredential -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)})
$ProjectId = $projects.value.id
Write-Host "Projects = $ProjectId"
ForEach ($Pt in $ProjectId)
{
$baseUrl = "https://vsrm.dev.azure.com/$($ORGName)/$($Pt)/_apis/release/releases?api-version=5.1"
$ReleaseId = (Invoke-RestMethod -Uri $baseUrl -Method Get -UseDefaultCredential -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)})
$ReleaseIds = $ReleaseId.value.id
Write-Host "ReleaseID = $ReleaseId"
foreach($ReleaseID in $ReleaseIds){
$url = "https://vsrm.dev.azure.com/$($ORGName)/$($Pt)/_apis/release/releases/$($ReleaseID)?api-version=5.1"
$GatesStatus = (Invoke-RestMethod -Uri $url -Method Get -UseDefaultCredential -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)})
Write-Host "GatesStatus For ORGName $ORGName Project $Pt And Release ID $ReleaseID= $($GatesStatus.environments.deploySteps.preDeploymentGates.status | ConvertTo-Json -Depth 100)"
}
}
}
The test result:
Hope this helps.

Equivalent for TeamCity's system.build.start.date in Azure Pipelines

Is there an equivalent for the TeamCity variable system.build.start.date in Azure Pipelines? I couldn't find it in the predefined variables (https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=vsts)
It doesn't appear like there is. It is available on the Get Builds API though. You could add a custom PowerShell task as the first step and use it to set a variable for future tasks to reference within the same Agent phase.
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes((":{0}" -f $env:SYSTEM_ACCESSTOKEN)))
$uri = "https://dev.azure.com/{oraganization}/{project}/_apis/build/builds/$($env:BUILD_BUILDID)?api-version=5.0"
$buildStartTime = Invoke-RestMethod -Uri $uri -Method Get -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} |
Select-Object -ExpandProperty startTime
Write-Host "##vso[task.setvariable variable=BuildStartTime;]$buildStartTime"