How do you manage building from branches when using Azure DevOps Classic Build Pipelines and TFVC?
I believe that the only viable option is to copy the build pipeline with a new name and update the source code mapping to point to the new TFVC branch.
I see the ADO web UI provides the option to clone an individual build definition, yet as I have over 200+ build pipelines to clone whenever I branch is there a more efficient way to do this? Or is the only option to write a custom tool to leverage the ADO REST Api?
Since you need to clone pipelines in batches, using scripts to run the Rest API will be a reasonable method. As far as I know, there is no easy way out of the box other than this.
You could try the following PowerShell Script Sample:
$DefinitionIds = "PipelineIDs" #InPut All Pipelineids(e.g. "324,323,xxx" )
$DefinitionId = $DefinitionIds.split(",");
$token = "PAT Token"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
foreach ($i in $DefinitionId)
{
echo $i
$url="https://dev.azure.com/{OrganizationName}/{ProjectName}/_apis/build/definitions/$($i)?api-version=6.0"
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method Get -ContentType application/json
Write-Host "$($response | ConvertTo-Json -Depth 100)"
$response.repository.properties.tfvcMapping= '{"mappings":[{"serverPath":"$/TFVCBranchName","mappingType":"map","localPath":"\\"}]}' # ServerPath is the Branch name
$response.repository.name = "TFVCRepoName" #Repo Source Name
$response.name = "Pipeline $i Clone" # Cloned PipelineName
echo $response.name
$url1= "https://dev.azure.com/{OrganizationName}/{ProjectName}/_apis/build/definitions?api-version=6.0"
$json = #($response) | ConvertTo-Json -Depth 100
$response1 = Invoke-RestMethod -Uri $url1 -Headers #{Authorization = "Basic $token"} -Method Post -Body $json -ContentType application/json
}
Here are the Two Rest APIs used in the Script:
Definitions - Get
Definitions - Create
Result:
The cloned Pipeline will be set to the new TFVC branch and Build definition name.
Related
I am publishing build artifact of each run over azure pipeline, whatever artifact I have published than can be accessible to that specific stage(run), But my query is in Azure is there any build artifacts repository where we can get all the build artifacts for all the builds of that pipeline ?
To get all the build artifacts for all pipeline runs, you need to combine the following two Rest APIs: Builds - List and Artifacts - List
Here is a PowerShell example:
$token = "PAT"
$url="https://dev.azure.com/{organization}/{project}/_apis/build/builds?api-version=6.0"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method Get -ContentType application/json
ForEach( $buildid in $response.value.id )
{
echo $buildid
$url1= "https://dev.azure.com/{organization}/{project}/_apis/build/builds/$($buildid)/artifacts?api-version=4.1"
$response1 = Invoke-RestMethod -Uri $url1 -Headers #{Authorization = "Basic $token"} -Method Get -ContentType application/json
echo $response1 | ConvertTo-Json
}
This sample will traverse all pipeline runs and get relevant artifacts information.
Update:
When you use Classic Pipeline, you could add a PowerShell task and input the PowerShell Script. Please change the parameters in the script to make it suitable for your organization
Looking at az pipelines documentation it seems it's not possible to clone a pipeline using cli.
I've looked at getting the yaml (az pipelines show -name=x > x_orig.yaml) and then trying to change json and create pipeline from modified yaml, but that feels like a lot of work that could break after next update.
Is there a way to clone a pipline without going the the Web UI?
Currently, there indeed is not available Azure CLI that can clone or export/import a pipeline to create a new pipeline.
I also searched and tried the Azure DevOps REST API for Pipelines, but did not find the available API.
Ideally, the Azure CLI "az pipelines create" can provide an input parameter that allows users specify an existing pipeline as a starting point for the new pipeline.
If your projects really need this feature, I recommend that you can directly report a feature request on the "Azure/azure-cli" repository to ask adding the parameter like as above mentioned. That will allow you directly interact with the appropriate engineering team, and make it more convenient for the engineering team to collect and categorize your suggestions.
As a workaround, we could clone the build definition via power shell script to call REST API.
Note: We need to change the original build definition name.
REST API
Get build definition:
GET https://dev.azure.com/{organization}/{project}/_apis/build/definitions/{definitionId}?api-version=6.0
Create build definition
POST https://dev.azure.com/{organization}/{project}/_apis/build/definitions?api-version=6.0
Power shell script
$connectionToken="{pat}"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$BuildDefinitionInfoURL = "https://dev.azure.com/{org name}/{project name}/_apis/build/definitions/386"
$BuildDefinitionInfo = Invoke-RestMethod -Uri $BuildDefinitionInfoURL -Headers #{authorization = "Basic $base64AuthInfo"} -Method Get
Write-Host $BuildDefinitionInfo.name
$BuildDefinitionInfo.name = $BuildDefinitionInfo.name +" clone"
Write-Host $BuildDefinitionInfo.name
$body = $BuildDefinitionInfo | ConvertTo-Json -Depth 99
$createBuildDefinitionURL = "https://dev.azure.com/{org name}/{project name}/_apis/build/definitions?api-version=6.0"
$response = Invoke-RestMethod -Uri $createBuildDefinitionURL -ContentType "application/json" -Body $body -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -Method POST
Write-Host $response.id
Result:
We have an Azure DevOps Server (on prem) and different build/release pipelines.
A build/release is depending to other systems. Now if we plan to do maintenance work on this other systems, no Azure build/release pipeline should be run during this time because of the dependency of this systems.
We can go to every pipeline and set the pipeline to "pause". This is working well for a small numbers of build/release pipelines, but if we have a lot of pipelines this would be time-consuming to enabled-disable all pipelines.
Is there any way to pause/resume all Azure Pipelines at the same time? (e.g. TeamCity has a simple flag to pause/resume the whole queue).
I checked the API, but there is also no way to disable the queue itself (change setting on the build/release pipeline). It this would be possible, we could loop through every pipeline definition and pause/resume the queue.
You can disable the agents to prevent the pipeline from running.
Go the Agent Pools under Pipelines in Project settings-->Select the agent pool -->Go to Agents tab-->Disable all the agents.
You can also use rest api to pause the build pipeline mentioned in the comments. See below example in powershell scripts: See here to get a personal access token.
$PAT="Personal Access Token"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($PAT)"))
#list build definiations api
$url = "https://dev.azure.com/org/proj/_apis/build/definitions?api-version=6.1-preview.7"
#list all build definitions
$AllDefinitons = Invoke-RestMethod -Uri $url1 -Headers #{Authorization = ("Bearer {0}" -f $base64AuthInfo)} -Method get
#get all the definition ids
$Ids = $AllDefinitons.value | select id
foreach($id in $Ids){
$definitionUrl="https://dev.azure.com/org/proj/_apis/build/definitions/$($id.id)?api-version=6.1-preview.7"
#get the definition of each build pipeline
$definiton = Invoke-RestMethod -Uri $definitionUrl-Headers #{Authorization = ("Bearer {0}" -f $base64AuthInfo)} -Method get
#set queueStatus to paused
$definiton.queueStatus= "paused"
#update the definition
Invoke-RestMethod -Uri $definitionUrl-Headers #{Authorization = ("Basic {0}" -f $base64AuthInfo)} -Method put -Body (ConvertTo-Json $definiton-Depth 100) -ContentType "application/json"
}
I had my first question answered here Is it possible to read the PR tag on a pipeline task? but my scenario is a little bit different. I need to read the PR tag from a pipeline that was triggered by another pipeline.
PR triggers the CI which checks if everything's ok for merge. If it is, the CI triggers the CD which will in turn read the PR tag.
PR -> CI -> CD (access the tag here)
I have a PowerShell task named Get PR tag with the following script (courtesy of Lance):
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/git/repositories/$($env:BUILD_REPOSITORY_NAME)/pullRequests/$($env:SYSTEM_PULLREQUEST_PULLREQUESTID)/labels?api-version=5.1-preview.1"
$response = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "##vso[task.setvariable variable=PullRequestTag;isOutput=true]$($response.value.name)"
But I keep getting "The request is invalid.":
========================== Starting Command Output ===========================
/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/74b14931-e33a-4389-b19f-3db7faa53e8d.ps1'
Invoke-RestMethod: /home/vsts/work/_temp/74b14931-e33a-4389-b19f-3db7faa53e8d.ps1:3
Line |
3 | $response = Invoke-RestMethod -Uri $url -Method Get -Headers #{
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| {"count":1,"value":{"Message":"The request is invalid."}}
##[error]PowerShell exited with code '1'.
Finishing: Get PR tag
My agent job is set to use the OAuth token:
Update
In release pipeline, the variable name is not the same as the variable name in build, we need update the url info in the script, we can also check the release pipeline variable in the Initialize job log.
Steps:
a. Configure branch policy and add the policy Build Validation-> add build pipeline A
b. Create release->select the build A as the Source type->Enable the feature Pull request trigger->open Pre-deployment conditions and enable the option Pull request deployment
c. Open the release->enable the feature Allow scripts to access the OAuth token (Click Agent Job Name=>Additional options) add task powershell and enter the script below
$url = "$($env:SYSTEM_TASKDEFINITIONSURI)$env:BUILD_PROJECTID/_apis/git/repositories/$($env:BUILD_REPOSITORY_NAME)/pullRequests/$($env:BUILD_PULLREQUEST_ID)/labels?api-version=5.1-preview.1"
$response = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "##vso[task.setvariable variable=PullRequestTag;isOutput=true]$($response.value.name)"
d. Configure the Reference name as PS and add task cmd to output the tags.
CMD script:
echo $(PS.PullRequestTag)
e. Create pull request and add tags
Result:
Update2
The pull request triggers the CI build pipeline(power shell), after the build pipeline is completed, another build pipeline(power shell test) will be triggered.
b. Open the build pipeline power shell test and add a new variable PullRequestID and grant test Build Service (xxx) account the Edit build pipeline permission. (open the build pipeline(power shell test)--> ... --> Security --> Edit build pipeline set to Allow)
c. enable the feature Allow scripts to access the OAuth token (Click Agent Job Name=>Additional options) add task powershell(Get the tag value) and enter the script below. click the powershell task->Output Variables->enter PS->add a task cmd and use the code echo $(PS.PullRequestTag) to output the tag value
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/git/repositories/$($env:BUILD_REPOSITORY_NAME)/pullRequests/$(PullRequestID)/labels?api-version=5.1-preview.1"
$response = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "##vso[task.setvariable variable=PullRequestTag;isOutput=true]$($response.value.name)"
d. Open build pipeline power shell, enable the feature Allow scripts to access the OAuth token (Click Agent Job Name=>Additional options) add task powershell and enter the script below to update the pipeline(power shell test) variable PullRequestID value.
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/build/definitions/55?api-version=5.1"
Write-Host "URL: $url"
$pipeline = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "Pipeline = $($pipeline | ConvertTo-Json -Depth 100)"
# Update an existing variable named PullRequestID to its new value pull request ID
$pipeline.variables.PullRequestID.value= $($env:SYSTEM_PULLREQUEST_PULLREQUESTID)
####****************** update the modified object **************************
$json = #($pipeline) | ConvertTo-Json -Depth 99
$updatedef = Invoke-RestMethod -Uri $url -Method Put -Body $json -ContentType "application/json" -Headers #{Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"}
write-host "=========================================================="
Write-host "The value of Varialbe 'PullRequestID ' is updated to" $updatedef.variables.PullRequestID.value
write-host "=========================================================="
I would like to get info on the number of commits per pull request. For now, semi-automated would be fine. Just looking for a simple approach, say, a PowerShell script. I'll likely need to create similar types of reports with different data in the future.
Does any approach lend itself to such a quick and dirty approach with Azure DevOps data?
There is a special Rest API for this: Pull Request Commits - Get Pull Request Commits:
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullRequests/{pullRequestId}/commits?api-version=5.1
So, simple PowerShell script:
$pat = "YOUR-PERSONAL-ACCESS-TOKEN"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,"$pat")))
$headers = #{Authorization=("Basic {0}" -f $base64AuthInfo)}
$url = "https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullRequests/{pullRequestId}/commits?api-version=5.1"
$commitsCount = (Invoke-RestMethod -Method Get -Uri $url -Headers $headers -Body $jsonBody -ContentType 'application/json').count