Execute YAML templates from Azure DevOps classic pipeline - azure-devops

I would put my questions through following points, hope it's make clear now:
The application source code is in application_code repo.
The pipeline code(YAMLs) are in pipeline_code repo. Because I'd like to version it and don't like to keep in application_code repo. Just to avoid giving control to Dev team to manage it.
Problem statement:
The pipeline YAML won't be triggered unless it's in the source code repository based on the events pr, commit etc.
Can we trigger or execute YAML file which is in pipeline_repo whenever there's event triggered in application_code repo?
I've tried achieving above using Classic pipeline and YAML template but this don't work together. As I can execute a YAML template from a YAML pipeline only not from a classic pipeline like below:
#azure-pipeline.yaml
jobs:
- job: NewJob
- template: job-template-bd1.yaml
Any ideas or better solution than above?

The feature Multi-repository support for YAML pipelines will be available soon for azure devops service. This feature will support for triggering pipelines based on changes made in one of multiple repositories. Please check Azure DevOps Feature Timeline or here. This feature is expected to be rolled out in 2020 Q1 for azure devops service.
Currently you can follow below workaround to achieve above using Build Completion(the pipeline will be triggered on the completion of another build).
1, Setup the triggering pipeline
Create an empty classic pipeline for application_code repo as the triggering pipeline, which will always succeed and do nothing.
And check Enable continuous integration under Triggers tab and setup Bracnh filters
2, setup the triggered pipeline
In the pipeline_code repo using Checkout to Check out multiple repositories in your pipeline. You can specifically checkout the source code of application_code repo to build. Please refer below example:
steps:
- checkout: git://MyProject/application_code_repo#refs/heads/master # Azure Repos Git repository in the same organization
- task: TaskName
...
Then in the yaml pipeline edit page, click the 3dots on the top right corner and click Triggers. Then click +Add beside Build Completion and select above triggering pipeline created in step 1 as the triggering build.
After finishing above two steps, when changes made to application_code repo, the triggering pipeline will be executed and completed with success. Then the triggered pipeline will be triggered to run the real build job.
Update:
Show Azure DevOps Build Pipeline Status in Bitbucket.
you can add a python script task at the end of the yaml pipeline to update the Bitbucket build status. You need to set a condtion: always() to always run this task even if other tasks are failed.
You can get the build status with env variable Agent.JobStatus. For below example:
For more information, please refer to document Integrate your build system with Bitbucket Cloud, and also this thread.
- task: PythonScript#0
condition: always()
inputs:
scriptSource: inline
script: |
import os
import requests
# Use environment variables that your CI server provides to the key, name,
# and url parameters, as well as commit hash. (The values below are used by
# Jenkins.)
data = {
'key': os.getenv('BUILD_ID'),
'state': os.getenv('Agent.JobStatus'),
'name': os.getenv('JOB_NAME'),
'url': os.getenv('BUILD_URL'),
'description': 'The build passed.'
}
# Construct the URL with the API endpoint where the commit status should be
# posted (provide the appropriate owner and slug for your repo).
api_url = ('https://api.bitbucket.org/2.0/repositories/'
'%(owner)s/%(repo_slug)s/commit/%(revision)s/statuses/build'
% {'owner': 'emmap1',
'repo_slug': 'MyRepo',
'revision': os.getenv('GIT_COMMIT')})
# Post the status to Bitbucket. (Include valid credentials here for basic auth.
# You could also use team name and API key.)
requests.post(api_url, auth=('auth_user', 'auth_password'), json=data)

Related

Sonarqube results display in azure pull request

I need a solution to display the results of sonar in the azure pull request.
I tried to do it with a status check by selecting the sonar pipeline in branch policy. It is showing success/fail and redirecting to sonar portal on click.
Is it really possible to show the actual results(vulnarabilities,duplications,etc.,) in the pull requets itself?
please help.
Thanks
After got the result of Sonarqube, you could use DevOps REST API to update the result to Azure pull request.
The flow is : a new pull request created > trigger a pipeline > run REST API to update the pull request description or title.
Add a Powershell task in the pipeline with follow script to update the pull request description and title. You could also refer to the document above to update other properties of the pull request. Please pay attention to PAT, the result of Sonarqube, organization name, project name, repository ID. Here we could use $(System.PullRequest.PullRequestId) to get the pull request ID, thus, the build will fail if it was not triggered by pull request.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$connectionToken="<PAT>"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($connectionToken)"))
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", "Basic $base64AuthInfo")
$headers.Add("Content-Type", "application/json")
$body = '{"description": "<the result of Sonarqube>","title": "<the result of Sonarqube>"}'
$response = Invoke-RestMethod 'https://dev.azure.com/<organization name>/<project name>/_apis/git/repositories/<repository ID>/pullrequests/$(System.PullRequest.PullRequestId)?api-version=5.0' -Method 'PATCH' -Headers $headers -Body $body
$response | ConvertTo-Json
After configure the pipeline, please enable Build Validation for the branch in project setting >> repositories >> your repo >> policies >> branch >> Build Validation. Then, every time a new pull request created for that branch will trigger the pipeline. You could also find the repository ID in the URL.

Azure Devops pipelines to trigger ONLY on Merge

I'm looking on a way to trigger a Azure pipeline ONLY on successful (or attempted) pull request merge.
Now I have :
trigger:
branches:
include:
- DEV
steps:
- script: FOO
But this runs EVERY time there is a change on the DEV branch and I would like to avoid that.
Besides, I want a programmatic response not going trough the UI each time.
EDIT:
A weird thing is happnening
condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
gets:
Expanded: and(True, eq('IndividualCI', 'PullRequest'))"
When doing a PR, and thus doesn't work as intented
I'm looking on a way to trigger a Azure pipeline ONLY on successful (or attempted) pull request merge.
There is no such out of box way to achieve this at this moment.
We could only set the CI trigger on the target branch, but we could set the condotion for the pipeline to avoid build any task:
and(succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
For example:
trigger:
branches:
include:
- DEV
steps:
- script: FOO
condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest'))
Or you could set the condition for the stage, job and so on.
Please check the document Specify conditions for some more details.
If there is a change on the DEV branch and it would be avoided by the condition.
Note: With above way, the pipeline will be triggered, but no task will be executed.
And if you even do not want the pipeline be triggered. You could add new pipeline with powershall task to invoke REST API to trigger above pipeline and set the condition to the powershell task.
In this way, the pipeline will only triggered when the commit comes from the PR.
Update:
Doing a PR on the DEV branch results in : "Expanded: and(True,
eq('IndividualCI', 'PullRequest'))"
Yes,you are correct. That because azure devops does not have the feature to trigger the pipeline after the PR completed. Pull request trigger and Build Validation both trigger the pipeline when the PR starts.
To resolve this request, we could try create a service hook to monitor PR status. If the PR status changes, the pipeline is triggered through API or Application, you could check this document for some more details.
And another way to achieve is using the REST API.
The main idea is:
create a pipeline and set it as Build validation, but not set it as Required, should set it as Optional:
Add powershell task in above pipeline to invoke REST API to monitor the PR status until it complated, and add another task to invoke the REST API to trigger your current pipeline.
So, you could remove the:
trigger:
branches:
include:
- DEV
in your current pipeline.
The trigger you have set is a CI trigger, it will work whenever the target branch has a new commit.
Currently, there isn't a trigger that works when a pull request is completed.
The feature closest to your needs is PR triggers and build validation branch policy.
They will work when a pull request is created or when it has been changed.
If you are using Azure Repos Git, please use branch policy for build validation. If you are using GitHub or Bitbucket Could, please use pr triggers. Click the documents for the detailed information.
Besides, you can use branch policy to prevent the direct commits. When you set the branch policy of any type, only users with "Bypass policies" permission can commit to the branch directly. The rest of the users must commit the branch through a pull request.
How to create branch policy: Branch policies and settings.
How to set "Bypass policies" permission: Set Git repository permissions.

How do you get the Pull request Variable from Github to use in Azue Devops Yaml Pipeline?

I want to run a pipeline when a pull request is triggered on master branch from github.
I am using azure devops to run the pipeline
I want to pass the pull request number from github to use as a variable in the azure devops yaml pipeline
Can this be done?
How do you get the Pull request Variable from Github to use in Azue Devops Yaml Pipeline?
You could use the predefined variables System.PullRequest.PullRequestNumber to get the pull request number from github.
pr:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: |
echo The PullRequest Number is that: $(System.PullRequest.PullRequestNumber)
displayName: 'Get the PullRequest Number'
You should create YAML file for the github with above scripts:
The test result:

Azure build pipeline sporadic error, [CredentialProvider]Device flow authentication failed

This happened maybe once every 2 weeks, but lately it happens a few times until the build succeeds once.
GET https://api.nuget.org/v3-flatcontainer/xamarin.uitest/3.0.9/xamarin.uitest.3.0.9.nupkg
##[error]C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets(128,5): Error : [CredentialProvider]Device flow authentication failed. User was presented with device flow, but didn't react within 90 seconds.
----- Update ----
After further digging in the logs, I found out that it was due to an 401 (Unauthorized) access to a private feed (in the same organization as the project where the pipeline fails).
A temporary solution was to remove the specific *.csproj, that needed this private feed, from the solution build config, and also remove the private feed from the pipeline. This got the pipeline working again.
---- More Pipeline infos ----
I can't post the entire code here.
pool:
vmImage: 'windows-latest'
variables:
configuration: NugetB
steps:
- task: MSBuild
displayName: 'MSBuild'
inputs:
solution: '**\xxSolutionxx.sln'
msbuildArchitecture: 'x64'
configuration: '$(configuration)'
msbuildArguments: '/t:restore;build;pack /p:PackageOutputPath=$(Build.ArtifactStagingDirectory) /p:RestoreAdditionalProjectSources=https://urlToPrivateFeedInTheSameOrganization /p:configuration=$(configuration) /p:NuGetInteractive="true"'
Azure build pipeline sporadic error, [CredentialProvider]Device flow authentication failed
If you have task/script access to azure ariftact feed in your pipeline, then please try to use the System.AccessToken variable to authenticate a pipeline to a private Azure Artifacts feed or try to reset your PAT.
You could check this document for some more details.
Besides, if above not resolve your question, please share how did you access the artifact and the build pipeline configuration.

Add terraform plan output as PR comment in Azure Devops Build Pipeline

In my current assignment, I am integrating terraform to our Azure DevOps CI/CD pipeline. My requirements are as follows:
The creation of PR should trigger a build pipeline.
A task in the build pipeline should publish a comment to the raised PR. The content of comment will
be the terraform plan output i.e the new infrastructure that is going to be deployed.
Once the PR is approved and code merged to master, a CD pipeline would be triggered and that will
deploy the infrastructure to Azure.
Up until now, I am all sorted on 1 and 3 requirements but I have no clue how can I publish a comment on PR with the contents from terraform plan command. Is there any built-in task for this? If not how can I achieve this? Is this possible at all? If so, can someone point out a resource that can help or just show the same .yml file?
I have searched a lot but did not find anything. My guess is you cannot add a comment from the build pipeline. Need your suggestions.
Add terraform plan output as PR comment in Azure Devops Build Pipeline
I am afraid there is no such a out of box way to Add terraform plan output as PR comment at this moment.
The workaround I am thinking of right now is invoke Rest API to create a PR comment with the contents from terraform plan command.
We could use the Rest API Pull Request Thread Comments - Create:
POST https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullRequests/{pullRequestId}/threads/{threadId}/comments?api-version=5.1
to create comment to the PR. We could write the contents from terraform plan command as Request Body:
{
"content": "YourContentsFromTerraformPlan ",
"parentCommentId": 1,
"commentType": 1
}
But above API need the pullRequestId. So, we also need another API to get the pullRequestId for current project Pull Requests - Get Pull Requests By Project:
GET https://dev.azure.com/{organization}/{project}/_apis/git/pullrequests?api-version=5.1
It will return a series of pullRequestIds, then we could use powershell argument Select-Object -first 1 like: $LastPullRequestId= $pullRequestIds.value.id | Select-Object -first 1 to get the latest pullRequestId.
So, we could add two inline powershell task in the build pipeline to invoke Rest API to get the latest pullRequestId, then use this pullRequestId to create a PR comment.
Hope this helps.