Excuse my ignorance upfront, I am still a newbie.
Goal: SonarQube Analysis and Publish results on PR creation.
What I have at the moment:
Azure Devops with built-in Repo.
local installs of git to push/pull
Redgate SQL Source Control (our code is all SQL)
SonarQube Analysis and Publish results works in my build pipeline, I even have a build breaker built in and everything works.
The issue I am facing and wish to change is that the pipeline only triggers after the PR is completed and the merge is done to my default branch. I want the analysis to be done at the time the PR is created (and not completed) as the following step is code review. The result being that if it fails SQ Analysis that the code review step can be avoided and the required fixes be returned to the developer instead.
Having the Analysis done (on the feature branch) before it is checked again during the build phase (to check the code of the whole project) will be a massive plus.
Thanks in advance
trigger:
- dev
jobs:
- job: Rcs_Dev
timeoutInMinutes: 0
pool: Rcs Build
cancelTimeoutInMinutes: 1
steps:
- checkout: self
clean: true
- task: SonarQubePrepare#4
inputs:
SonarQube: 'SonarQubeServiceConnection'
scannerMode: 'CLI'
configMode: 'manual'
cliProjectKey: 'Rcs_Rcs'
cliProjectName: 'Rcs'
cliSources: '.'
- task: SonarQubeAnalyze#4
displayName: "Run Code Analysis"
condition: and(succeeded(), or(eq(variables['Build.SourceBranchName'], 'dev'), contains(variables['Build.Reason'], 'PullRequest')))
- task: SonarQubePublish#4
inputs:
pollingTimeoutSec: '300'
- task: sonar-buildbreaker#8
inputs:
SonarQube: 'SonarQubeServiceConnection'
- task: RedgateSqlChangeAutomationBuild#4
inputs:
operation: Build
dbFolder: RootFolder
packageName: Rcs
tempServerTypeBuild: localDB
buildAdvanced: true
compareOptionsBuild: 'NoTransactions, IgnoreFileGroups'
dataCompareOptionsBuild: 'DisableAndReenableDMLTriggers, SkipFkChecks'
transactionIsolationLevelBuild: readCommitted
queryBatchTimeoutBuild: '0'
nugetPackageVersionSelector: Specific
nugetPackageVersion: '1.0'
nugetPackageVersionUseBuildId: true
requiredVersionOfDlma: latestInstalled
If you want a concrete Pipeline to be triggered on PRs, then you have two options:
You trigger the Pipeline always (including all the branches and PRs) using:
trigger:
branches:
include:
- '*'
Or if you want to trigger on PRs against dev (what it seems the most probable scenario taking into account your description) then instead of trigger you should use:
pr:
- dev
Of course if you don't want to write twice the same Pipeline you can use the templates functionality, define the whole Pipeline only once and then extend it on each YAML, one for the normal builds with the branch trigger and one for the PRs trigger.
Documentation about PR triggers: https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#pr-triggers
Documentation about Templates: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
Related
I need to run some tasks if a pull request is opened. Im new to this so apologies .
eg A pull request is rasied on github.
If this happens I want to build some review apps based on whether the above condition is true
I need to do this using YAML
This is how pr trigger works. The below example defines two triggers. When a new pr is created for the develop branch then the pipeline will be triggered. This means that pr is open for develop branch until it merges. The second trigger will be for main branch. This means that when you merge or commit code on your main branch then the pipeline will be also triggered.
trigger:
branches:
include:
- 'main'
pr:
branches:
include:
- develop
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#pr-triggers
This worked for me
Starter pipeline
Start with a minimal pipeline that you can customize to build and deploy your code.
Add steps that build, run tests, deploy, and more:
https://aka.ms/yaml
pr:
branches:
include:
- '*'
#trigger:
#- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "pullrequest"
**Write-Host "PR Number is:-" $(System.PullRequest.PullRequestId)"
Write-Host "PR Number is:-" $(System.PullRequest.PullRequestNumber)"**
I have 30 or so Java Microservices that run of the same ci and cd template. i.e. Each of my Microservices has a build pipeline as follows and as shown below it runs automatically on a merge to master:
name: 0.0.$(Rev:r)
trigger:
- master
pr: none
variables:
- group: MyCompany
resources:
repositories:
- repository: templates
type: git
name: <id>/yaml-templates
stages:
- stage: Build
jobs:
- job: build
displayName: Build
steps:
- template: my-templates/ci-build-template.yaml#templates
- stage: PushToContainerRegistry
dependsOn: Build
condition: succeeded()
jobs:
- job: PushToContainerRegistry
displayName: PushToContainerRegistry
steps:
- template: my-templates/ci-template.yaml#templates
Where ci-build-template.yaml contains...
steps:
- checkout: self
path: s
- task: PowerShell#2
- task: Gradle#2
displayName: 'Gradle Build'
- task: SonarQubePrepare#4
displayName: SonarQube Analysis
- task: CopyFiles#2
displayName: Copy build/docker to Staging Directory
I would like to implement pr build validation wherever someone raises a pr to merge into master. In the PR build only the Build stage should run and from the build template only some tasks within ci-build-template.yaml should run.
A few questions for my learning:
How can i uplift the yaml pipeline above to make the "PushToContainerRegistry" skip if it is a pr build?
How can i uplift ci-build-template.yaml to make the "SonarQubePrepare#4" and "CopyFiles#2" skip if it is a pr build?
And lastly how can i uplift the yaml pipeline above to enable build validation for all pr's that have a target branch of master?
Whilst doing my own research i know you can do this via clickops but I am keep on learning on how to implement via yaml.
thanks
How can i uplift the yaml pipeline above to make the
"PushToContainerRegistry" skip if it is a pr build?
How can i uplift ci-build-template.yaml to make the
"SonarQubePrepare#4" and "CopyFiles#2" skip if it is a pr build?
Just need to use condition of task:
For example,
pool:
name: Azure Pipelines
steps:
- script: |
echo Write your commands here
echo Hello world
echo $(Build.Reason)
displayName: 'Command Line Script'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
Above definition will skip the step if Pull request trigger the pipeline.
Please refer to these documents:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#using-the-trigger-type-in-conditions
https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#build-variables-devops-services
And lastly how can i uplift the yaml pipeline above to enable build
validation for all pr's that have a target branch of master?
You can use this expression in the condition:
eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/master')
If you are based on DevOps git repo, then just need to add branch policy is ok:
https://learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops&tabs=browser#configure-branch-policies
I have four YAML "release" pipelines where I use the same YAML syntax to define a continuation trigger. Here is the YAML definition for the trigger.
resources:
pipelines:
- pipeline: Build # Name of the pipeline resource
source: BuildPipeline # Name of the pipeline as registered with Azure DevOps
trigger: true
Not really sure about this syntax where I don't specify any branch but everything was working fine till recently. More recently I updated two of the YAML release pipelines and they now are not getting triggered when the build pipeline completes. All pipelines if executed manually work fine.
All release pipelines have the same YAML for the continuation trigger definition (see above) and have the same branch set for "Default branch for manual and scheduled builds".
I don't know how to investigate why some of the release pipelines are not triggered (any log available somewhere?) and I don't see them executed and failing, they simply are not being triggered. How do I investigate this issue?
For your question about investigating the logs - you can see what pipeline runs were created, but unfortunately you can't see what wasn't. So far as Azure DevOps is concerned, if "nothing occurred" to set off a trigger, then there's nothing to log.
As for the pipelines themselves not triggering, from the pipeline editor, check the trigger settings to ensure that nothing is set there - UI and YAML settings tend to cancel one another out:
Finally, if you want to specify a branch, you can use some combination of the following options:
resources:
pipelines:
- pipeline: Build # Name of the pipeline resource
source: BuildPipeline # Name of the pipeline as registered with Azure DevOps
trigger:
branches:
include: # branch names which will trigger a build
exclude: # branch names which will not
tags:
include: # tag names which will trigger a build
exclude: # tag names which will not
paths:
include: # file paths which must match to trigger a build
exclude: # file paths which will not trigger a build
I believe I found the issue and it's the removal of the following statements from my deploy pipelines
pool:
vmImage: windows-2019
I removed these statements because I transformed all jobs into deployment jobs as follows
- deployment: MyDeployJob
displayName: 'bla bla bla'
environment:
name: ${{ parameters.AzureDevopsEnv }}
resourceType: VirtualMachine
resourceName: ${{ parameters.AzureDevopsVM }}
The pipelines with no pool statement run perfectly well if started manually but I'm convinced fail at being triggered if started via the pipeline completion trigger. I do not understand this behavior but I placed the pool statement back in all deploy pipelines and all are now getting triggered as the build pipeline completes.
I found that when defining the resource pipeline (trigger) in a template that you extend in the depending pipeline, there are two things that can prevent builds from being triggered:
There are syntax errors in the template (or the parent .yaml)
The depending pipeline needs to be updated before Azure Devops realizes you made edits to the template it extends
This worked for me:
template.yaml
stages:
- stage: SomeBuildStage
displayName: Build The Project
jobs:
- job: SomeJob
displayName: Build NuGet package from Project
pool:
name: My Self-hosted Agent Pool # Using Pool here works fine for me, contrary to #whatever 's answer
steps:
- pwsh: |
echo "This template can be extended by multiple pipelines in order to define a trigger only once."
# I still use CI triggers as well here (optional)
trigger:
branches:
include:
- '*'
# This is where the triggering pipeline is defined
resources:
pipelines:
- pipeline: trigger-all-builds # This can be any string
source: trigger-all-builds # This is the name defined in the Azure Devops GUI
trigger: true
depending-pipeline.yaml
extends:
template: template.yaml
# I still use CI triggers as well here (optional)
trigger:
paths:
include:
- some/subfolder
triggering-pipeline.yaml
stages:
- stage: TriggerAllBuilds
displayName: Trigger all package builds
jobs:
- job: TriggerAllBuilds
displayName: Trigger all builds
pool:
name: My Self-hosted Agent Pool
steps:
- pwsh: |
echo "Geronimooo!"
displayName: Geronimo
trigger: none
pr: none
My pipeline trigger in Azure DevOps does not fire.
Enviroment:
PipelineA (branch dev)
PipelineB (branch dev)
PipelineB should be fired if PipelineB was running successfully
Here is my current code of PipelineB.yaml
trigger: none
resources:
pipelines:
- pipeline: build_pipeline
source: PipelineA
branch: dev
trigger:
branches:
- dev
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'Hello world'
It worked in the past, but suddenly it stopped
Here's the document of Pipeline Triggers, please check this statement:
However, if the two pipelines use different repositories, then the triggered pipeline will use the latest version of the code from its default branch.
Cause of your issue:
The triggered pipeline is PipelineB in your scenario, and the default branch is always the master. Since the pipeline trigger for dev branch of PipelineA is defined in your dev branch instead of the default master branch, it's expected behavior that the pipeline trigger won't fire.
To make the pipeline trigger work:
1.You can choose to define the pipeline trigger for dev of PipelineA in default master of PipelineB.
2.Or you can change the Default branch for manual and scheduled builds of PipelineB (Change it from master to dev). You can find detailed steps about how to find this setting from #2 of this answer.
Both two choices above can resolve your issue and make the pipeline trigger work.
I just created almost identical pipeline
trigger: none
resources:
pipelines:
- pipeline: build_pipeline
source: kmadof.devops-manual (14)
branch: master
trigger:
branches:
- master
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'Hello world'
and all works fine. Are you sure you run PipelineA on dev branch?
Problem
I have two repositories A and B in one project, each with their own pipeline A-CI and B-CI. The repos are Azure Repos Git (so not external ones). I got it working to trigger pipeline B-CI whenever A-CI has completed. If A-CI got triggered by a commit to branch develop, then B-CI is triggered to build master although B also has a develop branch.
I want to build a new release of B for the dev environment, when a new dev version of A has been built.
Is it possible to let a pipeline-resource trigger the B-CI pipeline to build the branch with the same name as the branch which was just built by the pipeline-resource? It would be fine for me if it would fallback to master in case a matching branch is not available in B.
This scenario is working however if A-C and B-CI would both refer to different pipeline yamls of the same repository.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
B-CI
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- master
- develop
- feature/*
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is always refs/heads/master
Write-Host $(Build.Reason) # is ResourceTrigger
Background Info
The main idea behind that is, that A contains the IaC project and whenever the infrastructure of the project changes, then all apps should be deployed, too.
I do not want to put the IaC into the app repo because we have multiple apps, so I would have to split the IaC code into several chunks.
And then I would probably still have the same problem because some resources, like Azure KeyVault, are shared among the apps so A would still include the common stuff used by all apps and changes to it would require re-deployments of all apps.
Please check pipeline triggers:
If the triggering pipeline and the triggered pipeline use the same repository, then both the pipelines will run using the same commit when one triggers the other. This is helpful if your first pipeline builds the code, and the second pipeline tests it.
However, if the two pipelines use different repositories, then the triggered pipeline will use the latest version of the code from its default branch.
In this case, since master if the default branch of your B-CI, $(Build.SourceBranch) is always refs/heads/master.
As a workaround:
You can create a new yaml pipeline for repository B. You can use similar content of the yaml file for B-CI. And you only need to change something in it to:
resources:
pipelines:
- pipeline: Pipeline_A
source: 'A-CI'
trigger:
branches:
- develop
When we create the new yaml file, it's always placed in master branch. For me, I created a file with same name in dev branch, and copy the same content in it. Then i delete the new yaml file in master branch, now when dev of A-CI pipeline is built, dev of B repos will be used.
I think I have found a nicely working workaround as the built-in pipeline-triggers are not addressing our specific problem (though I can't say if we have an odd approach and there are better ways).
What I am doing now it to use the Azure CLI DevOps extension based on this docs entry and trigger the pipelines manually.
Pipeline YAMLs
A-CI
trigger:
- '*'
stages:
- stage: Build
displayName: Build something and create a pipeline artifact
jobs:
- job: BuildJob
pool:
name: 'MyBuildPool'
steps:
- powershell: |
Write-Host "Building A"
# steps to publish artifact omitted
- stage: TriggerAppPipelines
displayName: Trigger App Pipeline
jobs:
- job: TriggerAppPipelinesJob
displayName: Trigger App Pipeline
steps:
- bash: az extension list | grep azure-devops
displayName: 'Ensure Azure CLI DevOps extension is installed'
- bash: |
echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
az devops configure --defaults organization=https://dev.azure.com/MyOrg project="MyProject" --use-git-aliases true
displayName: 'Login Azure CLI'
env:
AZURE_DEVOPS_CLI_PAT: $(System.AccessToken)
# By passing the build Id of this A-CI run, I can use that in B-CI to download the matching artifact of A-CI.
# If there is no matching branch, then the command fails.
- bash: |
az pipelines run --branch $(Build.SourceBranch) --name "B-CI" --variables a_Version="$(Build.BuildId)" -o none
displayName: 'Trigger pipeline'
B-CI
trigger:
- '*'
stages:
- stage: Build
jobs:
- job: BuildJob
pool:
name: MyBuildPool
steps:
- powershell: |
Write-Host $(Build.SourceBranch) # is same as the the triggering A-CI branch
Write-Host $(Build.Reason) # B-CI is triggered manually but the user is Project Collection Build Service, so automated runs can be distinguished
As B-CI is triggered manually now, there is no need for a resource node anymore.