I have a yaml pipeline which I want to make it to run for more branches. Therefore I am trying to checkout one branch, specified in pipeline variables. I'm unable to do so, the error being Unexpected value 'ref'.
The pipeline file is:
trigger:
batch: true
branches:
include:
- 'release/dev'
- 'release/test'
- 'release/prod'
paths:
include:
- '*'
resources:
- repo: self
ref: $(branch)
variables:
dockerRegistryServiceConnection: 'conn'
containerRegistry: 'conn.reg'
tag: '$(Build.BuildId)'
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- template: docker-build-template.yml
parameters:
dockerfilePath: $(Build.SourcesDirectory)/Dockerfile.release
imageName: conn.reg/container
imageRepository: container
pushToRegistry: true
How can I checkout different branches for building the container out of them?
Later edit: I want the pipeline to automatically run after a PR or a commit is pushed on any of the braches. (Manually it can be run with specifiyng a branch.)
Azure DevOps Pipelines: how to check out branch of the self repo?
You could specify the name of the self repo in the resource with a specific ref,like:
resources:
repositories:
- repository: MyTestProject
type: git
name: MyTestProject
ref: $(branch)
Then checkout with another path:
steps:
- checkout: MyTestProject
path: Another path/xxxx/xxx
You don't define the self repo as a resource. If you want to run your build on another branch, just choose your branch in the "Run pipeline" screen:
As for running automatically after completing a PR, you already have the triggers.branches.include set, so merges (or pushes) to all these branches will trigger a build in which the relevant branch will be checked out.
Related
I have a shared Azure pipeline yaml definition with the purpose to define one CodeAnalysis pipeline per repository.
How can I define the repository name dynamically?
I tried with name: '$(projectName)' which leads to the error:
The repository $(projectName) in project 8ab9d22b-6819-483b-829d-******* could not be retrieved. Verify the name and credentials being used.
azure-pipelines.yml
resources:
repositories:
- repository: codeAnalysisRepo
type: git
name: shared/codeanalysis
- repository: SourceRepo
type: git
name: '$(projectName)'
jobs:
- job: 'BackendCodeAnalysis'
pool:
name: '$(AgentPool)'
steps:
- checkout: SourceRepo
clean: true
- template: sonarqube_msbuild_prepare.yml#codeAnalysisRepo
parameters:
projectKey: '$(project)'
projectName: '$(project)'
- task: DotNetCoreCLI#2
displayName: "build DestRepo"
inputs:
command: 'build'
projects: '$(Build.Repository.LocalPath)/**/*.csproj'
configuration: Release
- template: sonarqube_execute.yml#codeAnalysisRepo
It works when I hardcode the name
Currently, set parameter and variable is not supported in resources -> repositories.
A work around for this, you could set this at the checkout step. Here is s sample: Check out multiple repositories in your pipeline - Azure Pipelines | Microsoft Docs. Please note that, the repos should be in the same organization.
resources:
repositories:
- repository: Repo1
type: git
name: Artifacts/Repo1
jobs:
- job: 'BackendCodeAnalysis'
pool:
vmimage: windows-latest
steps:
- checkout: git://$(projectName)
clean: true
For your demand, you could create a suggestion ticket via: https://developercommunity.visualstudio.com/report?space=21&entry=problem.
I have 30 or so Java Microservices that run of the same ci and cd template. i.e. Each of my Microservices has a build pipeline as follows and as shown below it runs automatically on a merge to master:
name: 0.0.$(Rev:r)
trigger:
- master
pr: none
variables:
- group: MyCompany
resources:
repositories:
- repository: templates
type: git
name: <id>/yaml-templates
stages:
- stage: Build
jobs:
- job: build
displayName: Build
steps:
- template: my-templates/ci-build-template.yaml#templates
- stage: PushToContainerRegistry
dependsOn: Build
condition: succeeded()
jobs:
- job: PushToContainerRegistry
displayName: PushToContainerRegistry
steps:
- template: my-templates/ci-template.yaml#templates
Where ci-build-template.yaml contains...
steps:
- checkout: self
path: s
- task: PowerShell#2
- task: Gradle#2
displayName: 'Gradle Build'
- task: SonarQubePrepare#4
displayName: SonarQube Analysis
- task: CopyFiles#2
displayName: Copy build/docker to Staging Directory
I would like to implement pr build validation wherever someone raises a pr to merge into master. In the PR build only the Build stage should run and from the build template only some tasks within ci-build-template.yaml should run.
A few questions for my learning:
How can i uplift the yaml pipeline above to make the "PushToContainerRegistry" skip if it is a pr build?
How can i uplift ci-build-template.yaml to make the "SonarQubePrepare#4" and "CopyFiles#2" skip if it is a pr build?
And lastly how can i uplift the yaml pipeline above to enable build validation for all pr's that have a target branch of master?
Whilst doing my own research i know you can do this via clickops but I am keep on learning on how to implement via yaml.
thanks
How can i uplift the yaml pipeline above to make the
"PushToContainerRegistry" skip if it is a pr build?
How can i uplift ci-build-template.yaml to make the
"SonarQubePrepare#4" and "CopyFiles#2" skip if it is a pr build?
Just need to use condition of task:
For example,
pool:
name: Azure Pipelines
steps:
- script: |
echo Write your commands here
echo Hello world
echo $(Build.Reason)
displayName: 'Command Line Script'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
Above definition will skip the step if Pull request trigger the pipeline.
Please refer to these documents:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops&tabs=yaml#using-the-trigger-type-in-conditions
https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#build-variables-devops-services
And lastly how can i uplift the yaml pipeline above to enable build
validation for all pr's that have a target branch of master?
You can use this expression in the condition:
eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/master')
If you are based on DevOps git repo, then just need to add branch policy is ok:
https://learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops&tabs=browser#configure-branch-policies
Well, I have the following code that uses a template file in azure Devops:
resources:
repositories:
- repository: templates
type: git
name: "Framework Back-end/templates-devops"
extends:
template: azure-pipelines-template.yml#templates
This works very well, downloading yml file from another project inside same organization. But inside my "azure-pipelines-template.yml" I'm trying to do the following:
- job: Deploy
pool: ${{parameters.agent}}
displayName: Deploy on Kubernetes
dependsOn: Push
condition: and(succeeded(), in(variables['Build.SourceBranchName'], 'master', 'main', 'qas', 'develop'))
steps:
- checkout: self
- checkout: templates
But I got the error:
remote: TF401019: The Git repository with name or identifier templates-devops does not exist or you do not have permissions for the operation you are attempting.
fatal: repository 'https://xx#xx/xxx/Framework%20Back-end/_git/templates-devops/' not found
I need to make a checkout because in other steps I will need to use the files that exist in "template-devops" repository. I can't understand why my pipeline can download the "azure-pipelines-template.yml" file but can't checkout the repository.
SOLVED
Was a permission problem in Settings , I disabled the flags:
Limit job authorization scope to referenced Azure DevOps repositories
Limit job authorization scope to current project for non-release pipelines
Limit job authorization scope to current project for release pipelines
I create a demo to reproduce your environment, but it works well on my side. The checkout step works well. Here is my yaml file and temp file:
Main.yaml
resources:
repositories:
- repository: templates
type: git
name: MyAgile TestPro/yaml
pool:
name: 'default'
extends:
template: azure-pipelines-template.yml#templates
Temp.yaml
stages:
- stage:
jobs:
- job: Deploy
steps:
- checkout: self
- checkout: templates
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Hello World MyAgileTestPro temp"
My test result:
At present, we recommend you can check if your account has the project admin level for your project Framework Back-end. And please check if there is any name changed about your project.
When a pipeline is triggered by a repository resource trigger I need to be able to determine metadata about the triggering repository (such as repo name and branch).
It sounds like these variables should be set during a pipeline run that was triggered by a repository resource; however, their values are blank when I do echo $(resources.triggeringAlias) or env | sort in a pipeline run that was triggered by a repository resource.
UPDATE: the predefined variables for Build.Repository.Name and Build.SourceBranchName now work as expected when used with a repository trigger. So, while I'm no longer in need of resources.triggeringAlias and resources.triggeringCategory, they still aren't working.
I need to be able to determine metadata about the triggering
repository (such as repo name and branch)
We can use $(Build.Repository.Name) and $(Build.Repository.Uri) to get repo name and repo uri.
And $(Build.SourceVersion) can be used to get CommitId, $(Build.SourceBranch) or $(Build.SourceBranchName) can be used to get branch info.
Just note we must also checkout the triggering repo to make above variables work to fetch the info of triggering repo, otherwise those variables will always represents the value of triggered repo:
steps:
- checkout: self
- checkout: TheTriggeringRepo
Some details:
I have a triggering repo PipelineA, and the triggered repo PipelineB. PipelineB's azure-pipelines.yml file:
resources:
repositories:
- repository: PipelineA
type: git
name: PipelineA
trigger:
- master
steps:
- checkout: self
- checkout: PipelineA
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo $(Build.Repository.Name)
echo $(Build.SourceBranch)
So this pipeline will be triggered by both PipelineA repo and PipelineB repo.
When PipelineB repo has changes:
When PipelineA repo has changes:
It's clear the $(Build.Repository.Name) variable can work well to output the real trigger repo if we checkout both these two repos. So just make sure you checkout those triggering repos, then my variables above would work for you.
resources:
pipelines:
- pipeline: client-qa-deployment ## Resources.TriggeringAlias will be 'client-qa-deployment' if this resource gets triggered.
source: benjose.dev-code
trigger:
stages:
- client_deploy_qa
- stage: qa_smoketest
displayName: Smoke Test on QA Environment
condition: eq(variables['Resources.TriggeringAlias'], 'client-qa-deployment')
pool:
vmImage: 'windows-latest'
jobs:
- job: TestQA
steps:
- script: echo Hello, $(Resources.TriggeringAlias)!
displayName: 'Running Smoke Test on QA Environment'
Hello, client-staging-deployment!
I believe the above snippet will help you to understand the syntax and usage of Resources.TriggeringAlias. This variable will be helpful if we have multiple resources in the same pipeline and to find out which resource got trigged.
I'm studying about azuredevops pipelines to a new project. I'm totally new about devops.
In this project, I want to use lerna.js to manage my monorepo and my packages.
Considering that I have those three projects inside my monorepo:
Package 1 (1.0.1)
Package 2 (1.0.0)
Package 3 (1.0.3)
And I want to create a new TAG, which will increase the Package 3 to (1.0.4). How can I trigger an Azure Pipeline just to Package 3? There is any guide?
I watched one talk about monorepos with lerna.js and I'm trying to figure out if Azure Pipelines has one feature similar to what Sail CI does. In the exemple we have this approch:
tasks:
build-package-1:
image: sailci/demo
when:
paths:
- "packages/package-1/**/*"
The company I'm working at is using Azure DevOps, would be awesome know if I can have that feature there.
The closest thing similar to that one is like #Simon Ness wrote multiple pipelines with path filters. Additionally if your packages have similar strucutre and require the same steps to creeate/test/publish package you should consider templates.
The conspet toi handle packages like you described can be similar to below steps.
template.yaml
parameters:
- name: workingDir
type: string
default: package-1
steps:
- script: npm install
workingDiretory: ${{ parameters.workingDir }}
- script: yarn install
workingDiretory: ${{ parameters.workingDir }}
- script: npm run compile
workingDiretory: ${{ parameters.workingDir }}
then pipeline for Package-1 may look like this:
trigger:
branches:
include:
- master
- releases/*
paths:
include:
- Package-1/*
steps:
- template: templates/template.yaml
parameters:
workingDir: Package-1
and for Package-2:
trigger:
branches:
include:
- master
- releases/*
paths:
include:
- Package-2/*
steps:
- template: templates/template.yaml
parameters:
workingDir: Package-2
EDIT
For tag part all you need to do is change trigger section:
trigger:
branches:
include:
- master
- refs/tags/*
and when you create a tag and push it:
git tag release-05
git push origin --tags
your pipeline wil start:
However, trigger works like or condition, so for any change on master or new tag piepline will start. So if you tag another branch (not master) pipeline will start.
This is why you may need to check if your source branch is the one from trigger section:
trigger:
branches:
include:
- master
- refs/tags/*
stages:
- stage: A
condition: eq(variables['Build.SourceBranch'], 'master')
jobs:
- job: JA
steps:
- script: |
echo "This is job Foo."
Above pipeline will run for:
change in master branch
any tag pushed to server, but it runs stage A only if you push tag to master branch
If you're happy to define a separate pipeline for each package take a look at paths in CI triggers.
Tags can be used as a trigger by including the branch - refs/tags/* in the triggers section.