Background -
I have multiple web service projects under multiple azure repos. All these projects have same structure as they are ASP.NET core web applications. I have written separate yaml pipelines and created separate yaml jobs for each project which has similar steps and placed them in separate azure repos.
Issue -
I have created separate yaml pipeline jobs (multistage - build and deploy) depending on each yaml. Is there any way to consolidate these yaml to one and place to shared azure repo and trigger the pipelines on the code commit to respective azure repos and deploy it to their related environments on azure web services?
Is there any way to consolidate these yaml to one and place to shared azure repo and trigger the pipelines on the code commit to respective azure repos and deploy it to their related environments on azure web services?
The answer is yes.
You could create a new repo with a new YAML file in it, or you could select one repo as main repo and set the YAML with Repository resource:
resources:
repositories:
- repository: A
type: git
name: MyProject/A
ref: main
trigger:
- main
- repository: B
type: git
name: MyProject/B
ref: release
trigger:
- main
- release
The best way to manage these is a separate repo as template. Essentially your template contains the pipeline steps that you want to run, and for each pipeline you need you have a yaml file that extends the template by defining what triggers you want for that pipeline and pass any pipeline dependant parameters you are using to the template.
Templates
Related
Like many before me I'm struggling hard with configuring pipeline triggers in Azure DevOps.
Background:
single project in the Organization
three branches: main, Infrastructure, Application
The branches are kind of independent of each other. They are never merged into main either.
I have a pipeline which deploys two App Services. The YAML file for this pipeline is in the Infrastructure branch. The Default branch for manual and scheduled builds is set to Infrastructure.
Then I have 2 pipelines, each to deploy a different App to the App Service. The YAMLs for those pipelines are in the Application branch. The Default branch for manual and scheduled builds is set to Application.
By themselves, the pipelines work perfectly fine. However what I am trying to achieve is to trigger the App pipelines after the App Service pipeline finishes. And no matter what combination of settings I try, I can't get it to work.
This is currently how it looks like in the n-th version of the YAML:
name: 'deploy-webapp-002'
pool:
vmImage: windows-latest
resources:
pipelines:
- pipeline: 'Deploy App Services' # Internal name of the source pipeline, used elsewhere within this YAML
# e.g. to reference published artifacts
source: deploy-appservices # Azure Pipelines name of the source pipeline referenced
project: HomeLab # Required only if the source pipeline is in another project
trigger:
branches:
include:
- Infrastructure
- Application
pr: none
trigger: none
Is it even possible to do what I'm trying to do?
If yes, what settings should be specified in the Resources/Pipelines section in the YAML, and how should the Default branch for manual and scheduled builds look like for each of those pipelines?
I can reproduce the same issue when I put the YAML files separately in two branches and set the default branch.
Refer to this doc: Define a pipelines resource
When you define a resource trigger, if its pipeline resource is from the same repository (say self) as the current pipeline, triggering follows the same branch and commit on which the event is raised. But, if the pipeline resource is from a different repository, the current pipeline triggers on the default branch of the self repository.
In your caseļ¼ you are using the same repo. So triggering follows the same branch and commit on which the event is raised.
To solve this issue, you need to copy the YAML file in Application Branch to Infrastructure Branch.
On the other hand, you can also try to set Build completion trigger manually on UI.
For example:
I currently have an Azure DevOps yaml file set up to release our project to its necessary environments. In this file, I reference a repository that holds our template files we use within the company.
resources:
repositories:
- repository: templates
type: git
name: 'Project/Repo'
ref: 'refs/tags/x.x.x'
trigger: none
What is odd is that today I made a change to the template repository, and it triggered a build in all our pipelines that use this repository as a resource. I have a trigger set to none, so why is it being triggered with a change to an outside repository?
Edit 1:
Adding an image to show some additional information
One doubt, I am using Azure DevOps for a number of automations. I would like to keep all the pipelines that I am creating in a different repository where the code is located, but both within the same Azure DevOps project. How can I tell the YAML that this pipeline does not point to the repository in which it is located, but to the one in which the code is located?
The tool to do this is a repository. You will add a structure like this:
resources:
repositories:
- repository: Tuto-Ressources
ref: main
type: git
name: TemplateRepository
name is the name of the repository and repository the name of the ressource.
Yu can the invoke it with template.
This is the doc:
https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema#resources
I've read the official documents to put the yaml file in the root of a project. I'm thinking to create a some sort of pipeline repo that contains several yaml files in charge of different pipeline workflow for different project. But Azure pipeline only recognise the azure-pipeline.yaml file name.
Issue:
It is obviously not possible to create several yaml files with the same azure-pipeline.yaml name under the same folder. What's the best practice to organise the azure pipeline yaml files? Shall it be just put in the root of the project?
It sounds like templates might be what you're looking for. This assumes you have a single project/repo and a large pipeline that you'd like to split up so it's easier to read or reason about individual parts.
Taking an example from the linked documentation page, you can define a template yaml file like this (ex: include-npm-steps.yml):
steps:
- script: npm install
- script: yarn install
- script: npm run compile
And then include it as a "module" in the main azure-pipelines.yml file like this:
jobs:
- job: Linux
pool:
vmImage: 'ubuntu-latest'
steps:
- template: templates/include-npm-steps.yml # Template reference
- job: Windows
pool:
vmImage: 'windows-latest'
steps:
- template: templates/include-npm-steps.yml # Template reference
It is obviously not possible to create several yaml files with the
same azure-pipeline.yaml name under the same folder.
Yes, it's not possible to create several yaml pipelines with same name under same folder. Cause the yaml pipeline is under version control and Azure Devops git doesn't support two files with same name in same folder...
What we can do is to create several pipelines with different names in same folder, like azure-pipeline.yaml,azure-pipelines-1.yml,azure-pipelines-2.yml and so on.
Not sure if you know this option when editing yaml pipeline:
We can easily change the yaml file's name in source control, and we just need to modify the path here:
What's the best practice to organise the azure pipeline yaml files?
Shall it be just put in the root of the project?
Assuming you own one Team Project with two repos A and B:
If A and B both represent the module of one final product, then you should have corresponding pipelines for A and B. It means in most scenarios, you should have at least one pipeline in RepoA and one in RepoB. They all need corresponding azure-pipeline.yaml file.
Now if azure-pipeline.yaml in RepoA and azure-pipeline.yaml in RepoB have many same variables/tasks/jobs, we can consider moving the duplicate contents into templates. We can create a RepoC in same project to store the templates, and in this templates repo, we don't need to create yaml pipeline here.
About how to reference templates in RepoC in RepoA's pipeline, see this document. If the source is in github, you can check Krzysztof's link. And if the RepoC is in Azure Devops Repos and same project with your RepoA and RepoB, you can should this format:
resources:
repositories:
- repository: templates
type: git
name: RepoC
ref: refs/heads/master
To sum up, functional repos (those with source code) should have corresponding yaml pipeline in it. And if you want to monitor the changes in one repo (without source code) for some purpose, you can also have one yaml pipeline in that. For templates repo, yaml pipelines are not necessary.
Also, apart from yaml pipelines you may sometimes use Classic Build/Release pipelines which are not under Version Control. See this.
I have a project on Azure DevOps containing multiple forks of the same main repository. I created a build pipeline for that repository which unfortunately cannot be reused for the present forks since a pipeline can only be configured for a single repository.
This solution is not ideal because leads to multiple identical pipelines, one for each fork, and maintaining all of them can be difficult.
Is there a way to use one pipeline for multiple repositories?
you can create a template file and reference that file from each pipeline, that way you can edit a single file and every pipeline will change.
example how to reuse a step file from different repo
resources:
repositories:
- repository: DevOps
type: git
name: DevOps
trigger: none
jobs:
- template: vsts/yaml/build.yaml#DevOps
parameters:
solutionName: xxx
registryName: yyy
You can take a look at the official docs for more examples
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
It's on the roadmap for 2019 Q3:
Multi-repository support for YAML pipelines
https://dev.azure.com/mseng/AzureDevOpsRoadmap/_workitems/edit/1454026
Update: this is now implemented:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#triggers
"Triggers
You can trigger a pipeline when an update is pushed to the self repository or to any of the repositories declared as resources."