I am facing a problem that is confusing me a bit and I would like to find the best approach to avoid to repeat my pipelines.
in this GitHub repos I have my yaml file to build the project, and this yaml targets a folder template on which it runs the C# Build and Publish. Approximately the GitHub repo is structure as follow:
- Folder 1
- Folder 2
- Folder 3
- Azure-Pipelines(build and Publish)
- Azure-pipeline.yaml
During the pipeline run, my yaml targets the Àzure-Pipelines(Build and Publish) folder and build the project. This is my Azure-pipeline.yaml file
stages:
- stage: Build
displayName: 'Build'
jobs:
- template: Azure-Pipelines/build.yaml
parameters:
solution: 'Solution-to-build'
- stage: Publish
displayName: 'Release and Push'
jobs:
- template: Azure-Pipelines/publish.yaml
parameters:
<All the parameters configured for this yaml file>
The template and the structure of my GitHub, keeps repeating themselves, as in each gitrepo I have that Azure-pipeline folder. What I am trying to to. Is to have a GitHub repo Where I keep the build.yaml and publish.yaml. and make all the other repos refer to this folder when the pipeline runs.
Is there any way how I can achieve this?
Please if I am missing any details to make my point clear, just ask. Thank you so much in advance
Is there any way how I can achieve this?
You could try to use the Resources in YAML Pipeline.
Here are the steps:
Step1: Create a Github Service Connection in Project Settings -> Service connection .
Step2: You could try to use the following sample to use the yaml template from another Github Repo.
Example
resources:
repositories:
- repository: MyGitHubRepo # The name used to reference this repository in the checkout step
type: github
endpoint: serviceconnectionname
name: githuborg/reponame #e.g. yy/test
ref: main
pool:
vmImage: ubuntu-latest
stages:
- stage: Build
displayName: 'Build'
jobs:
- template: Azure-Pipelines/build.yml#MyGitHubRepo
Related
I have a repo called REPO A and have created a YAML pipeline in here but I also have some code in REPO B which I need to use in this pipeline
In YAML how can tell the pipeline in REPO A to use files/folders from REPO B?
Im a newbie to this so can you please keep it simple?
resources:
repositories:
- repository: automation
type: git
name: MYORG/ automation
ref: develop
endpoint: YourEndpoint
steps:
- checkout: self
path: "s/Source" : ***>>>>>> what should this be??***
- checkout: automation
You should check out multiple repositories on your YAML pipeline. You can define a new resource repository and checkout this repository along with your source (the one that triggers the pipeline).
resources:
repositories:
- repository: devops_sf
type: github
name: Organization/Repository
ref: master
endpoint: YourEndpoint
steps:
- checkout: self
path: "s/Source"
- checkout: devops_sf
Then you will be able to access your repositories under
$(Pipeline.Workspace)/s/Repository
$(Pipeline.Workspace)/s/Source
Multi repo checkout for Azure DevOps:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops
ok simply put at the top of the yaml i added
> resources:
> repositories:
> - repository: automation
> type: git
> name: name of yourpoject/name of repo you want to use
> ref: whatever branch of that repo(under repository above) you want to use
THEN!! under steps in the YAML (if you havent one you need to add it!)
> steps:
> - checkout: self
> - checkout: your repository you want to use as above
Also if you use npm to run a task you need to add the below as well to not get any file not found errors! a common error if you dont add the below code is : npm WARN saveError ENOENT: no such file or directory, open '/home/vsts/work/package.json
The below code is put in the -Task section
> > npm install
> > displayName: 'a task using npm'
> > workingDirectory: '$(Build.SourcesDirectory)/your repository you want to use as above
ALSO be aware of indentation when inserting code in YAML it's annoying and a pain if you havent got the indentation right it gives you errors just tab until they go away
What I want to achieve:
I have a repository on Azure DevOps which hosts my web application. I wrote a test suite for UI Automation using Cypress. I created a separate repository for my test cases to check if they are working properly or not. I created a pipeline which has the following content:
trigger:
- manual-tests
pool:
vmImage: 'ubuntu-latest'
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
npm install
displayName: 'npm install'
- task: Npm#1
inputs:
command: 'custom'
customCommand: 'run test'
continueOnError: true
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/test-output-*.xml'
testRunTitle: 'My Test Cases'
I have a trigger set to a branch of the repository in which my UI Automation code is stored. What I want is, to trigger my automation script, when there is a push on some branch of the web application repository. Is there a way of doing this? Can we store our test case files in the application repository and give the path of the test script?
It seems that the UI Automation Repo and Web Application Repo are two separate repos.
To trigger my automation script, when there is a push on some branch of the web application repository. Is there a way of doing this?
The function: "trigger a pipeline from a different repo" is not available now.
This feature is still under development. Multi-repository support for YAML pipelines will be available soon for azure devops service.
Please check the function:"Multi-repository support for YAML pipelines" in Azure DevOps Feature Timeline 2020 Q2. This feature will roll out to everyone by the end of July 2020.
Workaround：
You could try to use the Pipeline triggers.
Here are the steps:
Step1: Create a pipeline with web application repository, then you could set the trigger branch.
Step2: Add the Pipeline trigger in the Yaml file (UI Automation Repo).
For example:
resources:
pipelines:
- pipeline: Name
source: Pipeline name
trigger:
branches:
- releases/*
- master
When you make changes in web application repository, the pipeline with the web application will be triggered.
After running the pipeline , the pipeline with UI Automation repo will be triggered.
Can we store our test case files in the application repository and give the path of the test script?
Of cource. You can do it.
If you want to use the test file in the pipeline (UI Automation repo), you could add the repo resouces in the pipeline.
For example:
resources:
repositories:
- repository: MyAzureReposGitRepository
type: git
name: MyProject/WebapplicationRepo
...
steps:
- checkout: MyAzureReposGitRepository
Note: the repo will be check out to the Agent Source Folder.
Hope this helps.
I have a project which depends on 2-3 other projects, is there a way to pull them together with the master project?
When the build process starts projects will be on the file system and the master project can locate the other dependency projects?
As #Kehinde's said in comment, what you want could be achieved by the feature Multi-repo checkout.
Note:
Multi-repo checkout is the feature which only supported YAML. Because what the design logic is Checkouts from multiple repos in combination with YAML builds enable focusing the source level dependency management to one structured descriptor file in Git (the YAML biuld definition) for good visibility.
But for pipeline that configured by classic UI, you had to add those other repositories/projects as submodules, or as manual scripts to run git checkout in pipeline.
For personal, I strongly suggest you use YAML to achieve what you want.
Simple sample YAML definition:
resources:
repositories:
- repository: tools
name: Tools
type: git
steps:
- checkout: self
- checkout: tools
- script: dir $(Build.SourcesDirectory)
Here, image I have a repository called "MyCode" with a YAML pipeline, plus a second repository called "Tools".
In above third step(dir $(Build.SourcesDirectory)), it will show you two directories, "MyCode" and "Tools", in the sources directory.
Hope this helps.
For Bitbucket:
resources:
repositories:
- repository: MyBitBucketRepo
type: bitbucket
endpoint: MyBitBucketServiceConnection
name: {BitBucketOrg}/{BitBucketRepo}
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- checkout: MyBitBucketRepo
- script: dir $(Build.SourcesDirectory)
In Azure DevOps we have created both build and release pipeline using classic way and now we are planning this to convert to yaml file.
But it seems in yaml method, the code can be put only on the root of the repo, where we want to keep the build yaml files in a separate repo, where the developers won't have access.
How can achieve this?
You can use templates, put in the main repo only the minimal yaml that refers to a template with all the steps, the template exits in another repo.
For example, your main repo yaml:
resources:
repositories:
- repository: templates
type: git
name: Contoso/BuildTemplates
jobs:
- template: common.yml#templates # Template reference
In in the repo: Contoso/BuildTemplates put the full yaml:
# Repo: Contoso/BuildTemplates
# File: common.yml
parameters:
vmImage: 'ubuntu 16.04'
jobs:
- job: Build
pool:
vmImage: ${{ parameters.vmImage }}
steps:
- script: npm install
- script: npm test
Restrict the access to the second repo (unless the agent pipeline user).
Read here more info about the resources.
I agree that one solution could be the one proposed by #Shayki Abramczyk
but to have standalone *.yml in dedicated repository you can use 'git clone' while using 'Git Credentials' to access the other repository that contains the files you want to build by the pipeline.
If your repository dedicated for *.yml is in the same Azure Devops project then you should not have any problem with the release definition.
Please see example *.yml that works for us as described:
pool:
vmImage: 'your-preferred-image'
variables:
solution: '$(Agent.BuildDirectory)/**/YourSolution.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Debug'
urlWithCreds: 'https://YourUser:YourPassword#dev.azure.com/YourOrganization/YourProject/
_git/YourOtherRepository'
steps:
- task: CmdLine#2
inputs:
script: |
git --version
git clone --quiet $(urlWithCreds)
git checkout master
- task: VSBuild#1
inputs:
solution: '$(solution)'
msbuildArgs: 'your build args'
platform: '$(buildPlatform)'
configuration: '$(buildConfiguration)'
You don't have to keep the YAML files in the root of the repository; ours are in a dedicated sub-folder:
That's crucial, because it means that we can add a PR policy which restricts who can approve changes to any of the pipeline YAML files.
I have the following repositories:
my-app-repo - Contains the code for the app
pipeline-repo - A collection of pipelines that build my-app-repo as well as others
I'm trying to build a yml pipeline within pipeline-repo that will be triggered on a commit to my-app-repo.
According to the official yml documentation, it sounds possible but I'm not able to get it working.
Here's what I've tried:
pipeline-repo/my-app-repo-build.yml
resources:
repositories:
- repository: target_repo
type: git
name: my-project/my-app_repo
trigger:
branches:
include:
- master
jobs:
- job:
steps:
- script: echo "Should be triggered from a push to my-app-repo!"
The build is not triggering when I push to my-app-repo. It only kicks off for commits to the source repo (pipeline-repo) which I cannot change since that holds the yml definition.
Am I missing something easy?
Edit:
I see you've added an issue to the github repository you mentioned. I agree that it seems to be broken.
While you aren't exactly trying to provide template functionality to your pipelines, you might use it as a work around until the issue you created is addressed.
I've tested this in my playground and it seems to work.
In the app repository
resources:
repositories:
- repository: templates
type: git
name: Pipeline-Templates
trigger:
branches:
include:
- master
jobs:
- template: azure-pipelines-template.yml#templates
In the pipelines repository
jobs:
- job: Get_Last_10_Commits
pool:
vmImage: 'vs2017-win2016'
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
cd $(Build.SourcesDirectory)
Write-Host "Show git log (last 10):"
git log --oneline -10
One "advantage" of this work around is that you no longer need to specify that the repository to checkout is the resource repository and not the self (your template-repository) as the self repository is the app-repo.
This would allow you to restrict changes to the pipeline core by having it in a separate repo like you want, but would still trigger on the app-repo master.
Not ideal b/c you now have 2x .yml files for each build you need, but that's basically the definition of work around: not ideal.
It looks like your include syntax is wrong. Have you tried to use the simple syntax?
All of the examples (and my experience) show that you should use the wildcard syntax when your trigger has an include or exclude specification.
resources:
repositories:
- repository: myPHPApp
type: GitHub
connection: myGitHubConnection
source: ashokirla/phpApp
trigger:
branches:
include:
- features/*
exclude:
- features/experimental/*
paths:
exclude:
- README.md