How to use powershell/bash script file in azure pipeline template - azure-devops

example: Azure Devops, I have one organization, few projects, and few repositories inside each project (most of them contains build pipelines):
ORGANIZATION:
..-- Project1
.....-- Repo1
.....-- Repo2
..-- Project2
.....-- Repo1
.....-- Repo2
..-- BuildTemplates
.....-- BuildTemplatesRepository
........-- Template1.yml
........-- Template2.yml
........-- Template1.ps1
Template1.yml contains powershell task or step:
- pwsh: ./Template1.ps1
Problem:
When Template1.yml executes inside pipelines from another repo (Project1/Repo1/azure-pipelines.yml) I get error:
[error]ENOENT: no such file or directory, stat '/home/vsts/work/1/s/Template1.ps1'
I understand why there is error, because *.ps1 file isn't copied inside container where process is going on, but how to solve this issue in best way (without coping this script manually)?

For this issue , you can check out multiple repositories in your pipeline.
Pipelines often rely on multiple repositories. You can have different repositories with source, tools, scripts, or other items that you need to build your code. By using multiple checkout steps in your pipeline, you can fetch and check out other repositories in addition to the one you use to store your YAML pipeline.
So you can check out other repo that contains Template1.ps1 in the pipeline .
steps:
- checkout: git://MyProject/MyRepo # Azure Repos Git repository in the same organization
For details ,please refer to this official document.

Related

Azure Devops : Multiple Project and Repo checkout

I would like to run a yaml pipeline from one project. I have a task in my yaml to scan all the source code. Using this Yaml I would like to scan all the source code in master branch for all the project and all the repository inside the same Org.
How can I get all the repo for all the project and iterate? Can someone help me ?
test.yaml
repositories:
- repository: justAnotherName
type: github
name: myGitRepo
endpoint: myGitServiceConnection
trigger:
branches:
include:
- master
steps:
- task: CredScan#2
inputs:
toolMajorVersion: 'V2'
outputFormat: 'tsv'
scanFolder: '$(Build.SourcesDirectory)'
If you're looking to pull every repo within a project, you have one of two options (see below). However, I'd advise caution before attempting this on a Microsoft-hosted agent, they have a 60-minute timeout by default. If you're using a self-hosted agent, you need not worry. I'd still advise breaking this up to avoid creating a long-running release that also consumes a large amount of disk space with each run.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/phases?view=azure-devops&tabs=yaml#timeouts
That being said, here are the options you have:
Option 1 (Not the best)
Manually add a repository: dependency for every project and a checkout: task for every repo within the projects.
This is heavily manual and would require maintenance every time a report is added.
Option 2
You can write a custom PowerShell/bash script that uses the Azure DevOps API and git to automatically scan all projects and repos within the org and pull them onto the machine.
Start by issuing a request to get all of the projects within the org:
https://learn.microsoft.com/en-us/rest/api/azure/devops/core/projects/list?view=azure-devops-rest-6.0
Then, iterate through every project and get all repos:
https://learn.microsoft.com/en-us/rest/api/azure/devops/git/repositories/list?view=azure-devops-rest-6.0
Finally, iterate through each repo and run git clone [repository URL] to clone it onto the build agent.
NOTE: You will want to ensure to have a lot of free disk space on the agent machine and that you clean up the build space after this operation.

Azure DevOps Yaml file location

When a pipeline is created it must be specified its file path and branch. When this pipeline runs a branch is asked again. What is the purpose of this second branch? My initial thought was that the branch where my code is located and the branch where the Yaml file is located are two different things.
However, it seems, after some issues, that when I select a branch in the Run pipeline dialog both the Yaml file and the source code are checked out from this same branch. Is it correct?
Thanks for any help.
This is the expected behavior.
In Azure Devops, the location of the Yaml file is in the branch of the repo, so thehe executed yaml file and the source branch are synchronized.
The first operation of selecting a branch is used to create the pipeline, if you run directly for the first time, you don’t need to choose the execution branch.
When you run a DevOps pipeline(select execution branch), you choose a branch to execute the pipeline. The Yaml file in that branch is the one that will be executed by default and the pipeline will checkout the source code from the same branch.
The version that gets executed will be determined by which branch you're running the pipeline for.
If you want to checkout the source files of other branches in a yaml file, you need to add an additional repo source.
For example:
resources:
repositories:
- repository: TestRepo
ref: refs/heads/branchname
type: git
name: Projectname/RepoName
stages:
- stage: deploy
jobs:
- job: test
steps:
- checkout: TestRepo

Including the build of one repo to another repo in Azure DevOps

I have a repo in Azure devops which is using 'ant all' command to build an ear file. This ear file contain should a war file built from another repo in the azure.
How can I include this war file in the build?
Thanks.
If your war file is built from code hosted in another repo and not is a part of the repo itself. You can publish pipeline artifact and download it in your next pipeline. Here you have doc article about this. And here description of the task itself.
In short to publish you should add code like this:
steps:
- publish: $(System.DefaultWorkingDirectory)/bin/WebApp
artifact: WebApp
and to download:
# Download artifacts from a specific pipeline.
- task: DownloadPipelineArtifact#2
inputs:
source: 'specific'
project: 'FabrikamFiber'
pipeline: 12
runVersion: 'latest'
If you mean the war file is already in another repo in the same organization, you could check multiple checkout, by using multiple checkout steps in your pipeline, you can fetch and check out other repositories in addition to the one you use to store your YAML pipeline.
steps:
- checkout: git://MyProject/MyRepo # Azure Repos Git repository in the same organization
- checkout: self

How to organize azure-pipeline.yaml files

I've read the official documents to put the yaml file in the root of a project. I'm thinking to create a some sort of pipeline repo that contains several yaml files in charge of different pipeline workflow for different project. But Azure pipeline only recognise the azure-pipeline.yaml file name.
Issue:
It is obviously not possible to create several yaml files with the same azure-pipeline.yaml name under the same folder. What's the best practice to organise the azure pipeline yaml files? Shall it be just put in the root of the project?
It sounds like templates might be what you're looking for. This assumes you have a single project/repo and a large pipeline that you'd like to split up so it's easier to read or reason about individual parts.
Taking an example from the linked documentation page, you can define a template yaml file like this (ex: include-npm-steps.yml):
steps:
- script: npm install
- script: yarn install
- script: npm run compile
And then include it as a "module" in the main azure-pipelines.yml file like this:
jobs:
- job: Linux
pool:
vmImage: 'ubuntu-latest'
steps:
- template: templates/include-npm-steps.yml # Template reference
- job: Windows
pool:
vmImage: 'windows-latest'
steps:
- template: templates/include-npm-steps.yml # Template reference
It is obviously not possible to create several yaml files with the
same azure-pipeline.yaml name under the same folder.
Yes, it's not possible to create several yaml pipelines with same name under same folder. Cause the yaml pipeline is under version control and Azure Devops git doesn't support two files with same name in same folder...
What we can do is to create several pipelines with different names in same folder, like azure-pipeline.yaml,azure-pipelines-1.yml,azure-pipelines-2.yml and so on.
Not sure if you know this option when editing yaml pipeline:
We can easily change the yaml file's name in source control, and we just need to modify the path here:
What's the best practice to organise the azure pipeline yaml files?
Shall it be just put in the root of the project?
Assuming you own one Team Project with two repos A and B:
If A and B both represent the module of one final product, then you should have corresponding pipelines for A and B. It means in most scenarios, you should have at least one pipeline in RepoA and one in RepoB. They all need corresponding azure-pipeline.yaml file.
Now if azure-pipeline.yaml in RepoA and azure-pipeline.yaml in RepoB have many same variables/tasks/jobs, we can consider moving the duplicate contents into templates. We can create a RepoC in same project to store the templates, and in this templates repo, we don't need to create yaml pipeline here.
About how to reference templates in RepoC in RepoA's pipeline, see this document. If the source is in github, you can check Krzysztof's link. And if the RepoC is in Azure Devops Repos and same project with your RepoA and RepoB, you can should this format:
resources:
repositories:
- repository: templates
type: git
name: RepoC
ref: refs/heads/master
To sum up, functional repos (those with source code) should have corresponding yaml pipeline in it. And if you want to monitor the changes in one repo (without source code) for some purpose, you can also have one yaml pipeline in that. For templates repo, yaml pipelines are not necessary.
Also, apart from yaml pipelines you may sometimes use Classic Build/Release pipelines which are not under Version Control. See this.

Reuse same build pipeline for different repository on Azure DevOps

I have a project on Azure DevOps containing multiple forks of the same main repository. I created a build pipeline for that repository which unfortunately cannot be reused for the present forks since a pipeline can only be configured for a single repository.
This solution is not ideal because leads to multiple identical pipelines, one for each fork, and maintaining all of them can be difficult.
Is there a way to use one pipeline for multiple repositories?
you can create a template file and reference that file from each pipeline, that way you can edit a single file and every pipeline will change.
example how to reuse a step file from different repo
resources:
repositories:
- repository: DevOps
type: git
name: DevOps
trigger: none
jobs:
- template: vsts/yaml/build.yaml#DevOps
parameters:
solutionName: xxx
registryName: yyy
You can take a look at the official docs for more examples
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
It's on the roadmap for 2019 Q3:
Multi-repository support for YAML pipelines
https://dev.azure.com/mseng/AzureDevOpsRoadmap/_workitems/edit/1454026
Update: this is now implemented:
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#triggers
"Triggers
You can trigger a pipeline when an update is pushed to the self repository or to any of the repositories declared as resources."