I have a shared Azure pipeline yaml definition with the purpose to define one CodeAnalysis pipeline per repository.
How can I define the repository name dynamically?
I tried with name: '$(projectName)' which leads to the error:
The repository $(projectName) in project 8ab9d22b-6819-483b-829d-******* could not be retrieved. Verify the name and credentials being used.
azure-pipelines.yml
resources:
repositories:
- repository: codeAnalysisRepo
type: git
name: shared/codeanalysis
- repository: SourceRepo
type: git
name: '$(projectName)'
jobs:
- job: 'BackendCodeAnalysis'
pool:
name: '$(AgentPool)'
steps:
- checkout: SourceRepo
clean: true
- template: sonarqube_msbuild_prepare.yml#codeAnalysisRepo
parameters:
projectKey: '$(project)'
projectName: '$(project)'
- task: DotNetCoreCLI#2
displayName: "build DestRepo"
inputs:
command: 'build'
projects: '$(Build.Repository.LocalPath)/**/*.csproj'
configuration: Release
- template: sonarqube_execute.yml#codeAnalysisRepo
It works when I hardcode the name
Currently, set parameter and variable is not supported in resources -> repositories.
A work around for this, you could set this at the checkout step. Here is s sample: Check out multiple repositories in your pipeline - Azure Pipelines | Microsoft Docs. Please note that, the repos should be in the same organization.
resources:
repositories:
- repository: Repo1
type: git
name: Artifacts/Repo1
jobs:
- job: 'BackendCodeAnalysis'
pool:
vmimage: windows-latest
steps:
- checkout: git://$(projectName)
clean: true
For your demand, you could create a suggestion ticket via: https://developercommunity.visualstudio.com/report?space=21&entry=problem.
Related
I am testing a POC build pipeline and I am trying to use templates as a sort of reference library, so I can reuse the same code in multiple pipelines, while also allowing for simpler editing/updating, if changes are required.
I want to be able to run the pipeline from a DevOps branch and refer to the same branch (or tag) when validating/running all the template pipelines in this build pipeline.
The ultimate goal is to pass multiple variables to the template (via a loop/for each) to create multiple build artifacts referencing public repositories (in this test a public terraform github repo). We could then maybe only need to maintain the variable data and not the source/reference code.
I thought DevOps did this 'OOTB', but realised that this was probably not the case. Something like this:
Repo: InfraAsData (all pipelines)
Branch: feature/sparse-clone-repo
Updated azure-pipeline.yaml and added template templates/sparse-clone.yaml (FYI - this template is untested as well)
azure-pipeline.yaml:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'IAC/InfraAsData'
refs: 'refs/heads/$(branch_name)'
parameters:
- name: branch
displayName: branch
type: string
default: $(Build.SourceBranch)
variables:
- name: branch_name
value: ${{ parameters.branch }}
- template: ./templates/variables/resource-groups.yaml#InfraAsData #variables template
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template-Sparse-Clone-To-Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: $(Build.SourceBranch)/templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: '${{ variables[template.repoUrl] }}'
repoPath: '${{ variables[template.repoPath] }}'
artifactPath: '${{ variables[template.artifactPath] }}'
sparse-clone.yaml
parameters:
repoUrl: ''
repoPath: ''
artifactPath: ''
#testing
steps:
- script: |
md $(Build.BinariesDirectory)/$(parameters.artifactPath)
git clone --filter=blob:none --sparse $(parameters.repoUrl) $(parameters.artifactPath)
cd $(Build.BinariesDirectory)/$(parameters.artifactPath)
git sparse-checkout init --cone
git sparse-checkout set $(parameters.repoPath)
git checkout main
dir $(Build.BinariesDirectory)/$(parameters.artifactPath)
displayName: 'Clone Github Repo Subdirectory - filter blob none'
When I run this from the feature/sparse-clone-repo branch in DevOps I get an error that the new template does not exist in the main branch (it doesn't of course).
/build/test/azure-pipeline.yaml: File /templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/main version
I have also tried using no resources.repository, using $(Build.SourceBranch) $(Build.SourceBranchName) as refs value and as the template path reference (like in the 2nd template example).
EDIT:
If i remove the resource.repositories reference, then the correct branch is identified, but the azure-pipeline.yaml relative path is appended to the template path:
/build/test/azure-pipeline.yaml: File /build/test/$(Build.SourceBranch)/templates/variables/resource-groups.yaml not found in repository https://dev.azure.com//IAC/_git/InfraAsData branch refs/heads/feature/pipelines
So correct branch, but not the correct relative path.
EDIT 2:
I can remove the resources section and use a relative path
../../templates/variables/resource-groups.yaml
but this is not dynamic, to a 'root' reference, so would not work if the folder structure were to change (EG I moved the azure-pipeline.yaml up a level)
I'm sure I am missing something obvious or misunderstanding the docs (or pipeline templates). Any pointers gratefully received!
When I run this from the feature/sparse-clone-repo branch in DevOps I
get an error that the new template does not exist in the main branch
(it doesn't of course).
From your description, seems you thought the 'refs' of the resources determine the branch of the alias of the template part.
But it is not, the 'resources.repositories.repository' doesn't have a section named 'refs', only 'ref' is allowed.
In your situation, you used a section named 'refs' which doesn't exist. Pipeline will use the default branch of the repository to looking for the template YAML. So change the default branch of the repository or change the 'refs' to 'ref' will solve the first issue.
And I notice you were using variable in resources section, you have two mistakes.
One is '$()' can't use in compile time part. Another is even compile time variables are also not allowed in 'ref' section.
ref
string
ref name to checkout; defaults to 'refs/heads/main'. The branch
checked out by default whenever the resource trigger fires. Does not
accept variables.
You can use your pipeline like this:
trigger: none
resources:
repositories:
- repository: InfraAsData #resource name to be used in the build pipeline
type: git #Azure git
name: 'BowmanCP/template_branch'
ref: 'refs/heads/${{parameters.branch}}'
parameters:
- name: branch
displayName: branch
type: string
default: main2
pool:
vmImage: 'ubuntu-latest'
stages:
#sparse clone the target public repo blobs to build pipeline artifacts directory
- stage: Template_Sparse_Clone_To_Artifacts
displayName: 'Test Sparse Clone to Artifacts'
jobs:
- deployment: sparseCloneTfModule
displayName: 'Sparse Clone Terraform module to Artifacts'
environment: Test
strategy:
runOnce:
deploy:
steps:
- template: ./templates/sparse-clone.yaml#InfraAsData #Create artifacts based on template variables
parameters: #module specific var from var template
repoUrl: 'xxx'
repoPath: 'xxx'
artifactPath: 'xxx'
Basically, variables can't be used in your situation and there doesn't have a parameter reuse feature. That's everything.
I have a yaml pipeline which I want to make it to run for more branches. Therefore I am trying to checkout one branch, specified in pipeline variables. I'm unable to do so, the error being Unexpected value 'ref'.
The pipeline file is:
trigger:
batch: true
branches:
include:
- 'release/dev'
- 'release/test'
- 'release/prod'
paths:
include:
- '*'
resources:
- repo: self
ref: $(branch)
variables:
dockerRegistryServiceConnection: 'conn'
containerRegistry: 'conn.reg'
tag: '$(Build.BuildId)'
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build and push stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- template: docker-build-template.yml
parameters:
dockerfilePath: $(Build.SourcesDirectory)/Dockerfile.release
imageName: conn.reg/container
imageRepository: container
pushToRegistry: true
How can I checkout different branches for building the container out of them?
Later edit: I want the pipeline to automatically run after a PR or a commit is pushed on any of the braches. (Manually it can be run with specifiyng a branch.)
Azure DevOps Pipelines: how to check out branch of the self repo?
You could specify the name of the self repo in the resource with a specific ref,like:
resources:
repositories:
- repository: MyTestProject
type: git
name: MyTestProject
ref: $(branch)
Then checkout with another path:
steps:
- checkout: MyTestProject
path: Another path/xxxx/xxx
You don't define the self repo as a resource. If you want to run your build on another branch, just choose your branch in the "Run pipeline" screen:
As for running automatically after completing a PR, you already have the triggers.branches.include set, so merges (or pushes) to all these branches will trigger a build in which the relevant branch will be checked out.
Well, I have the following code that uses a template file in azure Devops:
resources:
repositories:
- repository: templates
type: git
name: "Framework Back-end/templates-devops"
extends:
template: azure-pipelines-template.yml#templates
This works very well, downloading yml file from another project inside same organization. But inside my "azure-pipelines-template.yml" I'm trying to do the following:
- job: Deploy
pool: ${{parameters.agent}}
displayName: Deploy on Kubernetes
dependsOn: Push
condition: and(succeeded(), in(variables['Build.SourceBranchName'], 'master', 'main', 'qas', 'develop'))
steps:
- checkout: self
- checkout: templates
But I got the error:
remote: TF401019: The Git repository with name or identifier templates-devops does not exist or you do not have permissions for the operation you are attempting.
fatal: repository 'https://xx#xx/xxx/Framework%20Back-end/_git/templates-devops/' not found
I need to make a checkout because in other steps I will need to use the files that exist in "template-devops" repository. I can't understand why my pipeline can download the "azure-pipelines-template.yml" file but can't checkout the repository.
SOLVED
Was a permission problem in Settings , I disabled the flags:
Limit job authorization scope to referenced Azure DevOps repositories
Limit job authorization scope to current project for non-release pipelines
Limit job authorization scope to current project for release pipelines
I create a demo to reproduce your environment, but it works well on my side. The checkout step works well. Here is my yaml file and temp file:
Main.yaml
resources:
repositories:
- repository: templates
type: git
name: MyAgile TestPro/yaml
pool:
name: 'default'
extends:
template: azure-pipelines-template.yml#templates
Temp.yaml
stages:
- stage:
jobs:
- job: Deploy
steps:
- checkout: self
- checkout: templates
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Hello World MyAgileTestPro temp"
My test result:
At present, we recommend you can check if your account has the project admin level for your project Framework Back-end. And please check if there is any name changed about your project.
In my graphical version of a release, I am connecting to an "Azure Repo" to pull in a script from it, not a BUILD artifact
Is this possible to do in YAML pipelines? So far it is very unclear, thanks in advance
The repo has a lot of scripts in it, so it doesn't need building or packaging up as a zip/drop package.
Yeah sorry,my bad. Yes it is possible. Following doc
If your pipeline has templates in another repository, or if you want to use multi-repo checkout with a repository that requires a service connection, you must let the system know about that repository. The repository keyword lets you specify an external repository.
resources:
repositories:
- repository: string # identifier (A-Z, a-z, 0-9, and underscore)
type: enum # see the following "Type" topic
name: string # repository name (format depends on `type`)
ref: string # ref name to use; defaults to 'refs/heads/master'
endpoint: string # name of the service connection to use (for types that aren't Azure Repos)
A resource is any external service that is consumed as part of your pipeline. An example of a resource is another CI/CD pipeline that produces:
resources:
pipelines: [ pipeline ]
repositories: [ repository ]
containers: [ container ]
Artifacts like Azure Pipelines or Jenkins.
Code repositories like GitHub, Azure Repos, or Git.
Container-image registries like Azure Container Registry or Docker
hub.
Resources in YAML represent sources of pipelines, containers, repositories, and types. For more information on Resources, see here.
#Hugh Lin - MSFT
#Krzysztof Madej
My Azure DevOps GIT repo called "AZDO_Scripts" (all part of the same project) and has a structure like this...
/Scripts/AzureCLI/ResourceGroup/Provision_ResourceGroup_withTags.ps1
My YAML is like this now...
name: $(date:yyyyMMdd)$(rev:.r)-$(SourceBranchName)
trigger:
- master
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
resources:
repositories:
- repository: commonscripts
type: git
name: AZDO_Scripts
stages:
- stage: Test
variables:
- name: resourcegroup
value: 'rg-testthescript'
- name: location
value: 'WestEurope'
- name: environmenttag
value: 'Test'
jobs:
- job : TestJob
pool:
vmImage: 'windows-latest'
steps:
- task: AzureCLI#2
inputs:
azureSubscription: 'My-Azure-Sub-ServiceConnection'
scriptType: 'ps'
scriptLocation: 'scriptPath'
scriptPath: 'Scripts/AzureCLI/ResourceGroup/Provision_ResourceGroup_withTags.ps1#commonscripts'
arguments: '-resourcegroup $(resourcegroup) -location $(location) -environmenttag $(environmenttag)'
What do I need to do in the scriptPath: statement to reference that script? (2nd line from the bottom)
scriptPath: 'Scripts/AzureCLI/ResourceGroup/Provision_ResourceGroup_withTags.ps1#commonscripts'
I'm working with azure multistage pipelines, using deployment jobs with templates in a separate repo. I'm currently starting to use ARM templates in my deployment process and want to run ARM templates that are located in a different repository as well. This is where I get a little stuck, any help/advice appreciated.
To illustrate my setup:
Repo A -> Source code that has to be build and deployed to azure
Repo B -> Azure pipeline templates (only consists of yml files)
Repo C -> ARM templates
So what I want to accomplish: A uses B uses C.
REPO A: Documentation build and release yml
resources:
repositories:
- repository: templates
type: git
name: <ACCOUNT>/Azure.Pipelines.Templates
ref: refs/tags/2.2.40
stages:
- stage: Build
jobs:
- template: src/jobs/doc-build.yml#templates
- stage: DEV
jobs:
- template: src/deployment-jobs/doc.yml#templates
....
REPO B: Documentation deployment
parameters:
webAppName: ''
connectedServiceName: 'DEV'
jobs:
- deployment: doc_deploy
pool:
name: 'DOC'
environment: 'doc'
strategy:
runOnce:
deploy:
steps:
- template: ../deployment/arm-template.yml
parameters:
connectedServiceName: ${{ parameters.connectedServiceName }}
resourceGroupName: 'documentation'
templateFile: $(Build.SourcesDirectory)/Azure.ARM.Templates/src/web-app/documentation.jsonc
paramFile: $(Build.SourcesDirectory)/Azure.ARM.Templates/src/web-app/documentation-params.json
parameters: -name ${{ parameters.webAppName }}
...
REPO C: contains arm template + param file
The issue I'm facing is that I can't seem to be able to get to the files of repo c. I tried adding another repository entry on multiple levels but it does not seem to clone the dependent repo at all.
My current workaround/solution:
Use a powershell script to manually clone repo C and directly reference the file on disk.
Related github issue: https://github.com/microsoft/azure-pipelines-yaml/issues/103
I've also stumbled upon this issue, having to load arm templates from another repo into the current build. What I did was setting up a build on the arm-template-containing repo, producing a build artifact with following azure-pipelines.yml: (this would be your repo c)
trigger:
- master
steps:
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(System.DefaultWorkingDirectory)/templates'
ArtifactName: 'templates'
publishLocation: 'Container'
Afterwards I could add following step into the actual pipeline:
- task: DownloadPipelineArtifact#2
displayName: 'Get ARM Templates'
inputs:
buildType: 'specific'
project: <YOUR-PROJECT-ID>'
definition: '<ARM-BUILD-DEFINITION-ID>'
buildVersionToDownload: 'latest'
artifactName: 'scripts'
targetPath: '$(Pipeline.Workspace)/templates'
and I was able to access the files as follows:
- task: AzureResourceGroupDeployment#2
displayName: 'Create Queues $(ResourceGroup.Name) '
inputs:
azureSubscription: '<YOUR-SUBSCRIPTION>'
resourceGroupName: '$(ResourceGroup.Name)'
location: '$(ResourceGroup.Location)'
csmFile: '$(Pipeline.Workspace)/templates/servicebus.json'
For more information about the Download Pipeline Artifact task check out following link:
Download Pipeline Artifact task