Template:
parameters:
- name: PathPrefix
displayName: 'Path prefix'
type: string
default: ''
steps:
- task: DotNetCoreCLI#2
displayName: 'dotnet restore'
inputs:
command: restore
projects: ${{parameters.PathPrefix}}**/$(Build.DefinitionName).sln
Pipeline:
resources:
repositories:
- repository: devops
name: foo/devops
type: git
ref: master
trigger:
branches:
include:
- refs/heads/*
jobs:
- job: Job_1
displayName: Agent job 1
pool:
vmImage: windows-latest
steps:
- checkout: self
- template: azure/pipelines/pipeline.yaml#devops
parameters:
PathPrefix: $(Build.DefinitionName)
Run error during the restore step:
##[error]No files matched the search pattern.
Even setting verbosityRestore: detailed on the restore step doesn't give me any more information.
If I don't set PathPrefix, it seems to use the default empty string and find the solution file (in some cases). However, if I do set the prefix, which is needed in some repos, it can't find the file. I've tried various ways of referencing the parameter within the template (${{}}, $(), $[] and others) and different ways of specifying it within the pipeline, including hard-coding the path (though I want to use the variable instead), but nothing works.
I thought maybe variables would work instead, so I also tried specifying variables in the pipeline and using them in the template, but that results in the same error. Defining the variable in the template gave me a compilation error for the template (unexpected token 'variable' or something similar).
Look at what you're passing and mentally expand the results.
${{parameters.PathPrefix}}**/$(Build.DefinitionName).sln
If Build.DefinitionName is foo, and you pass that in as PathPrefix, then what you get is:
foo**/foo.sln
It looks like you want an extra forward slash in there, so you get foo/**/foo.sln.
Related
I have run in to an odd problem after converting a bunch of my YAML pipelines to use templates for holding job logic as well as for defining my pipeline variables. The pipelines run perfectly fine, however I get a "Some recent issues detected related to pipeline trigger." warning at the top of the pipeline summary page and viewing details only states: "Configuring the trigger failed, edit and save the pipeline again."
The odd part here is that the pipeline works completely fine, including triggers. Nothing is broken and no further details are given about the supposed issue. I currently have YAML triggers overridden for the pipeline, but I did also define the same trigger in the YAML to see if that would help (it did not).
I'm looking for any ideas on what might be causing this or how I might be able to further troubleshoot it given the complete lack of detail that the error/warning provides. It's causing a lot of confusion among developers who think there might be a problem with their builds as a result of the warning.
Here is the main pipeline. the build repository is a shared repository for holding code that is used across multiple repos in the build system. dev.yaml contains dev environment specific variable values. Shared holds conditionally set variables based on the branch the pipeline is running on.
name: ProductName_$(BranchNameLower)_dev_$(MajorVersion)_$(MinorVersion)_$(BuildVersion)_$(Build.BuildId)
resources:
repositories:
- repository: self
- repository: build
type: git
name: Build
ref: master
# This trigger isn't used yet, but we want it defined for later.
trigger:
batch: true
branches:
include:
- 'dev'
variables:
- template: YAML/variables/shared.yaml#build
- template: YAML/variables/dev.yaml#build
jobs:
- template: ProductNameDevJob.yaml
parameters:
pipelinePool: ${{ variables.PipelinePool }}
validRef: ${{ variables.ValidRef }}
Then this is the start of the actual job yaml. It provides a reusable definition of the job that can be used in more than one over-arching pipeline:
parameters:
- name: dependsOn
type: object
default: {}
- name: pipelinePool
default: ''
- name: validRef
default: ''
- name: noCI
type: boolean
default: false
- name: updateBeforeRun
type: boolean
default: false
jobs:
- job: Build_ProductName
displayName: 'Build ProductName'
pool:
name: ${{ parameters.pipelinePool }}
demands:
- msbuild
- visualstudio
dependsOn:
- ${{ each dependsOnThis in parameters.dependsOn }}:
- ${{ dependsOnThis }}
condition: and(succeeded(), eq(variables['Build.SourceBranch'], variables['ValidRef']))
steps:
**step logic here
Finally, we have the variable YAML which conditionally sets pipeline variables based on what we are building:
variables:
- ${{ if or(eq(variables['Build.SourceBranch'], 'refs/heads/dev'), eq(variables['Build.SourceBranch'], 'refs/heads/users/ahenderson/azure_devops_build')) }}:
- name: BranchName
value: Dev
** Continue with rest of pipeline variables and settings of each value for each different context.
You can check my post here : Azure DevOps pipeline trigger issue message not going away
As I can see in your YAML file, you are using this branch : 'refs/heads/users/ahenderson/azure_devops_build'.
I think some YAML files you are refering are missing from the branch defined as default in your build there :
Switch to your branch
I think I may have figured out the problem. It appears that this is related to the use of conditionals in the variable setup. While the variables will be set in any valid trigger configuration, it appears that the proper values are not used during validation and that may have been causing the problem. Switching my conditional variables to first set a default value and then replace the value conditionally seems to have fixed the problem.
It would be nice if Microsoft would give a more useful error message here, something to the extent of the values not being found for a given variable, but adding defaults does seem to have fixed the problem.
In our case it was because the path to the YAML file started with a slash: /builds/build.yaml
Removing the slash fixed the error: builds/build.yaml
In my case I had replaced the trigger in the yaml-file. The pipeline then does not now where to start.
# ASP.NET
# Build and test ASP.NET projects.
# Add steps that publish symbols, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/apps/aspnet/build-aspnet-4
trigger:
- "main" # <-- That was name wrong
For me it was this causing the problem...
The following pipeline.yaml causes the error “Configuring the trigger failed, edit and save the pipeline again”. It has to do with the environment name name: FeatureVMs.${{ variables.resourceName }}, if i replace ${{ variables.resourceName }} with something else e.g. FeatureVMs.develop then the error does not occur. The strange thing is, if i once save the pipeline with all triggers I want and a valid environment FeatureVMs.develop then it saves the triggers, if i then change it to what i actually want a dynamic environment resource selection FeatureVMs.${{ variables.resourceName }} then the error occurs but Azure Dev Ops the pipeline works as i expect it. So the workaround is to save it once without a variable and the triggers you want and then with the variable and live with the error on top of the pipeline
This causes the error
trigger: none
variables:
- name: resourceName
value: $(Build.SourceBranchName)
- name: sourcePipeline
value: vetsxl-ci
resources:
pipelines:
- pipeline: vetsxl-ci
source: vetsxl-ci
trigger:
branches:
include:
- develop
- feature/F*
- release/*
- review/*
- demo/*
- hotfix/H*
- tests/*
- test/*
stages:
- stage: Deploy
displayName: 'Deploy'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
jobs:
- deployment: DeployVM
displayName: 'Deploy to develop VM'
environment:
name: FeatureVMs.${{ variables.resourceName }}
strategy:
rolling:
deploy:
steps:
- template: deploy.yml
parameters:
sourcePipeline: ${{ variables.sourcePipeline }}
This works without any errors.
trigger: none
variables:
- name: resourceName
value: $(Build.SourceBranchName)
- name: sourcePipeline
value: vetsxl-ci
resources:
pipelines:
- pipeline: vetsxl-ci
source: vetsxl-ci
trigger:
branches:
include:
- develop
- feature/F*
- release/*
- review/*
- demo/*
- hotfix/H*
- tests/*
- test/*
stages:
- stage: Deploy
displayName: 'Deploy'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
jobs:
- deployment: DeployVM
displayName: 'Deploy to develop VM'
environment:
name: FeatureVMs.develop
strategy:
rolling:
deploy:
steps:
- template: deploy.yml
parameters:
sourcePipeline: ${{ variables.sourcePipeline }}
I have a solution where a git branch is directly related to an environment (this has to be this way, so please do not discuss whether this is good or bad, I know it is not best practice).
We have the option to run a verification deployment (including automatic tests) towards an environment, without actually deploying the solution to the environment. Because of this, I would like to set up a pipeline that runs this verification for an environment, whenever a pull request is opened towards that environment's branch. Moreover, I am using a template for the majority of the pipeline. The actual pipeline in the main repository is just a tiny solution that points towards the template pipeline in another repository. This template, in turn, has stages for each respective environment.
I have, in the main pipeline, successfully added a solution that identifies the current branch, which for pull requests should be the target branch:
variables:
- name: currentBranch
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
value: $(System.PullRequest.TargetBranch)
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
value: $(Build.SourceBranch)
I would like to send this variable currentBranch down to the template through a parameter, as my template pipeline has different stages depending on the branch. My solution was to use the pipeline like this:
extends:
template: <template-reference>
parameters:
branch: $(currentBranch)
...and then for a stage in my pipeline do this:
- stage: TestAndDeployBranchName
condition: eq('${{ parameters.branch }}', 'refs/heads/branchName')
jobs:
- job1... etc.
Basically, the stage should run if the current branch is either "branchName", or (for pull requests) when the target branch is "branchName", which comes from the "branch" parameters that is sent to the template.
However, I see here that System.PullRequest.TargetBranch is not available for templates and further here that the parameters are not available for templates (the variable is empty) when the template is expanded. Thus my pipeline does not work as expected (the condition does not trigger when it should, ie. when there is a match on the branch name).
Is there any way that I can use System.PullRequest.TargetBranch in a condition within a template, or should I look for another solution?
After investigating this further I concluded that what I am trying to do is not possible.
In short, System.PullRequest.TargetBranch (and I assume at least some other variables within System.PullRequest are not available in compile time for template, which is when conditions are evaluated. Thus, using these variables in a condition in a template is not possible.
As my goal was to have certain steps run for pull requests only, based on the target branch of the pull request, I solved this by creating duplicate pipelines. Each pipeline is the same and references the same template, except for that the input parameter for the template is different. I then added each "PR pipelines" to run as part of the branch policy each respective branch this was applicable.
This works great, however it requires me to create a new pipeline if I have the same requirement for another branch. Moreover, I have to maintain each PR pipeline separately (which can be both good and bad).
Not an ideal solution, but it works.
Reference PR pipeline:
trigger: none # no trigger as PR triggers are set by branch policies
#This references the template repository to reuse the basic pipeline
resources:
repositories:
- repository: <template repo>
type: git # "git" means azure devops repository
name: <template name> # Syntax: <project>/<repo>
ref: refs/heads/master # Grab latest pipeline template from the master branch
stages:
- stage: VerifyPullRequest
condition: |
and(
not(failed()),
not(canceled()),
eq(variables['Build.Reason'], 'PullRequest')
)
displayName: 'Verify Pull Request'
jobs:
- template: <template reference> # Template reference
parameters:
image: <image>
targetBranch: <targetBranch> # Adjust this to match each respective relevant branch
The targetBranch parameter is the used in relevant places in the template to run PR verification.
Example of branch policy:
(Set this up for each relevant branch)
Picture of branch policy set up
After checking your script, we find we can not use the
variables:
- name: currentBranch
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
value: $(System.PullRequest.TargetBranch)
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
value: $(Build.SourceBranch)
in the variables.
The variables will duplicate the second value to first one.
This will cause your issue.
So, on my side, I create a work around and hope this will help you. Here is my main yaml:
parameters:
- name: custom_agent
displayName: Use Custom Agent
type: boolean
default: true
- name: image
type: string
default: default
resources:
repositories:
- repository: templates
type: git
name: Tech-Talk/template
trigger: none
pool:
vmImage: windows-latest
# vmImage: ubuntu-20.04
stages:
- stage: A
jobs:
- job: A1
steps:
- task: PowerShell#2
name: printvar
inputs:
targetType: 'inline'
script: |
If("$(Build.Reason)" -eq "PullRequest"){
Write-Host "##vso[task.setvariable variable=currentBranch;isOutput=true]$(System.PullRequest.TargetBranch)"
}
else{
Write-Host "##vso[task.setvariable variable=currentBranch;isOutput=true]$(Build.SourceBranch)"
}
- stage: B
condition: eq(dependencies.A.outputs['A1.printvar.currentBranch'], 'refs/heads/master')
dependsOn: A
jobs:
- job: B1
variables:
varFromA: $[ stageDependencies.A.A1.outputs['printvar.currentBranch'] ]
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "$(varFromA)"
- template: temp.yaml#templates
parameters:
branchName: $(varFromA)
agent_pool_name: ''
db_resource_path: $(System.DefaultWorkingDirectory)
Please Note:
If we use this, we need to modified your temp yaml.
We need to move the condition to the main yaml and make the temp yaml only steps is left.
Let's suppose I have 3 environments on Azure: Dev, Test and Prod. I have the same pipeline for building and deploying the resources and the code for each one of the environments except for two differences:
different trigger branch
different variable values
What is the correct approach for this scenario? Because at least 3 come to my mind, none of which is perfect:
Option 1: I guess I could create a single pipeline on Azure DevOps (triggered by any of 3 branches) with 3 stages for each environment and for each stage add a condition to run depending on the source branch, like this:
condition: eq(variables['Build.SourceBranch'], 'refs/heads/a-branch-name')
and in each stage reference different variables. But this would introduce code duplication in each stage - when adding or modifying a step I would have to remember to edit 3 stages - not desirable.
Option 2: Create 3 separate YAML files in my repository, each one of them with specified trigger branch and referencing the same variable names, then create 3 different pipeline on Azure DevOps, each one of them with different variable values. But this would also introduce code duplication.
Option 3: Create 1 build-and-deploy.yaml file as a template with the steps defined in it and then create another 3 YAML files referring to that template, each with different trigger branch and with different variable values in each Azure Pipeline, like this:
trigger:
branches:
include:
- a-branch-name
steps:
- template: build-and-deploy.yaml
parameters:
parameterName1: $(parameterValue1)
parameterName2: $(parameterValue2)
This seems to be the best option but I haven't seen it used anywhere in the examples so maybe I'm just unaware of downsides of it, if there are any.
Here's how to do it with a shared pipeline config that gets included into env-specific pipelines.
To support 2 environments (dev and prod) you'd need:
1 shared pipeline yaml
2 env-specific yamls, one for each env
2 pipelines created in Azure DevOps, one for each env; each pipeline referencing corresponding yaml
pipeline-shared.yml:
variables:
ARTIFACT_NAME: ApiBuild
NPM_CACHE_FOLDER: $(Pipeline.Workspace)/.npm
stages:
- stage: Build
displayName: Build
pool:
vmImage: 'ubuntu-latest'
demands: npm
jobs:
...
- stage: Release
displayName: Release
dependsOn: Build
pool:
vmImage: 'ubuntu-latest'
jobs:
...
pipeline-dev.yml:
# Trigger builds on commits to branches
trigger:
- dev
# Do not trigger builds on PRs
pr: none
extends:
template: pipeline-shared.yml
pipeline-prod.yml
trigger:
- master
pr: none
extends:
template: pipeline-shared.yml
According to your description, if you want different stages to share the same repo resource, but their trigger branch and variable values are different.
Regarding trigger branch, you can use expression {{if ......}} to determine the trigger branch condition.
Regarding variable values, you can define templates and variable groups to specify them through parameters.
Here is an example, you can refer to it：
First go to Library under Pipelines, click on the Variable group to add a variable group. You can add multiple variables to this variable group.
Repo structure:
azure-pipelines.yml：
sample：
stages:
- template: stage/test.yml
parameters:
${{if contains(variables['Build.SourceBranch'], 'master')}}:
variableGroup: devGroup
stageName: Dev
test: a
${{if contains(variables['Build.SourceBranch'], 'test')}}:
stageName: test
test: b
stage/test. yml：
parameters:
- name: stageName
displayName: Test
type: string
default: test
- name: test
displayName: Test
type: string
default: test
- name: variableGroup
displayName: Test
type: string
default: test
stages:
- stage: Test_${{ parameters.stageName }}
variables:
- group: ${{parameters.variableGroup}}
jobs:
- job: Test1
pool:
vmImage: vs2017-win2016
steps:
- script: echo "Hello Test1"
- script: echo ${{ parameters.test }}
- script: echo $(dev1)
Of course, if you want to use a single variable, you can define the variable directly in yaml without adding a variable group.
I'm creating a docker build and push task in an Azure Devops Build pipeline YAML file, and I'd like to tag the image with the combination of two variables, the project name and the build number, so the tag would be service_22 (service is the project, 22 is the build number).
How do I join two variables together, e.g. $(variable)_$(variable2)
- task: Docker#2
inputs:
containerRegistry: 'Azure'
repository: 'jdmcontainers'
command: 'buildAndPush'
Dockerfile: 'Dockerfile'
tags: |
$(Build.BuildId)
$(imageName)
That's the current file, the tags are added as individual separate tags.
For those who come here and are actually looking for a way to combine two strings/variables in the job variable declaration, you can use the format() expression (docs link):
- job: Build
variables:
job_variable: $[format('Hey-{0}', variables['Build.Reason'])]
Hope that helps someone, at leased I searched for this for hours and found nothing. ^^
Try with below format:
steps:
- task: Docker#2
displayName: buildAndPush
inputs:
xxxx
tags: '$(System.TeamProject)_$(Build.Buildid)'
$(System.TeamProject) is one environment variable which can get the current project name.
Try this it works in Azure pipeline yaml
parameters:
- name: String1
displayName: StringName
type: string
values: 'Hello '
variables:
- name: String2
value: 'World'
- name: StringConcat
value: ${{parameters.String1}}${{variables.String2}}
Normal (non-template) jobs in Azure DevOps yaml support inter-job variable passing as follows:
jobs:
- job: A
steps:
- script: "echo ##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- job: B
condition: and(succeeded(), ne(dependencies.A.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
steps:
- script: echo hello from B
How do I do something similar in the following, given that templates don't support the dependsOn syntax? I need to get an output from the first template and pass it as 'environmentSlice' to the second template.
- stage: Deploy
displayName: Deploy stage
jobs:
- template: build-templates/get-environment-slice.yml#templates
parameters:
configFileLocation: 'config/config.json'
- template: build-templates/node-app-deploy.yml#templates
parameters:
# Build agent VM image name
vmImageName: $(Common.BuildVmImage)
environmentPrefix: 'Dev'
environmentSlice: '-$(dependencies.GetEnvironmentSlice.outputs['getEnvironmentSlice.environmentSlice'])'
The reason I want the separation between the two templates is the second one is a deployment template and I would like input from the first template in naming the environment in the second template. I.e. initial part of node-app-deploy.yml (2nd template) is:
jobs:
- deployment: Deploy
displayName: Deploy
# Because we use the environmentSlice to name the environment, we have to have it passed in rather than
# extracting it from the config file in steps below
environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}
Update:
The accepted solution does allow you to pass variables between separate templates, but won't work for my particular use case. I wanted to be able to name the 'environment' section of the 2nd template dynamically, i.e. environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}, but this can only be named statically since templates are compiled on pipeline startup.
The downside of the solution is that it introduces a hidden coupling between the templates. I would have preferred the calling pipeline to orchestrate the parameter passing between templates.
You can apply the depend on and dependency variable into templates.
See below sample:
To make sample more clear, here has 2 template files, one is azure-pipelines-1.yml, and another is azure-pipeline-1-copy.yml.
In azure-pipelines-1.yml, specify the environment value as output variable:
parameters:
environment: ''
jobs:
- job: preDeploy
variables:
EnvironmentName: preDeploy-${{ parameters.environment }}
steps:
- checkout: none
- pwsh: |
echo "##vso[task.setvariable variable=EnvironmentName;isOutput=true]$($env:ENVIRONMENTNAME)"
name: outputVars
And then, in azure-pipeline-1-copy.yml use dependency to get this output variable:
jobs:
- job: deployment
dependsOn: preDeploy
variables:
EnvironmentNameCopy: $[dependencies.preDeploy.outputs['outputVars.EnvironmentName']]
steps:
- checkout: none
- pwsh: |
Write-Host "$(EnvironmentNameCopy)"
name: outputVars
At last, in YAML pipeline, just need to pass the environment value
stages:
- stage: deployQA
jobs:
- template: azure-pipelines-1.yml
parameters:
environment: FromTemplate1
- template: azure-pipeline-1-copy.yml
Now, you can see the value get successfully in the second template job:
It is possible to avoid the dependency in the called template. However, as the OP says, the environment name cannot be created dynamically.
Here is an example of the "calling" template, which firstly calls another template (devops-variables.yml) that sets some environment variables that we wish to consume in a later template (devops-callee.yml):
stages:
- stage: 'Caller_Stage'
displayName: 'Caller Stage'
jobs:
- template: 'devops-variables.yml'
parameters:
InitialEnvironment: "Development"
- template: 'devops-callee.yml'
parameters:
SomeParameter: $[dependencies.Variables_Job.outputs['Variables_Job.Variables.SomeParameter']]
In the devops-variables.yml file, I have this:
"##vso[task.setvariable variable=SomeParameter;isOutput=true;]Wibble"
Then, in the "devops-callee.yml", I just consume it something like this:
parameters:
- name: SomeParameter
default: ''
jobs:
- deployment: 'Called_Job'
condition: succeeded()
displayName: 'Called Job'
environment: "Development"
pool:
vmImage: 'windows-2019'
dependsOn:
- Variables_Job
variables:
SomeParameter: ${{parameters.SomeParameter}}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: AzureCLI#2
condition: succeeded()
displayName: 'An Echo Task'
inputs:
azureSubscription: "$(TheServiceConnection)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
echo "Before"
echo "$(SomeParameter)"
echo "After"
Output:
2021-04-10T09:22:29.6188535Z Before
2021-04-10T09:22:29.6196620Z Wibble
2021-04-10T09:22:29.6197124Z After
This way, the callee doesn't reference the caller. Unfortunately, setting the environment in the callee thus:
environment: "$(SomeParameter)"
doesn't work - you'll just get an environment with the literal characters '$(SomeParameter)'.