Object as pipeline variable - azure-devops

I have the following pipeline variables:
---
variables:
models:
- name: model1
image_name: image1
- name: model2
image_name: image2
However, this is not allowed: A sequence was not expected. It seems that pipeline variables can only be single line strings. Is there a clever way to work around this? I've read about convertToJson, but it's not desirable/readable to write the models variable in a single line json string.
For context, I'm passing this variable as a parameter to a template, where I will be looping over parameters.models and run a stage for each model, like this:
- ${{each model in parameters.models}}:
- stage: {{ model.name }}
Apparently there is a very old unresolved issue that could provide a solution... What would be the best workaround?

DevOps YAML pipeline don't support such YAML structure:
variables:
models:
- name: model1
image_name: image1
- name: model2
image_name: image2
The variables in DevOps pipeline concept is only string.
And this place:
- ${{each model in parameters.models}}:
- stage: {{ model.name }}
Here are two usages in DevOps YAML tech, were named conditional insertion and template expression.
Both of them need valid structure in DevOps YAML. Variables in DevOps YAML concept doesn't support such structure, so these usages are not possible.
A possible way is make your YAML pipeline like below:
trigger:
- none
pool:
vmImage: ubuntu-latest
parameters:
- name: models
type: object
default:
- name: model1
image_name: image1
- name: model2
image_name: image2
stages:
- ${{each model in parameters.models}}:
- ${{ each modelcontent in model }}:
- ${{ if eq(modelcontent.Key, 'name') }}:
- stage: '${{ modelcontent.Value }}'
jobs:
- job:
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
Results:
Parameters can pass YAML object, and then use them in compile time usages. that's why I use parameters.
If you want handle the parameters automatically, the only way it to use another script/code/app to parse the YAML content getting from the repository and then change it. And after you finish the change, follow this to push back the changed YAML:
Git push back changes to repo

John Folberth almost does what you want, except he isn't passing it as a parameter to the yaml but to underlying templates:
https://blog.johnfolberth.com/advanced-azure-devops-yaml-objects/
The ConvertToJson is the other alternative, André van der Goes explains it in this blog post:
https://www.automagical.eu/posts/passing-complex-variables-devops-yaml/

Related

How to set Azure DevOps yaml variables conditionally based on parameter value

I am trying to set variables based on a parameter value in a yaml pipeline. It seems that I've read many other posts which show examples like the one below that the authors have said worked, but I cannot get past issues when trying to do something like below.
I've tried many variations on this example as well, too many to list here. Sometimes it will show 'values' as a duplicate key. In other cases I've been able to try and start a run and get the prompt with environment selection, but then opening the stage dialog throws a parse error.
Is there some sort of difference between variable declaration at the top of the file vs in a stage or job? That seems to be the difference that I notice when reading through other examples.
Ultimately what I'm trying to do is set the ServiceConnection variable value based on the value of the environment parameter.
parameters:
- name: environment
displayName: Environment
type: string
values:
- DEV
- TEST
pr: none
trigger: none
pool: PrivateAgentPool
variables:
- name: 'isMain'
value: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]
- name: 'buildConfiguration'
value: 'Release'
- name: 'environment'
value: ${{ parameters.environment }}
- name: 'ServiceConnection'
${{ if eq(variables['environment'], 'DEV') }}:
value: 'svcConnectionDev'
${{ if eq(variables['environment'], 'TEST') }}:
value: 'svcConnectionTest'
Looks like your solution is almost correct. Consider the below example.
parameters:
- name: region
type: string
default: westeurope
values:
- westeurope
- northeurope
variables:
${{ if eq(parameters['region'], 'westeurope') }}:
ServiceConnection: "svcConnectionDev"
${{ else }}:
enter code here
if you want to used this ServiceConnectionvar across you can do it just by calling $ServiceConnection
you could use bash with conditions:
steps:
- bash: |
echo "##vso[task.setvariable variable=ServiceConnection]svcConnectionDev"
condition: eq('${{parameters.environment}}', 'DEV')

"Configuring the trigger failed, edit and save the pipeline again" with no noticeable error and no further details

I have run in to an odd problem after converting a bunch of my YAML pipelines to use templates for holding job logic as well as for defining my pipeline variables. The pipelines run perfectly fine, however I get a "Some recent issues detected related to pipeline trigger." warning at the top of the pipeline summary page and viewing details only states: "Configuring the trigger failed, edit and save the pipeline again."
The odd part here is that the pipeline works completely fine, including triggers. Nothing is broken and no further details are given about the supposed issue. I currently have YAML triggers overridden for the pipeline, but I did also define the same trigger in the YAML to see if that would help (it did not).
I'm looking for any ideas on what might be causing this or how I might be able to further troubleshoot it given the complete lack of detail that the error/warning provides. It's causing a lot of confusion among developers who think there might be a problem with their builds as a result of the warning.
Here is the main pipeline. the build repository is a shared repository for holding code that is used across multiple repos in the build system. dev.yaml contains dev environment specific variable values. Shared holds conditionally set variables based on the branch the pipeline is running on.
name: ProductName_$(BranchNameLower)_dev_$(MajorVersion)_$(MinorVersion)_$(BuildVersion)_$(Build.BuildId)
resources:
repositories:
- repository: self
- repository: build
type: git
name: Build
ref: master
# This trigger isn't used yet, but we want it defined for later.
trigger:
batch: true
branches:
include:
- 'dev'
variables:
- template: YAML/variables/shared.yaml#build
- template: YAML/variables/dev.yaml#build
jobs:
- template: ProductNameDevJob.yaml
parameters:
pipelinePool: ${{ variables.PipelinePool }}
validRef: ${{ variables.ValidRef }}
Then this is the start of the actual job yaml. It provides a reusable definition of the job that can be used in more than one over-arching pipeline:
parameters:
- name: dependsOn
type: object
default: {}
- name: pipelinePool
default: ''
- name: validRef
default: ''
- name: noCI
type: boolean
default: false
- name: updateBeforeRun
type: boolean
default: false
jobs:
- job: Build_ProductName
displayName: 'Build ProductName'
pool:
name: ${{ parameters.pipelinePool }}
demands:
- msbuild
- visualstudio
dependsOn:
- ${{ each dependsOnThis in parameters.dependsOn }}:
- ${{ dependsOnThis }}
condition: and(succeeded(), eq(variables['Build.SourceBranch'], variables['ValidRef']))
steps:
**step logic here
Finally, we have the variable YAML which conditionally sets pipeline variables based on what we are building:
variables:
- ${{ if or(eq(variables['Build.SourceBranch'], 'refs/heads/dev'), eq(variables['Build.SourceBranch'], 'refs/heads/users/ahenderson/azure_devops_build')) }}:
- name: BranchName
value: Dev
** Continue with rest of pipeline variables and settings of each value for each different context.
You can check my post here : Azure DevOps pipeline trigger issue message not going away
As I can see in your YAML file, you are using this branch : 'refs/heads/users/ahenderson/azure_devops_build'.
I think some YAML files you are refering are missing from the branch defined as default in your build there :
Switch to your branch
I think I may have figured out the problem. It appears that this is related to the use of conditionals in the variable setup. While the variables will be set in any valid trigger configuration, it appears that the proper values are not used during validation and that may have been causing the problem. Switching my conditional variables to first set a default value and then replace the value conditionally seems to have fixed the problem.
It would be nice if Microsoft would give a more useful error message here, something to the extent of the values not being found for a given variable, but adding defaults does seem to have fixed the problem.
In our case it was because the path to the YAML file started with a slash: /builds/build.yaml
Removing the slash fixed the error: builds/build.yaml
In my case I had replaced the trigger in the yaml-file. The pipeline then does not now where to start.
# ASP.NET
# Build and test ASP.NET projects.
# Add steps that publish symbols, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/apps/aspnet/build-aspnet-4
trigger:
- "main" # <-- That was name wrong
For me it was this causing the problem...
The following pipeline.yaml causes the error “Configuring the trigger failed, edit and save the pipeline again”. It has to do with the environment name name: FeatureVMs.${{ variables.resourceName }}, if i replace ${{ variables.resourceName }} with something else e.g. FeatureVMs.develop then the error does not occur. The strange thing is, if i once save the pipeline with all triggers I want and a valid environment FeatureVMs.develop then it saves the triggers, if i then change it to what i actually want a dynamic environment resource selection FeatureVMs.${{ variables.resourceName }} then the error occurs but Azure Dev Ops the pipeline works as i expect it. So the workaround is to save it once without a variable and the triggers you want and then with the variable and live with the error on top of the pipeline
This causes the error
trigger: none
variables:
- name: resourceName
value: $(Build.SourceBranchName)
- name: sourcePipeline
value: vetsxl-ci
resources:
pipelines:
- pipeline: vetsxl-ci
source: vetsxl-ci
trigger:
branches:
include:
- develop
- feature/F*
- release/*
- review/*
- demo/*
- hotfix/H*
- tests/*
- test/*
stages:
- stage: Deploy
displayName: 'Deploy'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
jobs:
- deployment: DeployVM
displayName: 'Deploy to develop VM'
environment:
name: FeatureVMs.${{ variables.resourceName }}
strategy:
rolling:
deploy:
steps:
- template: deploy.yml
parameters:
sourcePipeline: ${{ variables.sourcePipeline }}
This works without any errors.
trigger: none
variables:
- name: resourceName
value: $(Build.SourceBranchName)
- name: sourcePipeline
value: vetsxl-ci
resources:
pipelines:
- pipeline: vetsxl-ci
source: vetsxl-ci
trigger:
branches:
include:
- develop
- feature/F*
- release/*
- review/*
- demo/*
- hotfix/H*
- tests/*
- test/*
stages:
- stage: Deploy
displayName: 'Deploy'
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
jobs:
- deployment: DeployVM
displayName: 'Deploy to develop VM'
environment:
name: FeatureVMs.develop
strategy:
rolling:
deploy:
steps:
- template: deploy.yml
parameters:
sourcePipeline: ${{ variables.sourcePipeline }}

Azure DevOps Pipeline variables in templates

I am working with Azure pipeline templates. I would like the developer that kicks off a pipeline to either set a variable of a specific branch OR leave as the $(Build.SourceBranch)
The reason is to pull down artifacts from different repositories/branches to combine.
So on my yml I added parameters (only showing 1 for simplicity)
parameters:
- name : source_branch
displayName: Which Branch (e.g. refs/head/foo)
type: string
default: $(Build.SourceBranch)
Then I call a template
- template: download_artifact.yml
parameters:
artifacts:
- project: 'XXX'
pipeline: 291
artifact: 'artifcat'
branch: ${{ parameters.source_branch }}
I use a template as there are approx 30 different artifacts to combine.
Within the template it downloads extracts and manipulates but I will simplify to only download.
parameters:
artifacts: []
steps:
- ${{ each step in parameters.artifacts }}:
- task: DownloadPipelineArtifact#2
displayName: '${{ step.artifact }}'
inputs:
source: 'specific'
project: ${{step.project}}
pipeline: ${{step.pipeline}}
runVersion: 'latestFromBranch'
runBranch: ${{step.branch}}
artifact: ${{step.artifact}}
path: '$(Pipeline.Workspace)\${{ step.artifact }}'
So the end result is that the variable does not get resolved within the template. I think this is due to templates being expanded at queue time. Does anyone have any workarounds for this scenario?
Your reference to the source_branch parameter when calling the template from the pipeline needs to be a template expression:
- template: download_artifact.yml
parameters:
artifacts:
- project: 'XXX'
pipeline: 291
artifact: 'artifcat'
branch: ${{ parameters.source_branch }}
Also, if the $(Build.SourceBranch) doesn't work in your parameter declaration, you can try:
parameters:
- name : source_branch
displayName: Which Branch (e.g. refs/head/foo)
type: string
default: ${{ variables['Build.SourceBranch'] }}
The parameters in the template should be expanded at pipeline compile time. And according to my tests and tries, we seem have no any available workaround for your scenario.

How can I invoke a YAML pipeline that has both variables and runtime parameters?

I have a scenario where I need to have both:
runtime parameters, so that the pipeline can be triggered manually from the UI, where users triggering it can choose from a predefined set of options (defined in YAML)
variables, so that the pipeline can be invoked via REST APIs
Regarding runtime parameters, I was able to create the following sample pipeline:
parameters:
- name: image
displayName: Pool Image
type: string
default: ubuntu-latest
values:
- windows-latest
- ubuntu-latest
trigger: none
stages:
- stage: A
jobs:
- job: A
steps:
- pwsh: |
echo "This should be triggering against image: $MY_IMAGE_NAME"
env:
MY_IMAGE_NAME: ${{ parameters.image }}
When I run it, I can see the dropdown list where I can choose the image name and it is reflected in the output message of the PowerShell script.
Regarding variables, I have defined one called "image" here (notice the value is empty):
The idea now is to invoke the pipeline from REST APIs and have the image name replaced by the value coming from the variable:
{
"definition": {
"id": 1
},
"sourceBranch": "master",
"parameters": "{\"image\": \"windows-latest\" }"
}
In order to make the step print the value I'm passing here, I need to correct the environment variable in some way. I thought it would be sufficient to write something like:
env:
MY_IMAGE_NAME: ${{ coalesce(variables.image, parameters.image) }}
That's because I want to give the priority to the variables, then to parameters, so that in case none is specified, I always have a default value the pipeline can use.
However, this approach doesn't work, probably because we're dealing with different expansion times for variables, but I don't really know what I should be writing instead (if there is a viable option, of course).
What I also tried is:
env:
MY_IMAGE_NAME: ${{ coalesce($(image), parameters.image) }}
MY_IMAGE_NAME: ${{ coalesce('$(image)', parameters.image) }}
MY_IMAGE_NAME: $[ coalesce(variables.image, parameters.image) ]
MY_IMAGE_NAME: $[ coalesce($(image), parameters.image) ]
None of those are working, so I suspect this may not be feasible at all.
There is a workaround that I'm currently thinking of, which is to create two different pipelines so that those can be invoked independently, but while this is quite easy for me to accomplish, given I'm using a lot of templates, I don't find it the right way to proceed, so I'm open to any suggestion.
I tested and found you might need to define a variable and assign the parameter's value to it (eg. Mimage: ${{parameters.image}}). And define another variable(eg. Vimage) and assign $[coalesce(variables.image, variables.Vimage)] to it. Then refer to $(Vimage) in the env field of powershell task. Please check out below yaml.
parameters:
- name: image
displayName: Pool Image
type: string
default: ubuntu-latest
values:
- windows-latest
- ubuntu-latest
trigger: none
stages:
- stage: A
jobs:
- job: A
variables:
Mimage: ${{parameters.image}}
Vimage: $[coalesce(variables.image, variables.Mimage)]
steps:
- pwsh: |
echo "This should be triggering against image: $env:MY_IMAGE_NAME"
env:
MY_IMAGE_NAME: $(Vimage)
Env field of powershell task is usually for mapping secret variables. You can directly refer to $(Vimage) in the powershell script: echo "This should be triggering against image: $(Vimage).
Note: To queue a build via REST API with provided parameters, you need to check Let users override this value when running this pipeline to make the varilabe to be settable at queue time.
Update:
You can try passing the variables to the parameters of the template to make the parameters for template dynamic. Please check below simple yaml.
jobs:
- template: template.yaml
parameters:
MTimage: ${{parameters.image}}
VTimage: $(Vimage)
template.yaml:
parameters:
MTimage:
VTimage:
jobs:
- job: buildjob
steps:
- powershell: |
echo "${{parameters.VTimage}}"
echo "${{parameters.MTimage}}"

Azure Devops - passing variables between job templates

Normal (non-template) jobs in Azure DevOps yaml support inter-job variable passing as follows:
jobs:
- job: A
steps:
- script: "echo ##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- job: B
condition: and(succeeded(), ne(dependencies.A.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
steps:
- script: echo hello from B
How do I do something similar in the following, given that templates don't support the dependsOn syntax? I need to get an output from the first template and pass it as 'environmentSlice' to the second template.
- stage: Deploy
displayName: Deploy stage
jobs:
- template: build-templates/get-environment-slice.yml#templates
parameters:
configFileLocation: 'config/config.json'
- template: build-templates/node-app-deploy.yml#templates
parameters:
# Build agent VM image name
vmImageName: $(Common.BuildVmImage)
environmentPrefix: 'Dev'
environmentSlice: '-$(dependencies.GetEnvironmentSlice.outputs['getEnvironmentSlice.environmentSlice'])'
The reason I want the separation between the two templates is the second one is a deployment template and I would like input from the first template in naming the environment in the second template. I.e. initial part of node-app-deploy.yml (2nd template) is:
jobs:
- deployment: Deploy
displayName: Deploy
# Because we use the environmentSlice to name the environment, we have to have it passed in rather than
# extracting it from the config file in steps below
environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}
Update:
The accepted solution does allow you to pass variables between separate templates, but won't work for my particular use case. I wanted to be able to name the 'environment' section of the 2nd template dynamically, i.e. environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}, but this can only be named statically since templates are compiled on pipeline startup.
The downside of the solution is that it introduces a hidden coupling between the templates. I would have preferred the calling pipeline to orchestrate the parameter passing between templates.
You can apply the depend on and dependency variable into templates.
See below sample:
To make sample more clear, here has 2 template files, one is azure-pipelines-1.yml, and another is azure-pipeline-1-copy.yml.
In azure-pipelines-1.yml, specify the environment value as output variable:
parameters:
  environment: ''
jobs:
- job: preDeploy
  variables:
    EnvironmentName: preDeploy-${{ parameters.environment }}
  steps:
  - checkout: none
  - pwsh: |
      echo "##vso[task.setvariable variable=EnvironmentName;isOutput=true]$($env:ENVIRONMENTNAME)"
    name: outputVars
And then, in azure-pipeline-1-copy.yml use dependency to get this output variable:
jobs:
- job: deployment
  dependsOn: preDeploy
  variables:
    EnvironmentNameCopy: $[dependencies.preDeploy.outputs['outputVars.EnvironmentName']]
  steps:
  - checkout: none
  - pwsh: |
      Write-Host "$(EnvironmentNameCopy)"
    name: outputVars
At last, in YAML pipeline, just need to pass the environment value
stages:
  - stage: deployQA
    jobs:
    - template: azure-pipelines-1.yml
      parameters:
        environment: FromTemplate1
    - template: azure-pipeline-1-copy.yml
Now, you can see the value get successfully in the second template job:
It is possible to avoid the dependency in the called template. However, as the OP says, the environment name cannot be created dynamically.
Here is an example of the "calling" template, which firstly calls another template (devops-variables.yml) that sets some environment variables that we wish to consume in a later template (devops-callee.yml):
stages:
- stage: 'Caller_Stage'
displayName: 'Caller Stage'
jobs:
- template: 'devops-variables.yml'
parameters:
InitialEnvironment: "Development"
- template: 'devops-callee.yml'
parameters:
SomeParameter: $[dependencies.Variables_Job.outputs['Variables_Job.Variables.SomeParameter']]
In the devops-variables.yml file, I have this:
"##vso[task.setvariable variable=SomeParameter;isOutput=true;]Wibble"
Then, in the "devops-callee.yml", I just consume it something like this:
parameters:
- name: SomeParameter
default: ''
jobs:
- deployment: 'Called_Job'
condition: succeeded()
displayName: 'Called Job'
environment: "Development"
pool:
vmImage: 'windows-2019'
dependsOn:
- Variables_Job
variables:
SomeParameter: ${{parameters.SomeParameter}}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: AzureCLI#2
condition: succeeded()
displayName: 'An Echo Task'
inputs:
azureSubscription: "$(TheServiceConnection)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
echo "Before"
echo "$(SomeParameter)"
echo "After"
Output:
2021-04-10T09:22:29.6188535Z Before
2021-04-10T09:22:29.6196620Z Wibble
2021-04-10T09:22:29.6197124Z After
This way, the callee doesn't reference the caller. Unfortunately, setting the environment in the callee thus:
environment: "$(SomeParameter)"
doesn't work - you'll just get an environment with the literal characters '$(SomeParameter)'.