Intermediate YAML template, resources repository's ref and parameters "Internal error reading the template. Expected a scalar,a sequence,or a mapping" - azure-devops

Azure Pipelines docs state that we can use template expressions in resources.repositories repository.ref argument.
It already works in some parent template, but using $(Build.SourceBranchName) instead of template expressions. In some child template, the template expression produces the error "Internal error reading the template. Expected a scalar, a sequence, or a mapping". BTW, if I use template expressions in parent template, I get there the same error.
Alternatively, if I use $(Build.SourceBranchName), child template throws error "Could not get the latest source version for repository xxxx-dotnet-solution hosted on Azure Repos using ref $(Build.SourceBranchName). (in this case, Azure Pipelines won't parse the variable).
I can't figure out how to fix this problem.
Here's the affected child template:
parameters:
- name: buildConfiguration
type: string
values: [Debug, Release]
- name: api
type: string
- name: solutionLocation
type: string
resources:
repositories:
- repository: xxxx-apis-parent-templates
name: xxxx-dotnet-solution
type: git
ref: ${{ variables['Build.SourceBranchName'] }}
extends:
template: .az-devops-cicd/templates/aks.yaml#xxxx-apis-parent-templates
parameters:
solutionLocation: ${{ parameters.solutionLocation }}
projectRepository: xxxxxx-apis
projectBaseDir: ${{ parameters.solutionLocation }}/${{ parameters.api }}/
projectName: XXXX.${{ parameters.solutionLocation }}.Apis.${{ parameters.api }}
buildConfiguration: ${{ parameters.buildConfiguration }}

Related

Alternate solution to using variable in resources/repositories/repository/ref

I am trying to find an alternate solution to using variable in resources/repositories/repository/ref because using a variable is technically not allowed.
resources:
resources:
repositories:
- repository: devops
name: MyProjects/devops
type: git
ref: master
The workaround of doing a git clone of the external repository will not work for me because my dependency on that repository is for referencing the templates.
Example:
- template: Build/Templates/downloadFiles.yaml#devops
Does anyone have a solution? Thank you for reading my post!
Refer to this doc: Template Expressions in Repository Resource Definition
Now, you can use template expressions to choose the branch of a repository resource.
Azure DevOps has supported to use variable in Repo Resource to set the ref. We can use template expression: ${{ variables.var }} to define the ref.
Here is an example:
variables:
branchname: main
resources:
repositories:
- repository: devops
name: 123/Repo90
type: git
ref: ${{ variables['branchname'] }}
pool:
vmImage: ubuntu-latest
steps:
- template: test.yml#devops

For-Each an Object in Azure Devops pipeline?

I starting to write an appication in microservices and want to have a build step to push the image from my pipeline. For this at the moment I have 3 services to push:
- stage: build_and_publish_containers
displayName: 'Docker (:Dev Push)'
jobs:
- template: "docker/publish.yaml"
parameters:
appName: Authorization_Service
projectPath: "Services/AuthorizationService"
imageName: authorization-service
imageTag: ${{variables.imageTag}}
- template: "docker/publish.yaml"
parameters:
appName: Registration_Service
projectPath: "Services/RegistrationService"
imageName: registration-service
imageTag: ${{variables.imageTag}}
- template: "docker/publish.yaml"
parameters:
appName: Tennant_Service
projectPath: "Services/TennantService"
imageName: tennant-service
imageTag: ${{variables.imageTag}}
Even with only this 3 services (and I want to have much more) I have a lot of duplicated code here I want to reduce.
I tried it with an array and an each-function but I have several information here (name / path / imagename) and that could grow.
Is there a better way?
If that would be a programming language I would have an array of a data model, is that something that is possible in azure devops?
Or maybe could each information saved in a json file (so 3 files at the moment and growing) and azure could get all files and informations out of this?
you could check the example below to define your complex object nested loops in Azure pipelines. By the way, you could also look into the github doc for more reference.
parameters:
- name: environmentObjects
type: object
default:
- environmentName: 'dev'
result: ['123']
- environmentName: 'uat'
result: ['223', '323']
pool:
vmimage: ubuntu-latest
steps:
- ${{ each environmentObject in parameters.environmentObjects }}:
- ${{ each result in environmentObject.result }}:
- script: echo ${{ result }}

Is there any possibilities to call AzDo templated with parameters?

I would like to run a few templates based on the initial value chosen from the parameter and as soon as the value is chosen then a template will be issued which will further ask for more parameters only required for that template.
Let's say in main azure-pipelines.yml if a user chooses dev then simply a template will be called. However, if a user chooses test then template create-stack-tst-template.yml will be issued but along with that, it should prompt the parameters needed for this template. Is it possible?
If not, is there any possibility to club all the parameters only needed for dev and the same for test. So that when the individual templates are called then clubbed parameter will be passed which is necessary for that template to run but not for others.
Is there any kind of segregation exists?
trigger:
- none
parameters:
- name: DeployToEnvType
displayName: |
Select the env type to be deployed
type: string
values:
- dev
- test
stages:
- ${{ if eq(parameters['DeployToEnvType'], 'dev' ) }}:
- template: templates/create-stack-dev-template.yml
- ${{ if ne(parameters['DeployToEnvType'], 'test' ) }}:
- template: templates/create-stack-tst-template.yml
parameters:
- name: ProjectName
type: string
- name: ImageSource
type: string
it should prompt the parameters needed for this template. Is it possible?
This is not possible. You need to provide all parameters and pass further only those which are needed by particular template.
trigger:
- none
parameters:
- name: DeployToEnvType
displayName: |
Select the env type to be deployed
type: string
values:
- dev
- test
- name: ImageSource
type: string
stages:
- ${{ if eq(parameters['DeployToEnvType'], 'dev' ) }}:
- template: templates/create-stack-dev-template.yml
parameters:
ProjectName: projectA
ImageSource: ${{ parameters.ImageSource }}
- ${{ if ne(parameters['DeployToEnvType'], 'test' ) }}:
- template: templates/create-stack-tst-template.yml
parameters:
ProjectName: projectA
ImageSource: ${{ parameters.ImageSource }}
If you need control at runtime you need to make corresponding runtime parameter and pass it down. If you want to have some values fixed you can just put them inline.

Passing parameters through nested templates (or declaring IF conditions on variables)

I would like to be able to pass a pipeline parameter all the way through my YAML pipeline without having to define a parameter in each and every YAML file.
Essentially I have a main YAML file which calls a stage YAML, that has multiple nested jobs YAML, which in turn calls nested steps YAML; essentially building up my pipeline as I should using templates: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
Here's a tree list sample folder;
E:.
├───01_stage (many files per folder)
├───02_jobs (many files per folder)
├───03_steps (many files per folder)
└───...main pipeline files
Ideally I want to run an IF condition on checking out a repository depending upon the pipeline being PROD or NON-PROD. I am fine with defining this as a parameter, but I am also open to it being defined as a variable. As far as I'm aware; you can't use IF condition on variables.
This is fine
- ${{ if eq(parameters.pester, true) }}: # or even as variables['pester']
- name: pester
value: yes
This is not fine
- ${{ if eq(variables.pester, true) }}: # or even as variables['pester']
- name: pester
value: yes
The condition I want this to run is nest far below many templates, and it would be absolutely painful to have to re-code everything to confirm to the parameters value being declared and passed down in each file.
This is where I want it:
steps:
- ${{ if eq(parameters['masterTagged'], 'true') }}: # here
- checkout: masterTagged
displayName: Repo Tagged
- ${{ if ne(parameters['masterTagged'], 'true') }}: # here
- checkout: self
displayName: Repo Self
- template: /.pipelines/03_steps/ssh_install.yml
- template: /.pipelines/03_steps/tf_install.yml
parameters:
terraformVersion: ${{ parameters['terraformVersion'] }}
- ...many more templates
Here is my main YAML pipeline file:
parameters:
- name: artifactory_base
type: boolean
default: true
# ...many more params
- name: pester
type: boolean
default: true
- name: planDeploy
type: boolean
default: true
- name: useBackupAgent
type: boolean
default: false
- name: masterTagged # key param
type: boolean
default: true
name: Team2
pr: none
resources:
repositories:
- repository: masterTagged
endpoint: nationwide-ccoe
name: my-github-org/my-github-repo
type: github
ref: refs/tags/v2.0.3
trigger: none
variables:
- template: /.pipelines/config/sub-asdfasdf.config.yml
- template: /.pipelines/config/namingstd.config.yml
- ${{ if eq(parameters.artifactory_base, true) }}:
- name: artifactory_base
value: yes
# ...many more conditions
- ${{ if eq(parameters.pester, true) }}:
- name: pester
value: yes
- ${{ if eq(parameters.planDeploy, true) }}:
- name: planDeploy
value: yes
stages:
- template: /.pipelines/01_stage/lz_deploy.yml
parameters:
${{ if eq(parameters.useBackupAgent, false) }}:
pool:
vmImage: Ubuntu 18.04
${{ if eq(parameters.useBackupAgent, true) }}:
pool:
name: backupAgents
terraformVersion: $(TERRAFORM_VERSION)
Is it possible to set this masterTagged parameter and for it to filter all the way down without having to declare it each time?
Also; is it even possible to use variables instead of parameters in this manner (I understand that parameters expand before variables):
- ${{ if eq(variables.pester, true) }}: # or even as variables['pester']
- name: pester
value: yes
...if it is, have I been doing it wrong all this time?
Note:
I do understand that you can use a standard task condition on the checkout task (shown below); however, having a 'switch' on two tasks ruins the folder path of the checked out repository. Even though we're only checking out on repository, it adds another folder level to the $SYSTEM_DEFAULTWORKINGDIRECTORY. Doing it this way would require more re-coding on the current structure of my YAML piplines.
- checkout: masterTagged
condition: eq(variables['masterTagged'], 'true')
displayName: Repo Tagged
- checkout: self
condition: ne(variables['masterTagged'], 'true')
displayName: Repo Self
If I could, but I know it's not possible (as seen by other peoples requests), I would enable a parameter or variable on the repository reference:
resources:
repositories:
- repository: masterTagged
endpoint: nationwide-ccoe
name: my-github-org/my-github-repo
type: github
ref: ${{ parameters.repoRef }} # here
Is it possible to set this masterTagged parameter and for it to filter all the way down without having to declare it each time?
No, because parameters are “scoped” to the file they are defined with. This is due to them being expanded when the pipeline is first compiled. (See > Pipeline run sequence)
You can use IF conditions on variables, however you can’t use template expressions (wrapped with {{}}) on variables inside templates as the variables do not exist/have not been populated at the point of template expansion.
One option is just using the conditions on the checkout tasks as you suggested, and dealing with the extra folder level to the default working directory. I had to do something similar a while back, our solution was to copy the contents of the repo folder up a level into the default working directory.
Your other option is to do the checkout in the top level pipeline file. This will allow you to template the checkout step/s using the parameter without having to pass it all the way through the files. This is the option I would suggest as you do not have to deal with the folder structure issues of the first option.
This would look something like this:
parameters:
- name: masterTagged
default: true
type: boolean
resources:
repositories:
- repository: masterTagged
endpoint: nationwide-ccoe
name: my-github-org/my-github-repo
type: github
ref: refs/tags/v2.0.3
steps:
- ${{ if eq(parameters.masterTagged, true) }}:
- checkout: masterTagged
- ${{ if eq(parameters.masterTagged, false) }}:
- checkout: self
- template: ./path/to/template.yml
I hope this answers your question.

How to use yaml template parameters in Azure DevOps Server 2019 on-prem?

According to the official documentation at https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops-2019 the Azure DevOps Server 2019 on-prem supports everything.
Yet, I cannot make it work with a simple yaml template with parameters.
Here is my yaml template (named prepare-sonar-qube.yml):
parameters:
- name: projectKey
type: string
- name: projectName
type: string
default: ${{ parameters.projectKey }}
- name: useDotCover
type: boolean
default: false
steps:
- template: install-java.yml
- task: SonarQubePrepare#4
displayName: 'Prepare SQ Analysis'
inputs:
SonarQube: 'SonarQube'
scannerMode: 'MSBuild'
projectKey: parameters.projectKey
projectName: parameters.projectName
${{ if parameters.useDotCover }}:
extraProperties: |
sonar.cs.dotcover.reportsPaths=$(Common.TestResultsDirectory)\coverage\*.CoverageResult.html
sonar.inclusions=**/*.cs
${{ if !parameters.useDotCover }}:
extraProperties: |
sonar.cs.opencover.reportsPaths=$(Common.TestResultsDirectory)\coverage\*.CoverageResult.xml
sonar.inclusions=**/*.cs
Here is the azure-pipelines.yml:
trigger:
- master
name: 1.0.$(Date:yy)$(DayOfYear)$(Rev:.r)
jobs:
- job: Build
pool:
demands: DotNetFramework
workspace:
clean: all
variables:
- template: variables.yml
steps:
- template: prepare-sonar-qube.yml
parameters:
projectKey: logs2db
...
Running the build I get the following lovely error message:
/prepare-sonar-qube.yml (Line: 2, Col: 1): A sequence was not expected
So what am I doing wrong? (Besides being a loyal TFS customer who got stuck with an outdated on-prem Azure DevOps Server 2019 that does not seem to go anywhere when compared against the ever evolving hosted Azure DevOps Services)
You do not did anything wrong. Need sorry to say, that is our document issue.
The syntax you are trying is our new richer YAML syntax feature, which haven't supported in Azure DevOps Server 2019 until now.
To let public know and avoid this before we make changes to document, I create a thread and announce this unsupport here.
At present, the on-prem server only support the older syntax, where defaults are declared as a mapping without type or value constraints:
parameters:
solution: '**/*.sln'
Or
parameters:
solution: ''
This looks like a simple indentation issue.
Here's a snip from one of my working templates.
parameters:
- name: sln
type: string
default: ''
- name: slnName
type: string
default: ''
- name: proj
type: string
default: ''
Try using space space - name: [name]