Pass array as inputs to Azure DevOps YAML task - azure-devops

I'm trying to configure a CI that will produce NuGet packages as artifacts on Azure DevOps (that will be later pushed to my NuGet server).
For this, I'm using Builds Pipelines on Azure DevOps, the YAML version.
I have 3 projects that should build packages. I'm using NuGetCommand#2 to accomplish this task :
- task: NuGetCommand#2
inputs:
command: pack
packagesToPack: $(Build.SourcesDirectory)/src/HTS_MessageEngine.Communication/HTS_MessageEngine.Communication.csproj
majorVersion: $(majorVersion)
minorVersion: $(minorVersion)
patchVersion: $(patchVersion)
versioningScheme: byPrereleaseNumber
However, I have to duplicate this block 3 times, for each project. Is there a way to specify an array of projects in the packagesToPack parameter? So far, the version is the same for each package, so I don't need three different blocks...
Note: these 3 projects are all 3 NetStandard and properties for package building are stored in csproj directly

you can use each function (if its available at all at this point in time):
# my-template.yml
parameters:
steps:
- ${{ each project in parameters.projects }}:
- task: PublishBuildArtifacts#1
displayName: Publish ${{ project }}
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ project }}.zip'
# ci.yml
steps:
- template: my-template.yml
parameters:
projects:
- test1
- test2
Github PR for this functionality: https://github.com/Microsoft/azure-pipelines-yaml/pull/2#issuecomment-452748467

For above code I got this exception running a build:
my-template.yml (Line: 1, Col: 12): Unexpected value ''
but this works for me:
# my-template.yml
parameters:
- name: projects
type: object
default: {}
steps:
- ${{ each project in parameters.projects }}:
- task: PublishBuildArtifacts#1
displayName: 'Publish ${{ project }}
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ project }}.zip'
and then:
# ci.yml
steps:
- template: my-template.yml
parameters:
projects:
- test1
- test2

Related

Matrix in Azure DevOps yaml pipeline: The pipeline is not valid

For our mobile app, I am trying to use matrix to set different pipeline values in Debug and Release:
jobs:
- job: Job_1
displayName: .Net MAUI Job
strategy:
maxParallel: 2
matrix:
Debug:
BuildConfiguration: Debug
ProvProfile: 'My_Testing_Profile.mobileprovision'
CertSecureFile: 'ios_development.p12'
CertPwd: $(IOSP12Password-testing)
Release:
BuildConfiguration: Release
ProvProfile: 'My_Distribution_Profile.mobileprovision'
CertSecureFile: 'ios_distribution.p12'
CertPwd: $(IOSP12Password-distribution)
...
- task: InstallAppleCertificate#2
displayName: Install Apple Certificate
inputs:
certSecureFile: $(CertSecureFile)
certPwd: $(CertPwd)
setUpPartitionIdACLForPrivateKey: false
deleteCert: false
deleteCustomKeychain: false
- task: InstallAppleProvisioningProfile#1
displayName: Install Testing Apple Provisioning Profile
inputs:
provisioningProfileLocation: 'secureFiles'
provProfileSecureFile: $(ProvProfile)
...
- task: DotNetCoreCLI#2
displayName: 'dotnet publish ($(BuildConfiguration))'
inputs:
command: 'publish'
publishWebProjects: false
projects: 'My_MobileApp.sln'
arguments: '-f:net6.0-ios -c:$(BuildConfiguration) -r ios-arm64 /p:ArchiveOnBuild=true /p:EnableAssemblyILStripping=false'
zipAfterPublish: false
modifyOutputPath: false
IOSP12Password-testing and IOSP12Password-distribution are variables set in the pipeline.
I am getting the following error:
There was a resource authorization issue: "The pipeline is not valid.
Job Job_1: Step InstallAppleCertificate input certSecureFile references secure file $(CertSecureFile) which could not be found. The secure file does not exist or has not been authorized for use.
Job Job_1: Step InstallAppleProvisioningProfile input provProfileSecureFile references secure file $(ProvProfile) which could not be found. The secure file does not exist or has not been authorized for use.
I suspect that CertPwd is also wrong.
I don't understand why it is not working, if there is no problem with BuildConfiguration at all.
Azure Devops Services now don't support using variable in secure file.
So if your profile and certificate files have already been added into the library-secure file, you need to directly write the name of the file into your yaml instead of variables.
If you do need the feature, you can directly report the feature requests. That will allow you to directly interact with the appropriate engineering team and make it more convenient for the engineering team to collect and categorize your suggestions.
updates:
You could try to use 'condition sytax' here to repeat the two tasks with different hard code value, if you need to choose the value in runtime, you could use parameters, it should be something like this:
parameters:
- name: CertSecureFile
values:
- ios_development.p12
- ios_distribution.p12
- name: ProvProfile
values:
- My_Testing_Profile.mobileprovision
- My_Distribution_Profile.mobileprovision
stages:
- stage: A
jobs:
- job: Job_1
displayName: .Net MAUI Job
strategy:
maxParallel: 2
matrix:
Debug:
BuildConfiguration: Debug
ProvProfile: 'My_Testing_Profile.mobileprovision'
CertSecureFile: 'ios_development.p12'
CertPwd: $(IOSP12Password-testing)
Release:
BuildConfiguration: Release
ProvProfile: 'My_Distribution_Profile.mobileprovision'
CertSecureFile: 'ios_distribution.p12'
CertPwd: $(IOSP12Password-distribution)
steps:
- task: InstallAppleCertificate#2
condition: eq('${{ parameters.CertSecureFile }}', 'ios_development.p12')
displayName: Install Apple Certificate
inputs:
certSecureFile: ios_development.p12
certPwd: $(CertPwd)
setUpPartitionIdACLForPrivateKey: false
deleteCert: false
deleteCustomKeychain: false
- task: InstallAppleProvisioningProfile#1
condition: eq('${{ parameters.ProvProfile }}', 'My_Testing_Profile.mobileprovision')
displayName: Install Testing Apple Provisioning Profile
inputs:
provisioningProfileLocation: 'secureFiles'
provProfileSecureFile: My_Testing_Profile.mobileprovision
- task: InstallAppleCertificate#2
condition: eq('${{ parameters.CertSecureFile }}', 'ios_distribution.p12')
displayName: Install Apple Certificate
inputs:
certSecureFile: ios_distribution.p12
certPwd: $(CertPwd)
setUpPartitionIdACLForPrivateKey: false
deleteCert: false
deleteCustomKeychain: false
- task: InstallAppleProvisioningProfile#1
condition: eq('${{ parameters.ProvProfile }}', 'My_Distribution_Profile.mobileprovision')
displayName: Install Testing Apple Provisioning Profile
inputs:
provisioningProfileLocation: 'secureFiles'
provProfileSecureFile: My_Distribution_Profile.mobileprovision
updates:
If you want to use template with parameters, you could put the repeated two jobs with two fixed value in two templates yaml in the same repo and branch.(here we name it for example, template1.yaml and template2.yaml)
Then you could try something like this:
steps:
- ${{ if and(eq(parameters.CertSecureFile, 'ios_development.p12'),eq(parameters.ProvProfile, 'My_Testing_Profile.mobileprovision')) }}:
- template: template1.yaml
- ${{ else }}:
- template: template2.yaml

How to fix Azure pipeline ( /templates/test-template.yaml (Line: 5, Col: 8): Unexpected value '') Error

I am trying to set up an gradle azure yaml pipeline that uses test_template.yaml, and when i run pipeline i get Unexpected value ''
My question is how can i improve my template or fix it ?
trigger: none
variables:
- group: azurecr #azure container registry
pool:
vmImage: ubuntu-latest
stages:
stage: test
jobs:
- template: templates/test_template.yaml
parameters:
app: :mortgage:mortgage-app:test
Build and test template
# build and test template
jobs:
- job:
steps:
- task:
inputs:
gradleWrapperFile: 'gradlew' #gradle wrapper file
gradleOptions: '-Xmx3072m' # max memory
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.11' #java version
jdkArchitectureOption: 'x64'
publishJUnitResults: true
testResultsFiles: '**/TEST-*.xml'
tasks: 'build'
- bash: |
echo "Build and run unit tests ${{ parameters.app }}".
echo "##vso[task.setvariable variable=${{ parameters.tag }};]$(./gradlew ${{ parameters.app }}} -q printVersion)" #build and run test
Your job requires to have name
job: string # Required as first property. ID of the job. Valid names may only contain alphanumeric characters and '_' and may not start
with a number.
https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job?view=azure-pipelines
For example
- job:MyJobName
EDIT 1
Also Azure DevOps will not know what task do you want to run in this place, you need to provide task name
- task:
inputs:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/build/gradle?view=azure-devops
EDIT 2
If you provide parameters to template you need to state in this template that it will accept this parameter
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#passing-parameters
So on the beginning of the template should be
parameters:
- name: app
type: string

Azure DevOps: Populating secure file references with job matrix variables

For context, I am trying to use an Azure build pipeline to build multiple flavors of an Android app. Each flavor has its own separate signing keystore, and all of those keystores are stored in my 'secure files' in the library.
However, when I try to dereference the $(Keystore) variable during the 'android signing' task, it doesn't seem to recognize that that is a variable that exists, and tries instead to locate a file called '$(Keystore)'
Am I doing something wrong here? This seems like it should work.
A sanitized example looks like this:
# Android
# Build your Android project with Gradle.
# Add steps that test, sign, and distribute the APK, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/android
trigger:
- feat/ci-setup
pool:
vmImage: 'macos-latest'
variables:
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/feat/') }}:
Branch_Type: 'feature'
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/hotfix/') }}:
Branch_Type: 'hotfix'
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/release/') }}:
Branch_Type: 'release'
${{ if eq(variables['Branch_Type'], 'release') }}:
Configuration: 'release'
ConfigurationCC: 'Release'
${{ if ne(variables['Branch_Type'], 'release') }}:
Configuration: 'debug'
ConfigurationCC: 'Debug'
jobs:
- job: Build
variables:
- group: android_keystores
strategy:
maxParallel: 2
matrix:
Flavor_1:
AppFlavor: '1'
AppFlavorCC: '1'
Keystore: 'flavor1.keystore'
KeyAlias: 'flavor1'
KeystorePass: '$(flavor1_storepass)'
KeyPass: '$(flavor1_keypass)'
Flavor_2:
AppFlavor: '2'
AppFlavorCC: '2'
Keystore: 'flavor2.keystore'
KeyAlias: 'flavor2'
KeystorePass: '$(flavor2_storepass)'
KeyPass: '$(flavor2_keypass)'
steps:
- task: Gradle#2
inputs:
workingDirectory: ''
gradleWrapperFile: 'gradlew'
gradleOptions: '-Xmx3072m'
publishJUnitResults: false
tasks: 'assemble$(AppFlavorCC)$(ConfigurationCC)'
- task: AndroidSigning#3
displayName: Signing .apk
inputs:
apkFiles: 'app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk'
apksign: true
apksignerKeystoreFile: '$(Keystore)'
apksignerKeystorePassword: '$(KeystorePass)'
apksignerKeystoreAlias: '$(KeyAlias)'
apksignerKeyPassword: '$(KeyPass)'
zipalign: true
- task: Bash#3
displayName: Move APK to Artifact Folder
continueOnError: true
inputs:
targetType: 'inline'
script: |
mv \
app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk \
$(Build.ArtifactStagingDirectory)/$(ArtifactName)/
- task: PublishBuildArtifacts#1
displayName: Publish Build Artifacts
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'Blueprint-Build'
publishLocation: 'Container'
But when the pipeline runs I am told this:
There was a resource authorization issue: "The pipeline is not valid. Job Build: Step AndroidSigning input keystoreFile references secure file $(Keystore) which could not be found. The secure file does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Azure DevOps: Populating secure file references with job matrix variables
This is a limitation from the task itself.
When we test it with Classic mode, we could find out that the value of the option Keystore file could not be entered manually, we could only select a certain file through the drop-down menu:
That the reason why it doesn't seem to recognize that that is a variable that exists, and tries instead to locate a file called '$(Keystore)'.
To resolve this issue, you could change the task version from 3 to 1, which supports manual input:
And as another solution, you could also use the command line to sign the *.apk:
Android apk signing: sign an unsigned apk using command line
You're missing the step to download the Secure File. Unlike variable groups, you need to explicitly download them to have access via the secure file name.
You'll want to add something similar to the example task below to your steps to pull the secure file. Then, you'll access your secure file via NAME_PARAMETER.secureFilePath:
- task: DownloadSecureFile#1
displayName: "Download Keyfile 1"
name: "YOUR_SECUREFILE_NAME"
inputs:
secureFile: keyfile1
- task: AndroidSigning#3
displayName: Signing .apk
inputs:
apkFiles: 'app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk'
apksign: true
apksignerKeystoreFile: '$(YOUR_SECUREFILE_NAME.secureFilePath)'
apksignerKeystorePassword: '$(KeystorePass)'
apksignerKeystoreAlias: '$(KeyAlias)'
apksignerKeyPassword: '$(KeyPass)'
zipalign: true

File from previous step cannot be found in Azure DevOps-Pipeline

In a pipeline I have two different steps. The first one generates some files, the second should take these files as an input.
the Yaml for that pipeline is the following:
name: myscript
stages:
- stage: Tes/t
displayName: owasp-test
jobs:
- job: owasp_test
displayName: run beasic checks for site
pool:
name: default
demands: Agent.OS -equals Windows_NT
steps:
- task: DotNetCoreCLI#2
inputs:
command: 'build'
projects: '**/*.sln'
- task: dependency-check-build-task#5
inputs:
projectName: 'DependencyCheck'
scanPath: '**/*.dll'
format: 'JUNIT'
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
the dependency-check-build-task returns an XML-File:
File upload succeed.
Upload 'P:\Azure-Pipelines-Agent\_work\2\TestResults\dependency-check\dependency-check-junit.xml' to file container: '#/11589616/dependency-check'
Associated artifact 53031 with build 21497
The following step (PublishTestResults) SHOULD take that file but returns
##[warning]No test result files matching **/*-junit.xml were found.
instead. I can see that file in the artifact after the pipeline is run.
This is because your report is written to Common.TestResultsDirectory which is c:\agent_work\1\TestResults (for Microsoft Hosted agents), and publish test task looks in System.DefaultWorkingDirectory which is c:\agent_work\1\s.
Please try:
- task: PublishTestResults#2
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/*-junit.xml'
searchFolder: '$(Common.TestResultsDirectory)'
I had the same trouble:
I fixed changing the Agent Specification

Using ARM Templates from external repository

I'm working with azure multistage pipelines, using deployment jobs with templates in a separate repo. I'm currently starting to use ARM templates in my deployment process and want to run ARM templates that are located in a different repository as well. This is where I get a little stuck, any help/advice appreciated.
To illustrate my setup:
Repo A -> Source code that has to be build and deployed to azure
Repo B -> Azure pipeline templates (only consists of yml files)
Repo C -> ARM templates
So what I want to accomplish: A uses B uses C.
REPO A: Documentation build and release yml
resources:
repositories:
- repository: templates
type: git
name: <ACCOUNT>/Azure.Pipelines.Templates
ref: refs/tags/2.2.40
stages:
- stage: Build
jobs:
- template: src/jobs/doc-build.yml#templates
- stage: DEV
jobs:
- template: src/deployment-jobs/doc.yml#templates
....
REPO B: Documentation deployment
parameters:
webAppName: ''
connectedServiceName: 'DEV'
jobs:
- deployment: doc_deploy
pool:
name: 'DOC'
environment: 'doc'
strategy:
runOnce:
deploy:
steps:
- template: ../deployment/arm-template.yml
parameters:
connectedServiceName: ${{ parameters.connectedServiceName }}
resourceGroupName: 'documentation'
templateFile: $(Build.SourcesDirectory)/Azure.ARM.Templates/src/web-app/documentation.jsonc
paramFile: $(Build.SourcesDirectory)/Azure.ARM.Templates/src/web-app/documentation-params.json
parameters: -name ${{ parameters.webAppName }}
...
REPO C: contains arm template + param file
The issue I'm facing is that I can't seem to be able to get to the files of repo c. I tried adding another repository entry on multiple levels but it does not seem to clone the dependent repo at all.
My current workaround/solution:
Use a powershell script to manually clone repo C and directly reference the file on disk.
Related github issue: https://github.com/microsoft/azure-pipelines-yaml/issues/103
I've also stumbled upon this issue, having to load arm templates from another repo into the current build. What I did was setting up a build on the arm-template-containing repo, producing a build artifact with following azure-pipelines.yml: (this would be your repo c)
trigger:
- master
steps:
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(System.DefaultWorkingDirectory)/templates'
ArtifactName: 'templates'
publishLocation: 'Container'
Afterwards I could add following step into the actual pipeline:
- task: DownloadPipelineArtifact#2
displayName: 'Get ARM Templates'
inputs:
buildType: 'specific'
project: <YOUR-PROJECT-ID>'
definition: '<ARM-BUILD-DEFINITION-ID>'
buildVersionToDownload: 'latest'
artifactName: 'scripts'
targetPath: '$(Pipeline.Workspace)/templates'
and I was able to access the files as follows:
- task: AzureResourceGroupDeployment#2
displayName: 'Create Queues $(ResourceGroup.Name) '
inputs:
azureSubscription: '<YOUR-SUBSCRIPTION>'
resourceGroupName: '$(ResourceGroup.Name)'
location: '$(ResourceGroup.Location)'
csmFile: '$(Pipeline.Workspace)/templates/servicebus.json'
For more information about the Download Pipeline Artifact task check out following link:
Download Pipeline Artifact task