Matrix in Azure DevOps yaml pipeline: The pipeline is not valid - azure-devops

For our mobile app, I am trying to use matrix to set different pipeline values in Debug and Release:
jobs:
- job: Job_1
displayName: .Net MAUI Job
strategy:
maxParallel: 2
matrix:
Debug:
BuildConfiguration: Debug
ProvProfile: 'My_Testing_Profile.mobileprovision'
CertSecureFile: 'ios_development.p12'
CertPwd: $(IOSP12Password-testing)
Release:
BuildConfiguration: Release
ProvProfile: 'My_Distribution_Profile.mobileprovision'
CertSecureFile: 'ios_distribution.p12'
CertPwd: $(IOSP12Password-distribution)
...
- task: InstallAppleCertificate#2
displayName: Install Apple Certificate
inputs:
certSecureFile: $(CertSecureFile)
certPwd: $(CertPwd)
setUpPartitionIdACLForPrivateKey: false
deleteCert: false
deleteCustomKeychain: false
- task: InstallAppleProvisioningProfile#1
displayName: Install Testing Apple Provisioning Profile
inputs:
provisioningProfileLocation: 'secureFiles'
provProfileSecureFile: $(ProvProfile)
...
- task: DotNetCoreCLI#2
displayName: 'dotnet publish ($(BuildConfiguration))'
inputs:
command: 'publish'
publishWebProjects: false
projects: 'My_MobileApp.sln'
arguments: '-f:net6.0-ios -c:$(BuildConfiguration) -r ios-arm64 /p:ArchiveOnBuild=true /p:EnableAssemblyILStripping=false'
zipAfterPublish: false
modifyOutputPath: false
IOSP12Password-testing and IOSP12Password-distribution are variables set in the pipeline.
I am getting the following error:
There was a resource authorization issue: "The pipeline is not valid.
Job Job_1: Step InstallAppleCertificate input certSecureFile references secure file $(CertSecureFile) which could not be found. The secure file does not exist or has not been authorized for use.
Job Job_1: Step InstallAppleProvisioningProfile input provProfileSecureFile references secure file $(ProvProfile) which could not be found. The secure file does not exist or has not been authorized for use.
I suspect that CertPwd is also wrong.
I don't understand why it is not working, if there is no problem with BuildConfiguration at all.

Azure Devops Services now don't support using variable in secure file.
So if your profile and certificate files have already been added into the library-secure file, you need to directly write the name of the file into your yaml instead of variables.
If you do need the feature, you can directly report the feature requests. That will allow you to directly interact with the appropriate engineering team and make it more convenient for the engineering team to collect and categorize your suggestions.
updates:
You could try to use 'condition sytax' here to repeat the two tasks with different hard code value, if you need to choose the value in runtime, you could use parameters, it should be something like this:
parameters:
- name: CertSecureFile
values:
- ios_development.p12
- ios_distribution.p12
- name: ProvProfile
values:
- My_Testing_Profile.mobileprovision
- My_Distribution_Profile.mobileprovision
stages:
- stage: A
jobs:
- job: Job_1
displayName: .Net MAUI Job
strategy:
maxParallel: 2
matrix:
Debug:
BuildConfiguration: Debug
ProvProfile: 'My_Testing_Profile.mobileprovision'
CertSecureFile: 'ios_development.p12'
CertPwd: $(IOSP12Password-testing)
Release:
BuildConfiguration: Release
ProvProfile: 'My_Distribution_Profile.mobileprovision'
CertSecureFile: 'ios_distribution.p12'
CertPwd: $(IOSP12Password-distribution)
steps:
- task: InstallAppleCertificate#2
condition: eq('${{ parameters.CertSecureFile }}', 'ios_development.p12')
displayName: Install Apple Certificate
inputs:
certSecureFile: ios_development.p12
certPwd: $(CertPwd)
setUpPartitionIdACLForPrivateKey: false
deleteCert: false
deleteCustomKeychain: false
- task: InstallAppleProvisioningProfile#1
condition: eq('${{ parameters.ProvProfile }}', 'My_Testing_Profile.mobileprovision')
displayName: Install Testing Apple Provisioning Profile
inputs:
provisioningProfileLocation: 'secureFiles'
provProfileSecureFile: My_Testing_Profile.mobileprovision
- task: InstallAppleCertificate#2
condition: eq('${{ parameters.CertSecureFile }}', 'ios_distribution.p12')
displayName: Install Apple Certificate
inputs:
certSecureFile: ios_distribution.p12
certPwd: $(CertPwd)
setUpPartitionIdACLForPrivateKey: false
deleteCert: false
deleteCustomKeychain: false
- task: InstallAppleProvisioningProfile#1
condition: eq('${{ parameters.ProvProfile }}', 'My_Distribution_Profile.mobileprovision')
displayName: Install Testing Apple Provisioning Profile
inputs:
provisioningProfileLocation: 'secureFiles'
provProfileSecureFile: My_Distribution_Profile.mobileprovision
updates:
If you want to use template with parameters, you could put the repeated two jobs with two fixed value in two templates yaml in the same repo and branch.(here we name it for example, template1.yaml and template2.yaml)
Then you could try something like this:
steps:
- ${{ if and(eq(parameters.CertSecureFile, 'ios_development.p12'),eq(parameters.ProvProfile, 'My_Testing_Profile.mobileprovision')) }}:
- template: template1.yaml
- ${{ else }}:
- template: template2.yaml

Related

Devops Nuget versioning byBuildNumber does not work

I want the version numbers of the Nuget package to be incremented and human readable with every version pushed to the Artifacts store. This is my yaml.
variables:
feedName : 'MyNugets'
buildConfiguration: 'release'
name: $(Year:yyyy).$(Month).$(DayOfMonth).$(Rev:.r)
pool:
vmImage: windows-latest
steps:
- task: DotNetCoreCLI#2
displayName: 'Build'
inputs:
command: 'build'
projects: '**/*.csproj'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: 'Pack'
inputs:
command: pack
versioningScheme: byBuildNumber
- task: NuGetAuthenticate#1
displayName: 'NuGet Authenticate'
- task: NuGetCommand#2
displayName: 'NuGet push'
inputs:
command: push
publishVstsFeed: '$(feedName)'
allowPackageConflicts: true
Whatever suggestion I could find on SO or other google result. None of them seems to be working. I always get this error.
##[error]Could not find version number data in the following environment variable: BUILD_BUILDNUMBER. The value of the variable should contain a substring with the following formats: X.Y.Z or X.Y.Z.A where A, X, Y, and Z are positive integers.
Any suggestions ?
Don't store the name in a variable, put in in a separate section above variables.
name: $(Year:yyyy).$(Month).$(DayOfMonth).$(Rev:.r)
variables:
feedName : 'MyNugets'
buildConfiguration: 'release'
See: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/run-number?view=azure-devops&tabs=yaml

Azure DevOps: Populating secure file references with job matrix variables

For context, I am trying to use an Azure build pipeline to build multiple flavors of an Android app. Each flavor has its own separate signing keystore, and all of those keystores are stored in my 'secure files' in the library.
However, when I try to dereference the $(Keystore) variable during the 'android signing' task, it doesn't seem to recognize that that is a variable that exists, and tries instead to locate a file called '$(Keystore)'
Am I doing something wrong here? This seems like it should work.
A sanitized example looks like this:
# Android
# Build your Android project with Gradle.
# Add steps that test, sign, and distribute the APK, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/android
trigger:
- feat/ci-setup
pool:
vmImage: 'macos-latest'
variables:
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/feat/') }}:
Branch_Type: 'feature'
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/hotfix/') }}:
Branch_Type: 'hotfix'
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/release/') }}:
Branch_Type: 'release'
${{ if eq(variables['Branch_Type'], 'release') }}:
Configuration: 'release'
ConfigurationCC: 'Release'
${{ if ne(variables['Branch_Type'], 'release') }}:
Configuration: 'debug'
ConfigurationCC: 'Debug'
jobs:
- job: Build
variables:
- group: android_keystores
strategy:
maxParallel: 2
matrix:
Flavor_1:
AppFlavor: '1'
AppFlavorCC: '1'
Keystore: 'flavor1.keystore'
KeyAlias: 'flavor1'
KeystorePass: '$(flavor1_storepass)'
KeyPass: '$(flavor1_keypass)'
Flavor_2:
AppFlavor: '2'
AppFlavorCC: '2'
Keystore: 'flavor2.keystore'
KeyAlias: 'flavor2'
KeystorePass: '$(flavor2_storepass)'
KeyPass: '$(flavor2_keypass)'
steps:
- task: Gradle#2
inputs:
workingDirectory: ''
gradleWrapperFile: 'gradlew'
gradleOptions: '-Xmx3072m'
publishJUnitResults: false
tasks: 'assemble$(AppFlavorCC)$(ConfigurationCC)'
- task: AndroidSigning#3
displayName: Signing .apk
inputs:
apkFiles: 'app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk'
apksign: true
apksignerKeystoreFile: '$(Keystore)'
apksignerKeystorePassword: '$(KeystorePass)'
apksignerKeystoreAlias: '$(KeyAlias)'
apksignerKeyPassword: '$(KeyPass)'
zipalign: true
- task: Bash#3
displayName: Move APK to Artifact Folder
continueOnError: true
inputs:
targetType: 'inline'
script: |
mv \
app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk \
$(Build.ArtifactStagingDirectory)/$(ArtifactName)/
- task: PublishBuildArtifacts#1
displayName: Publish Build Artifacts
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'Blueprint-Build'
publishLocation: 'Container'
But when the pipeline runs I am told this:
There was a resource authorization issue: "The pipeline is not valid. Job Build: Step AndroidSigning input keystoreFile references secure file $(Keystore) which could not be found. The secure file does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Azure DevOps: Populating secure file references with job matrix variables
This is a limitation from the task itself.
When we test it with Classic mode, we could find out that the value of the option Keystore file could not be entered manually, we could only select a certain file through the drop-down menu:
That the reason why it doesn't seem to recognize that that is a variable that exists, and tries instead to locate a file called '$(Keystore)'.
To resolve this issue, you could change the task version from 3 to 1, which supports manual input:
And as another solution, you could also use the command line to sign the *.apk:
Android apk signing: sign an unsigned apk using command line
You're missing the step to download the Secure File. Unlike variable groups, you need to explicitly download them to have access via the secure file name.
You'll want to add something similar to the example task below to your steps to pull the secure file. Then, you'll access your secure file via NAME_PARAMETER.secureFilePath:
- task: DownloadSecureFile#1
displayName: "Download Keyfile 1"
name: "YOUR_SECUREFILE_NAME"
inputs:
secureFile: keyfile1
- task: AndroidSigning#3
displayName: Signing .apk
inputs:
apkFiles: 'app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk'
apksign: true
apksignerKeystoreFile: '$(YOUR_SECUREFILE_NAME.secureFilePath)'
apksignerKeystorePassword: '$(KeystorePass)'
apksignerKeystoreAlias: '$(KeyAlias)'
apksignerKeyPassword: '$(KeyPass)'
zipalign: true

Azure DevOps Multi Stage Pipeline Error: No package found with specified pattern: /home/vsts/work/1/s/**/*.zip - How do I fix?

I have an Azure DevOps Build (yaml) and Release Pipeline (Classic) successfully deploying to Azure.
I am trying to convert these 2 separate steps in a Multi Stage Yaml Pipeline.
On the Azure App Service Deploy task (AzureRmWebAppDeployment#4), I am getting the following error:
No package found with specified pattern: /home/vsts/work/1/a/*.zip
Below is my Multi Stage Yaml Pipeline
stages:
- stage: Build
jobs:
- job: 'Build'
pool:
vmImage: 'windows-latest'
variables:
buildConfiguration: 'Release'
steps:
- task: DotNetCoreCLI#2
displayName: Restore
inputs:
command: restore
projects: '**/*.csproj'
vstsFeed: 'dd55642d-8943-411f-8856-9714dd0da8af'
- task: DotNetCoreCLI#2
displayName: Build
inputs:
projects: '**/*.csproj'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: Test
inputs:
command: test
projects: '**/*[Tt]ests/*.csproj'
arguments: '--configuration $(buildConfiguration)'
- task: DotNetCoreCLI#2
displayName: Publish
inputs:
command: publish
publishWebProjects: false
projects: '**/Tools.Client.Blazor.ServerApp.csproj'
arguments: '--configuration $(buildConfiguration) --output $(build.artifactstagingdirectory)'
- task: PublishSymbols#2
displayName: 'Publish symbols path'
inputs:
SearchPattern: '**\bin\**\*.pdb'
PublishSymbols: false
continueOnError: true
- task: CopyFiles#2
displayName: 'Copy Files to: $(build.artifactstagingdirectory)\AzureDeploy'
inputs:
SourceFolder: AzureDeploy
TargetFolder: '$(build.artifactstagingdirectory)\AzureDeploy'
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
condition: succeededOrFailed()
- stage: Systest
jobs:
- job: 'Systest'
variables:
resourceGroupName: '$(appName)-rg-$(environment)'
location: 'East US'
appServiceName: '$(appName)-svc-$(environment)'
appInsightsName: '$(appName)-ins-$(environment)'
appServicePlanName: '$(appName)-asp-$(environment)'
appName: 'tools'
owner: 'Pod'
environment: 'systest'
steps:
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'ARM Template deployment: Resource Group scope'
inputs:
azureResourceManagerConnection: 'Dev/Test Connection'
subscriptionId: ''
resourceGroupName: '$(resourceGroupName)'
location: '$(location)'
csmFile: '$(System.DefaultWorkingDirectory)/AzureDeploy/Tools.azureDeploy.json'
csmParametersFile: '$(System.DefaultWorkingDirectory)/AzureDeploy/Tools.azureDeploy.parameter.json'
overrideParameters: '-appServiceName "$(appServiceName)" -appInsightsName "$(appInsightsName)" -appServicePlanName "$(appServicePlanName)" -owner "$(owner)" -environment "$(environment)" -location "$(location)"'
- task: AzureRmWebAppDeployment#4
displayName: 'Azure App Service Deploy: $(appServiceName)'
inputs:
ConnectionType: 'AzureRM'
azureSubscription: ''
appType: 'webApp'
WebAppName: '$(appServiceName)'
packageForLinux: '$(Build.ArtifactStagingDirectory)/*.zip'
Any help / suggestions would be appreciated.
Because it's 2 stages the second stage doesn't have the file you published in the first stage, you need to download it.
You can use Pipeline artifacts instead of build artifacts.
Pipeline artifacts provide a way to share files between stages in a
pipeline or between different pipelines. They are typically the output
of a build process that needs to be consumed by another job or be
deployed. Artifacts are associated with the run they were produced in
and remain available after the run has completed.
To publish (upload) an artifact for the current run:
steps:
- publish: $(build.artifactstagingdirectory)
artifact: drop
And in the second stage, you download the artifact:
steps:
- download: current
artifact: drop
You can also achieve it with build artifacts and download with DownloadBuildArtifacts#0 task.
During Publish it will not work like this. Instead of using path "/home/vsts/work/1/a/.zip", this path can be used "$(System.DefaultWorkingDirectory)/_Releasepipelinename/drop/.zip"

Azure DevOps YAML pipeline manual intervention job run in parallel with other job

I want the jobs to go one after another and the first job should control the execution of the following one.
As there are no approvals currently available in the YAML pipeline for deployment outside Kubernetes, I'm using the Manual Intervention to stop the job from being run. But apparently, it doesn't stop the job before instead, it stops the upcoming stage. What do I do wrong? I would expect some notification on the intervention, but it fails immediately and doesn't stop the next job at all.
This is the part of the code for the Deploy STG stage, where the parameters.interventionEnabled is set to true
jobs:
- job: RunOnServer
displayName: 'Reject or resume'
pool: server
continueOnError: false
steps:
- task: ManualIntervention#8
displayName: 'Manual Intervention'
timeoutInMinutes: 0
inputs:
instructions: 'reject or resume'
enabled: ${{ parameters.interventionEnabled }}
- job: Deploy
displayName: ${{ parameters.name }}
pool:
name: ${{ parameters.agentPoolName }}
steps:
- checkout: none # skip checking out the default repository resource
- task: DownloadPipelineArtifact#2
displayName: Download NPM build artifact
inputs:
artifact: ${{ parameters.artifactName }}
buildType: 'current'
targetPath: ${{ parameters.artifactPath }}
Hey Andree ManualIntervention#8 is not supported in YAML. It is roadmapped for 2020\Q2.
I think the route you want to go down is to use approvals with generic environment types.
So you define a deployment job and environment in your yaml like so
- deployment: DeploymentHosted Context
displayName: Runs in Hosted Pool
pool:
vmImage: 'Ubuntu-16.04'
# creates an environment if it doesn't exist
environment: 'Dev'
strategy:
runOnce:
deploy:
steps:
- bash: |
echo This multiline script always runs in Bash.
echo Even on Windows machines!
And you use the GUI to protect the Environment.
Navigate to Pipelines -> Environments.
Select the Environment (you can pre-create them).
Then add and an Approval
There are some drawbacks when compared to classic release definitions and being able to manual trigger to stages. You may not want every artifact to be a candidate for each stage, and if you don't approve the environment it will eventually timeout and report failure. Other good discussion in the comments here.
This is now available:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/manual-validation?view=azure-devops&tabs=yaml
- task: ManualValidation#0
timeoutInMinutes: 1440 # task times out in 1 day
inputs:
notifyUsers: |
test#test.com
example#example.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'resume'

Pass array as inputs to Azure DevOps YAML task

I'm trying to configure a CI that will produce NuGet packages as artifacts on Azure DevOps (that will be later pushed to my NuGet server).
For this, I'm using Builds Pipelines on Azure DevOps, the YAML version.
I have 3 projects that should build packages. I'm using NuGetCommand#2 to accomplish this task :
- task: NuGetCommand#2
inputs:
command: pack
packagesToPack: $(Build.SourcesDirectory)/src/HTS_MessageEngine.Communication/HTS_MessageEngine.Communication.csproj
majorVersion: $(majorVersion)
minorVersion: $(minorVersion)
patchVersion: $(patchVersion)
versioningScheme: byPrereleaseNumber
However, I have to duplicate this block 3 times, for each project. Is there a way to specify an array of projects in the packagesToPack parameter? So far, the version is the same for each package, so I don't need three different blocks...
Note: these 3 projects are all 3 NetStandard and properties for package building are stored in csproj directly
you can use each function (if its available at all at this point in time):
# my-template.yml
parameters:
steps:
- ${{ each project in parameters.projects }}:
- task: PublishBuildArtifacts#1
displayName: Publish ${{ project }}
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ project }}.zip'
# ci.yml
steps:
- template: my-template.yml
parameters:
projects:
- test1
- test2
Github PR for this functionality: https://github.com/Microsoft/azure-pipelines-yaml/pull/2#issuecomment-452748467
For above code I got this exception running a build:
my-template.yml (Line: 1, Col: 12): Unexpected value ''
but this works for me:
# my-template.yml
parameters:
- name: projects
type: object
default: {}
steps:
- ${{ each project in parameters.projects }}:
- task: PublishBuildArtifacts#1
displayName: 'Publish ${{ project }}
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ project }}.zip'
and then:
# ci.yml
steps:
- template: my-template.yml
parameters:
projects:
- test1
- test2