Unable to download secure files conditionally in Azure Pipelines - azure-devops

Question
I am using DownloadSecureFile#1 task to download Secure files.
The issue occurs when in Azure DevOps, in the Library's secure files section, only file_A.txt exists.
The script works fine when both files exists.
In my case, a user A will only need file_A.txt, user B will only need file_B.txt.
Is this an expected behavior? Any possible workarounds to fulfill the use-case?
Error Message:
There was a resource authorization issue: "The pipeline is not valid. Job Job: Step fileB input secureFile references secure file file_B.txt which could not be found. The secure file does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Code:
parameters:
- name: file_name
type: string
default: ''
values:
- file_A.txt
- file_B.txt
pool:
vmImage: ubuntu-latest
steps:
- task: DownloadSecureFile#1
displayName: Download File A
condition: eq('${{ parameters.file_name }}', 'file_A.txt')
name: fileA
inputs:
secureFile: 'file_A.txt'
- task: DownloadSecureFile#1
displayName: Download file B
condition: eq('${{ parameters.file_name }}', 'file_B.txt')
name: fileB
inputs:
secureFile: 'file_B.txt'

Is this an expected behavior?
Yes, this is expected behavior. To turn a pipeline into a run, Azure Pipelines goes through several steps in this order:
First, expand templates and evaluate template expressions.
Next, evaluate dependencies at the stage level to pick the first
stage(s) to run.
For each stage selected to run, two things happen:
All resources used in all jobs are gathered up and validated for
authorization to run.
Evaluate dependencies at the job level to pick the first job(s) to
run.
For each job selected to run, expand multi-configs (strategy: matrix
or strategy: parallel in YAML) into multiple runtime jobs.
For each runtime job, evaluate conditions to decide whether that job
is eligible to run.
Request an agent for each eligible runtime job.
So, your secure files will be downloaded before evaluating conditions. Please refer to the document about Pipeline run sequence. As a workaround, you can refer to the sample shared by #danielorn.

Instead of using the condition on the tasks you can surround the step with an if-statement as described in use parameters to determine what steps run
parameters:
- name: file_name
type: string
default: ''
values:
- file_A.txt
- file_B.txt
pool:
vmImage: ubuntu-latest
steps:
- ${{ if eq(parameters.file_name, 'file_A.txt') }}:
- task: DownloadSecureFile#1
displayName: Download File A
name: fileA
inputs:
secureFile: 'file_A.txt'
- ${{ if eq(parameters.file_name, 'file_B.txt') }}:
- task: DownloadSecureFile#1
displayName: Download file B
name: fileB
inputs:
secureFile: 'file_B.txt'
However if every user needs exactly one file, a common (and cleaner) option would be to provide the name of the file needed as a parameter. If a secure file is not needed (i.e the parameter is the default empty) the step can be excluded using an if statement
parameters:
- name: file_name
type: string
default: ''
values:
- file_A.txt
- file_B.txt
pool:
vmImage: ubuntu-latest
steps:
- ${{ if ne(parameters.file_name, '') }}:
- task: DownloadSecureFile#1
displayName: Download Secure File
name: secureFileDownload
inputs:
secureFile: '${{ parameters.file_name }}'

Related

conditional ManualValidation step in Azure DevOps pipeline

Using the Azure DevOps ManualValidation task, can the job run conditionally, based on variables defined earlier in the pipeline?
The job accepts an enabled parameter, but it seems this must be hard-coded to true or false.
- stage: Approve_${{ targetPath.stageName }}_${{ parameters.planEnvironment }}
jobs:
- job: waitForValidation
displayName: Wait for external validation
pool: server
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
steps:
- task: ManualValidation#0
environment: development ## environment not accepted here
#enabled: $[destroy] ## Unexpected value '$[destroy]'",
#enabled: $(destroy) ## fails - syntax error (does not like this to be a var)
# manually setting true/false works
#enabled: true
#enabled: false
inputs:
notifyUsers: |
alert#test.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'reject'
enabled control option is boolean type, which means whether or not to run this step, defaults to 'true'. If you want to use enabled control option to condition ManualValidation step, you could check the following syntax:
variables:
- name: destroy
value: true
jobs:
- job: waitForValidation
displayName: Wait for external validation
pool: server
timeoutInMinutes: 4320 # job times out in 3 days
steps:
- task: ManualValidation#0
timeoutInMinutes: 1440 # task times out in 1 day
enabled: ${{ variables.destroy }}
inputs:
notifyUsers: |
test#test.com
example#example.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'resume'
Otherwise can specify the conditions under the steps as #Shayki Abramczyk mentioned.
You can use a custom condition like in every task:
- task: ManualIntervention#8
inputs:
instructions: 'test'
condition: and(succeeded(), eq(variables['VariableName'], 'VariableValue'))
Another option is to use template parameter conditionals to make the stage, job, or task go away when you don't want it there. That is, when the preprocessing is done on the pipeline code, use conditionals to avoid inserting the item into the expanded yaml file that is actually run when the pipeline runs.
- ${{ if ne(parameters.EnvironmentName, 'Dev') }}:
- stage: Approve_${{ targetPath.stageName }}_${{ parameters.planEnvironment }}
jobs:
- job: waitForValidation
displayName: Wait for external validation
pool: server
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
steps:
- task: ManualValidation#0
environment: development ## environment not accepted here
inputs:
notifyUsers: |
alert#test.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'reject'
This above example is using a pipeline template parameter named EnvironmentName. You have the ability to use some variables, as explained in Azure DevOps's Predefined variables documentation (see the "Available in templates?" table column).
Note that any subsequent stages that refer to this stage (or job, if you put the condition on the job) via the dependsOn functionality will also need to make that dependsOn disappear with the same logic.
More info at the Conditional Insertion official documentation.

Azure DevOps: Populating secure file references with job matrix variables

For context, I am trying to use an Azure build pipeline to build multiple flavors of an Android app. Each flavor has its own separate signing keystore, and all of those keystores are stored in my 'secure files' in the library.
However, when I try to dereference the $(Keystore) variable during the 'android signing' task, it doesn't seem to recognize that that is a variable that exists, and tries instead to locate a file called '$(Keystore)'
Am I doing something wrong here? This seems like it should work.
A sanitized example looks like this:
# Android
# Build your Android project with Gradle.
# Add steps that test, sign, and distribute the APK, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/android
trigger:
- feat/ci-setup
pool:
vmImage: 'macos-latest'
variables:
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/feat/') }}:
Branch_Type: 'feature'
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/hotfix/') }}:
Branch_Type: 'hotfix'
${{ if startsWith(variables['build.sourceBranch'], 'refs/heads/release/') }}:
Branch_Type: 'release'
${{ if eq(variables['Branch_Type'], 'release') }}:
Configuration: 'release'
ConfigurationCC: 'Release'
${{ if ne(variables['Branch_Type'], 'release') }}:
Configuration: 'debug'
ConfigurationCC: 'Debug'
jobs:
- job: Build
variables:
- group: android_keystores
strategy:
maxParallel: 2
matrix:
Flavor_1:
AppFlavor: '1'
AppFlavorCC: '1'
Keystore: 'flavor1.keystore'
KeyAlias: 'flavor1'
KeystorePass: '$(flavor1_storepass)'
KeyPass: '$(flavor1_keypass)'
Flavor_2:
AppFlavor: '2'
AppFlavorCC: '2'
Keystore: 'flavor2.keystore'
KeyAlias: 'flavor2'
KeystorePass: '$(flavor2_storepass)'
KeyPass: '$(flavor2_keypass)'
steps:
- task: Gradle#2
inputs:
workingDirectory: ''
gradleWrapperFile: 'gradlew'
gradleOptions: '-Xmx3072m'
publishJUnitResults: false
tasks: 'assemble$(AppFlavorCC)$(ConfigurationCC)'
- task: AndroidSigning#3
displayName: Signing .apk
inputs:
apkFiles: 'app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk'
apksign: true
apksignerKeystoreFile: '$(Keystore)'
apksignerKeystorePassword: '$(KeystorePass)'
apksignerKeystoreAlias: '$(KeyAlias)'
apksignerKeyPassword: '$(KeyPass)'
zipalign: true
- task: Bash#3
displayName: Move APK to Artifact Folder
continueOnError: true
inputs:
targetType: 'inline'
script: |
mv \
app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk \
$(Build.ArtifactStagingDirectory)/$(ArtifactName)/
- task: PublishBuildArtifacts#1
displayName: Publish Build Artifacts
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'Blueprint-Build'
publishLocation: 'Container'
But when the pipeline runs I am told this:
There was a resource authorization issue: "The pipeline is not valid. Job Build: Step AndroidSigning input keystoreFile references secure file $(Keystore) which could not be found. The secure file does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Azure DevOps: Populating secure file references with job matrix variables
This is a limitation from the task itself.
When we test it with Classic mode, we could find out that the value of the option Keystore file could not be entered manually, we could only select a certain file through the drop-down menu:
That the reason why it doesn't seem to recognize that that is a variable that exists, and tries instead to locate a file called '$(Keystore)'.
To resolve this issue, you could change the task version from 3 to 1, which supports manual input:
And as another solution, you could also use the command line to sign the *.apk:
Android apk signing: sign an unsigned apk using command line
You're missing the step to download the Secure File. Unlike variable groups, you need to explicitly download them to have access via the secure file name.
You'll want to add something similar to the example task below to your steps to pull the secure file. Then, you'll access your secure file via NAME_PARAMETER.secureFilePath:
- task: DownloadSecureFile#1
displayName: "Download Keyfile 1"
name: "YOUR_SECUREFILE_NAME"
inputs:
secureFile: keyfile1
- task: AndroidSigning#3
displayName: Signing .apk
inputs:
apkFiles: 'app/build/outputs/apk/$(AppFlavor)/$(Configuration)/*.apk'
apksign: true
apksignerKeystoreFile: '$(YOUR_SECUREFILE_NAME.secureFilePath)'
apksignerKeystorePassword: '$(KeystorePass)'
apksignerKeystoreAlias: '$(KeyAlias)'
apksignerKeyPassword: '$(KeyPass)'
zipalign: true

Azure DevOps YAML pipeline manual intervention job run in parallel with other job

I want the jobs to go one after another and the first job should control the execution of the following one.
As there are no approvals currently available in the YAML pipeline for deployment outside Kubernetes, I'm using the Manual Intervention to stop the job from being run. But apparently, it doesn't stop the job before instead, it stops the upcoming stage. What do I do wrong? I would expect some notification on the intervention, but it fails immediately and doesn't stop the next job at all.
This is the part of the code for the Deploy STG stage, where the parameters.interventionEnabled is set to true
jobs:
- job: RunOnServer
displayName: 'Reject or resume'
pool: server
continueOnError: false
steps:
- task: ManualIntervention#8
displayName: 'Manual Intervention'
timeoutInMinutes: 0
inputs:
instructions: 'reject or resume'
enabled: ${{ parameters.interventionEnabled }}
- job: Deploy
displayName: ${{ parameters.name }}
pool:
name: ${{ parameters.agentPoolName }}
steps:
- checkout: none # skip checking out the default repository resource
- task: DownloadPipelineArtifact#2
displayName: Download NPM build artifact
inputs:
artifact: ${{ parameters.artifactName }}
buildType: 'current'
targetPath: ${{ parameters.artifactPath }}
Hey Andree ManualIntervention#8 is not supported in YAML. It is roadmapped for 2020\Q2.
I think the route you want to go down is to use approvals with generic environment types.
So you define a deployment job and environment in your yaml like so
- deployment: DeploymentHosted Context
displayName: Runs in Hosted Pool
pool:
vmImage: 'Ubuntu-16.04'
# creates an environment if it doesn't exist
environment: 'Dev'
strategy:
runOnce:
deploy:
steps:
- bash: |
echo This multiline script always runs in Bash.
echo Even on Windows machines!
And you use the GUI to protect the Environment.
Navigate to Pipelines -> Environments.
Select the Environment (you can pre-create them).
Then add and an Approval
There are some drawbacks when compared to classic release definitions and being able to manual trigger to stages. You may not want every artifact to be a candidate for each stage, and if you don't approve the environment it will eventually timeout and report failure. Other good discussion in the comments here.
This is now available:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/manual-validation?view=azure-devops&tabs=yaml
- task: ManualValidation#0
timeoutInMinutes: 1440 # task times out in 1 day
inputs:
notifyUsers: |
test#test.com
example#example.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'resume'

Azure Devops - passing variables between job templates

Normal (non-template) jobs in Azure DevOps yaml support inter-job variable passing as follows:
jobs:
- job: A
steps:
- script: "echo ##vso[task.setvariable variable=skipsubsequent;isOutput=true]false"
name: printvar
- job: B
condition: and(succeeded(), ne(dependencies.A.outputs['printvar.skipsubsequent'], 'true'))
dependsOn: A
steps:
- script: echo hello from B
How do I do something similar in the following, given that templates don't support the dependsOn syntax? I need to get an output from the first template and pass it as 'environmentSlice' to the second template.
- stage: Deploy
displayName: Deploy stage
jobs:
- template: build-templates/get-environment-slice.yml#templates
parameters:
configFileLocation: 'config/config.json'
- template: build-templates/node-app-deploy.yml#templates
parameters:
# Build agent VM image name
vmImageName: $(Common.BuildVmImage)
environmentPrefix: 'Dev'
environmentSlice: '-$(dependencies.GetEnvironmentSlice.outputs['getEnvironmentSlice.environmentSlice'])'
The reason I want the separation between the two templates is the second one is a deployment template and I would like input from the first template in naming the environment in the second template. I.e. initial part of node-app-deploy.yml (2nd template) is:
jobs:
- deployment: Deploy
displayName: Deploy
# Because we use the environmentSlice to name the environment, we have to have it passed in rather than
# extracting it from the config file in steps below
environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}
Update:
The accepted solution does allow you to pass variables between separate templates, but won't work for my particular use case. I wanted to be able to name the 'environment' section of the 2nd template dynamically, i.e. environment: ${{ parameters.environmentPrefix }}${{ parameters.environmentSlice }}, but this can only be named statically since templates are compiled on pipeline startup.
The downside of the solution is that it introduces a hidden coupling between the templates. I would have preferred the calling pipeline to orchestrate the parameter passing between templates.
You can apply the depend on and dependency variable into templates.
See below sample:
To make sample more clear, here has 2 template files, one is azure-pipelines-1.yml, and another is azure-pipeline-1-copy.yml.
In azure-pipelines-1.yml, specify the environment value as output variable:
parameters:
  environment: ''
jobs:
- job: preDeploy
  variables:
    EnvironmentName: preDeploy-${{ parameters.environment }}
  steps:
  - checkout: none
  - pwsh: |
      echo "##vso[task.setvariable variable=EnvironmentName;isOutput=true]$($env:ENVIRONMENTNAME)"
    name: outputVars
And then, in azure-pipeline-1-copy.yml use dependency to get this output variable:
jobs:
- job: deployment
  dependsOn: preDeploy
  variables:
    EnvironmentNameCopy: $[dependencies.preDeploy.outputs['outputVars.EnvironmentName']]
  steps:
  - checkout: none
  - pwsh: |
      Write-Host "$(EnvironmentNameCopy)"
    name: outputVars
At last, in YAML pipeline, just need to pass the environment value
stages:
  - stage: deployQA
    jobs:
    - template: azure-pipelines-1.yml
      parameters:
        environment: FromTemplate1
    - template: azure-pipeline-1-copy.yml
Now, you can see the value get successfully in the second template job:
It is possible to avoid the dependency in the called template. However, as the OP says, the environment name cannot be created dynamically.
Here is an example of the "calling" template, which firstly calls another template (devops-variables.yml) that sets some environment variables that we wish to consume in a later template (devops-callee.yml):
stages:
- stage: 'Caller_Stage'
displayName: 'Caller Stage'
jobs:
- template: 'devops-variables.yml'
parameters:
InitialEnvironment: "Development"
- template: 'devops-callee.yml'
parameters:
SomeParameter: $[dependencies.Variables_Job.outputs['Variables_Job.Variables.SomeParameter']]
In the devops-variables.yml file, I have this:
"##vso[task.setvariable variable=SomeParameter;isOutput=true;]Wibble"
Then, in the "devops-callee.yml", I just consume it something like this:
parameters:
- name: SomeParameter
default: ''
jobs:
- deployment: 'Called_Job'
condition: succeeded()
displayName: 'Called Job'
environment: "Development"
pool:
vmImage: 'windows-2019'
dependsOn:
- Variables_Job
variables:
SomeParameter: ${{parameters.SomeParameter}}
strategy:
runOnce:
deploy:
steps:
- download: none
- task: AzureCLI#2
condition: succeeded()
displayName: 'An Echo Task'
inputs:
azureSubscription: "$(TheServiceConnection)"
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
echo "Before"
echo "$(SomeParameter)"
echo "After"
Output:
2021-04-10T09:22:29.6188535Z Before
2021-04-10T09:22:29.6196620Z Wibble
2021-04-10T09:22:29.6197124Z After
This way, the callee doesn't reference the caller. Unfortunately, setting the environment in the callee thus:
environment: "$(SomeParameter)"
doesn't work - you'll just get an environment with the literal characters '$(SomeParameter)'.

Azure DevOps Using Taskgroup in YAML Build

I've created a Task group to encapsulate some functionality.
If I use a regular build, I can add the task group through the normal wizard.
Unfortunately, I need to use the task group inside a YAML build. I can't view the YAML of the "old" build to view how this should gonna happen.
The things I've tried:
- task: TaskGroupName#1
displayName: 'RunTests'
inputs:
TestConfiguration: 'some.xml'
TestCaseFilter: $(TestCaseFilter)
UnitTestFolders: $(UnitTestFolders)
According to the docs, Task groups are not supported in the YAML pipelines.
Instead, in that case you can use templates.
Documentation for templates: See here
You can pass parameter 'objects' into a template YAML file to pretty much do what you want; the only tricky bit I found was to have multiple properties per instance parameter 'object' and using the new template {{ each }} expression to iterate over them.
Below is how I constructed my yaml files for this solution:
azure_pipelines.yml
pool:
name: Hosted VS2017
demands:
- npm
- msbuild
- visualstudio
- vstest
steps:
- template: azure_webapp_template.yml
parameters:
webapps:
- name: Customer 1
url: customer1.azurewebsites.net
- name: Customer 2
url: customer2.azurewebsites.net
- name: Customer 3
url: customer3.azurewebsites.net
- name: Customer 4
url: customer4.azurewebsites.net
As you can see above, we are creating an object webapps and then we have some nested properties for each 'webapp'.
Then in our 'template' we can iterate over each of the objects in the webapps parameter and expand the property in our iterated tasks.
azure_webapp_template.yml
# Proving ability to loop over params a number of times
parameters:
- name: 'webapps'
type: object
default: {}
steps:
- ${{ each webapp in parameters.webapps }}:
- task: PowerShell#2
displayName: 'Task Group Test 1 ${{webapp.name}}'
inputs:
targetType: 'inline'
script: |
Write-Host "Name: ${{webapp.name}} with url ${{webapp.url}}"
failOnStderr: true
workingDirectory: '$(Build.SourcesDirectory)'
- task: PowerShell#2
displayName: 'Task Group Test 2 ${{webapp.name}}'
inputs:
targetType: 'inline'
script: |
Write-Host "Name: ${{webapp.name}} with url ${{webapp.url}}"
failOnStderr: true
workingDirectory: '$(Build.SourcesDirectory)'