Set Azure DevOps pipeline variable to array of values - powershell

I am trying to run a set of tests across a set of ADO builds.
I can retrieve the list of ADO builds using PowerShell. However, once I have the list, I need to export that to an ADO variable and then iterate across the values.
I've seen how to export values from Powershell to ADO using logging, but that appears to export the value as a string, not a list.
Is there a way to export variables so that I could iterate across them; e.g., using ${{ each foo in exportedVars }}?

First, for the usage you mentioned:
${{ each foo in exportedVars }}
This is a compile-time usage, it is expanded at the beginning, and you can't get the variables generated by the pipeline runtime through it.
Second, the pipeline can output variables through the logging command, but the variables set in this way can only be strings. This is by design, and the documentation has said it very clearly:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-variables-in-scripts
All variables set by this method are treated as strings.
It is not difficult to parse a string and put it into an array, just use the function(split()) that comes with the string type to split and restore.
Here is an example:
trigger:
- none
# 1
stages:
- stage: s1
displayName: setvars
jobs:
- job: testJob
steps:
- task: PowerShell#2
name: setvar
inputs:
targetType: 'inline'
script: |
# logic here. For example you get the vars and put it into this format:
$testvars = "testvar1,testvar2,testvar3"
Write-Host "##vso[task.setvariable variable=outputvars;isOutput=true]$testvars"
# 2
- stage: s2
displayName: getvars
dependsOn: s1
variables:
vars: $[ stageDependencies.s1.testJob.outputs['setvar.outputvars'] ]
jobs:
- job:
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$varsArr = "$(vars)".Split(',')
foreach ($var in $varsArr)
{
Write-Host "$var`r`n"
}
Result:

Related

How to use a variable in each loop in Azure DevOps yaml pipeline

I have a PowerShell script in my Azure DevOps pipeline:
- task: PowerShell#2
displayName: Get_records
inputs:
targetType: 'inline'
script: |
<...>
$records.Records
$records.Records is some variable which contains array of records. I need to use this data this way: for each record in this array I need to perfome several tasks within one job. Something like this:
stages:
- stage: stage_1
jobs:
- job: Job_1
- task: PowerShell#2
displayName: Get_records
inputs:
targetType: 'inline'
script: |
<..getting this records..>
$records.Records
- ${{each records in variables.records.Records}}:
- task: not_powershell
displayName: name1
inputs:
AsyncOperation: true
MaxAsyncWaitTime: '60'
- task: not_powershell
displayName: name2
inputs:
AsyncOperation: true
MaxAsyncWaitTime: '60'
How can I do that? There are a couple questions about this:
How to use '$records.Records' variable in foreach loop? Must I save variable before using? If yes - how to save an array?
If it is imposible to do it this way, probably there are some ways around, for example use several stages, jobs... etc?
No, you can't. It only works for parameters, not variables:
You can use the each keyword to loop through parameters
This is because:
before it runs, the yaml is parsed and compiled into a pipeline structure, with all the stages, jobs and tasks defined. This is the point when the loop condition is evaluated, and the loop is expanded into multiple stages/jobs/tasks.
dynamic runtime variables are not yet available at compile-time, so they cannot be used at this point to define the each loop.
at runtime, when the dynamic variable is available, the pipeline structure is already fixed, so it is not possible to add extra tasks at this point.
Workaround
You can't loop over a dynamic array variable in the pipeline; but one place you can do so is within a task, for example within a powershell script:
- task: PowerShell#2
displayName: Loop Over Records
inputs:
targetType: 'inline'
script: |
# first, get the records
ForEach ($record in $records.Records) {
# do something with $record
}
That isn't as powerful as a pipeline loop, with all the different types of non-powershell tasks available, but it might be the only way to loop over this array.

Use query-time variables in pipeline YAML

I have set up a pipeline with variables users can enter using the UI like this:
UI for userinput of variable called 'forceRelease'
I now want to use this variable in the pipeline yaml inside an if-statement like this:
jobs:
- job: Demo
steps:
- ${{ if eq(variables['forceRelease'], 'true') }}:
...some more stuff...
This does'nt work. I've tried different approaches but could not find the right syntax.
If I use the variable inside a condition, it works fine. Like this:
jobs:
- job: MAVEN_Build
- task: Bash#3
condition: eq(variables['forceRelease'], 'true')
I also tried to map the variable inside the variables block to a new pipeline variable like this:
variables:
isReleaseBranch: ${{ startsWith(variables['build.sourcebranch'],'refs/heads/pipelines-playground') }}
isForceRelease: $(forceRelease)
The first variable using 'build.sourcebranch' works fine. My approach using forceRelease doesnt work :(
Any ideas would be appreciated!
Cheers,
Dirk
AFAIK this is working as intended. User set variables are not expanded during parsing of the template.
You can read more on pipeline processing here:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/runs?view=azure-devops
You should instead use parameters.
parameters:
- name: "forceRelease"
type: boolean
default: "false"
- name: "someOtherParameter"
type: string
default: "someValue"
stages:
- ${{ if eq(parameters['forceRelease'], true)}}:
- stage: build
jobs:
- job: bash_job
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
# Write your commands here
And then when you run the pipeline you have the option to enable the parameter forceRelease or add someOtherParameter string.

Is there a way to have a variable group defined at stage level? If so how to access it at Job Level?

I am trying to find a way to define a variable group at stage level and then access it in below jobs through a template? How would I go about doing this?
# Template file getchangedfilesandvariables.yaml
parameters:
- name: "previouscommitid"
type: string
steps:
- task: PowerShell#2
displayName: 'Get the changed files'
name: CommitIds
inputs:
targetType: 'filePath'
filePath: '$(Build.SourcesDirectory)\AzureDevOpsPipelines\Get-COChangedfiles.ps1'
arguments: >
-old_commit_id ${{ previouscommitid }}
- task: PowerShell#2
name: PassOutput
displayName: 'Getting Variables for Packaging'
inputs:
targetType: 'filepath'
filepath: '$(System.DefaultWorkingDirectory)\AzureDevOpsPipelines\Get-COADOvariables.ps1'
And below is my yaml file.
trigger: none
name: $(BuildID)
variables:
system.debug: true
CodeSigningCertThumbprint: "somethumbprint"
# Triggering builds on a branch itself.
${{ if startsWith(variables['Build.SourceBranch'], 'refs/heads/') }}:
branchName: $[ replace(variables['Build.SourceBranch'], 'refs/heads/', '') ]
# Triggering builds from a Pull Request.
${{ if startsWith(variables['Build.SourceBranch'], 'refs/pull/') }}:
branchName: $[ replace(variables['System.PullRequest.SourceBranch'], 'refs/heads/', '') ]
## it will create pipeline package and it will push it private or public feed artifacts
stages:
- stage: Stage1
variables:
- group: Cloudops
- name: oldcommitid
value: $[variables.lastcommitid]
jobs:
- job: IdentifyChangedFilesAndGetADOVariables
pool:
name: OnPrem
workspace:
clean: all # Ensure the agent's directories are wiped clean before building.
steps:
- powershell: |
[System.Version]$PlatformVersion = ((Get-Content "$(Build.SourcesDirectory)\AzureDevOpsPipelines\PlatformVersion.json") | ConvertFrom-Json).PlatformVersion
Write-Output "The repository's PlatformVersion is: $($PlatformVersion.ToString())"
$NewPackageVersion = New-Object -TypeName "System.Version" -ArgumentList #($PlatformVersion.Major, $PlatformVersion.Minor, $(Build.BuildId))
Write-Output "This run's package version is $($NewPackageVersion.ToString())"
echo "##vso[task.setvariable variable=NewPackageVersion]$($NewPackageVersion.ToString())"
echo "##vso[task.setvariable variable=commitidold;isOutput=true]$(oldcommitid)"
displayName: 'Define package version.'
name: commitidorpackageversion
errorActionPreference: stop
- template: getchangedfilesandvariables.yaml
parameters:
previouscommitid:
- $(commitidorpackageversion.commitidold)
# - $(oldcommitid)
I get the error at the second last line of the code that
/AzureDevOpsPipelines/azure-pipelines.yml (Line: 49, Col: 13): The 'previouscommitid' parameter is not a valid String.
I tried different combinations but I am still getting the errors.
Any ideas?
Thanks for your response. I already had the variable group setup in my library. I was just not able to use it.
The way I was able to achieve this I created another template file and supplied it to variables section under my stage. After doing this I was able to actually able to use the variables from my variable group in my successive jobs.
For more information you can review this doc : https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml
stagevariables.yaml
variables:
- group: Cloudops
azure-pipelines.yml
stages:
- stage: Stage1
variables:
- template: stagevariables.yaml
jobs:
- job: CheckwhichfeedsAreAvailable
In YAML pipeline, you can't define a new variable group under the variables key.
Actually, we do not have the syntax can be available to create new variable group when running the YAML pipeline.
Under the variables key, you can:
Define new variables with the specified values.
Override the existing variables with new values.
Reference the variables from the existing variable groups and variable templates.
So, if you want to use a variable group with some variables in the pipeline, you should manually define the variable group on the Pipelines > Library page, then reference it in the pipeline.

Azure DevOps yaml: use a powershell task output parameter to generate a loop in dependent job

I have the following yaml as used in an Azure DevOps pipeline (this is not the full pipeline - it's just a portion of yaml that is in a template):
jobs:
- job: CheckExcludedWorkspaces
displayName: Check Excluded Workspaces
pool:
name: DefaultWindows
steps:
- task: PowerShell#2
name: GetWorkspaces
displayName: Check Excluded Workspaces
inputs:
filePath: "$(System.DefaultWorkingDirectory)/pipelines_v2/powershell/checkExcludedWorkspaces.ps1"
targetType: FilePath
errorActionPreference: 'stop'
arguments: -environmentFolder "$(rootFolderPrefix)\${{parameters.environmentFolder}}" -excludeFolderList "${{parameters.tagOutList}}"
pwsh: false
# - ${{ each folder in dependencies.CheckExcludedWorkspaces.outputs['GetWorkspaces.WorkspaceList'] }}:
- job: NewJob
dependsOn: CheckExcludedWorkspaces
variables:
testVar: $[ dependencies.CheckExcludedWorkspaces.outputs['GetWorkspaces.WorkspaceList'] ]
pool:
name: DefaultWindows
steps:
- powershell: |
Write-Host "Test var = $(testVar)"
displayName: Test workspaces output
this works correctly in that the second job retrieves a variable from a powershell task in the previous job and outputs that variable value. The task in the second job outputs a list of apps using variable testVar. The output contains:
app1,app2,app3,app4 etc
I would like to take this the next stage which is I would like to create a loop of jobs that repeated runs for this application list. Something like:
jobs:
- job: CheckExcludedWorkspaces
displayName: Check Excluded Workspaces
pool:
name: DefaultWindows
steps:
- task: PowerShell#2
name: GetWorkspaces
displayName: Check Excluded Workspaces
inputs:
filePath: "$(System.DefaultWorkingDirectory)/pipelines_v2/powershell/checkExcludedWorkspaces.ps1"
targetType: FilePath
errorActionPreference: 'stop'
arguments: -environmentFolder "$(rootFolderPrefix)\${{parameters.environmentFolder}}" -excludeFolderList "${{parameters.tagOutList}}"
pwsh: false
- ${{ each folder in dependencies.CheckExcludedWorkspaces.outputs['GetWorkspaces.WorkspaceList'] }}:
- job: NewJob
dependsOn: CheckExcludedWorkspaces
variables:
testVar: $[ dependencies.CheckExcludedWorkspaces.outputs['GetWorkspaces.WorkspaceList'] ]
pool:
name: DefaultWindows
steps:
- powershell: |
Write-Host "Test var = ${{folder}}"
displayName: Test workspaces output
This code gives me an error:
Unrecognized value: 'dependencies'. Located at position 1 within expression: dependencies.CheckExcludedWorkspaces.outputs['GetWorkspaces.WorkspaceList']
Is there a way i can use a powershell task output variable, to create a list of jobs in a dependent job? The problem is that i don't know at design time what the list of applications will be (the pipeline should ideally find this out when it runs). The list of applications is based on the list of folders that are created within a repository - which changes over time..
In current situation, we cannot use the 'each' key word for the variables. The 'each' keyword is used for the Obj type, but the variable is String.
For more details, you can refer the doc: Each keyword

Dynamic variables not available in other stages of azure pipeline

I am new to azure pipelines and am currently experimenting with the passing variables to the later jobs. Here is the current snippet whereby I am trying to extract the project version from the pom file using maven help: evaluate plugin. The pomVersion variable is populated but is not available in the same step or later steps in the second job via the projectVersionDynamic variable.
stages:
- stage: FirstStage
jobs:
- job: FirstJob
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
pomVersion=`mvn help:evaluate -Dexpression=project.version -q -DforceStdout`
echo $pomVersion ##Prints semantic version 2.27.0-SNAPSHOT
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion'
echo '##vso[task.setvariable variable=projectVersionStatic;isOutput=true]2.27.0'
echo $(Task1.projectVersionDynamic) ##Error message
echo $projectVersionDynamic ##Is empty
name: Task1
displayName: Task1 in JobOne of FirstStage
- job: SecondJob
dependsOn: FirstJob
variables:
DynVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionDynamic'] ]
StaVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionStatic'] ]
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'SecondJob'
echo $(DynVar) ##Is empty
echo $DynVar ##Is empty
echo $(StaVar) ##Prints 2.27.0
echo $StaVar ##Is empty
displayName: Task in JobTwo of FirstStage
Observation: projectVersionDynamic value does not get populated and is not available in the same task or subsequent tasks within or in later jobs / stages. However, the static variable gets populated in projectVersionStatic without any issues.
Is it possible to set dynamic values for user-defined variables in azure pipelines or is it that I am doing something wrong? I see an example here under the Stages section where it seems to be working.
Variables in Azure Pipelines can be really tricky sometimes. The documentation isn't always crystal-clear on how it's supposed to work. Looking at your example, a cpuple of observations:
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion' - your single quotes need to be double quotes to expand the value of $pomVersion into the echo statement
(and this is where things get fuzzy): The purpose of task.setvariable is to communicate values to other tasks or jobs. From the documentation: "This doesn't update the environment variables, but it does make the new variable available to downstream steps within the same job."
$variable syntax won't work because the task.setVariable doesn't inject into the running environment - rather, it's a signal to the pipeline to capture the output and store it away for later use.
$(variable) syntax won't work because it's expanded just before the job starts, so it's too late to capture here.
If you use my suggestion in point 1) about double-quoting the task.setVariable, you should see the value available in the second job.
For anybody having a use case where variables defined in one azure pipeline stage needs to be used in another stage, this is how it can be done:
Following snipett of azure pipeline that defines a variable and make it available to be used in a job in the same stage and in a job in another stage:
stages:
- stage: stage1
jobs:
- job: job1
steps:
- task: "AzureCLI#2"
name: step1
inputs:
inlineScript: |
my_variable=a-value
echo "##vso[task.setvariable variable=var_name;isOutput=true]$my_variable"
- job: job2
variables:
stage1_var: $[ dependencies.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage1_var)
- stage: stage2
jobs:
- job: job1
variables:
stage2_var: $[ stageDependencies.stage1.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage2_var)