Accessing variables in Azure loops using templates - azure-devops

I want to loop through the pipeline artifacts and pass them as variables to a task.
Followed this answer here, but no luck:
https://stackoverflow.com/a/59451690/5436341
Have this powershell script in the build stage to get the artifact names and store them in a variable:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Fetching value files"
cd $(Build.ArtifactStagingDirectory)\MSI
$a=dir -Filter "*.msi"
$List = $a | foreach {$_}
Write-Host $List
$d = '"{0}"' -f ($List -join '","')
Write-Host $d
Write-Host "##vso[task.setvariable variable=MSINames;isOutput=true]$d"
name: getMSINames
And passing them as parameters to a template from another stage as below:
- stage: deployPoolsStage
displayName: Deploy Pools
dependsOn:
- Build
jobs:
- job: CloudTest_AgentBased_Job
displayName: 'CloudTest AgentBased Job'
timeoutInMinutes: 120
variables:
MSIFiles: $[dependencies.Build.outputs['getMSINames.MSINames']]
steps:
- template: TestPipeline.yml
parameters:
files : $(MSIFiles)
Now, my template looks like this:
parameters:
files : []
steps:
- ${{ each filename in parameters.files }}:
- task: SomeTask
inputs:
Properties: worker:VsTestVersion=V150;worker:MSIFile=${{ filename }}
displayName: 'tests'
Now this is failing with an error saying: "Expected a sequence or mapping. Actual value '$(MSIFiles)'". It's the same error even without using the template and directly accessing the variable in the original yml file.
Please let me know of a way to loop through my pipeline artifacts and pass them to my task.

You are exporting the variable MSINames from a task in Build stage that needs to be inside a job (say BuildJob). Also, you are trying to access the exported variable from another stage but inside a job. Thus, you should use the format to read variable from another stage but within a job. What you have used is a wrong format in deployPoolsStage stage. Try correcting the format as below inside CloudTest_AgentBased_Job job,
variables:
MSIFiles: $[stageDependencies.Build.BuildJob.outputs['getMSINames.MSINames']]
NOTE: I have assumed that the getMSINames task is defined inside the BuildJob job under Build stage according to what you have provided.
See docs related to this here.

Related

how to get and set array inside the ADO pipeline Powershell from variable?

I have below code in yaml file:
trigger:
branches:
include:
- maventest
pool:
vmImage: ubuntu-latest
steps:
- task: PowerShell#2
inputs:
filePath: './1.ps1'
Below is 1.ps1 file
$user1=#($(user))
Write-Host $(user)
write-host $user1[0]
As I can pass the value of variable inside while run pipeline but user is not getting from variable.
I had also used the $(env:user) but it didn't get value from variable.
From your YAML sample, the cause of the issue is that the pipeline variable is not able to directly expand in PowerShell file.
To solve this issue, you can pass the pipeline variable to ps file via argument of PowerShell task.
Here is the example:
Ps file sample:
param($user)
$user1=#($user)
Write-Host $user
write-host $user1[0]
Pipeline sample:
steps:
- task: PowerShell#2
inputs:
filePath: './1.ps1'
arguments: '-user $(user)'
Result:

Evaluating Azure Devops Expressions within Powershell

I want to be able to invoke the Counter expression within a template but I am unsure how to do so; my current template yml file looks like this:
parameters:
- name: major
type: string
default: '1'
- name: minor
type: string
default: '0'
steps:
- task: PowerShell#2
displayName: Set NuGet package version
inputs:
targetType: 'inline'
script: |
$isMain = ('$(Build.SourceBranch)' -eq 'refs/heads/main')
$minor = ${{ parameters.minor }}
$revision1 = $[counter($minor, 1)]
Write-Host $revision
exit 0
But I get:
The term '$[counter' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
I am guessing there is way I can invoke the function via Powershell.
The reason for me wanting to do this in the template as opposed to passing it in as a parameter from the consumer of the template is because seemingly parameters are not evaluated when passed into a template.
Evaluating Azure Devops Expressions within Powershell
Just as Daniel said that:
The powershell script you're trying to run is getting finalized
during compile time. This line: $revision1 = $[counter($minor, 1)]
cannot possibly work. You are trying to take the results of a
powershell script expression and use it in a YAML function.
That is reason why it is not work for you.
And personally think you can go the easier way, just define the variable in the main YAML file:
azure-pipelines.yml:
variables:
- name: minor
value: 0
- name: revision1
value: $[counter(variables['minor'], 0)]
stages:
- stage: Run
jobs:
- job:
steps:
- template: Template.yml
And we could use the the Counter in the template directly:
Template.yml:
steps:
- task: PowerShell#2
displayName: Set NuGet package version
inputs:
targetType: 'inline'
script: |
Write-Host '$(revision1)'
The test result:

Set Azure DevOps pipeline variable to array of values

I am trying to run a set of tests across a set of ADO builds.
I can retrieve the list of ADO builds using PowerShell. However, once I have the list, I need to export that to an ADO variable and then iterate across the values.
I've seen how to export values from Powershell to ADO using logging, but that appears to export the value as a string, not a list.
Is there a way to export variables so that I could iterate across them; e.g., using ${{ each foo in exportedVars }}?
First, for the usage you mentioned:
${{ each foo in exportedVars }}
This is a compile-time usage, it is expanded at the beginning, and you can't get the variables generated by the pipeline runtime through it.
Second, the pipeline can output variables through the logging command, but the variables set in this way can only be strings. This is by design, and the documentation has said it very clearly:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-variables-in-scripts
All variables set by this method are treated as strings.
It is not difficult to parse a string and put it into an array, just use the function(split()) that comes with the string type to split and restore.
Here is an example:
trigger:
- none
# 1
stages:
- stage: s1
displayName: setvars
jobs:
- job: testJob
steps:
- task: PowerShell#2
name: setvar
inputs:
targetType: 'inline'
script: |
# logic here. For example you get the vars and put it into this format:
$testvars = "testvar1,testvar2,testvar3"
Write-Host "##vso[task.setvariable variable=outputvars;isOutput=true]$testvars"
# 2
- stage: s2
displayName: getvars
dependsOn: s1
variables:
vars: $[ stageDependencies.s1.testJob.outputs['setvar.outputvars'] ]
jobs:
- job:
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$varsArr = "$(vars)".Split(',')
foreach ($var in $varsArr)
{
Write-Host "$var`r`n"
}
Result:

Capturing build date in deployment pipleine

I'm trying to capture the build date of an artifact to be logged elsewhere. Join me on my convoluted journey to solve this.
I know we have these handy variables
$(Build.SourceVersion)
$(Build.BuildNumber)
EDIT: These are not as handy as I thought. These are just the identifers for the deploy pipeline, not the original build pipeline that generated the artefact. So I can repeatedly deploy the same build / artefact, and these numbers will continue to increment, having no relevance to what I built - I'm not interested in that.
But there is no build date. I know it can be derived from the BuildNumber but it seems over the top to call a REST API to get that info.
So in my build pipeline I am writing Get-Date to a file then publishing that as an artefact
- powershell: (Get-Date).ToString("yyyy-MM-dd HH:mm:ss") | Out-File -FilePath $(Build.ArtifactStagingDirectory)\BuildDt.txt
Then I pick that up in the deploy pipeline and save to a variable using the kludgy Write-Host method
- stage: DownloadDBArtifacts
displayName: Download DB Artifacts
dependsOn: []
jobs:
- job: GetArtefacts
displayName: Get Artefacts
steps:
- download: DBBuild
- task: PowerShell#2
displayName: Get Build timestamp
name: GetBuildDt
inputs:
targetType: inline
script: |
$BuildDt = Get-Content -Path $(Pipeline.Workspace)\DBBuild\drop\BuildDt.txt
Write-Host "##vso[task.setvariable variable=BuildDt;isoutput=true]$BuildDt"
Write-Host "##[debug]Artifact Creation Date: $BuildDt"
This is done in stage DownloadDBArtifacts
Now I need to use it in a later stage, that is also in a child YAML template
I beleive this is the syntax for extracting the variable:
stageDependencies.DownloadDBArtifacts.GetArtefacts.outputs['GetBuildDt.BuildDt']
I'm having difficulty getting this recognised in later stages. Here is a subsequent stage that tries to capture the value based on examples from here:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#use-outputs-in-a-different-stage
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#job-to-job-dependencies-across-stages
- stage: DeployDBtoTST
displayName: Deploy DB to TST
dependsOn: DownloadDBArtifacts
variables:
- group: vgTST
jobs:
- deployment: DeployDBtoTST
displayName: Deploy DB to TST
environment: TST Environment
variables:
BuildDt: $[ stageDependencies.DownloadDBArtifacts.GetArtefacts.outputs['GetBuildDt.BuildDt'] ]
strategy:
runOnce:
deploy:
steps:
- powershell: |
Write-Host "var: $(BuildDt)"
however the value is not being passed through as the final powershell step just produces this output:
var:
Capturing build date in deployment pipleine
I could reproduce this issue with your YAML sample.
To resolve this issue, please update your DownloadDBArtifacts by following code:
- stage: DownloadDBArtifacts
displayName: Download DB Artifacts
dependsOn: []
jobs:
- job: GetArtefacts
displayName: Get Artefacts
steps:
- download: DBBuild
- task: InlinePowershell#1
displayName: 'Get Artefacts'
inputs:
Script: |
$BuildDt = Get-Content -Path $(Pipeline.Workspace)\DBBuild\drop\BuildDt.txt
Write-Host "##vso[task.setvariable variable=BuildDt;isOutput=true]$BuildDt"
name: GetBuildDt
The test result:
Update:
Sorry, I tried changing the DownloadDBArtifacts as you mentioned above
andit made no difference.
You have a slight letter error in your code that is causing the issue.
One is name: GetBuiltDt and another is outputs['GetBuildDt.BuildDt'] ]. The Built should be Build:

Azure YAML Get variable from a job run in a previous stage

I am creating YAML pipeline in Azure DevOps that consists of two stages.
The first stage (Prerequisites) is responsible for reading the git commit and creates a comma separated variable containing the list of services that has been affected by the commit.
The second stage (Build) is responsible for building and unit testing the project. This Stage consists of many templates, one for each Service. In the template script, the job will check if the relevant Service in in the variable created in the previous stage. If the job finds the Service it will continue to build and test the service. However if it cannot find the service, it will skip that job.
Run.yml:
stages:
- stage: Prerequisites
jobs:
- job: SetBuildQueue
steps:
- task: powershell#2
name: SetBuildQueue
displayName: 'Set.Build.Queue'
inputs:
targetType: inline
script: |
## ... PowerShell script to get changes - working as expected
Write-Host "Build Queue Auto: $global:buildQueueVariable"
Write-Host "##vso[task.setvariable variable=buildQueue;isOutput=true]$global:buildQueueVariable"
- stage: Build
jobs:
- job: StageInitialization
- template: Build.yml
parameters:
projectName: Service001
projectLocation: src/Service001
- template: Build.yml
parameters:
projectName: Service002
projectLocation: src/Service002
Build.yml:
parameters:
projectName: ''
projectLocation: ''
jobs:
- job:
displayName: '${{ parameters.projectName }} - Build'
dependsOn: SetBuildQueue
continueOnError: true
condition: and(succeeded(), contains(dependencies.SetBuildQueue.outputs['SetBuildQueue.buildQueue'], '${{ parameters.projectName }}'))
steps:
- task: NuGetToolInstaller#1
displayName: 'Install Nuget'
Issue:
When the first stages runs it will create a variable called buildQueue which is populated as seen in the console output of the PowerShell script task:
Service001 Changed
Build Queue Auto: Service001;
However when it gets to stage two and it tries to run the build template, when it checks the conditions it returns the following output:
Started: Today at 12:05 PM
Duration: 16m 7s
Evaluating: and(succeeded(), contains(dependencies['SetBuildQueue']['outputs']['SetBuildQueue.buildQueue'], 'STARS.API.Customer.Assessment'))
Expanded: and(True, contains(Null, 'service001'))
Result: False
So my question is how do I set the dependsOn and condition to get the information from the previous stage?
It because you want to access the variable in a different stage from where you defined them. currently, it's impossible, each stage it's a new instance of a fresh agent.
In this blog you can find a workaround that involves writing the variable to disk and then passing it as a file, leveraging pipeline artifacts.
To pass the variable FOO from a job to another one in a different stage:
Create a folder that will contain all variables you want to pass; any folder could work, but something like mkdir -p $(Pipeline.Workspace)/variables might be a good idea.
Write the contents of the variable to a file, for example echo "$FOO" > $(Pipeline.Workspace)/variables/FOO. Even though the name could be anything you’d like, giving the file the same name as the variable might be a good idea.
Publish the $(Pipeline.Workspace)/variables folder as a pipeline artifact named variables
In the second stage, download the variables pipeline artifact
Read each file into a variable, for example FOO=$(cat $(Pipeline.Workspace)/variables/FOO)
Expose the variable in the current job, just like we did in the first example: echo "##vso[task.setvariable variable=FOO]$FOO"
You can then access the variable by expanding it within Azure Pipelines ($(FOO)) or use it as an environmental variable inside a bash script ($FOO).