Azure Devops - Setting variable in YAML script to datetime - azure-devops

I'm trying to generate a name for a NuGet package in an Azure DevOps YAML pipeline.
The context is creating packages on each automated build with the name of the package, the name of the branch, the date and the incrementing revision number.
packageName-branchName-year-month-day-revision
This way when a new feature branch is created it will generate a unique package which can be used for testing in projects which need it.
I'm struggling to set a variable in the YAML file from environmental variables such as the date or revision number.
Using the ##vso command I'm able to set 'myVariable' to hello and print it out.
- script: |
echo '##vso[task.setvariable variable=myVariable]hello'
- script: |
echo my variable is $(myVariable)
When I try setting the variable from PowerShell as below I get the following error '#$dateStr' is not recognized as an internal or external command'.
# Create a variable
- script: |
#$dateStr = (Get-Date).ToString('yyyy-MM-dd')
echo '##vso[task.setvariable variable=myVariable]#$dateStr'
# Print the variable
- script: |
echo my variable is $(myVariable)
When I try to set a variable in the variables section of the YAML file as so.
variables:
solution: '**/*.sln'
foo: $(Date:yyyyMMdd)
- script: |
echo my variable is $(foo)
The variable is not interpolated and it outputs as.
'my variable is $(Date:yyyyMMdd)'
How do I create variables based on environmental variables such as $(rev) and $(Date)?

i dont think there is a built-in date variable, but for the powershell case you just need to drop # before variable and it has to be enclosed with " else powershell wont expand your variable
echo "##vso[task.setvariable variable=myVariable]$dateStr"

Thanks for the help.
This is how I solved the issue in the end. With non release branches I use build# + beta + branch name to generate a unique name for the nuget packages I'm creating. With release branches I just use the build #.
# Create version number for non release package(s) - 1.0.xxx-beta-develop for example
- powershell: |
[String]$buildNumber = $Env:BUILD_BUILDNUMBER.Substring($Env:BUILD_BUILDNUMBER.LastIndexOf('.') + 1)
[String]$branchName = $Env:BUILD_BUILDNUMBER.Substring(0, $Env:BUILD_BUILDNUMBER.LastIndexOf('.'))
Write-Host "##vso[task.setvariable variable=nugetVersion]1.0.$($buildNumber)-beta-$($branchName)"
displayName: 'Create beta version number for production nuget packages - run when in non release branch'
condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'release/'))
# Create version number for release package(s) - 1.0.xxx for example
- powershell: |
[String]$buildNumber = $Env:BUILD_BUILDNUMBER.Substring($Env:BUILD_BUILDNUMBER.LastIndexOf('.') + 1)
Write-Host "##vso[task.setvariable variable=nugetVersion]1.0.' + $($buildNumber)
displayName: 'Create version number for production nuget packages - run when in release branch'
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'release/'))
- task: NuGetCommand#2
inputs:
command: 'pack' # Options: restore, pack, push, custom
versioningScheme: byEnvVar
versionEnvVar: nugetVersion
packDestination: '$(Build.ArtifactStagingDirectory)\nuget'
packagesToPack: '**/Floww.*.nuspec'

using the -script is great for running commands that can be executed cross-platform like npm i
As the date is not a global variable you will need to create it. The issue is that depending on what build agent you are running on (window|mac|linx) you will have different ways to set that variable, see the documentation...
The following will allow you to use a $d variable like other variables
steps:
- bash: |
export d=$(date +%F)
echo "The Date is $d"
echo "##vso[task.setvariable variable=fileName]$d"
condition: eq( variables['Agent.OS'], 'Linux' )
- powershell: |
Set-Variable -Name d -Value (Get-Date).ToString('yyyy-MM-dd')
Write-Host "##vso[task.setvariable variable=fileName]$d"
condition: eq( variables['Agent.OS'], 'Windows_NT' )

Related

how can I get the agent/_work/[##] folder number in build script?

I am trying to access the folder number when running a build. How can I get this value as a $(someparameter.name) for use in the script?
I've scrubbed through the Pipelines predefined variables, and that working folder number is included in some of the values, but not JUST the number.
For example: ...Build\$(Agent.Name)\_work\47\s\... <<<< what contains the "47" value?
Please advise.
A
I am afraid that there is no out-of-box predefined variable can get the agent folder name.
To get the agent folder name, you can use the Pipeline variable: $(Pipeline.Workspace) to get the workspace path(e.g. /home/vsts/work/1). Then you can use the PowerShell script to get the folder number of the workspace path and set the number as Pipeline variable for next tasks.
For example:
steps:
- powershell: |
echo $(Pipeline.Workspace)
$foldernumber = "$(Pipeline.Workspace)".Split("\")[-1]
echo $foldernumber
echo "##vso[task.setvariable variable=foldernumber]$foldernumber"
displayName: 'PowerShell Script'
- powershell: |
echo $(foldernumber)
After much trial and error, this works to update a declared DevOps Build Pipeline variable using a PowerShell step in the build. It's pretty simple.
variables:
workDirectory: ''
steps:
- powershell: |
$env:workDirectory = "NewValueHere"
echo "##vso[task.setvariable variable=workDirectory]$($env:workDirectory)"
- later steps in script
$(workDirectory) has correct updated value

Accessing variables in Azure loops using templates

I want to loop through the pipeline artifacts and pass them as variables to a task.
Followed this answer here, but no luck:
https://stackoverflow.com/a/59451690/5436341
Have this powershell script in the build stage to get the artifact names and store them in a variable:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Fetching value files"
cd $(Build.ArtifactStagingDirectory)\MSI
$a=dir -Filter "*.msi"
$List = $a | foreach {$_}
Write-Host $List
$d = '"{0}"' -f ($List -join '","')
Write-Host $d
Write-Host "##vso[task.setvariable variable=MSINames;isOutput=true]$d"
name: getMSINames
And passing them as parameters to a template from another stage as below:
- stage: deployPoolsStage
displayName: Deploy Pools
dependsOn:
- Build
jobs:
- job: CloudTest_AgentBased_Job
displayName: 'CloudTest AgentBased Job'
timeoutInMinutes: 120
variables:
MSIFiles: $[dependencies.Build.outputs['getMSINames.MSINames']]
steps:
- template: TestPipeline.yml
parameters:
files : $(MSIFiles)
Now, my template looks like this:
parameters:
files : []
steps:
- ${{ each filename in parameters.files }}:
- task: SomeTask
inputs:
Properties: worker:VsTestVersion=V150;worker:MSIFile=${{ filename }}
displayName: 'tests'
Now this is failing with an error saying: "Expected a sequence or mapping. Actual value '$(MSIFiles)'". It's the same error even without using the template and directly accessing the variable in the original yml file.
Please let me know of a way to loop through my pipeline artifacts and pass them to my task.
You are exporting the variable MSINames from a task in Build stage that needs to be inside a job (say BuildJob). Also, you are trying to access the exported variable from another stage but inside a job. Thus, you should use the format to read variable from another stage but within a job. What you have used is a wrong format in deployPoolsStage stage. Try correcting the format as below inside CloudTest_AgentBased_Job job,
variables:
MSIFiles: $[stageDependencies.Build.BuildJob.outputs['getMSINames.MSINames']]
NOTE: I have assumed that the getMSINames task is defined inside the BuildJob job under Build stage according to what you have provided.
See docs related to this here.

Azure Pipelines - Output variable from python script file to pipeline variable

I've tried several articles and threads from Stackoverflow but can't seem to get anywhere. I am trying to take a variable from a .py file which is called in a YAML step and set that variable globally to be used.
In my .py file i have
print(f'##vso[task.setvariable variable=AMLPipelineId;isOutput=true]{pipelineId}')
Then in my YAML pipeline step i have
- task: AzurePowerShell#5
displayName: 'Run AML Pipeline'
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
name: AmlPipeline
azurePowerShellVersion: 'LatestVersion'
pwsh: true
Inline: |
$username = "$(ARM_CLIENT_ID)"
$password = "$(ARM_CLIENT_SECRET)"
$tenantId = "$(ARM_TENANT_ID)"
python $(Pipeline.Workspace)/AML_Pipeline/build_aml_pipeline.py --wsName $(wsName) --resourceGroup $(ResourceGroupName) --subscriptionId $(subId)
$MLPipelineId = $AmlPipeline.AMLPipelineId
But it seems like this variable is empty. I know there are other ways of using the "set variable" but this is my latest attempt i.e. something like print('##vso[task.setvariable variable=version;]%s' % (version))
My current approach i followed: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
You don't need isOutput=true - that's only needed for referencing variables between different jobs or stages.
"You cannot use the variable in the step that it is defined." - split that script into two steps: one that runs your .py file, second one that uses the newly defined variable.
I used print('##vso[task.setvariable variable=<Variable-in-Pipeline]+<output-variable>')
Variable-in-Pipeline // the given name should be used in Azure Devops pipeline and should be added to pipeline variables as an empty string
A very minimal example for everyone struggling with this. The documentation is kind of lacking on this for my taste. As #qbik said, dont set and use the variable in the same step, make it seperate steps.
set_variable.py
if __name__ == '__main__':
# set name of the variable
name = 'COLOR'
# set value of the variable
value = 'red'
# set variable
print(f'##vso[task.setvariable variable={name};]{value}')
azure-pipelines.yml
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.9'
displayName: 'Use Python 3.9'
# run the script to set the variable
- task: PythonScript#0
inputs:
scriptSource: filePath
scriptPath: set_variable.py
# now you can use the variable in the next step
- bash: echo my favorite color is: $(COLOR)
Now you can technically do all kinds of cool stuff in python, then set, and reference the variables in the following steps. In my case I have to extract specific package version numbers from a JSON/YAML file based on an id that is set earlier in the pipeline and parse the information as an args for a docker build. Hope that helps other people stumbling across this answer looking for a minimal working example :)

Set bash command output into an azure yml variable

I'm working with Azure DevOps and I need to set the return of bash commands into some variables, for example I have the following:
variables:
VERSION: 7.2 # works fine
FILE_VERSION: ${{cat public/VERSION}} # syntax error
I tried some some variations from ${{}} without success and I could not found the right syntax for it, but I think it must be possible.
You should use a bash step for that.
Like this:
steps:
- bash: |
echo "##vso[task.setvariable variable=FILE_VERSION]$(cat public/VERSION)"
You need to do this in two steps
STEP 1: Set static value first
variables:
VERSION: 7.2
STEP 2: Create step to calculate the value
Note this will only be available for subsequent steps
- bash: |
FILE_VERSION="$(cat public/VERSION)"
echo "##vso[task.setvariable variable=FILE_VERSION]$FILE_VERSION"
Resources:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
We shouldn't use cat command when defining variables. The command should be located in tasks/steps.
According to your description, I assume you're trying to pass the content of public/7.2 to FILE_VERSION variable. Here's my test:
1.Azure Devops Git repo:
2.Define the VERSION variable:
variables:
VERSION: 7.2
Run the cat command and set job-scoped variable:
- bash: |
FILE_VERSION=$(cat public/$(VERSION))
echo "##vso[task.setvariable variable=FILE_VERSION]$FILE_VERSION"
The whole yaml:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
variables:
VERSION: 7.2
steps:
- bash: |
FILE_VERSION=$(cat public/$(VERSION))
echo "##vso[task.setvariable variable=FILE_VERSION]$FILE_VERSION"
#Use second bash task to test the variable.
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo $(FILE_VERSION)

Is there a short 7-digit version of $(SourceVersion) in Azure Devops?

I am trying to set our build names to a format of...
$(BuildDefinitionName)_$(versionMajor).$(versionMinor).$(versionPatch)+$(SourceBranchName).$(SourceVersion) e.g.
OurBigLibraryCI_1.2.3+master.10bbc577
However I coudn't find any predefined variable holding the "short" (7-digit) version of the commit hash. $(SourceVersion) holds the full SHA-1 hash.
How would one shorten that in yaml based pipeline?
You can use traditional command substitution via backticks to obtain the short git hash (SHA-1), assuming that the code is being checked out in $(Build.SourcesDirectory):
- bash: |
short_hash=`git rev-parse --short=7 HEAD` ## At least 7 digits, more if needed for uniqueness
echo ""
echo "Full git hash: $(Build.SourceVersion)"
echo "Short git hash: $short_hash"
echo "##vso[task.setvariable variable=short_hash]$short_hash" ## Store variable for subsequent steps
workingDirectory: $(Build.SourcesDirectory)
displayName: Get short git hash
Output:
Full git hash: f8d63b1aaa20cf348a9b5fc6477ac80ed23d5ca0
Short git hash: f8d63b1
The following steps in the pipeline can then use the short hash via the variable $(short_hash).
(This is better than manually trimming down the full git hash to seven characters, since this will add extra digits if needed to uniquely identify the commit, see https://stackoverflow.com/a/21015031/1447415.)
Update: Improved version
The following improved version checks that the git hashes match (that the full hash starts with the short hash) and fails the step otherwise:
- bash: |
short_hash=`git rev-parse --short=7 HEAD`
echo ""
echo "Full git hash: $(Build.SourceVersion)"
echo "Short git hash: $short_hash"
echo ""
## Fail step if full hash does not start with short hash
if [[ $(Build.SourceVersion) != $short_hash* ]]; then
echo "--> Hashes do not match! Aborting."
exit 1
fi
echo "--> Hashes match. Storing short hash for subsequent steps."
## Store variable for subsequent steps
echo "##vso[task.setvariable variable=short_hash]$short_hash"
workingDirectory: $(Build.SourcesDirectory)
displayName: Get short git hash
- script: |
echo $sourceVersion
commitHash=${sourceVersion:0:7}
echo $commitHash
echo "##vso[task.setvariable variable=commitHash]$commitHash" ## Set variable for using in other tasks.
env: { sourceVersion: $(Build.SourceVersion) }
displayName: Git Hash 7-digit
workingDirectory: #workingDirectory
- task: Docker#2
displayName: Build & Push image
inputs:
command: 'buildAndPush'
containerRegistry: '$(myRegistry)'
repository: $(myContainerRepository)
Dockerfile: $(myDockerfile)
buildContext: '$(myBuildContext)'
tags: $(commitHash) ## The variable was defined above.
Here's example for vmImage: "ubuntu-latest". Step:
Split 7-characters from Pre-defined GitHash
Assign it to Pipeline variable. Don't confuse $(azure_pipeline_variable) with ${bash_shell_variable} or $bash_shell_variable.
Use it by $(commitHash)
Read more:
Assign variable in Azure pipeline
Using script in Azure pipeline
How would one shorten that in yaml based pipeline?
There is no out of box variable to get the 7-digit version of $(SourceVersion) in Azure Devops. Because the ShortSha is 8-digit version.
So, to resolve this issue, just like #4c74356b41 said, we have to use bash\powershell script to split long sha into short sha.
You can check my following sample for some more details:
steps:
- script: |
echo $(Build.SourceVersion)
set TestVar=$(Build.SourceVersion)
set MyCustomVar= %TestVar:~0,7%
echo %MyCustomVar%
displayName: 'Command Line Script'
The result:
========================== Starting Command Output ===========================
##[command]"C:\WINDOWS\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "C:\VS2017Agent\_work\_temp\be5f6293-77d8-41b7-a537-49e3b2e7bc6c.cmd""
cb124539c4cb7f19dc8e50e1b021f93c5ffaf226
cb12453
##[section]Finishing: Command Line Script
So, we could get the 7-digit version of $(SourceVersion) is cb12453.
Hope this helps.
you could use gitversion for that, it would expose shortsha under $(GitVersion.ShortSha) variable after you run the gitversion task.
on the other hand shortsha is just first 7 characters of a real sha, so you can just use some sort of bash\powershell script to split long sha into short sha
In Git, what is the difference between long and short hashes?
- task: gittools.gittools.setup-gitversion-task.gitversion/setup#0
displayName: gitversion/setup
inputs:
versionSpec: 5.x
- task: gittools.gittools.execute-gitversion-task.gitversion/execute#0
displayName: gitversion/execute
replacement for the deprecated extension
As the docs say you can't use $(Build.SourceVersion) variable directly in the build name, nor is there an in-built substring Expression
But you can update the Build Number variable dynamically:
pool:
vmImage: 'ubuntu-latest' #'windows-latest' - works on either
steps:
- checkout: none
- powershell: |
$shortHash = "$(Build.SourceVersion)".Substring(0, 7)
Write-Host "##vso[task.setvariable variable=shortHash]$shortHash"
displayName: Set Short Hash
- powershell: Write-Host $(shortHash)
displayName: Display Short Hash
- powershell: Write-Host "##vso[build.updatebuildnumber]$(Build.DefinitionName)_AnyOtherValues_$(shortHash)"
displayName: Update Build Number
- powershell: Write-Host $(Build.BuildNumber)
displayName: Display Build Number