Azure DevOps Variable Groups Password not working - azure-devops

I'm having userName and passWord field in my Azure DevOps Variable group, and I would like to pass the userName and passWord to my python script as a command line argument, I'm able to get the userName without any issue but passWord is not coming when I pass it from my pipeline script. I tried printing using echo - its printing empty - not evening printing masked value like this '*****'
Please find my pipeline code below:
trigger:
- none
variables:
- group: myVariableGrp
pool:
name: myAgents
stages:
- stage: Testing
jobs:
- job: Test
displayName: Test script
steps:
- task: Bash#3
displayName: Test variable
inputs:
targetType: inline
script: |
echo "hello variables"
echo $USER_NAME
echo $MY_PASSWORD
- task: CmdLine#2
displayName: Test Python script
inputs:
script: python test.py $USER_NAME $MY_PASSWORD
test.py
import sys
# total arguments
#n = len(sys.argv)
#print("Total arguments passed:", n)
print("UserName:", sys.argv[1])
print("Password:", sys.argv[2])
I'm getting the below error:
Traceback (most recent call last):
File "test.py", line 12, in <module>
print("Password:", sys.argv[2])
IndexError: list index out of range
##[error]Bash exited with code '1'.
Finishing: Test Python script

Secret variables are not automatically exposed as env variables. You have to explicitly map them: Reference secret variables in variable groups.
In this case, it would be:
- task: Bash#3
displayName: Test variable
inputs:
targetType: inline
env:
MAPPED_PASSWORD: $(MY_PASSWORD)
script: |
echo "hello variables"
echo $USER_NAME
echo $MAPPED_PASSWORD

Related

Runtime Variable in Yaml Azure DevOps

If have created the Pipeline for One of our CRM Solution in Yaml
I have Created variables SolutionExit,SolutionName,SolutionVersion
#back-up.yaml
parameters:
- name: Solution
displayName: Solution Need to be backed up
type: object
default: [solution1,Solution2]
trigger:
- none
variables:
SolutionExit: ''
SolutionVersion: ''
SolutionName: ''
pool:
vmImage: windows-2019
steps:
- task: MSCRMToolInstaller#12
inputs:
nugetFeed: 'official'
psFeed: 'official'
- ${{each value in parameters.Solution}}:
- task: MSCRMGetSolution#12
inputs:
crmConnectionString: '$(ConnectStr)'
solutionName: '${{value}}'
existsVariableName: 'SolutionExit' #replace existing variable with output value True or False
versionVariableName: 'SolutionVersion' #replace existing variable with output value
displayVariableName: 'SolutionName' #replace existing variable with output value
- template: template/validate-solution.yml
parameters:
SolutionExit: $(SolutionExit)
#validate-solution.yml
parameters:
SolutionExit: ""
steps:
- ${{if eq(parameters.SolutionExit, 'True')}}:
- script: echo "Solution exist $(SolutionExit),$(SolutionVersion),$(SolutionName),${{parameters.SolutionExit}}"
- ${{else}}:
- bash: echo "solution doesnt exit $(SolutionExit),$(SolutionVersion),$(SolutionName),${{parameters.SolutionExit}}"
But runtime parameters are not accepted If condition never satisfies. How to pass the runtime parameter
This Is the result for both cases
If the solution available
Starting: Bash
==============================================================================
Task : Bash
Description : Run a Bash script on macOS, Linux, or Windows
Version : 3.214.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
Script contents:
echo "solution doesnt exit True,1.0.0.11,Solution1,True"
"C:\Program Files\Git\bin\bash.exe" -c pwd
/d/a/_temp
========================== Starting Command Output ===========================
"C:\Program Files\Git\bin\bash.exe" /d/a/_temp/237f7dc2-debc-4c47-ba88-41a6a97c49d9.sh
solution doesnt exit True,1.0.0.11,soultion1,True
Finishing: Bash
if solution not available
Starting: Bash
==============================================================================
Task : Bash
Description : Run a Bash script on macOS, Linux, or Windows
Version : 3.214.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
==============================================================================
Generating script.
Script contents:
echo "solution doesnt exit False,,,False"
"C:\Program Files\Git\bin\bash.exe" -c pwd
/d/a/_temp
========================== Starting Command Output ===========================
"C:\Program Files\Git\bin\bash.exe" /d/a/_temp/49e5bea1-b380-4a74-a0ce-b89d0dfb3a07.sh
solution doesnt exit False,,,False
Finishing: Bash
How to fix this issue. There is compile time and Runtime variables,as per microsoft I tried ${{if eq(variables['SolutionExit'], 'True')}}: this too didnt work. We cannot use the macro variable in conditions like $(SolutionExit)
From your YAML definition, the variable will generate during the process of Pipeline Run.
The Variable should be Runtime variable. It will be expand at runtime. But the If expression and parameters will be expand at Compile time.
In this case, If expression will not work.
To solve this issue, you need to change to use Condition to replace the If expression.
At the same time, you don't need to use parameters to pass the variable value. You can directly use the variable in the condition.
For example:
Main YAML :
parameters:
- name: Solution
displayName: Solution Need to be backed up
type: object
default: [solution1,Solution2]
trigger:
- none
variables:
SolutionExit: ''
SolutionVersion: ''
SolutionName: ''
pool:
vmImage: windows-2019
steps:
- task: MSCRMToolInstaller#12
inputs:
nugetFeed: 'official'
psFeed: 'official'
- ${{each value in parameters.Solution}}:
- task: MSCRMGetSolution#12
inputs:
crmConnectionString: '$(ConnectStr)'
solutionName: '${{value}}'
existsVariableName: 'SolutionExit' #replace existing variable with output value True or False
versionVariableName: 'SolutionVersion' #replace existing variable with output value
displayVariableName: 'SolutionName' #replace existing variable with output value
- template: template/validate-solution.yml
Template YAML:
steps:
- script: echo "Solution exist $(SolutionExit),$(SolutionVersion),$(SolutionName),${{parameters.SolutionExit}}"
condition: eq(variables['SolutionExit'], 'true')
- bash: echo "solution doesnt exit $(SolutionExit),$(SolutionVersion),$(SolutionName),${{parameters.SolutionExit}}"
condition: ne(variables['SolutionExit'], 'true')

Pass Variable Group as Dictionary To Python Script

I have a variable group that i'm using from a python script. Something like this:
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
print('variableInVariableGroup: $(variableInVariableGroup)')
I'd like to write my script so that I can iterate over the variable group without explicitly referencing individual variables. Is there a way to feed in the entire variable group to the script as a dictionary or something similar?
You could not do that directly, for the workaround is to get the vars via azure cli and set with task variable, then get them in the python script task.
Something like below:
# 'Allow scripts to access the OAuth token' was selected in pipeline. Add the following YAML to any steps requiring access:
# env:
# MY_ACCESS_TOKEN: $(System.AccessToken)
# Variable Group 'vargroup1' was defined in the Variables tab
resources:
repositories:
- repository: self
type: git
ref: refs/heads/testb2
jobs:
- job: Job_1
displayName: Agent job 1
pool:
vmImage: ubuntu-20.04
steps:
- checkout: self
persistCredentials: True
- task: PowerShell#2
name: TestRef
displayName: PowerShell Script
inputs:
targetType: inline
script: >-
echo $(System.AccessToken) | az devops login
$a=az pipelines variable-group variable list --org 'https://dev.azure.com/orgname/' --project testpro1 --group-id 3 --only-show-errors --output json
echo "$a"
echo "##vso[task.setvariable variable=allvars;isOutput=true]$a"
- task: PythonScript#0
displayName: Run a Python script
inputs:
scriptSource: inline
script: "b=$(TestRef.allvars)\nprint(b)\n\n "
...

Dynamic variables not available in other stages of azure pipeline

I am new to azure pipelines and am currently experimenting with the passing variables to the later jobs. Here is the current snippet whereby I am trying to extract the project version from the pom file using maven help: evaluate plugin. The pomVersion variable is populated but is not available in the same step or later steps in the second job via the projectVersionDynamic variable.
stages:
- stage: FirstStage
jobs:
- job: FirstJob
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
pomVersion=`mvn help:evaluate -Dexpression=project.version -q -DforceStdout`
echo $pomVersion ##Prints semantic version 2.27.0-SNAPSHOT
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion'
echo '##vso[task.setvariable variable=projectVersionStatic;isOutput=true]2.27.0'
echo $(Task1.projectVersionDynamic) ##Error message
echo $projectVersionDynamic ##Is empty
name: Task1
displayName: Task1 in JobOne of FirstStage
- job: SecondJob
dependsOn: FirstJob
variables:
DynVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionDynamic'] ]
StaVar: $[ dependencies.FirstJob.outputs['Task1.projectVersionStatic'] ]
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: |
echo 'SecondJob'
echo $(DynVar) ##Is empty
echo $DynVar ##Is empty
echo $(StaVar) ##Prints 2.27.0
echo $StaVar ##Is empty
displayName: Task in JobTwo of FirstStage
Observation: projectVersionDynamic value does not get populated and is not available in the same task or subsequent tasks within or in later jobs / stages. However, the static variable gets populated in projectVersionStatic without any issues.
Is it possible to set dynamic values for user-defined variables in azure pipelines or is it that I am doing something wrong? I see an example here under the Stages section where it seems to be working.
Variables in Azure Pipelines can be really tricky sometimes. The documentation isn't always crystal-clear on how it's supposed to work. Looking at your example, a cpuple of observations:
echo '##vso[task.setvariable variable=projectVersionDynamic;isOutput=true]$pomVersion' - your single quotes need to be double quotes to expand the value of $pomVersion into the echo statement
(and this is where things get fuzzy): The purpose of task.setvariable is to communicate values to other tasks or jobs. From the documentation: "This doesn't update the environment variables, but it does make the new variable available to downstream steps within the same job."
$variable syntax won't work because the task.setVariable doesn't inject into the running environment - rather, it's a signal to the pipeline to capture the output and store it away for later use.
$(variable) syntax won't work because it's expanded just before the job starts, so it's too late to capture here.
If you use my suggestion in point 1) about double-quoting the task.setVariable, you should see the value available in the second job.
For anybody having a use case where variables defined in one azure pipeline stage needs to be used in another stage, this is how it can be done:
Following snipett of azure pipeline that defines a variable and make it available to be used in a job in the same stage and in a job in another stage:
stages:
- stage: stage1
jobs:
- job: job1
steps:
- task: "AzureCLI#2"
name: step1
inputs:
inlineScript: |
my_variable=a-value
echo "##vso[task.setvariable variable=var_name;isOutput=true]$my_variable"
- job: job2
variables:
stage1_var: $[ dependencies.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage1_var)
- stage: stage2
jobs:
- job: job1
variables:
stage2_var: $[ stageDependencies.stage1.job1.outputs['step1.var_name'] ]
steps:
- bash: |
echo "print variable value"
echo $(stage2_var)

azure pipeline variable created in powershell script not displaying value

In an Azure Pipeline, I want to set a variable in a Powershell script and then use that variable in a later build step. The variable doesn't print out a value.
In my Powrshell script I hardcode a value for testing purposes:
$packageFolder = 'dotnet/TestPackage/'
##vso[task.setvariable variable=changedPackage;isOutput=true]$packageFolder
Here is my YAML file:
steps:
- task: PowerShell#2
displayName: 'Detect Subfolder Changes'
name: setvarStep
inputs:
targetType: 'filePath'
filePath: $(System.DefaultWorkingDirectory)\detectchanges.ps1
failOnStderr: true
- script: echo "$(setvarStep.changedPackage)"
- task: DotNetCoreCLI#2
displayName: 'dotnet build'
condition: ne('$(setvarStep.changedPackage)', '')
inputs:
projects: '$(setvarStep.changedPackage)'
When the pipeline gets to the script step it just prints out this:
Script contents:
echo "$(setvarStep.changedPackage)"
I don't think the variable is actually set. When the pipeline gets to the dotnet build step it throws an error: ##[error]Project file(s) matching the specified pattern were not found.
While having debugging turned on I noticed this in the dotnet build step logs: ##[debug]findPath: 'C:\agents\librarywin-1\_work\2\s\$(setvarStep.changedPackage)'
I think my issue is the pipeline variable syntax is wrong even though I'm following the example on Microsoft's website. I can't figure it out.
You need double quote the logging command:
detectchanges.ps1:
$packageFolder = 'dotnet/TestPackage/'
"##vso[task.setvariable variable=changedPackage;isOutput=true]$packageFolder"
YAML:
steps:
- task: PowerShell#2
displayName: 'Detect Subfolder Changes'
name: setvarStep
inputs:
targetType: 'filePath'
filePath: $(System.DefaultWorkingDirectory)\detectchanges.ps1
failOnStderr: true
- script: echo "$(setvarStep.changedPackage)"
You should have Write-Host
$packageFolder = 'dotnet/TestPackage/'
Write-Host ##vso[task.setvariable variable=changedPackage;isOutput=true]$packageFolder
like here

Access build pipeline output variable in Environment Approval / Check

I am trying to build an automatic Environment Approval with Invoke Azure Function. I need to pass a variable that is created in a previous build stage. I have added the variable to the function body, but the variable is not evaluated:
Task that creates the output variable (in previous stage)
- task: CmdLine#2
inputs:
script: |
# Debug output (value visible in logs)
pulumi stack output resourceGroupName
echo "##vso[task.setvariable variable=resourceGroupName;isoutput=true]$(pulumi stack output resourceGroupName)"
workingDirectory: 'infra'
Body for the Azure function:
{ "ResourceGroup": "$(resourceGroupName)" }
Log:
2020-03-26T15:57:01.2590351Z POST https://azure-function-url
Request body: {
"ResourceGroup": "$(resourceGroupName)"
}
Added the variable to the function body, but the variable is not
evaluated
This is a expected action. Here what you are doing is share variable across stages, which does not supported directly until now.
Output variable just used to share values between steps instead of stages.
Work around:
If you want to use the generated variable in next stage, a workaround you can consider to use is writing the variable to a file, leveraging pipeline artifacts.
Sample steps:
Here I will pass one variable which name is resourceGroupName to next stage.
1) Create a folder which will contain the variables you want to pass
mkdir -p $(Pipeline.Workspace)/variables
2) Write the contents of the variable to file StageUsed:
echo "$resourceGroupName" > $(Pipeline.Workspace)/variables/StageUsed
3) In next stage, add one job before your InvokeAzureFunction job. Download the variables pipeline artifact that previous stage published.
4) Transfer each file into a variable:
resourceGroupName=$(cat $(Pipeline.Workspace)/variables/StageUsed)
5) Make the variable exposed in the current job, and set it reference name as Out:
echo "##vso[task.setvariable variable=resourceGroupName;isoutput=true]$resourceGroupName"
6) Now, you can access the variable in your InvokeAzureFunction job by calling dependencies.secondjob.outputs['output.resourceGroupName']
Sample Script:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: FirstStage
jobs:
- job: firstjob
pool:
vmImage: 'Ubuntu-16.04'
steps:
- bash: |
resourceGroupName="value"
mkdir -p $(Pipeline.Workspace)/variables
echo "$resourceGroupName" > $(Pipeline.Workspace)/variables/resourceGroupName
- publish: $(Pipeline.Workspace)/variables
artifact: variables
- stage: SecondStage
jobs:
- job: secondjob
pool:
vmImage: 'Ubuntu-16.04'
steps:
- download: current
artifact: variables
- bash: |
resourceGroupName=$(cat $(Pipeline.Workspace)/variables/resourceGroupName)
echo "##vso[task.setvariable variable=resourceGroupName;isoutput=true]$resourceGroupName"
name: output
- bash: |
echo "$(output.resourceGroupName)"
- job: ServerJob
dependsOn: secondjob
pool: server
variables:
resourceGroupName: $[dependencies.secondjob.outputs['output.resourceGroupName']]
steps:
- task: AzureFunction#1
inputs:
function:
method: 'POST'
body: '$(sdf)'
waitForCompletion: 'false'