I am trying to create a simple Release Pipeline that:
sets a value via a script
asks a user to manually check the value, and modify it if needed
goes on with the pipeline, using the value
What is the easiest way to achieve this functionality in Azure DevOps?
I have tried to define a Pipeline Variable called TestVariable, with initial value of 1, where I will store the value.
A Pipeline Variable seems a good option to me, because its value can be easily modified by the user, during manual check.
The problem is, I cannot set its value via a script (as per point 1. above).
I have tried to set its value to 2 via a Bash Script, but it does not work as you can see.
Here is the code, if you want to try yourself:
echo $(TestVariable)
echo "##vso[task.setvariable variable=TestVariable;]2"
echo $(TestVariable)
I searched thoroughly the Azure documentation but I could not find any solution. Is there any way to set a Pipeline Variable through a script, or to achieve the 3 above mentioned points with another strategy?
The method you used to update the value is correct.
Refer to this doc: Set variables in scripts
A script in your pipeline can define a variable so that it can be consumed by one of the subsequent steps in the pipeline.
It will not work on the current task, but the updated value can be used in the next tasks.
Here is an example:
steps:
- bash: |
echo $(TestVariable)
echo "##vso[task.setvariable variable=TestVariable;]2"
displayName: 'Bash Script'
- bash: |
echo $(TestVariable)
Result:
Related
I am trying to pass a variable "$a" from one step to another step in Azure DevOps. The $hash contains Name,IsReadOnly and Length as NoteProperty. I was successful in passing the variable from one step to next step as $b, but when i try to access NoteProperty of $b variable i am not able to access it
1.My question is it possible pass variable from one step to another step, How do i access the variables NoteProperty ??
2. Does Azure DevOps support passing complex objects from one step to another step ??
For your first question:
is it possible pass variable from one step to another step, How do i access the variables NoteProperty ??
Sure, you can pass variables from a step to the subsequent steps in the same job, and you even can pass the variables to the steps in the subsequent jobs.
To pass a variable value generated in a step to other subsequent steps, you can use the 'SetVariable' command in the step that generates the value.
If you want to share the variable value only to the subsequent steps in the same job, you can just set it as a general job-level variable using the following command.
Bash
echo "##vso[task.setvariable variable=<var_name>;]<var_value>"
or PowerShell
Write-Host "##vso[task.setvariable variable=<var_name>;]<var_value>"
With this way, in the subsequent steps, you can use the expression '$(var_name)' to access this variable.
After executing above command to set up the variable, like as other general pipeline variables, this variable will be automatically mapped as an environment variable. So, you also can use the expression '%VAR_NAME%' (CMD), '$env:VAR_NAME' (PowerShell) or '$VAR_NAME' (Bash) to access the variable.
If you want to share the variable value to the subsequent steps in the same job and the steps in the subsequent jobs, you can set it as an output variable using the following command.
Bash
echo "##vso[task.setvariable variable=<var_name>;isoutput=true]<var_value>"
or PowerShell
Write-Host "##vso[task.setvariable variable=<var_name>;isoutput=true]<var_value>"
With this way, to access this output variable in the subsequent steps of the same job, you can use the expression '$(step_name.var_name)'.
To access this output variable in the steps of the subsequent jobs, the subsequent jobs need to depend on (dependsOn) the job that contains the step to set up the output variable. And you also need to map this output variable as a job-level variable on each subsequent job, and then use this job-level variable to access the output value.
To view more details about variables in Azure Pipelines, you can see "Define variables".
A simple example as reference.
azure-pipelines.yml
jobs:
- job: JobA
displayName: 'Job A'
steps:
- bash: echo "##vso[task.setvariable variable=MyVar;isoutput=true]This is an output variable!"
name: set_output
displayName: 'Set up the output variable'
- bash: echo "set_output.MyVar = $(set_output.MyVar)"
displayName: 'Show the value of output variable'
- job: JobB
displayName: 'Job B'
dependsOn: JobA
variables:
output_from_JobA: $[ dependencies.JobA.outputs['set_output.MyVar'] ]
steps:
- bash: echo "output_from_JobA = $(output_from_JobA)"
displayName: 'Show the value of output variable from JobA'
Result:
For your second question:
Does Azure DevOps support passing complex objects from one step to another step ??
Normally, the value of the variable in Azure Pipelines should be string. To pass the content of an object as a variable in the pipeline, you can try to convert the object (such as JSON node, XML node, etc.) to be a string, then like as above example, pass this string as a variable. In the subsequent steps that need to access this object, after receiving the string variable, you can try to convert the string back to the object. This is the most common method I can think of.
I have an Azure Devops release pipeline (using the Classic Editor rather than yaml) and am having trouble setting a variable in an inline powershell script. The relevant part of the script is this:
Write-Host "Web is $web"
Write-Host "App is $app"
Write-Host "##vso[task.setvariable variable=MyWebEnv;]$web"
Write-Host "Set Web to $($MyWebEnv)"
Write-Host "##vso[task.setvariable variable=MyAppEnv;]$app"
Write-Host "Set App to $(MyAppEnv)"
The log files indicate that my $web and $app variables hold the correct values, however the setting is not working.
I've tried writing out the value in both the $($env:MyWebEnv) and $(MyWebEnv) formats and both formats indicate that the ADO variable has not been set correctly.
My variable is defined in the variables tab on the pipeline. It is set to Settable at queue time. I have tried with no value set in the definition and an initial value and I can see that the initial value does not get overwritten.
I've tried running this on both a self-hosted agent and an Azure Pipelines Hosted agent - same results. I have another release pipeline that does something similar without issues.
I'm trying to use the variable in a subsequent Stage.
I don't want to write the variable to a file or anything as I want to use the variable to control whether the stage even runs.
Edit:
Based on Levi's comment I tried adding isOutput=true and inspecting the variable in a subsequent step in the same stage - it's not visible even within a subsequent step:
=========================================================================
Edit 2:
I must have had a non visible character somewhere. I was composing the powershell in an external editor (just so I had more space) and pasting into the inline textbox in ADO - when I typed in the exact same thing I had into the editor it started working - sort of.
I can set the variable now and it's appearing in the task below, but what I really want is to use it in the next stages - in the IIS deployment group to control that IIS server A runs, but IIS server B doesn't. I'm attempting to do this in the Additional Options Run Expression.
I've tried
and(succeeded(), eq(variables['MyWebEnv'], 'myvalue')
or (where MyOutput. is the name given to my output reference when setting the variable)
and(succeeded(), eq(variables['MyOutpub.MyWebEnv'], 'myvalue')
or
and(succeeded(), eq(variables['MyOutpub_MyWebEnv'], 'myvalue')
To clarify, I'm doing this on the IIS Deployment Group itself, and not on any of the tasks within that group.
I keep seeing that the deployment is skipped due to:
Evaluating: eq(variables['MyOutput,MyWebEnv'], 'myvalue') Expanded: eq(Null, 'myvalue') Result: False
The variable you were trying to set in the powershell task will only take effect in the following tasks, which means if you echo out the variable in its own powershell task, it will not output the overriden value. See here.
To set a variable from a script, you use the task.setvariable logging command. This doesn't update the environment variables, but it does make the new variable available to downstream steps within the same job.
So you can only refer to it in the downstream tasks by wrapping it around $(). See below:
steps:
- powershell: |
Write-Host "##vso[task.setvariable variable=MyWebEnv]$web"
- powershell: |
Write-Host "Set Web to $(MyWebEnv)"
If you want to use it in a subsequent stage. You will have to make the variable to be an output variable by using isOutput=true. See here
- powershell: |
Write-Host "##vso[task.setvariable variable=MyWebEnv;isOutput=true]$web"
name: MyVariable
- powershell: echo "$(MyVariable.MyWebEnv)"
You should also give this powershell task a name in order to refer to it in the following stages. See here to learn how to refer to multiple stage output variables.
variables:
web: $[stageDependencies.stageName.jobName.outputs['stepName.variableName']]
You also need add dependency on this stage for the following stages.
See below example:
stages:
- stage: VariableStage
jobs:
- job: VariableJob
steps:
- powershell: |
$web = "---"
Write-Host "##vso[task.setvariable variable=MyWebEnv;isOutput=true]$web"
name: MyVariable
- stage:
dependsOn: VariableStage
variables:
web: $[stageDependencies.VariableStage.VariableJob.outputs['MyVariable.MyWebEnv']]
jobs:
- job:
steps:
- powershell: echo "$(web)"
As Levi Lu mentioned you can access pipeline variables in the next step. So if you set variables like this here:
$web ="some-web"
$app ="some-app"
Write-Host "##vso[task.setvariable variable=MyWebEnv;]$web"
Write-Host "##vso[task.setvariable variable=MyAppEnv;]$app"
then you can read them in this way:
Write-Host "Set Web to $(MyWebEnv)"
Write-Host "Set App to $(MyAppEnv)"
And if you want to access them via env variables you should use this syntax:
Write-Host "Set Web to $($env:MYWEBENV)"
Write-Host "Set App to $($env:MYAPPENV)"
Answer for EDIT 2:
What you actually need is output cross stage variables which is supported in YAML but not in classic pipelines. Please check this topics. You can overcome this but it requires an extra effort as it shown here. However, please remember that you can't publish an artifact from release pipeline.
Here you have docs for cross stage variables but as above, it works on YAML.
I have a pipeline in Azure DevOps somewhat like this:
parameters:
- name: Scenario
displayName: Scenario suite
type: string
default: 'Default'
variables:
Scenario: ${{ parameters.Scenario }}
...
steps:
- script: echo Scenario is $(Scenario)
And I'm executing the pipeline via the VSTS CLI like this:
vsts build queue ... --variables Scenario=Test
When I run my pipeline, it seems that the parameter default value overwrites my cmd line specified variable value and I get the step output Scenario is Default. I tried something like Scenario: $[coalesce(variables['Scenario'], ${{ parameters.Scenario }})] but I think I got the syntax wrong because that caused a parsing issue.
What would be the best way to only use the parameter value if the Scenario variable has not already been set?
What would be the best way to only use the parameter value if the
Scenario variable has not already been set?
Sorry but as I know your scenario is not supported by design. The Note here has stated that:
When you set a variable in the YAML file, don't define it in the web editor as settable at queue time. You can't currently change variables that are set in the YAML file at queue time. If you need a variable to be settable at queue time, don't set it in the YAML file.
The --variables switch in command can only be used to overwrite the variables which are marked as Settable at queue time. Since yaml pipeline doesn't support Settable variables by design, your --variables Scenario=Test won't actually be passed when queuing the yaml pipeline.
Here're my several tests to prove that:
1.Yaml pipeline which doesn't support Settable variable at Queue time:
pool:
vmImage: 'windows-latest'
variables:
Scenario: Test
steps:
- script: echo Scenario is $(Scenario)
I ran the command vsts build queue ... --variables Scenario=Test123, the pipeline run started but the output log would always be Scenario is Test instead of expected Scenario is Test123. It proves that it's not Pipeline parameter overwrites variable value, instead the --variables Scenario=xxx doesn't get passed cause yaml pipeline doesn't support Settable variables.
2.Create Classic UI build pipeline with pipeline variable Scenario:
Queuing it via command az pipelines build queue ... --variables Scenario=Test12345(It has the same function like vsts build queue ... --variables Scenario=Test) only gives this error:
Could not queue the build because there were validation errors or warnings.
3.Then enable the Settable at queue time option of this variable:
Run the same command again and now it works to queue the build. Also it succeeds to overwrite the original pipeline variable with the new value set in command-line.
You can do similar tests like what I did to figure out the cause of the behavior you met.
In addition:
VSTS CLI has been deprecated and replaced by Azure CLI with the Azure DevOps extension for a long time. So now it's more recommend to use az pipelines build queue
instead.
Lance had a great suggestion, but here is how I ended up solving it:
- name: Scenario
displayName: Scenario suite
type: string
default: 'Default'
variables:
ScenarioFinal: $[coalesce(variables['Scenario'], '${{ parameters.Scenario }}')]
...
steps:
- script: echo Scenario is $(ScenarioFinal)
In this case we use the coalesce expression to assign the value of a new variable, ScenarioFinal. This way we can still use --variables Scenario=Test via the CLI or use the parameter via the pipeline UI. coalesce will take the first non-null value and effectively "reorder" the precedence Lance linked to here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#expansion-of-variables
(Note that there need to be single quotes around the parameter reference '${{}}' because the ${{}} is simply converted to to the value, but then the coalesce expression doesn't know how to interpret the raw value unless it has the single quotes around it to denote it as a string)
Note that the ability to set parameters via the CLI is a current feature suggestion here: https://github.com/Azure/azure-devops-cli-extension/issues/972
Is it possible in Azure Devops YAML pipelines to dynamically create additional steps based on some variable data (without creating our own plugin)
The thing is I see that I want to iterate through several directories, but I don't want to just lump it all in a single step since it makes it harder to scan through to find an error.
Is it possible in Azure Devops YAML pipelines to dynamically create
additional steps based on some variable data (without creating our own
plugin)
No, Yaml pipelines(azure-pipeline.yml) are under Version Control. So what you want (for your original title) is to dynamically commit changes to the azure-pipeline.yml file when executing the pipeline. That's not a recommended workflow.
1.Instead you can consider using Azure Devops Conditions to dynamically enable/disable the additional steps.
Use a template parameter as part of a condition
Use the output variable from a job in a condition in a subsequent job
Or Use some predefined variables:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
2.If you're not using Conditions, you can check conditional template as Simon suggests above.
Also, both #1 and #2 can work with new feature runtime parameters.
3.However, if the dynamic variable you mean comes from the result of components = result of ls -1 $(Pipeline.Workspace)/components command, above tips won't work for this situation. For this you can try something like this:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
# some logic to run `components = result of ls -1 $(Pipeline.Workspace)/components` and determine whether to set the WhetherToRun=true.
'Write-Host "##vso[task.setvariable variable=WhetherToRun]True"'
- task: CmdLine#2
inputs:
script: |
echo Hello world
condition: eq(variables['WhetherToRun'], 'True')
It is possible to include steps conditionally with an if statement.
I think the example of extending a template on the same page will give you a good indication of how to iterate through a list parameter and create / run a step based on each value.
I am writing azure pipelines and using variables from variable groups. As part of adding flexibility, we are trying to make most of the parts of pipeline configurable ( more variables). So, i was looking for a way to add default value to variable if it is not present in variable group.
The way we use is like below:
variables:
dockerId: $(docker_id)
imageName: $(DOCKER_IMAGE_NAME)
Is there option to specify default for the variable, if it doesn't find it from variable group. Something like below:
variables:
dockerId: $(docker_id:"defaultDockerId")
imageName: $(DOCKER_IMAGE_NAME:"defaultDockerImageName")
You cannot achieve this directly in azure pipeline. Azure pipeline doesnot have this feature currently and this syntax $(docker_id:"defaultDockerId") is not supported.
As workaround, you may have to write scripts in script tasks to assign the default value if the variables are not exist.
Please check out below python script:
The first python script task checks if the variable exists and set a default value for it, and define the variable using print("##vso[task.setvariable variable=variableName]value").
(Please note the variable variableTest value can only be referenced in the following tasks.)
- task: PythonScript#0
displayName: 'setDefaultValue'
inputs:
scriptSource: inline
script: |
import os
b = os.getenv("variableTest","default value for variableTest")
print("##vso[task.setvariable variable=variableTest]{b}".format(b=b))
- task: PythonScript#0
displayName: 'Run a Python script'
inputs:
scriptSource: inline
script: |
print("$(variableTest)")
Hope above helps!
The comment is correct, you cannot have defaults (unless you are using templates and parameters, which you might want to look at, but they are not exactly the thing you are after). You can either use an if condition if you know when the variable exists or not (in the yaml file) or use a script task like the comment suggests and calculate the value in the script task and emit it back to the pipeline.