I'm having an issue with a new pipeline I'm setting up for a microservice on Azure DevOps where the pipeline seems to be unable to reference environment variables provided by the build environment.
I have a step in my pipeline that runs a Powershell script that prints out all the environment variables for debugging purposes and this allows me to confirm that the environment does indeed contain the variables I am trying to use.
These variables are BUILD_DEFINITIONNAME = my-microservice-repo and BUILD_BUILDNUMBER = 20200519.2. Essentially I want to combine these two variables to create another variable to name my docker container so I'd have a variable equal to $(BUILD_DEFINITIONNAME):$(BUILD_BUILDNUMBER) which should expand out into my-microservice-repo:20200519.2. We will call this variable imageName. I have defined it in my variables section on my .yml script as
variables:
- name: imageName
value: $(BUILD_DEFINITIONNAME):$(BUILD_BUILDNUMBER)
However, if I try to use this variable on my Docker steps:
- task: Docker#0
inputs:
dockerFile: Dockerfile
azureSubscriptionEndpoint: 'endpoint'
azureContainerRegistry: 'my container registry'
imageName: $(imageName)
action: 'Build an image'
- task: Docker#0
inputs:
azureSubscriptionEndpoint: 'endpoint'
azureContainerRegistry: 'my container registry'
action: 'Push an image'
imageName: $(imageName)
Then when the script runs, I get this error: ##[error]invalid argument "myazurecontainerregistry.azurecr.io/$(build_definitionname):$(build_buildnumber)" for "-t, --tag" flag: invalid reference format
So clearly it's concatenating the strings correctly since I get $(build_definitionname):$(build_buildnumber), however, they're not being expanded, despite the fact that they both exist within the printed variables as I mentioned earlier.
Is there something special I need to do to allow environment variables to be used within my script? We have other pipeline scripts that are working just fine that I copied this yml script from, but mine refuses to work and I can't figure out why, nor can my colleagues.
Any help would be greatly appreciated.
Figured it out. I was misnaming the variables I wanted to use.
$(BUILD_BUILDNUMBER) should have been $(Build.BuildNumber).
Related
So I have a variable in a variables.yaml file called keyVaultSecretToRetrieve. It can vary based on environment. Previously this pipeline was coded for single environment. So lets say the keyVaultSecretToRetrieve is "secret1". If in the task I put SecretsFilter: "secret1" and run the task and try to reference $(secret1) in the following task everything works great.
But my problem is that if I put it like the example below as SecretsFilter $(keyVaultSecretToRetrieve). How can I then retrieve the value? Its almost like I would want to do something like if it where possible $($(keyVaultSecretToRetrieve)).
I know my other option is to just run the command in a script using azure CLI which I tried but the build server says it does not have azure cli installed and I'd rather not mess with it and just wrap this up quickly. Not sure if there is some way to achieve what I want?
It's a windows build agent and the steps after this are mostly PowerShell. I wish the task just returns a data structure vs. dynamically named variables.... This is throwing me off it there is some trick I can do to make it work that I'm not aware of vs. just being forced to go the azure cli route or have the variable name coded in the variables name vs. as another variables.....
- task: AzureKeyVault#2
displayName: Retrieve from keyvault
inputs:
azureSubscription: $(serviceConnection)
KeyVaultName: $(keyVaultName)
SecretsFilter: $(keyVaultSecretToRetrieve)
RunAsPreJob: false
Based on your requirement, you need to use nested variable: $($(keyVaultSecretToRetrieve)). There is no built-in feature can support this requirement.
To achieve your requirement, you can use the Variable Set task from extension: Variable Toolbox.
For example:
steps
- task: VariableSetTask#2
inputs:
variableName: NewVariable
Value: '$($(keyVaultSecretToRetrieve))'
It will create a new variable based on the nested variable:$($(keyVaultSecretToRetrieve)). Then you can use the $(NewVariable) in the next tasks.
I've tried several articles and threads from Stackoverflow but can't seem to get anywhere. I am trying to take a variable from a .py file which is called in a YAML step and set that variable globally to be used.
In my .py file i have
print(f'##vso[task.setvariable variable=AMLPipelineId;isOutput=true]{pipelineId}')
Then in my YAML pipeline step i have
- task: AzurePowerShell#5
displayName: 'Run AML Pipeline'
inputs:
azureSubscription: '$(azureSubscription)'
ScriptType: 'InlineScript'
name: AmlPipeline
azurePowerShellVersion: 'LatestVersion'
pwsh: true
Inline: |
$username = "$(ARM_CLIENT_ID)"
$password = "$(ARM_CLIENT_SECRET)"
$tenantId = "$(ARM_TENANT_ID)"
python $(Pipeline.Workspace)/AML_Pipeline/build_aml_pipeline.py --wsName $(wsName) --resourceGroup $(ResourceGroupName) --subscriptionId $(subId)
$MLPipelineId = $AmlPipeline.AMLPipelineId
But it seems like this variable is empty. I know there are other ways of using the "set variable" but this is my latest attempt i.e. something like print('##vso[task.setvariable variable=version;]%s' % (version))
My current approach i followed: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
You don't need isOutput=true - that's only needed for referencing variables between different jobs or stages.
"You cannot use the variable in the step that it is defined." - split that script into two steps: one that runs your .py file, second one that uses the newly defined variable.
I used print('##vso[task.setvariable variable=<Variable-in-Pipeline]+<output-variable>')
Variable-in-Pipeline // the given name should be used in Azure Devops pipeline and should be added to pipeline variables as an empty string
A very minimal example for everyone struggling with this. The documentation is kind of lacking on this for my taste. As #qbik said, dont set and use the variable in the same step, make it seperate steps.
set_variable.py
if __name__ == '__main__':
# set name of the variable
name = 'COLOR'
# set value of the variable
value = 'red'
# set variable
print(f'##vso[task.setvariable variable={name};]{value}')
azure-pipelines.yml
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- task: UsePythonVersion#0
inputs:
versionSpec: '3.9'
displayName: 'Use Python 3.9'
# run the script to set the variable
- task: PythonScript#0
inputs:
scriptSource: filePath
scriptPath: set_variable.py
# now you can use the variable in the next step
- bash: echo my favorite color is: $(COLOR)
Now you can technically do all kinds of cool stuff in python, then set, and reference the variables in the following steps. In my case I have to extract specific package version numbers from a JSON/YAML file based on an id that is set earlier in the pipeline and parse the information as an args for a docker build. Hope that helps other people stumbling across this answer looking for a minimal working example :)
I have a standard .NET Core (Ubuntu) pipeline on Azure Devops and within my Test project, I use environment variables. Within my pipeline, I have defined my group variables like so
variables:
- group: MyApiVariables
Whenever I run the tests for my project
- task: DotNetCoreCLI#2
displayName: "Testing Application"
inputs:
command: test
projects: '**/*Tests/*.csproj'
arguments: '--configuration $(buildConfiguration)'
The actual environment variables aren't passed in. They are blank.
What am I missing to get this running? I've even defined variables in the edit pipeline page too with no luck
- task: Bash#3
inputs:
targetType: 'inline'
script: echo $AppConfigEndpoint
env:
AppConfigEndpoint: $(AppConfigEndpoint)
ApiConfigSection: $(ApiConfigSection)
Thanks!
CASING Strikes again! MyVariableName was turned into MYVARIABLENAME on Azure Devops. I changed my variable names in my group to all caps and it worked. I spent way too much time on this.
Mike, I see that you use variable groups. I assume it may cause your issue. Take a look at variable passing example I made:
First I had to create new variable group in Library:
Here is a pipeline code that reference created variables:
# Set variables group reference
variables:
- group: SampleVariableGroup
steps:
- powershell: 'Write-Host "Config variable=$(configuration) Platform variable=$(platform)"'
displayName: 'Display Sample Variable'
I used PowerShell task to verify if variables were properly passed to the job.
As you can see both configuration & platform values were displayed correctly.
In fact you can't go wrong that way, unless you start to mix variable groups with variables defined in a yaml. In such scenario you'll have to use name/value syntax for the individual (non-grouped) variables.
Please see Microsoft Variable Groups documentation. Such example is well explained there. I also suggest to take closer look at general Variables Documentation.
In case of referencing variables in other tasks here is a great example from MS (it should work everywhere in same manner):
# Set variables once
variables:
configuration: debug
platform: x64
steps:
# Use them once
- task: MSBuild#1
inputs:
solution: solution1.sln
configuration: $(configuration) # Use the variable
platform: $(platform)
# Use them again
- task: MSBuild#1
inputs:
solution: solution2.sln
configuration: $(configuration) # Use the variable
platform: $(platform)
Good luck!
I am writing azure pipelines and using variables from variable groups. As part of adding flexibility, we are trying to make most of the parts of pipeline configurable ( more variables). So, i was looking for a way to add default value to variable if it is not present in variable group.
The way we use is like below:
variables:
dockerId: $(docker_id)
imageName: $(DOCKER_IMAGE_NAME)
Is there option to specify default for the variable, if it doesn't find it from variable group. Something like below:
variables:
dockerId: $(docker_id:"defaultDockerId")
imageName: $(DOCKER_IMAGE_NAME:"defaultDockerImageName")
You cannot achieve this directly in azure pipeline. Azure pipeline doesnot have this feature currently and this syntax $(docker_id:"defaultDockerId") is not supported.
As workaround, you may have to write scripts in script tasks to assign the default value if the variables are not exist.
Please check out below python script:
The first python script task checks if the variable exists and set a default value for it, and define the variable using print("##vso[task.setvariable variable=variableName]value").
(Please note the variable variableTest value can only be referenced in the following tasks.)
- task: PythonScript#0
displayName: 'setDefaultValue'
inputs:
scriptSource: inline
script: |
import os
b = os.getenv("variableTest","default value for variableTest")
print("##vso[task.setvariable variable=variableTest]{b}".format(b=b))
- task: PythonScript#0
displayName: 'Run a Python script'
inputs:
scriptSource: inline
script: |
print("$(variableTest)")
Hope above helps!
The comment is correct, you cannot have defaults (unless you are using templates and parameters, which you might want to look at, but they are not exactly the thing you are after). You can either use an if condition if you know when the variable exists or not (in the yaml file) or use a script task like the comment suggests and calculate the value in the script task and emit it back to the pipeline.
I'm attempting to create multiple pipelines in Azure DevOps but I would like to reuse the same pipeline YAML file with the differences per environment being loaded from a separate template variables file.
For that purpose I've created two variable files, which are located in the same folder as the pipeline definition:
# vars.dev.yml
variables:
- name: EnvironmentName
value: Development
# vars.prd.yml
variables:
- name: EnvironmentName
value: Production
And the definition of the pipeline is the following:
trigger: none
pr: none
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.$('EnvironmentCode').yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: $(EnvironmentFileName)
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
After importing the pipelines using the Azure DevOps UI, I can go to settings of each and set the Environment Code variable to whatever desired environment code:
However I'm always getting the same error when I try to run the pipeline, regardless of the code I fill in the variable value:
So the question here is: Is this kind of variable expansion not supported or is there a different way that I should use to accomplish this?
Thanks!
EDIT
I was able to expand the variables using another method. The new version of the pipeline is as such:
variables:
- name: EnvironmentCode
value: dev
- name: EnvironmentFileName
value: vars.${{ variables.EnvironmentCode }}.yml
stages:
- stage: LoadVariablesPerEnvironment
displayName: Load Variables Per Environment
variables:
- template: ${{ variables.EnvironmentFileName }}
jobs:
- job: ShowcaseLoadedVariables
steps:
- pwsh: Write-Host "Variables have been loaded for the '$ENV:ENVIRONMENTNAME' environment"
displayName: Output Environment Variables
However there is yet the issue of loading different files. I made different attempts and verified the following:
If you give a different environment code using the UI, when running
the pipeline, the value it assumes is still the one that's on the
pipeline definition;
If you remove from the pipeline definition the
default value or the variable entirely the expression
${{variables.EnvironmentCode}} will return an empty string
assuming the filename to be vars..yml which doesn't exist.
Is this kind of variable expansion not supported or is there a
different way that I should use to accomplish this?
If I am not misunderstand, at first, you want to use $() to get the variable you defined using the UI but failed. But later, ${{ }} can give you the value of the variable EnvironmentCode.
In fact, while you change to use ${{ }}, it just accessing the variable you predefined in the YAML files instead of the one you defined with UI. Just see this doc: Variable templates.
For the variable you defined with UI, it can be get and used with the format $()(Note: ${{ }} is the format of get the variables which defined in YAML file). But also, there some things you need to pay attention is for the variables you defined in UI, it can only be get/accessed after the build begin to run, because the variable which defined with UI exists in environment only after the build compiled begin. In one word, they are the agent-scope variable. That's why the value it used is still the one that's on the pipeline definition instead of on the UI.
If you remove from the pipeline definition the default value or the
variable entirely the expression ${{variables.EnvironmentCode}} will
return an empty string assuming the filename to be vars..yml which
doesn't exist.
As the document defined and I mentioned before, ${{}} is format which used to get the value of variable which defined in YAML files rather than the one which defined using UI.
In the steps of job, the variable that defined in the UI or defined in the YAML file can all be get/accessed with the format $(). Still, for the variable which defined in the YAML file can also get with ${{variables.xxxx}}. But at this time, if the variable name which defined in YAML file is same with the one defined with UI, the server can only get the one defined in YAML file.