I've got yaml pipeline that referes some templates.
I have variable group linked to this main yaml file and I want to pass one of its variables to the template.
It's simple when I want to "just use it" as bellow:
example:
stages:
- stage: Deployment
variables:
- group: My_group_variables
jobs:
- template: /templates/jobs/myJobTemplate.yml
parameters:
someParameter: $(variable_from_my_variable_group)
myJobTemplate.yml:
parameters:
- name: someParameter
default: ''
jobs:
- job: Myjob
steps:
- task: Bash#3
inputs:
targetType: 'inline'
script: cat ${{ parameters.someParameter}}
It does not cooperate when I want to have parameters validation like:
parameters:
- name: environmentName
type: string
values:
- Development
- Test
- UAT
- Production
Or when I want to use "service connection" name as a variable.
...
- task: KubernetesManifest#0
displayName: Deploy to Kubernetes cluster
inputs:
action: 'deploy'
kubernetesServiceConnection: ${{ parameters.kubernetesServiceConnection }}
namespace: ${{ parameters.kubernetesNamespace }}
manifests: ${{ variables.manifestFile }}
...
Does anyone know how should I use those variables with pre-validated parameters or service connections?
It's most probably an issue with the time of resolving values. Pre-defined parameters and service connections names are checked on compile-time, while values from $() are resolved during runtime.
I cannot use extends and variables in this template.
Maybe someone has a pattern for those kinds of usage?
Related
Given I have following step in Azure Devops pipeline:
- task: DotNetCoreCLI#2
displayName: Run tests
inputs:
command: "test"
env:
"ASPNETCORE_ENVIRONMENT": "test"
"Case_sensitive_environment_variable_key": "This is the value"
I'd like to move this step into the template and enable users of that template to pass in env key values making sure it keeps case sensitivity of keys and values.
How can I do that?
According to your description and to transfer the parameter as the environmental varailbles on the level of a task, you may try Passing parameters into the template and using the parameter type of object.
Kindly take the following main YAML pipeline and template.yml for a reference.
Main
trigger: none
parameters:
- name: pEnvironment
type: object
default:
ASPNETCORE_ENVIRONMENT: "TEST 1"
Case_sensitive_environment_variable_key: "This is the value 1"
pool:
vmImage: windows-latest
stages:
- stage:
jobs:
- template: Templates/template1.yml
parameters:
transferedEnv: ${{ parameters.pEnvironment }}
Template
parameters:
- name: transferedEnv
type: object
jobs:
- job:
steps:
- task: DotNetCoreCLI#2
displayName: Run tests
inputs:
command: "test"
env:
${{ parameters.transferedEnv }}
When we run the main pipeline, we can input as well the values of the env to transferred into the template.
I am trying to make a pipeline template that takes a JobList a parameter and runs all the jobs, while ensuring that they run on the same agent every time. Basically the approach I've been taking is to try to adapt this answer into a genericized template format.
This is what I have so far, and I've tried a lot of slight tweaks of this with nothing passing the Validate test on the pipeline that calls it.
parameters:
- name: jobsToRun
type: jobList
- name: pool
type: string
default: Default
- name: demands
type: object
default: []
jobs:
- job:
steps:
- script: echo "##vso[task.setvariable variable=agentName;isOutput=true;]$(Agent.Name)"
pool:
name: ${{ parameters.pool }}
demands:
- ${{ each demand in parameters.demands }}:
${{ demand }}
- ${{ each j in parameters.jobsToRun }}:
${{ each pair in j }}:
${{ pair.key }} : ${{ pair.value }}
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
What am I doing wrong here? It seems like it should be possible if that answer I reference is correct, but it seems like I'm just a bit off.
Name missing on the job.., example below.
- job: 'test-Name'
Steps need a associated job and pool to run is declared inside
jobsToRun:
- job: sample_job1
displayName: "sample_job1"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"
On this bottom pool declaration...
pool:
name: Default
demands:
- Agent.Name -equals $(agentName)
i am not sure but i have tried this many times but i think this can't be included separate to the job since each individual job is passed as parameter. pool definition needs to be inside the job or inside the job template if you are using templates..
example:
jobsToRun:
- job: output_message_job1
displayName: "in pipe Output Message Job"
pool:
name: "your_PoolName"
steps:
- script: |
echo "Hi"
Problem Description
I was having some problems trying to use variables created in one stage in another stage and managed to find various articles old and new describing how this can be done. The more recent articles/posts identifying the new syntax
$[stageDependencies.{stageName}.{jobName}.outputs['{stepName}.{variableName}']
Used like this:
variables:
myVariable: $[stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage']]
This works great until you needed to use job templates.
None of the samples I found online covered the situation of templates. They just demonstrated how multiple stages in the same yaml file could obtain the value.
The syntax depends on being able to put the expression into a variable. Unfortunately, when you use a template for a job, it's not possible to declare variables and passing it as a parameter results in it being unevaluated.
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
In this snippet, it comes through as-is. The value does not get substituted in.
Theoretically, you could set your job to have the expression present in the variables block but that sort of hard-coding undermines one of the main benefits of templates.
Related Articles
Share variables across stages in Azure DevOps Pipelines
Azure DevOps Release Notes - sprint 168
I know the question asks about template jobs, but for future reference I want to describe how it can be achieved with template stages as well.
It is done with a variable workaround as in the accepted answer, and with a reference to stagedependencies when no dependsOn exists. (Templates don't allow dependsOn). Somehow, this still works.
Example YAML using stage template (I have modified the code from the accepted answer):
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
# stage template cannot use dependsOn, but is still allowed to refer to stagedependencies...
- template: templates/stage-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
Stage template:
parameters:
- name: ValueToOutput
type: string
stages:
- stage: ShowStageVarStage
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
Solution
The answer isn't actually far away. The original expression just need to be passed through a variable in the template job. Basically, set a variable to be the value of the parameter and use the macro syntax to evaluate the variable.
parameters:
- name: ValueToOutput
type: string
...
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
Using the macro syntax of $(LocalVarOfValueToOutputParam) will result in the value making its way into the template job correctly.
Example
If we have a yaml file for the build definition:
stages:
- stage: CreateStageVarStage
displayName: 'Create StageVar Stage'
jobs:
- job: CreateStageVarJob
displayName: 'Create StageVar Job'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
steps:
- checkout: none
- pwsh: |
[string]$message = 'This is the value from the first stage'
Write-Host "Setting output variable 'VariableFromFirstStage' to '$message'"
Write-Output "##vso[task.setvariable variable=VariableFromFirstStage;isOutput=$true]$message"
name: SetValueStep
- stage: UseJobTemplateStage
displayName: 'Use Job Template Stage'
dependsOn: CreateStageVarStage
jobs:
- template: templates/job-showstagevars.yml
parameters:
ValueToOutput: $[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
That uses the job template templates/job-showstagevars.yml
parameters:
- name: ValueToOutput
type: string
jobs:
- job: ShowStageVarJob
displayName: 'Show stage var'
timeoutInMinutes: 5
pool:
name: 'Azure Pipelines'
vmImage: 'windows-2019'
variables:
- name: LocalVarOfValueToOutputParam
value: ${{ parameters.ValueToOutput }}
steps:
- checkout: none
- pwsh: |
Write-Host "ValueToOutput parameter=${{ parameters.ValueToOutput }}"
Write-Host "LocalVarOfValueToOutputParam (pre-processor syntax)=${{ variables.LocalVarOfValueToOutputParam }}"
Write-Host "LocalVarOfValueToOutputParam (macro syntax)=$(LocalVarOfValueToOutputParam)"
displayName: 'Show StageVariable'
What we get in our output of the second stage is:
ValueToOutput parameter=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (pre-processor syntax)=$[ stagedependencies.CreateStageVarStage.CreateStageVarJob.outputs['SetValueStep.VariableFromFirstStage'] ]
LocalVarOfValueToOutputParam (macro syntax)=This is the value from the first stage
In my YAML pipeline I have a deployment job:
- stage: deployment_to_development
jobs:
- deployment: deployment_to_development
displayName: Deploy to Development
environment: Development
variables:
- template: migrate-vars.yml
strategy:
runOnce:
deploy:
steps:
- template: migrate-data.yml
The deployment template is a simple DbUp task:
steps:
- task: UpdateDatabaseWithDbUp#2
displayName: Migrate data
inputs:
ConnectionString: 'Data Source=$(DatabaseServer);Initial Catalog=$(DatabaseName);Integrated Security=SSPI'
ScriptPath: '$(Pipeline.Workspace)\data-migrations'
JournalSchemaName: dbo
JournalTableName: _SchemaVersions
LogScriptOutput: true
IncludeSubfolders: true
Order: FolderStructure
TransactionStrategy: SingleTransaction
The variables template defines the server and db name:
variables:
DatabaseServer: 'server'
DatabaseName: 'dbName'
Instances: '_1,_2'
This all works fine for a single database. However, I wish to repeat the task for each instance defined in the Instances variable, i.e for databases named dbName_1, dbName_2. This seemingly simple feat is proving difficult.
I have tried passing the instances as a parameter array and iterating them using
parameters:
param: []
steps:
- ${{each p in parameters.param}}:
but the 'p' variable isn't evaluated in the task.
There have been many more futile attempts. I must be missing something very obvious here. What is it?
I can't test UpdateDatabaseWithDbUp#2 but I have sth what explain how you can achieve your goal. First define template.yaml
parameters:
- name: 'instances'
type: object
default: {}
- name: 'server'
type: string
default: ''
steps:
- ${{ each instance in parameters.instances }}:
- script: echo ${{ parameters.server }}:${{ instance }}
then reuse this template in your build:
steps:
- template: template.yaml
parameters:
instances:
- test1
- test2
server: someServer
And here is the result:
I have two templates created -- one for getting and setting some configurations such as region names and one for deploying. I'm trying to use the variables set in the configuration template task as parameter inputs to the deploy template. Is there an actual way of doing this?
My config template:
steps:
- task: AzureCLI#2
name: Config
displayName: Get Config and Generate Variables
inputs:
azureSubscription: '$(Subscription)'
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
Environment="prod"
echo "##vso[task.setvariable variable=Environment;isOutput=true]prod"
echo "##vso[task.setvariable variable=EastName;isOutput=true]$(AppNamePrefix)-$Environment-eastus"
echo "##vso[task.setvariable variable=East2Name;isOutput=true]$(AppNamePrefix)-$Environment-eastus2"
echo "##vso[task.setvariable variable=CentralName;isOutput=true]$(AppNamePrefix)-$Environment-centralus"
echo "##vso[task.setvariable variable=WestName;isOutput=true]$(AppNamePrefix)-$Environment-westus"
and my deploy template looks like this:
parameters:
- name: artifactName
type: string
default: MyBuildOutputs
- name: appFullName
type: string
- name: condition
type: boolean
default: true
steps:
- task: AzureFunctionApp#1
condition: ${{ parameters.condition }}
displayName: 'Production deploy'
inputs:
azureSubscription: '$(Subscription)'
appType: 'functionApp'
appName: ${{ parameters.appFullName }}
package: '$(System.ArtifactsDirectory)/${{ parameters.artifactName }}/$(Build.BuildId).zip'
deploymentMethod: 'auto'
and my stage for this looks like so (with unnecessary bits cutout):
- template: ../../tasks/azure/getConfig.yml
- template: ../../tasks/azure/deployToFA.yml
parameters:
appFullName: $(EastName)
I have tried the following for appFullName: <name>:
$(EastName)
${{ EastName }}
$[ EastName ]
$EastName
but, sadly, none of these seem to work as they all get pulled in as literals. Is there a way of doing this? I have seen ways of using dependsOn but I don't want the hidden dependency between the two templates (if that's even possible)
but, sadly, none of these seem to work as they all get pulled in as
literals. Is there a way of doing this? I have seen ways of using
dependsOn but I don't want the hidden dependency between the two
templates (if that's even possible)
Sorry but I'm afraid your template structure is not supported for now. You can check Process the pipeline:
To turn a pipeline into a run, Azure Pipelines goes through several steps in this order:
First, expand templates and evaluate template expressions.
So the ${{ parameters.appFullName }} in deploy template is evaluated before the config template runs the AzureCli task. That's why none of these seem to work as they all get pulled in as literals. It's by design that your $(EastName)(runtime variable) has no meaning when being passed to parameter.
As an alternative way, check Use variables as task inputs. It describes another way to meet your needs.
Your config template:
steps:
- task: AzureCLI#2
name: Config
displayName: Get Config and Generate Variables
inputs:
xxx
Your deploy template:
parameters:
- name: artifactName
type: string
default: MyBuildOutputs
- name: appFullName
type: string
- name: condition
type: boolean
default: true
steps:
- task: AzureFunctionApp#1
condition: ${{ parameters.condition }}
displayName: 'Production deploy'
inputs:
azureSubscription: '$(Subscription)'
appType: 'functionApp'
appName: $(Config.EastName) // Changes here. ********* Config is the name of your AzureCLI task.
package: '$(System.ArtifactsDirectory)/${{ parameters.artifactName }}/$(Build.BuildId).zip'
deploymentMethod: 'auto'
Your stage:
- template: ../../tasks/azure/getConfig.yml
- template: ../../tasks/azure/deployToFA.yml
Hope it helps.