Create Stage that can be Called from Another Stage or Manually? - azure-devops

Hi I have a pipeline dependency challenge and have thought up a number of possible solutions. I can try them all out in a lab but the problem I am wondering if they work well "in the field" and so would like to know if anyone has tried them?
I have 3 stages, each in their own YML file. Each one is called from a main YML which is called from a main pipeline.
- template: 'build.yml'
- template: 'deploy.yml'
- template: 'test.yml'
The 'deploy.yml' generates a large number of output environment variables and 4 of these are consumed by the 'test.yml' using the "stageDependencies" syntax:
stages:
- stage: 'Test_Stage'
dependsOn: Deploy_Stage
jobs:
job: 'Test_Job'
variables:
MyWebSite: [ stageDependencies.Deploy_Stage.Variables_Job.outputs['Variables_Job.Variables.MyWebSite'] ]
This works nicely.
But, I'd like to be able to create a pipeline that just does the test stage (to test a pre-existing web site). That doesn't work of course because of the dependency dependsOn: Deploy_Stage.
I can think of a few possible solutions:
Instead of having a dependency and using the [ stageDependencies... ] syntax, send the MyWebSite as a pipeline parameter between stages. (Note that there are actually parameters not 1, I just simplified to demonstrate the challenge.) If I do that, the tester gets prompted to fill out (or choose from a list) the various parameters. But, it does create linkage between Deploy_Stage and Test_Stage - I don't know if that's a bad thing?
Pass a Boolean parameter from Deploy_Stage to Test_Stage such as "CalledFromDeployStage" and then in Test_Stage, do this:
stages:
- stage: 'Test_Stage'
${{ if eq(parameters.CalledFromDeployStage, true) }}:
dependsOn: Deploy_Stage
jobs:
job: 'Test_Job'
variables:
MyWebSite: [ stageDependencies.Deploy_Stage.Variables_Job.outputs['Variables_Job.Variables.MyWebSite'] ]
This feels a bit clunky.
Create a new YML called "Test_Stage_Manual" and get it to prompt for the various parameters and leave the rest as-is. (If I do this, I would probably put the jobs into their own YML file and call that YML from both Test stages.)
Something else?

You can try like as below:
Create an individual YAML pipeline to only run the test.
In the "Deploy_Stage" of the main pipeline, add a step or job at the end of this stage to execute the API "Runs - Run Pipeline" to trigger the pipeline for test after all the previous steps and jobs in this stage are completed successfully.
When calling the "Runs - Run Pipeline" API, you can pass the variables and parameters generated in the "Deploy_Stage" to the pipeline for test via the Request Body (JSON type) of the API.
Due to the test is in an individual pipeline, you can manually trigger this pipeline if you like. When manually trigger, you can manually set the value of the required variables and parameters in the pipeline.
With this way, you can trigger the pipeline for test both via the "Deploy_Stage" and manually.

Related

Azure Pipeline root variables in templates

I am trying to use variables defined at the root level on a YAML pipeline inside Azure DevOps inside templates via the template syntax, but it seems that the variables are not available inside the templates, but when adding the steps directly to the pipeline, the exact same thing works perfectly.
So with a pipeline snippet like that
variables:
- name: test
value: asdf
stages:
- stage:
jobs:
- job: test_job
steps:
- script: echo "${{ variables.test }}"
- template: ./test.yaml
And a test.yaml like that
jobs:
- job: test
steps:
- script: echo "${{ variables.test }}"
The script inside the test_job job writes out asdf while the job inside the template just resolves to echo "".
Since my understanding of pipeline templates is, that those basically get inserted into the main pipeline, this seems like a bug. Any ideas on how to use the root variables in a template syntax inside templates or why this is not working? (Macro synatx is not an option as I need the variable inside a templated condition like ${{ if eq(variables['test'], 'asdf') }})
For security reasons, we only allow you to pass information into
templated code via explicit parameters.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
The means the author of the pipeline using your template needs to
commit changes where they explicitly pass the needed info into your
template code.
There are some exceptions to this, where the variable is statically
defined in the same file or at pipeline compile time, but generally
speaking, it’s probably better to use parameters for everything that
does not involve system-defined read-only dynamic variable and
custom-defined dynamic output variables.
This behavior is by design, check this thread in the developer community
So you can either pass the variable as a parameter to the template or define a centralized variables file to include in the template like here

Conditions on Stage template not working in Azure yml pipeline

I have below main YML pipeline
And below is the template that is being called.
When trying to run the main pipeline is showing the error per below
Agree with Krzysztof Madej.
When you add condition to the stages level, you need to make sure that there is at least one stage that meets the condition at all times.
In addition to adding an empty stage, you could also add the negative condition in the Yaml sample:
For example:
stages:
- ${{if eq(variables['envName'],'sbx')}}:
- template: test.yml
parameters:
buildSteps: test
- ${{if ne(variables['envName'],'sbx')}}:
- stage: Test
jobs:
- job: TestJob
steps:
- script: echo Test
displayName: 'Test Stage'
In this case, if the envName = sbx, it will run the template, or it will run another stage.
It looks that your condition is not met. So envname is not sbx and thus your template is skipped, and you can't run pipeline without any stage. Please make sure you have always at least one stage regardless of the result of the condition, or sth like empty stage for non sbx env.

Fill runtime azure pipeline parameters from external source

We looking to create a pipeline to update our multi-tenant azure environment. We need to perform some actions during the update per tenant. To accomplish this, we would like to create a job per tenant, so we can process tenants in parallel. To accomplish this, I want to use a runtime parameter to pass the tenants to update to my pipeline as follows:
parameters:
- name: tenants
type: object
the value of the tenants parameter might look like something like this:
- Name: "customer1"
Someotherproperty: "some value"
- Name: "customer2"
Someotherproperty: "some other value"
to generate the jobs, we do something like this:
stages:
- stage:
jobs:
- job: Update_Tenant
strategy:
matrix:
${{ each tenant in parameters.Tenants }}:
${{ tenant.tenantName }}:
name: ${{ tenant.tenantName }}
someproperty: ${{ tenant.otherProperty }}
maxParallel: 2
steps:
- checkout: none
- script: echo $(name).$(someproperty)
Now what we need, is some way to fill this tenants parameter. Now I tried a few solutions:
Ideally I would like to put a build stage before the Update_Tenants stage to call a REST api to get the tenants, and expand the tenants parameter when the Update_Tenants stage starts, but this is not supported AFAIK, since parameter expansion is done when the pipeline starts.
A less ideal but still workable option would have been to create a variable group yaml file containing the tenants, and include this variable group in my pipeline, and use the ${{ variables.Tenants }} syntax to reference them. However, for some reason, variables can only be strings.
The only solution I can currently think of, is to create a pipeline that calls a REST api to get the tenants to update, and then uses the azure devops api to queue the actual update process with the correct parameter value. But this feels like a bit of a clunky workaround to accomplish this.
Now my question is, are there any (better?) alternatives to accomplish what I want to do?
Maybe this can help. I was able to use external source (.txt file) to fill array variable in azure pipelines.
Working example
# Create a variable
- bash: |
arrVar=()
for images in `cat my_images.txt`;do
arrVar+=$images
arrVar+=","
done;
echo "##vso[task.setvariable variable=list_images]$arrVar"
# Use the variable
# "$(list_images)" is replaced by the contents of the `list_images` variable by Azure Pipelines
# before handing the body of the script to the shell.
- bash: |
echo my pipeline variable is $(list_images)
Sources (there is also example for matrix)
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-a-job-scoped-variable-from-a-script
Other sources
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/runtime-parameters?view=azure-devops&tabs=script
To accomplish this, we would like to create a job per tenant, so we
can process tenants in parallel.
Apart from rolling deployment strategy, you can also check Strategies and Matrix.
You can try something like this unless you have to use Runtime parameters:
jobs:
- job: Update
strategy:
matrix:
tenant1:
Someotherproperty1: '1.1'
Someotherproperty2: '1.2'
tenant2:
Someotherproperty1: '2.1'
Someotherproperty2: '2.2'
tenant3:
Someotherproperty1: '3.1'
Someotherproperty2: '3.2'
maxParallel: 3
steps:
- checkout: none
- script: echo $(Someotherproperty1).$(Someotherproperty2)
displayName: 'Echo something'

Pass output variable to different stages

I have a pipeline in Azure DevOps that essentially look like this
stage: BuildStage
job: SetUp
job: Compile
stage: DeployStage
job: Deploy
In the SetUp job I define an output variable, which I can define in the Compile job using e.g.
variables:
MyVariableFromSetUp: $[ dependencies.SetUp.outputs['MyVariable'] ]
The question is, how can I do the same in the Deploy job? I don't want to run the SetUp stage twice as it is time consuming to calculate the value of MyVariable hence I must cache it.
The DeployStage has a dependsOn to BuildStage, but it appears I cannot use the dependencies as I expected. The documentation fails to mention the multi-stage case when dealing with variables.
Hey there is no direct way to do this at present based on what I have found, you would be following one of the following 3 methods
Passing it as artifact value using this method https://medium.com/microsoftazure/how-to-pass-variables-in-azure-pipelines-yaml-tasks-5c81c5d31763
Passing it via a API Endpoint using VSTeam powershell module https://www.donovanbrown.com/post/Passing-variables-from-stage-to-stage-in-Azure-DevOps-release, similar method can also be found here https://stefanstranger.github.io/2019/06/26/PassingVariablesfromStagetoStage/

How to conditionally run a build agent job with a pipeline variable?

In Azure DevOps pipelines there's an option to conditionally run a task based on a pipeline variable. This is handled under the Run this task > Custom conditions field and it uses the syntax:
eq(variables['VarName'], 'Desired Value')
An agent job has a similar field for conditional execution under Run this job > Custom condition using variable expressions.
However, when I use the same syntax as a conditional task the result always evaluates to 'false'.
So how can I conditionally run an agent job?
Screenshots:
Something like this worked for me:
- job: Job1
steps:
- powershell: |
if (some condition)
{
Write-Host ("##vso[task.setvariable variable=RunJob2;isOutput=true]True")
}
name: ScriptStep
- job: Job2
dependsOn: Create_Build_Matrix
condition: and(succeeded(), eq(dependencies.Job1.outputs['ScriptStep.RunJob2'], 'True'))
I discovered the answer. Unfortunately, it is not possible to conditionally run an agent job with a variable that is modified during build execution.
From the Azure DevOps Pipeline documentation under Pipeline Variables:
To define or modify a variable from a script, use the task.setvariable
logging command. Note that the updated variable value is scoped to
the job being executed, and does not flow across jobs or stages.
Try this one: https://stefanstranger.github.io/2019/06/26/PassingVariablesfromStagetoStage/
This one gives you to pass variables from one stage/job to another stage/job in the same release pipeline. I tried and it's working fine.
Also to run this you need to give some permissions for release pipeline. To allow for updating the Release Definition during the Release you need to configure the Release Permission Manage releases for the Project Collection Build Service.