Resolving Predefined Agent Variables in Runtime Expressions - azure-devops

I have the following scenario, it is simplified for the sake of brevity, but outlines my problem.
I have a 2 Job pipeline.
BreakMatrix Job: A job that runs on an AdminPool and outputs 2 variables with the following names ENV1 and ENV2. The names are important because each of them matches the name of an Environment running in a separate MachinePool VM deployment pool.
Upgrade Job: A deployment job that depends on the BreakMatrix job and that runs on a VM MachinePool, with a tag that will select ENV1 and ENV2 environments.
I am trying to pass in one of the variables to each of the corresponding Environments:
- job: BreakMatrix
pool: AdminPool
steps:
- checkout: none
- powershell: |
$result1 = #{"Hostname" = "Env1Value"}
$result2 = #{"Hostname" = "Env2Value"}
Write-Host "##vso[task.setvariable variable=ENV1;isOutput=true;]$result1"
Write-Host "##vso[task.setvariable variable=ENV2;isOutput=true;]$result2"
name: outputter
- deployment: Upgrade
dependsOn: BreakMatrix
variables:
agentName: $(Environment.ResourceName)
agentName2: $(Agent.Name)
formatted: $[ format('outputter.{0}', variables['agentName']) ]
result1: $[ dependencies.BreakMatrix.outputs[format('outputter.{0}', variables['agentName'])] ]
result2: $[ dependencies.BreakMatrix.outputs[format('outputter.{0}', variables['agentName2'])] ]
result3: $[ dependencies.BreakMatrix.outputs[format('outputter.{0}', variables['Agent.Name'])] ]
hardcode: $[ dependencies.BreakMatrix.outputs['outputter.ENV2'] ]
json: $[ convertToJson(dependencies.BreakMatrix.outputs) ]
environment:
name: MachinePool
resourceType: VirtualMachine
tags: deploy-dynamic
strategy:
rolling:
preDeploy:
steps:
- powershell: |
echo 'Predeploy'
echo "agentName: $(agentName)"
echo "agentName2: $(agentName2)"
echo "env: $(Environment.ResourceName)"
echo "formatted: $(formatted)"
echo "harcode: $(harcode)"
echo "result1: $(result1)"
echo "result2: $(result2)"
echo "result3: $(result3)"
echo "json: $(json)"
deploy:
steps:
- powershell: |
echo 'Deploy'
Output for ENV2 pre-deploy step:
Predeploy
agentName: ENV2
agentName2: ENV2
env: ENV2
formatted: outputter.ENV2
hardcode: {
Hostname:Env2Value}
result1:
result2:
result3:
json: {
outputter.ENV2:
\"Hostname\":\"Env2Value\"
}
If i try to use a predefined variable in a dependency expression they don't seem to properly resolve, but if i just simply map them to a variable / format them they work.
Note: The environment names are actually dynamically calculated before these jobs run so I cannot use parameters or static/compile time variables.
Any suggestions on how to pass in only the relevant variable to each of the environments ?

I got confirmation on the developer community that this is not possible.
Relevant part:
But I am afraid that the requirement couldnโ€™t be achieved.
The reason is that the dependencies.xx.outputs expression canโ€™t read
the variable(variables['Agent.Name']) and format expression value
defined in the YAML pipeline.
It now only supports hard-coding the name of the variable to get the
corresponding job variable value.

Related

github remove newline from the env

I have below workflow with workflow dispatcher
jobs:
deploy_infra:
name: Deploy Infra
env:
INFRA_ACCOUNT: >
${{ fromJson('{
"dev": "12345",
"preprd": "678234",
"prd": "91056"
}')[github.event.inputs.stage] }}
steps:
- name: Creating ARN
shell: bash
run: |
echo "ARN is arn:aws:iam::${{ env.INFRA_ACCOUNT }}:role/aws-github-role"
Now this give me output as for stage as "dev"
ARN is arn:aws:iam::12345
:role/aws-github-role
How to fix the line break after the account number?
You are specifying the environment variable as follows:
INFRA_ACCOUNT: >
${{ fromJson('{
"dev": "12345",
"preprd": "678234",
"prd": "91056"
}')[github.event.inputs.stage] }}
With the > operator, it adds a line break at the end.
However, this can be configured using the Block Chomping Indicator.
The default behavior ("Clipping") preserves the last line break without any trailing empty lines.
You can change this to Stripping by using >- instead of >. This results in the final line break being removed:
INFRA_ACCOUNT: >-
${{ fromJson('{
"dev": "12345",
"preprd": "678234",
"prd": "91056"
}')[github.event.inputs.stage] }}

Using variables in DevOps YAML Pipelines

I'm trying to add some conditional logic to my Azure DevOps pipeline to perform actions based on if there are pending changes in the Git repository. I've created a PowerShell script to check for changes and set a variable, which is working:
$gitStatus = (git status --porcelain) | Out-String
if ($gitStatus) {
Write-Host "##vso[task.setvariable variable=changes;]true"
Write-Host "##[debug]Changes found"
} else {
Write-Host "##vso[task.setvariable variable=changes;]false"
Write-Host "##[debug]No changes found"
}
I can then output the resulting value of "changes" in my pipeline as follows:
- script: echo Output - $(changes)
This returns "Output - true" as expected
If I then add the following to my YAML...
- ${{ if eq(variables.changes, true) }}:
- script: echo Changes = True
- ${{ else }}:
- script: echo Changes = False
I always receive "Changes = False"
Any help would be gratefully received.
Thanks to input from 4c74356b41 I've come up with the following solution:
- script: echo Changes = True
condition: eq(variables.changes, true)
- script: echo Changes = False
condition: ne(variables.changes, true)
The final pipeline will call templates with the required actions to be performed based on the result of the check, but the above works well enough to prove the concept.

Azure Devops Yaml Python Script task: PythonScriptV0 , how to pass arrays as the field supports only a strings

I am trying to pass to a python scripts 3 parameters one of them is an array, this works when i run the script locally, i am using sys.argv to achieve this.
However the argument field support only strings as far i can see. How can i go around this any ideas? Thanks
the array is ${{ parameters.packageVersion }}
Code:
- task: PythonScript#0
displayName: 'Modify ansible inventory files (wms_common.yml) with deployed versions'
inputs:
scriptSource: filePath
scriptPath: deployment/s/azure-devops/scripts/script.py
arguments: |
../../inventories/${{ variables.inventory }}/group_vars/all/wms_common.yml
../../inventories/central/${{ variables.inventory }}/group_vars/all/wms_common.yml
${{ parameters.packageVersion }}
Error:
/azure-devops/wms.full.pipeline.yml (Line: 95, Col: 18): Unable to convert from Array to String. Value: Array
Edit: Reframed question
I think the below YAML will help you convert the array to String objects and use them.
variables:
myVariable1: 'value1'
myVariable2: 'value2'
system.debug: true
parameters:
- name: InstanceArgs
type: object
default: [1,2]
steps:
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
import argparse
parse = argparse.ArgumentParser(description="Test")
parse.add_argument("test1")
parse.add_argument("test2")
args = parse.parse_args()
print(args.test1)
print(args.test2)
print('this is a test.')
# arguments: $(myVariable1) $(myVariable2)'
arguments: ${{join(' ',parameters.InstanceArgs)}}
You need to use the join expression in YAML to get the elements:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#join

Getting `Argument list too long` in GitHub Actions

I am following HashiCorp's learning guide on how to set up GitHub Actions and terraform. All is running great besides the step to update the PR with the Terraform Plan.
I am hitting the following error:
An error occurred trying to start process '/home/runner/runners/2.287.1/externals/node12/bin/node' with working directory '/home/runner/work/ccoe-aws-ou-scp-manage/ccoe-aws-ou-scp-manage'. Argument list too long
The code I am using is:
- uses: actions/github-script#0.9.0
if: github.event_name == 'pull_request'
env:
PLAN: "terraform\n${{ steps.plan.outputs.stdout }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Format and Style ๐Ÿ–Œ\`${{ steps.fmt.outcome }}\`
#### Terraform Initialization โš™๏ธ\`${{ steps.init.outcome }}\`
#### Terraform Plan ๐Ÿ“–\`${{ steps.plan.outcome }}\`
<details><summary>Show Plan</summary>
\`\`\`${process.env.PLAN}\`\`\`
</details>
*Pusher: #${{ github.actor }}, Action: \`${{ github.event_name }}\`*`;
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})
A clear COPY/Paste from the docs: https://learn.hashicorp.com/tutorials/terraform/github-actions
I have tried with
actions/github-script version 5 and 6 and still the same problem, But when I copy paste the plan all is great. If I do not use the output variable and use some place holder text for the body all is working great. I can see that the step.plan.outputs.stdout is Ok if I print only that.
I will be happy to share more details if needed.
I also encountered a similar issue.
I seem github-script can't give to argument for too long script.
reference:
https://github.com/robburger/terraform-pr-commenter/issues/6#issuecomment-826966670
https://github.community/t/maximum-length-for-the-comment-body-in-issues-and-pr/148867
my answer:
- name: truncate terraform plan result
run: |
plan=$(cat <<'EOF'
${{ format('{0}{1}', steps.plan.outputs.stdout, steps.plan.outputs.stderr) }}
EOF
)
echo "${plan}" | grep -v 'Refreshing state' >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
- name: create comment from plan result
uses: actions/github-script#0.9.0
if: github.event_name == 'pull_request'
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Initialization โš™๏ธ\`${{ steps.init.outcome }}\`
#### Terraform Plan ๐Ÿ“–\`${{ steps.plan.outcome }}\`
<details><summary>Show Plan</summary>
\`\`\`\n
${ process.env.PLAN }
\`\`\`
</details>
*Pusher: #${{ github.actor }}, Action: \`${{ github.event_name }}\`, Working Directory: \`${{ inputs.TF_WORK_DIR }}\`, Workflow: \`${{ github.workflow }}\`*`;
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})```
Based on #Zambozo's hint in the comments, this worked for me great:
- name: Terraform Plan
id: plan
run: terraform plan -no-color -input=false
- name: generate random delimiter
run: echo "DELIMITER=$(uuidgen)" >> $GITHUB_ENV
- name: truncate terraform plan result
run: |
echo "PLAN<<${{ env.DELIMITER }}" >> $GITHUB_ENV
echo '[maybe truncated]' >> $GITHUB_ENV
tail --bytes=10000 <<'${{ env.DELIMITER }}' >> $GITHUB_ENV
${{ format('{0}{1}', steps.plan.outputs.stderr, steps.plan.outputs.stdout) }}
${{ env.DELIMITER }}
echo >> $GITHUB_ENV
echo "${{ env.DELIMITER }}" >> $GITHUB_ENV
- name: post plan as sticky comment
uses: marocchino/sticky-pull-request-comment#v2
with:
header: plan
message: |
#### Terraform Plan ๐Ÿ“–\`${{ steps.plan.outcome }}\`
<details><summary>Show Plan</summary>
```
${{ env.PLAN }}
```
</details>
Notably GitHub does have an upper limit on comment size, so this only displays the last 10kB of the plan (showing the summary and warnings).
This also implements secure heredoc delimiting to avoid malicious output escaping.
Also note that the empty lines before and after the triplebacktics in the message are significant to avoid destroying the formatting.

AZP: Is there a best practice to be able to "namespace" script tasks in yaml templates for usage of variables?

In Azure Pipelines: my main problem is, if I create a yml template and have some logic inside that template in a script task where I want to set a variable, i need the
name: "pseudonamespace" to reference that variable further down in that template via
$(pseudonamespace.variablename)
An example, where the script part does nothing overtly useful, but should demonstrate my problem:
mytemplate.yml:
parameters:
- name: isWindowsOnTarget
type: boolean
default: true
steps:
- script: |
if [ "${{lower(parameters.isWindowsOnTarget)}}" == "true" ]; then
delimiter="\\"
else
delimiter="/"
fi
echo "##vso[task.setvariable variable=myCoolVariable;isOutput=true]$delimiter"
name: MyFakeNameSpace
...
- task: SomeTask#0
inputs:
myInput: $(MyFakeNameSpace.myCoolVariable)
This codeblock works; but only, if, in a job, I only instanciate it once:
- template: mytemplate.yml#templates
parameters:
isWindowsOnTarget: true
If I would need that template twice, differently parameterized, I get the error that the name of the script block needs to be unique.
Is there any useful possibility I'm not currently thinking about other than to have an extra parameter for the template that I could basically just call "UniqueNamespace"?
There is no much space to move. Your task needs a unique name as later as you mention for output parameters it works like a namespace. So the best and the only way you have is to provide another parameter which would be task name.
parameters:
- name: isWindowsOnTarget
type: boolean
default: true
- name: taskName
type: string
steps:
- script: |
if [ "${{lower(parameters.isWindowsOnTarget)}}" == "true" ]; then
delimiter="\\"
else
delimiter="/"
fi
echo "##vso[task.setvariable variable=myCoolVariable;isOutput=true]$delimiter"
name: ${{ parameters.taskName }}
...
- task: SomeTask#0
inputs:
myInput: $(MyFakeNameSpace.myCoolVariable)
and then:
- template: mytemplate.yml#templates
parameters:
isWindowsOnTarget: true
taskName: MyFakeNameSpace
- template: mytemplate.yml#templates
parameters:
isWindowsOnTarget: true
taskName: MyFakeNameSpace2
In fact when you do not provide a name Azure DevOps assign a unique name. However, in this way you don't know the name till runtime.