Microsoft Documentation explains the use of parameters in Yaml Pipeline jobs as
# File: azure-pipelines.yml
trigger:
- master
extends:
template: simple-param.yml
parameters:
yesNo: false # set to a non-boolean value to have the build fail
But instead of statically specifying the value of yesNo: I'd prefer to load it from a completely separate json config file. Preferably a json file that both my Build Job and my Application could share so that parameters specified for the Application could also be used in the Build Job.
Thus the question:
How do I load values from a .json file into a Devops Yaml Pipeline Parameter?
I've been using this marketplace task:
https://marketplace.visualstudio.com/items?itemName=OneLuckiDev.json2variable
And it's been working great so far. Haven't tried it builds, but can't see why it wouldn't work with separate build pipelines/multi-staged builds. There are a few things you have to be aware of/stumble upon, like double escaping slashes in directory paths - and you'll have to fetch secrets from someplace else, like traditional variable groups.
Related
A step in my workflow file will return some IDs of EC2 instances in my aws account and then i set these IDs as a github output to be used in other jobs in my workflow file
I have done this in many workflows and step will return something like this:
["i-0d945b001544f2614","i-0b90ba69d37aad78c"]
However, in one workflow file github is masking the IDs as it thinks it is a secret for some reason, so it will return:
["i-***2d571abc6d7d***4ef","i-***186ce12c5cd8e744"]
Therefore i get this error message on the workflow job summary:
Skip output 'instanceIDs' since it may contain secret.
And so the other jobs in my workflow file that rely on this output will fail as github won't set an output.
I have tried to use base64 as suggested in this post but i haven't been able to get that to work
Is there any other work around?
Recently, GitHub released a new feature - configuration variables in workflows.
Configuration variables allow you to store your non sensitive data as plain text variables that can be reused across your workflows in your repository or organization.
You can define variables at Organization, Repository, or Environment level based on your requirement.
These variables are accessible from the workflow by vars context.
Example:
jobs:
display-variables:
runs-on: ${{ vars.RUNNER }}
steps:
- name: Use variables
run: |
echo "Repository variable : ${{ vars.REPOSITORY_VAR }}"
echo "Organization variable : ${{ vars.ORGANIZATION_VAR }}"
In this example, we have the following configuration variables: RUNNER, REPOSITORY_VAR, ORGANIZATION_VAR. As opposed to the repository secrets, the values of these variables won't be masked.
For more details, see the Defining configuration variables for multiple workflows.
I am migrating from .gitlab-ci.yml to azure-pipelines.yml
In one of the lines inside the .gitlab-ci.yml, there are variables
$CI_PROJECT_ID,$CI_PIPELINE_ID, $CI_JOB_ID.
I have figured out the equivalent for $CI_PROJECT_ID. In azure it is $(System.TeamProjectId)
However, need help in figuring out : $CI_PIPELINE_ID and $CI_JOB_ID
Looking forward for some suggestions
I'm not sure what these variables mean but consider this on Azure Pipelines:
$(System.DefinitionId) - The ID of the build pipeline.
$(System.JobId) - A unique identifier for a single attempt of a single job. The value is unique to the current pipeline.
$(Build.BuildId) - The ID of the record for the completed build.
All predefined variables you can find here
I have an Azure DevOps pipeline which is failing to run because it seems to be using an old connection string.
The pipeline is for a C# project, where a FileTransform task updates an appsettings.json file with variables set on the pipeline.
The variables were recently updated to use a new connection string, however, when running a Console.PrintLn before using it and viewing it on the pipeline, it shows an outdated value.
Many updates similar to this have been run in the past without issue.
I've also recently added a Powershell task to echo what the value is in the variables loaded while the pipeline is running, which does display the new value.
I've checked the order of precedence of variables and there shouldn't be any other variables being used.
There is no CacheTask being used in this pipeline.
Does anyone have any advice to remedy this? It seems that the pipeline itself is just ignoring the variables set on the pipeline.
There is a problem with the recent File transform task version v1.208.0.
It will shows the warning message and not update the variable value correctly.
Warning example:
Resource file haven't been set, can't find loc string for key: JSONvariableSubstitution
Refer to this ticket: File transform task failing to transform files, emitting "Resource file haven't been set" warnings
The issue is from Task itself instead of the Pipeline configuration. Many users have the same issue.
Workaround:
You can change to use the File Transform task Version 2 to update the appsettings.json file.
Here is an example: Please remove the content in XML Transformation rules field and set the JSON file path
I've got release pipelines defined that have worked. I've got a config transform that will write a API url to a config file (currently with a hardcoded api url).
What I'd like to do is be able to have the config be re-written based on the agent its being deployed on.
eg. if the machine being deployed to is TEST-1, I'd like to write https://TEST-1.somedomain.com/api into a config using that transform step.
The .somedomain.com/api can be static.
I've tried modifying the pipeline variable's value to be https://${{Environment.Name}}.somedomain.com/api, but it just replaces the API_URL in the config with that literal string (does not populate machine name in that variable).
Being that variables are the source of value that is being written to configs during the transform, I'm struggling to see another way to do this.
some gotchas
Using non yaml pipeline definitions (I know I saw people put logic in variable definitions within yaml pipelines)
Can't just use localhost, as the configuration is being read into a javascript rich app that would have js trying to connect to localhost vs trying to connect to the server.
I'm interested in any ways I could solve this problem
${{Environment.Name}} is not valid syntax for either YAML or classic pipelines.
In classic pipelines it would be $(Environment.Name).
In YAML, $(Environment.Name) or ${{ variables['Environment.Name'] }} would work.
We have a requirement to somehow pass a dynamic runtime parameter to a pipeline task.
For example below paramater APPROVAL would be different for each run of the task.
This APPROVAL parameter is for the change and release number so that the task can tag it on the terraform resources created for audit purposes.
Been searching the web for a while but with no luck in finding a solution, is this possible in a concourse pipeline or best practice?
- task: plan-terraform
file: ci/concourse-jobs/pipelines/tasks/terraform/plan-terraform.yaml
params:
ENV: dev
APPROVAL: test
CHANNEL: Developement
GITLAB_KEY: ((gitlab_key))
REGION: eu-west-2
TF_FOLDER: terraform/squid
input_mapping:
ci: ci
tf: squid
output_mapping:
plan: plan
tags:
- dev
From https://concourse-ci.org/tasks.html:
ideally tasks are pure functions: given the same set of inputs, it should either always succeed with the same outputs or always fail.
A dynamic parameter would break that contract and produce different outputs from the same set of inputs. Could you possibly make APPROVAL an input? Then you'd maintain your build traceability. If it's a (file) input, you could then load it into a variable:
APPROVAL=$(cat <filename>)