I need to use github actions for deploying my stack to many environments, e.g., development and production. My prior implementation won't work anymore, and whilst I have a new working workflow, setting env vars for multiple envs seems repetitive and makes me think there must be a better way.
What I had before:
jobs:
build-and-deploy:
name: Build Container and Deploy
runs-on: ubuntu-latest
steps:
- name: Set env to development
if: ${{ !endsWith(github.ref, '/main') }}
run: |
echo "GCP_PROJECT=my-project-dev" >> $GITHUB_ENV
echo "NODE_ENV=development" >> $GITHUB_ENV
- name: Set env to production
if: ${{ endsWith(github.ref, '/main') }}
run: |
echo "GCP_PROJECT=my-project-prod" >> $GITHUB_ENV
echo "NODE_ENV=production" >> $GITHUB_ENV
...
What I have now:
jobs:
build-and-deploy:
name: Build Container and Deploy
runs-on: ubuntu-latest
env:
GCP_PROJECT: ${{ (!endsWith(github.ref, '/main') && 'my-project-dev') || 'my-project-prod' }}
NODE_ENV: ${{ (!endsWith(github.ref, '/main') && 'development') || 'production' }}
steps:
...
Obviously, it doesn't seem so bad in the example, but for many more env vars the ${{ (!endsWith(github.ref, '/main') && 'dev-value') || 'prod-value' }} seems noisy and dirty.
So, what would be the proper way for doing this?
I would structure your workflow differently and use one step for production and one step for development.
While this might seem repetitive, it makes it very clear what variables you're passing in the case of production and which ones for development.
This also means you only need one if and it's immediately clear to the reader what happens only on the main branch.
NB: you're condition endsWith(/main) would also trigger on a branch named feature/main. As I have shown below, I'd recommend using the entire reference.
steps:
- name: Deploy to Development
if: github.ref != 'refs/head/main'
uses: my-action-to-deploy
env:
GCP_PROJECT: my-dev-project
NODE_ENV: development
- name: Deploy to Production
if: github.ref == 'refs/head/main'
uses: my-action-to-deploy
env:
GCP_PROJECT: my-prod-project
NODE_ENV: production
I read some similar posts but none seem to answer this question. I can set individual GitHub secrets into environment variables in an Action if I know the name of the secret:
env:
PW_ID0007: "${{secrets.PW_ID0007}}"
How can I expose all secrets as environment variables without knowing their names (either in bulk or some way to iterate through them and set them individually?)
There is a way to do that. Please check here
- name: view the secrets context
shell: bash
run: echo "$SECRETS_CONTEXT"
env:
SECRETS_CONTEXT: ${{ toJson(secrets) }}
In that way you will expose all secrets without knowing names:
And know what you need is go through this json using for instance jq and set them as env variable suing following syntax:
echo "variable_name=variable_value" >> $GITHUB_ENV
I created an action exactly for that - takes all the secrets and exports them to environment variables.
An example would be:
- run: echo "Value of MY_SECRET1: $MY_SECRET1"
env:
MY_SECRET1: ${{ secrets.MY_SECRET1 }}
MY_SECRET2: ${{ secrets.MY_SECRET2 }}
MY_SECRET3: ${{ secrets.MY_SECRET3 }}
MY_SECRET4: ${{ secrets.MY_SECRET4 }}
MY_SECRET5: ${{ secrets.MY_SECRET5 }}
MY_SECRET6: ${{ secrets.MY_SECRET6 }}
...
You could convert it to:
- uses: oNaiPs/secrets-to-env-action#v1
with:
secrets: ${{ toJSON(secrets) }}
- run: echo "Value of MY_SECRET1: $MY_SECRET1"
Link to the action, which contains more documentation about configuration: https://github.com/oNaiPs/secrets-to-env-action
In its current format, you can define git hub secrets in the repository UI, and add them to your github actions CI like the following:
- name: test secrets
shell: bash
run:
env:
SECRET_1: ${{ secrets.secret_1 }}
SECRET_2: ${{ secrets.secret_2 }}
However this approach is cumbersome if you want to dynamically add all secrets attached to that environment, especially if you secrets change regularly. With the above methodology each alteration would require a code deploy which is inconvenient. Has anyone come up with a solution? Or is there any built in syntax which will load all secrets? Or secrets that follow a certain pattern?
No this is not possible (and rather won't be in the future). As it is written here you have to clearly show what secret you want to use in your workflow like here:
steps:
- name: Hello world action
with: # Set the secret as an input
super_secret: ${{ secrets.SuperSecret }}
env: # Or as an environment variable
super_secret: ${{ secrets.SuperSecret }}
This is done by purpose. As authors wanted to force developers to use exactly secrets what are needed. This just address security concern.
It is possible to interpolate secret ID so say you are conditioning your secret based on input upon triggering the workflow, here is how I implemented:
workflow_dispatch:
inputs:
environment:
type: choice
description: Deployment environment
required: true
options:
- staging
- production
jobs:
steps:
# this is because the secrets in my GH is on uppercase so I need to manipulate the whole string based on the input
- id: AWSAccessKeyId
run: export AWSAKI=${{ format('AWS_ACCESS_KEY_ID_GREEN_{0}', github.event.inputs.environment) }} && echo "::set-output name=AWSAccessKeyIdValue::${AWSAKI^^}"
- id: AWSSecretAccessKey
run: export AWSSAK=${{ format('AWS_SECRET_ACCESS_KEY_GREEN_{0}', github.event.inputs.environment) }} && echo "::set-output name=AWSSecretAccessKeyValue::${AWSSAK^^}"
- run : npm run deploy-${{ github.event.inputs.environment }}
env:
AWS_ACCESS_KEY_ID: ${{ secrets[steps.AWSAccessKeyId.outputs.AWSAccessKeyIdValue] }}
AWS_SECRET_ACCESS_KEY: ${{ secrets[steps.AWSSecretAccessKey.outputs.AWSSecretAccessKeyValue] }}
AWS_DEFAULT_REGION: ${{ github.event.inputs.aws_region }}
In the settings of my repository I created 2 environments: development and production. I also have 2 branches of the same name. I want my workflow to execute in the corresponding environment (so as to grab the correct git secrets).
This is what I have:
jobs:
branch-based-execution:
name: Run external workflow
environment: ${{ github.ref }}
runs-on: ubuntu-latest
steps:
- name: Echo env var
env:
my_var: ${{ secrets.my_var }}
shell: bash
run: echo $my_var
However, I get the following error on the environment line:
Unrecognized named-value: 'github'. Located at position 1 within expression: github.ref
Looks like I can't use github context to set the environment attribute. Is there a way to set the environment of the workflow dynamically? Thanks!
You can set it dynamically, you just have to specify that you are setting the name
jobs:
branch-based-execution:
name: Run external workflow
environment:
name: ${{ github.ref }}
I'm trying to dynamically pull back a GitHub secret using GitHub Actions at runtime:
Let's say I have two GitHub Secrets:
SECRET_ORANGES : "This is an orange secret"
SECRET_APPLES : "This is an apple secret"
In my GitHub Action, I have another env variable which will differ between branches
env:
FRUIT_NAME: APPLES
Essentially I want to find a way to do some sort of variable substitution to get the correct secret. So in one of my child jobs, I want to do something like:
env:
FRUIT_SECRET: {{ 'SECRET_' + env.FRUIT_NAME }}
I've tried the following approaches with no luck:
secrets['SECRET_$FRUIT_NAME'] }}
I even tried a simpler approach without concatenation just to try and get it working
secrets['$FRUIT_NAME'] }}
and
{{ secrets.$FRUIT_NAME }}
None of the above worked.
Apologies if I have not explained this very well. I tried to keep my example as simple as possible.
Anyone have any idea of how to achieve this?
Alternatively, what I am trying to do is to store secrets on a per-branch basis
For example:
In customer1 code branch:
SECRET_CREDENTIAL="abc123"
In customer2 code branch:
SECRET_CREDENTIAL="def456"
Then I can access the correct value for SECRET_CREDENTIAL depending on which branch I am in.
Thanks!
Update: I'm getting a bit closer to what I am trying to achieve:
name: Test
env:
CUSTOMER: CUSTOMER1
jobs:
build:
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ env.CUSTOMER }}_AWS_ACCESS_KEY_ID
steps:
- uses: actions/checkout#v2
- run: |
AWS_ACCESS_KEY_ID=${{ secrets[env.AWS_ACCESS_KEY_ID] }}
echo "AWS_ACCESS_KEY_ID = $AWS_ACCESS_KEY_ID"
There is a much cleaner option to achieve this using the format function.
Given set secrets DEV_A and TEST_A, the following two jobs will use those two secrets:
name: Secrets
on: [push]
jobs:
dev:
name: dev
runs-on: ubuntu-18.04
env:
ENVIRONMENT: DEV
steps:
- run: echo ${{ secrets[format('{0}_A', env.ENVIRONMENT)] }}
test:
name: test
runs-on: ubuntu-18.04
env:
ENVIRONMENT: TEST
steps:
- run: echo ${{ secrets[format('{0}_A', env.ENVIRONMENT)] }}
This also works with input provided through manual workflows (the workflow_dispatch event):
name: Secrets
on:
workflow_dispatch:
inputs:
env:
description: "Environment to deploy to"
required: true
jobs:
secrets:
name: secrets
runs-on: ubuntu-18.04
steps:
- run: echo ${{ secrets[format('{0}_A', github.event.inputs.env)] }}
Update - July 2021
I found a better way to prepare dynamic secrets in a job, and then consume those secrets as environment variables in other jobs.
Here's how it looks like in GitHub Actions.
My assumption is that each secret should be fetched according to the branch name. I'm getting the branch's name with this action rlespinasse/github-slug-action.
Go through the inline comments to understand how it all works together.
name: Dynamic Secret Names
# Assumption:
# You've created the following GitHub secrets in your repository:
# AWS_ACCESS_KEY_ID_master
# AWS_SECRET_ACCESS_KEY_master
on:
push:
env:
AWS_REGION: "eu-west-1"
jobs:
prepare:
name: Prepare
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout#v2
- name: Inject slug/short variables
uses: rlespinasse/github-slug-action#v3.x
- name: Prepare Outputs
id: prepare-step
# Sets this step's outputs, that later on will be exported as the job's outputs
run: |
echo "::set-output name=aws_access_key_id_name::AWS_ACCESS_KEY_ID_${GITHUB_REF_SLUG}";
echo "::set-output name=aws_secret_access_key_name::AWS_SECRET_ACCESS_KEY_${GITHUB_REF_SLUG}";
# Sets this job's, that will be consumed by other jobs
# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idoutputs
outputs:
aws_access_key_id_name: ${{ steps.prepare-step.outputs.aws_access_key_id_name }}
aws_secret_access_key_name: ${{ steps.prepare-step.outputs.aws_secret_access_key_name }}
test:
name: Test
# Must wait for `prepare` to complete so it can use `${{ needs.prepare.outputs.{output_name} }}`
# https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#needs-context
needs:
- prepare
runs-on: ubuntu-20.04
env:
# Get secret names
AWS_ACCESS_KEY_ID_NAME: ${{ needs.prepare.outputs.aws_access_key_id_name }}
AWS_SECRET_ACCESS_KEY_NAME: ${{ needs.prepare.outputs.aws_secret_access_key_name }}
steps:
- uses: actions/checkout#v2
- name: Test Application
env:
# Inject secret values to environment variables
AWS_ACCESS_KEY_ID: ${{ secrets[env.AWS_ACCESS_KEY_ID_NAME] }}
AWS_SECRET_ACCESS_KEY: ${{ secrets[env.AWS_SECRET_ACCESS_KEY_NAME] }}
run: |
printenv | grep AWS_
aws s3 ls
Update - August 2020
Following some hands-on experience with this project terraform-monorepo, here's an example of how I managed to use secret names dynamically
Secrets names are aligned with environments names and branches names - development, staging and production
$GITHUB_REF_SLUG comes from the Slug GitHub Action which fetches the name of the branch
The commands which perform the parsing are
- name: set-aws-credentials
run: |
echo "::set-env name=AWS_ACCESS_KEY_ID_SECRET_NAME::AWS_ACCESS_KEY_ID_${GITHUB_REF_SLUG}"
echo "::set-env name=AWS_SECRET_ACCESS_KEY_SECRET_NAME::AWS_SECRET_ACCESS_KEY_${GITHUB_REF_SLUG}"
- name: terraform-apply
run: |
export AWS_ACCESS_KEY_ID=${{ secrets[env.AWS_ACCESS_KEY_ID_SECRET_NAME] }}
export AWS_SECRET_ACCESS_KEY=${{ secrets[env.AWS_SECRET_ACCESS_KEY_SECRET_NAME] }}
Full example
name: pipeline
on:
push:
branches: [development, staging, production]
paths-ignore:
- "README.md"
jobs:
terraform:
runs-on: ubuntu-latest
env:
### -----------------------
### Available in all steps, change app_name to your app_name
TF_VAR_app_name: tfmonorepo
### -----------------------
steps:
- uses: actions/checkout#v2
- name: Inject slug/short variables
uses: rlespinasse/github-slug-action#v2.x
- name: prepare-files-folders
run: |
mkdir -p ${GITHUB_REF_SLUG}/
cp live/*.${GITHUB_REF_SLUG} ${GITHUB_REF_SLUG}/
cp live/*.tf ${GITHUB_REF_SLUG}/
cp live/*.tpl ${GITHUB_REF_SLUG}/ 2>/dev/null || true
mv ${GITHUB_REF_SLUG}/backend.tf.${GITHUB_REF_SLUG} ${GITHUB_REF_SLUG}/backend.tf
- name: install-terraform
uses: little-core-labs/install-terraform#v1
with:
version: 0.12.28
- name: set-aws-credentials
run: |
echo "::set-env name=AWS_ACCESS_KEY_ID_SECRET_NAME::AWS_ACCESS_KEY_ID_${GITHUB_REF_SLUG}"
echo "::set-env name=AWS_SECRET_ACCESS_KEY_SECRET_NAME::AWS_SECRET_ACCESS_KEY_${GITHUB_REF_SLUG}"
- name: terraform-apply
run: |
export AWS_ACCESS_KEY_ID=${{ secrets[env.AWS_ACCESS_KEY_ID_SECRET_NAME] }}
export AWS_SECRET_ACCESS_KEY=${{ secrets[env.AWS_SECRET_ACCESS_KEY_SECRET_NAME] }}
cd ${GITHUB_REF_SLUG}/
terraform version
rm -rf .terraform
terraform init -input=false
terraform get
terraform validate
terraform plan -out=plan.tfout -var environment=${GITHUB_REF_SLUG}
terraform apply -auto-approve plan.tfout
rm -rf .terraform
After reading this - Context and expression syntax for GitHub Actions
, focusing on
env object, I found out that:
As part of an expression, you may access context information using one of two syntaxes.
Index syntax: github['sha']
Property dereference syntax: github.sha
So the same behavior applies to secrets, you can do secrets[secret_name], so you can do the following
- name: Run a multi-line script
env:
SECRET_NAME: A_FRUIT_NAME
run: |
echo "SECRET_NAME = $SECRET_NAME"
echo "SECRET_NAME = ${{ env.SECRET_NAME }}"
SECRET_VALUE=${{ secrets[env.SECRET_NAME] }}
echo "SECRET_VALUE = $SECRET_VALUE"
Which results in
SECRET_NAME = A_FRUIT_NAME
SECRET_NAME = A_FRUIT_NAME
SECRET_VALUE = ***
Since the SECRET_VALUE is redacted, we can assume that the real secret was fetched.
Things that I learned -
You can't reference env from another env, so this won't work
env:
SECRET_PREFIX: A
SECRET_NAME: ${{ env.SECRET_PREFIX }}_FRUIT_NAME
The result of SECRET_NAME is _FRUIT_NAME, not good
You can use context expressions in your code, not only in env, you can see that in SECRET_VALUE=${{ secrets[env.SECRET_NAME] }}, which is cool
And of course - here's the workflow that I tested - https://github.com/unfor19/gha-play/runs/595345435?check_suite_focus=true - check the Run a multi-line script step
In case this can help, after reading the above answers which truly helped, the strategy I decided to use consists of storing my secrets as follow:
DB_USER_MASTER
DB_PASSWORD_MASTER
DB_USER_TEST
DB_PASSWORD_TEST
Where MASTER is the master branch for the prod environment and TEST is the test branch for the test environment.
Then, using the suggested solutions in this thread, the key is to dynamically generate the keys of the secrets variable. Those keys are generated via an intermediate step (called vars in the sample below) using outputs:
name: Pulumi up
on:
push:
branches:
- master
- test
jobs:
up:
name: Update
runs-on: ubuntu-latest
steps:
- name: Create variables
id: vars
run: |
branch=${GITHUB_REF##*/}
echo "::set-output name=DB_USER::DB_USER_${branch^^}"
echo "::set-output name=DB_PASSWORD::DB_PASSWORD_${branch^^}"
- uses: actions/checkout#v2
with:
fetch-depth: 1
- uses: docker://pulumi/actions
with:
args: up -s ${GITHUB_REF##*/} -y
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
DB_USER: ${{ secrets[steps.vars.outputs.DB_USER] }}
DB_PASSWORD: ${{ secrets[steps.vars.outputs.DB_PASSWORD] }}
Notice the hack to get the branch on uppercase: ${branch^^}. This is required because GitHub forces secrets to uppercase.
I was able to achieve this using the workflow name as the branch specific variable.
For each branch I create, I simply update this single value at the top of the YML file, then add GitHub Secrets to match the workflow name:
name: CUSTOMER1
jobs:
build:
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ github.workflow }}_AWS_ACCESS_KEY_ID
steps:
- uses: actions/checkout#v2
- run: echo "::set-env name=AWS_ACCESS_KEY_ID::${{ secrets[env.AWS_ACCESS_KEY_ID] }}"
- run: echo $AWS_ACCESS_KEY_ID
Don't use ::set-env, it is depreacated.
Use instead
echo "env_key=env_value" >> $GITHUB_ENV
You can set the env variable on a branch basis by setting env as in this example.
Suppose you have at least two secrets with different prefixes in your repository, like this: (DEV_SERVER_IP, OTHER_SERVER_IP)
I use 'format', '$GITHUB_ENV' which are workflow commands and function provide on Github.
- name: Set develop env
if: ${{ github.ref == 'refs/heads/develop' }}
run: echo "branch_name=DEVELOP" >> $GITHUB_ENV
- name: Set other env
if: ${{ github.ref == 'refs/heads/other' }}
run: echo "branch_name=OTHER" >> $GITHUB_ENV
- name: SSH Test
env:
SERVER_IP: ${{ secrets[format('{0}_SERVER_IP', env.branch_name)] }}
run: ssh -T user#$SERVER_IP
New solution as of December 2020
If you are reading this question because you need to use different secret values based on the environment you are deploying to, GitHub Actions now has a new feature called "Environments" in beta: https://docs.github.com/en/free-pro-team#latest/actions/reference/environments
This allows us to define environment secrets, and allow only jobs that are assigned to the environment to access them. This not only leads to better user experience as a developer, but also to better security and isolation of different deployment jobs.
Below is an example for how to dynamically determine the environment that should be used, based on the branch name:
jobs:
get-environment-name:
name: "Extract environment name"
runs-on: ubuntu-latest
outputs:
environment: ${{ steps.extract.outputs.environment }}
steps:
- id: extract
# You can run any logic you want here to map refs to environment names.
# The GITHUB_REF will look like this: refs/heads/my-branchname
# The example logic here simply removes "refs/heads/deploy-" from the beginning,
# so a branch name deploy-prod would be mapped to the environment "prod"
run: echo "::set-output name=environment::$(echo $GITHUB_REF | sed -e '/^refs\/heads\/deploy-\(.*\)$/!d;s//\1/')"
- env:
EXTRACTED: ${{ steps.extract.outputs.environment }}
run: 'echo "Extracted environment name: $EXTRACTED"'
deploy:
name: "Deploy"
if: ${{ github.event_name == 'push' && needs.get-environment-name.outputs.environment }}
needs:
- get-environment-name
# - unit-tests
# - frontend-tests
# ... add your unit test jobs here, so they are executed before deploying anything
environment: ${{ needs.get-environment-name.outputs.environment }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
# ... Run your deployment actions here, with full access to the environment's secrets
Note that in the if: clause of the deployment job, it's not possible to use any environment variables or bash scripts. So using a previous job that extracts the environment name from the branch name is the simplest I could make it at the current time.
I came across this question when trying to implement environment-based secret selection for a Github action.
This variable-mapper action (https://github.com/marketplace/actions/variable-mapper) implements the desired concept of mapping a key variable or an environment name to secrets or other pre-defined values.
The example use of it shows this:
on: [push]
name: Export variables corresponding to regular expression-matched keys
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: kanga333/variable-mapper#v1
with:
key: ${{GITHUB_REF#refs/heads/}}
map: |
{
"master": {
"environment": "production",
"AWS_ACCESS_KEY_ID": ${{ secrets.PROD_AWS_ACCESS_KEY_ID }},
"AWS_SECRET_ACCESS_KEY": ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
},
"staging": {
"environment": "staging",
"AWS_ACCESS_KEY_ID": ${{ secrets.STG_AWS_ACCESS_KEY_ID }},
"AWS_SECRET_ACCESS_KEY": ${{ secrets.STG_AWS_ACCESS_KEY_ID }}
},
".*": {
"environment": "development",
"AWS_ACCESS_KEY_ID": ${{ secrets.DEV_AWS_ACCESS_KEY_ID }},
"AWS_SECRET_ACCESS_KEY": ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
}
}
- name: Echo environment
run: echo ${{ env.environment }}