So what I am trying to do is to organize my ci workflow. I have a self-hosted runner that is my personal computer. Since the workflow files is getting too large I am thinking about reducing the size by having different workflow files instead.
so this is my files so far:
name: "Pipeline"
run-name: ${{ github.actor }} just pushed.
on:
schedule:
- cron: "0 2 * * *"
push:
branches-ignore:
- "docs/*"
tags:
- "v*"
workflow_dispatch:
inputs:
cc_platform:
description: "Input of branch name to build metalayers."
required: true
default: "dev"
jobs:
build:
runs-on: self-hosted
name: Build job.
steps:
- name: Checking out the repository.
uses: actions/checkout#v3
- name: Settings Credentials.
uses: webfactory/ssh-agent#v0.6.0
with:
ssh-private-key: ${{ secrets.SSH_KEY_CC }}
- name: Creating config file.
run:
touch conf
- name: Removing old folders & running build script.
run: |
if [[ -d "my_folder" ]]; then rm -rf my_folder; fi
./build
- name: Generate a zip file of artifacts.
uses: actions/upload-artifact#v3
with:
name: art_${{ github.sha }}
path: |
.*.rootfs.wic.xz
.*.rootfs.manifest
if-no-files-found: error
so what I want to do it to have the artifacts and the build step in their own workflow file. But it does give me som concersn.
Will each workflow file get run in their own runner?
Do they share memory?
I'm learning CI CD to deploy REST API to VPS using Github Actions.
I am confused about how to make a dotenv file so that it is on the VPS server using Github Actions, because this REST API requires a Jsonwebtoken key and I have to insert it in the .env file. I've tried several ways but nothing works.
If you have the answer, please modify my .yml file below and include your answer so I can understand it clearly.
I really appreciate any answer.
name: Node.js CICD
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14.x]
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- name: create env file
run: |
touch .env
echo JWT_ACCESS_TOKEN_SECRET=${{ secrets.JWT_ACCESS_TOKEN_SECRET }} >> .env
- name: install and test
run: |
npm i
npm run build --if-present
npm run test
env:
CI: true
deploy:
needs: [test]
runs-on: ubuntu-latest
steps:
- name: deploy with SSH
uses: appleboy/ssh-action#master
with:
host: ${{ secrets.HOST }}
username: ${{ secrets.USERNAME }}
key: ${{ secrets.PRIVATE_KEY }}
port: 22
script: |
cd ~/apps/routeros-api
git pull origin master
npm i --production
pm2 restart routeros-api
I'm trying to dynamically pull back a GitHub secret using GitHub Actions at runtime:
Let's say I have two GitHub Secrets:
SECRET_ORANGES : "This is an orange secret"
SECRET_APPLES : "This is an apple secret"
In my GitHub Action, I have another env variable which will differ between branches
env:
FRUIT_NAME: APPLES
Essentially I want to find a way to do some sort of variable substitution to get the correct secret. So in one of my child jobs, I want to do something like:
env:
FRUIT_SECRET: {{ 'SECRET_' + env.FRUIT_NAME }}
I've tried the following approaches with no luck:
secrets['SECRET_$FRUIT_NAME'] }}
I even tried a simpler approach without concatenation just to try and get it working
secrets['$FRUIT_NAME'] }}
and
{{ secrets.$FRUIT_NAME }}
None of the above worked.
Apologies if I have not explained this very well. I tried to keep my example as simple as possible.
Anyone have any idea of how to achieve this?
Alternatively, what I am trying to do is to store secrets on a per-branch basis
For example:
In customer1 code branch:
SECRET_CREDENTIAL="abc123"
In customer2 code branch:
SECRET_CREDENTIAL="def456"
Then I can access the correct value for SECRET_CREDENTIAL depending on which branch I am in.
Thanks!
Update: I'm getting a bit closer to what I am trying to achieve:
name: Test
env:
CUSTOMER: CUSTOMER1
jobs:
build:
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ env.CUSTOMER }}_AWS_ACCESS_KEY_ID
steps:
- uses: actions/checkout#v2
- run: |
AWS_ACCESS_KEY_ID=${{ secrets[env.AWS_ACCESS_KEY_ID] }}
echo "AWS_ACCESS_KEY_ID = $AWS_ACCESS_KEY_ID"
There is a much cleaner option to achieve this using the format function.
Given set secrets DEV_A and TEST_A, the following two jobs will use those two secrets:
name: Secrets
on: [push]
jobs:
dev:
name: dev
runs-on: ubuntu-18.04
env:
ENVIRONMENT: DEV
steps:
- run: echo ${{ secrets[format('{0}_A', env.ENVIRONMENT)] }}
test:
name: test
runs-on: ubuntu-18.04
env:
ENVIRONMENT: TEST
steps:
- run: echo ${{ secrets[format('{0}_A', env.ENVIRONMENT)] }}
This also works with input provided through manual workflows (the workflow_dispatch event):
name: Secrets
on:
workflow_dispatch:
inputs:
env:
description: "Environment to deploy to"
required: true
jobs:
secrets:
name: secrets
runs-on: ubuntu-18.04
steps:
- run: echo ${{ secrets[format('{0}_A', github.event.inputs.env)] }}
Update - July 2021
I found a better way to prepare dynamic secrets in a job, and then consume those secrets as environment variables in other jobs.
Here's how it looks like in GitHub Actions.
My assumption is that each secret should be fetched according to the branch name. I'm getting the branch's name with this action rlespinasse/github-slug-action.
Go through the inline comments to understand how it all works together.
name: Dynamic Secret Names
# Assumption:
# You've created the following GitHub secrets in your repository:
# AWS_ACCESS_KEY_ID_master
# AWS_SECRET_ACCESS_KEY_master
on:
push:
env:
AWS_REGION: "eu-west-1"
jobs:
prepare:
name: Prepare
runs-on: ubuntu-20.04
steps:
- uses: actions/checkout#v2
- name: Inject slug/short variables
uses: rlespinasse/github-slug-action#v3.x
- name: Prepare Outputs
id: prepare-step
# Sets this step's outputs, that later on will be exported as the job's outputs
run: |
echo "::set-output name=aws_access_key_id_name::AWS_ACCESS_KEY_ID_${GITHUB_REF_SLUG}";
echo "::set-output name=aws_secret_access_key_name::AWS_SECRET_ACCESS_KEY_${GITHUB_REF_SLUG}";
# Sets this job's, that will be consumed by other jobs
# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idoutputs
outputs:
aws_access_key_id_name: ${{ steps.prepare-step.outputs.aws_access_key_id_name }}
aws_secret_access_key_name: ${{ steps.prepare-step.outputs.aws_secret_access_key_name }}
test:
name: Test
# Must wait for `prepare` to complete so it can use `${{ needs.prepare.outputs.{output_name} }}`
# https://docs.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#needs-context
needs:
- prepare
runs-on: ubuntu-20.04
env:
# Get secret names
AWS_ACCESS_KEY_ID_NAME: ${{ needs.prepare.outputs.aws_access_key_id_name }}
AWS_SECRET_ACCESS_KEY_NAME: ${{ needs.prepare.outputs.aws_secret_access_key_name }}
steps:
- uses: actions/checkout#v2
- name: Test Application
env:
# Inject secret values to environment variables
AWS_ACCESS_KEY_ID: ${{ secrets[env.AWS_ACCESS_KEY_ID_NAME] }}
AWS_SECRET_ACCESS_KEY: ${{ secrets[env.AWS_SECRET_ACCESS_KEY_NAME] }}
run: |
printenv | grep AWS_
aws s3 ls
Update - August 2020
Following some hands-on experience with this project terraform-monorepo, here's an example of how I managed to use secret names dynamically
Secrets names are aligned with environments names and branches names - development, staging and production
$GITHUB_REF_SLUG comes from the Slug GitHub Action which fetches the name of the branch
The commands which perform the parsing are
- name: set-aws-credentials
run: |
echo "::set-env name=AWS_ACCESS_KEY_ID_SECRET_NAME::AWS_ACCESS_KEY_ID_${GITHUB_REF_SLUG}"
echo "::set-env name=AWS_SECRET_ACCESS_KEY_SECRET_NAME::AWS_SECRET_ACCESS_KEY_${GITHUB_REF_SLUG}"
- name: terraform-apply
run: |
export AWS_ACCESS_KEY_ID=${{ secrets[env.AWS_ACCESS_KEY_ID_SECRET_NAME] }}
export AWS_SECRET_ACCESS_KEY=${{ secrets[env.AWS_SECRET_ACCESS_KEY_SECRET_NAME] }}
Full example
name: pipeline
on:
push:
branches: [development, staging, production]
paths-ignore:
- "README.md"
jobs:
terraform:
runs-on: ubuntu-latest
env:
### -----------------------
### Available in all steps, change app_name to your app_name
TF_VAR_app_name: tfmonorepo
### -----------------------
steps:
- uses: actions/checkout#v2
- name: Inject slug/short variables
uses: rlespinasse/github-slug-action#v2.x
- name: prepare-files-folders
run: |
mkdir -p ${GITHUB_REF_SLUG}/
cp live/*.${GITHUB_REF_SLUG} ${GITHUB_REF_SLUG}/
cp live/*.tf ${GITHUB_REF_SLUG}/
cp live/*.tpl ${GITHUB_REF_SLUG}/ 2>/dev/null || true
mv ${GITHUB_REF_SLUG}/backend.tf.${GITHUB_REF_SLUG} ${GITHUB_REF_SLUG}/backend.tf
- name: install-terraform
uses: little-core-labs/install-terraform#v1
with:
version: 0.12.28
- name: set-aws-credentials
run: |
echo "::set-env name=AWS_ACCESS_KEY_ID_SECRET_NAME::AWS_ACCESS_KEY_ID_${GITHUB_REF_SLUG}"
echo "::set-env name=AWS_SECRET_ACCESS_KEY_SECRET_NAME::AWS_SECRET_ACCESS_KEY_${GITHUB_REF_SLUG}"
- name: terraform-apply
run: |
export AWS_ACCESS_KEY_ID=${{ secrets[env.AWS_ACCESS_KEY_ID_SECRET_NAME] }}
export AWS_SECRET_ACCESS_KEY=${{ secrets[env.AWS_SECRET_ACCESS_KEY_SECRET_NAME] }}
cd ${GITHUB_REF_SLUG}/
terraform version
rm -rf .terraform
terraform init -input=false
terraform get
terraform validate
terraform plan -out=plan.tfout -var environment=${GITHUB_REF_SLUG}
terraform apply -auto-approve plan.tfout
rm -rf .terraform
After reading this - Context and expression syntax for GitHub Actions
, focusing on
env object, I found out that:
As part of an expression, you may access context information using one of two syntaxes.
Index syntax: github['sha']
Property dereference syntax: github.sha
So the same behavior applies to secrets, you can do secrets[secret_name], so you can do the following
- name: Run a multi-line script
env:
SECRET_NAME: A_FRUIT_NAME
run: |
echo "SECRET_NAME = $SECRET_NAME"
echo "SECRET_NAME = ${{ env.SECRET_NAME }}"
SECRET_VALUE=${{ secrets[env.SECRET_NAME] }}
echo "SECRET_VALUE = $SECRET_VALUE"
Which results in
SECRET_NAME = A_FRUIT_NAME
SECRET_NAME = A_FRUIT_NAME
SECRET_VALUE = ***
Since the SECRET_VALUE is redacted, we can assume that the real secret was fetched.
Things that I learned -
You can't reference env from another env, so this won't work
env:
SECRET_PREFIX: A
SECRET_NAME: ${{ env.SECRET_PREFIX }}_FRUIT_NAME
The result of SECRET_NAME is _FRUIT_NAME, not good
You can use context expressions in your code, not only in env, you can see that in SECRET_VALUE=${{ secrets[env.SECRET_NAME] }}, which is cool
And of course - here's the workflow that I tested - https://github.com/unfor19/gha-play/runs/595345435?check_suite_focus=true - check the Run a multi-line script step
In case this can help, after reading the above answers which truly helped, the strategy I decided to use consists of storing my secrets as follow:
DB_USER_MASTER
DB_PASSWORD_MASTER
DB_USER_TEST
DB_PASSWORD_TEST
Where MASTER is the master branch for the prod environment and TEST is the test branch for the test environment.
Then, using the suggested solutions in this thread, the key is to dynamically generate the keys of the secrets variable. Those keys are generated via an intermediate step (called vars in the sample below) using outputs:
name: Pulumi up
on:
push:
branches:
- master
- test
jobs:
up:
name: Update
runs-on: ubuntu-latest
steps:
- name: Create variables
id: vars
run: |
branch=${GITHUB_REF##*/}
echo "::set-output name=DB_USER::DB_USER_${branch^^}"
echo "::set-output name=DB_PASSWORD::DB_PASSWORD_${branch^^}"
- uses: actions/checkout#v2
with:
fetch-depth: 1
- uses: docker://pulumi/actions
with:
args: up -s ${GITHUB_REF##*/} -y
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
DB_USER: ${{ secrets[steps.vars.outputs.DB_USER] }}
DB_PASSWORD: ${{ secrets[steps.vars.outputs.DB_PASSWORD] }}
Notice the hack to get the branch on uppercase: ${branch^^}. This is required because GitHub forces secrets to uppercase.
I was able to achieve this using the workflow name as the branch specific variable.
For each branch I create, I simply update this single value at the top of the YML file, then add GitHub Secrets to match the workflow name:
name: CUSTOMER1
jobs:
build:
runs-on: ubuntu-latest
env:
AWS_ACCESS_KEY_ID: ${{ github.workflow }}_AWS_ACCESS_KEY_ID
steps:
- uses: actions/checkout#v2
- run: echo "::set-env name=AWS_ACCESS_KEY_ID::${{ secrets[env.AWS_ACCESS_KEY_ID] }}"
- run: echo $AWS_ACCESS_KEY_ID
Don't use ::set-env, it is depreacated.
Use instead
echo "env_key=env_value" >> $GITHUB_ENV
You can set the env variable on a branch basis by setting env as in this example.
Suppose you have at least two secrets with different prefixes in your repository, like this: (DEV_SERVER_IP, OTHER_SERVER_IP)
I use 'format', '$GITHUB_ENV' which are workflow commands and function provide on Github.
- name: Set develop env
if: ${{ github.ref == 'refs/heads/develop' }}
run: echo "branch_name=DEVELOP" >> $GITHUB_ENV
- name: Set other env
if: ${{ github.ref == 'refs/heads/other' }}
run: echo "branch_name=OTHER" >> $GITHUB_ENV
- name: SSH Test
env:
SERVER_IP: ${{ secrets[format('{0}_SERVER_IP', env.branch_name)] }}
run: ssh -T user#$SERVER_IP
New solution as of December 2020
If you are reading this question because you need to use different secret values based on the environment you are deploying to, GitHub Actions now has a new feature called "Environments" in beta: https://docs.github.com/en/free-pro-team#latest/actions/reference/environments
This allows us to define environment secrets, and allow only jobs that are assigned to the environment to access them. This not only leads to better user experience as a developer, but also to better security and isolation of different deployment jobs.
Below is an example for how to dynamically determine the environment that should be used, based on the branch name:
jobs:
get-environment-name:
name: "Extract environment name"
runs-on: ubuntu-latest
outputs:
environment: ${{ steps.extract.outputs.environment }}
steps:
- id: extract
# You can run any logic you want here to map refs to environment names.
# The GITHUB_REF will look like this: refs/heads/my-branchname
# The example logic here simply removes "refs/heads/deploy-" from the beginning,
# so a branch name deploy-prod would be mapped to the environment "prod"
run: echo "::set-output name=environment::$(echo $GITHUB_REF | sed -e '/^refs\/heads\/deploy-\(.*\)$/!d;s//\1/')"
- env:
EXTRACTED: ${{ steps.extract.outputs.environment }}
run: 'echo "Extracted environment name: $EXTRACTED"'
deploy:
name: "Deploy"
if: ${{ github.event_name == 'push' && needs.get-environment-name.outputs.environment }}
needs:
- get-environment-name
# - unit-tests
# - frontend-tests
# ... add your unit test jobs here, so they are executed before deploying anything
environment: ${{ needs.get-environment-name.outputs.environment }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
# ... Run your deployment actions here, with full access to the environment's secrets
Note that in the if: clause of the deployment job, it's not possible to use any environment variables or bash scripts. So using a previous job that extracts the environment name from the branch name is the simplest I could make it at the current time.
I came across this question when trying to implement environment-based secret selection for a Github action.
This variable-mapper action (https://github.com/marketplace/actions/variable-mapper) implements the desired concept of mapping a key variable or an environment name to secrets or other pre-defined values.
The example use of it shows this:
on: [push]
name: Export variables corresponding to regular expression-matched keys
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: kanga333/variable-mapper#v1
with:
key: ${{GITHUB_REF#refs/heads/}}
map: |
{
"master": {
"environment": "production",
"AWS_ACCESS_KEY_ID": ${{ secrets.PROD_AWS_ACCESS_KEY_ID }},
"AWS_SECRET_ACCESS_KEY": ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
},
"staging": {
"environment": "staging",
"AWS_ACCESS_KEY_ID": ${{ secrets.STG_AWS_ACCESS_KEY_ID }},
"AWS_SECRET_ACCESS_KEY": ${{ secrets.STG_AWS_ACCESS_KEY_ID }}
},
".*": {
"environment": "development",
"AWS_ACCESS_KEY_ID": ${{ secrets.DEV_AWS_ACCESS_KEY_ID }},
"AWS_SECRET_ACCESS_KEY": ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
}
}
- name: Echo environment
run: echo ${{ env.environment }}
I have a GitHub workflow for releasing nightly snapshots of the repository. It uses the create-release action. This is how the workflow file looks right now:
name: Release Nightly Snapshot
on:
schedule:
- cron: "0 0 * * *"
jobs:
build:
name: Release Nightly Snapshot
runs-on: ubuntu-latest
steps:
- name: Checkout master Branch
uses: actions/checkout#v2
with:
ref: 'master'
- name: Create Release
id: nightly-snapshot
uses: actions/create-release#latest
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
tag_name: 'nightly snapshot'
release_name: 'nightly snapshot'
draft: false
prerelease: false
I want tag_name and release_name to use the current date and time, instead of hard-coded values. However, I couldn't find any documentation on it. How should I do it?
From this post you can create a step that set its output with the value $(date +'%Y-%m-%d')
Then use this output using ${{ steps.date.outputs.date }}. The following show an example for environment variables and for inputs :
on: [push, pull_request]
name: build
jobs:
build:
name: Example
runs-on: ubuntu-latest
steps:
- name: Get current date
id: date
run: echo "::set-output name=date::$(date +'%Y-%m-%d')"
- name: Test with environment variables
run: echo $TAG_NAME - $RELEASE_NAME
env:
TAG_NAME: nightly-tag-${{ steps.date.outputs.date }}
RELEASE_NAME: nightly-release-${{ steps.date.outputs.date }}
- name: Test with input
uses: actions/hello-world-docker-action#master
with:
who-to-greet: Mona-the-Octocat-${{ steps.date.outputs.date }}
Outputs :
* Test with environment variables
nightly-tag-2020-03-31 - nightly-release-2020-03-31
* Test with input
Hello Mona-the-Octocat-2020-03-31
Here's another way to do this via environment variables (from this post):
name: deploy
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Set current date as env variable
run: echo "NOW=$(date +'%Y-%m-%dT%H:%M:%S')" >> $GITHUB_ENV
- name: Echo current date
run: echo $NOW # Gives "2022-12-11T01:42:20"
This sets the date as an environment variable, which is useful if you want to consume it in scripts / programs in subsequent steps.
A clean solution is to use ${{ github.event.repository.updated_at}} which is pretty close to current datetime$(date +%Y%m%d%H%M)
Format is ISO 8601
e.g 2022-05-15T23:33:38Z
Pros:
Doesn't require shell execution
Directly accessible from all your workflow
Allows cross-referencing with GitHub events
Cons:
Strict format
you can still modify the format in a shell
e.g. echo ${{ github.event.repository.updated_at}} | sed 's/:/./g'
Not available with Act testing framework
References:
Github context
Event object
Update
The set-output command is deprecated and will be disabled soon. Please upgrade to using Environment Files.
Documentation can be found here
name: deploy
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Get current date
id: date
run: |
echo "{date}={$(date +'%Y-%m-%d')}" >> $GITHUB_STATE
- name: Test with environment variables
run: echo $TAG_NAME - $RELEASE_NAME
env:
TAG_NAME: nightly-tag-${{ env.date }}
RELEASE_NAME: nightly-release-${{ env.date }}
if $GITHUB_ENV doesn't work, use $GITHUB_OUTPUT instead:
name: Flutter CI
run-name: ${{ github.actor }} is testing GitHub Actions 🚀
on:
push:
tags:
- '*'
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: "Set current date as env variable"
run: |
echo "builddate=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
id: version # this is used on variable path
- name: Publishing Release
uses: softprops/action-gh-release#v1
with:
tag_name: ${{ steps.version.outputs.builddate }}
name: ${{ steps.version.outputs.builddate }}
body: "your date"
- name: print date
run: |
echo $DEBUG
env:
DEBUG: ${{ steps.version.outputs.builddate }}-DEBUG
RELEASE: ${{ steps.version.outputs.builddate }}-RELEASE
This is the project structure:
- parent/
|- .github/workflows/
|- frontend/
|- ...
This is the .yml file within workflows:
name: CI
on:
push:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- uses: actions/setup-node#v1
with:
node-version: "12.x"
- name: Install dependencies
working-directory: frontend
run: npm install
- name: Build
working-directory: frontend
run: npm run build
- name: Deploy Files
uses: appleboy/scp-action#master
env:
HOST: ${{ secrets.aws_pull_host }}
USERNAME: ${{ secrets.aws_pull_username }}
KEY: ${{ secrets.aws_pull_private_key }}
with:
working-directory: frontend
source: build/
target: "/home/build/site/testDir/"
strip_components: 1
Whenever the action gets to the step Deploy Files, i get the error:
tar: empty archive
tar all files into /tmp/891353322/bC24rHhFAi.tar
exit status 1
##[error]Docker run failed with exit code 1
First time working with github actions so I am pretty lost with this error. Appreciate any help.
Kept messing with it, finally figured it out:
- name: Deploy Files
uses: appleboy/scp-action#master
env:
HOST: ${{ secrets.aws_pull_host }}
USERNAME: ${{ secrets.aws_pull_username }}
KEY: ${{ secrets.aws_pull_private_key }}
with:
source: frontend/build/
target: "/home/build/site/testDir/"
strip_components: 1