I have a self hosted Github Action runner on a Windows server, my goal is to migrate from Jenkins to GitHub's workflow. I have a very hard to time understand how to use the environment variables for some reason.
I have a folder on my self hosted server, where I want to place some generic build scripts, which is created in Powershell.
So my main.yml look like this
# This is a basic workflow to help you get started with Actions
name: Build integration
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the "develop" branch
push:
branches: ["*"]
pull_request:
branches: ["*"]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
continuous-integration:
# The type of runner that the job will run on
runs-on: [self-hosted, .net]
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- name: Checkout code
uses: actions/checkout#v3
# get environment
- name: Get system environments
uses: FranzDiebold/github-env-vars-action#v2
# show environment
- name: Show all enronment variables
run: SET
shell: cmd
# read meta file
- name: read project meta file
run: |
.\$BUILD_SCRIPTS_PATH}\Powershell\meta.ps1 -MetaFilePath ProjectMeta.json
shell: powershell
I have tried all kinds of variants to get the environment variable BUILD_SCRIPTS_PATH out.
.\${env.BUILD_SCRIPTS_PATH}\Powershell\meta.ps1 -MetaFilePath ProjectMeta.json
.\${{BUILD_SCRIPTS_PATH}}\Powershell\meta.ps1 -MetaFilePath ProjectMeta.json
.\$env.BUILD_SCRIPTS_PATH\Powershell\meta.ps1 -MetaFilePath ProjectMeta.json
.\$BUILD_SCRIPTS_PATH\Powershell\meta.ps1 -MetaFilePath ProjectMeta.json
I keep getting this error
+ ${BUILD_SCRIPTS_PATH}\Powershell\meta.ps1 -MetaFilePath ProjectMe ...
+ ~~~~~~~~~~~~~~~~~~~~
Unexpected token '\Powershell\meta.ps1' in expression or statement.
+ CategoryInfo : ParserError: (:) [], ParseException
+ FullyQualifiedErrorId : UnexpectedToken
a minor part of the output from the SET command
Run SET
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\Administrator\AppData\Roaming
BUILD_SCRIPTS_PATH=E:\github\BuildScripts
ChocolateyInstall=C:\ProgramData\chocolatey
I have looked into the documentation from GitHub GitHub Documentation but I don't really understand what i am doing wrong. I would really appreciate is someone could explain to me what i am doing wrong.
You can read more about using environment variables in GitHub actions here.
From what you've posted it doesn't seem like you're providing the BUILD_SCRIPTS_PATH to the workflow's environment.
You can add it like this:
# read meta file
- name: read project meta file
run: |
.\$BUILD_SCRIPTS_PATH\Powershell\meta.ps1 -MetaFilePath ProjectMeta.json
shell: powershell
env:
BUILD_SCRIPTS_PATH: ${{ secrets:build_scripts_path}}
Then just make sure to add the build_scripts_path as a GitHub secret.
I had two github action, one of merge-staging-in-to-master, the other one is CI.
The merge-staging-in-to-master was triggered by manually.
And CI was triggered by push to master.
But, when I triggered the merge-staging-in-to-master manually, the CI does nothing...
The expectation I want is invoke the CI action after merge-staging-in-to-master was done.
There is my code below.
merge-staging-in-to-master.yml
# This is a basic workflow to help you get started with Actions
name: merge-staging-to-master
# Controls when the workflow will run
on:
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# Allows external webhook trigger
repository_dispatch:
types:
- merge-staging-to-master
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
merge-staging-to-master:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v2
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
git fetch origin --unshallow
git checkout -t remotes/origin/master
git pull origin master
git config --global user.name "user"
git config --global user.email "user#mail.com"
git merge staging -m "[AUTO] merge staging back to master"
git push
CI.yml
# This is a basic workflow to help you get started with Actions
name: CI
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the main branch
push:
branches: [ master ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-18.04
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: '14'
- id: 'auth'
name: 'Authenticate to Google Cloud'
uses: 'google-github-actions/auth#v0'
with:
credentials_json: '${{ secrets.GOOGLE_CREDENTIALS }}'
- name: Install Dependencies
run: |
sudo apt update
sudo apt install -y libcairo2-dev libjpeg-dev libpango1.0-dev libgif-dev librsvg2-dev
- name: 'Set up Cloud SDK'
uses: 'google-github-actions/setup-gcloud#v0'
- name: Build
run: sudo npm install --build-from-source && npx prisma generate dev && npm run build
- name: Deploy
run: npm run deploy
This is a builtin feature of GitHub actions to prevent jobs from triggering more jobs and potentially burning through your action minutes in a very short time (all the while blocking hosted runners for others as well).
The GitHub actions tokens used to authenticate have a special flag on them by which GitHub knows the token is from an Actions workflow.
If you know what you're doing you can use a PAT or Oauth App token to authenticate your git push instead, that will trigger further workflows. If you search the GitHub marketplace you'll find a few actions that can retrieve an OAuth App token.
I have this workflow in a repo called terraform-do-database and I'm trying to use a reusable workflow coming from the public repo foo/git-workflows/.github/workflows/tag_validation.yaml#master
name: Tag Validation
on:
pull_request:
branches: [master]
push:
branches:
- '*' # matches every branch that doesn't contain a '/'
- '*/*' # matches every branch containing a single '/'
- '**' # matches every branch
- '!master' # excludes master
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
tag_check:
uses: foo/git-workflows/.github/workflows/tag_validation.yaml#master
And this is the reusable workflow file from the public git-workflows repo that has the script that should run on it. What is happening is that the workflow is trying to use a script inside the repo terraform-do-database
name: Tag Validation
on:
pull_request:
branches: [master]
workflow_call:
jobs:
tag_check:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v3
# Runs a single command using the runners shell
- name: Verify the tag value
run: ./scripts/tag_verify.sh
So the question: How can I make the workflow use the script stored in the git-worflows repo instead of the terraform-do-database?
I want to have a single repo where I can call the workflow and the scripts, I don't want to have everything duplicated inside all my repos.
I have found that if I wrap the script into a composite action. I can use GitHub context github.action_path to locate the scripts.
Example:
run: ${{ github.action_path }}/scripts/foo.sh
One way to go about this is perform a checkout inside your reusable workflow that essentially clones the content of the repo where your scripts are and only then you can access it. It's not the cleanest solution but it works.
Perform a second checkout, to clone your repo that has the reusable workflow into a dir reusable-workflow-repo
- name: Checkout reusable workflow dir
uses: actions/checkout#v3
with:
repository: <your-org>/terraform-do-database
token: ${{ secrets.GIT_ACCESS_TOKEN }}
path: reusable-workflow-repo
Now you have all the code you need inside reusable-workflow-repo. Use ${GITHUB_WORKSPACE} to find the current path and simply append the path to the script.
- name: Verify the tag value
run: ${GITHUB_WORKSPACE}/reusable-workflow-repo/scripts/tag_verify.sh
I was able to solve it adding a few more commands to manually download the script and execute it.
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v3
# Runs a single command using the runners shell
- name: Check current directory
run: pwd
- name: Download the script
run: curl -o $PWD/tag_verify.sh https://raw.githubusercontent.com/foo/git-workflows/master/scripts/tag_verify.sh
- name: Give script permissions
run: chmod +x $PWD/tag_verify.sh
- name: Execute script
run: $PWD/tag_verify.sh
Following Kaleby Cadorin example but for the case where the script is in a private repository
- name: Download & run script
run: |
curl --header "Authorization: token ${{ secrets.MY_PAT }}" \
--header 'Accept: application/vnd.github.raw' \
--remote-name \
--location https://raw.githubusercontent.com/COMPANY/REPO/BRANCH/PATH/script.sh
chmod +x script.sh
./script.sh
Note: GITHUB_TOKEN doesn't seem to work here, a PAT is required.
According to this thread on github-community the script needs to be downloaded/checked out separatly.
The "reusable" workflow you posted is not reusable in this sense, because since it is not downloading the script the workflow can only run within its own repository (or a repository that already has the script).
I want to run periodically ipynb file in my github repository (Like every 30 minutes).
I know that I can use Github Action to create yml file for this progress but I have no idea how to reorganize yml file.
How can I do it?
Here is my test tml file defined below.
# This is a basic workflow to help you get started with Actions
name: CI
# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
schedule:
- cron: '*/5 * * * *'
push:
branches: [ master ]
pull_request:
branches: [ master ]
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v2
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo Hello, world!
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
echo Add other actions to build,
echo test, and deploy your project.
You can check out this GitHub action that runs your jupyter notebook and lets you upload the artifacts. As for how to organize your workflow file, you can read the documentation here.
I'm relatively new to GitHub Actions and I have 2 jobs–one that runs my tests, and one that deploys my project onto a server.
Obviously I want the tests to run on every branch, but deploying should only happen when something gets pushed to master.
I'm struggling to find a way to run a job on a specific branch. I know it's possible to only run entire workflows on a specific branch, however that would mean I would have a "test" workflow and a "deploy" workflow.
This sounds like a solution, however they would run parallel. In an ideal world, the tests would run first, and only if they succeed, then the deploy job would start. This isn't the case when using 2 separate workflows.
How would I be able to achieve this? Is it possible to run jobs on a specific branch?
In a recent update you can now put if conditionals at job level. See the documentation here. https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idif
I tested this workflow which runs the job test on every push, but only runs deploy on the master branch.
name: my workflow
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Execute tests
run: exit 0
deploy:
runs-on: ubuntu-latest
needs: test
if: github.ref == 'refs/heads/master'
steps:
- name: Deploy app
run: exit 0
What follows is my original answer, and an alternative solution if you prefer to have separate workflows.
The first workflow runs for every branch except master. In this workflow you run tests only.
on:
push:
branches:
- '*'
- '!master'
The second workflow runs for just master and runs both your tests and deploys if the tests were successfully passed.
on:
push:
branches:
- master
Most answers provide a solution for one single branch. To restrict the job to run on any specific set of branches, you can do it using the if conditional with multiple disjunction (||) operators; but this is too verbose and doesn't respect the DRY principle.
The same can be archived with less repetition using the contains function.
Using contains:
contains('
refs/heads/dev
refs/heads/staging
refs/heads/production
', github.ref)
compared to using multiple ||:
github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/staging' || github.ref == 'refs/heads/production' || …
Full example:
---
on: push
jobs:
test:
name: Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Run tests
run: …
deployment:
name: Deployment
runs-on: ubuntu-latest
needs: [test]
if:
contains('
refs/heads/dev
refs/heads/staging
refs/heads/production
', github.ref)
steps:
- uses: actions/checkout#v2
- name: Deploy
run: …
Here is what I've done for steps that should only run on a specific branch.
- name: Publish Image
# Develop branch only
if: github.ref == 'refs/heads/develop'
run: |
... publish commands ...
2021 update
I know it's possible to only run entire workflows on a specific
branch, however that would mean I would have a "test" workflow and a
"deploy" workflow.
This sounds like a solution, however they would run parallel. In an
ideal world, the tests would run first, and only if they succeed, then
the deploy job would start. This isn't the case when using 2 separate
workflows.
You can now use the event workflow_run to achieve the part the tests would run first, and only if they succeed, then the deploy job would start (read on to see how):
Documentation page of workflow_run
https://docs.github.com/en/actions/learn-github-actions/events-that-trigger-workflows#workflow_run
This event occurs when a workflow run is requested or completed, and
allows you to execute a workflow based on the finished result of
another workflow. A workflow run is triggered regardless of the result
of the previous workflow.
For example, if your pull_request workflow generates build artifacts,
you can create a new workflow that uses workflow_run to analyze the
results and add a comment to the original pull request.
Now, considering OP's initial issue:
I want the tests to run on every branch, but deploying should only happen when something gets pushed to master
This now could be solved like this:
The following setup is working, a few minutes ago I just implemented the same logic in one of my repository
Workflow <your_repo>/.github/workflows/tests.yml
name: My tests workflow
on:
push:
branches:
- master
pull_request: {}
jobs:
test:
# ... your implementation to run your tests
Workflow <your_repo>/.github/workflows/deploy.yml
name: My deploy workflow
on:
workflow_run:
workflows: My tests workflow # Reuse the name of your tests workflow
branches: master
types: completed
jobs:
deploy:
# `if` required because a workflow run is triggered regardless of
# the result of the previous workflow (see the documentation page)
if: ${{ github.event.workflow_run.conclusion == 'success' }}
# ... your implementation to deploy your project
Although this discussion is very old but recently I came across the same problem with a slight addition. if condition to check whether the branch is main works but what if someone pushes their branch and updates the workflow yml file to remove the if condition? The deploy job will get triggered without their branch even being reviewed or merged in main and could potentially break the production environment. This could be a concern in open source projects.
I couldn't find an answer for this anywhere so wanted to share my findings. I hope this is the right thread for it.
To ensure that there's no way a job is triggered except for in specific branches, environments can be used. A deploy job will most likely have some api keys to connect to the destination server that might be stored in secrets. Instead of storing them in repository secrets which are accessible globally within the repository, we should store them in respective environments.
The official docs for environments contain detailed explanation with sample scripts but sharing a simple example here. Let's say we want to run production deployment only when main is updated
From repository Settings, create a production environment
Choose Selected Branches in Deployment Branches dropdown and add main in the pattern
Add the api keys in production environment secrets
In the workflow yml we just need to add environment information environment: production as follows (using script from #peterevans's answer)
name: my workflow
on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Execute tests
run: exit 0
deploy:
runs-on: ubuntu-latest
needs: test
if: github.ref == 'refs/heads/main'
environment: production
steps:
- name: Deploy app
run: exit 0
The environment information indicates where the secrets have to be read from. If the current branch name doesn't match the pattern provided in Selected Branches then the job will immediately fail with an error. Since we have a condition to only run this on main, normally that won't bother us because this job will be skipped on other branches anyways. But if someone, mistakenly or with ill intentions, modifies the yml file and removes the condition before pushing their branch, they'll get an error. So, our system remains secure at least from the threat here.
Hope this helps anyone wondering the same.
The following worked for me inside a job:
Where a PR target branch is one of the following:
if: contains(github.base_ref, 'staging')
|| contains(github.base_ref, 'production')
When the source branch of the pull request is one of them below:
if: contains(github.head_ref, 'feature')
|| contains(github.head_ref, 'release')
name: CI
on: push
jobs:
prod-check:
if: ${{ github.ref == 'refs/heads/main' }}
runs-on: ubuntu-latest
steps:
- run: echo "Deploying to production server on branch $GITHUB_REF"
According to the documentation, the if condition is wrapped like so
if: ${{ github.ref == 'refs/heads/main' }}
While you can't have conditions at job level at this moment, you can have conditions at step level - see Contexts and expression syntax for GitHub Actions.
For getting a branch name, current solution is to check GITHUB_REF environment variable - see Default environment variables and this question for details.
Putting it all together - if you decide to go with accepted answer in last link, your workflow might look like this:
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Run tests
run: ./my-tests.sh
deploy:
runs-on: ubuntu-latest
needs: test
steps:
- name: Extract branch name
shell: bash
run: echo "##[set-output name=branch;]$(echo ${GITHUB_REF##*/})"
id: extract_branch
- name: Deploy
run: ./deploy.sh
if: steps.extract_branch.outputs.branch == 'master'
If you'd rather keep everything in workflow file instead of separated script, you can always add if to each step in given job.
I hope it's just a temporary solution/workaround, and job conditions are going to be added before end of beta.
For steps or jobs you can also use github.ref_name which is the branch or tag name that triggered the workflow run.
name: my workflow
on: push
jobs:
if: github.ref_name == 'main'
test:
runs-on: ubuntu-latest
steps:
- name: Execute tests
run: exit 0
For more information about the github context check here.
You need to add these two script to the workflows
//main.yml
name: main
on:
push:
branches:
- main
//size.yml
name: size
on:
pull_request:
branches:
- main
There are already some pretty good answers. Here's a way to specify multiple branches in your conditional:
name: my workflow
on: push
jobs:
deploy:
if: contains(fromJson('["refs/heads/master", "refs/heads/main"]'), github.ref)
runs-on: ubuntu-latest
steps:
- name: Execute tests
run: exit 0
2022
Below workflow runs only for push in main branch:
name: Distribute to Firebase
on:
push:
branches: [ main ]
jobs:
...