I've tried to follow the official documentation, several blogs and answers on stackoverflow but nothing helped me to find a solution for this problem.
With github-action I've created a script to run unit tests after every push is done in our github server. For now the project is in early stage, so our dependency list is quite short and npm install command is done quite quickly (around 40 seconds). Obviously in future we'll need to add more dependencies and we are expecting a serious increment of that time, so I've tried to implement the cache mechanism, but for some reason the process still need to download the dependencies and the time for that step still be pretty much the same.
Below there is the script I've used. Following the documentation online I've found 2 different options, I've tried both (once per time) but without luck.
name: unit
on:
push:
branches: ['*']
pull_request:
branches: ['*']
workflow_dispatch:
jobs:
build:
runs-on: macos-latest
steps:
- uses: actions/checkout#v2
- name: setup node
uses: actions/setup-node#v2
with:
node-version: '15'
cache: npm
# start option 1
- name: Setup cache
uses: actions/cache#v2
with:
path: app/node_modules
key: ${{ runner.os }}-node-${{ hashFiles('app/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
# end option 1
# start option 2
- name: Cache node modules
uses: actions/cache#v2
env:
cache-name: cache-node-modules
with:
path: ~/.npm
key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-build-${{ env.cache-name }}-
${{ runner.os }}-build-
${{ runner.os }}-
# end option 2
- name: Install dependencies
run: npm install
- name: run unit tests
run: npm run test
Why the cache is not working? Am I doing some mistake in the script?
Related
Currently I am trying to have 1 job scanning SonarQube and 1 job checking for the quality gate in github actions. In order to get the report from gradle of the quality check in the second job, I have to cache it (or atleast with the limited knowledge I have). But I don't want when I rerun the workflow on the same PR or on different PRs use the same cache, since the report is only valid for a current workflow and not futher ones (those need to always create a new report and give the new report to the second job).
Here is my workflow:
name: SonarQube
on:
push:
branches:
- master # or the name of your main branch
pull_request:
types: [opened, synchronize, reopened]
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
- name: Set up JDK 11
uses: actions/setup-java#v3
with:
java-version: 11
distribution: corretto
cache: gradle
- name: Build and analyze
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
run: ./gradlew --info sonar -Dsonar.host.url=$SONAR_HOST_URL -Dsonar.login=$SONAR_TOKEN
- name: Cache report
uses: actions/cache#v3
with:
path: build/sonar/
key: report-task
qualityCheck:
needs: scan
runs-on: ubuntu-latest
steps:
- name: Get cache report
uses: actions/cache#v3
with:
path: build/sonar/
key: report-task
- name: Quality Gate check
id: sonarqube-quality-gate-check
uses: sonarsource/sonarqube-quality-gate-action#master
# Force to fail step after specific time.
timeout-minutes: 5
with:
scanMetadataReportFile: build/sonar/report-task.txt
env:
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
What needs to be changed to achieve this?
I have currently done this:
- name: Cache report
uses: actions/cache#v3
with:
path: build/sonar/
key: commit-${{ github.sha }}-workflow-${{ github.run_id }}-${{ github.run_number }}-report-task-${{ hashFiles('build/sonar/**') }}
This doesn't solve the problem that it keeps the cache after the workflow ends, but it now won't use the report of previous workflows.
I'm creating a Github Action Workflow.
There is a phing build file that creates multiple zips depending on product variations. This is done with a matrix.
Before the product is build there is a JS File that needs to be build. That's done with a rollup build file.
Everything works fine, but the JS build only needs to be done once and not with every matrix combination.
I don't know how that should be done. Maybe with a single separate workflow running at start, but how would the finished JS file be pushed into the next workflow? Or maybe one workflow is fine?
name: Build
on: [push]
jobs:
build:
runs-on: ubuntu-latest
env:
VERSION: 1.0.1
strategy:
matrix:
edition: ["product1", "product2", "product3"]
limits: [100, 200, 300]
exclude:
- edition: product1
limits: 100
- edition: product2
limits: 100
- edition: product1
limits: 200
- edition: product2
limits: 200
steps:
- uses: actions/checkout#v2
- name: Build JS
uses: actions/setup-node#v1
with:
node-version: 12.x
- name: Cache Node.js modules
uses: actions/cache#v2
with:
path: ~/.npm
key: ${{ runner.OS }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.OS }}-node-
${{ runner.OS }}-
- run: npm ci
- run: npm run build # builds main.js
- name: Phing Build
uses: phingofficial/phing-github-action#main
with:
repo-token: "${{ secrets.GITHUB_TOKEN }}"
buildfile: build/phing/build.xml
targets: main
user-properties: editions=${{ matrix.edition }} limits=${{ matrix.limits }}
version: 3.0.0-alpha4
- name: Archive code
uses: actions/upload-artifact#v2
with:
name: file-${{ matrix.edition }}${{ matrix.limits }}-v${{ env.VERSION }}
path: build/${{ matrix.edition }}/zip/
retention-days: 1
You can execute creating the JS File in a separate job (let's name it prepare) and then reference that job in the job with the matrix definition (build) using the needs keyword. Sharing files across jobs is something that GitHub is very specific about: not each job needs all files from a previous job, so you will need to upload and download that file yourself.
Example setup:
jobs:
prepare:
runs-on: ubuntu-latest
steps:
...
# upload the file here for later use
- uses: actions/upload-artifact
build:
needs: prepare
strategy:
matrix:
edition: ["product1", "product2", "product3"]
steps:
# download the file here so you can use it
- uses: actions/download-artifact
See the docs on the needs keyword here.
I'm having hard time figuring out how to make the pipeline use the restored cache of npm modules.
Here's the manifest file:
jobs:
setup-build-push-deploy:
name: Setup, Build, Push, and Deploy
runs-on: ubuntu-latest
steps:
# Checkout the repo
- name: Checkout
uses: actions/checkout#v2
- name: Setup NodeJS
uses: actions/setup-node#v1
with:
node-version: '12.14'
# Cache the npm node modules
- name: Cache node modules
uses: actions/cache#v1
id: cache-node-modules
env:
CACHE_NAME: cache-node-modules
with:
# npm cache files are stored in `~/.npm` on Linux/macOS
path: ~/.npm
key: ${{ env.CACHE_NAME }}-${{ hashFiles('app/package-lock.json') }}
restore-keys: |
${{ env.CACHE_NAME }}-
# Install NPM dependencies
- name: Install Dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: |
cd app/
npm install
# Compile Typescript to Javascript
- name: Compile Typescript
run: |
cd app/
npm run make
The "cache" step does make a successful cache hit, and therefore the "install dependencies" step is skipped. But then the "compile" step fails, because it cannot find any installed npm modules.
The docs here and here and the SO question, eg. here, do not specify any certain step or configuration that points towards where to pick up the cached modules from. Seems like they should be picked up automatically from ~/.npm, but that's not the case.
I was able to solve this by changing the path to node_modules:
Here's an example:
jobs:
prepare-npm-cache:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: 12.22.0
- name: Cache node modules
uses: actions/cache#v2
id: npm_cache_id
with:
path: node_modules
key: ${{ runner.os }}-npm-cache-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-npm-cache-
${{ runner.os }}-
- name: Install Dependencies
if: steps.npm_cache_id.outputs.cache-hit != 'true'
run: npm ci
I found myself in the exact same situation.
The node_modules seems to be cached but then the next job immediately fails due to missing dependencies.
I ended up using this GitHub action by Gleb Bahmutov.
Specifically this part:
- uses: bahmutov/npm-install#v1.4.5
- run: npm install
- run: npm run ***
Detailed description can be found in this article:
https://glebbahmutov.com/blog/do-not-let-npm-cache-snowball/
I have a github action which is building the React app (based on create-react-app) and deploying it to AWS S3. I have to pass some environment variables to correctly run yarn build command.
I could hold them directly in .env file, but I don't want to hold them inside the repository. Currently I'm just adding environment variables right before the yarn build command, but it's annoying solution and seems to be a bit hacky. Ideally, I'd like to inject .env.local file with my own configuration, but I don't have any good idea how to do it.
Here's my build.yml file:
name: Build
on:
push:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [12.13.1]
steps:
- uses: actions/checkout#v1
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- name: Yarn install
run: yarn install
- name: Build
run: REACT_APP_GRAPHQL_URL=https://some.url/graphql CI=false yarn build
- name: Deploy to S3
uses: jakejarvis/s3-sync-action#master
with:
args: --acl public-read --delete
env:
AWS_S3_BUCKET: my-bucket-name
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ${{ secrets.AWS_REGION }}
SOURCE_DIR: "build"
So as you can see the magic happens here:
run: REACT_APP_GRAPHQL_URL=https://some.url/graphql CI=false yarn build
How can I make it look nicer? It's quite ok when I have two variables, but what if I'll have dozens of them?
By the way - it's a private repository, if it makes any difference.
And I don't want to use another CI solution, currently Github Actions seems to be enough for me.
you can do magic like this,
name: Build
on:
push:
branches:
- master
env:
CI : false
REACT_APP_GRAPHQL_URL : https://some.url/graphql
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [12.13.1]
steps:
- uses: actions/checkout#v1
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- name: Yarn install
run: yarn install
- name: Build
run: yarn build
- name: Deploy to S3
uses: jakejarvis/s3-sync-action#master
with:
args: --acl public-read --delete
env:
AWS_S3_BUCKET: my-bucket-name
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ${{ secrets.AWS_REGION }}
SOURCE_DIR: "build"
I think it makes look nicer
In the following workflow, I want to add cache functionality so that, every time it will build from scratch. this is the workflow:
# This workflow will do
# a clean install of node deps
# build the source code
# run test across different versions of node
name: Conflict Check
on:
push:
branches:
- staging
pull_request:
branches:
- staging
jobs:
triage:
runs-on: ubuntu-latest
steps:
- uses: mschilde/auto-label-merge-conflicts#master
with:
CONFLICT_LABEL_NAME: 'has conflicts'
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
How can I achieve this?
You can use actions/cache action for purposes of caching in Github Actions.
jobs:
triage:
runs-on: ubuntu-latest
steps:
- name: Cache build files
uses: actions/cache#v2
with:
path: ${{ PATH_TO_CACHE }}
key:${{ runner.os }}-${{ hashFiles(<glob_pattern_for_files>) }}
- uses: mschilde/auto-label-merge-conflicts#master
with:
CONFLICT_LABEL_NAME: 'has conflicts'
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The example above assumes you want to cache your files between runs on different refs but your actual key declaration would depend on what you are trying to do.
For example if you are trying to cache between jobs or workflow runs on same ref:
key: ${{ runner.os }}-${{ github.sha }}