Saving cache on job failure in GitHub Actions - github

I am using the GitHub cache action, but I noticed that the no cache will be created if the job fails. From the docs:
If the job completes successfully, the action creates a new cache with the contents of the path directory.
A stripped down version of my workflow YAML file:
name: Build
on: [push]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v1
- name: Setup Node.js
uses: actions/setup-node#master
with:
node-version: '10.x'
- name: Get yarn cache path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"
- name: Restore yarn cache
uses: actions/cache#v1
id: yarn-cache
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Install yarn dependencies
run: yarn install
- name: Build
run: yarn build
I noticed that if my Build step fails the cache post-step will be skipped unnecessarily, which causes the installed dependencies to not be cached. This requires subsequent runs to download dependencies again, slowing down the job.
Is there a way to always cache the dependencies, even when the build step fails?

In the official version of the action, no it's not possible to cache dependencies if the build fails. See this line in the cache action's manifest:
runs:
using: 'node12'
main: 'dist/restore/index.js'
post: 'dist/save/index.js'
post-if: 'success()'
It will only run the post-step if the job succeeds. I don't know the original reasoning for this, but there are a few issues opened around this idea. In this issue a user forked the action to change the post-if to always(), which may be what you're after, assuming you're willing to run an unofficial action.

Related

Github actions: Dependencies lock file is not found in runners/path

I have a single Github repository for both server and frontend. The directory structure looks like:
root
|- frontend
|- server (Express App)
Github Action:
name: Node.js CI
on:
push:
branches: [ main ]
jobs:
build:
runs-on: self-hosted
strategy:
matrix:
node-version: [14.x]
# See supported Node.js release schedule at https://nodejs.org/en/about/releases/
steps:
- uses: actions/checkout#v2
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v2
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- run: npm ci
working-directory: './server'
- run: npm run start
working-directory: './server'
I only have a single job to build the Express server (and not the frontend yet) so I set the working-directory to ./server. However, I still get an error:
Dependencies lock file is not found in /home/{username}/runners.../repository_name. Supported file patterns: package-lock.json,yarn.lock
So apparently it's not trying to run in .../reposirtoy_name/server.
I'm just trying to build both server and frontend in single Github action.
There might be a chance that your problem is specifically with "uses: actions/setup-node". They mention in the docs that if you have multiple lock files or a lock file(s) in a directory that is not the root
In my case I had a single project with nested projects/dir. In my GitHub actions I wanted to run npm test on the nested project/dir so I had to specify to use my package.json inside the specific sub-directory. Double check to see that you are specifying the right directories with cache-dependency-path.
Specified here
https://github.com/actions/setup-node#caching-packages-dependencies
Try out this solution. It worked in my case.
In the build insert the default working directory
build:
runs-on: self-hosted
defaults:
run:
working-directory: ./server/
strategy:
matrix:
node-version: [14.x]
Then include cache dependency path. This should be the location of your package-lock.json file
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
cache-dependency-path: './server/package-lock.json'
tldr
Make sure your checkout repo step is BEFORE setup node step, if using cache property with actions/setup-node#v3.
For me, it was caused by cache property on actions/setup-node#v3.
Without it - everything worked fine.
With it - failed. Reason is, it uses as cache key the package-lock.json (or yarn.lock) file.
See: https://github.com/actions/setup-node
My checkout repo step (actions/checkout#v2) was AFTER the setup node step, so it didn't find the package-lock.json file - because it wasn't checked out yet.

EACCES: permission denied, unlink '/home/runner/work/gos_front/gos_front/node_modules/.yarn-integrity

I have this workflow.yaml for github actions:
name: CI/CD
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14]
steps:
- name: Checkout repository
uses: actions/checkout#v2
- name: Set up Node.js ${{ matrix.node-version }}
uses: actions/setup-node#v2
with:
node-version: ${{ matrix.node-version }}
- name: Install dependencies
uses: borales/actions-yarn#v2.0.0
with:
cmd: install --ignore-engines
- name: Run linters
uses: borales/actions-yarn#v2.0.0
with:
cmd: lint:prettier
- name: Run cypress
uses: cypress-io/github-action#v2
with:
build: npm run build
start: npm start
- name: Build
uses: borales/actions-yarn#v2.0.0
with:
cmd: build
Then it runs, on run cypress occurs an error:
error An unexpected error occurred: "EACCES: permission denied, unlink
'/home/runner/work/gos_front/gos_front/node_modules/.yarn-integrity'"
.
Before that I add cypress run all was right. I tried to add sudo but it not helped. I use yarn.
cypress-io/github-action tries to install dependencies as well and runs into an access issue. The problem is the way your node_modules folder is created.
I ran into this when using Docker, mapping node_modules from inside the docker image to the outside. This is a known issue.
You use borales/actions-yarn. This is an outdated package, that probably does something similar. You should instead use yarn directly. From their readme:
Please keep in mind that this Action was originally written for GitHub Actions beta (when Docker was the only way of doing things). Consider using actions/setup-node to work with Yarn. This repository will be mostly supporting the existing flows.

Caching npm dependency with github action

I want to cache npm dependencies so that I does not do npm install every time I push and instead just load it from cache.
I think github action support this now?: How do I cache steps in GitHub actions?
Here are few cases
If package.json changes, which means yarn.lock or package-lock.json changed so do npm install and update cache
Extending my above point, the contributor could be doing both yarn install and npm install
From the same above question, I changed my github action to something like this
name: Tsc compilation test
on: [push, pull_request]
jobs:
build:
name: Build
runs-on: ubuntu-18.04
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Cache NPM dependencies
uses: actions/cache#v1
with:
path: ~/.npm
key: ${{ runner.OS }}-npm-cache-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.OS }}-npm-cache-
- name: Install dependencies
run: npm install
- name: Test tsc
run: npm run ts-compile-check
This still does npm install and haven't reduced my computation time for installing dependencies (So I am not sure if this is working correctly or not)
Then I did yarn install axios hoping it would update my cache but in post-install I see this as logged
Post job cleanup.
Cache hit occurred on the primary key Linux-npm-cache-, not saving cache.
So here is my questions, Is it possible to achieve
If package.json changes, which means yarn.lock or package-lock.json changed so do npm install and update cache
Extending my above point, the contributor could be doing both yarn install and npm install
And can someone explain me this
with:
path: ~/.npm
key: ${{ runner.OS }}-npm-cache-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.OS }}-npm-cache-
In order to have efficient caching with GitHub actions there needs to be a package-lock.json or yarn.lock present. This file is automatically generated when installing packages. If you want more information about package-lock.json, check out the docs.
Now there's the subject of whether npm and yarn should be used in the same project. More about that subject here.
Based on the question, let's assume that both a package-lock.json and yarn.lock exists. If you're only using one of both, feel free to remove one from below. The config below is for yarn version 2 which uses yarn config get cacheFolder to get the cache folder. For another yarn version see the docs.
name: Tsc compilation test
on: [push, pull_request]
jobs:
build:
name: Build
runs-on: ubuntu-18.04
steps:
- name: Checkout
uses: actions/checkout#v2
- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn config get cacheFolder)"
- name: Cache yarn dependencies
uses: actions/cache#v2
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-yarn-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-yarn-
- name: Cache npm dependencies
uses: actions/cache#v2
with:
path: '~/.npm'
key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.os }}-node-
- name: Install dependencies
run: npm install # or yarn install
- name: Test tsc
run: npm run ts-compile-check
That's it! Well except for one more thing. Above we're using ~/.npm for caching npm dependencies. Performance wise it would be quicker to cache **/node_modules however in some cases this would introduce conflicts. Feel free to experiment to see which one works for you. More about that here and here.
From your question you also asked to explain the following code:
with:
path: ~/.npm
key: ${{ runner.OS }}-npm-cache-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
${{ runner.OS }}-npm-cache-
https://github.com/actions/cache#usage:
path - A list of files, directories, and wildcard patterns to cache and restore. See #actions/glob for supported patterns.
key - An explicit key for restoring and saving the cache
restore-keys - An ordered list of keys to use for restoring the cache if no cache hit occurred for key
Therefore the code above:
path is the folder that will be cached/restored (where the dependencies are installed)
key is a unique identifier for the path that will be cached. In this case it hashes the contents of any **/package-lock.json file using hashFiles. Basically when the package-lock.json file changes, that means that the dependencies changed, and the cache shouldn't be used.
restore-keys is basically a default key to use in case there's no match for the key

How to replace a API key with a Secret in code during Github Actions Job?

I usually hide my API key in an XML file and use .gitignore along with getString() to retrieve my API key.
I'm trying to use Github Actions to automate Gradle build and Release of debug apk's. Due to the XML file not being uploaded to the repo, it obviously fails.
Is there any way to store my key in Github Secrets and then replace the code with the secret?
Current code: headers["token"]= getString(R.string.token)
Replaced code in Github actions server : headers["token"]="MY_API_KEY_FROM_SECRETS"
Here's my YAML file that I was using:
name: Gradle build test and release
on:
push:
branches: [ master ]
pull_request:
branches: [ master ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
#I'd like to replace the api key here in code from secrets
- name: Make Gradle executable
run: chmod +x ./gradlew
- name: Build with Gradle
run: ./gradlew build
- name: Build Debug APK
run: ./gradlew assembleDebug
- name: Releasing using Hub
uses: ShaunLWM/action-release-debugapk#master
env:
GITHUB_TOKEN: ${{ secrets.TOKEN }}
APP_FOLDER: app
RELEASE_TITLE: New Build
BODY: github.event.head_commit.message
prerelease: true
Sure, this is possible. You haven't said what file you'd like modified, but this should be easy enough to do with a one-liner (replacing FILENAME with your config file name):
- run: perl -pi -e 's/getString\(R\.string\.token\)/"$ENV{TOKEN}"/' FILENAME
env:
TOKEN: ${{ secrets.TOKEN }}
If you prefer Ruby or some other scripting language, you can use that instead. You could also use sed, but I prefer this approach because it means that the value is never available in ps output (even if this is a locked down VM).

Github action not uploading artifact

I'm having an issue with uploading artifacts to github from within a workflow.
This is my yaml file:
on:
push:
branches:
- master
jobs:
build:
name: build and test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Install robotframework and dependencies
run: |
pip install selenium
pip install robotframework
pip install robotframework-seleniumlibrary
pip install robotframework-imaplibrary
pip install robotframework-httplibrary
pip install robotframework-requests
- name: Download and install chromedriver
run: |
wget http://chromedriver.storage.googleapis.com/77.0.3865.10/chromedriver_linux64.zip
sudo unzip chromedriver_linux64.zip -d /usr/local/bin
export CHROME_BIN=chromium-browser
- name: Run robot tests
run: |
cd robot/tests
python -m robot -i ready bookingform.robot
- name: publish test results
uses: actions/upload-artifact#v1
with:
name: report
path: report.html
- name: clean up stuff
run: |
history
pwd
Everything runs fine up until "publish test results", at which point nothing is written to the logs and no artifacts are uploaded. If I view the workflow log, there is a grey icon alongside that step (not the usual check or red x), so I'm really baffled at what could be happening. I added arbitrary stuff to the "clean up stuff" step just to test what happens, and that step isn't run either.
I've tried messing around with the path, thinking that it could be related to the path being invalid or something, but that hasn't helped. No matter what I add near the bottom of that file, the same behaviour occurs.
I have tried running another workflow file which uploads artifacts and that ran fine, the logs showed that the upload action was invoked and that the artifact was saved, but I don't see anything like that when my yaml file is used.
Each job step resets the working path to GITHUB_WORKSPACE, which will be the root of your repository after actions/checkout runs.
The upload-artifact action most likely can't find report.html because it's no longer in the correct directory.
Try changing the path as follows:
- name: publish test results
uses: actions/upload-artifact#v1
with:
name: report
path: robot/tests/report.html
There is also a working-directory that you can set for a step. However, it seems to be incompatible with uses for actions. It can only apply to run script steps.
Using working-directory with uses will NOT work:
- name: publish test results
working-directory: robot/tests
uses: actions/upload-artifact#v1
with:
name: report
path: report.html
Using working-directory with run will work:
- name: print test results
working-directory: robot/tests
run: cat report.html
Tasks are not executed, if previous Tasks have failed; see Status check functions.
Using if: ${{ failure() }} a Task only executes, if the previous Task has failed.
So for me it solved the Problem to add if: ${{ failure() }}:
- name: Upload Cypress Artifacts
if: ${{ failure() }}
uses: actions/upload-artifact#v3
with:
path: ./cypress