trying to mitigate large LFS pulls from github. incurring cost.
so was happy to see caching of large blob coming from LFS is supported in github actions.
here is sample posted on issue #165 at the actions/checkout module repo
- name: Checkout code
uses: actions/checkout#v2
- name: Create LFS file list
run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id
- name: Restore LFS cache
uses: actions/cache#v2
id: lfs-cache
with:
path: .git/lfs
key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}-v1
- name: Git LFS Pull
run: git lfs pull
- name: run compile and package
run: python CompileAndPackage.py
my current flow is the following, i declare support for lfs and set submodules support to recursive. the git LFS i call is inside a submodule.
how should i integrate the above into my flow, below.
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
lfs: 'true'
# recursively checkout submodules.
submodules: recursive
i'm not certain the checkout and submodule update (where the lfs pulling i assume occurs) will happen before i check the cache.
here is what i think of using
steps:
- uses: actions/checkout#v2
with:
lfs: 'true'
submodules: recursive
ssh-key: ${{ secrets.git_key }}
- name: Create LFS file list
run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id
- name: Restore LFS cache
uses: actions/cache#v2
id: lfs-cache
with:
path: .git/lfs
key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}-v1
- name: Git LFS Pull
run: git lfs pull
should i be explicit about calling git submodule update --init AFTER i check the cache? afaik it happens automagicly by actions/checkout
UPDATE
hey, that looks like it actually worked!
unfrountly github doesnt go into details regarding who took a bite of the LFS quota. would be glad to hear someone says it is legit.
I created an action for this, action-cached-lfs-checkout.
Instead of
- name: Checkout code
uses: actions/checkout#v2
with:
lfs: true
simply do
- name: Checkout code
uses: nschloe/action-cached-lfs-checkout#v1
this is sorted, see update. im closing
Related
I recently started working on using some GitHub actions on my projects. I am able to setup them up to run automatically but am struggling with having them run manually. I know that you need the have the workflow_dispatch in the on section. I'm not sure if it's not working because I have it automatically run too. Is someone able to tell me what I am doing wrong?
Here is one of my workflow YAML files
name: Create-Doc-Nightly
on:
push:
branches: [ "nightly" ]
paths:
- 'src/**'
- 'pom.xml'
workflow_dispatch:
jobs:
doc:
name: Create Doc
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
name: Step 1 - Checkout Nightly Branch
with:
persist-credentials: false
fetch-depth: 0
- name: Step 2 - Setup JDK 17
uses: actions/setup-java#v3.4.1
with:
java-version: 17
distribution: 'temurin'
- name: Step 3 - Remove Doc
run: |
git remote set-url origin https://jnstockley:${{ secrets.TOKEN }}#github.com/jnstockley/BTTN.git
git config user.email "jack#jstockley.com"
git config --local user.name "Jack Stockley"
git rm -r docs
git commit -m "Removed Docs"
git push origin nightly
- name: Step 4 - Create Doc
run: mvn dokka:dokka -f pom.xml
- name: Step 5 - Move Docs
run: |
rm -rf docs
mkdir -p docs
mv target/dokka/* docs
- name: Step 6 - Publish docs
run: |
git remote set-url origin https://jnstockley:${{ secrets.TOKEN }}#github.com/jnstockley/BTTN.git
git config user.email "jack#jstockley.com"
git config --local user.name "Jack Stockley"
git add -f docs
git commit -m "Updated Docs"
git push origin nightly
Link to GitHub repo, nightly branch: https://github.com/jnstockley/BTTN/tree/nightly
The workflow must be on your default branch in order to use workflow_dispatch.
I believe in your case it's only on the branch nightly while it should also be on main.
To manually trigger a workflow, use the workflow_dispatch event. You can manually trigger a workflow run using the GitHub API, GitHub CLI, or GitHub browser interface. For more information, see Manually running a workflow
on: workflow_dispatch
Providing inputs
You can configure custom-defined input properties, default input values, and required inputs for the event directly in your workflow. When you trigger the event, you can provide the ref and any inputs. When the workflow runs, you can access the input values in the inputs context. For more information, see Contexts
This example defines inputs called logLevel, tags, and environment. You pass values for these inputs to the workflow when you run it. This workflow then prints the values to the log, using the inputs.logLevel, inputs.tags, and inputs.environment context properties.
yaml
on:
workflow_dispatch:
inputs:
logLevel:
description: 'Log level'
required: true
default: 'warning'
type: choice
options:
- info
- warning
- debug
tags:
description: 'Test scenario tags'
required: false
type: boolean
environment:
description: 'Environment to run tests against'
type: environment
required: true
jobs:
log-the-inputs:
runs-on: ubuntu-latest
steps:
- run: |
echo "Log level: $LEVEL"
echo "Tags: $TAGS"
echo "Environment: $ENVIRONMENT"
env:
LEVEL: ${{ inputs.logLevel }}
TAGS: ${{ inputs.tags }}
ENVIRONMENT: ${{ inputs.environment }}
If you run this workflow from a browser you must enter values for the required inputs manually before the workflow will run.
You might like the following documentation links
workflow_dispatch
github docs - events-that-trigger-workflows
How can I push some files that were generated by the runner (user1/repo1) to the main branch from another remote repo (user2/repo1) via GitHub actions?
Please note that:
I set-up a secret key (named ACCESS_TOKEN) in user1/repo1, such that it corresponds to the Personal Access Token from the destination repo (user2/repo1)
the GitHub actions needs to be repeated every ~30 minutes
there already exists a file.rds in the destination repo. The push thus needs to override that file every time
the runner needs to be macOS-latest
This is what I have tried so far:
name: gitaction
on:
schedule:
- cron: "*/30 * * * *"
workflow_dispatch:
jobs:
genFileAndPush:
runs-on: macOS-latest
steps:
- uses: actions/checkout#master
- uses: r-lib/actions/setup-r#master
with:
r-version: '4.1.2'
- name: Run R scripts and generate file
run: |
saveRDS(1:3, file = "file.rds")
shell: Rscript {0}
- name: Push to remote repository
run: |
git config --local user.name actions-user
git config --local user.email "actions#github.com"
git add file.rds
git commit -m "commit"
git remote set-url origin https://env.REPO_KEY#github.com/user2/repo1.git
git push -u origin main
env:
REPO_KEY: ${{secrets.ACCESS_TOKEN}}
username: github-actions
It returns the following error:
remote: Permission to user2/repo1.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/user2/repo1.git/': The requested URL returned error: 403
Error: Process completed with exit code 128.
What am I missing?
Edit
As suggested, I tried using GuillaumeFalourd/git-commit-push#v1.1:
name: gitaction
on:
workflow_dispatch:
jobs:
genFileAndPush:
runs-on: macOS-latest
steps:
- uses: actions/checkout#master
- uses: r-lib/actions/setup-r#master
with:
r-version: '4.1.2'
- name: Run R scripts and generate file
run: |
saveRDS(1:3, file = "file.rds")
shell: Rscript {0}
- uses: actions/checkout#v2.3.4
- uses: GuillaumeFalourd/git-commit-push#v1.1
with:
target_branch: main
files: file.rds
remote_repository: https://github.com/user2/repo1
access_token: ${{secrets.ACCESS_TOKEN}}
force: true
Although there were no error, the file was not pushed (because it was not detected?):
Run GuillaumeFalourd/git-commit-push#v1.1
Run CURRENT_BRANCH=${GITHUB_REF}
WARNING: No changes were detected. git commit push action aborted.
There are some actions on the Github Marketplace that can help you with pushing files to other repositories.
Here is an example of one supported on all OS runners.
The workflow would look like this:
name: gitaction
on:
workflow_dispatch:
jobs:
genFileAndPush:
runs-on: macOS-latest
steps:
- uses: actions/checkout#master
- uses: r-lib/actions/setup-r#master
with:
r-version: '4.1.2'
- name: Run R scripts and generate file
run: |
saveRDS(1:3, file = "file.rds")
shell: Rscript {0}
- uses: GuillaumeFalourd/git-commit-push#v1.3
with:
target_branch: main
files: file.rds
remote_repository: https://github.com/user2/repo1
access_token: ${{secrets.ACCESS_TOKEN}}
force: true
You can find more actions like this one on the marketplace.
Otherwise, you can also perform the whole operation manually using command lines to clone the remote repository, copy the files from the local repo wherever you want on the remote repo, then push the new files to the remote repository.
I'm trying to setup GitHub action to check for lint errors and fail the pull request if any error/ warnings detected.
Currently my logic works locally but when I try to run it via GitHub action, I get an error:
fatal: ambiguous argument 'origin/master...HEAD': unknown revision or
path not in the working tree.
I believe it's something to do with checkout#v2 not fetching the right amount of data, But I cant get my head around what
Code Sample
name: Initiate PR
on: push
jobs:
STEPS:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
with:
fetch-depth: 100
- name: Set up Node.js
uses: actions/setup-node#v1
with:
node-version: 14.18.0
- name: Install Node.js dependencies
run: npm ci --ignore-scripts
- name: lint-on-PR
shell: bash
run: |
npx eslint --max-warnings 0 $(git diff origin/master...HEAD --name-only --relative --diff-filter=MATR '***.js' '***.jsx' '***.ts' '***.tsx' | xargs)
You would probably need to do a checkout#v1 as in this example to get all the files.
- uses: actions/checkout#v1
...
- run: git diff ${{ github.event.pull_request.base.sha }} ${{ github.sha }}
v2 by default only fetches the sha that triggered the action.
In GitHub action on pull request, I need to run some code in the context of the "current master", and later re-run the same code in the context of the PR branch.
I can check out compare a pull request to the base it is being PR-ed against. How would I find the SHA of the base branch (e.g. current master if PR is against the master)?
jobs:
job_on_base:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
ref: "${{ github.base_ref }}"
- run: |
# Seems like I can get it here with $(git log -1 --format="%H")
echo "My current SHA is ... ?"
job_on_pr:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
ref: "${{ github.ref }}"
- run: |
echo "My current SHA is $GITHUB_SHA"
echo "The BASE SHA is ?"
If the job runs on a pull_request event the base sha is available as ${{ github.event.pull_request.base.sha }}
This turned out to be a git question, rather than Github actions. The actions/checkout#v2 creates a shallow --depth=1 clone, so to get PR's parent one can parse git cat-file -p output as described here. The first (base) parent could be accessed with
git cat-file -p <SHA> | awk 'NR > 1 {if(/^parent/){print $2; exit}}'
The better approach turned out to be using fetch-depth: 2 parameter. It allows just one job to handle both pull request and master merge cases, and can also be used with HEAD^1 to get to the parent.
steps:
- uses: actions/checkout#v2
with:
fetch-depth: 2
What worked for me was: github.event.workflow_run.head_sha.
The top-voted answer suggesting github.event.pull_request.head.sha didn't work for me.
I found this by examining all the possible data in the github.context object using the method suggested here - https://stackoverflow.com/a/70107953
I have an application which is in a private github repo, and am wondering if the releases could be made public so that the app can auto-update itself from github instead of us having to host it.
Additionally I'm wondering if it's possible to use the github api from the deployed app to check for new updates.
A workaround would be to create a public repo, composed of:
empty commits (git commit --allow-empty)
each commit tagged
each tag with a release
each release with the deliveries (the binaries of your private app)
That way, you have a visible repo dedicated for release hosting, and a private repos for source development.
As #VonC mentioned we have to create a second Repository for that. This is not prohibited and i am doing it already. With github workflows i automated this task, I'm using a develop / master branching, so always when I'm pushing anything to the master branch a new version is build and pushed to the public "Realease" Repo.
In my specific use case I'm building an android apk and releasing it via unoffical github api "hub". Some additional advantage of this is you can have an extra issue tracker for foreign issues and bugs.
name: Master CI CD
# using checkout#v2 instead of v1 caus it needs further configuration
on:
pull_request:
types: [closed]
jobs:
UnitTest:
runs-on: ubuntu-latest
if: github.event.pull_request.merged
steps:
- uses: actions/checkout#v2
- name: make executable
run: chmod +x gradlew
- name: Unit tests
run: |
./gradlew test
IncrementVersionCode:
needs: UnitTest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: make executable
run: chmod +x gradlew
- name: increment version
run: ./gradlew incrementVersionCode
- name: Push new version to master
run: |
git config --local user.email "workflow#bot.com"
git config --local user.name "WorkflowBot"
git commit -m "Increment Build version" -a
# maybe better amend commits to avoid bot commits
BuildArtifacts:
needs: IncrementVersionCode
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: make executable
run: chmod +x gradlew
- name: Build with Gradle
run: ./gradlew build -x lint
- name: Rename artifacts
run: |
cp app/build/outputs/apk/release/app-release.apk MyApp.apk
- name: Upload Release
uses: actions/upload-artifact#master
with:
name: Release Apk
path: MyApp.apk
- name: Upload Debug
uses: actions/upload-artifact#master
with:
name: Debug Apk
path: app/build/outputs/apk/debug/app-debug.apk
# https://dev.to/ychescale9/running-android-emulators-on-ci-from-bitrise-io-to-github-actions-3j76
E2ETest:
needs: BuildArtifacts
runs-on: macos-latest
strategy:
matrix:
api-level: [21, 27]
arch: [x86]
steps:
- name: checkout
uses: actions/checkout#v2
- name: Make gradlew executable
run: chmod +x ./gradlew
- name: run tests
uses: reactivecircus/android-emulator-runner#v2
with:
api-level: ${{ matrix.api-level }}
arch: ${{ matrix.arch }}
script: ./gradlew connectedCheck
Deploy:
needs: E2ETest
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/master'
steps:
- uses: actions/checkout#v2 # Needed for gradle file to get version information
- name: Get Hub
run: |
curl -fsSL https://github.com/github/hub/raw/master/script/get | bash -s 2.14.1
cd bin
chmod +x hub
cd ..
- name: Get Apk
uses: actions/download-artifact#master
with:
name: Release Apk
- name: Publish
env:
GITHUB_TOKEN: "${{ secrets.RELEASE_REPO_SECRET }}"
run: |
APP_NAME=MyApp
VERSION_NAME=`grep -oP 'versionName "\K(.*?)(?=")' ./app/build.gradle`
VERSION_CODE=`cat version.properties | grep "VERSION_CODE" | cut -d'=' -f2`
FILENAME="${APP_NAME}-v${VERSION_NAME}-${VERSION_CODE}"
TAG="v${VERSION_NAME}-${VERSION_CODE}"
TAG="latest-master"
echo $APP_NAME
echo $VERSION_NAME
echo $VERSION_CODE
echo $FILENAME
echo $TAG
git clone https://github.com/MyUser/MyApp-Releases
cd MyApp-Releases
./../bin/hub release delete "${TAG}" || echo "Failed deleting TAG: ${TAG}" # If release got lost catch error with message
./../bin/hub release create -a "../${APP_NAME}.apk" -m "Current Master Build: ${FILENAME}" -p "${TAG}"
EvaluateCode:
needs: Deploy
runs-on: ubuntu-latest
steps:
- name: Get Hub
run: |
echo "TDOO: Run Jacoco for coverage, and other profiling tools"
The 2022 answer to this question is even more straight-forward.
You'd just need to use the pre-installed gh CLI:
gh release create v0.0.1 foobar.zip -R https://github.com/your/repo-here
This command will create a tag v0.0.1 and a release with the local file foobar.zip attached on the public repository. You can run this in the GitHub Action of any private repository.
The -R argument points to the repository you'd like to create a tag/release on. foobar.zip would be located in your local directory.
One thing is important here: GITHUB_TOKEN must still be set as the token of the repository you'd like to release on!
Full example:
- name: Publish
env:
GITHUB_TOKEN: "${{ secrets.RELEASE_REPO_SECRET }}"
run: |
gh release create v0.0.1 foobar.zip -R https://github.com/your/repo-here
If you're planning to re-release and override existing versions, there is gh release delete as well. The -d flag creates a release as a draft etc. pp. Please take a look at the docs.
I'm using a slightly more advanced approach by setting:
shell: bash
run: $GITHUB_ACTION_PATH/scripts/publish.sh
And in file scripts/publish.sh:
#!/usr/bin/env node
const cp = require('child_process')
const fs = require('fs');
const path = require('path');
const APP_VERSION = JSON.parse(fs.readFileSync('package.json', { encoding: 'utf8' })).version
const TAG = `v${APP_VERSION}`
cp.execSync(`gh release create ${TAG} foobar.zip -R https://github.com/your/repo-name`, { stdio: 'inherit' })
This approach enables you to be able to for example, use Node.js or any other programming language available, to extract a version from the project management file of choice (e.g. package.json) and automatically come up with the right tag version and name.
A simple way to duplicate releases from a private repo to a public one may be this Release & Assets Github Action which can: Create a release, upload release assets, and duplicate a release to other repository.
Then you can use the regular electron-builder updater's support for public repos.