Github Actions environment variables not being set on Linux - github

I'm trying to set my Github secrets as env variables via Actions but for some reasons they aren't being set (Using Ubuntu 22.04.1 LTS).
name: My app
on:
push:
branches: [ "main" ]
env:
JWT_SECRET_KEY: ${{ secrets.JWT_SECRET_KEY }}
JWT_REFRESH_SECRET: ${{ secrets.JWT_REFRESH_SECRET }}
MONGO_CONNECTION_STRING: ${{ secrets.MONGO_CONNECTION_STRING }}
PLAI_APP_URL: ${{ secrets.APP_URL }}
SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }}
permissions:
contents: read
jobs:
build:
runs-on: self-hosted
steps:
- name: check out latest repo
uses: actions/checkout#v3
with:
ref: main
- name: Set up Python 3.10
uses: actions/setup-python#v3
with:
python-version: "3.10"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
Workflow runs successfully without errors but when I check my variables via env they are not listed. Any idea what I'm doing wrong?

You could follow the same idea as in "Managing secrets in an open-sourced flutter web app" from Mahesh Jamdade:
encode your confidential file to base64 and store the encoded output to GitHub secrets.
In your GitHub workflow decode the encrypted secret and convert it back to a file and then run your build scripts.
You can see an example in "How to access secrets when using flutter web with github actions".
Example workflow: maheshmnj/vocabhub .github/workflows/firebase-hosting-merge.yml.

Related

Releasing and Publishing via GH actions

I am trying to automate publishing the SDKs for Python, Java, GO, and Node. My main goal is to make the CI run whenever a new PR is created against main branch that will:
bump the version in all files.
publish the new release to the related public registry (for each language)
Problem:
right now the problem is that the publish step is not taking the artifacts from the release step, but rather the one before that, as if they are not synced.
For the release step, we're using semantic-release package with several plugins.
The ADMIN_TOKEN is a personal token of a user with write permissions.
The publishing step is different for each language, but I am certain this is unrelated since it worked before I complicated the workflow.
Possible issue:
Without the if statements, the release and publish steps are synced, but then the semantic-release creates another commit that creates another release (e.g. 2 releases and publishing in one run, not wanted). With the current if, the publish step takes the older release instead the newly created one (for example, if the new run creates release 1.0.40, the publish will take version 1.0.39).
Does anyone have some input on these 2 steps or the if statements? For example, this is the current variation of the Java workflow:
release:
runs-on: ubuntu-latest
if: "!startsWith(github.event.head_commit.message, 'chore')"
steps:
- name: Checkout code
uses: actions/checkout#v3
with:
fetch-depth: 0
token: ${{ secrets.ADMIN_TOKEN }}
- name: setup nodejs
uses: actions/setup-node#v3
with:
node-version: '16'
- name: release using semantic-release
env:
GITHUB_TOKEN: ${{ secrets.ADMIN_TOKEN }}
GIT_AUTHOR_NAME: ****
GIT_AUTHOR_EMAIL: ****
GIT_COMMITTER_NAME: ****
GIT_COMMITTER_EMAIL: ****
run: |
sudo apt-get update
sudo apt-get install python
pip install --user bumpversion
npm install #semantic-release/changelog
npm install #semantic-release/exec
npm install #semantic-release/git
npm install #semantic-release/github
npx semantic-release
publish:
runs-on: ubuntu-latest
needs: [release]
if: "!startsWith(github.event.head_commit.message, 'chore')"
steps:
- name: Checkout code
uses: actions/checkout#v3
with:
token: ${{ secrets.ADMIN_TOKEN }}
- name: Configure GPG Key
run: |
cat <(echo -e "${{ secrets.GPG_SIGNING_KEY }}") | gpg --batch --import
gpg --list-secret-keys --keyid-format LONG
- name: Set up Maven Central Repository
uses: actions/setup-java#v3
with:
java-version: 8
distribution: zulu
server-id: ossrh
server-username: ${{ secrets.MAVEN_USERNAME }}
server-password: ${{ secrets.MAVEN_PASSWORD }}
gpg-passphrase: ${{ secrets.GPG_PASSPHRASE }}
- name: Publish package
run: mvn clean deploy $MVN_ARGS -P central --no-transfer-progress --batch-mode -Dgpg.passphrase=${{ secrets.GPG_PASSPHRASE }}
env:
MAVEN_USERNAME: ${{ secrets.MAVEN_USERNAME }}
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
MVN_ARGS: "--settings build-settings.xml"
<more ENVS>
In case it is relevant, the .releaserc file is:
{
"debug": true,
"branches": [ "main" ],
"plugins": [
["#semantic-release/commit-analyzer", {
"preset": "angular",
"releaseRules": [
{"type": "release","release": "patch"}
]}],
"#semantic-release/release-notes-generator",
"#semantic-release/changelog",
[
"#semantic-release/exec",
{
"prepareCmd": "bump2version --allow-dirty --current-version ${lastRelease.version} --new-version ${nextRelease.version} patch"
}
],
[
"#semantic-release/git",
{
"message": "chore(release): ${nextRelease.version} release notes\n\n${nextRelease.notes}"
}
],
"#semantic-release/github"
]
}
I also asked in GH: https://github.com/orgs/community/discussions/40749
The quick fix I found is to split the release and publish steps into two different workflows (different files). I am certain with a bit more dive-in, one can merge those two with some proper if conditioning.
NOTE: The publish action steps are specific to Java, but can be changed to be valid for any other language. The main structure is the main answer here.
The release step:
The semantic-release creates a secondary commit to the main branch with "chore" commit message. in order to overcome this, I added the if to skip this type of commit.
name: release
on:
workflow_dispatch:
push:
branches:
- main
jobs:
release:
runs-on: ubuntu-latest
if: "github.event_name == 'push' && github.ref == 'refs/heads/main' && !startsWith(github.event.head_commit.message, 'chore')"
steps:
- name: Checkout code
uses: actions/checkout#v3
with:
fetch-depth: 0
token: ${{ secrets.ADMIN_TOKEN }}
- name: setup nodejs
uses: actions/setup-node#v3
with:
node-version: '16'
- name: release using semantic-release
env:
GITHUB_TOKEN: ${{ secrets.ADMIN_TOKEN }}
GIT_AUTHOR_NAME: secrets.automation.dev
GIT_AUTHOR_EMAIL: secrets.automation.dev#il.ibm.com
GIT_COMMITTER_NAME: secrets.automation.dev
GIT_COMMITTER_EMAIL: secrets.automation.dev#il.ibm.com
run: |
sudo apt-get update
sudo apt-get install python
pip install --user bumpversion
npm install #semantic-release/changelog
npm install #semantic-release/exec
npm install #semantic-release/git
npm install #semantic-release/github
npx semantic-release
The publish step:
The "release" event has several initiators so I added the published type to make sure the publishing happens only if a new release was published to GitHub.
name: publish artifact
on:
workflow_dispatch:
release:
types: [published]
jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout#v3
with:
token: ${{ secrets.ADMIN_TOKEN }}
- name: Configure GPG Key
run: |
cat <(echo -e "${{ secrets.GPG_SIGNING_KEY }}") | gpg --batch --import
gpg --list-secret-keys --keyid-format LONG
- name: Set up Maven Central Repository
uses: actions/setup-java#v3
with:
java-version: 8
distribution: zulu
server-id: ossrh
server-username: ${{ secrets.MAVEN_USERNAME }}
server-password: ${{ secrets.MAVEN_PASSWORD }}
gpg-passphrase: ${{ secrets.GPG_PASSPHRASE }}
- name: Publish package
run: mvn clean deploy $MVN_ARGS -P central --no-transfer-progress --batch-mode -Dgpg.passphrase=${{ secrets.GPG_PASSPHRASE }}
env:
MAVEN_USERNAME: ${{ secrets.MAVEN_USERNAME }}
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
MVN_ARGS: "--settings build-settings.xml"
<other envs>

How do I enable GitHub-hosted runners for a GitHub Action?

I created a workflow for my Python repo as follows:
name: Python package
on: [push, pull_request]
jobs:
build:
runs-on: [ubuntu-latest, macos-latest]
strategy:
fail-fast: false
matrix:
python-version: ["3.7", "3.8", "3.9", "3.10"]
steps:
- uses: actions/checkout#v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python#v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8 pytest semver
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings.
flake8 . --count --exit-zero --max-complexity=10 --ignore=E501 --statistics
- name: Test with pytest
run: |
pytest
Unfortunately, the action never runs and times out with the error:
This request was automatically failed because there were no enabled runners online to process the request for more than 1 days.
Did I do something silly in the configuration file?
I'm currently on a free GitHub account. Are GitHub-hosted runners available on free accounts? If so how do I enable one of those?
Turns out
runs-on: [ubuntu-latest, macos-latest]
doesn't run the action on each platforms. Instead it tries to find a runner that satisfies both conditions, i.e. running on ubuntu-latest and macos-latest which is, of course, never found.
The way to so what I originally intended is to, instead, do a two-dimensional matrix for os and python-version.

Deploy on Azure Function using GitHub Actions on push only if Function App is not executing

I have a function app, which basically scrapes data from the web. It is a long-running one, which generally takes 9 hrs a day.
I have configured on push event for build & deploy through GitHub Actions.
Problem: When we push any change to GitHub and the function app is running it will create a mess as the running function will be stopped and triggered after deployment.
I want a solution to deploy on every push but only when the function app is not running.
Content of yml file:
name: dev-workflow
on:
push:
branches:
- main
env:
AZURE_FUNCTIONAPP_NAME: test-github-actions
AZURE_FUNCTIONAPP_PACKAGE_PATH: '.'
PYTHON_VERSION: '3.7'
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- name: 'Checkout GitHub Action'
uses: actions/checkout#main
- name: Setup Python ${{ env.PYTHON_VERSION }} Environment
uses: actions/setup-python#v1
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: 'Resolve Project Dependencies Using Pip'
shell: bash
run: |
pushd './${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
python -m pip install --upgrade pip
pip install -r requirements.txt --target=".python_packages/lib/site-packages"
popd
- name: 'Run Azure Functions Action'
uses: Azure/functions-action#v1
id: fa
with:
app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
package: ${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}
publish-profile: ${{ secrets.PUBLISH_PROFILE }}
According to this post response from chrispat (6 months ago) there isn't
any native feature to prevent duplicate workflows currently on Github Actions.
However, there is this action that might help to prevent duplicated actions to start.
Another option could be to limit concurrent workflows runs, you can find more about it in this post.

Download private module from Github Package Registry via Yarn within a Github Action? Publishing works, but installing is met with '401 Unauthorized'

For various reasons we are stuck using yarn managing our packages so we can't rely on a package-lock.json to use npm with github actions.
We cannot get Yarn to authenticate as part of a github action.
We've got our repo npmrc configured as:
#COMPANY:registry=https://npm.pkg.github.com
registry=https://registry.npmjs.org/
And we're using this action for yarn.
Here's a basic setup where we're just trying to install the modules -- nothing more.
name: CI
on: [push]
jobs:
build:
name: Test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- uses: borales/actions-yarn#v2.1.0
with:
auth-token: ${{ secrets.GITHUB_TOKEN }}
registry-url: "https://npm.pkg.github.com"
scope: tlabs
cmd: version
env:
NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_REGISTRY_URL: https://npm.pkg.github.com
- name: Create NPMRC
run: |
echo "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}" > ~/.npmrc
- name: Install
run: |
yarn install --verbose
By default, this action will try to run install so to bypass that I provided a basic command there 'version' so it just displays the yarn version and nothing more.
Running yarn install will work for all other packages but when it gets to our private modules, it will try to get them from the right registry (github) but will be hit with a 401.
Full error:
verbose 7.614802156 Error: https://npm.pkg.github.com/download/#tlabs/utils/1.0.1/afe9eaa6f9565f95c31563cbecfe617d7970f44077302cbe9ca8ee3223550469: Request failed "401 Unauthorized"
at ResponseError.ExtendableBuiltin (/usr/share/yarn/lib/cli.js:696:66)
at new ResponseError (/usr/share/yarn/lib/cli.js:802:124)
at Request.<anonymous> (/usr/share/yarn/lib/cli.js:66996:16)
at Request.emit (events.js:210:5)
at Request.module.exports.Request.onRequestResponse (/usr/share/yarn/lib/cli.js:141441:10)
at ClientRequest.emit (events.js:210:5)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:583:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:115:17)
at TLSSocket.socketOnData (_http_client.js:456:22)
at TLSSocket.emit (events.js:210:5)
error An unexpected error occurred: "https://npm.pkg.github.com/download/#tlabs/utils/1.0.1/afe9eaa6f9565f95c31563cbecfe617d7970f44077302cbe9ca8ee3223550469: Request failed \"401 Unauthorized\"".
The default GITHUB_TOKEN is only scoped for the current repository. You cannot use it to access packages in another repository. Use a read:packages and repo scoped Personal Access Token instead of GITHUB_TOKEN.
I'm create a file .npmrc and .yarnrc.
Type:
name: Test
on: push
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [12.x]
steps:
- uses: actions/checkout#v2
- name: Node ${{ matrix.node-version }}
uses: actions/setup-node#v1
with:
node-version: ${{ matrix.node-version }}
- name: Create NPMRC
run: |
echo "//npm.pkg.github.com/:_authToken=${{ secrets.PACKAGES_TOKEN }}" >> ~/.npmrc
echo "#you-scope:registry=https://npm.pkg.github.com" >> ~/.npmrc
echo 'registry "https://registry.yarnpkg.com"' >> ~/.yarnrc
- run: yarn install
Replace #you-scope for you user of github or of your org in github in LowerCase.
Create a PACKAGES_TOKEN screte for this repository.
Have a .npmrc file in root of your project.
Content of .npmrc:
registry=https://registry.npmjs.org/
#{scope}:registry=https://npm.pkg.github.com/
//npm.pkg.github.com/:_authToken=********** (Token generated from github)
#{scope} is your organization-name or your username. It is case-sensitive.
Also, to access both private and public packages in github registry, you need to have a token.
Reference: You need an access token to publish, install, and delete packages.

Can I make releases public from a private github repo?

I have an application which is in a private github repo, and am wondering if the releases could be made public so that the app can auto-update itself from github instead of us having to host it.
Additionally I'm wondering if it's possible to use the github api from the deployed app to check for new updates.
A workaround would be to create a public repo, composed of:
empty commits (git commit --allow-empty)
each commit tagged
each tag with a release
each release with the deliveries (the binaries of your private app)
That way, you have a visible repo dedicated for release hosting, and a private repos for source development.
As #VonC mentioned we have to create a second Repository for that. This is not prohibited and i am doing it already. With github workflows i automated this task, I'm using a develop / master branching, so always when I'm pushing anything to the master branch a new version is build and pushed to the public "Realease" Repo.
In my specific use case I'm building an android apk and releasing it via unoffical github api "hub". Some additional advantage of this is you can have an extra issue tracker for foreign issues and bugs.
name: Master CI CD
# using checkout#v2 instead of v1 caus it needs further configuration
on:
pull_request:
types: [closed]
jobs:
UnitTest:
runs-on: ubuntu-latest
if: github.event.pull_request.merged
steps:
- uses: actions/checkout#v2
- name: make executable
run: chmod +x gradlew
- name: Unit tests
run: |
./gradlew test
IncrementVersionCode:
needs: UnitTest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: make executable
run: chmod +x gradlew
- name: increment version
run: ./gradlew incrementVersionCode
- name: Push new version to master
run: |
git config --local user.email "workflow#bot.com"
git config --local user.name "WorkflowBot"
git commit -m "Increment Build version" -a
# maybe better amend commits to avoid bot commits
BuildArtifacts:
needs: IncrementVersionCode
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: set up JDK 1.8
uses: actions/setup-java#v1
with:
java-version: 1.8
- name: make executable
run: chmod +x gradlew
- name: Build with Gradle
run: ./gradlew build -x lint
- name: Rename artifacts
run: |
cp app/build/outputs/apk/release/app-release.apk MyApp.apk
- name: Upload Release
uses: actions/upload-artifact#master
with:
name: Release Apk
path: MyApp.apk
- name: Upload Debug
uses: actions/upload-artifact#master
with:
name: Debug Apk
path: app/build/outputs/apk/debug/app-debug.apk
# https://dev.to/ychescale9/running-android-emulators-on-ci-from-bitrise-io-to-github-actions-3j76
E2ETest:
needs: BuildArtifacts
runs-on: macos-latest
strategy:
matrix:
api-level: [21, 27]
arch: [x86]
steps:
- name: checkout
uses: actions/checkout#v2
- name: Make gradlew executable
run: chmod +x ./gradlew
- name: run tests
uses: reactivecircus/android-emulator-runner#v2
with:
api-level: ${{ matrix.api-level }}
arch: ${{ matrix.arch }}
script: ./gradlew connectedCheck
Deploy:
needs: E2ETest
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/master'
steps:
- uses: actions/checkout#v2 # Needed for gradle file to get version information
- name: Get Hub
run: |
curl -fsSL https://github.com/github/hub/raw/master/script/get | bash -s 2.14.1
cd bin
chmod +x hub
cd ..
- name: Get Apk
uses: actions/download-artifact#master
with:
name: Release Apk
- name: Publish
env:
GITHUB_TOKEN: "${{ secrets.RELEASE_REPO_SECRET }}"
run: |
APP_NAME=MyApp
VERSION_NAME=`grep -oP 'versionName "\K(.*?)(?=")' ./app/build.gradle`
VERSION_CODE=`cat version.properties | grep "VERSION_CODE" | cut -d'=' -f2`
FILENAME="${APP_NAME}-v${VERSION_NAME}-${VERSION_CODE}"
TAG="v${VERSION_NAME}-${VERSION_CODE}"
TAG="latest-master"
echo $APP_NAME
echo $VERSION_NAME
echo $VERSION_CODE
echo $FILENAME
echo $TAG
git clone https://github.com/MyUser/MyApp-Releases
cd MyApp-Releases
./../bin/hub release delete "${TAG}" || echo "Failed deleting TAG: ${TAG}" # If release got lost catch error with message
./../bin/hub release create -a "../${APP_NAME}.apk" -m "Current Master Build: ${FILENAME}" -p "${TAG}"
EvaluateCode:
needs: Deploy
runs-on: ubuntu-latest
steps:
- name: Get Hub
run: |
echo "TDOO: Run Jacoco for coverage, and other profiling tools"
The 2022 answer to this question is even more straight-forward.
You'd just need to use the pre-installed gh CLI:
gh release create v0.0.1 foobar.zip -R https://github.com/your/repo-here
This command will create a tag v0.0.1 and a release with the local file foobar.zip attached on the public repository. You can run this in the GitHub Action of any private repository.
The -R argument points to the repository you'd like to create a tag/release on. foobar.zip would be located in your local directory.
One thing is important here: GITHUB_TOKEN must still be set as the token of the repository you'd like to release on!
Full example:
- name: Publish
env:
GITHUB_TOKEN: "${{ secrets.RELEASE_REPO_SECRET }}"
run: |
gh release create v0.0.1 foobar.zip -R https://github.com/your/repo-here
If you're planning to re-release and override existing versions, there is gh release delete as well. The -d flag creates a release as a draft etc. pp. Please take a look at the docs.
I'm using a slightly more advanced approach by setting:
shell: bash
run: $GITHUB_ACTION_PATH/scripts/publish.sh
And in file scripts/publish.sh:
#!/usr/bin/env node
const cp = require('child_process')
const fs = require('fs');
const path = require('path');
const APP_VERSION = JSON.parse(fs.readFileSync('package.json', { encoding: 'utf8' })).version
const TAG = `v${APP_VERSION}`
cp.execSync(`gh release create ${TAG} foobar.zip -R https://github.com/your/repo-name`, { stdio: 'inherit' })
This approach enables you to be able to for example, use Node.js or any other programming language available, to extract a version from the project management file of choice (e.g. package.json) and automatically come up with the right tag version and name.
A simple way to duplicate releases from a private repo to a public one may be this Release & Assets Github Action which can: Create a release, upload release assets, and duplicate a release to other repository.
Then you can use the regular electron-builder updater's support for public repos.