Github workflow with private repo & tag - github

I'll start with I asked this question here and got no response: https://github.community/t/private-repo-w-tag-in-workflow/229573
We have three private repos with tags in our package.json as dependencies, one example:
"Private-Repo1": "https://<PAT>:x-oauth-basic#github.com/project/Private-Repo.git#v1.0.0",
We use oauth keys to access our repos. My PAT is set to allow checking out the repo as well as workflow access.
When we run our Workflow action, it fails at npm ci for this line with an error of:
npm ERR! code 128
npm ERR! An unknown git error occurred
npm ERR! command git --no-replace-objects ls-remote ***github.com/project/Private-Repo.git
npm ERR! remote: Repository not found.
npm ERR! fatal: repository 'https://github.com/project/Private-Repo.git/' not found
Local testing is pointing to the reason that we’re failing is that git ls-remote fails when you point to a private repo with a tag number, if I remove the tag it works.
Can someone please point me to how we can use a PAT to pull a specific tag from a private repo in our workflow via our package.json? Everything I can find is how to access a private repo, but not how to access a private repo's tag.

For anyone that stumbles on this with a similar issue, the problem wasn't git ls-remote it was the token. I was calling it in the wrong place. It needs to be set in the checkout step, not setup-node step. Here is my working yaml that allows me to run a workflow with a private repo and tag that uses an oauth token. The only setup needed is to make a secret called GIT_TOKEN (or whatever you want to call it) and give it workflow access.
name: API auto test and lint workflow
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
token: ${{ secrets.GIT_TOKEN }}
- uses: actions/setup-node#v1
with:
node-version: 16.x
- run: npm ci
- run: npm run lint
- run: npm run test

Related

Github Actions failing to install a NPM and fails verification

Github Actions build script
name: Deploy
on:
push:
branches: [ "main" ]
jobs:
build_on_mac:
runs-on: macos-latest
steps:
- uses: actions/checkout#v3
with:
persist-credentials: false
- name: Use HTTP
run: >
git config --global url."https://github.com".insteadOf ssh://git#github.com
- uses: actions/setup-node#v3
with:
node-version: 14
- name: install dependencies
run: npm install
It keeps falling for a particular package randomly.. worked 1 out of 10 times
npm WARN tarball tarball data for buble#git+ssh://git#github.com/pemrouz/buble.git#4e639aeeb64712ac95dc30a52750d1ee4432c9c8 (sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA==) seems to be corrupted. Trying one more time.
7
npm ERR! Verification failed while extracting buble#git+ssh://git#github.com/pemrouz/buble.git#4e639aeeb64712ac95dc30a52750d1ee4432c9c8:
2895
npm ERR! sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA== integrity checksum failed when using sha512: wanted sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA== but got sha512-J+yRnScDV19Vr5+C8D5IJiIN2auC9t54tYpJmeqVxgyIyJQmF95mqBwBXzXKyvIH9aZvY6RlQqtMsQm0gdH7UQ==. (801561 bytes)
2635
So the problem is that when package-lock.json is generated locally it has
sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA==
But when getting downloaded by actions it is
sha512-J+yRnScDV19Vr5+C8D5IJiIN2auC9t54tYpJmeqVxgyIyJQmF95mqBwBXzXKyvIH9aZvY6RlQqtMsQm0gdH7UQ==
Workaround is to replace the SHA in GitHub with expected one which allows build to complete but obviously that looks wrong..
So options I can think are
Can someone please tell me how to add this package permanently into build so that GitHub doesn't need to install it
Or how to fix this issue? It doesn't seem to want to take https:// and takes ssh://git#github.com . Not sure if that is causing anything
Also if I use the option to replace the SHA in package-lock.json then the publish fails with another error
Publish Action
name: Release
on:
release:
types:
- created
jobs:
publish_on_mac:
runs-on: macos-latest
steps:
- uses: actions/checkout#v3
with:
persist-credentials: false
- name: HTTPS
run: >
git config --global url."https://github.com".insteadOf ssh://git#github.com
- uses: actions/setup-node#v3
with:
node-version: 14
- name: install dependencies
run: npm install
- name: publish
run: npm run publish
Error
npm ERR! /usr/local/bin/git ls-remote -h -t ssh://git#github.com/pemrouz/buble.git
1226
npm ERR!
1227
npm ERR! Warning: Permanently added the ECDSA host key for IP address '140.xxx.xxx.xxx' to the list of known hosts.
1228
npm ERR! git#github.com: Permission denied (publickey).
1229
npm ERR! fatal: Could not read from remote repository.
1230
npm ERR!
1231
npm ERR! Please make sure you have the correct access rights
1232
npm ERR! and the repository exists.
1233
npm ERR!
1234
npm ERR! exited with error code: 128
How do we fix this as well?

Npm install on GitHub Pull Request fails for the package referenced from a public GitHub repository

In the package.json file, I have added a dependency that is referencing one of our public repositories. The dependency in the package.json looks like below:
"ffprobe-static": "git+https://github.com/company-name/repo-name.git",
I can successfully run npm install locally and use this dependency, but when I push this code, our GitHub workflows where we execute npm install fails with the below error:
npm ERR! Warning: Permanently added the RSA host key for IP address 'x.x.x.x' to the list of known hosts.
npm ERR! git#github.com: Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR!
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
I don't understand the reason for this error, since the repository we are referencing is public, and also I can access the same repository when I install dependencies locally.
Note that the repository that is running this code is a private repository, but the referenced repository is public, but under the same organization.
I was able to fix it by adding the below step after checkout in the YAML file. Also, set the persist-credentials option to false in the checkout step.
steps:
- name: Checkout
uses: actions/checkout#v2
with:
persist-credentials: false
- name: Reconfigure git to use HTTP authentication
run: >
git config --global url."https://github.com/".insteadOf
ssh://git#github.com/
You might try a config to force https URLs, at least for testing, in your GitHub workflow:
- name: Fix URL access
run: echo -e '[url "https://github.com/"]\n insteadOf = "ssh://git#github.com/"' >> ~/.gitconfig
- name: Checkout server
uses: actions/checkout#v2
...
Or (as in here, just to illustrate where you can put the git config insteadOf command):
on: push
jobs:
check-elm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Checkout submodules
shell: bash
run: |
# From https://github.com/actions/checkout/issues/116#issuecomment-583221947
git config --global url."https://github.com/".insteadOf
ssh://git#github.com/
git submodule sync --recursive
git -c "http.extraheader=Authorization: basic ${{secrets.GITHUB_ACCESS_TOKEN}}" -c protocol.version=2 submodule update --init --force --recursive --depth=1
- uses: actions/setup-node#v1
with:
node-version: '8.16.0'
- run: npm run test

How to write a pipeline for a pull request for lint and run tests

I never wrote pipelines in github actions, I was asked to create a pipeline for a pull request for repositories in github using the commands:
npm run lint;
npm run test;
GitHub's own setup-node Action has examples of how to do this. Based on the docs there, something like this placed in a YAML file in the .github/workflows directory should work:
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v2
with:
node-version: '16'
- run: npm install
- run: npm lint
- run: npm test

How to push to protected main branches in a GitHub Action?

This is my github action workflow.
name: Release
on:
push:
branches:
- main
jobs:
release:
name: Release
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
with:
persist-credentials: false
- name: Setup java
uses: actions/setup-java#v1
with:
java-version: 11
- name: Setup node
uses: actions/setup-node#v1
with:
node-version: "14.x"
cache: npm
- name: Install dependencies
run: npm ci
- name: Build package
run: npm run build --if-present
- name: Semantic release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
HUSKY: 0
run: chmod +x script/prepare-release.sh && npx semantic-release
However, my workflow fails with the following error log.
[semantic-release] › ✖ An error occurred while running semantic-release: Error: Command failed with exit code 1: git push --tags https://x-access-token:[secure]#github.com/didrlgus/convention-template.git HEAD:main
remote: error: GH006: Protected branch update failed for refs/heads/main.
remote: error: At least 1 approving review is required by reviewers with write access.
Maybe it's because my main branch is a protected branch.
How can I push with a protected branch on github action?
There is a workaround. Steps as follows:
Create new Github user eg. my-org-bot
Generate Personal Access Token for this user on https://github.com/settings/tokens and save it somewhere (select repo scope for the token)
Go to your repo and add my-org-bot to contributors
Open your branch protection rules and add my-org-bot to the rule below:
Go to repository secrets and add new secret for Actions with key =BOT_ACCESS_TOKEN and the value = Personal Access Token generated previously
Modify your GH Workflow Checkout step with below:
Now your workflow should be able to push directly to your protected branch on behalf of my-org-bot user.
The solution that works for us is as follows:
name: Version and Package Repo
on:
push:
branches: [ master, main ]
jobs:
build:
if: github.event.commits[0].author.name != 'GitHubActions'
runs-on: ubuntu-18.04
steps:
- name: Checkout repo
uses: actions/checkout#v2
with:
fetch-depth: 0
token: ${{ secrets.PAT }}
- name: Configure git
run: |
git config user.name "GitHubActions"
git config user.email "<>"
- name: Install NPM Packages
run: npm install
env:
NODE_AUTH_TOKEN: $\{{ secrets.PAT }}
- name: Version and Package
run: npm version patch --force
env:
NODE_AUTH_TOKEN: $\{{ secrets.PAT }}
- name: Update git
run: |
git push
git push --tags
This runs on all pushes to master and main branches (we use the same script on multiple repos) and it:
checks the repo out
configures git
installs and then versions some NPM packages (not relevant to this issue, aside from the job making some kind of change to the repo) - this creates a new commit
pushes the changes back to the same branch
secrets.PAT is a personal access token of a user with admin rights and the repo has branch protection on, but excludes admins.
It is worth considering that if you run git push from an action with the on push trigger and you're using a PAT rather than GITHUB_TOKEN, then the action will run in a loop. If you are using GITHUB_TOKEN then GitHub Actions prevents the action running again automatically. We use the conditional if line at the top of the job to prevent the job running if the author name of the last commit is GitHubActions. This is the author name set in the Configure git stage, so the commits that happen within this job (as a result of npm version patch) are from an author with this name.
If the author variable doesn't work for you, there are plenty of others you can use:
https://docs.github.com/en/actions/learn-github-actions/contexts#github-context
https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#push
The downside of this approach is that you always get a second run appear in your list of actions which is immediately skipped.
I couldn’t find a solution that was acceptable to me/work. So, the only option I was left with was avoiding updates in CI that need to be pushed up. That means versioning and changelogs have to be done as part of a user commit/PR. And I created some tooling it make sure it’s done right, in case it helps anyone else: https://github.com/Shakeskeyboarde/anglerci

Lerna always lists all packages ready to publish when running workflow of Github actions

Lerna does not correctly detect packages change during running workflow of Github actions.
If I make none packages related changes, commit and runlerna updatedlocally. it tells me No changed packages found which is correct and expected.
If I make package related changes, commit and run lerna updated locally. it tells me found x packages ready to publish which is also correct and expected.
However, if I push the commit based on 1 or 2. the step which I run lerna updated in my github actions workflow always tells/lists me all the package are available to publish which is wrong.
I am wondering why and how to fix it ???
here is what I see locally if I made none packages related changes
lerna notice cli v3.20.2
lerna info versioning independent
lerna info Looking for changed packages since #xxx/bar#2.3.4
lerna info No changed packages found
here is what I see on workflow log after pusing the none packages related changes to Github
> lerna updated -l
lerna notice cli v3.20.2
lerna info versioning independent
lerna info Assuming all packages changed
#xxx/bar v2.3.4 packages/Bar
#xxx/foo v1.4.4 packages/Foo
#xxx/hulk v1.0.4 packages/Hulk
lerna success found 3 packages ready to publish
here is my workflows
name: Publish
on:
push:
branches:
- master
jobs:
unit-test:
name: UnitTest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: 12
- run: npm ci
- run: npm test
publish:
name: Publish NPM Packages
needs: unit-test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: 12
registry-url: https://registry.npmjs.org/
- run: npm ci
- run: git config --global user.email "xxx"
- run: git config --global user.name "xxx"
- run: npm run updated
env:
NODE_AUTH_TOKEN: ${{secrets.NPM_AUTH_TOKEN}}
here is my package.json
{
"name": "root",
"devDependencies": {
"jest": "^25.1.0",
"lerna": "^3.20.2"
},
"scripts": {
"updated": "lerna updated -l",
"test": "jest"
}
}
here is my lerna setting
{
"packages": [
"packages/*"
],
"version": "independent",
"command": {
"publish": {
"allowBranch": "master",
"conventionalCommits": true,
"message": "chore(release): updated release notes and package versions"
}
}
}
After hours of debugging. I found the answer myself and thanks to #peterevans for the tip
You have to combine both
fetch-depth: 0
run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
so that all git history and tag are exposed to lerna.
Wow - cannot believe that I've finally found a fix to the same issue - huge thanks!
I see this as a big issue with github actions (specifically #actions/checkout), and thus I've informed them here: https://github.com/actions/checkout/issues/217
I've also informed the lerna folks here: https://github.com/lerna/lerna/issues/2542
and semantic-release people here: https://github.com/semantic-release/semantic-release/issues/1526
Thanks again! You've helped me save a lot of time & fix an annoying issue, and I hope I'll help others with this too. Cheers
There is also the option include-merged-tags
So this should also solve the problem:
lerna updated --include-merged-tags
or for publishing:
lerna publish --include-merged-tags
Possibly you use git tag xxx without -m parameter. If you execute git tag yourself instead of using lerna version or lerna publish, you should add -m parameter to make the tag annotated.
Ref: https://github.com/lerna/lerna/issues/1357#issuecomment-438162152
Documenting my findings about this issue.
#eded found the correct issue, which was that by default, actions/checkout fetches only 1 commit for performance reasons. Therefore, lerna does not have access to the version tags (which are often several commits back) and assumes that all packages have changed by default :
lerna info Assuming all packages changed
You therefore need to force actions/checkout to get all commits and tags, by using the input fetch-depth: '0'
This is all you need to do, as actions/checkout was corrected to fetch all commits and tags since v2 after this issue surfaced.
This might be a performance issue as your monorepo grows.