Npm install on GitHub Pull Request fails for the package referenced from a public GitHub repository - github

In the package.json file, I have added a dependency that is referencing one of our public repositories. The dependency in the package.json looks like below:
"ffprobe-static": "git+https://github.com/company-name/repo-name.git",
I can successfully run npm install locally and use this dependency, but when I push this code, our GitHub workflows where we execute npm install fails with the below error:
npm ERR! Warning: Permanently added the RSA host key for IP address 'x.x.x.x' to the list of known hosts.
npm ERR! git#github.com: Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR!
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
I don't understand the reason for this error, since the repository we are referencing is public, and also I can access the same repository when I install dependencies locally.
Note that the repository that is running this code is a private repository, but the referenced repository is public, but under the same organization.

I was able to fix it by adding the below step after checkout in the YAML file. Also, set the persist-credentials option to false in the checkout step.
steps:
- name: Checkout
uses: actions/checkout#v2
with:
persist-credentials: false
- name: Reconfigure git to use HTTP authentication
run: >
git config --global url."https://github.com/".insteadOf
ssh://git#github.com/

You might try a config to force https URLs, at least for testing, in your GitHub workflow:
- name: Fix URL access
run: echo -e '[url "https://github.com/"]\n insteadOf = "ssh://git#github.com/"' >> ~/.gitconfig
- name: Checkout server
uses: actions/checkout#v2
...
Or (as in here, just to illustrate where you can put the git config insteadOf command):
on: push
jobs:
check-elm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Checkout submodules
shell: bash
run: |
# From https://github.com/actions/checkout/issues/116#issuecomment-583221947
git config --global url."https://github.com/".insteadOf
ssh://git#github.com/
git submodule sync --recursive
git -c "http.extraheader=Authorization: basic ${{secrets.GITHUB_ACCESS_TOKEN}}" -c protocol.version=2 submodule update --init --force --recursive --depth=1
- uses: actions/setup-node#v1
with:
node-version: '8.16.0'
- run: npm run test

Related

Github Actions failing to install a NPM and fails verification

Github Actions build script
name: Deploy
on:
push:
branches: [ "main" ]
jobs:
build_on_mac:
runs-on: macos-latest
steps:
- uses: actions/checkout#v3
with:
persist-credentials: false
- name: Use HTTP
run: >
git config --global url."https://github.com".insteadOf ssh://git#github.com
- uses: actions/setup-node#v3
with:
node-version: 14
- name: install dependencies
run: npm install
It keeps falling for a particular package randomly.. worked 1 out of 10 times
npm WARN tarball tarball data for buble#git+ssh://git#github.com/pemrouz/buble.git#4e639aeeb64712ac95dc30a52750d1ee4432c9c8 (sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA==) seems to be corrupted. Trying one more time.
7
npm ERR! Verification failed while extracting buble#git+ssh://git#github.com/pemrouz/buble.git#4e639aeeb64712ac95dc30a52750d1ee4432c9c8:
2895
npm ERR! sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA== integrity checksum failed when using sha512: wanted sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA== but got sha512-J+yRnScDV19Vr5+C8D5IJiIN2auC9t54tYpJmeqVxgyIyJQmF95mqBwBXzXKyvIH9aZvY6RlQqtMsQm0gdH7UQ==. (801561 bytes)
2635
So the problem is that when package-lock.json is generated locally it has
sha512-vBjSU8v5gODqXVUvlpgEzjtSMXz2Am8Yh7O1PUZmx6DAzKWT1CrYm5azoYhAGLCWOlkd4aN6LEOCZ+dgVt2rFA==
But when getting downloaded by actions it is
sha512-J+yRnScDV19Vr5+C8D5IJiIN2auC9t54tYpJmeqVxgyIyJQmF95mqBwBXzXKyvIH9aZvY6RlQqtMsQm0gdH7UQ==
Workaround is to replace the SHA in GitHub with expected one which allows build to complete but obviously that looks wrong..
So options I can think are
Can someone please tell me how to add this package permanently into build so that GitHub doesn't need to install it
Or how to fix this issue? It doesn't seem to want to take https:// and takes ssh://git#github.com . Not sure if that is causing anything
Also if I use the option to replace the SHA in package-lock.json then the publish fails with another error
Publish Action
name: Release
on:
release:
types:
- created
jobs:
publish_on_mac:
runs-on: macos-latest
steps:
- uses: actions/checkout#v3
with:
persist-credentials: false
- name: HTTPS
run: >
git config --global url."https://github.com".insteadOf ssh://git#github.com
- uses: actions/setup-node#v3
with:
node-version: 14
- name: install dependencies
run: npm install
- name: publish
run: npm run publish
Error
npm ERR! /usr/local/bin/git ls-remote -h -t ssh://git#github.com/pemrouz/buble.git
1226
npm ERR!
1227
npm ERR! Warning: Permanently added the ECDSA host key for IP address '140.xxx.xxx.xxx' to the list of known hosts.
1228
npm ERR! git#github.com: Permission denied (publickey).
1229
npm ERR! fatal: Could not read from remote repository.
1230
npm ERR!
1231
npm ERR! Please make sure you have the correct access rights
1232
npm ERR! and the repository exists.
1233
npm ERR!
1234
npm ERR! exited with error code: 128
How do we fix this as well?

Github workflow with private repo & tag

I'll start with I asked this question here and got no response: https://github.community/t/private-repo-w-tag-in-workflow/229573
We have three private repos with tags in our package.json as dependencies, one example:
"Private-Repo1": "https://<PAT>:x-oauth-basic#github.com/project/Private-Repo.git#v1.0.0",
We use oauth keys to access our repos. My PAT is set to allow checking out the repo as well as workflow access.
When we run our Workflow action, it fails at npm ci for this line with an error of:
npm ERR! code 128
npm ERR! An unknown git error occurred
npm ERR! command git --no-replace-objects ls-remote ***github.com/project/Private-Repo.git
npm ERR! remote: Repository not found.
npm ERR! fatal: repository 'https://github.com/project/Private-Repo.git/' not found
Local testing is pointing to the reason that we’re failing is that git ls-remote fails when you point to a private repo with a tag number, if I remove the tag it works.
Can someone please point me to how we can use a PAT to pull a specific tag from a private repo in our workflow via our package.json? Everything I can find is how to access a private repo, but not how to access a private repo's tag.
For anyone that stumbles on this with a similar issue, the problem wasn't git ls-remote it was the token. I was calling it in the wrong place. It needs to be set in the checkout step, not setup-node step. Here is my working yaml that allows me to run a workflow with a private repo and tag that uses an oauth token. The only setup needed is to make a secret called GIT_TOKEN (or whatever you want to call it) and give it workflow access.
name: API auto test and lint workflow
on: push
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
with:
token: ${{ secrets.GIT_TOKEN }}
- uses: actions/setup-node#v1
with:
node-version: 16.x
- run: npm ci
- run: npm run lint
- run: npm run test

Is there a way to use secrets in flutter web deployments on Netlify with or without using GitHub workflows?

I have a Flutter web app and I'm not much familiar with GitHub workflows. I have a dotenv file that stores a token needed by another file in the project.
For the deployments, I'm using this build command in Netlify:
if cd flutter; then git pull && cd ..; else git clone https://github.com/flutter/flutter.git; fi && flutter/bin/flutter config --enable-web && flutter/bin/flutter build web --release
One more reason why I chose to use this instead of a GitHub workflow is because this doesn't add the build directory in my repo itself.
Now a need has arised to use this dotenv in my project. But to build and deploy this using the aforementioned command, the dotenv should always be version controlled. Otherwise Netlify won't be able to detect it.
I have come across this stackoverflow post but this doesn't seem to solve my problem.
Is there any way I can directly pass the environment secrets to Netlify needed for build and deploy for Flutter? Or is there any workflow to directly deploy (on push) to Netlify, without storing the build files in my repo?
This is my current netlify build settings:
You can simply put your build files onto another branch using GitHub workflows.
First create a empty branch named build and initiate the branch using an empty commit. Follow these steps:
git checkout --orphan build
git rm -rf . This removes existing files
git commit --allow-empty -m "some message"
Now come back to the master branch. And run these steps:
base64 path/to/dontenv and copy the output of this command. This actually decodes the contents of your DOTENV into a string.
Paste this output in a new GitHub repo project secret and name it DOTENV.
Now simply add this DOTENV in .gitignore.
Create a new GitHub workflow.
Run mkdir -p .github/workflows.
nano .github/workflows/build.yml and paste this:
name: Build and Deploy
on:
push:
branches:
- master
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout#v2
#I am assuming your dontenv was in lib/
#This now decodes your dotenv back and writes the contents into lib/dotenv
- run: echo "${DOTENV// /}" | base64 -d > lib/dotenv
env:
MAIN: ${{ secrets.DOTENV }}
- name: Set up Flutter
uses: subosito/flutter-action#v1
with:
channel: 'stable'
- name: Get dependencies
run: flutter pub get
- name: Run analyze
run: flutter analyze .
- name: Build release
run: flutter build web --release
- name: Upload artifacts
uses: actions/upload-artifact#v1
with:
name: build
path: build
deploy-build:
needs: build
runs-on: ubuntu-latest
steps:
- name: Clone the repoitory
uses: actions/checkout#v2
with:
ref: build
- name: Configure git
run: |
git config --global user.email ${GITHUB_ACTOR}#gmail.com
git config --global user.name ${GITHUB_ACTOR}
- name: Download website build
uses: actions/download-artifact#v1
with:
name: build
path: build
- name: Commit and Push
run: |
if [ $(git status --porcelain=v1 2>/dev/null | wc -l) != "0" ] ; then
git add -f build
git commit -m "gh-actions deployed a new build"
git push --force https://github.com/${GITHUB_REPOSITORY}.git HEAD:build
fi
This will create the build files in the build branch. And your master branch will remain unaffected. After every push, this GitHub action will get triggered and build and commit your build files in the build branch. To deploy, simply deploy this build branch.

Lerna always lists all packages ready to publish when running workflow of Github actions

Lerna does not correctly detect packages change during running workflow of Github actions.
If I make none packages related changes, commit and runlerna updatedlocally. it tells me No changed packages found which is correct and expected.
If I make package related changes, commit and run lerna updated locally. it tells me found x packages ready to publish which is also correct and expected.
However, if I push the commit based on 1 or 2. the step which I run lerna updated in my github actions workflow always tells/lists me all the package are available to publish which is wrong.
I am wondering why and how to fix it ???
here is what I see locally if I made none packages related changes
lerna notice cli v3.20.2
lerna info versioning independent
lerna info Looking for changed packages since #xxx/bar#2.3.4
lerna info No changed packages found
here is what I see on workflow log after pusing the none packages related changes to Github
> lerna updated -l
lerna notice cli v3.20.2
lerna info versioning independent
lerna info Assuming all packages changed
#xxx/bar v2.3.4 packages/Bar
#xxx/foo v1.4.4 packages/Foo
#xxx/hulk v1.0.4 packages/Hulk
lerna success found 3 packages ready to publish
here is my workflows
name: Publish
on:
push:
branches:
- master
jobs:
unit-test:
name: UnitTest
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: 12
- run: npm ci
- run: npm test
publish:
name: Publish NPM Packages
needs: unit-test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- uses: actions/setup-node#v1
with:
node-version: 12
registry-url: https://registry.npmjs.org/
- run: npm ci
- run: git config --global user.email "xxx"
- run: git config --global user.name "xxx"
- run: npm run updated
env:
NODE_AUTH_TOKEN: ${{secrets.NPM_AUTH_TOKEN}}
here is my package.json
{
"name": "root",
"devDependencies": {
"jest": "^25.1.0",
"lerna": "^3.20.2"
},
"scripts": {
"updated": "lerna updated -l",
"test": "jest"
}
}
here is my lerna setting
{
"packages": [
"packages/*"
],
"version": "independent",
"command": {
"publish": {
"allowBranch": "master",
"conventionalCommits": true,
"message": "chore(release): updated release notes and package versions"
}
}
}
After hours of debugging. I found the answer myself and thanks to #peterevans for the tip
You have to combine both
fetch-depth: 0
run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
so that all git history and tag are exposed to lerna.
Wow - cannot believe that I've finally found a fix to the same issue - huge thanks!
I see this as a big issue with github actions (specifically #actions/checkout), and thus I've informed them here: https://github.com/actions/checkout/issues/217
I've also informed the lerna folks here: https://github.com/lerna/lerna/issues/2542
and semantic-release people here: https://github.com/semantic-release/semantic-release/issues/1526
Thanks again! You've helped me save a lot of time & fix an annoying issue, and I hope I'll help others with this too. Cheers
There is also the option include-merged-tags
So this should also solve the problem:
lerna updated --include-merged-tags
or for publishing:
lerna publish --include-merged-tags
Possibly you use git tag xxx without -m parameter. If you execute git tag yourself instead of using lerna version or lerna publish, you should add -m parameter to make the tag annotated.
Ref: https://github.com/lerna/lerna/issues/1357#issuecomment-438162152
Documenting my findings about this issue.
#eded found the correct issue, which was that by default, actions/checkout fetches only 1 commit for performance reasons. Therefore, lerna does not have access to the version tags (which are often several commits back) and assumes that all packages have changed by default :
lerna info Assuming all packages changed
You therefore need to force actions/checkout to get all commits and tags, by using the input fetch-depth: '0'
This is all you need to do, as actions/checkout was corrected to fetch all commits and tags since v2 after this issue surfaced.
This might be a performance issue as your monorepo grows.

Travis CI - How to push into master branch?

I have a Travis CI project connected to GitHub that tries to update content in the Github repo and push them back to GitHub, both master and gh-pages branches.
However, although my travis-ci log files says everything is ok, I only see the gh-pages branch updated, but not the master branch.
My travis.yml file is:
language: node_js
node_js: stable
language: python
python: 3.6
# Travis-CI Caching
cache:
directories:
- node_modules
- pip
# S: Build Lifecycle
install:
- npm install
- npm install -g gulp
- python -m pip install requests
- python -m pip install bs4
- python -m pip install lxml
before_script:
- cd archive_builder
- python build_archive.py
- cd ..
script:
- gulp dist
after_script:
- cd dist
- git init
- git config user.name "my git name"
- git config user.email "my git email"
- git add -A
- git commit -m "travis -- update gh-page"
- git push --force --quiet "https://${GH_TOKEN}#${GH_REF}" master:gh-pages
- sh ../purgeCF.sh $CF_ZONE $CF_KEY $CF_EMAIL
- cd ..
- git add -A
- git commit -m "travis -- update master files"
- git push --quiet "https://${GH_TOKEN}#${GH_REF}" HEAD:master
# E: Build LifeCycle
branches:
only:
- master
env:
global:
- GH_REF: github.com/mygitname/myprojectname.git
In this script, I first update and build website sourcefiles with gulp, storing them into "dist" folder. Then I push content in "dist" to my gh-pages branch, and push everything else to my master branch.
The credentials are stored as security keys with Travis and should work correctly.
To push "dist/", I created a new ".git/" under "dist/" and force push it as new.
To push everything else, I could not do it because the root repository already contains ".git" folder and I do not want to lose my previous commits. It should work.
Thanks for help.
I found most articles or answers were talking about how to deploy to gh-pages branch, and most ways is not work for me , i debug this issue on travis for several days, i will list key steps about how to push to master brach on travis
e.g. Below is my doc repository script, travis will update readme.md automated.
Generate github token, you can refer to the article https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
Set Environment Variables
GH_REF githu.com/clownvary/docs.git your repository address
GITHUB_API_KEY *********** your token generated on step 1
Script
os: osx
language: node_js
cache:
directories:
- node_modules
node_js:
- 'lts/*'
before_install:
- git pull
- brew install tree
install:
- npm install
script:
- npm run updateReadme
after_success:
- git config user.email "travis#travis.org"
- git config user.name "travis" # this email and name can be set anything you like
- git add README.md
- git commit --allow-empty -m "updated README.md"
- git push https://clownvary:${GITHUB_API_KEY}#${GH_REF} HEAD:master #clownvary is my username on github, you need to use yourself , do not use travis or others.
Hope this can help you
Even though #gary wang method is working, there is much simpler method that can push into GitHub master branch directly.
Just add target_branch variable under deploy section, and assign it with master.
Documentation on Travis CI GitHub Pages Deployment: https://docs.travis-ci.com/user/deployment/pages/
Sample contents of .travis.yml:
language: node_js
...
...
...
deploy:
provider: pages
skip_cleanup: true
keep_history: true
github_token: $github_token # Your GitHub token set in Travis CI console
target_branch: master # Add this line - To push into GitHub master branch
on:
branch: staging # Your GitHub repo default branch
This method is tested and working as per expected.