Publishing Github packages for a monorepo as part of an organization - github

I have a Lerna monorepo on Github Enterprise which currently has two npm packages that I'm trying to publish to the Github package registry under the same repo.
For reference say they are:
github.com/mycompany/package-a
github.com/mycompany/package-b
I followed these instructions: https://help.github.com/en/github/managing-packages-with-github-packages/configuring-npm-for-use-with-github-packages#publishing-multiple-packages-to-the-same-repository
So now my 2 package.json files look like the following (trimmed for formatting purposes):
"name": "#mycompany/package-a",
"repository": {
"url": "ssh://git#github.com:mycompany/monorepo.git"
},
"publishConfig": {
"registry": "https://npm.pkg.github.com/"
},
"name": "#mycompany/package-b",
"repository": {
"url": "ssh://git#github.com:mycompany/monorepo.git"
},
"publishConfig": {
"registry": "https://npm.pkg.github.com/"
},
So you can notice they both have the same URL for the repository as recommended.
First problem: One is that my company already has repos called package-a and package-b. It seems that you can't have a naming collision with a package in a monorepo and a package outside the monorepo.¹
Second and more important problem: This doesn't seem to work for me at all. I renamed the package in their respective package.json files to avoid the naming collision to package-a-mono and package-b-mono which I don't really want to do but I'm just trying to get it to work. I get a 404 when trying to run either lerna publish or npm publish inside of the repos themselves. Like it's not actually trying to read that repository.url field in that it tells you to modify.
^ This turned out to be temporary or was never actually an issue, it was just because of the naming conflicts.
npm publish output:
npm ERR! code E404
npm ERR! 404 Not Found - PUT https://npm.pkg.github.com/#mycompany%2fpackage-a - The expected resource was not found.
lerna publish output:
lerna http fetch PUT 404 https://npm.pkg.github.com/mycompany/#mycompany%2fpackage-a 327ms
lerna ERR! E404 The expected resource was not found.
Has anyone run into this and found a solution?
¹On a somewhat worse note, for some reason the very first time I ran this it actually did publish a package into the monorepo for package-a. But from then on I get the error lerna ERR! E422 Package "package-a" is already associated with another repository. Nothing changed and I couldn't publish another version to the same repo.

Another possible cause of this error (discussed and ruled out in the original question body) is if any package's package.json's repository field does not match the git URL of the repo - for example, if you transferred the repo to a different organisation, or renamed the repo, but didn't update every package's package.json with the new URL.
The error message will report 404 on https://npm.pkg.github.com/#org/package-name even if the problem is with the repository URL.
(credit to jonas-reif's comment)

The issue here was that we had repos in the organization with the same name as the package being published and it didn't like that.
When I looped back around to solving this I renamed the packages to not have a collision with another existing repo in the organization and it worked as expected.

I ran into the same issue, I had to generate a new personal access token that had more privileges. Just read:packages and write:packages was not sufficient, you also needed repo.

I had the same problem and I had to add the following line to my NPM package's URL:
"repository": "https://github.com/ACCOUNT/REPOSITORY"
This repository was the root of my mono repository.
More on this here:
https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-npm-registry#publishing-multiple-packages-to-the-same-repository

Related

Mirroring pub.dev packages to a custom package repository

As of Dart 2.15, you can create your own package repository.
https://dart.dev/tools/pub/custom-package-repositories#publishing-to-a-custom-package-repository
We need to mirror the subset of pub.dev packages (for corporative security reasons).
But there's no information, how to do a package mirroring.
Should I fork a GitHub source and publish every allowed package versions manually? It looks exremely annoying and time-consuming. And what about transitive dependencies? Should I upload all of them too?
I also found this article: https://medium.com/dartlang/hosting-a-private-dart-package-repository-774c3c51dff9 , it describes the pub.dev mirroring scenario too, but doesn't explains how to do it.
I don't know if this is the answer you are looking for, but I believe that with some experimentation this could be scripted using the APIs described in the dart package repository specification:
https://github.com/dart-lang/pub/blob/master/doc/repository-spec-v2.md
For example, you can query all versions of a package like so:
curl https://pub.dev/api/packages/test
{
"version": "1.23.1",
"pubspec": {
"name": "test",
"version": "1.23.1",
...
"dependencies": {
"analyzer": ">=2.0.0 <6.0.0",
...
"test_core": "0.4.24"
},
"dev_dependencies": {
...
}
},
"archive_url": "https://pub.dartlang.org/packages/test/versions/1.23.1.tar.gz",
...
}
....
}
You can then use the archive_url for the version(s) you want to download the package, potentially iterating over the dependencies and downloading them as well.
You would then need to upload each package version you downloaded to your private repo. This can be done by first doing a GET on the new package submission URL:
curl https://my-private-repo.tld/api/packages/versions/new
{
"url": "https://my-private-repo.tld/api/packages/versions/newUpload",
"fields": {}
}
And then POST the form described by fields plus the archive you previously downloaded to the url provided in the response. Note that when I test this against unpub, the fields map is empty, but depending on your private repo implementation that may be different.
One could imagine a script that starts with a list of packages to mirror, downloads them, potentially also downloads their dependencies, and then uploads them all to the private server.
For transitive dependencies it probably depends on your security requirements. If you use PUB_HOSTED_URL=https://my-private-repo.tld then running flutter pub get will download any dependencies it can find from your private repo, and any other dependencies from pub.dev. If that is not acceptable, then you'll probably need to upload them all.

Cannot install npm package in Github registry

I have published a package to the GitHub npm registry. I can see the package when I login onto GitHub, as per the screenshots below.
ScreenShot 1
ScreenShot 2
Locally, a different package has the former as a dependency. Therefore I have on my package.json:
"dependencies": {
"<#scope>/<package_name>": "1.2.0",
...
I also have, in-line with the documentation, a .npmrc file at the same directory as the package.json with the authToken. This authToken has all the permissions necessary (delete:packages, read:packages, repo, write:packages)
//npm.pkg.github.com/:_authToken=<my_auth_token>
registry=https://npm.pkg.github.com/OWNER
Lastly, I have on package.json the following two entries. Although I believe these are necessary to publish packages to the github registry, no to install them which is the current problem.
"publishConfig": {
"registry": "https://npm.pkg.github.com/"
},
"repository": {
"type": "git",
"url": "https://github.com/<OWNER>/<REPO>.git"
}
So, the error that I am having when I try to install the package is:
npm ERR! code E404
npm ERR! 404 Not Found: <#scope>/<package_name>#1.2.0
When I tried to understand what was going on with some helpful logs:
With npm install #<scope>/<package_name>#1.2.0 --loglevels verbose, I see:
....
npm http fetch GET 200 https://npm.pkg.github.com/OWNER/#<scope>%2f<package_name>
npm http fetch GET 404 https://npm.pkg.github.com/download/#<scope>/<package_name>/1.2.0/98e69e2adddec5c715a32c94352efac83ce586e4e6f3d06fef1fc1f82d238r34
...
The first http request is a success, which seems to indicate things are correctly wired up and authenticated. However, I have no idea why I am getting a 404 afterwards. The package is clearly there as we see on the image above.
Does anyone had a similar problem ?
Well, my own .npmrc (in my home directory) is configured as this:
//registry.npmjs.org/:_authToken=<token>
//npm.pkg.github.com/:_authToken=<token>
#myorg1:registry=https://npm.pkg.github.com
#myorg2:registry=https://registry.npmjs.org
I've not added my org in the github registry URL as you did, and specified the registry to use for each scope.
I just ran into the same issue. Removing package-lock.json fixed it for me. I suspect the problem was that I previously installed the offending package from a tarball and that the old installation information was somehow cached inside the package-lock.json
I had misread the checkboxes in my PAT. Make sure you've enabled read:packages.
My successful .npmrc file looks like this:
//npm.pkg.github.com/:_authToken=TOKEN_GOES_HERE

Yarn can't find private Github npm registry

I signed up for the Github private npm registry beta and followed their instruction: https://github.com/features/package-registry
Works great with npm but I'd prefer using yarn. And while npm has no issues finding the registered package, yarn can't find it at all.
yarn add #omniphx/adminite-adminite-ui-components outputs:
yarn add v1.19.0
info No lockfile found.
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
[1/4] 🔍 Resolving packages...
error Couldn't find package "#omniphx/adminite-ui-components" on the "npm" registry.
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.
After reading up on private repos with yarn, I thought the trick was due to yarn having a slightly different rc format. Unfortunately, that didn't work either and yarn is still unable to find the private registry.
.npmrc
registry=https://registry.npmjs.org
#omniphx:registry=https://npm.pkg.github.com/omniphx
.yarnrc
registry "https://registry.npmjs.org"
"#omniphx:registry" "https://npm.pkg.github.com/omniphx"
Also confirmed that my github token is set too with yarn config list:
yarn config v1.19.0
info yarn config
{
'version-tag-prefix': 'v',
'version-git-tag': true,
'version-commit-hooks': true,
'version-git-sign': false,
'version-git-message': 'v%s',
'init-version': '1.0.0',
'init-license': 'MIT',
'save-prefix': '^',
'bin-links': true,
'ignore-scripts': false,
'ignore-optional': false,
registry: 'https://registry.npmjs.org',
'strict-ssl': true,
'user-agent': 'yarn/1.19.0 npm/? node/v12.11.1 darwin x64',
email: 'mattjmitchener#gmail.com',
lastUpdateCheck: 1570679687836,
username: 'omniphx',
'#omniphx:registry': 'https://npm.pkg.github.com/omniphx'
}
info npm config
{
'//npm.pkg.github.com/:_authToken': 'fake12345',
registry: 'https://registry.npmjs.org',
'#omniphx:registry': 'https://npm.pkg.github.com/omniphx',
python: '/usr/bin/python'
}
Any idea?
Resolved
Changed "#myorg:registry" "https://npm.pkg.github.com/myorg"
To "#myorg:registry" "https://npm.pkg.github.com"
I've just run into a similar situation. It seemed that yarn was only looking in the main Yarn package registry for my organization's private package. I had copied the examples from GitHub's Packages documentation for constructing your .npmrc file directly to the .yarnrc file in the project that will be consuming the app, not knowing that the formats were different (I've never had to deal with .yarnrc files before).
However, after updating the .yarnrc file with the correct format that you've mentioned above (which I also found in googling around), yarn successfully found the private package and installed it correctly.
As a heads up, my yarn version: 1.17.3
Steps I Took
Start new terminal session
cd to the project
nvm use (if you have a specific node version to use)
Add the correctly-formatted .yarnrc file to the project. See below for what it looks like.
Manually add the package and version range to the package.json for my private package
Run npm login --registry=https://npm.pkg.github.com --scope=#MyOrg
See the note below on scope / org gotcha's
Run yarn
That worked for me.
.yarnrc
"#myorg:registry" "https://npm.pkg.github.com"
Note: See below for a note on the org / scope name gotcha's
Other Notes
I know that it appears that you don't have any issues with this, given your GH username / scope above, but for anyone else that comes here, the documentation on GH is a little sparse with regards to mapping your username / org name to a scope in the package name. Just remember these little gotcha's here:
The name of your package must always be scoped to your org (or username)
E.g., name: #johndturn/my-package
If your organization has capital letters in it, like MyOrg, just replace them in the name of the package in your package.json and your .yarnrc with lowercase
E.g., name: #myorg/my-package
Note: When authenticating with npm login, I still have kept the uppercase letters in the --scope= argument.
The name of your package doesn't have to be the same name of the repo.
E.g., for a repo called MyOrg/random-prefix.js-lib, you can have name: #myorg/js-lib in your package.json file for the project itself. Then, installing it in other projects will look something like #myorg/js-lib: 1.0.0.
The problem I had is slightly different.
After tried what John suggested I still can't add private registry packages with yarn (but perfectly fine with npm)
Then I realise two things:
For GitHub packages, npm is fine with either
registry=https://npm.pkg.github.com/my-org
or
#my-org:registry=https://npm.pkg.github.com
but yarn only allow the latter syntax.
Docs from Github website only show the first syntax which could cause problems for yarn users.
Another thing is that if you npm login to the private registry but use a .yarnrc file in your project, yarn can't really mix your npm credentials with it. Although it seems behave differently on different environment.
But it would seems to be a best practice to stick with either yarn login + .yarnrc, or npm login + .npmrc (you can still use yarn to manage your packages in both cases)
In Yarn v2+ the setup has changed quite a bit. ".yarnrc" is ignored and only ".yarnrc.yml" is used.
To setup a private registry with a scope and token from env, add something along these lines to the ".yarnrc.yml" file (fontawesome example):
npmScopes:
fortawesome:
npmRegistryServer: "https://npm.fontawesome.com"
npmAuthToken: ${FONTAWESOME_TOKEN}
Documentation: https://yarnpkg.com/configuration/yarnrc#npmScopes
I'm not an expert with npm/yarn so I might be misunderstanding what is happening here, but I don't think package proxying from the npm registry works with yarn yet. Could that be related? When package proxying was released for npm I remember reading comments on Twitter from people that tried it with yarn and it didn't work.
Found the Twitter thread here:
https://twitter.com/github/status/1171832034580451328
It doesn't work with Yarn. As soon as I change the registry url -> Couldn't find package.

Git hook update package json version

In our project we often forget to update version numbers in Package.json file. Ours is a AngularJS project. In our package JSON file we are specifying the below two version information
"version": "1.0.7",
"devVersion": "1.0.4"
Before Merging a branch to develop I want a automated script to update these above two version numbers. I am thinking Git Hooks will help me.
Where can i find the hooks, I am able to see the hooks in my local repo under .git folder. I am confused which hook to use. Searching on Google suggests I have to create hooks on server.
Where can i find them and can i update the above both keys (version and devVersion) ?
Pls suggest the location and hook to use, this will solve a lot of problem.
I am using husky and git-branch-is:
"scripts": {
...
"postmerge": "(git-branch-is master && npm version minor ||
(git-branch-is dev && npm --no-git-tag-version version patch)",
...
},
Read more about npm version
Webpack or Vue.js
If you are using webpack or Vue.js, you can display this in the UI using Auto inject version - Webpack plugin
NUXT
In nuxt.config.js:
var WebpackAutoInject = require('webpack-auto-inject-version');
module.exports = {
build: {
plugins: [
new WebpackAutoInject({
// options
// example:
components: {
InjectAsComment: false
},
}),
]
},
}
Inside your template for example in the footer:
<p> All rights reserved © 2018 [v[AIV]{version}[/AIV]]</p>
You have two kinds of hooks (both present in any .git/hooks folder): server and client hooks.
They are listed in "Customizing Git - Git Hooks"
A merge is a local operation, so if you wanted to automate any process during a merge, you would need a client hook, like a post-commit hook (meaning executed just after creating a merge commit).
If you need to update that file before a merge, you can try a pre-commit hook, and check if a merge is in progress (if not, your pre-commit hook would do nothing since you want to update the versions only before a merge).
You can see in this answer an example of a post-commit hook which generates a version.json file.
If is written in node, but you can write a hook ni any scripting language you want.
With Husky, it's extremely simple:
{
"name": "demo-project",
"version": "0.0.3",
"husky": {
"hooks": {
"pre-commit": "npm --no-git-tag-version version patch && git add ."
}
}
}
Note: I put git add . in the end because after we update package version, we need to stage it

Composer cannot find package for a Symfony2 bundle

I'm creating a Symfony2 bundle hosted on GitHub. I suppose that everything is configured well! However, when I try to install it something goes wrong.
Here follows a summary of the key info.
First, in my Symfony2 app I update the composer.json as follows:
"require": {
...,
"bundle-name": "dev-master"
},
When I try to install the bundle through the command php composer.phar update, I get the following error:
Loading composer repositories with package information
Installing dependencies
Your requirements could not be resolved to an installable set of packages.
Problem 1
- The requested package bundle-name could
not be found in any version, there may be a typo in the package name.
Potential causes:
- A typo in the package name
- The package is not available in a stable-enough version according
to your minimum-stability setting
see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion>
for more details.
Any idea?
You have to add repository for "bundle-name".
I couldn't find it on packagist, so register it first.