I have two questions about mirroring Nuget packages using the nuget mirror command (reference).
We are migrating our internal Nuget repository to Artifactory and for the time being I need to have all the packages (and all their versions) from the original nuget repo mirrored to Artifactory.
First, does nuget mirror take into account packages that are already present at the destination? Because so far no matter what I do, the package is always re-uploaded with every call, even though it is already there. Setting up a job responsible for mirroring all the packages would be pretty costly if it always re-uploaded all of them.
Second, is it possible to tell the command to mirror all packages? The documentation states I have to either specify a package ID or a file containing the IDs, but I want to mirror them all.
Related, is it possible to tell the command to mirror all versions of a package? Because if a version is not speficied, only the latest one gets mirrored.
Thanks
As per comments - nuget mirror can only mirror a single package/version. In order to mirror all packages, you need to make a script that scrapes the package info from the Nuget server and then mirror them one-by-one.
Related
I have a GitHub worflow which builds and deploys a snapshot version of a library as a GitHub package, e.g., mycompany.mytool.1.0.0-SNAPSHOT.jar. Whenever I make a new build and deploy, a new asset is created, like, e.g., mycompany.mytool.1.0.0-20210723.145233-1.jar instead which is then somehow associated with the SNAPSHOT tag. This all seems to work and I can access mycompany.mytool.1.0.0-SNAPSHOT.jar without problem.
My question now is, how can I get rid of all these older versions of this jar? Actually I just want to keep the latest version. I can delete them manually via the web-interface but that is a more than awkward task. I would somehow like to automate this too.
This is not possible as of this writing. GitHub staff member Jamie Cansdale wrote this in their community forum:
SNAPSHOT versions are exposed as artifacts inside a regular versions. There isn’t an API for cleaning up artifacts, only whole versions.
(source)
Which means that a single SNAPSHOT version (like 1.0.0-SNAPSHOT) will accumulate all builds you make, and all artifacts will show up on the Assets list to the right of the web page.
The only practical solution I can think of, is that you delete the whole version from a script, before publishing each build's artifacts. Then you'd have the effect of having a single set of artifacts stored as part of the 1.0.0-SNAPSHOT version name.
However this solution is not ideal: public package versions cannot be deleted if they are popular enough (probably to avoid squatting attacks):
If the package is public and the package version has more than 5,000 downloads, you cannot delete the package version. In this scenario, contact GitHub support for further assistance.
This is the method I use to download NuGet package using NuGet.Client API.
public DownloadResourceResult DownloadPackage(string packageId, NuGetVersion version)
{
var packageSource = new PackageSource(_Task.PackageSource);
var sourceRepository = new SourceRepository(packageSource, Repository.Provider.GetCoreV3());
DownloadResource downloadResource = _DownloadResource ?? (_DownloadResource = sourceRepository.GetResourceAsync<DownloadResource>().Result);
var packageIdentity = new PackageIdentity(packageId, version);
var packageDownloadContext = new PackageDownloadContext(NullSourceCacheContext.Instance);
string globalPackagesFolder = SettingsUtility.GetGlobalPackagesFolder(_Settings);
return downloadResource.GetDownloadResourceResultAsync(packageIdentity, packageDownloadContext, globalPackagesFolder, _Logger, CancellationToken.None).Result;
}
Overall it does what it's supposed to do. There is one thing I don't understand. Is there a way to tell GetDownloadResourceResultAsync method to ignore current cache and download and unpack the package again.
For example, I have a NuGet package called MyPackage in the NuGet repository. After I call this method to download the package. MyPackage is downloaded and unpacked into C:\Users[CurrentUser].nuget\packages. If I update MyPackage in the repository and then call DownloadPackage again it won't update the cache and the whole system will think the package was not updated. I can manually delete the package from C:\Users[CurrentUser].nuget\packages before downloading but it seems to invasive. I guess there should be a way for better cache control.
I understand normally it might not be necessary because every time you need to change something in the package you change the package version but it is necessary for automatic packaging and deploying in case you need to re-deploy and re-test something without updating the version.
I believe this is an example of an XY problem. NuGet is designed for packages to be immutable, so you're working against the design if you recreate the same package version with different content, even for testing. You only made a passing comment about automatic packaging and deploying, but didn't explain why you're writing custom code to download packages rather than using nuget.exe, so it's hard for me to make a good suggestion.
If you have a CI/CD environment where packages are created automatically, and other tests automatically use them, then I recommend the packages are built with prerelease version numbers and published to a dev nuget feed. For example, here you can see where the ASP.NET team's CI server publishes dev builds of a package, multiple per day. When they're ready to publish the production ready version, a different CI build packs without a prerelease version and pushes to nuget.org instead of their dev feed.
Depending on what you're trying to do, changing the global packages folder might work for you. Assuming you're using git for source control, if you make your test's global packages folder in an ignored part of your git repo, then git clean -Xdf will delete it, and I believe it's common for CI servers to clean before a build. This is also something I've done for local, manual testing where I didn't want to pollute my global cache. I just delete the folder once I'm done.
Ultimately the package cache is just a folder, which you can delete using System.IO APIs, no need for using NuGet APIs. Although, if you want to account for the fact that nuget.config files can change the global packages folder, you might want to use the NuGet APIs to find it. If you don't mind having all packages deleted, you could just run nuget.exe locals global-cache -clear and avoid writing any code.
I have a situation where we have multiple C# projects that use similar set of Nuget packages (ex. Newton Json, Microsoft Compilers and CodeDom, Owin, log4net, jQuery, EntityFramework etc.)
I am trying to see how we can have a shared location for all Nuget packages to reduce the footprint of those binaries in Git, having a single repo for them by centralizing them in one place.
One option I found is to use Nuget.config in each project with repositoryPath set to point at the shared location. This works great for adding/upgrading/restoring Nuget packages in the project but it is not very clean when a package gets removed from one project but is still required in a different one. Basically the package will get removed from the shared location and the change is committed to Git, then when the other project requires it, it would get restored and added back to Git. Not a perfect solution in my mind.
I have a two part question:
1. Is there a way to improve the above workflow when packages get removed?
2. What is the industry standard for handling third party libraries delivered via Nuget? Or if there is none, can you share your experience handling Nuget packages across multiple projects.
If the concern lies with the footprint/organization of the Git repository, maybe you can do a .git ignore for the dependencies folders to prevent git from committing them into the repositories. When you need to build the projects from source, just do a dotnet /nuget restore to get the dependencies from the source you configured in the Nuget.config
Not sure if it is the industry standard, but we host our own Nuget server to control the libraries that the different teams can use. Microsoft has an article on Hosting your own NuGet feeds
Hope it helps.
I'm actually thinking about the pro and cons about using NuGet. In our current software we're storing each external reference in a common reference folder (which is commited to our SW versioning system). Over time this approach becomes more and more painful because we've to store different versions to the same library.
Since our devs are sometimes at the customer site (where not all customers are offering internet connectivity ...) we won't use NuGet directly, because NuGet packages can't be restored.
Based on that I'm actually thinking about using NuGet and store the packages folder in our SW versioning system.
Does anybody know if there are some disadvantages about this solution? Does anybody have a better proposal?
Thx.
I would argue against storing external nuget packages in your version control system.
It's not your application's responsibility to archive third party packages. Should you need to take care of that risk then build a solution intended for such (for example: use private nuget repository that's properly backed up).
Avoid duplication in code base - provided you use properly released packages, then the packages.config file content is sufficient for reliably reproducing the exact dependencies your application needs.
Synchronization is an effort - keeping packages.config and packages folder in sync- once you start including them in source control every developer working with packages would monitor and add or remove packages to source control.
If devs ever forget to add then local build still fails.
If they forget to remove no longer necessary piece then your downloadable set would contain junk.
VCS dataset size - storing them would needlessly enlarge your version control storage. Quite often the packages contain N different platform dlls, tools and whatnot which add up quite fast. Should you keep your dependencies constantly up to date, then after 10 years your VCS history would contain huige amount of irrelevant junk. Storage is cheap, but still..
Instead, consider having a private nuget repository with the purpose of serving and archiving the packages your application needs and set up your project to check your project nuget repository first. If your developers need offline compile support then they can set up project repository mirrors on their build boxes and configure the following fallback structure for repos:
Developer local project repository (ex: folder)
Shared project repository (ex: Nuget.Server)
(nuget.org)
A guide how to configure multiple repositories can be found here: How to configure local Nuget Repository.
I have turned on TeamCity's NuGet Server and I want to push in common packages (i.e. from public sources such as NuGet.org) because the build server cannot see outside our company, so restoring packages on the build server from NuGet.org is not possible.
I cannot see how to push these packages on to our TeamCity server. I've seen various answers suggesting to use a package build still or some other means of publishing from within a build, but this is not appropriate for my use case.
If I try to publish from a command line it complains that it cannot find an API key (where do I get that from?) and it won't allow me to enter my credentials (I assume my team city login would be it) as it tells me "Cannot prompt for input in non-interactive mode." (I didn't set that mode and I can't see how to turn that off).
So, how do I push/publish an adhoc package that I obtained elsewhere into team city?
I believe that the nuget functionality provided by TeamCity is an API added on top of TeamCity's builtin artifact functionality.
There are a number of consequences of that:
When a build configuration is executed that produces any .nupkg files that are marked as artifacts, they will be available on the Teamcity nuget feed.
As with all other artifacts nupkgs published in TeamCity are subject to Teamcity's general artifact retention rules.
Access rules for nuget packages are the same as access to the TeamCity projects.
There is however as far as I know no implementation in the Teamcity Nuget API for pushing packages to it. The general practice for storing original or generated packages is to use a stand alone nuget server or service like a normal file share, a Nuget.Core based server, proget or myget.org.
Update:
If you end up with many packages of your own I've heard people reporting that Teamcity becomes quite slow when the clients are resolving the packages.
Update 2:
The last years I've adopted the notion of separating build artifact packages into the two categories library package and deployment package. A separate package repository can be used for both types but a repository such as the one available in for instance Octopus deploy should only be used for deployment packages.
Update 3:
Microsoft have a page for a number of nuget server options.