I have a GitHub worflow which builds and deploys a snapshot version of a library as a GitHub package, e.g., mycompany.mytool.1.0.0-SNAPSHOT.jar. Whenever I make a new build and deploy, a new asset is created, like, e.g., mycompany.mytool.1.0.0-20210723.145233-1.jar instead which is then somehow associated with the SNAPSHOT tag. This all seems to work and I can access mycompany.mytool.1.0.0-SNAPSHOT.jar without problem.
My question now is, how can I get rid of all these older versions of this jar? Actually I just want to keep the latest version. I can delete them manually via the web-interface but that is a more than awkward task. I would somehow like to automate this too.
This is not possible as of this writing. GitHub staff member Jamie Cansdale wrote this in their community forum:
SNAPSHOT versions are exposed as artifacts inside a regular versions. There isn’t an API for cleaning up artifacts, only whole versions.
(source)
Which means that a single SNAPSHOT version (like 1.0.0-SNAPSHOT) will accumulate all builds you make, and all artifacts will show up on the Assets list to the right of the web page.
The only practical solution I can think of, is that you delete the whole version from a script, before publishing each build's artifacts. Then you'd have the effect of having a single set of artifacts stored as part of the 1.0.0-SNAPSHOT version name.
However this solution is not ideal: public package versions cannot be deleted if they are popular enough (probably to avoid squatting attacks):
If the package is public and the package version has more than 5,000 downloads, you cannot delete the package version. In this scenario, contact GitHub support for further assistance.
Related
This is the method I use to download NuGet package using NuGet.Client API.
public DownloadResourceResult DownloadPackage(string packageId, NuGetVersion version)
{
var packageSource = new PackageSource(_Task.PackageSource);
var sourceRepository = new SourceRepository(packageSource, Repository.Provider.GetCoreV3());
DownloadResource downloadResource = _DownloadResource ?? (_DownloadResource = sourceRepository.GetResourceAsync<DownloadResource>().Result);
var packageIdentity = new PackageIdentity(packageId, version);
var packageDownloadContext = new PackageDownloadContext(NullSourceCacheContext.Instance);
string globalPackagesFolder = SettingsUtility.GetGlobalPackagesFolder(_Settings);
return downloadResource.GetDownloadResourceResultAsync(packageIdentity, packageDownloadContext, globalPackagesFolder, _Logger, CancellationToken.None).Result;
}
Overall it does what it's supposed to do. There is one thing I don't understand. Is there a way to tell GetDownloadResourceResultAsync method to ignore current cache and download and unpack the package again.
For example, I have a NuGet package called MyPackage in the NuGet repository. After I call this method to download the package. MyPackage is downloaded and unpacked into C:\Users[CurrentUser].nuget\packages. If I update MyPackage in the repository and then call DownloadPackage again it won't update the cache and the whole system will think the package was not updated. I can manually delete the package from C:\Users[CurrentUser].nuget\packages before downloading but it seems to invasive. I guess there should be a way for better cache control.
I understand normally it might not be necessary because every time you need to change something in the package you change the package version but it is necessary for automatic packaging and deploying in case you need to re-deploy and re-test something without updating the version.
I believe this is an example of an XY problem. NuGet is designed for packages to be immutable, so you're working against the design if you recreate the same package version with different content, even for testing. You only made a passing comment about automatic packaging and deploying, but didn't explain why you're writing custom code to download packages rather than using nuget.exe, so it's hard for me to make a good suggestion.
If you have a CI/CD environment where packages are created automatically, and other tests automatically use them, then I recommend the packages are built with prerelease version numbers and published to a dev nuget feed. For example, here you can see where the ASP.NET team's CI server publishes dev builds of a package, multiple per day. When they're ready to publish the production ready version, a different CI build packs without a prerelease version and pushes to nuget.org instead of their dev feed.
Depending on what you're trying to do, changing the global packages folder might work for you. Assuming you're using git for source control, if you make your test's global packages folder in an ignored part of your git repo, then git clean -Xdf will delete it, and I believe it's common for CI servers to clean before a build. This is also something I've done for local, manual testing where I didn't want to pollute my global cache. I just delete the folder once I'm done.
Ultimately the package cache is just a folder, which you can delete using System.IO APIs, no need for using NuGet APIs. Although, if you want to account for the fact that nuget.config files can change the global packages folder, you might want to use the NuGet APIs to find it. If you don't mind having all packages deleted, you could just run nuget.exe locals global-cache -clear and avoid writing any code.
I'm actually thinking about the pro and cons about using NuGet. In our current software we're storing each external reference in a common reference folder (which is commited to our SW versioning system). Over time this approach becomes more and more painful because we've to store different versions to the same library.
Since our devs are sometimes at the customer site (where not all customers are offering internet connectivity ...) we won't use NuGet directly, because NuGet packages can't be restored.
Based on that I'm actually thinking about using NuGet and store the packages folder in our SW versioning system.
Does anybody know if there are some disadvantages about this solution? Does anybody have a better proposal?
Thx.
I would argue against storing external nuget packages in your version control system.
It's not your application's responsibility to archive third party packages. Should you need to take care of that risk then build a solution intended for such (for example: use private nuget repository that's properly backed up).
Avoid duplication in code base - provided you use properly released packages, then the packages.config file content is sufficient for reliably reproducing the exact dependencies your application needs.
Synchronization is an effort - keeping packages.config and packages folder in sync- once you start including them in source control every developer working with packages would monitor and add or remove packages to source control.
If devs ever forget to add then local build still fails.
If they forget to remove no longer necessary piece then your downloadable set would contain junk.
VCS dataset size - storing them would needlessly enlarge your version control storage. Quite often the packages contain N different platform dlls, tools and whatnot which add up quite fast. Should you keep your dependencies constantly up to date, then after 10 years your VCS history would contain huige amount of irrelevant junk. Storage is cheap, but still..
Instead, consider having a private nuget repository with the purpose of serving and archiving the packages your application needs and set up your project to check your project nuget repository first. If your developers need offline compile support then they can set up project repository mirrors on their build boxes and configure the following fallback structure for repos:
Developer local project repository (ex: folder)
Shared project repository (ex: Nuget.Server)
(nuget.org)
A guide how to configure multiple repositories can be found here: How to configure local Nuget Repository.
I've been looking into artifact repositories for something that our release team can use for storing outputs of full builds from multiple projects. From what I've read, artifact repositories are mostly used for storing library files required for a build. My assumption is that their intended use is to ensure developers and build servers are using the exact same binary dependencies during build process.
Few questions:
Is it possible to store the build output of entire projects into an artifact repository (A full release), a place to store artifacts ready for deployment?
Is this common practice?
Is it possible to have analytics of what was changed since the last build? Ex: can I see which artifacts have changed since the last release?
So, the short answer to your questions are: yes, yes, and mostly yes.
While it is true that Binary Managers such as Artifactory are used for dependency management they are also used to host entire builds.
In Artifactory this can be easily achieved through the Build Integration features. If you are not using any CI server such as Jenkins (for example) you can use the JFrog CLI to upload your builds and their corresponding Build Info.
In addition, with regards to analytics, not exactly as such, but in Artifactory you have the option to perform Build Diff and see the changes between builds.
Hope I helped,
Eran
p.s. I work for JFrog
Using Sonatype Nexus woks for what you need, you are able to deploy not just Java artifacts (example: .ear, .jar, .war files) you are able to deploy any kind of binaries, we are using it for storing reports for Orace BI Publisher, or .exe binaries.
Is it possible to store the build output of entire projects into an artifact repository (A full release), a place to store artifacts ready for deployment?
Yes, as I said before, you can store any kind of binaries you want.
Is this common practice?
I don't know if it is a common practice, but in my case It helped us to keep an order. Just evaluate if it works for you.
Is it possible to have analytics of what was changed since the last build? Ex: can I see which artifacts have changed since the last release?
Sonatype Nexus handle a version for each artifact (or binary) so you are able to store all the "history" from your deployments, also it is able to handle security policy for example you could not deploy the same binary twice with the same version it forces you deploy a new version in this way you can verify when an artifact has changed, the date and who uploaded the artifact.
This is how it looks like:
I recently installed Nuget Gallery (https://github.com/NuGet/NuGetGallery) as a repository. Ideally I would like to create multiple feeds so that I could differentiate between nuget packages that will be reused in other projects (dll's, contracts etc) from the packages we use to deploy our projects to the production environment.
I know I can achieve this by creating multiple instances of the Nuget Gallery, but this seems to me a bit of an overkill, it would mean two websites two databases. I am also familiar with the fact that MyGet provides this functionally but I will not be able to get an approval for the purchase. I am also aware teamcity contains its own feed server but it doesn't allow this multiple feed scenario, nor its performs well enough to be used in a large scale.
In a nutshell the ideal deployment scenario would be as follow:
teamcity generate deployment package or dll/contract package, depending on the build scenario.
teamcity publishes deployment packages to a nuget gallery deploy feed
(say: nugetgallery.server.com/deploy/api/v2).
teamcity publishes dll/contract packages to a nuget gallery dev feed
(say: nugetgallery.server.com/dev/api/v2).
octopus always searches for packages in
nugetgallery.server.com/deploy/api/v2
devs / teamcity searches for packages in
nugetgallery.server.com/dev/api/v2
This way I keep things clean and I can even go as far as create a third type of feed that only contains release packages so that I can be sure nothing would ever be deployed to production if it wasn't on that feed.
I might have missed some fundamental approach, so alternatives to this one I picked are welcome.
As I couldn't find anything relevant I ultimately gave up and went with the two servers solution. I struggled a lot to find any documentation what functionalities the nuget gallery really has.
Right now we have something like deploy-nuget.server.com and dev-nuget.server.com, separate urls, iis instances and sql instances and folder location.
For someone that might look into this in the future, one of the solutions that could work is to make private repository based on the user, unfortunately in my case that would not be enough as I would also want the packages to be stored in different locations so we could enforce different backup policies based on the type of package. Another option would be to actually change to fork the project, but from my previous experience this never ends well as sooner rather than later you will want to upgrade and your custom changes will have to be sorted somehow.
I understand this is not the idea behind nuget gallery, as you are not supposed to delete packages. But we do have some space constraints so eventually we will remove certain deployment packages that were created for QA environments which we obviously dont care anymore.
you can try Proget. using this server you can easily manage multiple NuGet feeds.
it also provides free edition which supports all features.
We used nuget.server 2.8 to create a private feed for hosting nuget packages (mostly chocolatey packages) in our organization. I would like to improve/expand the indexing capability but I can't figure out how to do that.
I know in a typical nuget server feed, all the .NUPKG files would be in the root of the package path specified in the config. Long story short, we have a requirement for a folder structure in that package feed as different groups within the organization will be using SVN to commit data which ends up here. To easily manage this, we need a more complex folder structure.
However what I have found is that .NUPKGs in the root of the package path or one folder deep are indexed and available via the feed. Once you go two folders deep, the NUPKG files aren't indexed and aren't available via the nuget feed. Is there a relatively easy way I can change that? Is that a setting specified somewhere? I can't seem to find where this limitation is coming from. Any direction would be outstanding.
We've had a few users request such a feature for ProGet, but ultimately decided against implementing the feature because of the problem of not only dealing with duplicate packages, but communicating that problem to the user.
Remember that a valid NuGet package must have a file name that matches its version+id (e.g. MyPackage.1.2.nupkg can only be MyPackage v1.2). Thus if you have folderA\MyPackage.1.2.nupkg and folderB\MyPackage.1.2.nupkg, which is the valid? Do you invalidate both? Etc.
That said, it's trivial to implement, so you could simply use the ProGet SDK to build your own package store that inherits from the default, but iterates subdirectories as well.
As a side note, if you're serious about maintaining a private repository, you really should get something other than NuGet.Server. There are several available that can manage chocolately packages.
Symlinks is your best bet. You will just want to symlink those files up on a regular basis with a scheduled task.
I have to second Karl's answer on using something better than NuGet.Server. Depending on your growth potential, it can start to become unusable fast after you have 100+ packages in the repository. Note: I haven't checked this myself since 2012, it's possible it has better support now for multiple packages.