I have a private NuGet feed where I publish package A.
A has a version like 4.0.0.X (where X is the build number).
When I change code, the build number is incremented and the package is published.
In the csproj, I have referenced the package like this:
<PackageReference Include="A" Version="4.0.*" />
I want to get the newest version of A, which has no major changes (which would result in a bump the minor section...).
Unfortunately, if Nuget has downloaded a Version of A, it never attempts to check if there is a newer version.
I can check manually, but then Nuget automatically pins the version in the csproj, which I have to re-edit.
How can I fix this?
Fix means: I want a smooth dev experience for my CI workflow. Ideally, I have the newest package version on my dev computer without lots of manual work.
NuGet has caching strategies to avoid re-downloading packages/hitting the network all the time.
It dumps a little cache file in your obj that tells it whether you've changed your dependencies at all.
You will need to force NuGet to reevaluates the available packages from remote sources, by using the -force option. /p:RestoreForce=true, or --force in dotnet CLI.
In Visual Studio, currently a rebuild will do a force restore.
NuGet also has a http caching strategy that avoids hitting the remote feeds for 30 mins. To override that, use -NoCache from the commandline. Currently there's no option to override that in Visual Studio.
tl;dr;
NuGet caches a lot of things to improve performance and remote unnecessary remote calls.
Avoid that by calling restore with the --force/rebuilding.
Related
This is the method I use to download NuGet package using NuGet.Client API.
public DownloadResourceResult DownloadPackage(string packageId, NuGetVersion version)
{
var packageSource = new PackageSource(_Task.PackageSource);
var sourceRepository = new SourceRepository(packageSource, Repository.Provider.GetCoreV3());
DownloadResource downloadResource = _DownloadResource ?? (_DownloadResource = sourceRepository.GetResourceAsync<DownloadResource>().Result);
var packageIdentity = new PackageIdentity(packageId, version);
var packageDownloadContext = new PackageDownloadContext(NullSourceCacheContext.Instance);
string globalPackagesFolder = SettingsUtility.GetGlobalPackagesFolder(_Settings);
return downloadResource.GetDownloadResourceResultAsync(packageIdentity, packageDownloadContext, globalPackagesFolder, _Logger, CancellationToken.None).Result;
}
Overall it does what it's supposed to do. There is one thing I don't understand. Is there a way to tell GetDownloadResourceResultAsync method to ignore current cache and download and unpack the package again.
For example, I have a NuGet package called MyPackage in the NuGet repository. After I call this method to download the package. MyPackage is downloaded and unpacked into C:\Users[CurrentUser].nuget\packages. If I update MyPackage in the repository and then call DownloadPackage again it won't update the cache and the whole system will think the package was not updated. I can manually delete the package from C:\Users[CurrentUser].nuget\packages before downloading but it seems to invasive. I guess there should be a way for better cache control.
I understand normally it might not be necessary because every time you need to change something in the package you change the package version but it is necessary for automatic packaging and deploying in case you need to re-deploy and re-test something without updating the version.
I believe this is an example of an XY problem. NuGet is designed for packages to be immutable, so you're working against the design if you recreate the same package version with different content, even for testing. You only made a passing comment about automatic packaging and deploying, but didn't explain why you're writing custom code to download packages rather than using nuget.exe, so it's hard for me to make a good suggestion.
If you have a CI/CD environment where packages are created automatically, and other tests automatically use them, then I recommend the packages are built with prerelease version numbers and published to a dev nuget feed. For example, here you can see where the ASP.NET team's CI server publishes dev builds of a package, multiple per day. When they're ready to publish the production ready version, a different CI build packs without a prerelease version and pushes to nuget.org instead of their dev feed.
Depending on what you're trying to do, changing the global packages folder might work for you. Assuming you're using git for source control, if you make your test's global packages folder in an ignored part of your git repo, then git clean -Xdf will delete it, and I believe it's common for CI servers to clean before a build. This is also something I've done for local, manual testing where I didn't want to pollute my global cache. I just delete the folder once I'm done.
Ultimately the package cache is just a folder, which you can delete using System.IO APIs, no need for using NuGet APIs. Although, if you want to account for the fact that nuget.config files can change the global packages folder, you might want to use the NuGet APIs to find it. If you don't mind having all packages deleted, you could just run nuget.exe locals global-cache -clear and avoid writing any code.
I'm at wits end here. I've tried everything I can find to get NuGet to restore my packages on the TFS build server and nothing seems to work. (The latest changes I checked in I got here). I have read a lot about problems with older versions of the Build Process Template, so I just changed the build to use TfvcTemplate.12.xaml and still no packages are restored. This latest change (from the link above) had me delete the NuGet.targets file and now I get an error saying:
This project references NuGet package(s) that are missing on this computer.
Enable NuGet Package Restore to download them. For more information, see
http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is
:\Builds\TFS\WebApps\src\WebApps\Main\Src\\.nuget\NuGet.targets.
Since you have tried so many solutions based on google. Which may make your environment complex and confusing.
First there are two ways to do the nuget restore MSBuild-integrated restore and Automatic restore in Visual Studio. There are totally different things.
With NuGet 2.7 and later, Visual Studio automatically restores missing
packages by default at the beginning of a build.
Since you are using NuGet 3.4, suggest you to use Automatic restore.
You could only select one way to do the restore operation. Based your error message NuGet.targets not found, maybe you haven't finished Migrating to automatic restore step. Please do it again or double check this.
You could also test the migration, do the following:
Remove the packages folder from the solution.
Open the solution in Visual Studio and start a build.
Automatic restore should download and install each dependency
package, without adding them to source control.
Do not add $(SolutionDir).nuget\nuget.exe restore -SolutionDirectory ..\ to the pre-build event on my first project in the build order . This is the old way to do the restore. Just like the way describe here.
Nuget Restore.exe was added to the 2013 (v12) templates. Since you are using the TfvcTemplate.12.xaml , you don't have to add it again.
Highly doubt you were mixing up the two ways cause your present dilemma. Suggest you follow the automatic restore process, use a clean environment, complete migration to automatic restore step , test the migration locally, make sure NuGet.exe and NuGet.targets files from .nuget folder in source control are removed, checking pending changes, check your TfvcTemplate.12.xaml Process Templates with Nuget Restore functionality. Create a new build definition with old settings and use the template. Finally trigger the build...
I have a solution which produces several NuGet packages, and I pack the packages during build. I want my nightly builds to be marked as pre-release, so I version my packages accordingly: 1.2.3-PreRelease0001. However, once a nightly build passed testing, I want to publish the same build, with the same packages, but using a non-PreRelease version: 1.2.3.
My question: How can I re-package a NuGet package with a different version? I guess I could hack some unzip/edit/nuget pack script, but is there a better way?
Alternatives:
Don't package during build - package in a separate process, which I can re-run later.
Con: If I package during build, I get access to <Content> files directly from the sources
Run another build, this time packaging with the non-PreRelease version.
Con: Want to distribute the exact same bits I tested...
Don't mark nightly builds as PreRelease, and instead publish them to a separate repository.
Con: PreRelease packages are not marked as such, and could get mistaken as released.
Package during build twice: Once with PreRelease and once without.
Con: People might be tempted to ref the non-PreRelease versions. Maybe I could put them inside some GeneratedDoNotTouch folder...
There is no public API to change a package’s metadata in NuGet. I would say the last solution, i.e., produce both the prerelease and non-prerelease packages during build, is the best.
To prevent people from accidentally using the non-prerelease package, you can create it in a private directory first. Then, publish it only after the build passes testing.
Yes, you can extract the nuspec file from the package, make the necessary changes and then save the file back into the package. The problem is that this might stop working if the nuspec format is changed.
We use Nuget for our internal development to allow us to share code across teams. We run into issues however when one person is working on code that will be deployed across multiple nuget packages at the same time. For instance
A depends on B which depends on C.
A, B and C have their artifacts pushed to Nuget and that's how we manage the dependencies between A, B and C. The trouble we find is that if a developer wants to make changes in C and quickly see those changes reflected in A, they have to go through the following process.
Make change in C.
Push change up to git
CI picks up change to C and builds and deploys new nuget package.
Go into B and update reference to C using a nuget update package command.
Push up the change to the packages.config file up to git
CI picks up change to B and builds and deploys new nuget package for B
Now open A and change reference to B and nuget update package
Make changes in A to go along with the changes in B(and transitively C)
This seems extremely painful and is causing some of our developers to question the choice of Nuget for our internally developed code. Everyone still like it for consuming external packages.
Is there any better workflow for using Nuget internally?
In our company we have solved the cascading updates problem with the following setup. First we have the following setup for our NuGet repositories and build server.
There is an internal NuGet repository that holds all the published packages for the company. This repository is just a shared directory on one of our servers.
Each developer can have (but doesn't need to have) one or more directories on their own machine that serves as a local NuGet package repository. By using a user specific NuGet configuration the developer can control in which order NuGet searches through the package repositories to find packages.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageRestore>
<add key="enabled" value="True" />
</packageRestore>
<packageSources>
<add key="Dev" value="D:\dev\testpackages" />
<add key="Company" value="<UNC_ADDRESS_COMPANY_REPOSITORY>" />
<add key="NuGet official package source" value="https://nuget.org/api/v2/" />
</packageSources>
<disabledPackageSources />
<activePackageSource>
<add key="All" value="(Aggregate source)" />
</activePackageSource>
</configuration>
All solutions have automatic package restore turned on, so that we don't have to commit the packages to our version control system.
Developers only control 3 out of the 4 version numbers, e.g. if the version is <MAJOR>.<MINOR>.<BUILD>.<REVISION> then developers can only change the major, minor and build numbers, the revision number is set to 0 except in builds done by the build server where it is the build number of the build. This is important because it means that for a given version consisting of a major, minor and build number the build server will always produce the higher version number. This again means that NuGet will prefer to take the package version coming from the company package repository (which only gets packages through the build server).
In order to make a change to one of the base libraries there are two possible processes being used. The first process is:
Make the changes to the base library (A). Update the version of (A) if needed.
Run the MsBuild script to build the binaries and create the NuGet packages of (A)
Copy the new NuGet packages over to the package repository on the local machine
In the dependent project (B) upgrade to the new packages of (A) that were just placed in the local machine package repository (which should be of a higher version than the ones available on the company wide repository, or NuGet.org)
Make the changes to (B).
If more changes are required to (A) then repeat steps 1,2 and 3 and then delete the package of (A) from the working directory of (B). Next time the build runs NuGet will go looking for the specific version of (A), find it in the local machine repository and pull it back in. Note that the NuGet cache may thwart this process some of the time, although it looks like NuGet may not cache packages that come from the same machine(?).
Once the changes are complete, then we:
Commit the changes to (A). The build server will run the integration build to verify everything works.
Tell the build server to run the release build, which builds the binaries and pushes the NuGet packages to the company-wide NuGet repository.
In (B), upgrade to the latest version of (A) (which should have a higher version number than the test package because the test package should have version a.b.c.0 while the newly build version in the company-wide repository should be a.b.c. where > 0
Commit the changes to (B). Wait for the build server to finish the integration tests
Tell the build server to run the release build for (B).
Another way of doing the development work is by taking the following steps
Make the changes to the base library (A). Update the version of (A) if required.
Build the binaries
Copy the binaries over to the location where NuGet unpacks the package of (A) for project (B) (e.g. c:\mysource\projectB\packages\ProjectA.1.2.3.4)
Make the required changes to project (B)
The commit process is still the same, project (A) needs to be committed first, and in project (B) the NuGet reference to (A) needs to be upgraded.
The first approach is slightly neater because this process also warns if there are faults in the NuGet package of (A) (e.g. forgotten to add a new assembly) while in the second process the developer won't know until after the package for (A) has been published.
You have two choices here:
Run an instance of NuGet Gallery within your organisation. This is the code which runs nuget.org
Get a license for Artifactory Pro, which has in-built Nuget support and acts as a Nuget repository.
I have used both, and #1 is a reasonable choice to start with, but NuGet Galley is optimised and designed for nuget.org, not on-premise/enterprise use, so things like deleting packages is a pain (hand-rolled SQL required).
I'd say that you should pay the (low) license fee for Artifactory Pro - it's an excellent product, and the JFrog team are really keen and switched on.
You should not be using nuget.org for internal/enterprise packages; nuget.org is designed for 3rd party/open source libraries, not internal build dependencies.
EDIT: in terms of workflow, why are you putting shared code into multiple packages? If the code needs to be shared, it needs to go in its own separate package.
EDIT 2: To speed up the code change workflow for the developer, you can use nuget.exe (the command-line client) and use command-line accessible builds, so you can target a "developer" build run. Then in your "developer" build (as opposed to the CI build) you specify -Source as a local path (e.g. nuget install B -Source C:\Code\B) when you want to pull the newly-updated B as a dependency and build against that; likewise for C or other local, newly-updated packages. Then when A, B, and C all build fine, you can git push all of them (in reverse dependency order), and let CI do its thing.
However, you also should question whether your package separation is really appropriate if you have to do this build 'dance' often, as this suggests that all the code should be in a single package, or possibly split along different lines in separate packages. A key feature of a well-defined package is that it should not cause ripple effects on other packages, certainly not if you are using Semantic Versioning effectively.
Edit 3 Some clarifications requested by marcelo-oliveira: "command-line accessible builds" are builds which can take place entirely from the command-line, without using Visual Studio, usually via batch files. A "developer build" is a build which a developer runs from her workstation, as opposed to the CI build which runs on the CI server (both builds should essentially be the same).
If A, B and C are under the same solution, you can create NuGet packages for them in a single build.
Just make sure that the using package has the new version number (assuming your build doesn't randomly change it) of the package it depends on.
If A, B and C are intentionally under different solutions e.g. A is under an infrastructure solution and B is under a product solution, then the only suggestion I have for you is to define your CI builds run on check in and not periodically.
Edit:
Another option is to create a push pre-release packages (e.g. version 1.0.0-alpha) to your NuGet server during local builds, this way developers can experiment with the package prior to creating a release version.
Nuget was designed for sharing third party libaries. We plan to use Nuget internally for all common code that can be shared between projects. Things like a common data layer, string and other regular expression functions, email components and other artifacts like that.
The only way I see Nuget being of any help is when the components, code or artifacts you are sharing across teams/projects are stable and have their own release cycle that is different from your project's release cycle. For your needs Nuget is an overkill. You might have better productivity linking all such projects within the same solution. Your solution structure should include the three projects A,B and C as dependent projects referenced internally wherever required.
Not every time but quite frequently I get ##[error]The nuget command failed with exit code(1) and error(NU1102: Unable to find package MyPackage with version (>= 1.0.5)
in my Azure Pipelines builds (different packages and different versions):
The package definitely exists as it had just been built a couple of minutes ago and I can see it in the Artifact Feed:
Here is my pipeline:
The project consists of multiple packages so this is starting to get really inconvenient. If I try on different build agents I eventually get one to work but the pipelines are supposed to be more of a hands off process. As far as I can tell (or guess), Nuget is caching the index.json for the feed. The only other issues I can find related to this specifically happen in people's local environments. Is there some way to get Nuget to properly check if packages exist?
1.Not sure about the real cause of your issue, but if cleaning cache can help to resolve your issue, you only need to enable this option in Restore task.
2.Also, sometimes the package not found error could be related to feed permissions in Devops. Go Artifacts=>custom Feed=>Feed Settings in right corner:
Make sure your build service have access to that feed.
NuGet does cache what versions of a package are available on each feed for 30 minutes, so the package was published more recently than that, and that machine had already restored a different version of the package within the last 30 minutes, it will be a problem.
You could run dotnet nuget locals http-cache --clear or nuget.exe locals http-cache -clear before the restore, that will delete NuGet's HTTP cache.