We are experiencing issues with restoring Nuget packages in Azure DevOps builds due to failed cache on private Nuget server. Our current solution is clearing the cache on the server by deleting the nuget.cache file and restarting nuget.exe. This problem seems to increase frequency as the frequency of builds also increase. Does anyone know of a permanent solution to this?
Related
We changed our maven private repository Nexus to Azure Artifacts.
And when we upload a private maven dependency with a some version, it told us below info:
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy
(default-deploy) on project HmapBasic: Failed to deploy artifacts:
Could not transfer artifact com.hand:HmapBasic:pom:3.0.3-RELEASE
from/to dev-azure-com-kohlerdevops-konnect-unified
(https://pkgs.dev.azure.com/KohlerDevOps/_packaging/Konnect-Unified/maven/v1/):
Failed to transfer file:
https://pkgs.dev.azure.com/KohlerDevOps/_packaging/Konnect-Unified/maven/v1/com/hand/HmapBasic/3.0.3-RELEASE/HmapBasic-3.0.3-RELEASE.pom.
Return code is: 409
I searched the code 409 error. It means we have already uploaded the dependency with this version and we can't upload this again.
We want to use the new dependency cover the old one without changing version(because if we change the version, all of our projects need to change pom.xml)
how can we do that in Azure Artifacts?
How to update a maven dependency with a same version number in Azure Artifacts
I am afraid you have to change the pom.xml, I understand how frustrating this is for you. But this is what Azure Artifacts deliberately designed to ensure the correctness of the build results.
Check the document: Understand immutability of packages.
Once you publish a particular version of a package to a feed, that
version number is permanently reserved. You cannot upload a newer
revision package with that same version number, or delete it and
upload a new package at the same version.
The reason for this design:
That because many package clients, including NuGet, keep a local cache
of packages on your machine. Once a client has cached a particular
package#version, it will return that copy on future install/restore
requests. If, on the server, you replace package#version (rev 1) with
a new package#version (rev 2), the client is unable to tell the
difference. the client still use the old version in cache. This can lead to indeterminate build results from
different machines.
So we could not overwrite all packages that have already been
uploaded, even we could not delete them and re-uploaded. To resolve
this issue, the only way is that upload the packages with a new
version.
Check my another thread for some more info.
Hope this helps.
I just made a test on Azure DevOps and if you use the "SNAPSHOT" suffix it will overwrite the previous package/artifact. For instance 2.0.0-SNAPSHOT.
I'm trying to use the nuget install plugin for TeamCity but after running, it takes too long in restoring and installing all the packages in my solution.
Can someone help in nuget restore packages in TeamCity?
I was researching and solving the same problem. My solution took up to 20 minute(in bad times) to resolve all NuGet packages. Here is what I had found:
TeamCity doesn't use the nuget cache because they reassign the env variable for the directory where they keep resolved packages. The details about this you can find on here https://teamcity-support.jetbrains.com/hc/en-us/community/posts/115000160524/comments/360000087900
And here is the reason why they do it https://github.com/NuGet/Home/issues/4277
I'm not sure why but the version NuGet that provided by TeamCity v3.5, v4.6 proceed with restore operation slower than other versions.
What I did:
In my case I've added the override of the env.NUGET_PACKAGES to the home Agent directory (%teamcity.agent.home.dir%/nuget-cache) in my
TeamCity project. That made restore stable not more than 10 minutes. Be careful because the work and temp directory might be cleared by TeamCity Agent and your cache will not be stable.
I've added the latest stable version of the NuGet.exe file (v5.8.1 in my case) https://www.nuget.org/downloads. Then I'd chosen the Custom option for NuGet Install step in TeamCity (NuGet.exe select box) and set the path to my NuGet.exe file in the solution.
After both of these steps, NuGet restore takes up to 1.5 minutes for my solution and this is ok for me.
I have a private NuGet feed where I publish package A.
A has a version like 4.0.0.X (where X is the build number).
When I change code, the build number is incremented and the package is published.
In the csproj, I have referenced the package like this:
<PackageReference Include="A" Version="4.0.*" />
I want to get the newest version of A, which has no major changes (which would result in a bump the minor section...).
Unfortunately, if Nuget has downloaded a Version of A, it never attempts to check if there is a newer version.
I can check manually, but then Nuget automatically pins the version in the csproj, which I have to re-edit.
How can I fix this?
Fix means: I want a smooth dev experience for my CI workflow. Ideally, I have the newest package version on my dev computer without lots of manual work.
NuGet has caching strategies to avoid re-downloading packages/hitting the network all the time.
It dumps a little cache file in your obj that tells it whether you've changed your dependencies at all.
You will need to force NuGet to reevaluates the available packages from remote sources, by using the -force option. /p:RestoreForce=true, or --force in dotnet CLI.
In Visual Studio, currently a rebuild will do a force restore.
NuGet also has a http caching strategy that avoids hitting the remote feeds for 30 mins. To override that, use -NoCache from the commandline. Currently there's no option to override that in Visual Studio.
tl;dr;
NuGet caches a lot of things to improve performance and remote unnecessary remote calls.
Avoid that by calling restore with the --force/rebuilding.
I'm trying out VSO and it's taking over 2 minutes to sync with a GitHub repository. It appears that it's checking out the whole thing on every build. I made sure that the "clean" box is unchecked but it had no effect.
Any ideas on how to get it to cache the source or is this even possible in VSO?
Each build in VSO uses a new VM that is spun up just for your build. Short
of hosting your own Build Server connected your VSO, I don't think it can be avoided.
Unless there are ways to speed up a the process of downloading the code from a git repo, I think you're stuck.
Not every time but quite frequently I get ##[error]The nuget command failed with exit code(1) and error(NU1102: Unable to find package MyPackage with version (>= 1.0.5)
in my Azure Pipelines builds (different packages and different versions):
The package definitely exists as it had just been built a couple of minutes ago and I can see it in the Artifact Feed:
Here is my pipeline:
The project consists of multiple packages so this is starting to get really inconvenient. If I try on different build agents I eventually get one to work but the pipelines are supposed to be more of a hands off process. As far as I can tell (or guess), Nuget is caching the index.json for the feed. The only other issues I can find related to this specifically happen in people's local environments. Is there some way to get Nuget to properly check if packages exist?
1.Not sure about the real cause of your issue, but if cleaning cache can help to resolve your issue, you only need to enable this option in Restore task.
2.Also, sometimes the package not found error could be related to feed permissions in Devops. Go Artifacts=>custom Feed=>Feed Settings in right corner:
Make sure your build service have access to that feed.
NuGet does cache what versions of a package are available on each feed for 30 minutes, so the package was published more recently than that, and that machine had already restored a different version of the package within the last 30 minutes, it will be a problem.
You could run dotnet nuget locals http-cache --clear or nuget.exe locals http-cache -clear before the restore, that will delete NuGet's HTTP cache.