Is it possible to fetch a NuGet package from a private feed (i.e., requiring login/password) within a script?
I have a NuGet package with a single JSON file. It's hosted on a private TeamCity feed. I want to use that JSON in a script, so here's what I'm trying to do:
nuget sources add -Name tc -Source https://xxxxxxxxx/ -Username username -Password password
nuget install xxxxxxxx -source tc
nuget sources remove -Name tc
Which results into Unable to find version '1.0.7' of package 'xxxxxxx'. Which is strange, for it's obviously able to reach the feed since it see the version of the package, but can't download the package.
I also tried to install the package into a dummy .sln and run nuget restore xxxxx.sln. NuGet is then asks me to provide credentials, two times:
Please provide credentials for: https://xxxxxxxx/httpAuth/app/nuget/v1/xxxxx.svc/
Please provide credentials for: https://xxxxxxxx/httpAuth/repository/download/xxxxx/1148145:id/xxxxxxxx.1.0.7.nupkg
Creating a NuGet.config with a feed & credentials declaration also results into the same Unable to find error.
It works OK when I'm installing/restoring the package from VS.
These two questions seems related:
NuGet Package Restore Unable to prompt for credentials with custom feed
https://stackoverflow.com/questions/24682237/nuget-command-line-authentication-not-working
It looks like it's a problem with the TC feed. I didn't think that was the case because I can use the feed from VS Package Manager, the problem is only with command-line NuGet.
I turned on -Verbosity detailed (thanks danliu) and here's what I saw:
GET https://xxxxxx/httpAuth/app/nuget/v1/FeedService.svc/Packages(Id='xxxxxx',Version='1.0.7')
GET https://xxxxxx/httpAuth/app/nuget/v1/FeedService.svc/Packages(Id='xxxxxx',Version='1.0.7.0')
System.InvalidOperationException: Unable to find version '1.0.7' of package 'xxxxxx'.
The request GET https://xxxxxx/httpAuth/app/nuget/v1/FeedService.svc/Packages(Id='xxxxxx',Version='1.0.7') returns 404, I think it's related to server configuration problem which is mentioned in TeamCity NuGet feed settings:
NuGet Feed must contain server URL inside. Current TeamCity server configuration does not let TeamCity server to get original request URL from HTTP request. It looks like TeamCity server is wrongly configured with reverse proxy. Make sure reverse proxy and TeamCity server is configured to let TeamCity server know request real request URL.
I tried to host the package on MyGet and install it from there. It worked out.
Related
I've got the Azure DevOps pipeline which builds nuget package and deploys it to Azure DevOps feed. In the portal I can find download link to a specific version of the package.
How do I find url to download latest version of the nuget package in the feed?
Alternatively how do I download the latest version of the package?
Ideally just via curl, dotnet or anything what is resent on dev windows machine and in general docker sdk image.
I tend to go long way
dotnet new console
add package
restore
find the location of the file. But I really don't like this approach. Anything nicer?
How do I find url to download latest version of the nuget package in the feed?
Please follow below steps to find this url.
Use this Rest API: Feed Management - Get Feeds to get the feed
id.
Use this API: Artifact Details - Get Packages to get details about the target package in the feed. Url looks like: https://feeds.dev.azure.com/{organization}/{project}/_apis/packaging/Feeds/{feedId}/packages?packageNameQuery={packageName}&api-version=6.0-preview.1. From the response, you will find its versions array, and the latest version is marked by "isLatest":true, so you get the latest version of this package in this feed.
This Rest API: NuGet - Download Package provides the url to download latest version of the nuget package in the feed.
BTW, please note that the project parameter in above APIs must be supplied if the feed was created in a project. If the feed is not associated with any project, omit the project parameter from the request.
If you build the latest before you import NuGet packages and set the project(s) up to get latest, you can make this automagic. Neither of these are really part of pipeline, except that you are ordering steps.
But, I question NuGetting a project and then including in pipeline, especially if this is one project (with dependencies) in a larger solution. If you are to the point you are deploying to NuGet, you should be more intentional and move the items in question to another solution and productize the solution rather than leave it in a master solution. You would then create a separate pipeline for the new product and consume it just like any other NuGet package. Now, you may think something like "but I use this in a variety of solutions" so this is easier, but, in reality, that is a more compelling reason to separate it out and have it intentionally placed in NuGet and not automatically get latest (i.e. act if you are buying this assembly from another company and have governance that requires you test the new version before automatically deploying with your solutions).
If you are doing this because the project(s) in question are still in flux, then I would not set the consumer to automatically pick up latest and make it intentional. Even if you currently only have a single development group, you are best to commoditize the parts going to NuGet. Otherwise, there is really no need to build a NuGet package and consume (unless I am missing some compelling reason not to productize and continue this complex manner of compiling each time and versioning).
Probably TL;DR?
For anyone who found #edward-han-msft's answer useful (as I did) here's a python 3.6+ script to download all versions of all packages from an Azure DevOps artifact(sic) feed. In my example, I was migrating to another NPM feed so this script also publishes the downloaded packages to whatever npm registry is configured in your .npmrc. Adjust it to your needs.
import requests
from requests.auth import HTTPBasicAuth
import re
import json
import os
from os.path import exists
import subprocess
organization = '<Your organisation>'
project = '<Your project name>'
feed_name = '<The name of the artifact feed>'
feed_id = '<feedId - this can be found by examining the json response of the package feed>'
# Packages feed url
url_packages = f"https://feeds.dev.azure.com/{organization}/{project}/_apis/packaging/feeds/{feed_name}/packages?api-version=5.1-preview.1"
# ADO PAT
basic = HTTPBasicAuth("<I think this can be anything>", "<a DevOps PAT with Packaging Read scope>")
# fetch the packages feed and save locally
r1 = requests.get(url_packages, auth = basic)
open('packages.json', 'wb').write(r1.content) # for debug
# parse the json
packages = json.loads(r1.content)
for package in packages['value']:
package_name = package['normalizedName']
short_name = package_name.split('/')[1]
versions_url = package['_links']['versions']['href']
print(f'Package: {package_name} ({short_name})')
# create a folder for each package
package_folder = './'+short_name
if not exists(package_folder):
os.mkdir(package_folder)
# fetch versions json
r2 = requests.get(versions_url, auth = basic)
versions = json.loads(r2.content)
open(f'{package_folder}/versions.json', 'wb').write(r2.content) # for debug
# This block iterates though the versions and discards ones that fall outside of semver e.g. x.x.x-canary or similar
version_numbers = {}
for package_version in versions['value']:
# is it a release version? (semver compliant)
version_number = package_version['normalizedVersion']
match = re.search('^\d+.\d+.\d+$', version_number)
if match:
split = version_number.split('.')
sortable_version = '%02d.%02d.%02d' % (int(split[0]),int(split[1]),int(split[2]))
version_numbers[sortable_version] = version_number
# the dictionary keys are a sortable format of the version e.g. 00.00.00
version_numbers = sorted(version_numbers.items(), key = lambda kv: kv[0])
print(version_numbers) # for debug
for package_version in version_numbers:
version_number = package_version[1] # package_version is a tuple
package_filename = f'{package_folder}/{short_name}-{version_number}.tgz'
# dowload package if not previously downloaded
if not exists(package_filename):
print(f'Downloading : {short_name}-{version_number}')
package_url = f'https://pkgs.dev.azure.com/{organization}/{project}/_apis/packaging/feeds/{feed_id}/npm/packages/{package_name}/versions/{version_number}/content?api-version=6.0-preview.1'
r3 = requests.get(package_url, allow_redirects=True, auth = basic)
open(package_filename, 'wb').write(r3.content)
# publish the package if not previsouly published
if not exists(package_filename+'.published'):
npm_publish = subprocess.run(["npm", "publish", package_filename])
if npm_publish.returncode == 0:
# create a file matching the package with .published extension to indicate successful publish
subprocess.run(["touch", package_filename+'.published'])
else:
print(f'Error publishing {short_name}-{version_number}. Code: {npm_publish.returncode}')
print('done.')
The script is idempotent as it keeps a copy of downloaded packages and also touches a <package name>.published file after a successful call to npm publish.
I'm attempting to set up my TeamCity instance as a private NuGet package feed. I have successfully configured the feed and set up a build that publishes artifacts.
However, I'm running into a problem when attempting to install a package in my source code from the TeamCity feed:
By going to the "NuGet Feed" page in TeamCity, I can see that the feed URL is
http://localhost:8080/httpAuth/app/nuget/v1/FeedService.svc/.
I have set my local Visual Studio (running on my local machine, not the TeamCity server) as
http://[my-TC-server]/httpAuth/app/nuget/v1/FeedService.svc/, and I can view the package in the NuGet manager with the expected attributes.
However, when I try to install the package, it fails because the src attribute in the package configuration is still set to http://localhost:8080/httpAuth/app/nuget/v1/FeedService.svc/download/NuGet_[myPackage]/73:id/p[myPackage].nupkg (I can see this by viewing my TeamCity-hosted packages at http://[my-TC-server]/httpAuth/app/nuget/v1/FeedService.svc/Packages()), and the install fails because it is looking on my machine for the package file, and not on the TeamCity server. I assume the src attribute is set at pack time and grabbed from the TeamCity configuration somewhere.
If I navigate to to
http://[my-TC-server]/httpAuth/app/nuget/v1/FeedService.svc/download/NuGet_[myPackage]/73:id/p[myPackage].nupkg
in a browser, I can successfully download the package.
Is it possible to configure the feed URL in TeamCity to be http://[my-TC-server]/httpAuth/app/nuget/v1/FeedService.svc/ so that the src param will also use [my-TC-server] instead of localhost? I can't find an option in TC to do this, and I haven't found any documentation for dealing with this problem.
If it is not possible to configure the feed URL, what would a good solution be so that my local machine can download the package from my TeamCity server?
It was two things:
I needed to configure my TeamCity Server URL (Administration>General Settings)
I was running into this problem with my server's reverse proxy, and running this command on the server fixed the problem (the server was not preserving the Host header in requests)
After those two, everything's all good!
I have set up a private nuget feed using a empty web application and the nuget server package. It is all working, I can retrieve from the feed, and I can publish to the private feed using the Nuget Package Explorer.
But I cant publish via the command line - it prompts for credentials. I have set the ApiKey in the web application in the Appsettings, to a simple password, and I have tried adding the ApiKey to the push command. This Prompts me for a username and then a password.
I have also tried the SetApiKey command but I get the same behavior.
On the build server I am trying the same thing with the same results.
Here is the error messages from the build Log (identifying info x'd out)
nuget pack "C:\Builds\2\OE Phase II\Common\src\WebApi\Web.Http\Bxxxxxxxs.Web.Http.csproj" -IncludeReferencedProjects -Properties Configuration=Release
nuget SetApiKey Bxxxxxxx1 -Source http://tfs12.xxxxxxxrps.com/Nuget
nuget push *.nupkg -s http://tfs12.xxxxxxxrps.com/Nuget/
C:\Builds\2\OE Phase II\Common\bin\xxxxxxx.Web.Http.dll
1 File(s) copied
Attempting to build package from 'xxxxxxx.Web.Http.csproj'.
Packing files from 'C:\Builds\2\OE Phase II\Common\src\WebApi\Web.Http\bin\Release'.
Using 'xxxxxxxs.Web.Http.nuspec' for metadata.
Found packages.config. Using packages listed as dependencies
Successfully created package 'C:\Builds\2\OE Phase II\Common\bin\xxxxxxxs.Web.Http.1.0.0.0.nupkg'.
The API Key 'xxxxxxx' was saved for 'http://tfs12.xxxxxxxrps.com/Nuget'.
Pushing Bxxxxxxxrs.Data 1.0.0.0 to 'http://tfs12.xxxxxxxrps.com/Nuget/'...
Please provide credentials for: http://tfs12.xxxxxxxrps.com/Nuget/
Object reference not set to an instance of an object.
UserName: Password:
Found solution for Windows Server 2012 and Nuget.Server 2.8.5
Open IIS manager
find your "Nuget.Server" application
right click and select "Edit permissions"
go to "Security" tab
Add "write" permission for group "Everyone".
API key is for uploading/pushing packages. Looks like you have a http proxy enabled and you can set the username and password by calling nuget.exe config command.
Some example:
nuget config -Set HTTP_PROXY=http://*.*.. -Set HTTP_PROXY.USER=domain\user
nuget.config HTTP_PROXY
http://docs.nuget.org/docs/reference/command-line-reference
This worked for me: Site > Authentication > enable Windows Authentication (I only had anonymous by default) and push will use your domain account without user input.
I had this issue too and I knew it worked before, turns out I was using NuGet.exe 2.8.1 upgrading using
NuGet.exe update -self
took it to 2.8.3 and now works fine.
Also which version of NuGet.Server package did you use? NuGet already released 2.7.2 version of it.
Can you try and see if the null ref exception still reproes? Thanks.
You may find this SO answer helpful, as it covers how to configure credentials on a workstation and/or build server, which is required to push via a command-line to a server that requires authentication (e.g. private feed.)
An API key is not required to publish.
Is there a way to load a package from an alternative server when Visual Studio Package Manager (NuGet) is responding with a "The remote server returned an error: (503) Server Unavailable" message?
This is an obscure condition that will likely only occur on an "enterprisy" network environment. If these conditions apply you:
you are required to access the Internet via an HTTP proxy server
the HTTP proxy server requires a valid user ID & password (or AD authentication) to allow requests to proceed
you've been messing with cool developer tools that were ported to Windows from a Linux/Unix environment
the new cool tool(s) work after adding the HTTP_PROXY (or possibly HTTPS_PROXY or both) environment variable(s)
you can access the NuGet servers from a browser without getting a 503 error
Then it's likely you broke NuGet by inadvertently invoking this configuration feature. I'm not sure exactly how the environment variable breaks NuGet but I suspect NuGet is detecting & using the http_proxy URL but sending an empty user ID & password which causes the HTTP proxy to reject the request.
Fix: remove the environment variable(s) you added and see if the cool tool can be configured to use an HTTP proxy without them.
Update: Ran into a version of this issue with the NuGet config file referenced in the "this configuration feature" link above. Open this file:
%appdata%\nuget\nuget.config
in your favorite editor. If it contains elements with http_proxy or https_proxy then removing these elements may fix the issue too.
PS: Hopefully I'll get an up vote from Colonel Panic :-)
If you have used the package in the past it is probably in your cache. You can add the local cache as an available package source by going into the Library Package Manager Settings under the Tools menu in Visual Studio. For Visual Studio 2012, choose Tools, Library Package Manager, Package Manager Settings, and then click on Package Sources.
In the Available package sources section, type a name like "Cache" and then in for the source, browse to %LocalAppData%\NuGet\Cache. You may need to use Windows Explorer to translate %LocalAppData%\NuGet\Cache into the full path (usually C:\Users\YourAccountName\AppData\Local\NuGet\Cache).
Once you have the Cache as an available source, you can now use the Package Manager Console (found under the View menu under Other Windows or also under the Tools menu under Library Package Manager).
From the Console (which is a PowerShell window with commandlets for NuGet) you can type "get-help NuGet" to see available commands.
Then using Get-Package, you can get a list of Package ID's. Make sure the "Package source" is set to "Cache" (or whatever you called it) and the Default project is set to the project you need manipulate, both of these are dropdowns located at the top of the Page Manager Console. You can also use the Get-Project to verify you are working against the correct project in your solution.
Finally, you can type Install-Package and when prompted enter the Package ID from the output of the Get-Package commandlet.
i had also this problem, it was becouse of my network.
if you have any blocking on your Internet, (like in companies internet or etc..)
you may not allowed to download the nuget package.
try to download the package in another network, maybe it can help you!
Talbott's answer did not work for me, as my cache was empty. However, if you have used the package in another solution, you can copy the items you want from the "packages" folder in the other solution to a packages folder in your target solution.
If you have no packages installed in the target solution, you may need to add the following to a repositories.xml file in the packages folder:
<?xml version="1.0" encoding="utf-8"?>
<repositories>
</repositories>
After doing that, the packages appeared to be installed in my solution and I was able to add them to projects.
Additional Note: I had to use the "Manage NuGet Packages for Solution" option at the solution level to add the package to individual projects. Using Install-Package from the console still returns a 503 even though the packages is already installed in the solution.
You can also get this error if you are using a VPN client (e.g. Cisco AnyConnect) and you have recently renewed your VPN certificate. The issue can occur after you have updated your certificate, but before you have rebooted. A reboot resolves the issue.
It is a pretty old question, but I have just encountered the same problem. In my case it occurred because I had more than one nuget package source configured in the Visual Studio Package Manager. In my company we use NuGet to get mainstream packages and MyGet for our own stuff.
When I attempted to pull a pretty big package it failed with a 503 code and the error link looked pretty odd, it had MyGet in it istead of NuGet. Turns out Visual Studio package manager tried to pull it from another source despite having NuGet chosen as a current source. Disabling other sources and then proceeding with a download fixed it.
Hopefully it will help somebody who stumbled upon this thread just like I did.
Another possible reason for recieving 503: If you're using Azure DevOps feed, then NuGet packages are limited to 500 MB.
I’ve set up a private NuGet Gallery from github.com/NuGet/NuGetGallery using instructions from here: http://dotnet.dzone.com/news/how-set-local-nuget-gallery
The Gallery works ok, it allows users to log in and upload packages using the Web UI as well as retrieve package updates using nuget.exe and the Visual Studio add in.
However, when I try use nuget.exe (version 1.8) to push to the Gallery like this:
nuget.exe push myPackage.4.0.0.12.nupkg -Source "http://myServer/NuGetGallery/"
I get:
Failed to process request. 'OK'.
.
With upload working ok through the Web UI I'm pretty sure that file/directory permissions must be set correctly.
Unfortunately Failed to process request isn't as useful an error message as it could be.
So I'm not sure what else could be wrong?
When using the Push command from NuGet.exe (1.8) to a Privately setup NuGet Gallery the -Source option needs to include the full path to api/v2/package, so in my case it’s: nuget.exe
push myPackage.4.0.0.12.nupkg -Source "http://myserver/NuGetGallery/api/v2/package"
I get the feeling that this could also be fixed with a rewrite rule in the web.config or perhaps a Route somewhere isn’t working, so if someone has a better solution then I’d be keen to hear it.