Azure DevOps feed get download url for a latest version of the nuget package or symply download latest package - azure-devops

I've got the Azure DevOps pipeline which builds nuget package and deploys it to Azure DevOps feed. In the portal I can find download link to a specific version of the package.
How do I find url to download latest version of the nuget package in the feed?
Alternatively how do I download the latest version of the package?
Ideally just via curl, dotnet or anything what is resent on dev windows machine and in general docker sdk image.
I tend to go long way
dotnet new console
add package
restore
find the location of the file. But I really don't like this approach. Anything nicer?

How do I find url to download latest version of the nuget package in the feed?
Please follow below steps to find this url.
Use this Rest API: Feed Management - Get Feeds to get the feed
id.
Use this API: Artifact Details - Get Packages to get details about the target package in the feed. Url looks like: https://feeds.dev.azure.com/{organization}/{project}/_apis/packaging/Feeds/{feedId}/packages?packageNameQuery={packageName}&api-version=6.0-preview.1. From the response, you will find its versions array, and the latest version is marked by "isLatest":true, so you get the latest version of this package in this feed.
This Rest API: NuGet - Download Package provides the url to download latest version of the nuget package in the feed.
BTW, please note that the project parameter in above APIs must be supplied if the feed was created in a project. If the feed is not associated with any project, omit the project parameter from the request.

If you build the latest before you import NuGet packages and set the project(s) up to get latest, you can make this automagic. Neither of these are really part of pipeline, except that you are ordering steps.
But, I question NuGetting a project and then including in pipeline, especially if this is one project (with dependencies) in a larger solution. If you are to the point you are deploying to NuGet, you should be more intentional and move the items in question to another solution and productize the solution rather than leave it in a master solution. You would then create a separate pipeline for the new product and consume it just like any other NuGet package. Now, you may think something like "but I use this in a variety of solutions" so this is easier, but, in reality, that is a more compelling reason to separate it out and have it intentionally placed in NuGet and not automatically get latest (i.e. act if you are buying this assembly from another company and have governance that requires you test the new version before automatically deploying with your solutions).
If you are doing this because the project(s) in question are still in flux, then I would not set the consumer to automatically pick up latest and make it intentional. Even if you currently only have a single development group, you are best to commoditize the parts going to NuGet. Otherwise, there is really no need to build a NuGet package and consume (unless I am missing some compelling reason not to productize and continue this complex manner of compiling each time and versioning).
Probably TL;DR?

For anyone who found #edward-han-msft's answer useful (as I did) here's a python 3.6+ script to download all versions of all packages from an Azure DevOps artifact(sic) feed. In my example, I was migrating to another NPM feed so this script also publishes the downloaded packages to whatever npm registry is configured in your .npmrc. Adjust it to your needs.
import requests
from requests.auth import HTTPBasicAuth
import re
import json
import os
from os.path import exists
import subprocess
organization = '<Your organisation>'
project = '<Your project name>'
feed_name = '<The name of the artifact feed>'
feed_id = '<feedId - this can be found by examining the json response of the package feed>'
# Packages feed url
url_packages = f"https://feeds.dev.azure.com/{organization}/{project}/_apis/packaging/feeds/{feed_name}/packages?api-version=5.1-preview.1"
# ADO PAT
basic = HTTPBasicAuth("<I think this can be anything>", "<a DevOps PAT with Packaging Read scope>")
# fetch the packages feed and save locally
r1 = requests.get(url_packages, auth = basic)
open('packages.json', 'wb').write(r1.content) # for debug
# parse the json
packages = json.loads(r1.content)
for package in packages['value']:
package_name = package['normalizedName']
short_name = package_name.split('/')[1]
versions_url = package['_links']['versions']['href']
print(f'Package: {package_name} ({short_name})')
# create a folder for each package
package_folder = './'+short_name
if not exists(package_folder):
os.mkdir(package_folder)
# fetch versions json
r2 = requests.get(versions_url, auth = basic)
versions = json.loads(r2.content)
open(f'{package_folder}/versions.json', 'wb').write(r2.content) # for debug
# This block iterates though the versions and discards ones that fall outside of semver e.g. x.x.x-canary or similar
version_numbers = {}
for package_version in versions['value']:
# is it a release version? (semver compliant)
version_number = package_version['normalizedVersion']
match = re.search('^\d+.\d+.\d+$', version_number)
if match:
split = version_number.split('.')
sortable_version = '%02d.%02d.%02d' % (int(split[0]),int(split[1]),int(split[2]))
version_numbers[sortable_version] = version_number
# the dictionary keys are a sortable format of the version e.g. 00.00.00
version_numbers = sorted(version_numbers.items(), key = lambda kv: kv[0])
print(version_numbers) # for debug
for package_version in version_numbers:
version_number = package_version[1] # package_version is a tuple
package_filename = f'{package_folder}/{short_name}-{version_number}.tgz'
# dowload package if not previously downloaded
if not exists(package_filename):
print(f'Downloading : {short_name}-{version_number}')
package_url = f'https://pkgs.dev.azure.com/{organization}/{project}/_apis/packaging/feeds/{feed_id}/npm/packages/{package_name}/versions/{version_number}/content?api-version=6.0-preview.1'
r3 = requests.get(package_url, allow_redirects=True, auth = basic)
open(package_filename, 'wb').write(r3.content)
# publish the package if not previsouly published
if not exists(package_filename+'.published'):
npm_publish = subprocess.run(["npm", "publish", package_filename])
if npm_publish.returncode == 0:
# create a file matching the package with .published extension to indicate successful publish
subprocess.run(["touch", package_filename+'.published'])
else:
print(f'Error publishing {short_name}-{version_number}. Code: {npm_publish.returncode}')
print('done.')
The script is idempotent as it keeps a copy of downloaded packages and also touches a <package name>.published file after a successful call to npm publish.

Related

How to publish a package to private npm registry using yarn berry

I am working on a private internal package; it is a common components library used by a couple of different repositories for the company I work for. I recently migrated the repository containing the common components from yarn 1 to yarn berry (3.3.1) - there were no issues with the migration itself.
The problem I am experiencing is while publishing a new package of the library to our private npm repo. Prior to migrating, publishing was very simple:
I'd simply run yarn publish and the terminal would prompt me for my login info and to enter the package's new version (basically this: https://classic.yarnpkg.com/lang/en/docs/publishing-a-package/), and the package would be published and could be used.
Since upgrading I now run yarn npm publish and that takes whatever is in my files section of package.json and packages it up into a zip file, so in my case the following:
"files": [
"dist/*/**",
"src/assets"
],
This appears to be fine. However, when I go and install that package in another repo the contents do not match what was published. I used yarn link to verify the package was working during development, I also used yalc to test that the packaged version was working as well.
I'm well aware that this could entirely be a user error on my part. I've looked in the documentation for help with this but was not really able to find an answer:
How can I set up my project to properly package its contents and publish them to the private npm registry?
Contents of .yarnrc.yml file
npmRegistryServer: "<redacted>"
npmPublishRegistry: "<redacted>"
plugins:
- path: .yarn/plugins/#yarnpkg/plugin-constraints.cjs
spec: "#yarnpkg/plugin-constraints"
pnpMode: loose
yarnPath: .yarn/releases/yarn-3.3.1.cjs
enableStrictSsl: false

NuGet Package being marked as prerelease

I have built a NuGet Package and uploaded it to a locally hosted Sonatype Nexus repository.
I have given it the version 0.1.1+251019-020007-e3baff. My understanding of sem-ver 2.0 is this should be treated as a stable/release version (because the data after the + should only be treated as metadata), but nuget seems to be getting confused and showing it only if I include prerelease versions in the search.
For example in the cli if I run Find-package <my-package-name> I get no results. But if I run Find-Package -prerelease <my-package-name> I get
Id Versions Description
-- -------- -----------
<my-package> {0.1.1} <description>
Likewise if I use the GUI in Visual Studio I have to check the "include prerelease" option, but then the version that is available is marked as "latest stable"...
In Nexus there is a flag "is_prerelease" that is being set to true by something, not sure what, Is that flag being incorrectly set and then being used in the search?
Is there something else I am doing wrong? Is my understanding of the + character in sem-ver 2.0 not correct?
I am using NuGet version 4.9.3, and nexus is version 3.19.0-01
NuGet's logic is available as packages, the versioning logic in the NuGet.Versioning package. Using this little program:
static void Main(string[] args)
{
var version = NuGetVersion.Parse("0.1.1+251019-020007-e3baff");
Console.WriteLine($"Version is prerelease: {version.IsPrerelease}");
}
I get the output
Version is prerelease: False
I tried many versions of the NuGet.Versioning package, from the latest 5.3.1, to 4.9.4, 4.3.0, 3.5.0, 3.2.0 and even the oldest release version of the package, 1.0.1. All of them say that your version is not prerelease.
Therefore, it's not NuGet that thinks your package is prerelease. Given that - is the separator for prerelease labels, my guess is that Nexus is incorrectly doing a simple check similar to version.Contains('-') to determine if it's pre-release. This is a shame, as semver.org has two regex expressions on their website which do not have this behaviour (example, I have no idea how long this link will be valid). If your Nexus installation isn't running the latest version, I suggest trying to update if you can. If it's still a problem, you could try contacting the software vendor to report a bug.
As a work around, you could try avoid using the - character in the build metadata as long as you keep using Nexus. SemVer2 is quite restricted in the characters it lets you use, so I suggest using . instead (0.1.1+251019.020007.e3baff).

How to get latest version of Microsoft.TeamFoundation.DistributedTask.Task.Deployment.dll?

Where can I download the latest version of Microsoft.TeamFoundation.DistributedTask.Task.Deployment.dll for a release on Azure DevOps?
Microsoft-hosted agents are regularly updated to make sure the latest and applicable images of the softwares are applied to the agents.
If you want to use the latest version of softwares to run the related pipeline tasks on DevOps,you can install the latest softwares on the agents which you used .
If the agent you used is privated agent, you need to manually install or upgrade all the softwares you request on the agent machine.By downloading the latest package from Internet or using the related commands to install/upgrade the softwares.
You can refer to this case for details.
For other SO users could get clearly know more details about this issue when they are checking to this ticket, I add the following ticket which relevant with this ticket:
SO forum: Register-Environment Failing
DC forum: Where to download latest versions of deployment utilities for azure pipeline tasks?
When you using the task(such as AzureFileCopy V1.*) which need/import the Microsoft.TeamFoundation.DistributedTask.Task.Deployment.dll file, no matter whether the agent you used is hosted agent or private agent, its version are always be the latest version to the task, because these dll files are all exists depend with task.
According to the content you shared in the DC forum ticket:
I suspect that there's a newer version of these with a newer version
of Register-Environment, because with my version of this PowerShell
module, the following line fails claiming that I'm missing parameters,
specifically a "taskContext."
the error you are facing caused by the configuration of the Microsoft.TeamFoundation.DistributedTask.Task.Deployment.dll file. We did not defined the parameter taskContext in it.
To check and verify it, you can execute the PowerShellOnTargetMachines v2.* task in your private agent, then find the dll file in the path of ~\_work\_tasks\PowerShellOnTargetMachines_3b5693d4-5777-4fee-862a-bd2b7a374c68\2.0.7\DeploymentUtilities. And also, you can see its version is 16.0.0.0.
Then you can use decompile tool, here what I am using is Reflector, to decompile this dll file.
You can see that in the script block of RegisterEnvironmentCmdlet, there does not have parameter taskContext defined in it.
When you use PowerShellOnTargetMachines v2.* task, the parameter for Register-Environment supported only include EnvironmentName, EnvironmentSpecification, UserName, Password, WinRmProtocol, TestCertificate, ResourceFilter, ProjectName, TagsList.
For the parameter taskContext, it is the one which only used in v1.*(See v1.* source code) and does not supported in v2.*. In v1.*, taskContext used for Get-VssConnection cmdlet which dose not used any more in v2.*. That's why you facing the error message in PowerShellOnTargetMachines v2.* task.

How to ensure exact version of Installshield setup.exe is installed by Powershell DSC

I'm trying out Powershell DSC as a way of automating deployments. We have an EXE installer created by Installshield for a server application, and need to ensure the latest version is installed.
Installshield guidelines (http://www.flexerasoftware.com/producer/resources/white-papers/is-msipatches.html) suggest that the Package Code should change for every build, the Product Code should stay the same between minor versions, and the Upgrade Code should always stays the same.
Is there a way of telling Powershell DSC to install a particular minor version, i.e. to make sure that the Package Code matches exactly?
I'm using the following to create the MOF, but when I run it, it detects the Product as already installed and doesn't do anything, even though it's a different Package.
Package MyApp
{
Ensure = "Present"
Name = "MyApp"
Path = "\\path\to\specific\version\of\setup.exe"
ProductId = ''
Arguments = "/V`"ADDLOCAL=ALL /qb`""
}
The package resource will declare the resource as correctly configured if the package is already installed. So it will not work for your specific scenario. You will have to write a custom package resource or extend the existing one. If you want to modify feel free to fork this repository and extend the functionality https://github.com/PowerShell/xPSDesiredStateConfiguration You can also open an issue for someone to pick up and fix the same.

Installing a NuGet package from command-line from a private feed

Is it possible to fetch a NuGet package from a private feed (i.e., requiring login/password) within a script?
I have a NuGet package with a single JSON file. It's hosted on a private TeamCity feed. I want to use that JSON in a script, so here's what I'm trying to do:
nuget sources add -Name tc -Source https://xxxxxxxxx/ -Username username -Password password
nuget install xxxxxxxx -source tc
nuget sources remove -Name tc
Which results into Unable to find version '1.0.7' of package 'xxxxxxx'. Which is strange, for it's obviously able to reach the feed since it see the version of the package, but can't download the package.
I also tried to install the package into a dummy .sln and run nuget restore xxxxx.sln. NuGet is then asks me to provide credentials, two times:
Please provide credentials for: https://xxxxxxxx/httpAuth/app/nuget/v1/xxxxx.svc/
Please provide credentials for: https://xxxxxxxx/httpAuth/repository/download/xxxxx/1148145:id/xxxxxxxx.1.0.7.nupkg
Creating a NuGet.config with a feed & credentials declaration also results into the same Unable to find error.
It works OK when I'm installing/restoring the package from VS.
These two questions seems related:
NuGet Package Restore Unable to prompt for credentials with custom feed
https://stackoverflow.com/questions/24682237/nuget-command-line-authentication-not-working
It looks like it's a problem with the TC feed. I didn't think that was the case because I can use the feed from VS Package Manager, the problem is only with command-line NuGet.
I turned on -Verbosity detailed (thanks danliu) and here's what I saw:
GET https://xxxxxx/httpAuth/app/nuget/v1/FeedService.svc/Packages(Id='xxxxxx',Version='1.0.7')
GET https://xxxxxx/httpAuth/app/nuget/v1/FeedService.svc/Packages(Id='xxxxxx',Version='1.0.7.0')
System.InvalidOperationException: Unable to find version '1.0.7' of package 'xxxxxx'.
The request GET https://xxxxxx/httpAuth/app/nuget/v1/FeedService.svc/Packages(Id='xxxxxx',Version='1.0.7') returns 404, I think it's related to server configuration problem which is mentioned in TeamCity NuGet feed settings:
NuGet Feed must contain server URL inside. Current TeamCity server configuration does not let TeamCity server to get original request URL from HTTP request. It looks like TeamCity server is wrongly configured with reverse proxy. Make sure reverse proxy and TeamCity server is configured to let TeamCity server know request real request URL.
I tried to host the package on MyGet and install it from there. It worked out.