How to get the list of all artifacts from Azure feed.
Below command is working fine, but getting only first 1000 artifacts.
#ReposList = `curl -u username\#abc.com:key -X GET "https://feeds.dev.azure.com/abc/ProjectName/_apis/packaging/Feeds/FeedName/packages?api-version=5.1-preview.1"`;
But, how we can get rest of the artifacts as well ?
Can some one help on this !
Take a look at the documentation for Artifact Details - Get Packages: https://learn.microsoft.com/en-us/rest/api/azure/devops/artifacts/artifact-details/get-packages?view=azure-devops-rest-6.0.
There are a couple query parameters you can try passing to get all the artifacts.
$top is used in order to:
Get the top N packages (or package versions where getTopPackageVersions=true)
You can try setting this to an arbitrarily high number and see if that returns the total number of artifacts you expected.
There is also includeAllVersions which works in the following way:
True to return all versions of the package in the response. Default is false (latest version only).
I suggest using one (or both) and see if that helps resolve your situation.
Something like: curl -u username\#abc.com:key -X GET "https://feeds.dev.azure.com/abc/ProjectName/_apis/packaging/Feeds/FeedName/packages&$top=5000&includeAllVersions=true?api-version=5.1-preview.1"
For powershell,it seems like:
$UriFeedPackageslist = "https://feeds.dev.azure.com/$orgName/$projectName/_apis/packaging/Feeds/$feed" + '/packages?$top=30000&api-version=7.0'
Related
I have this structure of projects (folders) in git repository:
/src
/src/Sample.Backend.Common
/src/Sample.Backend.Common.Tests
/src/Sample.Backend.Common.Domain
/src/Sample.Backend.Common.Domain.Tests
/src/Sample.Backend.Pricing.Abstractions
/src/Sample.Backend.Pricing.Domain
/src/Sample.Backend.Pricing.Domain.Tests
/src/Sample.Backend.Pricing.Persistence
/src/Sample.Backend.Pricing.Persistence.Tests
/src/Sample.Backend.Accounting.Abstractions
/src/Sample.Backend.Accounting.Domain
/src/Sample.Backend.Accounting.Domain.Tests
/src/Sample.Backend.Accounting.Persistence
/src/Sample.Backend.Accounting.Persistence.Tests
/src/Sample.Backend.Api
/src/Sample.Common
/src/Sample.Frontend.Common
/src/Sample.Frontend.Web
/src/Sample.Tests.Common
(The sample is simplified, in real there are much more projects/folders.)
I want different pipelines for different parts. For example a pipeline to be triggered whenever any file is commited in master branch in any Backend project. Something like this:
trigger:
branches:
include:
- master
paths:
include:
- src/Sample.Backend.*
- src/Sample.Common
- src/Sample.Tests.Common
The problem is, that filter src/Sample.Backend.* is not working. I have to add exact name of each Backend folder to get it working. I could use exclude but I have the same problem - there are many other projects and I would have to name them all.
I found that wildcards are not supported: https://github.com/MicrosoftDocs/azure-devops-docs/issues/397#issuecomment-422958966
Is there any other way to achieve the same result?
Does Azure YAML pipelne support wildcards in path filter in trigger?
This is a known request on our main forum for product:
Support wildcards (*) in Trigger > Path Filters
This feature has not yet been implemented; you could add your comment and vote this on user voice.
As a workaround for us, we add an inline PowerShell task as the first task to execute the git command line git diff HEAD HEAD~ --name-only then get the modified file names and filter the files name in the latest submit, and use Logging Command to sets variables which are then referenced in custom conditions in the next steps in the build pipeline:
and(succeeded(), eq(variables['CustomVar'], 'True'))
Our inline PowerShell script:
cd $(System.DefaultWorkingDirectory)
$editedFiles = git diff HEAD HEAD~ --name-only
echo "$($editedFiles.Length) files modified:"
$editedFiles | ForEach-Object {
echo $_
Switch -Wildcard ($_ ) {
'XXXX/Src/Sample.Backend.*' {
Write-Host ("##vso[task.setvariable variable=CustomVar]True")
}
'XXXX/Src/Sample.Common*' {
Write-Host ("##vso[task.setvariable variable=CustomVar]True")}
'XXXX/Src/Sample.Tests.Common' {
Write-Host ("##vso[task.setvariable variable=CustomVar]True")}
}
}
Then add the condition for all remaining tasks:
In this case, if the changed files do not meet our filters, then all remaining tasks will be skipped.
UPDATE: 09/09/2021
This is possible now as it is written here
Wild cards can be used when specifying inclusion and exclusion branches for CI or PR triggers in a pipeline YAML file. However, they cannot be used when specifying path filters. For instance, you cannot include all paths that match src/app//myapp*. This has been pointed out as an inconvenience by several customers. This update fills this gap. Now, you can use wild card characters (, *, or ?) when specifying path filters.
Note: documentation seems to be not updated yet.
Old answer:
No this is not possible at the moment. You have even feature request here and I would recommend to upvote it. (I already did this) Rick in above mentioned topic shared his idea how to overcome the issue:
I currently achieve this by having 3 files:
azure-pipelines.yml ( This calls some python on each commit )
azure-pipelines.py (This checks for changed folders and has some parameters to ignore certain folders, then calls the API directly)
azure-pipelines-trigger.yml ( This is called by the python based on the changed folders )
It works well enough, but it is unfortunate for the need to go through these loops.
But it needs an extra work.
This feature will roll out over the next two to three weeks according to the latest release notes
Update on this.
It took a few weeks but the change mentioned by pavlo in the comments above finally got rolled out and path triggers are now supported in YAML.
So I see the Api to get the build by its build Id, but I cannot see one to get it by the build number. Is the only way to list all the builds?
You can use an optional parameter on the builds - List API
https://dev.azure.com/{organization}/{project}/_apis/build/builds?api-version=4.0&buildNumber=0.0.1836.3
There is no API for getting a build according to the build number because theoretically, it could be few builds with the same build number, so if you want to retrieve the build according to the build number you need to get all the builds with this Rest API and then filter the results.
For example (with PowerShell):
# Get the results with Invoke-RestMethod
$builds = Invoke-RestMethod ...
$buildNumber = "10.7"
$myBuild = $builds.value.Where({ $_.buildNumber -eq $buildNumber})
this is the call you are looking for:
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}?api-version=5.0
edit: NVM, I misread the question. there is no way to get the build by its build number.
I want to search for a filename pattern across entire JFrog ARM
without knowing the explicit repository name in the JFrog cli.
jfrog rt s "reponame/*pattern*"
is giving the results as expected in a specific repo.
But I have repo1, repo2, repo3, ... so on.
How do I search using wildcard for reponame, below is not working.
jfrog rt s "*/*pattern*"
Basically I want the jfrog cli equlivalent of the curl GET request search
"https://server/artifactory/api/search/artifact?name=*pattern*"
This is not for cli client, but an alternative way to get desired feature. Spent some time looking at API here:
https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API
I recommend to scroll down that page slowly and read in entirety as a lof of possible commands, syntax is excellent, I executed a few searches and they searched all local repositories. No need to recursively search 1 by 1. Command syntax:
export url="http://url/to/articatory"
curl --noproxy '*' -x GET "$url/api/search/artifact?name=log4j*"
Read link above for more granular search options/syntax.
How I set it up:
alias artpost='curl -X POST "http://url/artifactory/api/search/aql" -T - -u admin:password'
Some example usage:
echo 'items.find({"name": {"$match" : "log4j*"}})' | artpost
echo 'items.find({"$and" : [{"created" : {"$gt" : "2017-06-12"}},{"name": {"$nmatch" : "*surefire*"}}]})' | artpost
I'd like to have "Download Latest Version" button on my website which would represent the link to the latest release (stored at GitHub Releases). I tried to create release tag named "latest", but it became complicated when I tried to load new release (confusion with tag creation date, tag interchanging, etc.). Updating download links on my website manually is also a time-consuming and scrupulous task. I see the only way - redirect all download buttons to some html, which in turn will redirect to the actual latest release.
Note that my website is hosted at GitHub Pages (static hosting), so I simply can't use server-side scripting to generate links. Any ideas?
You don't need any scripting to generate a download link for the latest release. Simply use this format:
https://github.com/:owner/:repo/zipball/:branch
Examples:
https://github.com/webix-hub/tracker/zipball/master
https://github.com/iDoRecall/selection-menu/zipball/gh-pages
If for some reason you want to obtain a link to the latest release download, including its version number, you can obtain that from the get latest release API:
GET /repos/:owner/:repo/releases/latest
Example:
$.get('https://api.github.com/repos/idorecall/selection-menu/releases/latest', function (data) {
$('#result').attr('href', data.zipball_url);
});
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<a id="result">Download latest release (.ZIP)</a>
Github now provides a "Latest release" button on the release page of a project, after you have created your first release.
In the example you gave, this button links to https://github.com/reactiveui/ReactiveUI/releases/latest
You can use the following where:
${Organization} as the GitHub user or organization
${Repository} is the repository name
curl -L https://api.github.com/repos/${Organization}/${Repository}/tarball > ${Repository}.tar.gz
The top level directory in the .tar.gz file has the sha hash of the commit in the directory name which can be a problem if you need an automated way to change into the resulting directory and do something.
The method below will strip this out, and leave the files in a folder with a predictable name.
mkdir ${Repository}
curl -L https://api.github.com/repos/${Organization}/${Repository}/tarball | tar -zxv -C ${Repository} --strip-components=1
Since February 18th, 2015, the GitHUb V3 release API has a get latest release API.
GET /repos/:owner/:repo/releases/latest
See also "Linking to releases".
Still, the name of the asset can be tricky.
Git-for-Windows, for instance, requires a command like:
curl -IkLs -o NUL -w %{url_effective} \
https://github.com/git-for-windows/git/releases/latest|\
grep -o "[^/]*$"| sed "s/v//g"|\
xargs -I T echo \
https://github.com/git-for-windows/git/releases/download/vT/PortableGit-T-64-bit.7z.exe \
-o PortableGit-T-64-bit.7z.exe| \
sed "s/.windows.1-64/-64/g"|sed "s/.windows.\(.\)-64/.\1-64/g"|\
xargs curl -kL
The first 3 lines extract the latest version 2.35.1.windows.2
The rest will build the right URL
https://github.com/git-for-windows/git/releases/download/
v2.35.1.windows.2/PortableGit-2.35.1.2-64-bit.7z.exe
^^^^^^^^^^^^^^^^^ ^^^^^^^^^
Maybe could you use some client-side scripting and dynamically generate the target of the link by invoking the GitHub api, through some JQuery magic?
The Releases API exposes a way to retrieve the list of all the releases from a repository. For instance, this link return a Json formatted list of all the releases of the ReactiveUI project.
Extracting the first one would return the latest release.
Within this payload:
The html_url attribute will hold the first part of the url to build (ie. https://github.com/{owner}/{repository}/releases/{version}).
The assets array will list of the downloadable archives. Each asset will bear a name attribute
Building the target download url is only a few string operations away.
Insert the download/ keyword between the releases/ segment from the html_url and the version number
Append the name of the asset to download
Resulting url will be of the following format: https://github.com/{owner}/{repository}/releases/download/{version}/name_of_asset
For instance, regarding the Json payload from the link ReactiveUI link above, we've got html_url: "https://github.com/reactiveui/ReactiveUI/releases/5.99.0" and one asset with name: "ReactiveUI.6.0.Preview.1.zip".
As such, the download url is https://github.com/reactiveui/ReactiveUI/releases/download/5.99.0/ReactiveUI.6.0.Preview.1.zip
If you using PHP try follow code:
function getLatestTagUrl($repository, $default = 'master') {
$file = #json_decode(#file_get_contents("https://api.github.com/repos/$repository/tags", false,
stream_context_create(['http' => ['header' => "User-Agent: Vestibulum\r\n"]])
));
return sprintf("https://github.com/$repository/archive/%s.zip", $file ? reset($file)->name : $default);
}
Function usage example
echo 'Download';
As I didn't see the answer here, but it was quite helpful for me while running continuous integration tests, this one-liner that only requires you to have curl will allow to search the Github repo's releases to download the latest version
https://gist.github.com/steinwaywhw/a4cd19cda655b8249d908261a62687f8
I use it to run PHPSTan on our repository using the following script
https://gist.github.com/rvanlaak/7491f2c4f0c456a93f90e31774300b62
If you are trying to download form any linux — even old or tiny versions — or are trying to download from a bash script then the failproof way is using this command:
wget https://api.github.com/repos/$OWNER/$REPO/releases/latest -O - | awk -F \" -v RS="," '/browser_download_url/ {print $(NF-1)}' | xargs wget
do not forget to replace $OWNER and $REPO with the right owner and repository names. The command downloads a json page with the data of the latest release. then awk gets the value from the browser_download_url key.
If you are in a really old linux or a tiny embedded system with a small wget, the download name can be a problem. In such case you can always use the ultra-reliable:
URL=$(wget https://api.github.com/repos/$OWNER/$REPO/releases/latest -O - | awk -F \" -v RS="," '/browser_download_url/ {print $(NF-1)}'); wget $URL -O $(basename "$URL")
As noted by #Dan Dascalescu in a comment to accepted answer, there are some projects (roughly 30%) which do not bother to file formal releases, so neither "Latest release" button nor /releases/latest API call would return useful data.
To reliably fetch the latest release for a GitHub project, you can use lastversion.
All I'm looking for is a way to get a list of issues for a given milestone. It looks like Github treats milestones a bit like labels in that you can ask for the labels for an issue, but not the issues for a given label.
I know that I can filter my issues by milestone on the Github website, but this traverses multiple pages and I wanted an easy way to see all of the issues for a milestone in a more printer friendly version.
Any tips?
You could use GitHub's API for this. See here on how to get the list of issues for a repo and notice the milestone parameter. The response you will get is a big JSON document, so you would have to create a small script to pull only the titles of the issues, or use grep, or smething like jq.
Notice also that API responses are also paged, but you can set the paging to be 100 entries per page, which is usually enough. If not, you would again have to create a small script to fetch all the pages (or do it manually).
You can use the GraphQL API which is V4. and do something like:
{
repository(owner: "X", name: "X") {
milestone(number: X) {
id
issues(first: 100) {
edges {
node {
id,
title
}
}
}
}
}
}
I was not able to find any easy methods. This worked a treat for me:
brew install hub (on OSX). Hub is created by GitHub
cd to the local repo you want to access the origin for.
hub issue -M 21 -f "%I,%t,%L,%b,%au,%as" > save_here.csv
profit.
Find the issue # (21 in the example above) in the URL on GitHub when you are viewing the milestone.
Docs for hub and in particular the format (-f) flag can be found here: https://hub.github.com/hub-issue.1.html
First find the list of milestones using this
Then query this api by milestone number for each milestone
Given a milestone $title in $owner/$repo, we can list the issues in this milestone using curl and jq:
api_url="https://api.github.com/repos/$owner/$repo"
MS=$(curl -s "$api_url/milestones" | jq '.[] | select(.title == "QA")')
MS_number=$(echo "$MS" | jq .number -r)
MS_state=$(echo "$MS" | jq .state -r)
echo "Found $title milestone with state=$MS_state"
echo ""
issues=$(curl -s "$api_url/issues?milestone=$MS_number" | jq '.[].number' -r)
echo "The following issues are in the QA milestone:"
for i in $issues; do
issue_title=$(curl -s "$api_url/issues/$i" | jq '.title' -r)
echo " https://github/$owner/$repo/issues/$i - $issue_title"
done
echo ""