Download multiple files from the artifactory repo using powershell - powershell

I was trying to write a powershell script which downloads multiple files from my artifactory repo. I could use some kind of logic as below by passing file names.
$files = #("test1.zip", "test.zip")
foreach($file in $files)
{
Invoke-WebRequest -Uri "$artifactory_url/$file" -OutFile "D:\download\$file"
}
But, Is there any way to download all the files with out passing names? I tried with wildcards like (*zip) but, looks like Invoke-webrequest isn't accepting the wildcards. And found no luck with Start-bittransfer cmdlet as well as described in article https://blogs.technet.microsoft.com/heyscriptingguy/2012/08/17/use-powershell-3-0-to-easily-download-60-spanned-files .
I was able to pull up list of files in the repo using below command
((Invoke-WebRequest $url).links | Where href -match "zip$").href
How can I use this command to download the files? Is there any better way to download multiple files from the artifactory repo or http endpoint? I have to perform this action on multiple servers. So, I was not looking at usage of jfrog cli.
Thanks in advance

You may be missing the credentials to be sent with the request.
If you are using an Artifactory Key you could use the WebClient object like the following -
#example Artifactory url
$artifactory_url = "https://artifactory.company.com/artifactory/"
#example Artifactory Key
$ArtifactoryKey = "AKCp2VpEfLuMVkxpmH9rSiZT3RPoWCucL8kEiq4SjbEuuuCFdNf5t5E6dom32TCE3efy2RCyg"
$wc = New-Object System.Net.WebClient
$wc.Headers.Add("X-JFrog-Art-Api", $ArtifactoryKey)
$files = #("test1.zip", "test.zip")
try {
foreach($file in $files) {
$wc.DownloadFile("$artifactory_url/$file", "D:\download\$file")
}
}
catch {
$Host.UI.WriteErrorLine("Error while Trying to download Artifacts.")
$Host.UI.WriteErrorLine($_.Exception.Message)
exit
}

Related

store Windows PowerShell output as a table in Azure SQL DB

I use the below Windows PowerShell script to extract all members of particular Yammer group and store the output as .csv file on my local machine.
What I need is to save this output as a table in Azure SQL DB. What would be the right way to approach this:
Amend the below script to save data in Azure Table?
Use some Azure SQL functionality?
Anything else?
I would appreciate your help,
$GroupId=xxxxx
$Token = "xxxx"
$Headers = #{ "Authorization" = "Bearer "+$Token }
$GroupCycle = 1
DO
{
$GetMoreGroupsUri = "https://www.yammer.com/api/v1/users/in_group/$GroupId.xml?page=$GroupCycle"
write-host ("REST API CALL : $GetMoreGroupsUri")
[xml]$Xml = ((Invoke-WebRequest -Uri $GetMoreGroupsUri -Method Get -Headers $Headers).content)
$YammerGroups += $Xml.response.users.user
$GroupCycle ++
$GroupCount += $Xml.response.users.user.count
write-host ("GROUPMEMBER COUNT : $GroupCount")
}
While ($Xml.response.users.user.count -gt 0)
$YammerGroups | Where {$_} | Export-Csv "$GroupId.csv" -Delimiter ","
I would approach this problem slightly differently. I would use Azure to store the value of the variables I needed and keep the script somewhere else that is cloud accessible, Sharepoint comes to mind, so does GIT.
Here is the documentation on the Azure DB PowerShell Module AzTable:
https://learn.microsoft.com/en-us/azure/storage/tables/table-storage-how-to-use-powershell
Here are a couple of highlights:
#Install Module
Install-Module AzTable
#Get Table Contents
Get-AzStorageTable –Context $ctx | select Name
The Module is very well defined. I encourage you to take a look at all of the different calls listed in the Microsoft Doc.
Here is a breakdown:
Use Sharepoint or Git to store your script
If you want to store values, Azure DB is great. Use the PowerShell Module AzTable to interact with it. You can write and read tables as you see fit.
If you need dynamic variables, simply adjust the code to read the table value and variablize it. Should be easy to accomplish in some of the loops.

Download folder from the Artifactory repo using Powershell

I am using the following code to download files from Jfrog Artifactory using Powershell. It's working perfectly. I now have a repository on Artifactory that has couple of recursive folders with files. So I need to download all contents of that repository. Can anyone suggest What I need to change with the following code:
#example Artifactory url
$artifactory_url = "https://artifactory.company.com/artifactory/"
#example Artifactory Key
$ArtifactoryKey = "AKCp2VpEfLuMVkxpmH9rSiZT3RPoWCucL8kEiq4SjbEuuuCFdNf5t5E6dom32TCE3efy2RCyg"
$wc = New-Object System.Net.WebClient
$wc.Headers.Add("X-JFrog-Art-Api", $ArtifactoryKey)
$files = #("test1.zip", "test.zip")
try {
foreach($file in $files) {
$wc.DownloadFile("$artifactory_url/$file", "D:\download\$file")
}
}
catch {
$Host.UI.WriteErrorLine("Error while Trying to download Artifacts.")
$Host.UI.WriteErrorLine($_.Exception.Message)
exit
}
To download the entire folder under a repository you can utilize the JFrog CLI. First, configure the Artifactory with the JFrog CLI and download the entire folder as below,
$ jfrog rt dl "my-local-repo/*.jar" all-my-frogs/

Download secure file with PowerShell

I'm trying to create a new release task for VSTS which needs to download a secure file from the library. However, when I run the following PowerShell script no secure files are displayed but there are two in there. Could this be not having enough rights? What should be changed.
Another question: when I'm able to list the secure files I want to download a specific one. I haven't found any examples on how to do that. Does anyone know of an example?
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/securefiles"
Write-Host "URL: $url"
$secureFiles = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "SecureFiles: $secureFiles"
I was able to download Secure Files using a REST API, the task's Access Token, and an Accept header for application/octet-stream. I enabled "Allow scripts to access the OAuth token". Here my task.json is using a secureFile named "SecureFile."
$secFileId = Get-VstsInput -Name SecureFile -Require
$secTicket = Get-VstsSecureFileTicket -Id $secFileId
$secName = Get-VstsSecureFileName -Id $secFileId
$tempDirectory = Get-VstsTaskVariable -Name "Agent.TempDirectory" -Require
$collectionUrl = Get-VstsTaskVariable -Name "System.TeamFoundationCollectionUri" -Require
$project = Get-VstsTaskVariable -Name "System.TeamProject" -Require
$filePath = Join-Path $tempDirectory $secName
$token= Get-VstsTaskVariable -Name "System.AccessToken" -Require
$user = Get-VstsTaskVariable -Name "Release.RequestedForId" -Require
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $User, $token)))
$headers = #{
Authorization=("Basic {0}" -f $base64AuthInfo)
Accept="application/octet-stream"
}
Invoke-RestMethod -Uri "$($collectionUrl)$project/_apis/distributedtask/securefiles/$($secFileId)?ticket=$($secTicket)&download=true&api-version=5.0-preview.1" -Headers $headers -OutFile $filePath
I am using "$(Build.QueuedById)" to get the user id in build tasks, but honestly I don't think it matters what string you use there.
If you don't have the Accept header, you'll get JSON metadata back for the file you're attempting to download.
Unfortunately I cobbled this together from other SO posts and the github issues pages; I can't find anywhere official that documents the URL I'm using there.
There has no such REST API to download secure file, but you can use Download secure file task for assistants.
And since the secure file only exist in temporary location during build, you should download the secure file by Download secure file task firstly, and copy the secure file to another directory secondly:
1. Download secure file
You can add a Download secure file task (for VSTS) and specify the filename to download.
Note: since the task is not available for TFS, you can install the similar task like Download Secure File extension for your TFS account.
2. Copy secure file to another directory
Such as copy the secure file to $(Build.ArtifactStagingDirectory), you can use PowerShell script:
Copy-Item -Path $(Agent.WorkFolder)\_temp\filename -Destination $(Build.ArtifactStagingDirectory)
Or use Copy Files task to copy the secure file to $(Build.ArtifactStagingDirectory).
BTW:
Since you are using Download Secure File task (developed by Matt Labrum) which can only select secure file from DropDownList (variables can not be used). But there has an issue Enable to use a variable to specify the secure file to download which suggests this feature, you can follow up.
And for the REST API to download secure file, it's not available for now. But there has an user voice Access "Secure files" from .NET client library, and you can vote and follow up.

Powershell download file from redirecting url - TeamViewer & Intune

Thank you in advance for anyone taking a look into this.
I'm currently trying to deploy TeamViewer via Intune that only support MSI files for deployment. However, TeamViewer has a feature called account assignment which it comes in form of an executable. Since Intune doesn't allow you deploy exe files, please correct me if I'm wrong. I have resulted in using a PowerShell script that will download the necessary files and then install.
My goal is to have the files stored in the cloud like onedrive or Dropbox. The problem there is the public link doesn't point to the file directly as its a redirect.
For example https://www.dropbox.com/x/xyzd/TeamViewer_Assignment.exe?dl=0 --> https://www.dropbox.com/x/xyzd/TeamViewer_Assignment.exe
or
https://1drv.ms/u/s!Avjfi0upMYg9haNVTMpdoPGdstex --> https://1drv.ms/u/s/teamviewer.exe
if both links were to end with the file extension (.exe), then it would be no problem. But I would like to use Teamviewer links (get.teamviewer.com/myhost redirects https://download.teamviewer.com/u/id12345/TeamViewer.exe hoping this will help a lot more people. As opposed to having a cloud storage account.
https://download.teamviewer.com/u/id12345/TeamViewer.exe is not a permanent link either, and it has an expiration time.
Things I've tried:
$url = "https://get.teamviewer.com/mycustomhost"
$output = "$PSScriptRoot\myhost.exe"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds)
second(s)"
$url = "http://get.teamviewer.com/myhost"
$output = "$PSScriptRoot\myhost.exe"
$start_time = Get-Date
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
#OR
(New-Object System.Net.WebClient).DownloadFile($url, $output)
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds)
second(s)"
$rep=Invoke-WebRequest
http://www.get.teamviewer.com/myhost -MaximumRedirection
0
$rep.Links | where {$_.innerText -eq "click here"} |select -expand href
None of those examples worked I tried other combination from bits and pieces over the net but no go.
You can use the following URI for all of your examples:
https://customdesign.teamviewer.com/download/version_12x/myhost/TeamViewerQS.exe
You can get this URI for your download in Chrome in the following way:
Download TeamViewer
Open the Download History
Right click the entry for the TeamViewer download and copy the download URI.
Edit:
You can parse the download site for the real link with the following command:
$downloadPage = Invoke-WebRequest -Uri https://get.teamviewer.com/myhost
$downloadLink = $request.ParsedHtml.getElementById('MasterBodyContent_btnRetry').href
Now you can use the '$downloadLink' variable to download the executable with any of your scripts. You may have to change this if the download page for TeamViewer changes.
Just search for the id of the 'Try again' button on the download page. Then you can edit my code to get the appropriate element and attribute.

Trying to download a zip file from a weblink with powershell

Ok, I am trying to download a file off of a web link that we use with powershell. I am downloading a zip file where the begining of the name is always the same, but the the middle part will change based off of the version number of the zip. I have been able to get the file to download when I use the fully qualified web address and have the file name hard coded into the script. I have tried every version of using the wild cards to get all the most common version of the zip, but it errors out saying that it can't find the file on there server. This is the code that I have already, and any help would be greatly appreciated since I feel like I am at a wall with it.
$url = 'http://blah/blah/blah/My File Name 11.1111.11.zip'
$localFileName = 'C:\temp\MYzip.zip'
Invoke-WebRequest $url -UseDefaultCredentials -OutFile $localFileName
If the site has directory browsing enabled (unlikely unless you have control of the site and can turn it on), you can do this:
$url = 'http://blah/blah/blah/'
$wr = iwr $url
$filename = $wr.Links.href | Where {$_ -match 'My File Name.*?\.zip'}
$wr = iwr "$url/$filename"
If the site doesn't have directory browsing enabled then surely it has a page with a link to the ZIP file on it. Download that page and use the same $wr.Links.href trick to get all the links and look for the one that matches "My File Name.*?.zip".