I have a URL to a CSV file which, in a browser, I can download and open without issue.
I'm trying to download this file using PowerShell without success. I tried using Invoke-WebRequest, Start-BitsTransfer and using a webrequest object but no luck there.
Invoke-WebRequest comes with a parameter to store its result in a file: -OutFile
Invoke-WebRequest $myDownloadUrl -OutFile c:\file.ext
If you need authorization before you can send a request like this:
Invoke-WebRequest $myAuthUrl /* whatever is neccesary to login */ -SessionVariable MySession
Invoke-WebRequest $myDownloadUrl -WebSession $MySession
To determine the layout of the form where the login happens, you can use Invoke-WebRequests return object. It'll collect information about forms and fields on the HTML (might be Windows only). Mileage of logging in may vary with things like Two-Factor-Auth active or not. Probably you can create some secret link to your file which does not need Auth or possibly google allows you to create a private access token of some sort, which can be send aus Authorization-Header alongside your request.
TLDR answers*:
Method 1, by default synchronous**
Invoke-WebRequest $url -OutFile $path_to_file
(if you get error "...Could not create SSL/TLS secure channel." see Powershell Invoke-WebRequest Fails with SSL/TLS Secure Channel)
Method 2, by default synchronous**
(New-Object System.Net.WebClient).DownloadFile($url, $path_to_file)
Method 3, asynchronous and may be much slower than the other two but is very gentle on bandwidth usage (it uses the BITS service).
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $path_to_file
Notes:
*: This answer is for those that google for "how to download a file with PowerShell".
**: Read the help pages if you want asynchronous downloading
For a while now I've been using a PS script to download PowerBI bi-monthly and using the BITS, it's been pretty solid and now so much stronger now since I removed the -Asynchronous at the end of the Start-BitsTransfer
$url = "https://download.microsoft.com/download/8/8/0/880BCA75-79DD-466A-927D-1ABF1F5454B0/PBIDesktopSetup.exe"
$output = "%RandomPath%\PowerBI Pro\PBIDesktopSetup.exe"
$start_time = Get-Date
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $output
#Commented out below because it kept creating "Tmp files"
#Start-BitsTransfer -Source $url -Destination $output -Asynchronous
Related
I'm trying to download a msi Package from a Page which does a redirection in the background which downloads the file I want.
This is the Page I want to download the msi from:
https://remotedesktopmanager.com/de/home/thankyou/rdmmsi
I tried several PowerShell Scripts, but none of them extracted the right download URL or downloaded the file.
With Invoke-Webrequest -Outfile C:xxx only the HTML is saved.
the page uses javascript with a timeout to redirect to the current setup file. that's why you cannot use Invoke-WebRequest as this uses the Internet Explorer engine under the hood which also performs this javascript window.location redirect (resulting in opening the url in the default browser).
To only get the raw HTML you have to use Invoke-RestMethod
$website = Invoke-RestMethod -Uri 'https://remotedesktopmanager.com/de/home/thankyou/rdmmsi'
The full website is now stored in the variable $website without interpreting the javascript.
To find the line with the string window.location i use Select-String which requires a file to be parsed. Thus the content of the variable is first stored in the file which is then parsed.
$tmpFilePath = 'C:\tmp\t.txt'
Out-File -FilePath $tmpFilePath -InputObject $website
$urlRedirectLine = Select-String -Path $tmpFilePath -SimpleMatch "window.location"
Remove-Item -Path $tmpFilePath
The new variable $urlRedirectLine (which content now is C:\tmp\t.txt:999: setTimeout(function () { window.location = 'https://cdn.devolutions.net/download/Setup.RemoteDesktopManager.2021.1.25.0.msi'; }, 4500);) contains the string we are looking for.
To extract that url i convert to variable to string and then use SubString() to extract the url itself. For this i look for the first ' and last ' in that variable.
$urlString = $urlRedirectLine.ToString()
$url = $urlString.Substring($urlString.IndexOf("'")+1,$urlString.LastIndexOf("'")-$urlString.IndexOf("'")-1)
Resulting in $url having the url https://cdn.devolutions.net/download/Setup.RemoteDesktopManager.2021.1.25.0.msi
To download the file you can again use Invoke-RestMethod
Invoke-RestMethod -Uri $url -OutFile 'C:\tmp\rdm.msi'
I was facing similar error while trying to download httpd.
Following worked for me (-UserAgent "NativeHost")
$ENV:HTTPD_DOWNLOAD_URL = "https://www.apachelounge.com/download/VS16/binaries/httpd-2.4.52-win64-VS16.zip"
$ENV:HTTPD_DOWNLOAD_ZIP = "httpd.zip"
Invoke-WebRequest -Uri $ENV:HTTPD_DOWNLOAD_URL -OutFile $ENV:HTTPD_DOWNLOAD_ZIP -UserAgent "NativeHost";
Reference
https://social.technet.microsoft.com/Forums/en-US/23fccc30-5f84-4a84-8160-c6e95102b11c/powershell-invokewebrequest-sourceforge-urlsredirectiondynamic-content?forum=winserverpowershell
The Link you give is of main download page which redirects after few second to the download.
The downloading link is "https://cdn.devolutions.net/download/Setup.RemoteDesktopManager.2021.1.25.0.msi"
Use the following command to download that MSI.
$url = "https://cdn.devolutions.net/download/Setup.RemoteDesktopManager.2021.1.25.0.msi"
$dest = "C:\Setup.RemoteDesktopManager.2021.1.25.0.msi"
Start-BitsTransfer -Source $url -Destination $dest
You can also use Invoke-WebRequest using the above variable
Invoke-WebRequest -Uri $url -OutFile $dest
Thank You
I am trying to perform CI/CD using Perfecto and hence I am trying to upload a file to perfecto when my Bamboo build is finished.
I was trying with the following cURL command when we have a Linux server.
curl -X POST --upload-file test.apk 'https://****.perfectomobile.com/services/repositories/media/PRIVATE:test.apk?operation=upload&user=<email>&password=<password>&overwrite=true'
Now our server is changed to Windows and hence I want a powershell script which I can use as an Inline Scripts in Bamboo.
Can you please tell me what is an equivalent script in Powershell for windows.
Many thanks in advance.
# Gather your information.
$email = "myEmail#website.com";
$password = "powershellR0cks!";
$subDomain = "****";
$url = "https://$subDomain.perfectomobile.com/services/repositories/media/PRIVATE:test.apk?operation=upload&user=$email&password=$password&overwrite=true";
$filePath = ".\test.apk";
# Make the request.
$response = Invoke-WebRequest -Uri $URL -Method Post -InFile $filePath -ContentType "application/octet-stream";
# Check for success.
if (-not ($response.StatusCode -eq 200)) {
throw "There was an error uploading the APK manifest.";
}
You may want to check the value of -ContentType, but I think that's correct. You don't necessarily need to include the scheme (HTTPS) if you don't want to, and semicolons in PowerShell are optional, but you can include them if you want.
The $response variable is an HtmlWebResponseObject that has the content of the response, the status code, and a bunch of other useful info. You can check out the available properties and methods on the object by running $response | Get-Member.
Finally, the Invoke-WebRequest cmdlet also has other parameters that may be useful to you, such as -Credential, -Headers, and more.
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-5.1
As a side-note, if you run Get-Alias -Name "curl", you can see that anytime you use curl in PowerShell, you're really just calling Invoke-WebRequest. You can use the curl alias if you want, but it's generally not a good idea to use aliases in automation since they can be modified or deleted.
I'm trying to create a new release task for VSTS which needs to download a secure file from the library. However, when I run the following PowerShell script no secure files are displayed but there are two in there. Could this be not having enough rights? What should be changed.
Another question: when I'm able to list the secure files I want to download a specific one. I haven't found any examples on how to do that. Does anyone know of an example?
$url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/distributedtask/securefiles"
Write-Host "URL: $url"
$secureFiles = Invoke-RestMethod -Uri $url -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "SecureFiles: $secureFiles"
I was able to download Secure Files using a REST API, the task's Access Token, and an Accept header for application/octet-stream. I enabled "Allow scripts to access the OAuth token". Here my task.json is using a secureFile named "SecureFile."
$secFileId = Get-VstsInput -Name SecureFile -Require
$secTicket = Get-VstsSecureFileTicket -Id $secFileId
$secName = Get-VstsSecureFileName -Id $secFileId
$tempDirectory = Get-VstsTaskVariable -Name "Agent.TempDirectory" -Require
$collectionUrl = Get-VstsTaskVariable -Name "System.TeamFoundationCollectionUri" -Require
$project = Get-VstsTaskVariable -Name "System.TeamProject" -Require
$filePath = Join-Path $tempDirectory $secName
$token= Get-VstsTaskVariable -Name "System.AccessToken" -Require
$user = Get-VstsTaskVariable -Name "Release.RequestedForId" -Require
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $User, $token)))
$headers = #{
Authorization=("Basic {0}" -f $base64AuthInfo)
Accept="application/octet-stream"
}
Invoke-RestMethod -Uri "$($collectionUrl)$project/_apis/distributedtask/securefiles/$($secFileId)?ticket=$($secTicket)&download=true&api-version=5.0-preview.1" -Headers $headers -OutFile $filePath
I am using "$(Build.QueuedById)" to get the user id in build tasks, but honestly I don't think it matters what string you use there.
If you don't have the Accept header, you'll get JSON metadata back for the file you're attempting to download.
Unfortunately I cobbled this together from other SO posts and the github issues pages; I can't find anywhere official that documents the URL I'm using there.
There has no such REST API to download secure file, but you can use Download secure file task for assistants.
And since the secure file only exist in temporary location during build, you should download the secure file by Download secure file task firstly, and copy the secure file to another directory secondly:
1. Download secure file
You can add a Download secure file task (for VSTS) and specify the filename to download.
Note: since the task is not available for TFS, you can install the similar task like Download Secure File extension for your TFS account.
2. Copy secure file to another directory
Such as copy the secure file to $(Build.ArtifactStagingDirectory), you can use PowerShell script:
Copy-Item -Path $(Agent.WorkFolder)\_temp\filename -Destination $(Build.ArtifactStagingDirectory)
Or use Copy Files task to copy the secure file to $(Build.ArtifactStagingDirectory).
BTW:
Since you are using Download Secure File task (developed by Matt Labrum) which can only select secure file from DropDownList (variables can not be used). But there has an issue Enable to use a variable to specify the secure file to download which suggests this feature, you can follow up.
And for the REST API to download secure file, it's not available for now. But there has an user voice Access "Secure files" from .NET client library, and you can vote and follow up.
Thank you in advance for anyone taking a look into this.
I'm currently trying to deploy TeamViewer via Intune that only support MSI files for deployment. However, TeamViewer has a feature called account assignment which it comes in form of an executable. Since Intune doesn't allow you deploy exe files, please correct me if I'm wrong. I have resulted in using a PowerShell script that will download the necessary files and then install.
My goal is to have the files stored in the cloud like onedrive or Dropbox. The problem there is the public link doesn't point to the file directly as its a redirect.
For example https://www.dropbox.com/x/xyzd/TeamViewer_Assignment.exe?dl=0 --> https://www.dropbox.com/x/xyzd/TeamViewer_Assignment.exe
or
https://1drv.ms/u/s!Avjfi0upMYg9haNVTMpdoPGdstex --> https://1drv.ms/u/s/teamviewer.exe
if both links were to end with the file extension (.exe), then it would be no problem. But I would like to use Teamviewer links (get.teamviewer.com/myhost redirects https://download.teamviewer.com/u/id12345/TeamViewer.exe hoping this will help a lot more people. As opposed to having a cloud storage account.
https://download.teamviewer.com/u/id12345/TeamViewer.exe is not a permanent link either, and it has an expiration time.
Things I've tried:
$url = "https://get.teamviewer.com/mycustomhost"
$output = "$PSScriptRoot\myhost.exe"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds)
second(s)"
$url = "http://get.teamviewer.com/myhost"
$output = "$PSScriptRoot\myhost.exe"
$start_time = Get-Date
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
#OR
(New-Object System.Net.WebClient).DownloadFile($url, $output)
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds)
second(s)"
$rep=Invoke-WebRequest
http://www.get.teamviewer.com/myhost -MaximumRedirection
0
$rep.Links | where {$_.innerText -eq "click here"} |select -expand href
None of those examples worked I tried other combination from bits and pieces over the net but no go.
You can use the following URI for all of your examples:
https://customdesign.teamviewer.com/download/version_12x/myhost/TeamViewerQS.exe
You can get this URI for your download in Chrome in the following way:
Download TeamViewer
Open the Download History
Right click the entry for the TeamViewer download and copy the download URI.
Edit:
You can parse the download site for the real link with the following command:
$downloadPage = Invoke-WebRequest -Uri https://get.teamviewer.com/myhost
$downloadLink = $request.ParsedHtml.getElementById('MasterBodyContent_btnRetry').href
Now you can use the '$downloadLink' variable to download the executable with any of your scripts. You may have to change this if the download page for TeamViewer changes.
Just search for the id of the 'Try again' button on the download page. Then you can edit my code to get the appropriate element and attribute.
Using PowerShell 4.0 and Invoke-RestMethod cmdlet. I'm having trouble with the -OutFile and -PassThru options. Whenever I add the -PassThru option, my -OutFile is created but the contents are Empty!
According to the Invoke-RestMethod Documentation, both an output file and pipeline object should be available when these options are used together. "-OutFile Saves the response body in the specified output file. [...] To send the results to a file and to the pipeline, use the Passthru parameter."
Here's a test to repeat the problem I'm having. Here I'm calling a rest api attempting to BOTH save response to file AND deserialize into a powershell object.
"POWERSHELL VERSION $($host.Version.ToString())"
$date = Invoke-RestMethod "http://date.jsontest.com" -OutFile "OutFile.txt" -PassThru
Get-Content "OutFile.txt"
# FILE IS EMPTY!!! PASSTHRU SEEMS TO RESULT IN EMPTY FILE
$date
# powershell object has the date received from api
Here are two tests to verify the normal functionality of Invoke-RestMethod WITHOUT the PassThru option
# ... Test # 1, call rest api and deserialize into powershell object
$date = Invoke-RestMethod "http://date.jsontest.com"
$date
# Output shows the date retrieved from sample restful service
# ... Test # 2, call rest api and save response body directly to a file
Invoke-RestMethod "http://date.jsontest.com" -OutFile "OutFile.txt"
Get-Content "OutFile.txt"
# Output shows contents of rest api response body (json text)
I think these tests should help others see the trouble I'm having. My question is whether there is something I'm missing to make this work, or whether this may be a bug with the cmdlet? I've Googled a bit for solution and no obvious reports of this issue. I'm wanting to use -OutFile as part of a workaround for another Invoke-RestMethod issue related to content encoding as described at Bug? Invoke-RestMethod and UTF-8 data. The -PassThru option is helpful for me to look at the returned data and terminate iteration on a multi-request (paged) odata result set.
I believe the -PassThru switch redirects all output to the console only, and I think that is why your file is empty. However, since you have it a variable you could add one more line like so. . .
Write-Output -InputObject $date | Out-File -FilePath "OutFile.txt"