I have a set of URLs using which I need to download files which can be of any type. I tried to use below PowerShell script, but the problem with this script is that I need to mention the type of the file. e.g. in below example I had to specify the type as PDF. Is there a way I can download file of any type from a URL to a specified folder?
$Path = "D:\Downloads\test.pdf"
# below url turns into 'https://xyz.abc.com/efg/TempData/data/%7B5a8662ff-1cbf-4141-bc93-c2f65657%7D/G4068.pdf'
$Url = "http://xyz.abc.com/efg/url.asp?SessID=1904B8A0E7B358A45F5422BC751CB763666BAB2F6EE10098823DFC10BCA2051260D63DCC93C0D9E63"
$WebClient = New-Object System.Net.WebClient
Write-Host "Downloading" $Path -ForegroundColor Green
$WebClient.DownloadFile($url, $path)
You can use the response headers to learn the file type, then adjust the file as necessary.
$data = Invoke-WebRequest -Uri $Url -Method get
Switch ($data.Headers.'Content-Type'){
"*pdf*" {$Data.content | Out-File MyOutfile.pdf}
"*jpeg*" {$Data.content | Out-File MyOutfile.jpg}
}
Related
I am trying to download a file from a website with powershell and save it in a C:\temp dir with it's original name
if i try Invoke-WebRequest -Uri ("uri to file download") -Method Get with the "-OutFile" Parameter i can save the file but not with the original name.
if i save the WebRequest return value in a parameter i find the original filename under $r.Headers.'Content-Disposition'
but i don't know how to output "the file" out of this variable.
can any one help me ?
Kind regards
Florian
I just created a quick snippet that should help you. Sadly, I couldn't test it without a valid URL :(
# Vars
$url = ""
$outputDir = $PSScriptRoot
# Invoke request
$result = Invoke-WebRequest -Method GET -Uri $url
# Extract name
$contentDisposition = $result.Headers.'Content-Disposition'
$fileName = $contentDisposition.Split("=")[1].Replace("`"","")
$path = Join-Path $outputDir $fileName
# Write into file
$file = [System.IO.FileStream]::new($path, [System.IO.FileMode]::Create)
$file.write($result.Content, 0, $result.RawContentLength)
$file.close()
I uploaded a file to a cloud which gives me direct download link.
Downloading it by clicking on this link works fine, but when I try to download it via System.Net.WebClient.DownloadFile on Powershell, it downloads the file, but when I open it it says that the file is corrupted and unreadable.
That's the code:
$WebClient = New-Object System.Net.WebClient
$WebClient.DownloadFile("https://xxxxxx.com/xxxxx/xxx.exe","C:\Users\user\Desktop\xxx.exe")
Any solution?
Strange, this logic works for me.
You could try adding $WebClient.Dispose() ?
or other PowerShell download methods such as:
$uri = "https://xxxxxx.com/xxxxx/xxx.exe"
$path = "C:\Users\user\Desktop\xxx.exe"
Invoke-WebRequest -Uri $uri -OutFile $path
3 ways to download files with PowerShell
<#
# 1. Invoke-WebRequest
The first and most obvious option is the Invoke-WebRequest cmdlet. It is built
into PowerShell and can be used in the following method:
#>
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
<#
2. System.Net.WebClient
A common .NET class used for downloading files is the System.Net.WebClient class.
#>
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
# OR
(New-Object System.Net.WebClient).DownloadFile($url, $output)
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
<#
3. Start-BitsTransfer
If you haven't heard of BITS before, check this out. BITS is primarily designed
for asynchronous file downloads, but works perfectly fine synchronously too
(assuming you have BITS enabled).
#>
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $output
# Or
Start-BitsTransfer -Source $url -Destination $output -Asynchronous
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
Here is what I personally use daily, from a function in my personal module imported via my profile.
$webclient = New-Object System.Net.WebClient
$url = 'https://download.microsoft.com/download/B/A/4/BA4A7E71-2906-4B2D-A0E1-80CF16844F5F/dotNetFx45_Full_setup.exe'
$filename = [System.IO.Path]::GetFileName($url)
$file = "$TechToolsUNC\$filename"
$webclient.DownloadFile($url,$file)
Start-Process $file -Wait
I would speculate that his is not about powershell, but other factors on your machine or network, most likely antivirus agent etc.
I am trying to create function to download a file from internet where the path to the files is defined in multiple text files on different client computers.
This is what I have come up with so far.
$Company = Get-Content "C:\ProgramData\test\Company.txt"
$CompanyURLFile = "https://onegeek.dk/MSI/$Company.rar"
$CompanyUpdateFile="C:\ProgramData\test\conf\conf.rar"
Invoke-WebRequest -Uri $CompanyURLFile -OutFile $CompanyUpdateFile
The code above will fail because it cant use "$Company" in line 2
If I use this insteadt everything works fine.
$CompanyURLFile = "https://onegeek.dk/MSI/CYPL.rar"
How do I fix this
The solution was to add "| Select-Object -
First 1"
$Company = Get-Content "C:\ProgramData\test\Company.txt" | Select-Object -
First 1
$CompanyURLFile = "https://onegeek.dk/MSI/$Company.rar"
$CompanyUpdateFile="C:\ProgramData\test\conf\conf.rar"
Invoke-WebRequest -Uri $CompanyURLFile -OutFile $CompanyUpdateFile
I have an url in which there are 6 digits which are changing daily.
Sample website:
https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_160126.csv
This is the part which changes: 160126
I don't know the correct syntax but as a form of pseudo code:
$url = "https://www.ecb.europa.eu/paym/coll/assets
/html/dla/ea_MID/ea_csv_" + [0-9][0-9][0-9][0-9][0-9][0-9]+ ".csv"
How can I write this string?
To answer the comments, I use it to download that file to a folder, like this:
"https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_" + [0-9]+[0-9]+[0-9]+[0-9]+[0-9]+[0-9] +".csv"
$output = "C:\MyFolder\SomeSubFolder\ScriptDownload"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
You can parse the download page for your filenames by downloading the page or use get-ElementById etc.
I assume, this is the original download page
This is your Download URL:
$Url = "https://www.ecb.europa.eu/paym/coll/assets/html/list-MID.en.html"
$page = Invoke-WebRequest -Uri $Url
$a = ($page.ParsedHtml.getElementsByTagName('table') | ? {$_.classname -eq 'ecb-contentTable'}).textContent
$filename = $a.Substring($a.IndexOf('ea_csv_'), 17)
$DLURL = "https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/" + $filename
Gives:
$DLURL
https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_160126.csv
Complete that with your
$output = "C:\MyFolder\SomeSubFolder\ScriptDownload\" + $filename
Invoke-WebRequest -Uri $DLURL -OutFile $Output
and its done.
What you are asking for cannot be done. However there are better, more reliable ways to get the same result you are looking for.
I'm with Martin. I also found the download page he did. The better way to do this is get the link. Now this is probably not the best way to get the information but it is a start in the right direction.
Note this is slow as hell. Mostly because of Invoke-WebRequest
$start_time = Get-Date
$output = "C:\MyFolder\SomeSubFolder\ScriptDownload"
# Browse to the page hosting the csv file.
$request = Invoke-WebRequest "https://www.ecb.europa.eu/paym/coll/assets/html/list-MID.en.html"
# Locate the uncompressed CSV file name from the page
$filename = $request.ParsedHtml.getElementsByTagName("a") | Where-Object{$_.nameProp -match "^ea_csv_\d{6}\.csv$"} | Select -ExpandProperty nameProp
$fileurl = "https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/$filename"
# Get the file the is hosted today.
Invoke-WebRequest -Uri $fileurl -OutFile "$output\$filename"
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
The way we find the right file name is with ^ea_csv_\d{6}\.csv$ which matches a name where it is exactly "ea_csv_[6 digits].csv".
The 6 digits are the date encoded as YYMMDD, correct? If so, you can generate a URL for the current day with:
$currentDay = $(get-date).ToString("yyMMdd")
$url = "https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_$currentDay.csv"
Using the Test-Path PowerShell command we can check if the system drive path or file exists or not in local drive. Similarly how can we check if the SharePoint document library folder path or file in document library exists or not using Test-Path or any similar command?
Test-Path -Path "http://win-3:001/sites/Dev/Shared%20Documents/Test1"
Test-Path -Path "http://win-3:001/sites/Dev/Shared%20Documents/Test1/sample.txt"
You should be able to check that with an HTTP request:
$uri = 'http://win-3:001/sites/Dev/Shared%20Documents/Test1'
(Invoke-WebRequest -Method Head -Uri $uri -UseDefaultCredentials).StatusCode
If your PowerShell version is too old to provide the Invoke-WebRequest cmdlet you should upgrade. If for some reason you can't do that use the System.Net.WebRequest class instead:
$uri = 'http://win-3:001/sites/Dev/Shared%20Documents/Test1'
$req = [Net.WebRequest]::Create($uri)
$req.Method = 'HEAD'
$req.UseDefaultCredentials = $true
$req.PreAuthenticate = $true
$req.Credentials = [Net.CredentialCache]::DefaultCredentials
$req.GetResponse().StatusCode.value__
Either way, a status code of 200 means the request was OK, i.e. the document exists.