I have an url in which there are 6 digits which are changing daily.
Sample website:
https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_160126.csv
This is the part which changes: 160126
I don't know the correct syntax but as a form of pseudo code:
$url = "https://www.ecb.europa.eu/paym/coll/assets
/html/dla/ea_MID/ea_csv_" + [0-9][0-9][0-9][0-9][0-9][0-9]+ ".csv"
How can I write this string?
To answer the comments, I use it to download that file to a folder, like this:
"https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_" + [0-9]+[0-9]+[0-9]+[0-9]+[0-9]+[0-9] +".csv"
$output = "C:\MyFolder\SomeSubFolder\ScriptDownload"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
You can parse the download page for your filenames by downloading the page or use get-ElementById etc.
I assume, this is the original download page
This is your Download URL:
$Url = "https://www.ecb.europa.eu/paym/coll/assets/html/list-MID.en.html"
$page = Invoke-WebRequest -Uri $Url
$a = ($page.ParsedHtml.getElementsByTagName('table') | ? {$_.classname -eq 'ecb-contentTable'}).textContent
$filename = $a.Substring($a.IndexOf('ea_csv_'), 17)
$DLURL = "https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/" + $filename
Gives:
$DLURL
https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_160126.csv
Complete that with your
$output = "C:\MyFolder\SomeSubFolder\ScriptDownload\" + $filename
Invoke-WebRequest -Uri $DLURL -OutFile $Output
and its done.
What you are asking for cannot be done. However there are better, more reliable ways to get the same result you are looking for.
I'm with Martin. I also found the download page he did. The better way to do this is get the link. Now this is probably not the best way to get the information but it is a start in the right direction.
Note this is slow as hell. Mostly because of Invoke-WebRequest
$start_time = Get-Date
$output = "C:\MyFolder\SomeSubFolder\ScriptDownload"
# Browse to the page hosting the csv file.
$request = Invoke-WebRequest "https://www.ecb.europa.eu/paym/coll/assets/html/list-MID.en.html"
# Locate the uncompressed CSV file name from the page
$filename = $request.ParsedHtml.getElementsByTagName("a") | Where-Object{$_.nameProp -match "^ea_csv_\d{6}\.csv$"} | Select -ExpandProperty nameProp
$fileurl = "https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/$filename"
# Get the file the is hosted today.
Invoke-WebRequest -Uri $fileurl -OutFile "$output\$filename"
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
The way we find the right file name is with ^ea_csv_\d{6}\.csv$ which matches a name where it is exactly "ea_csv_[6 digits].csv".
The 6 digits are the date encoded as YYMMDD, correct? If so, you can generate a URL for the current day with:
$currentDay = $(get-date).ToString("yyMMdd")
$url = "https://www.ecb.europa.eu/paym/coll/assets/html/dla/ea_MID/ea_csv_$currentDay.csv"
Related
How do you search Azure Devops Code Repos w/ Powershell using Devops REST API and output to CSV.
Couldn't find a solution for this but found enough inspiration around the internet to put this together from this article written by Tenbulls
MS Documentation on this subject can be found here
Instead of exhaustively going through the step by step process of putting this together, I just want to highlight the important hurdles that I overcame.
When the documentation says to use almsearch.dev.azure.com, that's what it means. Don't do what I did and think, oh that's probably incorrect and use dev.azure.com.
Giving your PAT (personal access token) the appropriate amount of rights is important. I, out of laziness and a propensity for anarchy chose to give my token full access after failing to give it the right amount of read access.
Here's the entire script in all of it's glory.
#set your desired org, search and PAT variables
$orgname = "YourOrgName"
$textsearch = 'YourTextString'
$personalToken = "YourPersonalAccessTokenGeneratedInDevOps"
#set your desired file and path properties
$folderpath = "C:\OutBound\"
$file = "OutputSearchResults.csv"
$FileName = $folderpath + $file
#force create directory
New-Item -ItemType Directory -Force -Path $folderpath
#initial test to remove csv file if it already exists
if (Test-Path $FileName) {
Remove-Item $FileName
}
#converting textsearch to json format for later use
$body =
#{
searchText = $textsearch
'$top' = 300
}
$json = $body | ConvertTo-Json
#Setting token and header for Get and Post requests
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($personalToken)"))
$header = #{authorization = "Basic $token"}
#outer query getting a list of all projects in the azure devops organization
$url = "https://dev.azure.com/" + $orgname + "/_apis/projects?api-version=5.1"
$output = Invoke-RestMethod -Uri $url -Method Get -ContentType "application/json" -Headers $header
$output.value | ForEach-Object {
#inner query that searches repositories for the string entered in $textsearch
#-Body $json is the where the magic happens. Many additional request properties to choose from in MS documentation
$project = $_.name
$urlsearch = "https://almsearch.dev.azure.com/" + $orgname + "/" + $project + "/_apis/search/codesearchresults?api-version=6.1-preview.1"
$output = Invoke-RestMethod -Uri $urlsearch -Method Post -ContentType "application/json" -Headers $header -Body $json
$output.results |Export-Csv -Path $FileName -Append
}
I am trying to download a file from a website with powershell and save it in a C:\temp dir with it's original name
if i try Invoke-WebRequest -Uri ("uri to file download") -Method Get with the "-OutFile" Parameter i can save the file but not with the original name.
if i save the WebRequest return value in a parameter i find the original filename under $r.Headers.'Content-Disposition'
but i don't know how to output "the file" out of this variable.
can any one help me ?
Kind regards
Florian
I just created a quick snippet that should help you. Sadly, I couldn't test it without a valid URL :(
# Vars
$url = ""
$outputDir = $PSScriptRoot
# Invoke request
$result = Invoke-WebRequest -Method GET -Uri $url
# Extract name
$contentDisposition = $result.Headers.'Content-Disposition'
$fileName = $contentDisposition.Split("=")[1].Replace("`"","")
$path = Join-Path $outputDir $fileName
# Write into file
$file = [System.IO.FileStream]::new($path, [System.IO.FileMode]::Create)
$file.write($result.Content, 0, $result.RawContentLength)
$file.close()
I uploaded a file to a cloud which gives me direct download link.
Downloading it by clicking on this link works fine, but when I try to download it via System.Net.WebClient.DownloadFile on Powershell, it downloads the file, but when I open it it says that the file is corrupted and unreadable.
That's the code:
$WebClient = New-Object System.Net.WebClient
$WebClient.DownloadFile("https://xxxxxx.com/xxxxx/xxx.exe","C:\Users\user\Desktop\xxx.exe")
Any solution?
Strange, this logic works for me.
You could try adding $WebClient.Dispose() ?
or other PowerShell download methods such as:
$uri = "https://xxxxxx.com/xxxxx/xxx.exe"
$path = "C:\Users\user\Desktop\xxx.exe"
Invoke-WebRequest -Uri $uri -OutFile $path
3 ways to download files with PowerShell
<#
# 1. Invoke-WebRequest
The first and most obvious option is the Invoke-WebRequest cmdlet. It is built
into PowerShell and can be used in the following method:
#>
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
<#
2. System.Net.WebClient
A common .NET class used for downloading files is the System.Net.WebClient class.
#>
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
# OR
(New-Object System.Net.WebClient).DownloadFile($url, $output)
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
<#
3. Start-BitsTransfer
If you haven't heard of BITS before, check this out. BITS is primarily designed
for asynchronous file downloads, but works perfectly fine synchronously too
(assuming you have BITS enabled).
#>
$url = "http://mirror.internode.on.net/pub/test/10meg.test"
$output = "$PSScriptRoot\10meg.test"
$start_time = Get-Date
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $output
# Or
Start-BitsTransfer -Source $url -Destination $output -Asynchronous
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
Here is what I personally use daily, from a function in my personal module imported via my profile.
$webclient = New-Object System.Net.WebClient
$url = 'https://download.microsoft.com/download/B/A/4/BA4A7E71-2906-4B2D-A0E1-80CF16844F5F/dotNetFx45_Full_setup.exe'
$filename = [System.IO.Path]::GetFileName($url)
$file = "$TechToolsUNC\$filename"
$webclient.DownloadFile($url,$file)
Start-Process $file -Wait
I would speculate that his is not about powershell, but other factors on your machine or network, most likely antivirus agent etc.
I have a set of URLs using which I need to download files which can be of any type. I tried to use below PowerShell script, but the problem with this script is that I need to mention the type of the file. e.g. in below example I had to specify the type as PDF. Is there a way I can download file of any type from a URL to a specified folder?
$Path = "D:\Downloads\test.pdf"
# below url turns into 'https://xyz.abc.com/efg/TempData/data/%7B5a8662ff-1cbf-4141-bc93-c2f65657%7D/G4068.pdf'
$Url = "http://xyz.abc.com/efg/url.asp?SessID=1904B8A0E7B358A45F5422BC751CB763666BAB2F6EE10098823DFC10BCA2051260D63DCC93C0D9E63"
$WebClient = New-Object System.Net.WebClient
Write-Host "Downloading" $Path -ForegroundColor Green
$WebClient.DownloadFile($url, $path)
You can use the response headers to learn the file type, then adjust the file as necessary.
$data = Invoke-WebRequest -Uri $Url -Method get
Switch ($data.Headers.'Content-Type'){
"*pdf*" {$Data.content | Out-File MyOutfile.pdf}
"*jpeg*" {$Data.content | Out-File MyOutfile.jpg}
}
I am trying to create function to download a file from internet where the path to the files is defined in multiple text files on different client computers.
This is what I have come up with so far.
$Company = Get-Content "C:\ProgramData\test\Company.txt"
$CompanyURLFile = "https://onegeek.dk/MSI/$Company.rar"
$CompanyUpdateFile="C:\ProgramData\test\conf\conf.rar"
Invoke-WebRequest -Uri $CompanyURLFile -OutFile $CompanyUpdateFile
The code above will fail because it cant use "$Company" in line 2
If I use this insteadt everything works fine.
$CompanyURLFile = "https://onegeek.dk/MSI/CYPL.rar"
How do I fix this
The solution was to add "| Select-Object -
First 1"
$Company = Get-Content "C:\ProgramData\test\Company.txt" | Select-Object -
First 1
$CompanyURLFile = "https://onegeek.dk/MSI/$Company.rar"
$CompanyUpdateFile="C:\ProgramData\test\conf\conf.rar"
Invoke-WebRequest -Uri $CompanyURLFile -OutFile $CompanyUpdateFile