How to download a zip file with PowerShell from request stream - powershell

Currently, I make a POST request to a external website then I am supposed to get a zip file in return. I can get the zip file, but it comes in an xml with just the name.zip and nothing is downloaded. I have no idea why it is not downloading. My code is below on the piece where I make the actual request. I am not sure if I am over engineering this or what else I would have to do to get the actual file to download.
$url = "https://thewebsite.net/v6_1?id=$messageID"
Write-Output($url)
$Body = [byte[]][char[]]$xmlMessage
Write-Output($Body)
$Request = [System.Net.HttpWebRequest]::CreateHttp($url);
$Request.Method="POST"
$Request.ContentType = 'text/xml;charset=utf-8'
$Request.ContentLength = $Body.Length
$Request.ClientCertificates.Add($Certificate)
Write-Output($Request.ClientCertificates)
$Stream = $Request.GetRequestStream();
$Stream.Write($Body, 0, $Body.Length);
$Response = $Request.GetResponse()
$totalLength = [System.Math]::Floor($Response.get_ContentLength()/1024)
$responseStream = $Response.GetResponseStream()
$targetStream = New-Object -TypeName System.IO.FileStream -ArgumentList "D:\path\to\save\test.txt", Create
$buffer = new-object byte[] 1GB
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $count
while ($count -gt 0)
{
[System.Console]::CursorLeft = 0
[System.Console]::Write("Downloaded {0}K of {1}K", [System.Math]::Floor($downloadedBytes/1024), $totalLength)
$targetStream.Write($buffer, 0, $count)
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $downloadedBytes + $count
Write-Output($count)
}
$targetStream.Flush()
$targetStream.Close()
$targetStream.Dispose()
$responseStream.Dispose()

Unfortunately without certain download URI it's hard to clarify either you case is nontrivial or you just select non optimal way to get remote file. Routine way to get ".zip" (or any other 'octet/stream' file) with Power-Shell is execute the following command
Invoke-WebRequest -uri "https://thewebsite.net/v6_1?id=$messageID" -Method "GET" -Outfile (-join($messageID,".zip"))
then $messageID.zip file would be created in directory from which you execute Power-Shell
Progress would be shown in console window automatically. I test this example just before write the answer and it works independently on method "POST"/"GET" when remote host actually return "octet/stream" in the response. Maybe in you case file is not directly returned after requesting
thewebsite.net/v6_1?id=$messageID
but it is not a point of you original question.

Have you tried using Invoke-WebRequest?
$path = [Environment]::GetFolderPath("MyDocuments")
Invoke-WebRequest "example.com" -OutFile "$path\ZippedFile.zip"
A variable does not have to be used, as the path can be completely defined in the Invoke-WebRequest line if desired.

Related

URL health-check PowerShell script correctly gets HTTP 200 on most sites, but incorrect '0' status code on some...API timeout issue?

I have a URL health-checking PowerShell script which correctly gets an HTTP 200 status code on most of my intranet sites, but a '0' status code is returned on a small minority of them. The '0' code is an API return rather than from the web site itself, according to my research of questions from others who have written similar URL-checking PowerShell scripts. Thinking this must be a timeout issue, where API returns '0' before the slowly-responding web site returns its 200, I've researched yet more questions about this subject area on SO and implemented a suggestion from someone to insert a timeout in the script. The timeout setting though, no matter how high I set the timeout value, doesn't help. I still get the same '0' "response" code from the same web sites even though those web sites are up and running as checked from any regular web browser. Any thoughts on how I could tweak the timeout setting in the script below in order to get the correct 200 response code?
The Script:
$URLListFile = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$URLList = Get-Content $URLListFile -ErrorAction SilentlyContinue
#if((test-path $reportpath) -like $false)
#{
#new-item $reportpath -type file
#}
#For every URL in the list
$result = foreach($Uri in $URLList) {
try{
#For proxy systems
[System.Net.WebRequest]::DefaultWebProxy = [System.Net.WebRequest]::GetSystemWebProxy()
[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
#Web request
$req = [system.Net.WebRequest]::Create($uri)
$req.Timeout=5000
$res = $req.GetResponse()
}
catch {
#Err handling
$res = $_.Exception.Response
}
$req = $null
#Getting HTTP status code
$int = [int]$res.StatusCode
# output a formatted string to capture in variable $result
"$int - $uri"
#Disposing response if available
if($res){
$res.Dispose()
}
}
# output on screen
$result
#output to log file
$result | Set-Content -Path "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt" -Force
Current output:
200 - http://192.168.1.1/
200 - http://192.168.1.2/
200 - http://192.168.1.250/config/authentication_page.htm
0 - https://192.168.1.50/
200 - http://app1-vip-http.dev.local/
0 - https://CA/certsrv/Default.asp
Perhaps using PowerShell cmdlet Invoke-WebRequest works better for you. It has many more parameters and switches to play around with like ProxyUseDefaultCredentials and DisableKeepAlive
$pathIn = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$pathOut = "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt"
$URLList = Get-Content -Path $pathIn
$result = foreach ($uri in $URLList) {
try{
$res = Invoke-WebRequest -Uri $uri -UseDefaultCredentials -UseBasicParsing -Method Head -TimeoutSec 5 -ErrorAction Stop
$status = [int]$res.StatusCode
}
catch {
$status = [int]$_.Exception.Response.StatusCode.value__
}
# output a formatted string to capture in variable $result
"$status - $uri"
}
# output on screen
$result
#output to log file
$result | Set-Content -Path $pathOut -Force

Using htmlagilitypack to capture 'p class' in a website

I want to capture this 'text' shown int he picture below on a loop every minute, as this text changes every few minutes.
Here's the code I am using, and using HtmlAgilityPack.HtmlDocument
$metro = 'greatesthits'
$URL = "https://triplem.scadigital.com.au/stations/$metro/live"
[Reflection.Assembly]::LoadFile("C:\Users\makean\Downloads\htmlagilitypack.1.8.10\lib\Net45\HtmlAgilityPack.dll")
[HtmlAgilityPack.HtmlWeb]$web = #{}
[HtmlAgilityPack.HtmlDocument]$doc = $web.Load($url)
$doc.DocumentNode.SelectNodes(".//*[contains(#class,'sc-bdVaJa iHZvIS')]")
This is slimier code below, does the same thing, however just a different way of doing it
$metro = 'greatesthits'
$URL = "https://triplem.scadigital.com.au/stations/$metro/live"
Add-Type -path 'C:\Users\makean\Downloads\htmlagilitypack.1.8.10\lib\Net45\HtmlAgilityPack.dll'
$doc = New-Object HtmlAgilityPack.HtmlDocument
$wc = New-Object System.Net.WebClient
$doc.LoadHtml($wc.DownloadString($url))
$doc.DocumentNode.SelectNodes(".//*[contains(#class,'sc-bdVaJa iHZvIS')]")
This class sc-bdVaJa iHZvIS is a div and sits just a bit higher than PlayerNowPlaying__TrackInfo-kia103-1 gDXfGh and PlayerNowPlaying__TrackInfo-kia103-1 gDXfGh is what I want to capture, however when using this in my code, it returns blank.
How can I return just the text I want? Any help greatly appreciated.
I looked further at the thanks to the person above who pointed me in the right direction, checked the network option in Chrome 'inspect'. Grabbed the metadata from the stream URL.
$metro = '2classicrock'
$URL = 'https://wz2web.scahw.com.au/live/' + $metro + '_32.stream/playlist.m3u8'
$null = (Invoke-WebRequest -Uri $URL).RawContent -match '(https.*m3u8.*)'
$StreamURL = $Matches[0]
$streamMetaData = Invoke-WebRequest -Uri $StreamURL
$null = $streamMetaData.RawContent -match '#EXTINF:4.*?,(.*)'
$Matches[1]

Measure response time using Invoke-WebRequest similar to curl

I have a curl command which response time by breaking it by each action in invoking a service.
curl -w "#sample.txt" -o /dev/null someservice-call
I want to measure the response time in a similar way using PowerShell's built-in Invoke-WebRequest call. So far I am able to get total response time using Measure-Command. Can someone please help me with this?
Content of sample.txt used in curl:
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
time in milliseconds:
$url = "google.com"
(Measure-Command -Expression { $site = Invoke-WebRequest -Uri $url -UseBasicParsing }).Milliseconds
This seems to do it without any noticable overhead:
$StartTime = $(get-date)
Invoke-WebRequest -Uri "google.com" -UseBasicParsing
Write-Output ("{0}" -f ($(get-date)-$StartTime))
As the other solutions point out, there is a performance catch when using powershell only.
The most efficient solution would probably be to write some c# with the measurements built in. But when it's not properly compiled beforehand, the loading-time will increase dramatically when the C# needs to be compiled.
But there is another way.
Since you can use almost all dotnet constructs within powershell, you can just write the same request and measurement logic within powershell itself.
I have written a small method which should do the trick:
function Measure-PostRequest {
param(
[string] $Url,
[byte[]] $Bytes,
[switch] $Block
)
$content = [Net.Http.ByteArrayContent]::new($bytes);
$client = [Net.Http.HttpClient]::new();
$stopwatch = [Diagnostics.Stopwatch]::new()
$result = $null;
if ($block) {
# will block and thus not allow ctrl+c to kill the process
$stopwatch.Start()
$result = $client.PostAsync($url, $content).GetAwaiter().GetResult()
$stopwatch.Stop()
} else {
$stopwatch.Start()
$task = $client.PostAsync($url, $content)
while (-not $task.AsyncWaitHandle.WaitOne(200)) { }
$result = $task.GetAwaiter().GetResult()
$stopwatch.Stop()
}
[PSCustomObject]#{
Response = $result
Milliseconds = $stopwatch.ElapsedMilliseconds
}
}

Trying to do a simple post request in powershell v2.0, no luck

I'm simply trying to do a HTTP POST request with some keys and values. I can't get this to work for the life of me and yes I know this should be simple.
Here's what I've tried:
$Body = [byte[]][char[]]'username=asdf';
$Request = [System.Net.HttpWebRequest]::CreateHttp('http://mysite/test.php');
$Request.Method = 'POST';
$Stream = $Request.GetRequestStream();
$Stream.Write($Body, 0, $Body.Length);
$Request.GetResponse();
This doesn't work in Powershell v2.0 because I get the error
Method
invocation failed because [System.Net.HttpWebRequest] doesn't contain
a method named 'CreateHttp'.
Next, I've taken someone else's example of:
$URI1 = "http://mysite/test.php"
$request = [System.Net.WebRequest]::Create($URI1)
$request.ContentType = "application/xml"
$request.Method = "POST"
$body = "username=test"
# $request | Get-Member for a list of methods and properties
try
{
$requestStream = $request.GetRequestStream()
$streamWriter = New-Object System.IO.StreamWriter($requestStream)
$streamWriter.Write($body)
}
finally
{
if ($null -ne $streamWriter) { $streamWriter.Dispose() }
if ($null -ne $requestStream) { $requestStream.Dispose() }
}
$res = $request.GetResponse()
but for some reason "username" doesn't get noticed when test.php echos $_POST['username']
Can someone please help tell me what I'm missing here? I've been googling for hours and everything I try isn't working for some reason. Works fine on Powershell versions greater than 2.0, but not 2.0 (default in Windows 7).
Invoke-WebRequest and Invoke-RestMethod do not work on Powershell v2.0, so I'm forced to find all these annoying alternatives.
* EDIT *
I got it working after finding another HTTP POST request example:
$url = "http://mysite/test.php"
$postData = "username=test"
$buffer = [text.encoding]::ascii.getbytes($postData)
[net.httpWebRequest] $req = [net.webRequest]::create($url)
$req.method = "POST"
$req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
$req.Headers.Add("Accept-Language: en-US")
$req.Headers.Add("Accept-Encoding: gzip,deflate")
$req.Headers.Add("Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7")
$req.AllowAutoRedirect = $false
$req.ContentType = "application/x-www-form-urlencoded"
$req.ContentLength = $buffer.length
$req.TimeOut = 50000
$req.KeepAlive = $true
$req.Headers.Add("Keep-Alive: 300");
$reqst = $req.getRequestStream()
$reqst.write($buffer, 0, $buffer.length)
$reqst.flush()
$reqst.close()
[net.httpWebResponse] $res = $req.getResponse()
$resst = $res.getResponseStream()
$sr = new-object IO.StreamReader($resst)
$result = $sr.ReadToEnd()
$res.close()
This was discovered on another site before additional comments came here; however, I've tried the solutions from people's suggestions below and also was able to get this working.
In the first example code, the method should be named Create, Not CreateHttp
In the second block of code, you set the content-type to 'application/xml', but the body is plain-text.
This method, and your initial example itself, would work in a newer version of PowerShell. Try installing dotnet 4.5 on your system, then WMF 4.0, and this should work with no issue.
The dotnet class of [System.Net.HttpWebRequest] didn't have the static method CreateHttp() until dotnet 4.5, as seen here on MSDN Docs.
Why not just update this one machine to a newer version of PowerShell? It will be a LOT less pain.

ftp via powershell - how to indicate success

I'm using a Powershell script to automate sending a .txt to an FTP site. When I execute it in powershell, nothing happens. The root\prompt just appears...no messages that it was successful. How do I tell if it worked? Here is my script in case it helps.
$localfile = "D:\Export\TESTING.txt"
$remotefile = "/TESTING.txt"
$ftphost = "ftp://ftp.site.com"
$URI = $ftphost + $remotefile
$username="USERNAME"
$password="1234"
function Get-FTPFile
($URI,$localfile,$username,$password){
$credentials=New-Object System.Net.NetworkCredential
($username,$password)
$ftp=[System.Net.FtpWebRequest]::Create($URI)
$ftp.Credentials=$credentials
$ftp.UseBinary=1
$ftp.KeepAlive=0
$response=$ftp.GetResponse()
$responseStream = $response.GetResponseStream()
$file = New-Object
IO.FileStream ($localfile,[IO.FileMode]::Create)
[byte[]]$buffer = New-Object byte[] 1024
$read = 0
do{
$read=$responseStream.Read($buffer,0,1024)
$file.Write($buffer,0,$read)
}
while ($read -ne 0)$file.close()
}
Use WebRequestMethods.Ftp.GetFileSize following completion of the upload to confirm that the uploaded file size matches the local file size.
You could also use a try/catch block to check for exceptions during your read/write operations. A lack of exceptions would give you some confidence that the upload was successful (i.e. no news is good news).