Get response link from HTTP request on Powershell - powershell

The following Powershell script runs a Google search of an image stored within my hard drive.
How can I get the link which is followed to get to the results page? Is it possible to navigate to the different webpages displayed on it?
I've tried $request.Links | Select href to try and get a list of the links, but it didn't work. I've also tried to add Write-Output $respStream to the code, but then it doesn't run.
Set-ExecutionPolicy Bypass -scope Process -Force
function Get-GoogleImageSearchUrl
{
param(
[Parameter(Mandatory = $true)]
[ValidateScript({ Test-Path $_ })]
[string] $ImagePath
)
# extract the image file name, without path
$fileName = Split-Path $imagePath -Leaf
# the request body has some boilerplate before the raw image bytes (part1) and some after (part2)
# note that $filename is included in part1
$part1 = #"
-----------------------------7dd2db3297c2202
Content-Disposition: form-data; name="encoded_image"; filename="$fileName"
Content-Type: image/jpeg
"#
$part2 = #"
-----------------------------7dd2db3297c2202
Content-Disposition: form-data; name="image_content"
-----------------------------7dd2db3297c2202--
"#
# grab the raw bytes composing the image file
$imageBytes = [Io.File]::ReadAllBytes($imagePath)
# the request body should sandwich the image bytes between the 2 boilerplate blocks
$encoding = New-Object Text.ASCIIEncoding
$data = $encoding.GetBytes($part1) + $imageBytes + $encoding.GetBytes($part2)
# create the HTTP request, populate headers
$request = [Net.HttpWebRequest] ([Net.HttpWebRequest]::Create('http://images.google.com/searchbyimage/upload'))
$request.Method = "POST"
$request.ContentType = 'multipart/form-data; boundary=---------------------------7dd2db3297c2202' # must match the delimiter in the body, above
$request.ContentLength = $data.Length
# don't automatically redirect to the results page, just take the response which points to it
$request.AllowAutoredirect = $false
# populate the request body
$stream = $request.GetRequestStream()
$stream.Write($data, 0, $data.Length)
$stream.Close()
# get response stream, which should contain a 302 redirect to the results page
$respStream = $request.GetResponse().GetResponseStream()
# pluck out the results page link that you would otherwise be redirected to
(New-Object Io.StreamReader $respStream).ReadToEnd() -match 'HREF\="([^"]+)"' | Out-Null
$matches[1]
}
$url = Get-GoogleImageSearchUrl "C:\Users\Path\filename.jpeg"
Start-Process $url

As mentioned by #soc, you shouldn't need the stream to pull the url, the moved to location is in the response header:
$request = [Net.HttpWebRequest] ([Net.HttpWebRequest]::Create('http://images.google.com/searchbyimage/upload'))
$request.AllowAutoredirect = $false
...
$response = $request.GetResponse()
if ($response.StatusCode -eq 302) {
$redirect_url = $response.Headers["Location"]
write-host $redirect_url
}

Related

URL health-check PowerShell script correctly gets HTTP 200 on most sites, but incorrect '0' status code on some...API timeout issue?

I have a URL health-checking PowerShell script which correctly gets an HTTP 200 status code on most of my intranet sites, but a '0' status code is returned on a small minority of them. The '0' code is an API return rather than from the web site itself, according to my research of questions from others who have written similar URL-checking PowerShell scripts. Thinking this must be a timeout issue, where API returns '0' before the slowly-responding web site returns its 200, I've researched yet more questions about this subject area on SO and implemented a suggestion from someone to insert a timeout in the script. The timeout setting though, no matter how high I set the timeout value, doesn't help. I still get the same '0' "response" code from the same web sites even though those web sites are up and running as checked from any regular web browser. Any thoughts on how I could tweak the timeout setting in the script below in order to get the correct 200 response code?
The Script:
$URLListFile = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$URLList = Get-Content $URLListFile -ErrorAction SilentlyContinue
#if((test-path $reportpath) -like $false)
#{
#new-item $reportpath -type file
#}
#For every URL in the list
$result = foreach($Uri in $URLList) {
try{
#For proxy systems
[System.Net.WebRequest]::DefaultWebProxy = [System.Net.WebRequest]::GetSystemWebProxy()
[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
#Web request
$req = [system.Net.WebRequest]::Create($uri)
$req.Timeout=5000
$res = $req.GetResponse()
}
catch {
#Err handling
$res = $_.Exception.Response
}
$req = $null
#Getting HTTP status code
$int = [int]$res.StatusCode
# output a formatted string to capture in variable $result
"$int - $uri"
#Disposing response if available
if($res){
$res.Dispose()
}
}
# output on screen
$result
#output to log file
$result | Set-Content -Path "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt" -Force
Current output:
200 - http://192.168.1.1/
200 - http://192.168.1.2/
200 - http://192.168.1.250/config/authentication_page.htm
0 - https://192.168.1.50/
200 - http://app1-vip-http.dev.local/
0 - https://CA/certsrv/Default.asp
Perhaps using PowerShell cmdlet Invoke-WebRequest works better for you. It has many more parameters and switches to play around with like ProxyUseDefaultCredentials and DisableKeepAlive
$pathIn = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$pathOut = "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt"
$URLList = Get-Content -Path $pathIn
$result = foreach ($uri in $URLList) {
try{
$res = Invoke-WebRequest -Uri $uri -UseDefaultCredentials -UseBasicParsing -Method Head -TimeoutSec 5 -ErrorAction Stop
$status = [int]$res.StatusCode
}
catch {
$status = [int]$_.Exception.Response.StatusCode.value__
}
# output a formatted string to capture in variable $result
"$status - $uri"
}
# output on screen
$result
#output to log file
$result | Set-Content -Path $pathOut -Force

why this powershell http server code doesn't work

I copied and pasted this code https://community.idera.com/database-tools/powershell/powertips/b/tips/posts/creating-powershell-web-server in a powershell console directory which contains an index.html file
when browsing to http://localhost:8080/index.html I get an error Oops, the page is not available!
Is there something wrong with the code I can't see what ?
# enter this URL to reach PowerShell’s web server
$url = 'http://localhost:8080/'
# HTML content for some URLs entered by the user
$htmlcontents = #{
'GET /' = '<html><building>Here is PowerShell</building></html>'
'GET /services' = Get-Service | ConvertTo-Html
}
# start web server
$listener = New-Object System.Net.HttpListener
$listener.Prefixes.Add($url)
$listener.Start()
try
{
while ($listener.IsListening) {
# process received request
$context = $listener.GetContext()
$Request = $context.Request
$Response = $context.Response
$received = '{0} {1}' -f $Request.httpmethod, $Request.url.localpath
# is there HTML content for this URL?
$html = $htmlcontents[$received]
if ($html -eq $null) {
$Response.statuscode = 404
$html = 'Oops, the page is not available!'
}
# return the HTML to the caller
$buffer = [Text.Encoding]::UTF8.GetBytes($html)
$Response.ContentLength64 = $buffer.length
$Response.OutputStream.Write($buffer, 0, $buffer.length)
$Response.Close()
}
}
finally
{
$listener.Stop()
}
It does work if you go to just http://localhost:8080/, you'd have to also have an index.html listing in order to browse to that too.
Just modify the $htmlContents section like so:
# HTML content for some URLs entered by the user
$htmlcontents = #{
'GET /' = '<html><building>Here is PowerShell</building></html>'
'GET /services' = Get-Service | ConvertTo-Html
'GET /index.html' = '<html><building>this is my index page</building></html>'
}
You could also have a statement like this.
$htmlcontents = #{
'GET /' = '<html><building>Here is PowerShell</building></html>'
'GET /services' = Get-Service | ConvertTo-Html
'GET /index.html' = '<html><building>this is my index page</building></html>'
'GET /fromPage.html' = Get-content "C:\temp\fence.txt"
}

How to download a zip file with PowerShell from request stream

Currently, I make a POST request to a external website then I am supposed to get a zip file in return. I can get the zip file, but it comes in an xml with just the name.zip and nothing is downloaded. I have no idea why it is not downloading. My code is below on the piece where I make the actual request. I am not sure if I am over engineering this or what else I would have to do to get the actual file to download.
$url = "https://thewebsite.net/v6_1?id=$messageID"
Write-Output($url)
$Body = [byte[]][char[]]$xmlMessage
Write-Output($Body)
$Request = [System.Net.HttpWebRequest]::CreateHttp($url);
$Request.Method="POST"
$Request.ContentType = 'text/xml;charset=utf-8'
$Request.ContentLength = $Body.Length
$Request.ClientCertificates.Add($Certificate)
Write-Output($Request.ClientCertificates)
$Stream = $Request.GetRequestStream();
$Stream.Write($Body, 0, $Body.Length);
$Response = $Request.GetResponse()
$totalLength = [System.Math]::Floor($Response.get_ContentLength()/1024)
$responseStream = $Response.GetResponseStream()
$targetStream = New-Object -TypeName System.IO.FileStream -ArgumentList "D:\path\to\save\test.txt", Create
$buffer = new-object byte[] 1GB
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $count
while ($count -gt 0)
{
[System.Console]::CursorLeft = 0
[System.Console]::Write("Downloaded {0}K of {1}K", [System.Math]::Floor($downloadedBytes/1024), $totalLength)
$targetStream.Write($buffer, 0, $count)
$count = $responseStream.Read($buffer,0,$buffer.length)
$downloadedBytes = $downloadedBytes + $count
Write-Output($count)
}
$targetStream.Flush()
$targetStream.Close()
$targetStream.Dispose()
$responseStream.Dispose()
Unfortunately without certain download URI it's hard to clarify either you case is nontrivial or you just select non optimal way to get remote file. Routine way to get ".zip" (or any other 'octet/stream' file) with Power-Shell is execute the following command
Invoke-WebRequest -uri "https://thewebsite.net/v6_1?id=$messageID" -Method "GET" -Outfile (-join($messageID,".zip"))
then $messageID.zip file would be created in directory from which you execute Power-Shell
Progress would be shown in console window automatically. I test this example just before write the answer and it works independently on method "POST"/"GET" when remote host actually return "octet/stream" in the response. Maybe in you case file is not directly returned after requesting
thewebsite.net/v6_1?id=$messageID
but it is not a point of you original question.
Have you tried using Invoke-WebRequest?
$path = [Environment]::GetFolderPath("MyDocuments")
Invoke-WebRequest "example.com" -OutFile "$path\ZippedFile.zip"
A variable does not have to be used, as the path can be completely defined in the Invoke-WebRequest line if desired.

Azure storage API: download entire folder / prefix / directory

I need to be able to download a folder with its contents from a blob storage account using the Azure Storage REST API only.
I have created a function (New-StorageAccountAthorisationHeader) that creates the (authentication) header that I can download a single file, but I cannot find any reference on how I might go about downloading the whole folder.
If I pass the folder as the $blob parameter, I get a BlobNotFound error.
The URL of the said folder is: https://mystorageaccount.blob.core.windows.net/acontainer/somefolder. The contents of "somefolder" looks like:
Folder1
FolderA
FileA.txt
FolderB
FileB.txt
FileC.txt
New-StorageAccountAthorisationHeader:
function New-StorageAccountAuthorizationHeader
{
[cmdletbinding()]
param
(
[string]$StorageAccountName,
[string]$Container,
[string]$Blob,
[string]$accesskey ,
[string]$ResourceUri,
[string]$xmsversion = "2017-04-17"
)
$xmsdate = Get-Date
$xmsdate = $xmsdate.ToUniversalTime()
$xmsdate = $xmsdate.toString('r')
function GetRestApiParameters
{
[cmdletbinding()]
param
(
[Parameter(Mandatory=$true)]
[string]$Uri
)
if($Uri.Contains("?"))
{
Write-Verbose "URI to extract REST parameters: $uri"
return ($Uri.Split("?")[1]).Split("&")
}
}
Write-Verbose "Generating string for signature encryption..."
$partUrl = "/$StorageAccountName/"
if($Container)
{
$partUrl = $partUrl + "$Container/"
}
if($Blob)
{
$parturl = $partUrl + "$Blob"
}
######Don't change the line count or indentation of the here-string#####
$hereString = #"
GET
x-ms-date:$xmsdate
x-ms-version:$xmsversion
$partUrl
"#
$hereString =$hereString -replace "$([char]13)$([char]10)","$([char]10)" #Change `r`n to just `n
$empty = $oSignature = New-Object System.Text.StringBuilder
$empty = $oSignature.Append($hereString)
Write-Verbose "Appending parameters from URI into authorisation string..."
$restParameters = GetRestApiParameters -Uri $ResourceUri -Verbose
if ($restParameters -ne $null)
{
foreach ($param in $restParameters)
{
$empty = $oSignature.Append("$([char]10)$($param.Replace('=',':'))")
}
}
#$oSignature.toString()
Write-Verbose "Encrypting string..."
$hmacsha = New-Object System.Security.Cryptography.HMACSHA256
$hmacsha.key = [Convert]::FromBase64String($accesskey)
$signature = $hmacsha.ComputeHash([Text.Encoding]::UTF8.GetBytes($oSignature.ToString()))
$signature = [Convert]::ToBase64String($signature)
Write-Verbose "Building header..."
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("x-ms-version", $xmsversion)
$headers.Add("x-ms-date", $xmsdate)
$headers.Add("Authorization", "SharedKey " + $StorageAccountName + ":" + $signature)
#$headers.Add("x-ms-blob-type","BlockBlob")
#$headers.Add("Content-Type", "application\xml")
Write-Verbose ("Header: $($headers | Out-String)")
Return $headers
}
And I would call it:
$StorageAccountName = "mystorageaccount"
$container = "acontainer"
$blob = "somefile.txt"
$uriToDownloadBlobs = "https://" + $StorageAccountName + ".blob.core.windows.net/$container/$blob"
$header = $null
$header = New-StorageAccountAuthorizationHeader -StorageAccountName $StorageAccountName -ResourceUri $uriToDownloadBlobs -Verbose -Container $container -Blob $blob
$result = Invoke-WebRequest -Headers $header -Uri $uriToDownloadBlobs -OutFile C:\Temp\$blob -PassThru
$result
So this works, but as I said, I'm after any hints to help with downloading the whole folder.
It looks like this is not possible? Although I'd be interested to see how it's done with the likes of Azure Storage Explorer.
My solution was to zip the files up and then use the above to download the single ZIP file. A few extra lines of code to compress and extract at either end, but it was the quickest way at the time and it works well with VSTS tasks.

PowerShell WebRequest POST

In Windows PowerShell 3.0 was introduced Invoke-RestMethod cmdlet.
Invoke-RestMethod cmdlet accepts -Body<Object> parameter for setting the body of the request.
Due to a certain limitations Invoke-RestMethod cmdlet could not be used in our case. From the other hand, an alternative solution described in article InvokeRestMethod for the Rest of Us suits our needs:
$request = [System.Net.WebRequest]::Create($url)
$request.Method="Get"
$response = $request.GetResponse()
$requestStream = $response.GetResponseStream()
$readStream = New-Object System.IO.StreamReader $requestStream
$data=$readStream.ReadToEnd()
if($response.ContentType -match "application/xml") {
$results = [xml]$data
} elseif($response.ContentType -match "application/json") {
$results = $data | ConvertFrom-Json
} else {
try {
$results = [xml]$data
} catch {
$results = $data | ConvertFrom-Json
}
}
$results
But it is intended for a GET method only.
Could you please suggest how to extend this code sample with the ability to send the body of the request using POST method (similar to Body parameter in Invoke-RestMethod)?
First, change the line that updates the HTTP method.
$request.Method= 'POST';
Next, you need to add the message body to the HttpWebRequest object. To do that, you need to grab a reference to the request stream, and then add data to it.
$Body = [byte[]][char[]]'asdf';
$Request = [System.Net.HttpWebRequest]::CreateHttp('http://www.mywebservicethatiwanttoquery.com/');
$Request.Method = 'POST';
$Stream = $Request.GetRequestStream();
$Stream.Write($Body, 0, $Body.Length);
$Request.GetResponse();
NOTE: PowerShell Core edition is now open source on GitHub, and cross-platform on Linux, Mac, and Windows. Any issues with the Invoke-RestMethod cmdlet should be reported on the GitHub issue tracker for this project, so they can be tracked and fixed.
$myID = 666;
#the xml body should begin on column 1 no indentation.
$reqBody = #"
<?xml version="1.0" encoding="UTF-8"?>
<ns1:MyRequest
xmlns:ns1="urn:com:foo:bar:v1"
xmlns:ns2="urn:com:foo:xyz:v1"
<ns2:MyID>$myID</ns2:MyID>
</ns13:MyRequest>
"#
Write-Host $reqBody;
try
{
$endPoint = "http://myhost:80/myUri"
Write-Host ("Querying "+$endPoint)
$wr = [System.Net.HttpWebRequest]::Create($endPoint)
$wr.Method= 'POST';
$wr.ContentType="application/xml";
$Body = [byte[]][char[]]$reqBody;
$wr.Timeout = 10000;
$Stream = $wr.GetRequestStream();
$Stream.Write($Body, 0, $Body.Length);
$Stream.Flush();
$Stream.Close();
$resp = $wr.GetResponse().GetResponseStream()
$sr = New-Object System.IO.StreamReader($resp)
$respTxt = $sr.ReadToEnd()
[System.Xml.XmlDocument] $result = $respTxt
[String] $rs = $result.DocumentElement.OuterXml
Write-Host "$($rs)";
}
catch
{
$errorStatus = "Exception Message: " + $_.Exception.Message;
Write-Host $errorStatus;
}