Powershell WebScraping and Terminating Exception Handling - powershell

I'm using powershell to check several different sites for data. If the site has no data for today, it will throw a NullReferenceException. I'd like for the script to output a message that there is no data, then continue on to the other sites without halting.
In Java, I can simply just try/catch/finally, but Powershell isn't acting as nicely.
try {
$webRequest = Invoke-WebRequest -URI "http://###.##.###.##:####/abcd.aspx?"
} catch [System.NullReferenceException]{
Write-Host "There is no data"
}
The full error displays in console, and the Write-Host never actually appears.

try {
$webRequest = Invoke-WebRequest -URI "http://host/link" -erroraction stop
}
catch [System.NullReferenceException]{
Write-Host "There is no data"
}
Powershell distinguishes between terminating and non terminating errors, for catch to work, you need the error to be terminating. https://blogs.technet.microsoft.com/heyscriptingguy/2015/09/16/understanding-non-terminating-errors-in-powershell/
UPD: to get the type of exception, after you get the error just do:
$Error[0].Exception.GetType().FullName
and you use that to catch that specific error after
to continue on specific error with invoke-webrequest you can do something like this:
try { Invoke-WebRequest "url" }
catch { $req = $_.Exception.Response.StatusCode.Value__}
if ($req -neq 404) { do stuff }

That's probably because the exception may not be a Null Reference exception. I Initially thought it to be a Non-Terminating error, but invoke-webrequest throws a terminating error.
In this case, you may simply try (without catching a specific exception type)
--Edited Per OP Comments--
try
{
Invoke-WebRequest -URI "http://doc/abcd.aspx?" -ErrorAction Stop
}
catch
{
if($_.Exception.GetType().FullName -eq "YouranticipatedException")
{
Write-Host ("Exception occured in Invoke-WebRequest.")
# You can also get the response code thru "$_.Exception.Response.StatusCode.Value__" if there is response to your webrequest
}

Related

Download a file, get the status, then execute the file

I've tried invoke-restmethod, new-object and many other methods to achieve what I'm trying to do. Here are the latest two iterations:
$req = Invoke-WebRequest -uri $scripturl -OutFile "$($scriptpath)\fls.core.ps1"
Write-Host "StatusCode:" $req.StatusCode
$req = Invoke-WebRequest -uri $scripturl -OutFile "$($scriptpath)\fls.core.ps1" | Select-Object -Expand StatusCode
Write-Host "StatusCode:" $req
Basically I'm attempting to download another PowerShell script and execute it. So obviously it needs to be synchronous. I also need the status so I can determine if it updated or not.
Here is pseudo code for what I'm trying to accomplish:
try {
download file
} catch {
output error
if (local copy exists) {
log warning that local copy is being used
} else {
log error could not download and no local copy available
exit script
}
}
run script (only after downloading new one if available)
Here is my current code in full:
$param1=$args[0]
if ($param1 -eq "-d" -or $param1 -eq "-D") {
$isDev = $true
}
#todo: Move to config file
$logpath = "c:\company\logs\loginscript"
$scriptpath = "c:\company\scripts\"
$scripturl = "http://downloads.company.com/fls.core.ps1"
$logfile="$(Get-Date -Format "yyyy-MM-dd hhmmss").log"
Function log($message) {
Write-Output "[$(Get-Date -Format "yyyy-MM-dd hhmmss")] $message" | Out-file "$($logpath)\$($logfile)" -append
if ($isDev) { Write-Host "[$(Get-Date -Format "yyyy-MM-dd hhmmss")] $message" }
}
Function createFolder($path) {
if (-!(Test-Path $path)) { New-Item -Type Directory -Path $path }
}
function updateScripts() {
try {
$req = Invoke-WebRequest -uri $scripturl -OutFile "$($scriptpath)\fls.core.ps1"
Write-Host "StatusCode:" $req.StatusCode
} catch {
Write-Host "StatusCode:" $req.StatusCode
if ($req.StatusCode -eq 404) {
log "WARNING: Script not found at $scripturl"
} else {
log "ERROR: Script download error: $req.StatusCode"
}
if (Test-Path "$($scriptpath)\fls.core.ps1") {
log "WARNING: Using local script"
} else {
log "ERROR: Unable to update script and no local script found. Exiting."
exit
}
}
}
#----------------------------------------------#
#---- MAIN CODE BLOCK -------------------------#
#----------------------------------------------#
createFolder $logpath
createFolder $scriptpath
#update scripts
updateScripts
#execute core loginscript
& $scriptpath/fls.core.ps1
$req.StatusCode appears to be null.
Invoke-WebRequest reports errors as statement-terminating errors, which means that no assignment to variable $req (in statement $req = Invoke-WebRequest ...) takes place in case an error occurs.
Instead, unfortunately, if an error occurs, the response object[1] must be gleaned from the [ErrorRecord] instance representing the error, which is available via $Error[0] after the fact, or via $_ in the catch block of a try { ... } catch { ... } statement (adapted from this answer):
try {
Invoke-WebRequest -Uri $scripturl -OutFile "$scriptpath\fls.core.ps1"
} catch [Microsoft.PowerShell.Commands.HttpResponseException] {
# Get the status code...
$statuscode = $_.Exception.Response.StatusCode
# ... and work with it.
# if ($statusCode -eq 404) { ...
} catch {
# Unexpected error, re-throw
throw
}
Strictly speaking, $_.Exception.Response.StatusCode returns a value from an enumeration type, System.Net.HttpStatusCode, not an [int] value, but you can use it like an integer. To return an integer to begin with, append .Value__ or cast to [int].
Note that Invoke-WebRequest is always synchronous; if you download a file (successfully), the call won't return until the download is completed.
[1] As the linked answer explains, the response object contained in the error record is of a different type than the one that Invoke-WebRequest returns in case of success (which requires -PassThru if -OutFile is also specified): The error record's .Exception.Response property contains a System.Net.Http.HttpResponseMessage instance, whereas Invoke-WebRequest returns an instance (derived from) Microsoft.PowerShell.Commands.WebResponseObject, which incorporates an instance of the former type, in its .BaseResponse property.

How do I prevent PowerShell from adding exception data to $error

I am invoking a lot of REST calls in a function.
I know that some of them will fail, but that is expected.
My question is:
How do I prevent powershell from adding entries to the global $error variable?
foreach:
$oldErrorActionPreference = $ErrorActionPreference
$ErrorActionPreference = "Ignore"
try {
$response = Invoke-RestMethod -Uri $Uri -ea Ignore
} catch {
Write-Verbose "$_"
} finally {
$ErrorActionPreference = $oldErrorActionPreference
}
$error variable after invoking:
Invoke-RestMethod cmdlet always fails with terminating error, which can't be Ignored — it will always end up inside $Error list.
You can clear it: $Error.Clear()

Trying to Use Power shell Script to test availability of website

I am trying to use a powershell script to test the availability of certain websites. I have a script here that writes "Site is OK" if the site returns a http 200 code. It should return "The Site may be down, please check!" If it returns any other code. I put in 'https://www.google.com/cas76' which should obviously return a 404 error however the script returns "Site is ok" How should I go about fixing my code so it returns "The Site may be down, please check!"
Tried putting in websites that would obviously not return a 200 code.
# First we create the request.
$HTTP_Request = [System.Net.WebRequest]::Create('https://www.google.com/cas76')
# We then get a response from the site.
$HTTP_Response = $HTTP_Request.GetResponse()
# We then get the HTTP code as an integer.
$HTTP_Status = [int]$HTTP_Response.StatusCode
If ($HTTP_Status -eq 200) {
Write-Host "Site is OK!"
}
Else {
Write-Host "The Site may be down, please check!"
}
# Finally, we clean up the http request by closing it.
$HTTP_Response.Close()
Code acknowledges that there is a 404 error
Exception calling "GetResponse" with "0" argument(s): "The remote server returned an error: (404) Not Found."
At C:\Users\TX394UT\Desktop\Web_Bot_Project\WebsiteMonitoring.ps1:6 char:1
+ $HTTP_Response = $HTTP_Request.GetResponse()
However, Site is OK! Prints on the console.
the way that the webclient handles returned errors is ... odd. [grin]
that message is treated as a non-terminating error and needs to be handled as such. the most obvious way is with Try/Catch/Finally. when capturing the exception message, the 404 StatusCode is converted to the text value for it - NotFound.
$TestUrl = 'https://www.google.com/cas76'
try
{
$Response = (Invoke-WebRequest -Uri $TestUrl -ErrorAction Stop).StatusCode
}
catch
{
$Response = $_.Exception.Response.StatusCode
}
$Response
output for the bad url = NotFound
output for a good url = 200

Catching errors in Powershell

I have weird problem, when im using try/catch method for some cmdlets its working for some not.
Can you advice on that?
This one is working fine:
try
{
$LookingForRemoteMailboxOnPrem = Get-RemoteMailbox $info -ErrorAction Stop | select -ExpandProperty UserPrincipalName
}
catch
{
string]$t = $Error[0]
}
But this one is not:
try
{
$EnableRemoteMailbox = Enable-RemoteMailbox $info -RemoteRoutingAddress $remote -PrimarySmtpAddress $info2 -ErrorAction Stop
}
catch
{
[string]$t = $Error[0]
}
Not saving error to $t variable
The $ErrorActionPreference is set to Continue by default. This means if PowerShell can "recover" from an error it won't throw an exception. You can use the -ErrorAction parameter to change the behaviour at every cmdlet.
This link gives a good example:
Try {dir c:\missingFolder}
Catch [System.Exception] {"Caught the exception"}
Finally {$error.Clear() ; "errors cleared"}
The string "Caught the exception does not occur in PowerShell windows. If you set the -ErrorAction to Stop an exception is raised.
Details are described here.

Get HTTP request and tolerate 500 server error in powershell

In Powershell v3.0 I would like to return the response code from an HTTP GET, such as 200 OK or 500 Internal Server Error. (This is for a hack-deploy to do a quick warmup of a deployed site and see if it works, a sort of a mini acceptance test. The status code is truly all I want.)
Against my wishes, HttpWebRequest.GetResponse throws an error when it receives a 500 Internal Server Error. This is annoying because it isn't really an error to me in my use case. Anyway, I figured I could catch the exception and still peel out the underlying response code, but I'm having trouble with that.
Here's some almost-working code:
function WebResponseStatusCode (
[Parameter(Mandatory=$true)][string] $url
) {
$req = [system.Net.HttpWebRequest]::Create($url)
try {
$res = $req.GetResponse();
$statuscode = $res.statuscode;
}
catch [System.Net.WebException] {
#the outer error is a System.Management.Automation.ErrorRecord
Write-Host "error!"
return = $_.Response.statuscode; #nope
}
finally {
if (!($res -eq $null)) {
$res.Close();
}
}
return $statuscode;
}
The problem is of course that $_ has no Response property. Neither does $_.InnerException, even when cast:
return [System.Net.WebException]($_.InnerException)
I've played around with $_ | Get-Member and exploring all its properties. I thought $_.TargetObject had some promise but it doesn't appear to.
(Update) I also think I tried variations on $_.Exception.Response though may have gotten it wrong.
Getting just a response code seems like such a simple thing to do.
Here's an example, though it does a couple more things to allow you to test redirections as well as expected exceptions.
function GetWebSiteStatusCode {
param (
[string] $testUri,
$maximumRedirection = 5
)
$request = $null
try {
$request = Invoke-WebRequest -Uri $testUri -MaximumRedirection $maximumRedirection -ErrorAction SilentlyContinue
}
catch [System.Net.WebException] {
$request = $_.Exception.Response
}
catch {
Write-Error $_.Exception
return $null
}
$request.StatusCode
}
GetWebSiteStatusCode -testUri "https://www.google.com/"
GetWebSiteStatusCode -testUri "https://www.google.com/foobar"
GetWebSiteStatusCode -testUri "http://google.com/" -maximumRedirection 0
GetWebSiteStatusCode -testUri "https://accounts.google.com" -maximumRedirection 0
GetWebSiteStatusCode -testUri "https://www.googleapis.com/coordinate/v1/teams/1/custom_fields?fields=1111&key="
GetWebSiteStatusCode -testUri "https://www.googleapis.com/shopping/search/v1/test/products/sasdf/asdf/asdf?key="
#Next test would be for an expected 500 page.
#GetWebSiteStatusCode -testUri "https://www.somesite.com/someurlthatreturns500"