I have the following problem:
I use a PS script that scrapts a web page. Based on the response, it concludes if the site is up or down.
My code works
foreach ($site in $websitesArray) {
$Counter = 5
$ErrorCounter = 0
if ($site -like 'http://*') {
$siteSplit = $site.Replace("http://", "")
}
else {
$siteSplit = $site.Replace("https://", "")
}
$ip = [System.Net.Dns]::GetHostAddresses($siteSplit)
for (int i = 0 ; i -lt $Counter ; i++) {
Try {
$HTMLstring = $web.DownloadString($site)
}
Catch {
$ErrorCounter++
}
Start-Sleep 5s
}
if ($ErrorCounter > 3) {
$body += "The website " + $site + " (" + $ip + ") has returned an HTTP error and is down <br />"
}
}
The problem is that the following is returned for pages that have a Cookie Policy pop up:
The remote server returned an error: (500) Internal Server Error.
What workaround could I use to prevent this false positive from happening? Keep in mind that I have a list of websites so a hardcoded solution wouldn't be helping me.
Related
I am trying to pull the last 7 days of sign-in logs using Get-AzureADAuditSignInLogs from the AzureADPreview Module 2.0.2.105, however I am being throttled. I've tried using start-sleep within try/catch blocks, but no luck. Any other ideas on how I can get around the throttling limits? (other than using the Graph API which in this case isn't an option)
Error:
Fetching data from Azure Active Directory...
Get-AzureADAuditSignInLogs : Error occurred while executing GetAuditSignInLogs
Code: UnknownError
Message: This request is throttled. Please try again after the value specified in the Retry-After header. CorrelationId: e23fd131-xxxx-xxxx-xxxx-cbd125f5606b
InnerError:
RequestId: xxxxxxx-97be-4ae7-xxxx-8c7c4781ec0b
DateTimeStamp: Sun, 12 Jun 2022 09:27:04 GMT
HttpStatusCode: 429
HttpStatusDescription:
HttpResponseStatus: Completed
At line:4 char:12
+ $Records = Get-AzureADAuditSignInLogs -Filter "createdDateTime gt $St ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-AzureADAuditSignInLogs], ApiException
+ FullyQualifiedErrorId : Microsoft.Open.MSGraphBeta.Client.ApiException,Microsoft.Open.MSGraphBeta.PowerShell.GetAuditSignInLogs
This is the code:
# Fetches the last month's Azure Active Directory sign-in data
CLS; $StartDate = (Get-Date).AddDays(-7); $StartDate = Get-Date($StartDate) -format yyyy-MM-dd
Write-Host "Fetching data from Azure Active Directory..."
try {
$Records = Get-AzureADAuditSignInLogs -Filter "createdDateTime gt $StartDate" -all:$True
$Report = [System.Collections.Generic.List[Object]]::new()
}
catch {
Start-Sleep -Seconds 200
$Records = Get-AzureADAuditSignInLogs -Filter "createdDateTime gt $StartDate" -all:$True
$Report = [System.Collections.Generic.List[Object]]::new()
}
ForEach ($Rec in $Records) {
Switch ($Rec.Status.ErrorCode) {
"0" {$Status = "Success"}
default {$Status = $Rec.Status.FailureReason}
}
$ReportLine = [PSCustomObject] #{
TimeStamp = Get-Date($Rec.CreatedDateTime) -format g
User = $Rec.UserPrincipalName
Name = $Rec.UserDisplayName
IPAddress = $Rec.IpAddress
ClientApp = $Rec.ClientAppUsed
Device = $Rec.DeviceDetail.OperatingSystem
Location = $Rec.Location.City + ", " + $Rec.Location.State + ", " + $Rec.Location.CountryOrRegion
Appname = $Rec.AppDisplayName
Resource = $Rec.ResourceDisplayName
Status = $Status
Correlation = $Rec.CorrelationId
Interactive = $Rec.IsInteractive }
$Report.Add($ReportLine) }
Write-Host $Report.Count "sign-in audit records processed."
I tried to reproduce the same in my environment and got the below results:
As I have less sign-in logs, I got the count successfully.
Throttling error occurs if there are too many requests. To handle the throttling error, you can try the below:
Try to reduce number of requests by giving shorter time span.
As mentioned in the error message, try using Retry-After header to resolve the issue.
By using this header, when throttling occurs it delays and retry the request until it succeeds.
As a workaround, you can include Retry-After header in your PowerShell snippet mentioned by Mike-Crowley in this GitHub blog.
try {$webResponse = Get-AzureADAuditSignInLogs...}
catch {
$webError = $_.Exception
$mustRetry = 1
}
if ($mustRetry -and ($webError.Response.StatusCode -eq 429)) {
$SleepDuration = ($webError.Response.Headers[“Retry-After”])
Write-Output ("Too many requests. Waiting " + $SleepDuration + " seconds...")
sleep ([int32]$SleepDuration + 1)
$webResponse = Get-AzureADAuditSignInLogs..
}
For more in detail, please refer below links:
Error occurred while executing GetAuditSignInLogs. Message: This - Microsoft Community
How to handle throttling of Microsoft Graph in PowerShell - Stack Overflow
I am trying to run a Powershell script and display the error code and error message if it fails. It is supposed to output me a result in this format:
"FAILED;ErrorCode;ErrorMessage;"
Here is my script:
param([String]$Cab_Type)
$output
if(!(Test-Connection -Cn 165.100.10.10 -BufferSize 16 -Count 1 -quiet))
{
$output = "FAILED; " + $LASTEXITCODE + ";" + $error[0] + ";"
}
else
{
$output = "PASSED"
}
Write-Host $Cab_Type
Write-Host "<ScriptResult_Start>"
Write-Host $output
Write-Host "<ScriptResult_End>"
I am trying to intentionally ping an address that I know will fail.
When running the script, it returns me the error message but not the error code.
Does $LASTEXITCODE not return the error code of the script? Even if my script worked, does it only return 0 or 1? Is there any way of getting the actual error code of the script?
Perhaps this is what you are after?
# set up a hash with possible Ping status codes
$status = #{
11001 = 'Buffer Too Small'
11002 = 'Destination Net Unreachable'
11003 = 'Destination Host Unreachable'
11004 = 'Destination Protocol Unreachable'
11005 = 'Destination Port Unreachable'
11006 = 'No Resources'
11007 = 'Bad Option'
11008 = 'Hardware Error'
11009 = 'Packet Too Big'
11010 = 'Request Timed Out'
11011 = 'Bad Request'
11012 = 'Bad Route'
11013 = 'TimeToLive Expired Transit'
11014 = 'TimeToLive Expired Reassembly'
11015 = 'Parameter Problem'
11016 = 'Source Quench'
11017 = 'Option Too Big'
11018 = 'Bad Destination'
11032 = 'Negotiating IPSEC'
11050 = 'General Failure'
}
$server = '165.100.10.10'
$ping = (Get-WmiObject -Class Win32_PingStatus -Filter "Address='$server'").StatusCode
if (!$ping -or [int]$ping -ne 0) {
$err = if ( $status[[int]$ping]) { $status[[int]$ping] } else { "Unknown Failure" }
$output = "FAILED; $ping; $err"
}
else { $output = "PASSED" }
Write-Host $output
The above example outputs:
FAILED; 11010; Request Timed Out
There are multiple ways to handle errors and verifying the. First and foremost, I would like you to keep the main code under a try/catch block and capture the error message so that you get to know what exactly the error is. Below is the modification of your code.
param([String]$Cab_Type)
$output
try
{
if(!(Test-Connection -Cn 165.100.10.10 -BufferSize 16 -Count 1 -quiet))
{
$output = "FAILED; " + $LASTEXITCODE + ";" + $error[0] + ";"
}
else
{
$output = "PASSED"
}
Write-Host $Cab_Type
Write-Host "<ScriptResult_Start>"
Write-Host $output
Write-Host "<ScriptResult_End>"
}
catch
{
$_.Exception.Message
}
Also, go through about 1. TRY CATCH FINALLY in PS
Apart from that you can use $Error variable to get to know about all the errors.
Go through 2. About $Error variable and how to use it effectively
Also you need to understand 3. Powershell Exceptions and Everything you ever wanted to know
Hope it helps and gives you a direction.
I'm trying to get a 200 status code for a URL but I keep getting 0 instead. The url works, but the problem is it's a redirected URL. Even if I try the final URL after it redirects it still shows me a 0 status code.
How can I get the correct status code for a website that's down or up regardless of whether it's been redirected or not?
This is what I have now which works fine for regular URL's like http://google.com but not for redirected URL's. Unfortunately the urls I'm working with are private but it's in the format of http://example.com which winds up at https://example.com/index?redirectUrl=
If I run the PS script below with: .\CheckUrl.ps1 https://example.com/index?redirectUrl=
...it still fails to return a code of 200. The page comes up fine, whether I use the 1st url or the final redirected url but status code returns 0 which means it says the site is down and that's not true.
$url = $args[0]
function Get-WebStatus($url) {
try {
[Net.HttpWebRequest] $req = [Net.WebRequest]::Create($url)
$req.Method = "HEAD"
[Net.HttpWebResponse] $res = $req.GetResponse()
if ($res.StatusCode -eq "200") {
Write-Host "`nThe site $url is UP (Return code: $($res.StatusCode) - $([int] $res.StatusCode))`n"
} else {
Write-Host "`nThe site $url is DOWN (Return code: $($res.StatusCode) - $([int] $res.StatusCode))`n"
}
} catch {
Write-Host "`nThe site $url is DOWN (Return code: $($res.StatusCode) - $([int] $res.StatusCode))`n" -ForegroundColor Red -BackgroundColor Black
}
}
Get-WebStatus $url
Too long for a comment. Important: $res = $req.GetResponse() does not set any value into the $res variable in the catch case (the $res variable keeps unchanged).
#url1 = $args[0]
function Get-WebStatus($url) {
try {
$req = [System.Net.HttpWebRequest]::Create($url)
$req.Method = "HEAD"
$req.Timeout = 30000
$req.KeepAlive = $false
$res = $req.GetResponse()
if ($res.StatusCode.value__ -eq 200) {
Write-Host ("`nThe site $url is UP (Return code: " +
"$($res.StatusCode) - " +
"$($res.StatusCode.value__))`n") -ForegroundColor Cyan
} else {
Write-Host ("`nThe site $url is DOWN (Return code: " +
"$($res.StatusCode) - " +
"$($res.StatusCode.value__))`n") -ForegroundColor Yellow
}
} catch {
$res = $null ### or ### [System.Net.HttpWebRequest]::new()
Write-Host ("`nThe site $url is DOWN " +
"($($error[0].Exception.InnerException.Message))`n") -Foreground Red
}
$res ### return a value
}
#Get-WebStatus $url1
Output examples:
Get-WebStatus 'https://google.com/index?redirectUrl='
Get-WebStatus 'https://google.com/'
Get-WebStatus 'https://example.com/index?redirectUrl='
The site https://google.com/index?redirectUrl= is DOWN (The remote server
returned an error: (404) Not Found.)
The site https://google.com/ is UP (Return code: OK - 200)
The site https://example.com/index?redirectUrl= is DOWN (The operation has
timed out)
I was successfully able to open a port on my computer (using only PowerShell) and know when HTTP requests are done to that port. I came up with this simple code:
$listener = [System.Net.Sockets.TcpListener]5566;
$listener.Start();
while ($true) {
$client = $Listener.AcceptTcpClient();
Write-Host "Connected!";
$client.Close();
}
If I open my browser and type http://localhost:5566 in the PowerShell interface it will show a message that a user got connected.
What I need to do is to get the GET parameters of this HTTP request. For example, if instead I had opened my browser and typed http://localhost:5566/test.html?parameter1=xxx¶meter2=yyy.
How can I grab the GET parameters (parameter1 and parameter2) name and values using my simplified code above?
If you are comfortable using the HttpListener instead of the TcpListener. It's easier to do the job.
Below script will output in a browser
Path is /test.html
parameter2 is equal to yyy
parameter1 is equal to xxx
Quick and dirty script
$listener = New-Object System.Net.HttpListener
$listener.Prefixes.Add("http://localhost:5566/")
try {
$listener.Start();
while ($true) {
$context = $listener.GetContext()
$request = $context.Request
# Output the request to host
Write-Host $request | fl * | Out-String
# Parse Parameters from url
$rawUrl = $request.RawUrl
$Parameters = #{}
$rawUrl = $rawUrl.Split("?")
$Path = $rawUrl[0]
$rawParameters = $rawUrl[1]
if ($rawParameters) {
$rawParameters = $rawParameters.Split("&")
foreach ($rawParameter in $rawParameters) {
$Parameter = $rawParameter.Split("=")
$Parameters.Add($Parameter[0], $Parameter[1])
}
}
# Create output string (dirty html)
$output = "<html><body><p>"
$output = $output + "Path is $Path" + "<br />"
foreach ($Parameter in $Parameters.GetEnumerator()) {
$output = $output + "$($Parameter.Name) is equal to $($Parameter.Value)" + "<br />"
}
$output = $output + "</p></body></html>"
# Send response
$statusCode = 200
$response = $context.Response
$response.StatusCode = $statusCode
$buffer = [System.Text.Encoding]::UTF8.GetBytes($output)
$response.ContentLength64 = $buffer.Length
$output = $response.OutputStream
$output.Write($buffer,0,$buffer.Length)
$output.Close()
}
} finally {
$listener.Stop()
}
Cheers
Glenn
I am trying to write a PowerShell script that will restart an application pool in IIS if a 503 response code is received.
So far I have managed to retrieve the response code for every crm application under the default website in IIS. However I am unsure how I would go about finding the application pool name. I've tried the below, but it returns the same application pool for each site. Can anyone help?
$getSite = (Get-WebApplication -Site 'Default Web Site')
$SiteURL = ForEach ($site in $getSite.path) {("http://localhost")+$site}
ForEach ($crm in $SiteURL){
$req = [system.Net.WebRequest]::Create($crm)
try {
$res = $req.GetResponse()
} catch [System.Net.WebException] {
$res = $_.Exception.Response
}
$ApplicationPool = ForEach ($app in $getSite.applicationpool) {$app}
if([int]$res.StatusCode -eq 503) {write-host ($crm + ' ' + [int]$res.StatusCode) + $app}
}
I think you need to access $_.Exception.InnerException for the the Response property.
Your $ApplicationPool assignment doesn't make much sense, as you would only need one applicationPool name per $crm app you test:
foreach($App in #(Get-WebApplication -Site 'Default Web Site')){
# Uri for the application
$TestUri = 'http://localhost{0}' -f $App.path
# Create WebRequest
$Request = [system.Net.WebRequest]::Create($TestUri)
try {
# Get the response
$Response = $Request.GetResponse()
} catch [System.Net.WebException] {
# If it fails, get Response from the Exception
$Response = $_.Exception.InnerException.Response
}
# The numerical value of the StatusCode value is the HTTP status code, ie. 503
if(503 -eq ($Response.StatusCode -as [int])){
# Restart the app pool
Restart-WebAppPool -Name $App.applicationPool
}
}