Download a file, get the status, then execute the file - powershell

I've tried invoke-restmethod, new-object and many other methods to achieve what I'm trying to do. Here are the latest two iterations:
$req = Invoke-WebRequest -uri $scripturl -OutFile "$($scriptpath)\fls.core.ps1"
Write-Host "StatusCode:" $req.StatusCode
$req = Invoke-WebRequest -uri $scripturl -OutFile "$($scriptpath)\fls.core.ps1" | Select-Object -Expand StatusCode
Write-Host "StatusCode:" $req
Basically I'm attempting to download another PowerShell script and execute it. So obviously it needs to be synchronous. I also need the status so I can determine if it updated or not.
Here is pseudo code for what I'm trying to accomplish:
try {
download file
} catch {
output error
if (local copy exists) {
log warning that local copy is being used
} else {
log error could not download and no local copy available
exit script
}
}
run script (only after downloading new one if available)
Here is my current code in full:
$param1=$args[0]
if ($param1 -eq "-d" -or $param1 -eq "-D") {
$isDev = $true
}
#todo: Move to config file
$logpath = "c:\company\logs\loginscript"
$scriptpath = "c:\company\scripts\"
$scripturl = "http://downloads.company.com/fls.core.ps1"
$logfile="$(Get-Date -Format "yyyy-MM-dd hhmmss").log"
Function log($message) {
Write-Output "[$(Get-Date -Format "yyyy-MM-dd hhmmss")] $message" | Out-file "$($logpath)\$($logfile)" -append
if ($isDev) { Write-Host "[$(Get-Date -Format "yyyy-MM-dd hhmmss")] $message" }
}
Function createFolder($path) {
if (-!(Test-Path $path)) { New-Item -Type Directory -Path $path }
}
function updateScripts() {
try {
$req = Invoke-WebRequest -uri $scripturl -OutFile "$($scriptpath)\fls.core.ps1"
Write-Host "StatusCode:" $req.StatusCode
} catch {
Write-Host "StatusCode:" $req.StatusCode
if ($req.StatusCode -eq 404) {
log "WARNING: Script not found at $scripturl"
} else {
log "ERROR: Script download error: $req.StatusCode"
}
if (Test-Path "$($scriptpath)\fls.core.ps1") {
log "WARNING: Using local script"
} else {
log "ERROR: Unable to update script and no local script found. Exiting."
exit
}
}
}
#----------------------------------------------#
#---- MAIN CODE BLOCK -------------------------#
#----------------------------------------------#
createFolder $logpath
createFolder $scriptpath
#update scripts
updateScripts
#execute core loginscript
& $scriptpath/fls.core.ps1
$req.StatusCode appears to be null.

Invoke-WebRequest reports errors as statement-terminating errors, which means that no assignment to variable $req (in statement $req = Invoke-WebRequest ...) takes place in case an error occurs.
Instead, unfortunately, if an error occurs, the response object[1] must be gleaned from the [ErrorRecord] instance representing the error, which is available via $Error[0] after the fact, or via $_ in the catch block of a try { ... } catch { ... } statement (adapted from this answer):
try {
Invoke-WebRequest -Uri $scripturl -OutFile "$scriptpath\fls.core.ps1"
} catch [Microsoft.PowerShell.Commands.HttpResponseException] {
# Get the status code...
$statuscode = $_.Exception.Response.StatusCode
# ... and work with it.
# if ($statusCode -eq 404) { ...
} catch {
# Unexpected error, re-throw
throw
}
Strictly speaking, $_.Exception.Response.StatusCode returns a value from an enumeration type, System.Net.HttpStatusCode, not an [int] value, but you can use it like an integer. To return an integer to begin with, append .Value__ or cast to [int].
Note that Invoke-WebRequest is always synchronous; if you download a file (successfully), the call won't return until the download is completed.
[1] As the linked answer explains, the response object contained in the error record is of a different type than the one that Invoke-WebRequest returns in case of success (which requires -PassThru if -OutFile is also specified): The error record's .Exception.Response property contains a System.Net.Http.HttpResponseMessage instance, whereas Invoke-WebRequest returns an instance (derived from) Microsoft.PowerShell.Commands.WebResponseObject, which incorporates an instance of the former type, in its .BaseResponse property.

Related

Invoke-WebRequest to null if it fails

I'm running this part of the code in my script but I have small issue.
$response = Invoke-WebRequest -Uri "http://169.254.169.254/metadata/instance/compute?api-version=2019-06-01" -Headers #{Metadata = "true"} -TimeoutSec 1 -ErrorAction SilentlyContinue
if($response.StatusCode -ne 200) {
write-host "$env:computername is not in the cloud. Let's continuing configurations"
configuration code
}else{
write-host "$env:computername is in the cloud. Stop the script"
}
When the invoke fails which in my cases does 99% of the time it outputs the fail as big red error message which I would want to get rid of
Then I tried to suppress to output of the invoke-webrequest using these command but none of them worked
| Out-Null
> $null
$null =
Tried also to play with Try & Catch but did not manage to it work either because I somehow never got to my last if statement
Write-Verbose "Checking if this is an Azure virtual machine"
try {
$response = Invoke-WebRequest -Uri "http://169.254.169.254/metadata/instance/compute?api-version=2019-06-01" -Headers #{Metadata = "true"} -TimeoutSec 1 -ErrorAction SilentlyContinue
}
catch {
Write-Verbose "Error $_ checking if we are in Azure"
return $false
}
if ($null -ne $response -and $response.StatusCode -eq 200) {
Write-Verbose "Azure check indicates that we are in Azure"
return $true
}
return $false
if($false -eq 'False')
{
Write-Host "server is in Azure"
}
else{
Write-host "server is not in Azure"
}
What should I try next? I'm not super good at PowerShell so there might even just be some errors in syntax or misunderstandings.
I've done some tests with Invoke-WebRequest and was able to suppres the error with try & catch. Try something like that:
Write-Verbose "Checking if this is an Azure virtual machine"
try {
$response = Invoke-WebRequest -Uri "http://169.254.169.254/metadata/instance/compute?api-version=2019-06-01" -Headers #{Metadata = "true"} -TimeoutSec 1 -ErrorAction Stop
# Response was successful (200), otherwise script would run in catch at this point
Write-Verbose "$env:computername is in the cloud. Stop the script"
} catch {
# Webrequest failed (not 200)
Write-Verbose "$env:computername is not in the cloud. Let's continuing configurations"
# CONFIGURATION CODE
}

PowerShell Pester Mock Rest API Calls

Is there any simple approach on how to mock a Rest API Calls in Pester.
Here is my code, I just need to mock those Rest API Calls in Pester and test them, could someone help me here.
Describe 'Test worldclockapi.com' {
BeforeAll {
$response = Invoke-WebRequest -Method 'GET' -Uri 'http://worldclockapi.com/api/json/utc/now'
$responseContent = $response.Content | ConvertFrom-Json
Write-Output $responseContent
}
It 'It should respond with 200' {
$response.StatusCode | Should -Be 200
}
It 'It should have a null service response' {
$responseContent.serviceResponse | Should -BeNullOrEmpty
}
It 'It should be the right day of the week' {
$dayOfWeek = (Get-Date).DayOfWeek
$responseContent.dayOfTheWeek | Should -Be $dayOfWeek
}
It 'It should be the right year' {
$year = Get-Date -Format 'yyyy'
$responseContent.currentDateTime | Should -BeLike "*$year*"
}
It 'It should be the right month' {
$month = Get-Date -Format 'MM'
$responseContent.currentDateTime | Should -BeLike "*$month*"
}
# These two tests assume you are running this outside daylight savings (during the winter) .. hacky but good way to showcase the syntax ;)
It 'It should not be daylight savings time' {
$responseContent.isDayLightSavingsTime | Should -Not -Be $true
}
It 'It should not be daylight savings time another way' {
$responseContent.isDayLightSavingsTime | Should -BeFalse
}
}
It's probably easiest to use a real response as a template for your mock output.
Invoke-WebRequest -Method 'GET' -Uri 'http://worldclockapi.com/api/json/utc/now' |
Export-Clixml .\response.xml
The above command will serialize a real response from the API to a file.
Now we can import the file to use with our mock. All it takes is to use the Mock command to define our mock.
Mock
-CommandName : command we are mocking
-ParameterFilter: An optional filter to limit mocking behavior only to usages of CommandName where the values of the parameters passed to the command pass the filter. This ScriptBlock must return a boolean value.
-MockWith: A ScriptBlock specifying the behavior that will be used to mock CommandName. The default is an empty ScriptBlock. It is here that we import the file to be our output.
Mock -CommandName Invoke-WebRequest -ParameterFilter { $Method -eq 'GET' } -MockWith { Import-Clixml .\response.xml }
Now when Invoke-WebRequest is called with -Method 'Get' our mock will be called instead and will return our object which we import using Import-CliXml (or other method - json, xml, etc. Note: the mock command must come before any calls to the command we are mocking. Also note that the mock will be called even inside other functions that use the command.
BeforeAll {
Mock -CommandName Invoke-WebRequest -ParameterFilter { $Method -eq 'GET' } -MockWith { Import-Clixml .\response.xml }
# on this next line our mock will be called instead and will return our prepared object
$response = Invoke-WebRequest -Method 'GET' -Uri 'http://worldclockapi.com/api/json/utc/now'
$responseContent = $response.Content | ConvertFrom-Json
Write-Output $responseContent
}
I think Daniel's answer is great, but if you are working on a large or shared repository then you just need to be careful about managing those XML files too. Another option, which I use, is to have one large Json file for all your returned objects using real responses. It can be imported in either BeforeAll or BeforeDiscovery depending on how your tests are structured.
The reason for my supplementary answer is really just to cover error responses too, because it is important to have test cases that show how you deal with a REST call failing. Wrapping Invoke-WebRequest in your own function might be useful for returning personalised errors, handling header responses, and having constants for a site name or an allowed set of API paths. Depending on the version of PowerShell, this is how I might handle a 404, for example.
Context " When a path does not exist in the API" {
BeforeAll {
Mock Invoke-WebRequest {
# Use the actual types returned by your function or directly from Invoke-WebRequest.
if ($PSVersionTable.PSEdition -eq "Desktop") {
$WR = New-MockObject -Type 'System.Net.HttpWebResponse'
$Code = [System.Net.HttpStatusCode]::NotFound
# Use Add-Member because StatusCode is a read-only field on HttpWebResponse
$WR | Add-Member -MemberType NoteProperty -Name StatusCode -Value $Code -Force
$Status = [System.Net.WebExceptionStatus]::ProtocolError
$Ex = [System.Net.WebException]::new("404", $null, $Status, $WR)
}
else {
$Message = [System.Net.Http.HttpResponseMessage]::new()
$Message.StatusCode = [System.Net.HttpStatusCode]::NotFound
$Ex = [Microsoft.PowerShell.Commands.HttpResponseException]::new("404", $Message)
}
throw $Ex
} -ParameterFilter {
$Uri -eq "http://worldclockapi.com/api/json/utc/NEVER" -and
$Method -eq "Get"
}
$GetRestTimeParams = #{
Uri = "http://worldclockapi.com/api/json/utc/NEVER"
Method = "Get"
}
}
It " Get-RestTime should not run successfully" {
{ Get-RestTime #GetRestTimeParams } | Should -Throw
}
It " Get-RestTime should throw a 404 error" {
$ShouldParams = #{
# Use the actual types returned by your function or directly from Invoke-WebRequest.
ExceptionType = [System.Net.WebException]
ExpectedMessage = "404: NotFound"
}
{
Get-RestTime #GetRestTimeParams
} | Should -Throw #ShouldParams
}
}

Sqljob, make powershell step fail if file does not contain certain text

This is an attempt to get a specific answer to a possible solution to sqljob, on failed step, retry previous step, but not indefinitely
I have a job, which transfer a file using
powershell -command "Invoke-WebRequest https://someserver.dom/fetch.php -OutFile c:/tmp/data.xml"
Sometime the data from upstream is not proper XML. I just want to "grep" for the final line in the file, if it is not the expected </records> I want the step to fail, so the step is retried a few times before it hard-fails.
The filesize could also be a error-criterium. A failed transfer is a few lines, a successfil is megabytes.
So basically i made a function just for this. Set the amount of attempts and the script to run.
function TryAgain($Attempts = 1, $Scriptblock){
while($Attempts -gt 0){
try{
Invoke-Command -ScriptBlock $Scriptblock
Break
}catch{
$_.Exception
$Attempts -= 1
}
}
}
In this case we want to find out if it is valid XML so we are going to get the .RawContent of the Invoke-WebRequest and test against $(new-object System.Xml.XmlDocument).LoadXml(). If it fails throw a error, if it passes then out the raw content to a file. Errors will not be saved to the file only valid XML
$Site = "https://someserver.dom/fetch.php"
$OutFile = "c:/test/data.xml"
(Invoke-WebRequest $Site).RawContent | %{
try{
$(new-object System.Xml.XmlDocument).LoadXml($_)
$_ | out-file $OutFile
}catch{
throw "Bad XML"
}
}
Now we combine into a final product
$Site = "https://someserver.dom/fetch.php"
$OutFile = "c:/test/data.xml"
function TryAgain($Attempts = 1, $Scriptblock){
while($Attempts -gt 0){
try{
Invoke-Command -ScriptBlock $Scriptblock
Break
}catch{
$_.Exception
$Attempts -= 1
}
}
}
TryAgain -Attempts 3 -Scriptblock {
(Invoke-WebRequest $Site).RawContent | %{
try{
$(new-object System.Xml.XmlDocument).LoadXml($_)
$_ | out-file $OutFile
}catch{
throw "Bad XML"
}
}
}
since you dont need the retries and all the glam here is it in basic
(Invoke-WebRequest https://someserver.dom/fetch.php).RawContent | %{
try{
(new-object System.Xml.XmlDocument).LoadXml($_)
$_ | out-file C:\test\test.xml
}catch{}
}

PowerShell Write-Output not working in try/catch block?

Below is my Powershell environment (via the Get-Host command):
Name : Windows PowerShell ISE Host
Version : 5.1.15063.608
InstanceId : 46c51405-fc6d-4e36-a2ae-09dbd4069710
UI : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture : en-US
CurrentUICulture : en-US
PrivateData : Microsoft.PowerShell.Host.ISE.ISEOptions
DebuggerEnabled : True
IsRunspacePushed : False
Runspace : System.Management.Automation.Runspaces.LocalRunspace
I have a script that makes a REST request to a resource -- everything works fine.
While testing, I changed the URL to be invalid to see if error messages are written to the output file -- this is where I am confused. It appears that the only way to write exceptions to the log is by using Write-Host. When I use Write-Output the exception messages are NEVER written to the log file. Below is the code I am using:
function rest_request( $method, $uri )
{
# Code here create #req dictionary
try {
$json = Invoke-RestMethod #req
} catch {
# If using Write-Output - Nothing is written to output file???
# MUST use Write-Host for error info to be included in output file - WHY?
Write-Host "* REST Request Error:"
Write-Host $error[0] | Format-List -force
$json = $null
}
return $json
}
function do_something()
{
Write-Output "Start..."
# other functions called here but removed for clarity.
# The other functions contain Write-Output statements
# which are showing in the log file.
# Issue REST request with invalid URL to cause exception ...
$json = rest_request "Get" "https://nowhere.com/rest/api"
Write-Output "Done."
}
do_something | Out-File "D:\temp\log_file.txt" -Append
According to this blog post: using Write-Host should be avoided, and yet, it seems that Write-Host is the only way I can get error messages in the output file.
My question is: How do you write a function that makes a REST request and returns the result if successful, but if an error occurs, write the error message to an output file and return $null?
As I am fairly green when it comes to PowerShell, any advice would be appreciated.
Ok - I gave up completely on redirection and instead wrote a function that writes the parameter to the log file, as follows:
function log_write
{
param
(
[Parameter(ValueFromPipeline=$true)] $piped
)
$piped | Out-File $log_file -Append
}
Note: $log_file is set at initialization as I needed a new log file name for each day of the week.
Then, I replaced ALL Write-Output / Write-Host with log_write, i.e.:
log_write "* REST Request Error:"
$s = $error[0] | Format-List -force
log_write $s
Works like a charm and exceptions messages are written to the log file without any of the previous issues.
As far as I know Write-Host only writes to the host, nothing else.
In your example, do_something does not return anything, so it's no wonder you don't get anything in the log file.
Here is how you could do it, but many different approaches exist:
function Send-RESTRequest {
Param($method, $uri, $logFile)
# Code here create #req dictionary
try {
# make sure exception will be caught
$json = Invoke-RestMethod #req -ErrorAction Stop
} catch {
# send exception details to log file
$_ | Out-File $logFile -Append
$json = $null
}
return $json
}
function Do-Something {
# Issue REST request with invalid URL to cause exception ...
$json = Send-RESTRequest -method "Get" -uri "https://nowhere.com/rest/api" -logFile "D:\temp\log_file.txt"
}
Do-Something

How do you get the response from a 404 page requested from powershell

I have to call an API exposed by TeamCity that will tell me whether a user exists. The API url is this: http://myteamcityserver.com:8080/httpAuth/app/rest/users/monkey
When called from the browser (or fiddler), I get the following back:
Error has occurred during request processing (Not Found).
Error: jetbrains.buildServer.server.rest.errors.NotFoundException: No user can be found by username 'monkey'.
Could not find the entity requested. Check the reference is correct and the user has permissions to access the entity.
I have to call the API using powershell. When I do it I get an exception and I don't see the text above. This is the powershell I use:
try{
$client = New-Object System.Net.WebClient
$client.Credentials = New-Object System.Net.NetworkCredential $TeamCityAgentUserName, $TeamCityAgentPassword
$teamCityUser = $client.DownloadString($url)
return $teamCityUser
}
catch
{
$exceptionDetails = $_.Exception
Write-Host "$exceptionDetails" -foregroundcolor "red"
}
The exception:
System.Management.Automation.MethodInvocationException: Exception calling "DownloadString" with "1" argument(s): "The remote server returned an error: (404) Not Found." ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadString(Uri address)
at CallSite.Target(Closure , CallSite , Object , Object )
--- End of inner exception stack trace ---
at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
I need to be able to check that the page is returned contains the text described above. This way I know whether I should create a new user automatically or not.
I could just check for 404, but my fear is that if the API is changed and the call really returns a 404, then I would be none the wiser.
Change your catch clause to catch the more specific WebException, then you can use the Response property on it to get the status code:
{
#...
}
catch [System.Net.WebException]
{
$statusCode = [int]$_.Exception.Response.StatusCode
$html = $_.Exception.Response.StatusDescription
}
BrokenGlass gave the answer, but this might help:
try
{
$URI='http://8bit-museum.de/notfound.htm'
$HTTP_Request = [System.Net.WebRequest]::Create($URI)
"check: $URI"
$HTTP_Response = $HTTP_Request.GetResponse()
# We then get the HTTP code as an integer.
$HTTP_Status = [int]$HTTP_Response.StatusCode
}
catch [System.Net.WebException]
{
$statusCode = [int]$_.Exception.Response.StatusCode
$statusCode
$html = $_.Exception.Response.StatusDescription
$html
}
$HTTP_Response.Close()
Response:
check: http://8bit-museum.de/notfound.htm
404
Not Found
another approach:
$URI='http://8bit-museum.de/notfound.htm'
try {
$HttpWebResponse = $null;
$HttpWebRequest = [System.Net.HttpWebRequest]::Create("$URI");
$HttpWebResponse = $HttpWebRequest.GetResponse();
if ($HttpWebResponse) {
Write-Host -Object $HttpWebResponse.StatusCode.value__;
Write-Host -Object $HttpWebResponse.GetResponseHeader("X-Detailed-Error");
}
}
catch {
$ErrorMessage = $Error[0].Exception.ErrorRecord.Exception.Message;
$Matched = ($ErrorMessage -match '[0-9]{3}')
if ($Matched) {
Write-Host -Object ('HTTP status code was {0} ({1})' -f $HttpStatusCode, $matches.0);
}
else {
Write-Host -Object $ErrorMessage;
}
$HttpWebResponse = $Error[0].Exception.InnerException.Response;
$HttpWebResponse.GetResponseHeader("X-Detailed-Error");
}
if i understand the question then $ErrorMessage = $Error[0].Exception.ErrorRecord.Exception.Message contains the errormessage you are looking for.
(source: Error Handling in System.Net.HttpWebRequest::GetResponse() )
Another simple example, hope this helps:
BEGIN
{
# set an object to store results
$queries = New-Object System.Collections.ArrayList
Function Test-Website($Site)
{
try
{
# check the Site param passed in
$request = Invoke-WebRequest -Uri $Site
}
catch [System.Net.WebException] # web exception
{
# if a 404
if([int]$_.Exception.Response.StatusCode -eq 404)
{
$request = [PSCustomObject]#{Site=$site;ReturnCode=[int]$_.Exception.Response.StatusCode}
}
else
{
# set a variable to set a value available to automate with later
$request = [PSCustomObject]#{Site=$site;ReturnCode='another_thing'}
}
}
catch
{
# available to automate with later
$request = [PSCustomObject]#{Site=$site;ReturnCode='request_failure'}
}
# if successful as an invocation and has
# a StatusCode property
if($request.StatusCode)
{
$siteURI = $Site
$response = $request.StatusCode
}
else
{
$response = $request.ReturnCode
}
# return the data
return [PSCustomObject]#{Site=$Site;Response=$response}
}
}
PROCESS
{
# test all the things
$nullTest = Test-Website -Site 'http://www.Idontexist.meh'
$nonNullTest = Test-Website -Site 'https://www.stackoverflow.com'
$404Test = Test-Website -Site 'https://www.stackoverflow.com/thispagedoesnotexist'
# add all the things to results
$queries.Add($nullTest) | Out-Null
$queries.Add($nonNullTest) | Out-Null
$queries.Add($404Test) | Out-Null
# show the info
$queries | Format-Table
}
END{}
Output:
Site Response
---- --------
http://www.Idontexist.meh another_thing
https://www.stackoverflow.com 200
https://www.stackoverflow.com/thispagedoesnotexist 404
You could try using the Internet Explorer COM object instead. It allows you to check the browser return codes and navigate the HTML object model.
Note: I've found that you need to run this from an elevated PowerShell prompt in order to maintain the COM object definition.
$url = "http://myteamcityserver.com:8080/httpAuth/app/rest/users/monkey"
$ie = New-Object -ComObject InternetExplorer.Application
Add this to See the browser
$ie.visibility = $true
Navigate to the site
$ie.navigate($url)
This will pause the script until the page fully loads
do { start-sleep -Milliseconds 250 } until ($ie.ReadyState -eq 4)
Then verify your URL to make sure it's not an error page
if ($ie.document.url -ne $url) {
Write-Host "Site Failed to Load" -ForegroundColor "RED"
} else {
[Retrieve and Return Data]
}
You can navigate HTML Object model via $ie.document. Using Get-Member and HTML methods such as GetElementsByTagName() or GetElementById().
If credentials are an issue, build this into a function then use Invoke-Command with the -Credentials parameter to define your logon information.