RunSpacePool rebooting computers does not record success - powershell

I'm working on a script that will reboot a list of servers at a later time using the below script scheduled in Task Scheduler.
The script uses RunSpacePools to process multiple servers but what seems to happen is my script fails to record the successful reboot.
I checked a server rebooted using this script, and found an event ID 1074 from USER32 which logs the Shutdown/Restart event but the log I create for record keeping says Reboot Failed.
I'm just not sure how to begin troubleshooting this problem. Any assistance is appreciated!
param([Parameter(Mandatory=$true)][string]$InputServerList)
#Script block for runspaces
$ScriptBlock = {
Param([Parameter(Mandatory=$true)][string]$server)
$ErrorActionPreference = "SilentlyContinue"
Try{
$LastReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
Restart-Computer -ComputerName $server -Force
#New loop with counter, exit script if server did not reboot.
$max = 60;$i = 0
DO{
IF($i -gt $max){
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}#end custom object
$subject="Warning: $server Did not Reboot! Please Check!"
#$SendTo = <Email List>
$body = #"
LastRebootTime = $LastReboot
Script waited 10 minutes for $server to reboot.
"#
#Send-MailMessage <details>
exit}#exit script and log failed to reboot.
$i++
Start-Sleep -Seconds 10
}#end DO
#At this point, server went offline. Script waits 10 minutes for server to come online.
While (Test-path "\\$server\c$")
$max = 60;$i = 0
DO{
IF($i -gt $max){
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$body = #"
LastRebootTime = $LastReboot
Script waited 10 minutes for $server to come back online, please investigate.
"#
$subject="Warning!!!!: $server Did not Come Online! Please Check!"
#Send-MailMessage <details>
exit}#exit script and log failed to come online.
$i++
Start-Sleep -Seconds 10
}#end DO
While (-not(Test-path "\\$server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
[PsCustomObject]#{
"Server" = $server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = "$CurrentReboot"
}
}#End Try.
Catch{
$errMsg = $_.Exception.Message.ToString().Trim()
#"$server : Failed with $errMsg"
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$errMsg"
"CurrentRebootTime" = "FailedToReboot!"
}#end custom object
}
}#end script block
#Define computers to work with.
$Computers = Get-Content $InputServerList
#Create Runspace pool
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(30,30)
$RunspacePool.Open()
$Jobs =
foreach ( $Computer in $Computers )
{
$Job = [powershell]::Create().
AddScript($ScriptBlock).
AddArgument($Computer)
$Job.RunspacePool = $RunspacePool
[PSCustomObject]#{
Pipe = $Job
Result = $Job.BeginInvoke()
}
}
Write-Host 'Working..' -NoNewline
Do {
Start-Sleep -Seconds 5
} While ( $Jobs.Result.IsCompleted -contains $false)
Write-Host "Finished."
$(ForEach ($Job in $Jobs)
{ $Job.Pipe.EndInvoke($Job.Result) }) |
Export-Csv -Path "D:\Scripts\Reboot-Async-Results.csv" -NoTypeInformation -Append
#clean up runspace
$RunspacePool.Close()
$RunspacePool.Dispose()

To solve this, I added a flag that is set for rebootCheck true or rebootCheck false in the [PSCustomObject] output. The last thing the script does it check this flag and perform a final check that determines if the Reboot Failed email is needed.
$counter = 0
do{$counter++
Start-Sleep -Seconds 1
IF($counter -ge 600){#"Waited too long"
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
"ReCheckReboot" = $true
}#end custom logging object
exit
}#end if.
}#end do loop. #Break RebootSentWait continues script on next line.
until (-not (Test-Connection -ComputerName $server -Count 1 -Quiet))
Then the variable, named ReCheckReboot is checked for true/false using an IF statement before the script ends.

Related

Power Shell server live ping status

Good Day!
I work with multiple clients and one of my clients have their own toll that displays live server ping stats and this is very helpful during patching activity because of followed reboots etc. I want to design a tool like this or any other tool or means where I can see live ping stats of servers in my Domain. Can I do it using Power Shell by any chance? Is there a way that I could refresh my Power Shell page or the output HTML page which updates the ping stats of respective servers please! I have no clue about programming languages but I want to do this as my pet project and I cannot ask my client about all these details. Help is very much appreciated!
Thanks guys, I have tried below two codes:
param(
[int]$waitseconds = 3
)
while($true) {
$servers = Get-Content .\servers.txt
$collection = $()
foreach ($server in $servers)
{
$status = #{ "ServerName" = $server; "TimeStamp" = (Get-Date -f s) }
if (Test-Connection $server -Count 1 -ea 0 -Quiet)
{
$status["Results"] = "Up"
}
else
{
$status["Results"] = "Down"
}
New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
$collection += $serverStatus
}
$collection | out-gridview; sleep $waitseconds
}
This gives me output in a grid view and it also successfully refreshes the output but the new refreshed output is being presented in a new grid view window. I cannot afford opening numerous windows. So I have tried below code:
param(
[int]$waitseconds = 3
)
while($true) {
$servers = Get-Content .\servers.txt
$collection = $()
foreach ($server in $servers)
{
$status = #{ "ServerName" = $server; "TimeStamp" = (Get-Date -f s) }
if (Test-Connection $server -Count 1 -ea 0 -Quiet)
{
$status["Results"] = "Up"
}
else
{
$status["Results"] = "Down"
}
New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
$collection += $serverStatus
}
Start-Process -FilePath powershell.exe -ArgumentList "-WindowStyle -NoNewWindow -Command &{ $collection | out-gridview; sleep $waitseconds; exit }" -Wait
}
This code is not at all giving gridview output, it simply displays ping stats in shell itself but able to refresh.
I have tried below code too with -NoNewWWindow at the end and this gives output one time and then closes:
param(
[int]$waitseconds = 3
)
while($true) {
$servers = Get-Content .\servers.txt
$collection = $()
foreach ($server in $servers)
{
$status = #{ "ServerName" = $server; "TimeStamp" = (Get-Date -f s) }
if (Test-Connection $server -Count 1 -ea 0 -Quiet)
{
$status["Results"] = "Up"
}
else
{
$status["Results"] = "Down"
}
New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
$collection += $serverStatus
}
Start-Process -FilePath powershell.exe -ArgumentList "-WindowStyle -Command &{ $collection | out-gridview; sleep $waitseconds; exit }" -Wait -NoNewWindow
}
Help me with any modifications to my 1st code so that the grid view does not open for every refresh please!
Have you tried something like this:
$collection | out-gridview
sleep $waitseconds
(Get-Process -id $PID).CloseMainWindow()

Split-Pipeline parallel jobs script won't finish the last job

So I've been using Split-Pipeline module for some time now, and what bothers me about it, rarely does it really finish. I always get stuck with the last job in the queue hanging and have to either stop the script or kill ISE:
VERBOSE: Split-Pipeline: Jobs = 1; Load = End; Queue = 0
example: 300 servers, running a scan for hotfixes on them using split-pipeline. I'm using pretty standard parameters:
$servers | Split-Pipeline -Verbose -Count 10 { process {#some code for scanning#}}
So 10 jobs each loaded with 30 servers, first lets say 250 servers are scanned really fast, then it slows down a little and when the last job remains only, it never finishes...
Anyone experienced something similar? I've tested the same on several machines and it's always the same so I don't think it's related to the machine running the script.
EDIT: heres the code
$servers = (Get-Clipboard).trim() #server1..100
$KB = (Get-Clipboard).trim() -split ', ' #KB4022714, KB4022717, KB4022718, KB4022722, KB4022727, KB4022883, KB4022884, KB4022887, KB4024402, KB4025339, KB4025342, KB4021903, KB4021923, KB4022008, KB4022010, KB4022013, KB3217845
$servers | Split-Pipeline -Verbose -Variable KB -Count 10 { process {
$hash = [ordered]#{}
try
{
$hash.Hostname = $_
$os = gwmi win32_operatingsystem -ComputerName $_ -ErrorAction Stop
$hash.OS = $os.Caption
$hash.Architecture = $os.OSArchitecture
$today = [Management.ManagementDateTimeConverter]::ToDateTime($os.LocalDateTime)
$hash.LastReboot = [Management.ManagementDateTimeConverter]::ToDateTime($os.LastBootUpTime)
$hash.DaysSinceReboot = ($today - $hash.LastReboot).Days
}
catch
{
$hash.OS = $Error[0]
$hash.Architecture = 'N/A'
$hash.LastReboot = 'N/A'
$hash.DaysSinceReboot = 'N/A'
}
try
{
$hash.PendingReboot = icm -cn $_ {
if (Get-ChildItem "HKLM:\Software\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending" -EA SilentlyContinue) { return $true }
if (Get-Item "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired" -EA SilentlyContinue) { return $true }
if (Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager" -Name PendingFileRenameOperations -Ea SilentlyContinue) { return $true }
try
{
$util = [wmiclass]"\\.\root\ccm\clientsdk:CCM_ClientUtilities"
$status = $util.DetermineIfRebootPending()
if(($status -ne $null) -and $status.RebootPending)
{
return $true
}
}
catch{}
return $false
}
}
catch
{
$hash.PendingReboot = 'N/A'
}
try
{
$hotfix = Get-HotFix -ComputerName $_ -Id $KB -ErrorAction Stop
if ($hotfix)
{
$hash.Hotfix = $hotfix.HotFixID -join ','
}
else
{
$hash.Hotfix = "No Hotfix from the list applied"
}
}
catch
{
$hash.Hotfix = $Error[0]
}
$obj = New-Object -TypeName PSObject -Property $hash
$obj | Export-Csv c:\temp\hotfixes.csv -NoTypeInformation -Append -Force
}
}

Making variables visible inside Powershell workflow

I have five variables defined before a workflow that need to be available to the workflow, but I can't find out how to do it.
Putting the variables inside the workflow makes them visible, but that causes an issue with the CSV import that means extra properties are added to the object relating to the workflow that I don't want.
Code as follows:
$source = 'C:\Users\Koozer\a place\'
$rotateParams = 90, 90, 270
$cropParams = #(64, 64), (32, 0)
$images = Import-Csv "${source}images.csv"
$colNames = $images[0].psobject.properties.Name
Workflow StitchCropWorkflow {
foreach -parallel ($imageSet in $images) {
$magickRotParams = ''
$n = 0
foreach ($image in $colNames) {
$magickRotParams += '`( '''+$source+$imageSet.($image)+''' -rotate '+$rotateParams[$n]+' `) '
$n++
}
$finfo = [io.fileinfo]$imagePathSets[0]
$command = 'magick '+$magickRotParams+' +append -crop '+$cropParams[0][0]+'x'+$cropParams[0][1]+'+'+$cropParams[1][0]+'+'+$cropParams[1][1]+' +repage '''+$finfo.DirectoryName+'\'+$finfo.BaseName+'_stitch_crop'+$finfo.Extension+''''
echo $command
Invoke-Expression $command
}
}
StitchCropWorkflow
You can pass parameters to a workflow like you would do it for a function:
$source = 'C:\Users\Koozer\a place\'
$rotateParams = 90, 90, 270
$cropParams = #(64, 64), (32, 0)
$images = Import-Csv "${source}images.csv"
$colNames = $images[0].psobject.properties.Name
Workflow StitchCropWorkflow {
param (
$source,
$rotateParams,
$cropParams,
$images,
$colNames
)
# your code here
}
StitchCropWorkflow $source $rotateParams $cropParams $images $colNames
If you define a variable in a script containing a workflow, to access the variable within the workflow the syntax is $using:variable
As an example, here is a powershell script containing a workflow with a single param.
param([string]$ComputerName)
$VerbosePreference = "Continue"
workflow Start-Reboot {
param($server)
Write-Verbose "Input server is $server"
InlineScript{
New-Item -Path C:\Scripts\Logging\Start-Reboot-Single.log -ItemType File -ErrorAction SilentlyContinue | Out-Null
}
Sequence{
InlineScript
{
Try
{
Get-WmiObject -ComputerName $using:server -Class Win32_Service -Filter "Name='HealthService'" | Invoke-WmiMethod -Name PauseService | Out-Null
$operatingSystem = Get-WmiObject Win32_OperatingSystem -ComputerName $using:server -ErrorAction stop
$LastReboot = [Management.ManagementDateTimeConverter]::ToDateTime($operatingSystem.LastBootUpTime).ToString().Trim()
Write-Verbose "Last reboot time for $using:server is $LastReboot"
Write-Verbose "Here we restart $using:server, for testing no reboot is done"
Restart-Computer -ComputerName $using:server -Wait -For Wmi -Force
$OSRecheck = Get-WmiObject Win32_OperatingSystem -ComputerName $using:server -ErrorAction stop
$CurrentReboot = [Management.ManagementDateTimeConverter]::ToDateTime($OSRecheck.LastBootUpTime).ToString().Trim()
$props = [Ordered]#{
Server=$using:server
LastReboot=$LastReboot
CurrentReboot=$CurrentReboot
}
} Catch {
Write-Verbose "Oh no, problem with $using:server"
$rnd = Get-Random -Minimum 1 -Maximum 5
Start-Sleep -Seconds $rnd
$err = $_.Exception.GetType().FullName
$props = [Ordered]#{
Server=$using:server
LastReboot=$err
CurrentReboot=$null
}
} Finally {
$object = New-Object -TypeName PSObject -Property $props
$random = Get-Random -Minimum 2 -Maximum 15
Start-Sleep -Seconds $random
Write-Output $object | Out-File -Append -FilePath C:\Scripts\Logging\Start-Reboot-Single.log
}
}#inline end
}#sequence end
}#end workflow block
Start-Reboot -server $ComputerName

Creating workflow for parallel scheduled server reboots with logging

I'm currently using the following code to schedule a server reboot. This works pretty well for a handful of servers but becomes a problem when there are many servers (over 80) because Register-ScheduledJob takes a long time per server.
$user = Get-Credential -UserName $env:USERNAME -Message "UserName/password for scheduled Reboot"
$trigger = New-JobTrigger -once -at $date
$script = [ScriptBlock]::Create("D:\Scripts\Scheduled-Reboot-Single.ps1 -server $server")
Register-ScheduledJob -Name $server -Credential $user -Trigger $trigger -ScriptBlock $script
My research pointed to using workflow and foreach -parallel.
The problem I run into is accurate logging. My log file is created but the columns are not ordered correctly.
workflow Do-ScheduledReboot{
Param([string[]]$servers)
foreach -parallel($server in $servers) {
InlineScript {
try {
$LastReboot = Get-EventLog -ComputerName $using:server -LogName system |
Where-Object {$_.EventID -eq '6005'} |
Select -ExpandProperty TimeGenerated |
select -first 1
#New loop with counter, exit script if server did not reboot.
$max = 20; $i = 0
do {
if ($i -gt $max) {
$hash = #{
"Server" = $using:server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
exit
}#exit script and log failed to reboot.
$i++
Start-Sleep -Seconds 15
} while (Test-path "\\$using:server\c$")
$max = 20; $i = 0
do {
if ($i -gt $max) {
$hash = #{
"Server" = $using:server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
exit
}#exit script and log failed to come online.
$i++
Start-Sleep -Seconds 15
} while (-not(Test-path "\\$using:server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $using:server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
$hash = #{
"Server" = $using:server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = "$CurrentReboot"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
} catch {
$errMsg = $_.Exception
"Failed with $errMsg"
}#end catch
}#end inline script
}#end foreach parallel
}#end workflow
$mylist = gc D:\Servers.txt
Do-ScheduledReboot -servers $mylist
Create ordered hashtables:
$hash = [ordered]#{
'Server' = $using:server
'Status' = ...
"LastRebootTime" = ...
'CurrentRebootTime' = ...
}

Server reboot script with timed loop

I'm putting together a reboot script used for scheduling server reboots.
My goal is to improve functionality and error handling by using a loop with counter to break/exit the script if the server:
1) Does not restart. For example While (Test-path "\\$server\c$")
2) Does not come online. For example While (-not(Test-path "\\$server\c$"))
In addition, I'm trying to add logging to a CSV file if any of the above conditions are TRUE. Otherwise the script continues and creates the CSV with the previous reboot time stamp along with current reboot time stamp.
My current attempt does not seem to correctly exit the script and log the failure and honestly I'm a little unsure of how to test this before rebooting servers.
Param([Parameter(Mandatory=$true)][string]$server)
$ErrorActionPreference = "SilentlyContinue"
Try{
$LastReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
(Invoke-WmiMethod -ComputerName $server -Path "Win32_Service.Name='HealthService'" -Name PauseService).ReturnValue | Out-Null
Restart-Computer -ComputerName $server -Force
#New loop with counter, exit script if server did not reboot.
$max = 20;$i = 0
DO{
IF($i -gt $max){
$hash = #{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = $LastReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
;exit}#exit script and log failed to reboot.
$i++
"Wait for server to reboot"
Start-Sleep -Seconds 30
}#end DO
While (Test-path "\\$server\c$")
$max = 15;$i = 0
DO{
IF($i -gt $max){
$hash = #{
"Server" = $server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = $LastReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
;exit}#exit script and log failed to reboot.
$i++
"Wait for [$server] to come online"
Start-Sleep -Seconds 30
}#end DO
While (-not(Test-path "\\$server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
$hash = #{
"Server" = $server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = $CurrentReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
}#End Try.
Catch{
$errMsg = $_.Exception
"Failed with $errMsg"
}
In order to break out of a loop, use the break keyword, not return.
Assign $true to $test inside the if block immediately before break:
$test = false
while ($limit -lt $max){
Write-Host "Reboot Request Sent, waiting for server to reboot"
if(Test-Path \\Server1\C$){
Write-Host "Reboot Successful, continue script"
$test = $true
break
}
$limit++
Start-Sleep -Seconds 10
}
The switch statement you've shown is syntactically valid, but kinda weird, since it's the exact equivalent of a standard if/else:
if($test)
{
Write-Host "Reboot succeeded!"
}
else
{
Write-Host "Reboot failed!"
}
After trial and error I believe I have something working.
This example will exit the script after two seconds
#restart-computer $server
$max = 2;$i = 0
DO{
IF($i -gt $max){"Server failed to reboot!";exit}#exit script and log failed to reboot.
$i++
"Wait for server to reboot"
Start-Sleep -Seconds 1
}#end DO
While (Test-path "\\$server\c$")
"Server rebooted, continue script"