Creating workflow for parallel scheduled server reboots with logging - powershell

I'm currently using the following code to schedule a server reboot. This works pretty well for a handful of servers but becomes a problem when there are many servers (over 80) because Register-ScheduledJob takes a long time per server.
$user = Get-Credential -UserName $env:USERNAME -Message "UserName/password for scheduled Reboot"
$trigger = New-JobTrigger -once -at $date
$script = [ScriptBlock]::Create("D:\Scripts\Scheduled-Reboot-Single.ps1 -server $server")
Register-ScheduledJob -Name $server -Credential $user -Trigger $trigger -ScriptBlock $script
My research pointed to using workflow and foreach -parallel.
The problem I run into is accurate logging. My log file is created but the columns are not ordered correctly.
workflow Do-ScheduledReboot{
Param([string[]]$servers)
foreach -parallel($server in $servers) {
InlineScript {
try {
$LastReboot = Get-EventLog -ComputerName $using:server -LogName system |
Where-Object {$_.EventID -eq '6005'} |
Select -ExpandProperty TimeGenerated |
select -first 1
#New loop with counter, exit script if server did not reboot.
$max = 20; $i = 0
do {
if ($i -gt $max) {
$hash = #{
"Server" = $using:server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
exit
}#exit script and log failed to reboot.
$i++
Start-Sleep -Seconds 15
} while (Test-path "\\$using:server\c$")
$max = 20; $i = 0
do {
if ($i -gt $max) {
$hash = #{
"Server" = $using:server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
exit
}#exit script and log failed to come online.
$i++
Start-Sleep -Seconds 15
} while (-not(Test-path "\\$using:server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $using:server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
$hash = #{
"Server" = $using:server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = "$CurrentReboot"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
} catch {
$errMsg = $_.Exception
"Failed with $errMsg"
}#end catch
}#end inline script
}#end foreach parallel
}#end workflow
$mylist = gc D:\Servers.txt
Do-ScheduledReboot -servers $mylist

Create ordered hashtables:
$hash = [ordered]#{
'Server' = $using:server
'Status' = ...
"LastRebootTime" = ...
'CurrentRebootTime' = ...
}

Related

Powershell runspace unexpected output (misunderstanding?)

$ScriptBlock = {
Param (
$Vm,
$vc,
$session)
$VCenter = Connect-VIServer -Server $vc -session $session -force
$status = Get-VM $Vm
Write-host $status.Name
if ($status.PowerState -eq "PoweredOff")
{
$Clone = '_2.2'
$NewVM = $Vm + $Clone
Write-host $NewVM.Name
}
}
$Results = #()
$Throttle = 5 #threads
$sesssionstate = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$sesssionstate.ImportPSModule("Vmware.VimAutomation.Core")
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, $Throttle, $sesssionstate, $host)
$RunspacePool.Open()
$Jobs = #()
$Vms = gc .\vmlist.txt
Foreach ($Vm in $Vms)
{
#Start-Sleep -Minutes 1
$Job = [powershell]::Create().AddScript($ScriptBlock).AddArgument($Vm).AddArgument($con.Name).AddArgument($con.SessionId)
$Job.RunspacePool = $RunspacePool
$Jobs += New-Object PSObject -Property #{
Job = $Job
Result = $Job.BeginInvoke()
}
}
while ($Jobs -Contains $false) {}
foreach($Job in $jobs)
{
$results += $Job.Job.EndInvoke($Job.Result)
$Job.Job.Dispose()
}
$results
Input File:
INTCLWK8
INTCLWK82
INTCLWK83
Output:
INTCLWK8
INTCLWK8_2.2
INTCLWK82
INTCLWK82_2.2
INTCLWK83
INTCLWK83_2.2
2nd run:
INTCLWK8
INTCLWK8_2.2
INTCLWK82_2.2
INTCLWK83
INTCLWK82
INTCLWK83_2.2
Why is the output different at run? This caused confusion to write multithread.
The output should be always like the below.
INTCLWK8
INTCLWK8_2.2
INTCLWK82
INTCLWK82_2.2
INTCLWK83
INTCLWK83_2.2

Making variables visible inside Powershell workflow

I have five variables defined before a workflow that need to be available to the workflow, but I can't find out how to do it.
Putting the variables inside the workflow makes them visible, but that causes an issue with the CSV import that means extra properties are added to the object relating to the workflow that I don't want.
Code as follows:
$source = 'C:\Users\Koozer\a place\'
$rotateParams = 90, 90, 270
$cropParams = #(64, 64), (32, 0)
$images = Import-Csv "${source}images.csv"
$colNames = $images[0].psobject.properties.Name
Workflow StitchCropWorkflow {
foreach -parallel ($imageSet in $images) {
$magickRotParams = ''
$n = 0
foreach ($image in $colNames) {
$magickRotParams += '`( '''+$source+$imageSet.($image)+''' -rotate '+$rotateParams[$n]+' `) '
$n++
}
$finfo = [io.fileinfo]$imagePathSets[0]
$command = 'magick '+$magickRotParams+' +append -crop '+$cropParams[0][0]+'x'+$cropParams[0][1]+'+'+$cropParams[1][0]+'+'+$cropParams[1][1]+' +repage '''+$finfo.DirectoryName+'\'+$finfo.BaseName+'_stitch_crop'+$finfo.Extension+''''
echo $command
Invoke-Expression $command
}
}
StitchCropWorkflow
You can pass parameters to a workflow like you would do it for a function:
$source = 'C:\Users\Koozer\a place\'
$rotateParams = 90, 90, 270
$cropParams = #(64, 64), (32, 0)
$images = Import-Csv "${source}images.csv"
$colNames = $images[0].psobject.properties.Name
Workflow StitchCropWorkflow {
param (
$source,
$rotateParams,
$cropParams,
$images,
$colNames
)
# your code here
}
StitchCropWorkflow $source $rotateParams $cropParams $images $colNames
If you define a variable in a script containing a workflow, to access the variable within the workflow the syntax is $using:variable
As an example, here is a powershell script containing a workflow with a single param.
param([string]$ComputerName)
$VerbosePreference = "Continue"
workflow Start-Reboot {
param($server)
Write-Verbose "Input server is $server"
InlineScript{
New-Item -Path C:\Scripts\Logging\Start-Reboot-Single.log -ItemType File -ErrorAction SilentlyContinue | Out-Null
}
Sequence{
InlineScript
{
Try
{
Get-WmiObject -ComputerName $using:server -Class Win32_Service -Filter "Name='HealthService'" | Invoke-WmiMethod -Name PauseService | Out-Null
$operatingSystem = Get-WmiObject Win32_OperatingSystem -ComputerName $using:server -ErrorAction stop
$LastReboot = [Management.ManagementDateTimeConverter]::ToDateTime($operatingSystem.LastBootUpTime).ToString().Trim()
Write-Verbose "Last reboot time for $using:server is $LastReboot"
Write-Verbose "Here we restart $using:server, for testing no reboot is done"
Restart-Computer -ComputerName $using:server -Wait -For Wmi -Force
$OSRecheck = Get-WmiObject Win32_OperatingSystem -ComputerName $using:server -ErrorAction stop
$CurrentReboot = [Management.ManagementDateTimeConverter]::ToDateTime($OSRecheck.LastBootUpTime).ToString().Trim()
$props = [Ordered]#{
Server=$using:server
LastReboot=$LastReboot
CurrentReboot=$CurrentReboot
}
} Catch {
Write-Verbose "Oh no, problem with $using:server"
$rnd = Get-Random -Minimum 1 -Maximum 5
Start-Sleep -Seconds $rnd
$err = $_.Exception.GetType().FullName
$props = [Ordered]#{
Server=$using:server
LastReboot=$err
CurrentReboot=$null
}
} Finally {
$object = New-Object -TypeName PSObject -Property $props
$random = Get-Random -Minimum 2 -Maximum 15
Start-Sleep -Seconds $random
Write-Output $object | Out-File -Append -FilePath C:\Scripts\Logging\Start-Reboot-Single.log
}
}#inline end
}#sequence end
}#end workflow block
Start-Reboot -server $ComputerName

RunSpacePool rebooting computers does not record success

I'm working on a script that will reboot a list of servers at a later time using the below script scheduled in Task Scheduler.
The script uses RunSpacePools to process multiple servers but what seems to happen is my script fails to record the successful reboot.
I checked a server rebooted using this script, and found an event ID 1074 from USER32 which logs the Shutdown/Restart event but the log I create for record keeping says Reboot Failed.
I'm just not sure how to begin troubleshooting this problem. Any assistance is appreciated!
param([Parameter(Mandatory=$true)][string]$InputServerList)
#Script block for runspaces
$ScriptBlock = {
Param([Parameter(Mandatory=$true)][string]$server)
$ErrorActionPreference = "SilentlyContinue"
Try{
$LastReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
Restart-Computer -ComputerName $server -Force
#New loop with counter, exit script if server did not reboot.
$max = 60;$i = 0
DO{
IF($i -gt $max){
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}#end custom object
$subject="Warning: $server Did not Reboot! Please Check!"
#$SendTo = <Email List>
$body = #"
LastRebootTime = $LastReboot
Script waited 10 minutes for $server to reboot.
"#
#Send-MailMessage <details>
exit}#exit script and log failed to reboot.
$i++
Start-Sleep -Seconds 10
}#end DO
#At this point, server went offline. Script waits 10 minutes for server to come online.
While (Test-path "\\$server\c$")
$max = 60;$i = 0
DO{
IF($i -gt $max){
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$body = #"
LastRebootTime = $LastReboot
Script waited 10 minutes for $server to come back online, please investigate.
"#
$subject="Warning!!!!: $server Did not Come Online! Please Check!"
#Send-MailMessage <details>
exit}#exit script and log failed to come online.
$i++
Start-Sleep -Seconds 10
}#end DO
While (-not(Test-path "\\$server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
[PsCustomObject]#{
"Server" = $server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = "$CurrentReboot"
}
}#End Try.
Catch{
$errMsg = $_.Exception.Message.ToString().Trim()
#"$server : Failed with $errMsg"
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$errMsg"
"CurrentRebootTime" = "FailedToReboot!"
}#end custom object
}
}#end script block
#Define computers to work with.
$Computers = Get-Content $InputServerList
#Create Runspace pool
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(30,30)
$RunspacePool.Open()
$Jobs =
foreach ( $Computer in $Computers )
{
$Job = [powershell]::Create().
AddScript($ScriptBlock).
AddArgument($Computer)
$Job.RunspacePool = $RunspacePool
[PSCustomObject]#{
Pipe = $Job
Result = $Job.BeginInvoke()
}
}
Write-Host 'Working..' -NoNewline
Do {
Start-Sleep -Seconds 5
} While ( $Jobs.Result.IsCompleted -contains $false)
Write-Host "Finished."
$(ForEach ($Job in $Jobs)
{ $Job.Pipe.EndInvoke($Job.Result) }) |
Export-Csv -Path "D:\Scripts\Reboot-Async-Results.csv" -NoTypeInformation -Append
#clean up runspace
$RunspacePool.Close()
$RunspacePool.Dispose()
To solve this, I added a flag that is set for rebootCheck true or rebootCheck false in the [PSCustomObject] output. The last thing the script does it check this flag and perform a final check that determines if the Reboot Failed email is needed.
$counter = 0
do{$counter++
Start-Sleep -Seconds 1
IF($counter -ge 600){#"Waited too long"
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
"ReCheckReboot" = $true
}#end custom logging object
exit
}#end if.
}#end do loop. #Break RebootSentWait continues script on next line.
until (-not (Test-Connection -ComputerName $server -Count 1 -Quiet))
Then the variable, named ReCheckReboot is checked for true/false using an IF statement before the script ends.

Resolve-DnsName inside Test-Connection

I was wondering how I could return the Resolve-DnsName output from my Test-Connection script and add it to the CSV I created.
I like to capture the Name, Type, TTL, Section from that please.
Only invoke the Resolve-DnsName when the ping is not successful.
$servers = Get-Content "servers.txt"
$collection = $()
foreach ($server in $servers)
{
$status = #{ "ServerName" = $server; "TimeStamp" = (Get-Date -f s) }
$result = Test-Connection $server -Count 1 -ErrorAction SilentlyContinue
if ($result)
{
$status.Results = "Up"
$status.IP = ($result.IPV4Address).IPAddressToString
}
else
{
$status.Results = "Down"
$status.IP = "N/A"
$status.DNS = if (-not(Resolve-DnsName -Name $server -ErrorAction SilentlyContinue))
{
Write-Output -Verbose "$server -- Not Resolving"
}
else
{
"$server resolving"
}
}
New-Object -TypeName PSObject -Property $status -OutVariable serverStatus
$collection += $serverStatus
}
$collection | Export-Csv -LiteralPath .\ServerStatus3.csv -NoTypeInformation
but nothing new is added to the CSV.
You ran into a PowerShell gotcha. PowerShell determines the columns displayed in tabular/CSV output from the first object processed. If that object doesn't have a property DNS that column won't be shown in the output, even if other objects in the list do have it. If other objects don't have properties that were present in the first object they will be displayed as empty values.
Demonstration:
PS C:\> $a = (New-Object -Type PSObject -Property #{'a'=1; 'b'=2}),
>> (New-Object -Type PSObject -Property #{'a'=3; 'b'=4; 'c'=5}),
>> (New-Object -Type PSObject -Property #{'b'=6; 'c'=7})
>>
PS C:\> $a | Format-Table -AutoSize
a b
- -
1 2
3 4
6
PS C:\> $a[1..2] | Format-Table -AutoSize
c b a
- - -
5 4 3
7 6
If you want to generate tabular output always create your objects uniformly with the same set of properties. Choosing sensible defaults even allows you to reduce your total codebase.
$collection = foreach ($server in $servers) {
$status = New-Object -Type PSObject -Property #{
'ServerName' = $server
'TimeStamp' = Get-Date -f s
'Results' = 'Down'
'IP' = 'N/A'
'HasDNS' = [bool](Resolve-DnsName -Name $server -EA SilentlyContinue)
}
$result = Test-Connection $server -Count 1 -EA SilentlyContinue
if ($result) {
$status.Results = 'Up'
$status.IP = ($result.IPV4Address).IPAddressToString
}
$status
}

Server reboot script with timed loop

I'm putting together a reboot script used for scheduling server reboots.
My goal is to improve functionality and error handling by using a loop with counter to break/exit the script if the server:
1) Does not restart. For example While (Test-path "\\$server\c$")
2) Does not come online. For example While (-not(Test-path "\\$server\c$"))
In addition, I'm trying to add logging to a CSV file if any of the above conditions are TRUE. Otherwise the script continues and creates the CSV with the previous reboot time stamp along with current reboot time stamp.
My current attempt does not seem to correctly exit the script and log the failure and honestly I'm a little unsure of how to test this before rebooting servers.
Param([Parameter(Mandatory=$true)][string]$server)
$ErrorActionPreference = "SilentlyContinue"
Try{
$LastReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
(Invoke-WmiMethod -ComputerName $server -Path "Win32_Service.Name='HealthService'" -Name PauseService).ReturnValue | Out-Null
Restart-Computer -ComputerName $server -Force
#New loop with counter, exit script if server did not reboot.
$max = 20;$i = 0
DO{
IF($i -gt $max){
$hash = #{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = $LastReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
;exit}#exit script and log failed to reboot.
$i++
"Wait for server to reboot"
Start-Sleep -Seconds 30
}#end DO
While (Test-path "\\$server\c$")
$max = 15;$i = 0
DO{
IF($i -gt $max){
$hash = #{
"Server" = $server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = $LastReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
;exit}#exit script and log failed to reboot.
$i++
"Wait for [$server] to come online"
Start-Sleep -Seconds 30
}#end DO
While (-not(Test-path "\\$server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
$hash = #{
"Server" = $server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = $CurrentReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
}#End Try.
Catch{
$errMsg = $_.Exception
"Failed with $errMsg"
}
In order to break out of a loop, use the break keyword, not return.
Assign $true to $test inside the if block immediately before break:
$test = false
while ($limit -lt $max){
Write-Host "Reboot Request Sent, waiting for server to reboot"
if(Test-Path \\Server1\C$){
Write-Host "Reboot Successful, continue script"
$test = $true
break
}
$limit++
Start-Sleep -Seconds 10
}
The switch statement you've shown is syntactically valid, but kinda weird, since it's the exact equivalent of a standard if/else:
if($test)
{
Write-Host "Reboot succeeded!"
}
else
{
Write-Host "Reboot failed!"
}
After trial and error I believe I have something working.
This example will exit the script after two seconds
#restart-computer $server
$max = 2;$i = 0
DO{
IF($i -gt $max){"Server failed to reboot!";exit}#exit script and log failed to reboot.
$i++
"Wait for server to reboot"
Start-Sleep -Seconds 1
}#end DO
While (Test-path "\\$server\c$")
"Server rebooted, continue script"