Server reboot script with timed loop - powershell

I'm putting together a reboot script used for scheduling server reboots.
My goal is to improve functionality and error handling by using a loop with counter to break/exit the script if the server:
1) Does not restart. For example While (Test-path "\\$server\c$")
2) Does not come online. For example While (-not(Test-path "\\$server\c$"))
In addition, I'm trying to add logging to a CSV file if any of the above conditions are TRUE. Otherwise the script continues and creates the CSV with the previous reboot time stamp along with current reboot time stamp.
My current attempt does not seem to correctly exit the script and log the failure and honestly I'm a little unsure of how to test this before rebooting servers.
Param([Parameter(Mandatory=$true)][string]$server)
$ErrorActionPreference = "SilentlyContinue"
Try{
$LastReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
(Invoke-WmiMethod -ComputerName $server -Path "Win32_Service.Name='HealthService'" -Name PauseService).ReturnValue | Out-Null
Restart-Computer -ComputerName $server -Force
#New loop with counter, exit script if server did not reboot.
$max = 20;$i = 0
DO{
IF($i -gt $max){
$hash = #{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = $LastReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
;exit}#exit script and log failed to reboot.
$i++
"Wait for server to reboot"
Start-Sleep -Seconds 30
}#end DO
While (Test-path "\\$server\c$")
$max = 15;$i = 0
DO{
IF($i -gt $max){
$hash = #{
"Server" = $server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = $LastReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
;exit}#exit script and log failed to reboot.
$i++
"Wait for [$server] to come online"
Start-Sleep -Seconds 30
}#end DO
While (-not(Test-path "\\$server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
$hash = #{
"Server" = $server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = $CurrentReboot
}
$newRow = New-Object PsObject -Property $hash
Export-Csv c:\Scripts\RebootCheck.csv -InputObject $newrow -Append -Force
}#End Try.
Catch{
$errMsg = $_.Exception
"Failed with $errMsg"
}

In order to break out of a loop, use the break keyword, not return.
Assign $true to $test inside the if block immediately before break:
$test = false
while ($limit -lt $max){
Write-Host "Reboot Request Sent, waiting for server to reboot"
if(Test-Path \\Server1\C$){
Write-Host "Reboot Successful, continue script"
$test = $true
break
}
$limit++
Start-Sleep -Seconds 10
}
The switch statement you've shown is syntactically valid, but kinda weird, since it's the exact equivalent of a standard if/else:
if($test)
{
Write-Host "Reboot succeeded!"
}
else
{
Write-Host "Reboot failed!"
}

After trial and error I believe I have something working.
This example will exit the script after two seconds
#restart-computer $server
$max = 2;$i = 0
DO{
IF($i -gt $max){"Server failed to reboot!";exit}#exit script and log failed to reboot.
$i++
"Wait for server to reboot"
Start-Sleep -Seconds 1
}#end DO
While (Test-path "\\$server\c$")
"Server rebooted, continue script"

Related

Using while -or operator cause infinite loop

Hello I am trying to add -or operator to my Script to avoid and infinite script run using a counter but I don´t get that works, if I don´t use -or over while It´s work ok but if I add the program script, run infinite
Import-Module WebAdministration
$server = "server1"
$evtsrc = "AppPool Resurrector"
$loopcounter = 0
if ( [System.Diagnostics.EventLog]::SourceExists($evtsrc) -eq $false){
New-EventLog -LogName Application -Source $evtsrc
}
while((Get-ChildItem IIS:\AppPools | where {$_.state -eq "Stopped"}).count -gt 0 -or ($loopcounter -lt 20))
{
$appPools = Get-ChildItem -Path 'IIS:\AppPools' | where {$_.state -eq "Stopped"} | Foreach-Object {$_.Name}
foreach ($appPoolName in $appPools) {
Start-WebAppPool $appPoolName
Write-Host ($appPoolName)
Start-Sleep -Seconds 10
$loopcounter++
if ((Get-WebAppPoolState -Name $appPoolName).Value -eq "Started"){
Write-EventLog -LogName Application -Message "Start Application Pool `"$appPoolName`" because SMB Client is connected again." -Source $evtsrc -EntryType Warning -EventId 666 -Category 0
}
else{
write-Host ("test")
}
}
}
You should use -and instead of -or here, like Mathias R. Jessen already commented.
Also, I would suggest using Splatting the arguments for the Write-EventLog cmdlet to make the code more readable and easier to maintain.
Something like below:
$loopcounter = 0
$appPools = Get-ChildItem -Path 'IIS:\AppPools' | Where-Object {$_.State -eq "Stopped"}
# as long as there are stopped app pools AND we have not yet reached the max for the loop counter
while ($appPools.Count -gt 0 -and ($loopcounter -lt 20)) {
$appPools | Foreach-Object {
$appPoolName = $_.Name
Write-Host "Starting '$appPoolName'"
Start-WebAppPool $appPoolName
Start-Sleep -Seconds 10 # that's a mighty long wait..
$loopcounter++
if ((Get-WebAppPoolState -Name $appPoolName).Value -eq 'Started') {
$splat = #{
'LogName' = 'Application'
'Message' = "Start Application Pool '$appPoolName' because SMB Client is connected again."
'Source' = $evtsrc
'EntryType' = 'Warning'
'EventId' = 666
'Category' = 0
}
Write-EventLog #splat
}
else{
write-Host ("test: Counter = $loopcounter")
}
}
# refresh the app pools variable
$appPools = Get-ChildItem -Path 'IIS:\AppPools' | Where-Object {$_.State -eq "Stopped"}
}
Hope that helps

Function within a Function - Powershell

OK I am going to try to explain this as best as I can. What started out as a simple script has turned into a huge mess and now I cannot figure out how to get it working. I have been coming here for answers for some time so maybe you guys can help.
What I am trying to do is a import a list of systems and check to see if they are online. If they are online they go in one list and if not they go in another.
foreach ($server in $servers) {
if (Test-Connection $server -Count 1 -ea 0 -Quiet) {
Write-Host "$server Is Up" -ForegroundColor Green
$server | out-file -Append $liveSystems -ErrorAction SilentlyContinue
} else {
Write-Host "$server Is Down" -ForegroundColor Red
$server | out-file -Append $inactive -ErrorAction SilentlyContinue
}
}
From there I check to see if the application I need installed is on the systems. That is where things start to go off-track. When I run the function to process the $liveSystems file all I get is the last line of the file (or the same system over and over) and not each system as it should be.
function Is-Installed( $program ) {
$x86 = ((Get-ChildItem "HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall") |
Where-Object { $_.GetValue( "DisplayName" ) -like "*$program*" } ).Length -gt 0;
$x64 = ((Get-ChildItem "HKLM:\Software\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall") |
Where-Object { $_.GetValue( "DisplayName" ) -like "*$program*" } ).Length -gt 0;
}
$program
function process-file1 {
param($filename)
Get-Content $filename -PipelineVariable line | ForEach-Object {
Is-Installed -program "My_Service"
if (Is-Installed -eq "True") {
Write-Host "$server has agent installed" -ForegroundColor Green
$server | Out-File $installed -ErrorAction SilentlyContinue
}
else
{
Write-Host "$server does not have agent installed" -ForegroundColor Red
$server | Out-File -Append $notInstalled -ErrorAction SilentlyContinue
}
}
}
process-file1 -filename $liveSystems
Once I can get the systems to process through the list of installed and not installed I am trying to take the list of installed systems and check which ones have the service running and which ones do not.
$array = #()
foreach($i in (gc $installed)) {
$svc = Get-Service my_service -ComputerName $i -ea "0"
$obj = New-Object psobject -Property #{
Name = $svc.name
Status = $svc.status
Computer = $i
}
$array += $obj
}
$array | Select Computer,Name,Status | Export-Csv -Path $resultsFile -
NoTypeInformation
Last but not least I run through that list of running and not running and attempt to start the service on systems that are not running.
function process-CSVfile2 {
param($filename)
Import-Csv $filename |
ForEach-Object -PipelineVariable object {
if($_.Status -eq "Running") {
Write-Host "Your Service is currently Running on" $_.Computer
}
if($_.Status -eq "Stopped") {
$serviceName = 'my_service'
$service = Get-CimInstance Win32_Service -ComputerName $_.Computer -Filter "Name=$serviceName"
$service.Start()
$service.WaitForStatus("Started",'00:00:30')
Start-Sleep 10
}
}
}
Several of these blocks run separately but when put together they will not run. I can't seem to get past the second block where it just looks at the same line over and over.
In addition there is a piece I have been trying to get working that would install the application on systems that do not have the service installed but that is not working either but I will save that for a different time.
If anyone can help me with this I would really appreciate it. After 3 days of trying to get it running I am at my wits end.
I'd create objects and properties instead of files with computers online etc...
Something like:
$Computers=New-Object -TypeName System.Collections.ArrayList
$Servers = #(Get-Content -path c:\servers.txt)
$Servers = $Servers | ? {$_} | select-object -uniqe |ForEach-Object {$_.TrimEnd()}
$Servers|ForEach-Object {
$tempobj=New-Object -TypeName PSObject
$tempobj | Add-Member -type NoteProperty -name Name -value $_
$tempobj | Add-Member -type NoteProperty -name isOnline -value $FALSE
$tempobj | Add-Member -type NoteProperty -name Installed -value $FALSE
$tempobj | Add-Member -type NoteProperty -name serviceRunning -value $FALSE
[void]$Computers.Add($tempobj)
then You could work on array (no need for additional files)
$Computers|Where-Object {$_.isOnline -eq $TRUE}
etc

RunSpacePool rebooting computers does not record success

I'm working on a script that will reboot a list of servers at a later time using the below script scheduled in Task Scheduler.
The script uses RunSpacePools to process multiple servers but what seems to happen is my script fails to record the successful reboot.
I checked a server rebooted using this script, and found an event ID 1074 from USER32 which logs the Shutdown/Restart event but the log I create for record keeping says Reboot Failed.
I'm just not sure how to begin troubleshooting this problem. Any assistance is appreciated!
param([Parameter(Mandatory=$true)][string]$InputServerList)
#Script block for runspaces
$ScriptBlock = {
Param([Parameter(Mandatory=$true)][string]$server)
$ErrorActionPreference = "SilentlyContinue"
Try{
$LastReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
Restart-Computer -ComputerName $server -Force
#New loop with counter, exit script if server did not reboot.
$max = 60;$i = 0
DO{
IF($i -gt $max){
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}#end custom object
$subject="Warning: $server Did not Reboot! Please Check!"
#$SendTo = <Email List>
$body = #"
LastRebootTime = $LastReboot
Script waited 10 minutes for $server to reboot.
"#
#Send-MailMessage <details>
exit}#exit script and log failed to reboot.
$i++
Start-Sleep -Seconds 10
}#end DO
#At this point, server went offline. Script waits 10 minutes for server to come online.
While (Test-path "\\$server\c$")
$max = 60;$i = 0
DO{
IF($i -gt $max){
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$body = #"
LastRebootTime = $LastReboot
Script waited 10 minutes for $server to come back online, please investigate.
"#
$subject="Warning!!!!: $server Did not Come Online! Please Check!"
#Send-MailMessage <details>
exit}#exit script and log failed to come online.
$i++
Start-Sleep -Seconds 10
}#end DO
While (-not(Test-path "\\$server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
[PsCustomObject]#{
"Server" = $server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = "$CurrentReboot"
}
}#End Try.
Catch{
$errMsg = $_.Exception.Message.ToString().Trim()
#"$server : Failed with $errMsg"
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$errMsg"
"CurrentRebootTime" = "FailedToReboot!"
}#end custom object
}
}#end script block
#Define computers to work with.
$Computers = Get-Content $InputServerList
#Create Runspace pool
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(30,30)
$RunspacePool.Open()
$Jobs =
foreach ( $Computer in $Computers )
{
$Job = [powershell]::Create().
AddScript($ScriptBlock).
AddArgument($Computer)
$Job.RunspacePool = $RunspacePool
[PSCustomObject]#{
Pipe = $Job
Result = $Job.BeginInvoke()
}
}
Write-Host 'Working..' -NoNewline
Do {
Start-Sleep -Seconds 5
} While ( $Jobs.Result.IsCompleted -contains $false)
Write-Host "Finished."
$(ForEach ($Job in $Jobs)
{ $Job.Pipe.EndInvoke($Job.Result) }) |
Export-Csv -Path "D:\Scripts\Reboot-Async-Results.csv" -NoTypeInformation -Append
#clean up runspace
$RunspacePool.Close()
$RunspacePool.Dispose()
To solve this, I added a flag that is set for rebootCheck true or rebootCheck false in the [PSCustomObject] output. The last thing the script does it check this flag and perform a final check that determines if the Reboot Failed email is needed.
$counter = 0
do{$counter++
Start-Sleep -Seconds 1
IF($counter -ge 600){#"Waited too long"
[PsCustomObject]#{
"Server" = $server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
"ReCheckReboot" = $true
}#end custom logging object
exit
}#end if.
}#end do loop. #Break RebootSentWait continues script on next line.
until (-not (Test-Connection -ComputerName $server -Count 1 -Quiet))
Then the variable, named ReCheckReboot is checked for true/false using an IF statement before the script ends.

Creating workflow for parallel scheduled server reboots with logging

I'm currently using the following code to schedule a server reboot. This works pretty well for a handful of servers but becomes a problem when there are many servers (over 80) because Register-ScheduledJob takes a long time per server.
$user = Get-Credential -UserName $env:USERNAME -Message "UserName/password for scheduled Reboot"
$trigger = New-JobTrigger -once -at $date
$script = [ScriptBlock]::Create("D:\Scripts\Scheduled-Reboot-Single.ps1 -server $server")
Register-ScheduledJob -Name $server -Credential $user -Trigger $trigger -ScriptBlock $script
My research pointed to using workflow and foreach -parallel.
The problem I run into is accurate logging. My log file is created but the columns are not ordered correctly.
workflow Do-ScheduledReboot{
Param([string[]]$servers)
foreach -parallel($server in $servers) {
InlineScript {
try {
$LastReboot = Get-EventLog -ComputerName $using:server -LogName system |
Where-Object {$_.EventID -eq '6005'} |
Select -ExpandProperty TimeGenerated |
select -first 1
#New loop with counter, exit script if server did not reboot.
$max = 20; $i = 0
do {
if ($i -gt $max) {
$hash = #{
"Server" = $using:server
"Status" = "FailedToReboot!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
exit
}#exit script and log failed to reboot.
$i++
Start-Sleep -Seconds 15
} while (Test-path "\\$using:server\c$")
$max = 20; $i = 0
do {
if ($i -gt $max) {
$hash = #{
"Server" = $using:server
"Status" = "FailedToComeOnline!"
"LastRebootTime" = "$LastReboot"
"CurrentRebootTime" = "FailedToReboot!"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
exit
}#exit script and log failed to come online.
$i++
Start-Sleep -Seconds 15
} while (-not(Test-path "\\$using:server\c$"))
$CurrentReboot = Get-EventLog -ComputerName $using:server -LogName system | Where-Object {$_.EventID -eq '6005'} | Select -ExpandProperty TimeGenerated | select -first 1
$hash = #{
"Server" = $using:server
"Status" = "RebootSuccessful"
"LastRebootTime" = $LastReboot
"CurrentRebootTime" = "$CurrentReboot"
}
$newRow = New-Object PsObject -Property $hash
$rnd = Get-Random -Minimum 5 -Maximum 40
Start-Sleep -Seconds $rnd
Export-Csv D:\workflow-results.csv -InputObject $newrow -Append -Force
} catch {
$errMsg = $_.Exception
"Failed with $errMsg"
}#end catch
}#end inline script
}#end foreach parallel
}#end workflow
$mylist = gc D:\Servers.txt
Do-ScheduledReboot -servers $mylist
Create ordered hashtables:
$hash = [ordered]#{
'Server' = $using:server
'Status' = ...
"LastRebootTime" = ...
'CurrentRebootTime' = ...
}

Powershell script to run on all computers in domain - wait until computer is online

I have pieced together the following PowerShell script that deletes a file from the public desktop of every machine in an OU on our domain, and then copies a replacement file back to replace it. It works well, except for the machines that are offline. What would be the best way to have the script run on a machine once it comes online? My best guess is to have it put any offline machine in a list, then re-run a few hours later for the computers on that list. Is there a better way?
$DESTINATION = "c$\Users\Public\Desktop"
$REMOVE = "ComputerName"
$strFilter = "computer"
$objDomain = New-Object System.DirectoryServices.DirectoryEntry
$objSearcher = New-Object System.DirectoryServices.DirectorySearcher
$objSearcher.SearchRoot = "LDAP://OU=MyOU,DC=Domain,DC=com"
$objSearcher.SearchScope = "Subtree"
$objSearcher.PageSize = 1000
$objSearcher.Filter = "(objectCategory=$strFilter)"
$colResults = $objSearcher.FindAll()
foreach ($i in $colResults)
{
$objComputer = $i.GetDirectoryEntry()
$REMOVE = $objComputer.Name
#Ping system to see if it's on
$rtn = Test-Connection -CN $REMOVE -Count 2 -BufferSize 16 -Quiet
IF($rtn -match 'True') {
Remove-Item "\\$REMOVE\$DESTINATION\SparksNET.url" -Recurse
Copy-Item "\\spd3\PD IT stuff\Software\Desktop Icons\mySparks.website" "\\$REMOVE\$DESTINATION"
}
ELSE {
}
}
Everyone is 100% correct, you should do this through GPO. However, if you crazily want to do it through PowerShell, you could do it the way I have outlined below. I threw your code into it - so it might be a tad messy.
Start off by pulling in the list of PC's and then send them all off to a job...
if(Test-Path "c:\LogPath"){
}else {
mkdir "c:\LogPath"
}
$time=Get-Date -Format s
$date=Get-Date -Format F
"LOG FILE CREATED - $date" | Add-Content $log
$objDomain = New-Object System.DirectoryServices.DirectoryEntry
$objSearcher = New-Object System.DirectoryServices.DirectorySearcher
$objSearcher.SearchRoot = "LDAP://OU=MyOU,DC=Domain,DC=com"
$objSearcher.SearchScope = "Subtree"
$objSearcher.PageSize = 1000
$objSearcher.Filter = "(objectCategory=$strFilter)"
$colResults = $objSearcher.FindAll()
foreach($Obj in $colResults){
$objComputer = $Obj.GetDirectoryEntry()
$remotePC = $objComputer.Name
#Imports all functions used in script
. "C:\PathToYourJobScript.ps1"
#Creating Jobs
$jobs = (get-job -state running | Measure-Object).count
Get-job -State Completed | Remove-Job
while($jobs -ge 5)
{
#get currently running jobs after restart
get-job -state running | %{write-host $_.PSBeginTime} | ?{($_.PSBeginTime - $(Get-Date)).Minutes -gt 10} | Remove-Job
Write-host "Currently running maximum threads at: $jobs `n Sleeping 5 seconds" -ForegroundColor DarkYellow
start-sleep -seconds 5
$jobs = (get-job -state running | Measure-Object).count
}
Write-host "`t`tCreating Thread for $remotePC" -ForegroundColor Yellow
start-job $DeleteFile -ArgumentList $remotePC -Name $remotePC
Receive-Job $remotePC
"$remotePC; Starting Job at; $time" | Add-Content $log
}
In your actual job script PS1, wrap your code above into one function
$DeleteFile={Param(
[Parameter(Mandatory=$true)]
[string]$remotePC
)
<#
.SYNOPSIS
.DESCRIPTION
.NOTES
#>
##Globals
$DESTINATION = "c$\Users\Public\Desktop"
$REMOVE = "ComputerName"
$strFilter = "computer"
#Ping system to see if it's on
if(Test-Connection -ComputerName $remotePC -Count 2 -BufferSize 16 -ErrorAction SilentlyContinue){
Remove-Item "\\$REMOVE\$DESTINATION\SparksNET.url" -Recurse
Copy-Item "\\spd3\PD IT stuff\Software\Desktop Icons\mySparks.website" "\\$REMOVE\$DESTINATION"
"$remotePC; Removal Complete; $time" | Add-Content $log
}
ELSE
{
do {Start-Sleep -Seconds 300; "$remotePC; Ping Fail; $time" | Add-Content $log}
until (Test-Connection -ComputerName $remotePC -Count 2 -BufferSize 16 -ErrorAction SilentlyContinue)
Remove-Item "\\$REMOVE\$DESTINATION\SparksNET.url" -Recurse
Copy-Item "\\spd3\PD IT stuff\Software\Desktop Icons\mySparks.website" "\\$REMOVE\$DESTINATION"
"$remotePC; Removal Complete; $time" | Add-Content $log
}
}
This will check every 5 minutes if the PC is online and wait until it gets a response to proceed. Once an item is tossed into a job, you loose site of its progress and would want to log accordingly so you know its position.