Kill a process if it reaches x% of CPU usage - powershell

I want to stop processes that are running higher than 14% CPU usage.
$process = get-process
foreach ($proc in (Get-WmiObject Win32_Processor)){
if($proc.numberofcores -eq $null){
$cores++
}else{
$cores = $cores + $proc.numberofcores
}
}
foreach($name in $process){
$processName = $name.processName
foreach($hog in $processName){
$cpuusage = [Math]::round(((((Get-Counter "\Process($processName)\%
Processor Time" -MaxSamples 2).Countersamples)[0].CookedValue)/$cores),2)
if($cpuusage -gt 14){
Stop-Process -Name $processName
}
}
}
I am getting the following as an error, an nothing else. I expect the Idle(0) not to work, but nothing else is being killed.
Stop-Process : Cannot stop process "Idle (0)" because of the following
error: Access is denied
At line:14 char:17
+ Stop-Process -Name $processName
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (System.Diagnostics.Process
(Idle):Process) [Stop-Process], ProcessCommandException
+ FullyQualifiedErrorId :
CouldNotStopProcess,Microsoft.PowerShell.Commands.StopProcessCommand
I have tried to replace the $processName variables within the second foreach loop to $hog and I still get the same error.
After reading #JosefZ answer I got something that satisfies what I require for my class. Posting it here for reference;
$process = get-process
foreach ($pro in $process){
$name = $pro.ProcessName
$CpuCores = (Get-WmiObject -Class Win32_Processor).NumberOfCores
$CpuValue = ((Get-Counter "\Process($name)\% Processor Time").CounterSamples.CookedValue)/$CpuCores
$percent = [Decimal]::Round($CpuValue, 3)
if($percent -ge 15){
Stop-Process -Name $name
$wshell = New-Object -ComObject Wscript.Shell
$wshell.Popup("Process $name was using more than $percent % CPU. We have eliminated it.",0,"Ok",0x1)
}
}

Note that Performance counters are often protected by access control lists (ACLs). To get all available performance counters, open Windows PowerShell with the "Run as administrator" option and Your ability to stop processes depends on your permissions.
The following script was used for debugging with CPU-time consuming wmic path cim_datafile and antivirus full scan:
Set-StrictMode -Version latest
$process = get-process
$cores = 0
foreach ($proc in (Get-WmiObject Win32_Processor)){
if($proc.numberofcores -eq $null){
$cores++
}else{
$cores = $cores + $proc.numberofcores
}
}
### $cores
foreach($name in $process){
$processName = $name.processName
if ( $processName -notmatch "^Idle" ) {
foreach($hog in $processName){
$cpuusage = -1
$cpuusage = [Math]::round(((((Get-Counter "\Process($processName)\% Processor Time" -MaxSamples 2).Countersamples)[0].CookedValue)/$cores),2)
if ($cpuusage -gt 14) {
Stop-Process -Name $processName -PassThru -ErrorAction Continue
### "{0} {1} {2} {3}" -f '+', $cpuusage, $name.Id, $processName
} else {
if($cpuusage -ne 0){
### "{0} {1} {2} {3}" -f '-', $cpuusage, $name.Id, $processName
}
}
}
}
}
Dirty solution: Stop-Process -Name $processName -ErrorAction SilentlyContinue -PassThru
"Idle" means "inactive" (not operating or being used). When the "System Idle Process" is at 100 %, that means nothing is using your CPU resources.
Read Stopping Processes (Stop-Process) at MSDN:
Windows PowerShell gives you flexibility for listing processes, but
what about stopping a process?
The Stop-Process cmdlet takes a Name or Id to specify a process
you want to stop. Your ability to stop processes depends on your
permissions. Some processes cannot be stopped. For example, if you
try to stop the idle process, you get an error:
PS> Stop-Process -Name Idle
Stop-Process : Process 'Idle (0)' cannot be stopped due to the following error:
Access is denied
At line:1 char:13
+ Stop-Process <<<< -Name Idle
Read Get-Help 'Stop-Process' -Online
Outputs
None, System.Diagnostics.Process
This cmdlet returns a System.Diagnostics.Process object that represents the stopped process, if you specify the PassThru
parameter. Otherwise, this cmdlet does not generate any output.

Couple of additional pointers for you. Firstly you should use Win32_ComputerSystem and NumberOfLogicalProcessors instead of Win32_Processor with NumberOfCores. The reason being, that the performance counters account for HyperThreading on systems that have it, so your calculation based off the physical cores will give you a processor time value twice the size of what it should be. For example, on my machine with 6 Physical Cores, I have 12 logical processors due to HyperThreading. Using your original calculation, a process using 8% CPU would be reported as 16% and thus be incorrectly stopped. The Win32_ComputerSystem version will return all logical processors across all physical CPU's, so multi socketed servers would also be calculated correctly.
C:\WINDOWS\system32> (Get-WmiObject win32_processor).NumberofCores
6
C:\WINDOWS\system32> (Get-WmiObject win32_computersystem).Numberoflogicalprocessors
12
Second, you should be stopping the process by it's ID not by Name, as this will have unintended consequences. A simple example is Chrome, which has a process per tab, all of which have the Name Chrome. So a single tab could be having issues causing high CPU, but your script calling Stop-Process -Name Chrome would close all instances of Chrome, including the ones that aren't doing anything.
The following example script resolves both of these issues:
#Get all cores, which includes virtual cores from hyperthreading
$cores = (Get-WmiObject Win32_ComputerSystem).NumberOfLogicalProcessors
#Get all process with there ID's, excluding processes you can't stop.
$processes = ((Get-Counter "\Process(*)\ID Process").CounterSamples).where({$_.InstanceName -notin "idle","_total","system"})
#Get cpu time for all processes
$cputime = $processes.Path.Replace("id process", "% Processor Time") | get-counter | select -ExpandProperty CounterSamples
#Get the processes with above 14% utilisation.
$highUsage = $cputime.where({[Math]::round($_.CookedValue / $cores,2) -gt 14})
# For each high usage process, grab it's process ID from the processes list, by matching on the relevant part of the path
$highUsage |%{
$path = $_.Path
$id = $processes.where({$_.Path -like "*$($path.Split('(')[1].Split(')')[0])*"}) | select -ExpandProperty CookedValue
Stop-Process -Id $id -Force -ErrorAction SilentlyContinue
}
Note that the usage of the .where( syntax requires PowerShell 5 or above. This script also has the added bonus of being much faster than calling Get-Counter in a foreach loop.

Related

PowerShell Invoke-Command timeout if no answer from frozen AD-Server

I'm writing a PowerShell-Script that reads all shares from all AD-Servers and outputs them into a csv-file. At the same time the script is saving all occuring errors and outputs them into an error-log. The script will be run as a weekly task. When I run the script, all goes well until it gets to a server that has frozen. In that case, the script will just run forever because it gets no answer from the server.
Now I need to add some sort of timeout that skips a server after it doesn't recieve an answer for a specific amount of time. How would I do that with my existing code?
My Code:
$computers = (Get-Content C:\PowerShell\Shares\serverlist.txt).ForEach({
if(-not [string]::IsNullOrWhiteSpace($_))
{
"$_.domain.com"
}
})
$remoteCode = {
Get-SmbShare | Where-Object Path | Get-Acl |
Select-Object -Property "PSChildName", "Path", "Group", "AccessToString"
}
$results = Invoke-Command -ComputerName $computers -ScriptBlock $remoteCode 2>&1
$errors, $good = $results.Where({$_ -is [System.Management.Automation.ErrorRecord]}, 'Split')
$good | Sort-Object PSComputerName | Select-Object "PSComputerName", "PSChildName", "Path", "Group", #{ Name = "AccessToString"; Expression = { $_.AccessToString -replace("268435456", "FullControl") -replace("-1610612736", "ReadAndExecute")}} | export-csv -path C:\PowerShell\Shares\shares.csv -NoTypeInformation -delimiter ";"
$errors.Exception.Message | Set-Content $error_logfile -Encoding Unicode
NOTE: This answer is pretty much useless, in an ideal world, -OperationTimeout would do what it's name implies, however, as the helpful comment from mklement0 states:
Unfortunately, the OperationTimeout session option doesn't do what its name suggests: see GitHub issue #15696. Implementing an actual operation-duration timeout is the subject of GitHub proposal #5434, which suggest adding a -Timeout parameter to Invoke-Command.
If you feel this would be a great implementation for future versions of PowerShell, consider up-voting his proposal!
You could use PSSessionOption with a Operation Timeout and Open Timeout below the default values (3 minutes):
See the Parameter section of New-PSSessionOption documentation:
-OpenTimeout
Determines how long the client computer waits for the session connection to be established. When the interval expires, the command to establish the connection fails.
-OperationTimeout
Determines the maximum time WinRM waits for positive connection tests from a live connection before initiating a connection time-out.
$timeOut = 30000 # => 30 seconds
$psso = New-PSSessionOption -OpenTimeout $timeOut -OperationTimeout $timeOut
$session = (Get-Content C:\PowerShell\Shares\serverlist.txt).ForEach({
if(-not [string]::IsNullOrWhiteSpace($_)) {
try {
New-PSSession -ComputerName "$_.domain.com" -SessionOption $psso
}
catch {
Write-Warning $_.Exception.Message
}
}
})
Then the rest of the script is the same, except for the use of -Session instead -ComputerName for Invoke-Command:
$results = Invoke-Command -Session $session -ScriptBlock $remoteCode 2>&1
And lastly, after you're done with the remote connections, you would need to remove the PSSessions:
Remove-PSSession $session

Running commands as logged on user (remotely)

Thought I would share this quick function I made for myself, feel free to adapt it and improve it according to your needs.
Sometimes you want to run commands as the logged on user of a remote computer.
As you know, some commands show output for the user who runs it and if you run the same command with Invoke-Command, it won't return the user's information, but yours). Get-Printer is an example amongst many others.
There is no easy, quick way of running commands as the logged on user natively without any third-party apps like PsExec or others so I made this quick function that uses VBS, PS1 and Scheduled Task to make it happen.
It runs completly silently for the user (thanks to the VBS) and the output is shown in your console. Please note it assumes the remote computer has a C:\TEMP.
Created in a Windows 10, powershell v 5.1.17763.503 environement.
I don't pretend it's final and perfect, it's the simplest way I found to do what is needed and I just wanted to share it with you guys as it can be very useful!
Check the comments for explanation of the code and feel free to use it as you wish. Please share your version as I'm curious to see people improve it. A good idea would be to make it support multiple computers, but as I said it's a quick function I did I don't have too much time to put into refining it.
That being said, I had no problems using it multiple times as is :)
*Output returned is in form of a string, if you want to have a proper object, add '| ConvertFrom-String' and play with it :)
PLEASE NOTE: The surefire way of grabbing the username of who is currently logged on is via QWINSTA (since Win32_ComputerSystem - Username is only reliable if a user is logged on LOCALLY, it won't be right if a user is using RDP/RemoteDesktop). So this is what I used to grab the username, however, please note that in our french environement the name of the username property in QWINSTA is "UTILISATEUR",so you have to change that to your needs (english or other language) for it to work. If I remember correctly, it's "USERNAME" in english.
On this line:
$LoggedOnUser = (qwinsta /SERVER:$ComputerName) -replace '\s{2,22}', ',' | ConvertFrom-Csv | Where-Object {$_ -like "*Acti*"} | Select-Object -ExpandProperty UTILISATEUR
See code in the answer below.
function RunAsUser {
Param ($ComputerName,$Scriptblock)
#Check that computer is reachable
Write-host "Checking that $ComputerName is online..."
if (!(Test-Connection $ComputerName -Count 1 -Quiet)) {
Write-Host "$ComputerName is offline" -ForegroundColor Red
break
}
#Check that PsRemoting works (test Invoke-Command and if it doesn't work, do 'Enable-PsRemoting' via WMI method).
#*You might have the adjust this one to suit your environement.
#Where I work, WMI is always working, so when PsRemoting isn't, I enable it via WMI first.
Write-host "Checking that PsRemoting is enabled on $ComputerName"
if (!(invoke-command $ComputerName { "test" } -ErrorAction SilentlyContinue)) {
Invoke-WmiMethod -ComputerName $ComputerName -Path win32_process -Name create -ArgumentList "powershell.exe -command Enable-PSRemoting -SkipNetworkProfileCheck -Force" | Out-Null
do {
Start-Sleep -Milliseconds 200
} until (invoke-command $ComputerName { "test" } -ErrorAction SilentlyContinue)
}
#Check that a user is logged on the computer
Write-host "Checking that a user is logged on to $ComputerName..."
$LoggedOnUser = (qwinsta /SERVER:$ComputerName) -replace '\s{2,22}', ',' | ConvertFrom-Csv | Where-Object {$_ -like "*Acti*"} | Select-Object -ExpandProperty UTILISATEUR
if (!($LoggedOnUser) ) {
Write-Host "No user is logged on to $ComputerName" -ForegroundColor Red
break
}
#Creates a VBS file that will run the scriptblock completly silently (prevents the user from seeing a flashing powershell window)
#"
Dim wshell, PowerShellResult
set wshell = CreateObject("WScript.Shell")
Const WindowStyle = 0
Const WaitOnReturn = True
For Each strArg In WScript.Arguments
arg = arg & " " & strArg
Next 'strArg
PowerShellResult = wshell.run ("PowerShell " & arg & "; exit $LASTEXITCODE", WindowStyle, WaitOnReturn)
WScript.Quit(PowerShellResult)
"# | out-file "\\$ComputerName\C$\TEMP\RAU.vbs" -Encoding ascii -force
#Creates a script file from the specified '-Scriptblock' parameter which will be ran as the logged on user by the scheduled task created below.
#Adds 'Start-Transcript and Stop-Transcript' for logging the output.
$Scriptblock = "Start-Transcript C:\TEMP\RAU.log -force" + $Scriptblock + "Stop-Transcript"
$Scriptblock | out-file "\\$ComputerName\C$\TEMP\RAU.ps1" -Encoding utf8 -force
#On the remote computer, create a scheduled task that runs the .ps1 script silently in the user's context (with the help of the vbs)
Write-host "Running task on $ComputerName..."
Invoke-Command -ComputerName $ComputerName -ArgumentList $LoggedOnUser -ScriptBlock {
param($loggedOnUser)
$SchTaskParameters = #{
TaskName = "RAU"
Description = "-"
Action = (New-ScheduledTaskAction -Execute "wscript.exe" -Argument "C:\temp\RAU.vbs C:\temp\RAU.ps1")
Settings = (New-ScheduledTaskSettingsSet -AllowStartIfOnBatteries -DontStopIfGoingOnBatteries -StartWhenAvailable -DontStopOnIdleEnd)
RunLevel = "Highest"
User = $LoggedOnUser
Force = $true
}
#Register and Start the task
Register-ScheduledTask #SchTaskParameters | Out-Null
Start-ScheduledTask -TaskName "RAU"
#Wait until the task finishes before continuing
do {
Write-host "Waiting for task to finish..."
$ScheduledTaskState = Get-ScheduledTask -TaskName "RAU" | Select-Object -ExpandProperty state
start-sleep 1
} until ( $ScheduledTaskState -eq "Ready" )
#Delete the task
Unregister-ScheduledTask -TaskName "RAU" -Confirm:$false
}
Write-host "Task completed on $ComputerName"
#Grab the output of the script from the transcript and remove the header (first 19) and footer (last 5)
$RawOutput = Get-Content "\\$ComputerName\C$\temp\RAU.log" | Select-Object -Skip 19
$FinalOutput = $RawOutput[0..($RawOutput.length-5)]
#Shows output
return $FinalOutput
#Delete the output file and script files
Remove-Item "\\$ComputerName\C$\temp\RAU.log" -force
Remove-Item "\\$ComputerName\C$\temp\RAU.vbs" -force
Remove-Item "\\$ComputerName\C$\temp\RAU.ps1" -force
}
#____________________________________________________
#Example command
#Note: Sometimes Start-Transcript doesn't show the output for a certain command, so if you run into empty output, add: ' | out-host' or '| out-default' at the end of the command not showing output.
$Results = RunAsUser -ComputerName COMP123 -Scriptblock {
get-printer | Select-Object name,drivername,portname | Out-host
}
$Results
#If needed, you can turn the output (which is a string for the moment) to a proper powershell object with ' | ConvertFrom-String'

Loop through servers and output results along with errors

I wrote a simple PowerShell script to retrieve a list of servers' last boot time and output the results to grid view. The results are immediately shown in the grid window but and comes to a short pause whenever a server is not responding to the get command, either due to WMI not running or class not registered. It then displays the error in PS and move to the next server.
Now, the results aren't helpful unless the "not responding" servers are shown in the results windows.
$servers = ('serverx','serverb')
Get-WmiObject -Class Win32_OperatingSystem -ComputerName $servers |
select csname, #{LABEL='LastBootUpTime';EXPRESSION={$_.ConvertToDateTime($_.LastBootupTime)}},
#{LABEL='LocalTime';EXPRESSION={$_.ConvertToDateTime($_.LocalDateTime)}},
#{LABEL='UpTime';EXPRESSION={(Get-Date) - $_.ConvertToDateTime($_.LastBootupTime)}},
#{LABEL='OS';EXPRESSION={$_.Caption}} |
Out-GridView
Errors type shown in PS window in Red:
Get-WmiObject : Class not registered (Exception from HRESULT: 0x80040154 (REGDB_E_CLASSNOTREG)) At line:1 char:12
Get-WmiObject : The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) At line:1 char:12
Edit: How do I can i output the good results along with the server name if the servers that responded with an error?
For your desired result you need to query the servers individually and construct a custom object if the query fails:
$svr = 'serverx'
try {
Get-WmiObject Win32_OperatingSystem -Computer $svr -EA Stop |
select csname, #{n='LocalTime';e={...}},
#{n='UpTime';e={...}}, #{n='OS';e={...}}
} catch {
New-Object -Type PSObject -Property #{
csname = $svr
LocalTime = $null
UpTime = $null
OS = $null
}
}
Run this in a loop
$servers | ForEach-Object {
...
} | Out-GridView
Use background jobs (or something similar) instead of a plain loop to speed up the checks by running them in parallel rather than sequentially. Spawn each check as a job in the background and check for completed jobs in a loop until all jobs have completed. Collect the output from completed jobs.
Here is the full script that loops through the servers, catches non-terminating error and output to a window.
$svr = ('localhost','fail')
$Output = Foreach ($server in $svr)
{
try {
Get-WmiObject Win32_OperatingSystem -ComputerName $server -EA STOP |
select csname, #{n='LocalTime';e={$_.ConverttoDateTime($_.lastbootuptime)}},
#{n='UpTime';e={....}}, #{n='OS';e={"...."}}
} catch {
New-Object -Type PSObject -Property #{
Csname = $server
LocalTime = $null
UpTime = $null
OS = "Error" #$null
}
}
}
$output | Out-GridView

PowerShell Invoke-Command Returns Blank Data?

Been trying to solve this for a bit and can't seem to figure it out.
I have the following script:
$Servers = Get-Content -Path "C:\Utilities_PowerShell\ServerList.txt"
$IISServiceName1 = 'W3SVC'
$IISServiceName2 = 'IISAdmin'
$IISServiceName3 = 'WAS'
$IISarrService = Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3
$IISarrServiceCheck = Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3 -ErrorAction SilentlyContinue -ErrorVariable NoService
function IISServiceStatus # Checks for status of IIS services
{
param (
$IISServiceName1,
$IISServiceName2,
$IISServiceName3,
$IISarrService,
$IISarrServiceCheck
)
if (Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3)
{
Write-Host "Status of IIS service(s) on $env:ComputerName :"
Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3 | Select Name,DisplayName,Status | Format-Table -AutoSize
}
else
{
Write-Host " No IIS service(s) were found..." -foreground "red"
}
}
$Sessions = New-PSSession -ComputerName $Servers
$EndJobs = $Sessions | ForEach-Object {
Invoke-Command -Session $_ -ScriptBlock ${function:IISServiceStatus} -AsJob -ArgumentList $IISServiceName1, $IISServiceName2, $IISServiceName3, $IISarrService, $IISarrServiceCheck | Wait-Job | Receive-Job
Write-Host " "
}
Whenever I run it, all I get is the output of:
Status of IIS service(s) on *PC* :
If I run the function outside of a loop/invoke-command, the results are absolutely perfect. What is wrong with my remote loop?
I've tried putting the variables inside the function, I've tried running invoke-command without the argument list, etc.
Update: 3/17/16
Turns out...if I run my actual script as is, the result of $EndJobs is weird in that it outputs ALL services in one table and then the three IIS services in another table. This would explain why when I run my invoke-command (stopIIS) scriptblock...I had to reboot the whole server because it took all of the services down.
These functions run PERFECTLY when not run via remote/invoke-command.
What the heck...invoke-command is seriously screwing with my stuff!
Anyone have any ideas/tips on how I can run my local script (which works 100%) on a set of servers from a text file without weird issues like this? Is invoke-command the only way?
do you have the same problem if you wrap it all into the script block like this?
$Servers = Get-Content 'C:\Utilities_PowerShell\ServerList.txt'
$Sessions = New-PSSession -ComputerName $Servers
$EndJobs = $Sessions | ForEach-Object {
Invoke-Command -Session $_ -ScriptBlock {
$IISServiceName1 = 'W3SVC'
$IISServiceName2 = 'IISAdmin'
$IISServiceName3 = 'WAS'
$IISarrService = Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3
$IISarrServiceCheck = Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3 -ErrorAction SilentlyContinue -ErrorVariable NoService
function IISServiceStatus { # Checks for status of IIS services
param (
$IISServiceName1,
$IISServiceName2,
$IISServiceName3,
$IISarrService,
$IISarrServiceCheck
)
if (Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3) {
Write-Host "Status of IIS service(s) on $env:ComputerName :"
Get-Service -Name $IISServiceName1,$IISServiceName2,$IISServiceName3 | Select Name,DisplayName,Status | Format-Table -AutoSize
} else {
Write-Host ' No IIS service(s) were found...' -ForegroundColor Red
}
}
IISServiceStatus $IISServiceName1 $IISServiceName2 $IISServiceName3 $IISarrService $IISarrServiceCheck
} -AsJob | Wait-Job | Receive-Job
Write-Host ' '
}
$EndJobs
I'm having a similar issue. I'm using credssp to test 2nd hop auth for an automation for shutting down a production environment cleanly. My script has 3 sections; session setup, the invoke, session teardown. If I run each piece separately, I get output. If I run the whole script, I get blank lines matching the amount of output I get when I run them separately... there's nothing fancy in my invoke (backtick line continuation - I prefer Python's formatting paradigm better than Powershell/C#):
Invoke-Command `
-Session $workingSession `
-ScriptBlock {
get-service *spool* -ComputerName server01
}

Powershell: Run multiple jobs in parralel and view streaming results from background jobs

Overview
Looking to call a Powershell script that takes in an argument, runs each job in the background, and shows me the verbose output.
Problem I am running into
The script appears to run, but I want to verify this for sure by streaming the results of the background jobs as they are running.
Code
###StartServerUpdates.ps1 Script###
#get list of servers to update from text file and store in array
$servers=get-content c:\serverstoupdate.txt
#run all jobs, using multi-threading, in background
ForEach($server in $servers){
Start-Job -FilePath c:\cefcu_it\psscripts\PSPatch.ps1 -ArgumentList $server
}
#Wait for all jobs
Get-Job | Wait-Job
#Get all job results
Get-Job | Receive-Job
What I am currently seeing:
Id Name State HasMoreData Location Command
-- ---- ----- ----------- -------- -------
23 Job23 Running True localhost #patch server ...
25 Job25 Running True localhost #patch server ...
What I want to see:
Searching for approved updates ...
Update Found: Security Update for Windows Server 2003 (KB2807986)
Update Found: Windows Malicious Software Removal Tool - March 2013 (KB890830)
Download complete. Installing updates ...
The system must be rebooted to complete installation.
cscript exited on "myServer" with error code 3.
Reboot required...
Waiting for server to reboot (35)
Searching for approved updates ...
There are no updates to install.
cscript exited on "myServer" with error code 2.
Servername "myServer" is fully patched after 2 loops
I want to be able to see the output or store that somewhere so I can refer back to be sure the script ran and see which servers rebooted, etc.
Conclusion:
In the past, I ran the script and it went through updating the servers one at a time and gave me the output I wanted, but when I started doing more servers - this task took too long, which is why I am trying to use background jobs with "Start-Job".
Can anyone help me figure this out, please?
You may take a look at the module SplitPipeline.
It it specifically designed for such tasks. The working demo code is:
# import the module (not necessary in PS V3)
Import-Module SplitPipeline
# some servers (from 1 to 10 for the test)
$servers = 1..10
# process servers by parallel pipelines and output results immediately
$servers | Split-Pipeline {process{"processing server $_"; sleep 1}} -Load 1, 1
For your task replace "processing server $_"; sleep 1 (simulates a slow job) with a call to your script and use the variable $_ as input, the current server.
If each job is not processor intensive then increase the parameter Count (the default is processor count) in order to improve performance.
Not a new question but I feel it is missing an answer including Powershell using workflows and its parallel possibilities, from powershell version 3. Which is less code and maybe more understandable than starting and waiting for jobs, which of course works good as well.
I have two files: TheScript.ps1 which coordinates the servers and BackgroundJob.ps1 which does some kind of check. They need to be in the same directory.
The Write-Output in the background job file writes to the same stream you see when starting TheScript.ps1.
TheScript.ps1:
workflow parallelCheckServer {
param ($Servers)
foreach -parallel($Server in $Servers)
{
Invoke-Expression -Command ".\BackgroundJob.ps1 -Server $Server"
}
}
parallelCheckServer -Servers #("host1.com", "host2.com", "host3.com")
Write-Output "Done with all servers."
BackgroundJob.ps1 (for example):
param (
[Parameter(Mandatory=$true)] [string] $server
)
Write-Host "[$server]`t Processing server $server"
Start-Sleep -Seconds 5
So when starting the TheScript.ps1 it will write "Processing server" 3 times but it will not wait for 15 seconds but instead 5 because they are run in parallel.
[host3.com] Processing server host3.com
[host2.com] Processing server host2.com
[host1.com] Processing server host1.com
Done with all servers.
In your ForEach loop you'll want to grab the output generated by the Jobs already running.
Example Not Tested
$sb = {
"Starting Job on $($args[0])"
#Do something
"$($args[0]) => Do something completed successfully"
"$($args[0]) => Now for something completely different"
"Ending Job on $($args[0])"
}
Foreach($computer in $computers){
Start-Job -ScriptBlock $sb -Args $computer | Out-Null
Get-Job | Receive-Job
}
Now if you do this all your results will be mixed. You might want to put a stamp on your verbose output to tell which output came from.
Or
Foreach($computer in $computers){
Start-Job -ScriptBlock $sb -Args $computer | Out-Null
Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}
}
while((Get-Job -State Running).count){
Get-Job | ? {$_.State -eq 'Complete' -and $_.HasMoreData} | % {Receive-Job $_}
start-sleep -seconds 1
}
It will show all the output as soon as a job is finished. Without being mixed up.
If you're wanting to multiple jobs in-progress, you'll probably want to massage the output to help keep what output goes with which job straight on the console.
$BGList = 'Black','Green','DarkBlue','DarkCyan','Red','DarkGreen'
$JobHash = #{};$ColorHash = #{};$i=0
ForEach($server in $servers)
{
Start-Job -FilePath c:\cefcu_it\psscripts\PSPatch.ps1 -ArgumentList $server |
foreach {
$ColorHash[$_.ID] = $BGList[$i++]
$JobHash[$_.ID] = $Server
}
}
While ((Get-Job).State -match 'Running')
{
foreach ($Job in Get-Job | where {$_.HasMoreData})
{
[System.Console]::BackgroundColor = $ColorHash[$Job.ID]
Write-Host $JobHash[$Job.ID] -ForegroundColor Black -BackgroundColor White
Receive-Job $Job
}
Start-Sleep -Seconds 5
}
[System.Console]::BackgroundColor = 'Black'
You can get the results by doing something like this after all the jobs have been received:
$array=#()
Get-Job -Name * | where{$array+=$_.ChildJobs.output}
.ChildJobs.output will have anything that was returned in each job.
function OutputJoblogs {
[CmdletBinding(DefaultParameterSetName='Name')]
Param
(
[Parameter(Mandatory=$true, Position=0)]
[System.Management.Automation.Job] $job,
[Parameter(Mandatory=$true, Position=1)]
[string] $logFolder,
[Parameter(Mandatory=$true, Position=2)]
[string] $logTimeStamp
)
#Output All logs
while ($job.sate -eq "Running" -or $job.HasMoreData){
start-sleep -Seconds 1
foreach($remotejob in $job.ChildJobs){
if($remotejob.HasMoreData){
$output=(Receive-Job $remotejob)
if($output -gt 0){
$remotejob.location +": "+ (($output) | Tee-Object -Append -file ("$logFolder\$logTimeStamp."+$remotejob.Location+".txt"))
}
}
}
}
#Output Errors
foreach($remotejob in $job.ChildJobs){
if($remotejob.Error.Count -gt0){$remotejob.location +": "}
foreach($myerr in $remotejob.Error){
$myerr 2>&1 | Tee-Object -Append -file ("$logFolder\$logTimeStamp."+$remotejob.Location+".ERROR.txt")
}
if($remotejob.JobStateInfo.Reason.ErrorRecord.Count -gt 0){$remotejob.location +": "}
foreach($myerr in $remotejob.JobStateInfo.Reason.ErrorRecord){
$myerr 2>&1 | Tee-Object -Append -file ("$logFolder\$logTimeStamp."+$remotejob.Location+".ERROR.txt")
}
}
}
#example of usage
$logfileDate="$((Get-Date).ToString('yyyy-MM-dd-HH.mm.ss'))"
$job = Invoke-Command -ComputerName "servername1","servername2" -ScriptBlock {
for ($i=1; $i -le 5; $i++) {
$i+"`n";
if($i -gt 2){
write-error "Bad thing happened"};
if($i -eq 4){
throw "Super Bad thing happened"
};
start-sleep -Seconds 1
}
} -asjob
OutputJoblogs -Job $job -logFolder "$PSScriptRoot\logs" -logTimeStamp $logfileDate