Multi-threading with PowerShell - event-log

I have done lots of reading about multi-threading in PowwerShell with Get-Job and Wait-Job but still cant seem to work it out.
Eventually, I will have this as a GUI based script to run and don't want my GUI to freeze up while its doing its task.
The script is looking for Event Logs of my Domain Controllers and then getting the details I want, then outputting them, it works like I need it to.
I can start a job using Invoke-Command {#script goes here} -ComputerName ($_) -AsJob -JobName $_ and the jobs run.
Script below:
Clear-Host
Get-Job | Remove-Job
(Get-ADDomainController -Filter *).Name | ForEach-Object {
Invoke-Command -ScriptBlock {
$StartTime = (Get-Date).AddDays(-4)
Try{
Get-WinEvent -FilterHashtable #{logname='Security'; id=4740;StartTime=$StartTime} -ErrorAction Stop `
| Select-Object * | ForEach-Object {
$Username = $_.Properties[0].Value
$lockedFrom = $_.Properties[1].Value
$DC = $_.Properties[4].Value
$Time = $_.TimeCreated
Write-Host "---------------------------------------------"
Write-Host $Username
Write-Host $lockedFrom
Write-Host $DC
Write-Host $Time
Write-Host "---------------------------------------------"
}#ForEach-Object
}catch [Exception] {
If ($_.Exception -match "No events were found that match the specified selection criteria") {
Write-Host "No events for locked out accounts." -BackgroundColor Red
}#If
}#Try Catch
} -ComputerName ($_) -AsJob -JobName $_ | Out-Null # Invoke-Command
}#ForEach-Object
Currently I have a While loop to tell me its waiting then to show me the result:
(Get-ADDomainController -Filter *).Name | ForEach-Object {
Write-Host "Waiting for: $_."
While ($(Get-Job -Name $_).State -ne 'Completed') {
#no doing anything here
}#While
Receive-Job -Name $_ -Keep
}#ForEach-Object
#clean up the jobs
Get-Job | Remove-Job
Thinking of my GUI (to be created), I will have a column for each Domain Controller and showing results under each heading, how do make it not freeze my GUI and show the results when they arrive?
I know its been asked a few times, but the examples I cant work out.

I would avoid Start-Job for threading - for efficiency try a runspace factory.
This is a basic setup which could be useful (I also have PS 4.0), and open to suggestions/improvements.
$MaxThreads = 2
$ScriptBlock = {
Param ($ComputerName)
Write-Output $ComputerName
#your processing here...
}
$runspacePool = [RunspaceFactory]::CreateRunspacePool(1, $MaxThreads)
$runspacePool.Open()
$jobs = #()
#queue up jobs:
$computers = (Get-ADDomainController -Filter *).Name
$computers | % {
$job = [Powershell]::Create().AddScript($ScriptBlock).AddParameter("ComputerName",$_)
$job.RunspacePool = $runspacePool
$jobs += New-Object PSObject -Property #{
Computer = $_
Pipe = $job
Result = $job.BeginInvoke()
}
}
# wait for jobs to finish:
While ((Get-Job -State Running).Count -gt 0) {
Get-Job | Wait-Job -Any | Out-Null
}
# get output of jobs
$jobs | % {
$_.Pipe.EndInvoke($_.Result)
}

Related

Add a timeout within a foreach loop and write to host - Powershell

I have a small script that runs through all share paths within a csv file and invokes a quick Get-ChildItem to count the number of files within the folder/subfolders etc. This is working but it is taking an age, especially if a folder have more than 500k files.
I would like to add a timeout within the loop, so that it will just write-host and then output into another column 'TIMED OUT' but i can't seem to get it to work.
I have tried a few different ways but somehow my loop breaks and runs the top line in my csv over and over.
This is the code i have so far:
...
#Import middle to filter unique
$sharesMiddle = Import-Csv -Path './middle.csv' | Sort-Object NFTSPath -Unique
$result = foreach ($share in $sharesMiddle) {
#Replace the colon for $ within the path
$nftsPath = join-path \\ $share.AssetName $share.Path.Replace(':', '$')
$path = $share.Path
$count = Invoke-Command -computername $share.AssetName -ScriptBlock { param($path) (Get-ChildItem $path -File -Recurse).count } -Credential $Cred -Verbose -ArgumentList $path
Write-Host $nftsPath : $count
$share | Select-Object *, #{n = "Files"; e = { $count } }
}
$result | Export-CSV '.\newcsvfile.csv' -NoTypeInformation
Any help on this would be super!
Thanks.
I tested the following which should be suitable for you:
# define the time limit in seconds
$TimeoutLimitSeconds = 2
# Define the job, start job
$job = Start-Job -ScriptBlock {420+69 ; start-sleep -Seconds 3}
# Await job
$job | Wait-Job -Timeout ( $TimeoutLimitSeconds ) | Out-Null
# If job doesn't finish before the time limit is reached
if ($job.state -eq 'Running') {
# Job timed out
$job | Stop-Job | Remove-Job
$job = $null
$output = "TIMED OUT"
# If job managed to finalize in time
}Else{
# Finished on time
$job | Stop-Job
$output = Receive-Job $job
$job | Remove-Job
}
# Write the output to host
Write-Host "output is: $output"
output is: TIMED OUT
# define the time limit in seconds
$TimeoutLimitSeconds = 4
# Define the job, start job
$job = Start-Job -ScriptBlock {420+69 ; start-sleep -Seconds 3}
# Await job
$job | Wait-Job -Timeout ( $TimeoutLimitSeconds ) | Out-Null
# If job doesn't finish before the time limit is reached
if ($job.state -eq 'Running') {
# Job timed out
$job | Stop-Job | Remove-Job
$job = $null
$output = "TIMED OUT"
# If job managed to finalize in time
}Else{
# Finished on time
$job | Stop-Job
$output = Receive-Job $job
$job | Remove-Job
}
# Write the output to host
Write-Host "output is: $output"
output is: 489

Using Start-Job on script

Using a script in PowerShell to recursivly pass through all folders on multiple NAS boxes to display every folder with its full path in an Out-File.
Using the Get-FolderEntry script I found here.
Since I have multiple NAS boxes with more then 260 chars in the filename/pathlength I figured I'd use multithreading to speed the process up.
Code:
. C:\Users\mdevogea\Downloads\Get-FolderEntry.ps1
# list with the servers
$Computers = Get-Content C:\Users\mdevogea\Desktop\servers.txt
# scriptblock calling on get-FolderEntry
$sb = {
param ($Computer, $fname)
C:\Users\mdevogea\Downloads\Get-FolderEntry.ps1 -Path $Computer |
fl | Out-File -Append -Width 1000 -FilePath $fname
}
foreach($Computer in $Computers)
{
$name = $Computer.Replace("\", "")
$fname = $("C:\Users\mdevogea\Desktop\" + $name + ".txt")
#Get-FolderEntry -Path $Computer | fl | Out-File -Append -Width 1000 $fname
$res = Start-Job $sb -ArgumentList $Computer, $fname
}
# Wait for all jobs
Get-Job
while(Get-Job -State "Running")
{
Write-Host "Running..."
Start-Sleep 2
}
# Get all job results
Get-Job | Receive-Job | Out-GridView
So far:
I either get empty files with the correct naming of the file.
I get the correct named file with the code of Get-FolderEntry in it.
I get errors depend on what I pass along to the scriptblock.
In short, it's probably stupid but don't see it.
Found it eventually myself after some trial and error:
. C:\Users\mdevogea\Downloads\Get-FolderEntry.ps1
# list with the servers
$Computers = Get-Content C:\Users\mdevogea\Desktop\servers.txt
# scriptblock calling on get-FolderEntry
$sb = {
Param ($Computer, $fname)
. C:\Users\mdevogea\Downloads\Get-FolderEntry.ps1
(Get-FolderEntry -Path $Computer | fl | Out-File -Append -Width 1000 -FilePath $fname)
}
foreach ($Computer in $Computers)
{
$name = $Computer.Replace("\", "")
$fname = $("C:\Users\mdevogea\Desktop\" + $name + ".txt")
$res = Start-Job $sb -ArgumentList $Computer, $fname
}
# Wait for all jobs
Get-Job
while (Get-Job -State "Running")
{
Write-Host "Running..."
Start-Sleep 2
}
# Get all job results
Get-Job | Receive-Job | Out-GridView
Thanks a lot Ansgar for pointing my in the right direction!

Using Powershell To Distribute Script Level Jobs On Remote Servers

More of a theory question...
I have a powershell script that exists on three servers. In this example the three servers are:
server1
server2
server3
I am using another machine, server4, to call script C:\ExampleScript.ps1 remotely using Invoke-Command while specifying the remote machine via the ComputerName parameter. The ultimate goal of the script is to detect whether powershell is running, if it is not, then the computer is "not busy" and can open up the script being called remotely. If the computer is "busy", move onto the next server and continue on through the three machines until all the parameter values have been exhausted. If all machines are busy, it would be ideal if there was a way to periodically check the processes and see if they are still open. In this way, execution of the script can be balanced across the various machines, in an albeit primitive fashion.
Consider the following code:
$servers = "server1","server2","server3"
$data = "param1", "param2", "param3", "param4", "param5", "param6"
#somehow loop through the different servers/data using the above arrays
$job = Invoke-Command $servers[0] {
$ProcessActive = Get-Process powershell -ErrorAction SilentlyContinue
if($ProcessActive -eq $null)
{
"Running"
Invoke-Command -ComputerName $env:computername -FilePath C:\ExampleScript.ps1 -ArgumentList $data[0]
}
else
{
"Busy go to next machine"
}
} -AsJob
Wait-Job $job
$r = Receive-Job $job
$r
The expected result trying to be achieved is attempting to load balance the script across the machines based on whether there is an active powershell process, if not move onto the next machine and perform the same test and subsequent possible execution. The script should go through all the values as specified in the $data array (or whatever).
I found this question interesting, so I wanted to give it a try.
$servers = "server1","server2","server3"
$data = New-Object System.Collections.ArrayList
$data.AddRange(#("param1", "param2", "param3", "param4", "param5", "param6"))
$jobs = New-Object System.Collections.ArrayList
do
{
Write-Host "Checking job states." -ForegroundColor Yellow
$toremove = #()
foreach ($job in $jobs)
{
if ($job.State -ne "Running")
{
$result = Receive-Job $job
if ($result -ne "ScriptRan")
{
Write-Host " Adding data back to que >> $($job.InData)" -ForegroundColor Green
$data.Add($job.InData) | Out-Null
}
$toremove += $job
}
}
Write-Host "Removing completed/failed jobs" -ForegroundColor Yellow
foreach ($job in $toremove)
{
Write-Host " Removing job >> $($job.Location)" -ForegroundColor Green
$jobs.Remove($job) | Out-Null
}
# Check if there is room to start another job
if ($jobs.Count -lt $servers.Count -and $data.Count -gt 0)
{
Write-Host "Checking servers if they can start a new job." -ForegroundColor Yellow
foreach ($server in $servers)
{
$job = $jobs | ? Location -eq $server
if ($job -eq $null)
{
Write-Host " Adding job for $server >> $($data[0])" -ForegroundColor Green
# No active job was found for the server, so add new job
$job = Invoke-Command $server -ScriptBlock {
param($data, $hostname)
$ProcessActive = Get-Process powershell -ErrorAction SilentlyContinue
if($ProcessActive -eq $null)
{
# This will block the thread on the server, so the JobState will not change till it's done or fails.
Invoke-Command -ComputerName $hostname -FilePath C:\ExampleScript.ps1 -ArgumentList $data
Write-Output "ScriptRan"
}
} -ArgumentList $data[0], $env:computername -AsJob
$job | Add-Member -MemberType NoteProperty -Name InData -Value $data[0]
$jobs.Add($job) | Out-Null
$data.Remove($data[0])
}
}
}
# Just a manual check of $jobs
Write-Output $jobs
# Wait a bit before checking again
Start-Sleep -Seconds 10
} while ($data.Count -gt 0)
Basically I create an array, and keep it constantly populated with one job for each server.
Data is removed from the list when a new job starts, and is added back if a job fails. This is to avoid servers running the script with the same data/params.
I lack a proper environment to test this properly at the moment, but will give it a whirl at work tomorrow and update my answer with any changes if needed.

start-job Run command in parallel and output result as they arrive

I am trying to get specific KBXXXXXX existence on a list of servers , but once my script one server it takes time and return result and come back and then move to next one . this script works perfectly fine for me .
I want my script to kick off and get-hotfix as job and other process just to collect the results and display them.
$servers = gc .\list.txt
foreach ($server in $servers)
{
$isPatched = (Get-HotFix -ComputerName $server | where HotFixID -eq 'KBxxxxxxx') -ne $null
If ($isPatched)
{
write-host $server + "Exist">> .\patchlist.txt}
Else
{
Write-host $server +"Missing"
$server >> C:\output.txt
}
}
The objective it to make the list execute faster rather than running serially.
With Powershell V2 you can use jobs as in #Andy answer or also in further detail in this link Can Powershell Run Commands in Parallel?
With PowerShell V2 you may also want to check out this script http://gallery.technet.microsoft.com/scriptcenter/Foreach-Parallel-Parallel-a8f3d22b using runspaces
With PowerShell V3 you have the foreach -parallel option.
for example (NB Measure-Command is just there for timing so you could make a comparison)
Workflow Test-My-WF {
param([string[]]$servers)
foreach -parallel ($server in $servers) {
$isPatched = (Get-HotFix -ComputerName $server | where {$_.HotFixID -eq 'KB9s82018'}) -ne $null
If ($isPatched)
{
$server | Out-File -FilePath "c:\temp\_patchlist.txt" -Append
}
Else
{
$server | Out-File -FilePath "c:\temp\_output.txt" -Append
}
}
}
Measure-Command -Expression { Test-My-WF $servers }
For this use PowerShell jobs.
cmdlets:
Get-Job
Receive-Job
Remove-Job
Start-Job
Stop-Job
Wait-Job
Here's an untested example:
$check_hotfix = {
param ($server)
$is_patched = (Get-HotFix -ID 'KBxxxxxxx' -ComputerName $server) -ne $null
if ($is_patched) {
Write-Output ($server + " Exist")
} else {
Write-Output ($server + " Missing")
}
}
foreach ($server in $servers) {
Start-Job -ScriptBlock $check_hotfix -ArgumentList $server | Out-Null
}
Get-Job | Wait-Job | Receive-Job | Set-Content patchlist.txt
Rather than use jobs, use the ability to query multiple computer that's built into the cmdlet. Many of Microsoft's cmdlets, especially those used for system management, take an array of strings as the input for a -Computername parameter. Pass in your list of servers, and the cmdlet will query all of them. Most of the cmdlets that have this ability will query the servers in series, but Invoke-Command will do it in parallel.
I haven't tested this as I don't have Windows booted at the moment, but this should get you started (in sequence).
$servers = gc .\list.txt
$patchedServers = Get-HotFix -ComputerName $servers | where HotFixID -eq 'KBxxxxxxx'|select machinename
$unpatchedServers = compare-object -referenceobject $patchedServers -differenceobject $servers -PassThru
$unpatchedServers |out-file c:\missing.txt;
$patchedServers|out-file c:\patched.txt;
In parallel:
$servers = gc .\list.txt
$patchedServers = invoke-command -computername $servers -scriptblock {Get-HotFix | where HotFixID -eq 'KBxxxxxxx'}|select -expandproperty pscomputername |sort -unique
As before, I don't have the right version of Windows available at the moment to test the above & check the output but it's a starting point.

How to timeout PowerShell function call

I wrote a little powershell function that executes Get-EventLog against remote servers. On some servers this seems to just hang and never times out. Can I timeout a powershell function call? I see how to do this against a different process, but i want to do this for a power shell function.
thanks
#######################
function Get-Alert4
{
param($computer)
$ret = Get-EventLog application -after (get-date).addHours(-2) -computer $computer | select-string -inputobject{$_.message} -pattern "Some Error String" | select-object List
return $ret
} #
You can implement timeouts by using a background job like so:
function Get-Alert4($computer, $timeout = 30)
{
$time = (Get-Date).AddHours(-2)
$job = Start-Job { param($c) Get-EventLog Application -CN $c -After $time |
Select-String "Some err string" -inputobject{$_.message} |
Select-Object List } -ArgumentList $computer
Wait-Job $job -Timeout $timeout
Stop-Job $job
Receive-Job $job
Remove-Job $job
}
FYI - your argumentlist is only good for one parameter.
If you want to pass more than one argument to the job, you have to pass them as an array:
$job = Start-Job { param($c, $t) Get-EventLog Application -CN $c -After $t |
Select-String "Some err string" -inputobject{$_.message} |
Select-Object List } -ArgumentList #($computer, $time)
This is a one liner (due to semi-colons) that will display a count down while pausing similar (but not the same) as the timeout cmd command
$NumOfSecs = 15; $n = $NumOfSecs; write-host "Starting in " -NoNewLine; do {if($n -lt $NumOfSecs) {write-host ", " -NoNewLine}; write-host $n -NoNewLine; $n = $n - 1; Start-Sleep 1} while ($n -gt 0)