I recently finished a script to ping every computer/workstation on a list and output it in a nice format.
There are thousands of computers to ping on the next list so it would take a while to run normally. How might I be able to multithread this so the job can be completed within a reasonable time-frame?
workflow Ping
{
param($computers)
foreach -parallel ($computer in $computers)
{
$status = Test-Connection -ComputerName $computer -Count 1 -Quiet
if (!$status)
{ Write-Output "Could not ping $computer" }
}
}
$computers = #(
"wd1600023",
"sleipnir"
)
Ping $computers
You could spawn multiple jobs
or you could use a workflow and a Foreach -parallel loop.
I found and used this script as a baseline for a similar function Ping-IPRange.
I have modified it from the original and included it below.
You could easily adapt this to take in a list of hostnames if that is what you are using.
Ping-IPRange.ps1
function Ping-IPRange
{
<#
.SYNOPSIS
Sends ICMP echo request packets to a range of IPv4 addresses between two given addresses.
.DESCRIPTION
This function lets you sends ICMP echo request packets ("pings") to
a range of IPv4 addresses using an asynchronous method.
Therefore this technique is very fast but comes with a warning.
Ping sweeping a large subnet or network with many switches may result in
a peak of broadcast traffic.
Use the -Interval parameter to adjust the time between each ping request.
For example, an interval of 60 milliseconds is suitable for wireless networks.
The RawOutput parameter switches the output to an unformatted
[System.Net.NetworkInformation.PingReply[]].
.INPUTS
None
You cannot pipe input to this function.
.OUTPUTS
The function only returns output from successful pings.
Type: System.Net.NetworkInformation.PingReply
The RawOutput parameter switches the output to an unformatted
[System.Net.NetworkInformation.PingReply[]].
.NOTES
Author : G.A.F.F. Jakobs
Created : August 30, 2014
Version : 6
Revision History: Kory Gill, 2016/01/09
formatting
added better error handling
close progress indicator when complete
.EXAMPLE
Ping-IPRange -StartAddress 192.168.1.1 -EndAddress 192.168.1.254 -Interval 20
IPAddress Bytes Ttl ResponseTime
--------- ----- --- ------------
192.168.1.41 32 64 371
192.168.1.57 32 128 0
192.168.1.64 32 128 1
192.168.1.63 32 64 88
192.168.1.254 32 64 0
In this example all the ip addresses between 192.168.1.1 and 192.168.1.254 are pinged using
a 20 millisecond interval between each request.
All the addresses that reply the ping request are listed.
.LINK
http://gallery.technet.microsoft.com/Fast-asynchronous-ping-IP-d0a5cf0e
#>
[CmdletBinding(ConfirmImpact='Low')]
Param(
[parameter(Mandatory = $true, Position = 0)]
[System.Net.IPAddress]$StartAddress,
[parameter(Mandatory = $true, Position = 1)]
[System.Net.IPAddress]$EndAddress,
[int]$Interval = 30,
[Switch]$RawOutput = $false
)
$timeout = 2000
function New-Range ($start, $end) {
[byte[]]$BySt = $start.GetAddressBytes()
[Array]::Reverse($BySt)
[byte[]]$ByEn = $end.GetAddressBytes()
[Array]::Reverse($ByEn)
$i1 = [System.BitConverter]::ToUInt32($BySt,0)
$i2 = [System.BitConverter]::ToUInt32($ByEn,0)
for ($x = $i1;$x -le $i2;$x++)
{
$ip = ([System.Net.IPAddress]$x).GetAddressBytes()
[Array]::Reverse($ip)
[System.Net.IPAddress]::Parse($($ip -join '.'))
}
}
$ipRange = New-Range $StartAddress $EndAddress
$IpTotal = $ipRange.Count
Get-Event -SourceIdentifier "ID-Ping*" | Remove-Event
Get-EventSubscriber -SourceIdentifier "ID-Ping*" | Unregister-Event
$ipRange | ForEach-Object {
[string]$VarName = "Ping_" + $_.Address
New-Variable -Name $VarName -Value (New-Object System.Net.NetworkInformation.Ping)
Register-ObjectEvent -InputObject (Get-Variable $VarName -ValueOnly) -EventName PingCompleted -SourceIdentifier "ID-$VarName"
(Get-Variable $VarName -ValueOnly).SendAsync($_,$timeout,$VarName)
Remove-Variable $VarName
try
{
$pending = (Get-Event -SourceIdentifier "ID-Ping*").Count
}
catch [System.InvalidOperationException]
{
$pending = 0
}
$index = [array]::indexof($ipRange,$_)
Write-Progress -Activity "Sending ping to" -Id 1 -status $_.IPAddressToString -PercentComplete (($index / $IpTotal) * 100)
$percentComplete = ($($index - $pending), 0 | Measure-Object -Maximum).Maximum
Write-Progress -Activity "ICMP requests pending" -Id 2 -ParentId 1 -Status ($index - $pending) -PercentComplete ($percentComplete/$IpTotal * 100)
Start-Sleep -Milliseconds $Interval
}
Write-Progress -Activity "Done sending ping requests" -Id 1 -Status 'Waiting' -PercentComplete 100
while ($pending -lt $IpTotal) {
Wait-Event -SourceIdentifier "ID-Ping*" | Out-Null
Start-Sleep -Milliseconds 10
try
{
$pending = (Get-Event -SourceIdentifier "ID-Ping*").Count
}
catch [System.InvalidOperationException]
{
$pending = 0
}
$percentComplete = ($($IpTotal - $pending), 0 | Measure-Object -Maximum).Maximum
Write-Progress -Activity "ICMP requests pending" -Id 2 -ParentId 1 -Status ($IpTotal - $pending) -PercentComplete ($percentComplete/$IpTotal * 100)
}
Write-Progress -Completed -Id 2 -ParentId 1 -Activity "Completed"
Write-Progress -Completed -Id 1 -Activity "Completed"
$Reply = #()
if ($RawOutput)
{
Get-Event -SourceIdentifier "ID-Ping*" | ForEach {
if ($_.SourceEventArgs.Reply.Status -eq "Success")
{
$Reply += $_.SourceEventArgs.Reply
}
Unregister-Event $_.SourceIdentifier
Remove-Event $_.SourceIdentifier
}
}
else
{
Get-Event -SourceIdentifier "ID-Ping*" | ForEach-Object {
if ($_.SourceEventArgs.Reply.Status -eq "Success")
{
$pinger = #{
IPAddress = $_.SourceEventArgs.Reply.Address
Bytes = $_.SourceEventArgs.Reply.Buffer.Length
Ttl = $_.SourceEventArgs.Reply.Options.Ttl
ResponseTime = $_.SourceEventArgs.Reply.RoundtripTime
}
$Reply += New-Object PSObject -Property $pinger
}
Unregister-Event $_.SourceIdentifier
Remove-Event $_.SourceIdentifier
}
}
if ($Reply.Count -eq 0)
{
Write-Verbose "Ping-IPRange : No IP address responded" -Verbose
}
return $Reply
}
What is the correct way of determining if a process is running, for example FireFox, and stopping it?
I did some looking around and the best way I found was this:
if((get-process "firefox" -ea SilentlyContinue) -eq $Null){
echo "Not Running"
}
else{
echo "Running"
Stop-Process -processname "firefox"
}
Is this the ideal way of doing it? If not, what the correct way of doing so?
The way you're doing it you're querying for the process twice. Also Lynn raises a good point about being nice first. I'd probably try something like the following:
# get Firefox process
$firefox = Get-Process firefox -ErrorAction SilentlyContinue
if ($firefox) {
# try gracefully first
$firefox.CloseMainWindow()
# kill after five seconds
Sleep 5
if (!$firefox.HasExited) {
$firefox | Stop-Process -Force
}
}
Remove-Variable firefox
If you don't need to display exact result "running" / "not runnuning", you could simply:
ps notepad -ErrorAction SilentlyContinue | kill -PassThru
If the process was not running, you'll get no results. If it was running, you'll receive get-process output, and the process will be stopped.
#jmp242 - the generic System.Object type does not contain the CloseMainWindow method, but statically casting the System.Diagnostics.Process type when collecting the ProcessList variable works for me. Updated code (from this answer) with this casting (and looping changed to use ForEach-Object) is below.
function Stop-Processes {
param(
[parameter(Mandatory=$true)] $processName,
$timeout = 5
)
[System.Diagnostics.Process[]]$processList = Get-Process $processName -ErrorAction SilentlyContinue
ForEach ($Process in $processList) {
# Try gracefully first
$Process.CloseMainWindow() | Out-Null
}
# Check the 'HasExited' property for each process
for ($i = 0 ; $i -le $timeout; $i++) {
$AllHaveExited = $True
$processList | ForEach-Object {
If (-NOT $_.HasExited) {
$AllHaveExited = $False
}
}
If ($AllHaveExited -eq $true){
Return
}
Start-Sleep 1
}
# If graceful close has failed, loop through 'Stop-Process'
$processList | ForEach-Object {
If (Get-Process -ID $_.ID -ErrorAction SilentlyContinue) {
Stop-Process -Id $_.ID -Force -Verbose
}
}
}
To start with process-killing, here python, my 2 cents:
Get-Process python3.9|Stop-Process
Thanks #Joey. It's what I am looking for.
I just bring some improvements:
to take into account multiple processes
to avoid reaching the timeout when all processes have terminated
to package the whole in a function
function Stop-Processes {
param(
[parameter(Mandatory=$true)] $processName,
$timeout = 5
)
$processList = Get-Process $processName -ErrorAction SilentlyContinue
if ($processList) {
# Try gracefully first
$processList.CloseMainWindow() | Out-Null
# Wait until all processes have terminated or until timeout
for ($i = 0 ; $i -le $timeout; $i ++){
$AllHaveExited = $True
$processList | % {
$process = $_
If (!$process.HasExited){
$AllHaveExited = $False
}
}
If ($AllHaveExited){
Return
}
sleep 1
}
# Else: kill
$processList | Stop-Process -Force
}
}
$fullnamexp = ((net user $winxp /domain | Select-String "Full Name") -replace "Full Name","").Trim();
If $winxp cannot be found, the command will hang, is there a timeout I can use with this to make it move on after 5-10 seconds? Not sure where I would put it.
Edit- I use this to pull the username:
$reg = [Microsoft.Win32.RegistryKey]::OpenRemoteBaseKey('LocalMachine', $tag1)
$key = $reg.OpenSubKey('SOFTWARE\Microsoft\Windows NT\CurrentVersion\WinLogon')
$winxp = $key.GetValue('DefaultUserName') -replace '^.*?\\'
$winxp is then a login name such as ajstepanik then I put it into: $fullnamexp = ((net user $winxp /domain | Select-String "Full Name") -replace "Full Name","").Trim();
1.21.2014 Update
$timeoutSeconds = 5
$code = {
((net user $winxp /domain | Select-String "Full Name") -replace "Full Name","").Trim(); # your commands here, e.g.
}
$j = Start-Job -ScriptBlock $code
if (Wait-Job $j -Timeout $timeoutSeconds) { $fullnamexp = Receive-Job $j }
Remove-Job -force $j
While #mjolinor may have indeed provided you an alternative approach, here is a direct answer to your general question: how do you force a timeout in PowerShell?
Wrap whatever you wish to time-limit in a script block, run that as a job, then use the Wait-Job cmdlet to time-limit the operation. Wait-Job will return either at the end of the timeout period or when the script block completes, whichever occurs first. After Wait-Job returns, you can examine the job state ($j.state) to determine whether it was interrupted or not, if it matters to you.
$timeoutSeconds = 5 # set your timeout value here
$j = Start-Job -ScriptBlock {
# your commands here, e.g.
Get-Process
}
"job id = " + $j.id # report the job id as a diagnostic only
Wait-Job $j -Timeout $timeoutSeconds | out-null
if ($j.State -eq "Completed") { "done!" }
elseif ($j.State -eq "Running") { "interrupted" }
else { "???" }
Remove-Job -force $j #cleanup
2014.01.18 Update
Here is a bit more streamlining approach that also includes the practical step of getting information out of the script block with Receive-Job, assuming what you want is generated on stdout:
$timeoutSeconds = 3
$code = {
# your commands here, e.g.
Get-ChildItem *.cs | select name
}
$j = Start-Job -ScriptBlock $code
if (Wait-Job $j -Timeout $timeoutSeconds) { Receive-Job $j }
Remove-Job -force $j
You can use Start-Sleep to pause the script:
Start-Sleep -s 5
net doesn't explicitly allow you to set a time out on it's operations, but you could check out this link on changing the ipv4 timeout for your sockets:
http://www.cyberciti.biz/tips/linux-increasing-or-decreasing-tcp-sockets-timeouts.html
The only thing else I could imagine is spawning a worker thread but I don't even know if that's possible in bash, I'm not fluid enough in it to answer that; plus it opens you up to sync problems and all sorts of multi threaded issues beyond what you're trying to accomplish quickly in a bash script to begin with! :P
Does this help?
$query = (dsquery user -samid $winxp)
if ($query) {$fullnamexp = ($query | dsget user -display)[1].trim()}
$fullnamexp
This solution doesn't work for me. remove-job -force $j takes over 5 seconds in this example.
$timeoutseconds = 1
$start = get-date
$j = start-job -scriptblock { Resolve-DnsName 1.1.1.1 }
if (wait-job $j -timeout $timeoutseconds) { $fullnamexp = receive-job $j }
remove-job -force $j
(get-date) - $start
Days : 0
Hours : 0
Minutes : 0
Seconds : 5
Milliseconds : 342
Ticks : 53426422
TotalDays : 6.18361365740741E-05
TotalHours : 0.00148406727777778
TotalMinutes : 0.0890440366666667
TotalSeconds : 5.3426422
TotalMilliseconds : 5342.6422
Here's a simple timeout example with notepad:
notepad
if (-not $(wait-process notepad 10; $?)) { stop-process -name notepad }
$watchdog = 10 #seconds
$start_time = Get-Date
$j = Start-Job -ScriptBlock{
#timeout command
if ($true) {
$i = 0
while($true) {
Write-Host "Count: $i"
Start-Sleep -Milliseconds 100
$i++
}
}
write-host "Hello"
}
while($true) {
if ($j.HasMoreData) {
Receive-Job $j
Start-Sleep -Milliseconds 200
}
$current = Get-Date
$time_span = $current - $start_time
if ($time_span.TotalSeconds -gt $watchdog) {
write-host "TIMEOUT!"
Stop-Job $j
break
}
if (-not $j.HasMoreData -and $j.State -ne 'Running') {
write-host "Finished"
break
}
}
Remove-Job $j
I'm not sure whether to call this a need for multi-threading, job-based, or async, but basically I have a Powershell script function that takes several parameters and I need to call it several times with different parameters and have these run in parallel.
Currently, I call the function like this:
Execute "param1" "param2" "param3" "param4"
How can I call this multiple times without waiting for each call to Execute return to the caller?
Currently I'm running v2.0 but I can update if necessary
EDIT: here's what I have so far, which doesn't work:
$cmd = {
param($vmxFilePath,$machineName,$username,$password,$scriptTpath,$scriptFile,$uacDismissScript,$snapshotName)
Execute $vmxFilePath $machineName $username $password $scriptTpath $scriptFile $uacDismissScript $snapshotName
}
Start-Job -ScriptBlock $cmd -ArgumentList $vmxFilePath, $machineName, $username $password, $scriptTpath, $scriptFile, $uacDismissScript, $snapshotName
I get an error:
cannot convert 'system.object[]' to the type 'system.management.automation.scriptblock' required by parameter 'initializationscript'. specified method is not supported
EDIT2: I've modified my script but I still get the error mentioned above. Here's my mod:
$cmd = {
param($vmxFilePath,$machineName,$username,$password,$scriptTpath,$scriptFile,$uacDismissScript,$snapshotName)
Execute $vmxFilePath $machineName $username $password $scriptTpath $scriptFile $uacDismissScript $snapshotName
}
Start-Job -ScriptBlock $cmd -ArgumentList $vmxFilePath, $machineName, $username $password, $scriptTpath, $scriptFile, $uacDismissScript, $snapshotName
No update necessary for this. Define a script block and use Start-Job to run the script block as many times as necessary. Example:
$cmd = {
param($a, $b)
Write-Host $a $b
}
$foo = "foo"
1..5 | ForEach-Object {
Start-Job -ScriptBlock $cmd -ArgumentList $_, $foo
}
The script block takes 2 parameters $a and $b which are passed by the -ArgumentList option. In the example above, the assignments are $_ → $a and $foo → $b. $foo is just an example for a configurable, but static parameter.
Run Get-Job | Remove-Job at some point to remove the finished jobs from the queue (or Get-Job | % { Receive-Job $_.Id; Remove-Job $_.Id } if you want to retrieve the output).
Here's a quick bogus scriptblock for the purpose of testing:
$Code = {
param ($init)
$start = Get-Date
(1..30) | % { Start-Sleep -Seconds 1; $init +=1 }
$stop = Get-Date
Write-Output "Counted from $($init - 30) until $init in $($stop - $start)."
}
This scriptblock can then be passed on to Start-Job, with for example 3 parameters (10, 15, 35)
$jobs = #()
(10,15,35) | % { $jobs += Start-Job -ArgumentList $_ -ScriptBlock $Code }
Wait-Job -Job $jobs | Out-Null
Receive-Job -Job $jobs
This creates 3 jobs, assign them to the $jobs variable, runs them in parallel and then waits for these 3 jobs to finish, and retrieves the results:
Counted from 10 until 40 in 00:00:30.0147167.
Counted from 15 until 45 in 00:00:30.0057163.
Counted from 35 until 65 in 00:00:30.0067163.
This did not take 90 seconds to execute, only 30.
One of the tricky parts is to provide -Argumentlist to Start-Job, and include a param() block inside the ScriptBlock. Otherwise, your values are never seen by the scriptblock.
You can use an alternative which may be faster than invoking jobs if the function is not a long running one. Max thread is 25 and I'm only invoking this function 10 times, so I expect my total runtime to be 5 seconds. You can wrap Measure-Command around the 'results=' statement to view stats.
Example:
$ScriptBlock = {
Param ( [int]$RunNumber )
Start-Sleep -Seconds 5
Return $RunNumber
}
$runNumbers = #(1..10)
$MaxThreads = 25
$runspacePool = [RunspaceFactory ]::CreateRunspacePool(1, $MaxThreads)
$runspacePool.Open()
$pipeLines = foreach($num in $runNumbers){
$pipeline = [powershell]::Create()
$pipeline.RunspacePool = $runspacePool
$pipeline.AddScript($ScriptBlock) | Out-Null
$pipeline.AddArgument($num) | Out-Null
$pipeline | Add-Member -MemberType NoteProperty -Name 'AsyncResult' -Value $pipeline.BeginInvoke() -PassThru
}
#obtain results as they come.
$results = foreach($pipeline in $pipeLines){
$pipeline.EndInvoke($pipeline.AsyncResult )
}
#cleanup code.
$pipeLines | % { $_.Dispose()}
$pipeLines = $null
if ( $runspacePool ) { $runspacePool.Close()}
#your results
$results
I'm sorry that everyone missed your issue - I know it is far too late now, but...
This error is caused because you are missing a comma between $username and $password in your list.
You can test it out with this snippet, which I modelled off of the previous answers:
$cmd = {
param($a, $b, $c, $d)
}
$foo = "foo"
$bar = "bar"
start-job -scriptblock $cmd -ArgumentList "a", $foo, $bar, "gold" #added missing comma for this to work
I've made a very versitile function to do this for you, and unlike the other answers you dont need to rejig your code to get it to work.
Just pass your function as a parameter to Async and pipe your input in, each item on the pipeline will run your scriptblock in parallel asynchronously, and emit them as each one is completed.
For your question specifically it would look something like this
#(
#{vmxFilePath='a';machineName='b';username='c';password='d';scriptTpath='e';scriptFile='f';uacDismissScript='g';snapshotName'h'},
#{vmxFilePath='i';machineName='j';username='k';password='l';scriptTpath='m';scriptFile='n';uacDismissScript='o';snapshotName'p'}
...
) `
| Async `
-Func { Process {
Execute $_.vmxFilePath $_.machineName $_.username $_.password $_.scriptTpath $_.scriptFile $_.uacDismissScript $_.snapshotName
} }
On top of this my function supports not only automatic construction of [powershell] (#binarySalt's answer) but also Jobs (utilised in #Joost's, but dont use these as they are much slower than runspaces) and Tasks if you're using other people's code that already spawn them (use the -AsJob flag, which I explain at the bottom of this answer).
So, that's not useful for new visitors to this question, lets do something more demonstrable that you can run on your machine and see real-world results.
Take this simple code for example, it just takes in some test data for websites and checks if they're up.
$in=TestData | ?{ $_.proto -eq 'tcp' }
$in `
| %{
$WarningPreference='SilentlyContinue'
$_ `
| Add-Member `
-PassThru `
-MemberType NoteProperty `
-Name result `
-Value $(Test-NetConnection `
-ComputerName $_.address `
-Port $_.port `
-InformationLevel Quiet
)
} `
| Timer -name 'normal' `
| Format-Table
Here's the test data, just a good few of the same websites on repeat.
And also a timing function to see how performant it is.
Function TestData {
1..20 | %{
[PsCustomObject]#{proto='tcp' ; address='www.w3.org' ; port=443},
[PsCustomObject]#{proto='https'; address='www.w3.org' ; port=443},
[PsCustomObject]#{proto='icmp' ; address='www.w3.org' ; },
[PsCustomObject]#{proto='tcp' ; address='developer.mozilla.org' ; port=443},
[PsCustomObject]#{proto='https'; address='developer.mozilla.org' ; port=443},
[PsCustomObject]#{proto='icmp' ; address='developer.mozilla.org' ; },
[PsCustomObject]#{proto='tcp' ; address='help.dottoro.com' ; port=80 },
[PsCustomObject]#{proto='http' ; address='help.dottoro.com' ; port=80 },
[PsCustomObject]#{proto='icmp' ; address='help.dottoro.com' ; }
}
}
Function Timer {
Param ($name)
Begin {
$timer=[system.diagnostics.stopwatch]::StartNew()
}
Process { $_ }
End {
#(
$name,
' '
[math]::Floor($timer.Elapsed.TotalMinutes),
':',
($timer.Elapsed.Seconds -replace '^(.)$','0$1')
) -join '' | Out-Host
}
}
Okay, 15seconds, so how much faster can this get if we use Async?
And how much do we have to change to get it to work?
$in=TestData | ?{ $_.proto -eq 'tcp' }
$in `
| Async `
-Expected $in.Count `
-Func { Process {
$WarningPreference='SilentlyContinue'
$_ `
| Add-Member `
-PassThru `
-MemberType NoteProperty `
-Name result `
-Value $(Test-NetConnection `
-ComputerName $_.address `
-Port $_.port `
-InformationLevel Quiet
)
} } `
| Timer -name 'async' `
| Format-Table
It looks basically identical..
Okay, what's the speed?
Wow, cut it by two thirds!
Not only that, but because we know how many items are in the pipeline I wrote in some smarts to give you a progressbar and an ETA
Don't believe me? Have a video
Or run the code yourself :)
#Requires -Version 5.1
#asynchronously run a pool of tasks,
#and aggregate the results back into a synchronous output
#without waiting to pool all input before seeing the first result
Function Async { Param(
#maximum permitted simultaneous background tasks
[int]$BatchSize=[int]$env:NUMBER_OF_PROCESSORS * 3,
#the task that accepts input on a pipe to execute in the background
[scriptblock]$Func,
#because your task is in a subshell you wont have access to your outer scope,
#you may pass them in here
[array]$ArgumentList=#(),
[System.Collections.IDictionary]$Parameters=#{},
#the title of the progress bar
[string]$Name='Processing',
#your -Func may return a [Job] instead of being backgrounded itself,
#if so it must return #(job;input;args)
#optionally job may be a [scriptblock] to be backgrounded, or a [Task]
[switch]$AsJob,
#if you know the number of tasks ahead of time,
#providing it here will have the progress bar show an ETA
[int]$Expected,
#outputs of this stream will be #(job;input) where job is the result
[switch]$PassThru,
#the time it takes to give up on one job type if there are others waiting
[int]$Retry=5
)
Begin {
$ArgumentList=[Array]::AsReadOnly($ArgumentList)
$Parameters=$Parameters.GetEnumerator() `
| &{
Begin { $params=[ordered]#{} }
Process { $params.Add($_.Key, $_.Value) }
End { $params.AsReadOnly() }
}
#the currently running background tasks
$running=#{}
$counts=[PSCustomObject]#{
completed=0;
jobs=0;
tasks=0;
results=0;
}
#a lazy attempt at uniquely IDing this instance for Write-Progress
$asyncId=Get-Random
#a timer for Write-Progress
$timer=[system.diagnostics.stopwatch]::StartNew()
$pool=[RunspaceFactory]::CreateRunspacePool(1, $BatchSize)
$pool.Open()
#called whenever we want to update the progress bar
Function Progress { Param($Reason)
#calculate ETA if applicable
$eta=-1
$total=[math]::Max(1, $counts.completed + $running.Count)
if ($Expected) {
$total=[math]::Max($total, $Expected)
if ($counts.completed) {
$eta=`
($total - $counts.completed) * `
$timer.Elapsed.TotalSeconds / `
$counts.completed
}
}
$Reason=Switch -regex ($Reason) {
'^done$' { "Finishing up the final $($running.Count) jobs." }
'^(do|next)$' { "
Running
$($running.Count)
jobs concurrently.
$(#('Adding','Waiting to add')[!($Reason -eq 'do')])
job #
$($counts.completed + $running.Count + 1)
" -replace '\r?\n\t*','' }
Default { "
Running $($running.Count) jobs concurrently.
Emitting
$($counts.completed)
$(#{1='st';2='nd';3='rd'}[$counts.completed % 10] -replace '^$','th')
result.
" -replace '\r?\n\t*','' }
}
Write-Progress `
-Id $asyncId `
-Activity $Name `
-SecondsRemaining $eta `
-Status ("
$($counts.completed)
jobs completed in
$([math]::Floor($timer.Elapsed.TotalMinutes))
:
$($timer.Elapsed.Seconds -replace '^(.)$','0$1')
" -replace '\r?\n\t*','') `
-CurrentOperation $Reason `
-PercentComplete (100 * $counts.completed / $total)
}
#called with the [Job]'s that have completed
Filter Done {
++$counts.completed
$out=$running.Item($_.Id)
$running.Remove($_.Id)
Progress
$out.job=`
if ($_ -is [System.Management.Automation.Job]) {
--$counts.jobs
$_ | Receive-Job
}
elseif ($_.pwsh) {
--$counts.results
try {
$_.pwsh.EndInvoke($_)
}
catch {
#[System.Management.Automation.MethodInvocationException]
$_.Exception.InnerException
}
finally {
$_.pwsh.Dispose()
}
}
elseif ($_.IsFaulted) {
--$counts.tasks
#[System.AggregateException]
$_.Exception.InnerException
}
else {
--$counts.tasks
$_.Result
}
if ($PassThru) {
$out
}
else {
$out.job
}
}
$isJob={
$_ -is [System.Management.Automation.Job]
}
$isTask={
$_ -is [System.Threading.Tasks.Task]
}
$isResult={
$_ -is [IAsyncResult]
}
$isFinished={
$_.IsCompleted -or `
(
$_.JobStateInfo.State -gt 1 -and
$_.JobStateInfo.State -ne 6 -and
$_.JobStateInfo.State -ne 8
)
}
$handle={
$_.AsyncWaitHandle
}
Function Jobs { Param($Filter)
$running.Values | %{ $_.job } | ? $Filter
}
#called whenever we need to wait for at least one task to completed
#outputs the completed tasks
Function Wait { Param([switch]$Finishing)
#if we are at the max background tasks this instant
while ($running.Count -ge $BatchSize) {
Progress -Reason #('done','next')[!$Finishing]
$value=#('jobs', 'tasks', 'results') `
| %{ $counts.($_) } `
| measure -Maximum -Sum
$wait=if ($value.Maximum -lt $value.Sum) {
$Retry
}
else {
-1
}
$value=Switch -exact ($value.Maximum) {
$counts.jobs {
(Wait-Job `
-Any `
-Job (Jobs -Filter $isJob) `
-Timeout $wait
).Count -lt 1
break
}
Default {
[System.Threading.WaitHandle]::WaitAny(
(Jobs -Filter $handle | % $handle),
[math]::Max($wait * 1000, -1)
) -eq [System.Threading.WaitHandle]::WaitTimeout
break
}
}
(Jobs -Filter $isFinished) | Done
}
}
}
#accepts inputs to spawn a new background task with
Process {
Wait
Progress -Reason 'do'
$run=[PSCustomObject]#{
input=$_;
job=$Func;
args=$ArgumentList;
params=$Parameters;
}
if ($AsJob) {
$run.job=$NULL
Invoke-Command `
-ScriptBlock $Func `
-ArgumentList #($run) `
| Out-Null
}
if ($run.job | % $isJob) {
++$counts.jobs
}
elseif ($run.job | % $isTask) {
++$counts.tasks
}
#if we weren't given a [Job] we need to spawn it for them
elseif ($run.job -is [ScriptBlock]) {
$pwsh=[powershell]::Create().AddScript($run.job)
$run.args | %{ $pwsh.AddArgument($_) } | Out-Null
$pwsh.RunspacePool=$pool
$run.job=$pwsh.AddParameters($run.params).BeginInvoke(
[System.Management.Automation.PSDataCollection[PSObject]]::new(
[PSObject[]]($run.input)
)
)
$run.job | Add-Member `
-MemberType NoteProperty `
-Name pwsh `
-Value $pwsh `
-PassThru `
| Add-Member `
-MemberType NoteProperty `
-Name Id `
-Value $run.job.AsyncWaitHandle.Handle.ToString()
++$counts.results
}
else {
throw "$($run.job.GetType()) needs to be a ScriptBlock"
}
$running.Add($run.job.Id, $run) | Out-Null
}
End {
#wait for the remaining running processes
$BatchSize=1
Wait -Finishing
Write-Progress -Id $asyncId -Activity $Name -Completed
$pool.Close()
$pool.Dispose()
}
}
So you may have noticed three things above, I alluded to -AsJob (using a mix of Jobs, Tasks and scriptblocks), there are unused protocols mentioned in the test data, and the video had a third test in it.
Here it is. Instead of doing the basic tcp test, using the test data we will also do a http/s check and a icmp ping (idk, maybe the https fails, but you want to narrow down if it's because the machine is down or just the service).
Test-Connection -AsJob is a cmdlet that returns a Job and checks ping.
WebRequest.GetResponseAsync() which connects to a web resource, returning a Task
And finally Test-NetConnection, which will be the same as before, ran in what we programmed as a synchronous scriptblock, which will be ran asynchronously for us.
$in=TestData
$in `
| Async `
-Expected $in.Count `
-PassThru `
-AsJob `
<#this would be accessible as a named parameter if needed#>`
-Parameters #{proxy=[System.Net.WebRequest]::GetSystemWebProxy()} `
-Func { Param([parameter(Position=0)]$x)
$x.job=Switch -regex ($x.input.proto) {
'^icmp$' {
Test-Connection `
-ComputerName $x.input.address `
-Count 1 `
-ThrottleLimit 1 `
-AsJob
}
'^tcp$' {
$x.params=#{address=$x.input.address; port=$x.input.port}
{ Param($address, $port)
$WarningPreference='SilentlyContinue'
Test-NetConnection `
-ComputerName $address `
-Port $port `
-InformationLevel Quiet
}
}
'^(http|https)$' {
[Net.ServicePointManager]::SecurityProtocol=[Net.SecurityProtocolType]::Tls12
$request=[System.Net.HttpWebRequest]::Create((#(
$x.input.proto,
'://',
$x.input.address,
':',
$x.input.port
) -join ''))
$request.Proxy=$NULL
$request.Method='Get'
$request.GetResponseAsync()
}
}
} `
| %{
$result=$_
$result.input `
| Add-Member `
-PassThru `
-MemberType NoteProperty `
-Name result `
-Value $(Switch -regex (#($result.input.proto, $result.job.message)[$result.job -is [Exception]]) {
#[Win32_PingStatus]
'^icmp$' { $result.job.StatusCode -eq 0 }
#[bool]
'^tcp$' { $result.job }
#[System.Net.HttpWebResponse]
'^(http|https)$' {
$result.job.Close()
Switch ($result.job.StatusCode.value__) {
{ $_ -ge 200 -and $_ -lt 400 } { $True }
Default {$False}
}
}
#[Exception]
Default { $False }
})
} `
| Timer -name 'async asjob' `
| Format-Table
As you may have seen, this did more than double the work of the original code, but still finished in around half the time at 8seconds.
I would like to time my background jobs (started with start-job) and time them out after x seconds. I find it hard however to keep track of the running time on each separate job (I am running aprox 400 jobs).
I wish there was a way to time out the job and set it to failed if not completed in X seconds, but I find no timeout-parameter.
What would be a good way to track the individual run-time of the jobs?
I guess I could create a hashtable with start-time of each job and the job-id and check against the running state and do a manual timeout, but that sounds kinda "inventing the wheel".
Any ideas?
Edit
Thank you everyone for a fruitful discussion and great inspiration on this topic!
You can use a hash table of timers:
$jobtimer = #{}
foreach ($job in $jobs){
start-job -name $job -ScriptBlock {scriptblock commands}
$jobtimer[$job] = [System.Diagnostics.Stopwatch]::startnew()
}
The running time of each job will be in $jobtimer[$job].elapsed
Just walk through the list of running jobs and stop any that have run past your timeout spec e.g.:
$timeout = [timespan]::FromMinutes(1)
$now = Get-Date
Get-Job | Where {$_.State -eq 'Running' -and
(($now - $_.PSBeginTime) -gt $timeout)} | Stop-Job
BTW there are more properties to a job object than the default formatting shows e.g.:
3 > $job | fl *
State : Running
HasMoreData : True
StatusMessage :
Location : localhost
Command : Start-sleep -sec 30
JobStateInfo : Running
Finished : System.Threading.ManualResetEvent
InstanceId : de370ea8-763b-4f3b-ba0e-d45f402c8bc4
Id : 3
Name : Job3
ChildJobs : {Job4}
PSBeginTime : 3/18/2012 11:07:20 AM
PSEndTime :
PSJobType : BackgroundJob
Output : {}
Error : {}
Progress : {}
Verbose : {}
Debug : {}
Warning : {}
You can specify the timeout option of Wait-Job:
-Timeout
Determines the maximum wait time for each background job, in seconds.
The default, -1, waits until the job completes, no matter how long it
runs. The timing starts when you submit the Wait-Job command, not the
Start-Job command.
If this time is exceeded, the wait ends and the command prompt
returns, even if the job is still running. No error message is
displayed.
Here's some example code:
This part just makes some test jobs:
Remove-Job -Name *
$jobs = #()
1..10 | % {
$jobs += Start-Job -ScriptBlock {
Start-Sleep -Seconds (Get-Random -Minimum 5 -Maximum 20)
}
}
The variable $timedOutJobs contains jobs that timed out. You can then restart them or what have you.
$jobs | Wait-Job -Timeout 10
$timedOutJobs = Get-Job | ? {$_.State -eq 'Running'} | Stop-Job -PassThru
For completeness, this answer combines the maximum seconds per job and the maximum concurrent jobs running. As this is what most people are after.
The example below retrieves the printer configuration for each print server. There can be over 3000 printers, so we added throttling.
$i = 0
$maxConcurrentJobs = 40
$maxSecondsPerJob = 60
$jobTimer = #{ }
$StopLongRunningJobs = {
$jobTimer.GetEnumerator().where( {
($_.Value.IsRunning) -and
($_.Value.Elapsed.TotalSeconds -ge $maxSecondsPerJob)
}).Foreach( {
$_.Value.Stop()
Write-Verbose "Stop job '$($_.Name.Name)' that ran for '$($_.Value.Elapsed.TotalSeconds)' seconds"
Stop-Job $_.Name
})
}
Foreach ($Computer in #($GetPrinterJobResults.Where( { $_.Data }) )) {
foreach ($Printer in $Computer.Data) {
do {
& $StopLongRunningJobs
$running = #(Get-Job -State Running)
$Wait = $running.Count -ge $maxConcurrentJobs
if ($Wait) {
Write-Verbose 'Waiting for jobs to fininsh'
$null = $running | Wait-Job -Any -Timeout 5
}
} while ($Wait)
$i++
Write-Verbose "$I $($Computer.ComputerName) Get print config '$($Printer.Name)'"
$Job = $Printer | Get-PrintConfiguration -AsJob -EA Ignore
$jobtimer[$Job] = [System.Diagnostics.Stopwatch]::StartNew()
}
}
$JobResult = Get-Job | Wait-Job -Timeout $maxSecondsPerJob -EA Ignore
$JobResult = Get-Job | Stop-Job -EA Ignore # Add this line if you want to stop the Jobs that reached the max time to wait (TimeOut)
$JobResult = Get-Job | Receive-Job -EA Ignore
$JobResult.count