Related
I'm curious to test out the performance/usefulness of asynchronous tasks in PowerShell with Start-ThreadJob, Start-Job and Start-Process. I have a folder with about 100 zip files and so came up with the following test:
New-Item "000" -ItemType Directory -Force # Move the old zip files in here
foreach ($i in $zipfiles) {
$name = $i -split ".zip"
Start-Job -scriptblock {
7z.exe x -o"$name" .\$name
Move-Item $i 000\ -Force
7z.exe a $i .\$name\*.*
}
}
The problem with this is that it would start jobs for all 100 zip, which would probably be too much, so I want to set a value $numjobs, say 5, which I can change, such that only $numjobs will be started at the same time, and then the script will check for all 5 of the jobs ending before the next block of 5 will start. I'd like to then watch the CPU and memory depending upon the value of $numjobs
How would I tell a loop only to run 5 times, then wait for the Jobs to finish before continuing?
I see that it's easy to wait for jobs to finish
$jobs = $commands | Foreach-Object { Start-ThreadJob $_ }
$jobs | Receive-Job -Wait -AutoRemoveJobchange
but how might I wait for Start-Process tasks to end?
Although I would like to use Parallel-ForEach, the Enterprises that I work in will be solidly tied to PowerShell 5.1 for the next 3-4 years I expect with no chance to install PowerShell 7.x (although I would be curious for myself to test with Parallel-ForEach on my home system to compare all approaches).
ForEach-Object -Parallel and Start-ThreadJob have built-in functionalities to limit the number of threads that can run at the same time, the same applies for Runspace with their RunspacePool which is what is used behind the scenes by both cmdlets.
Start-Job does not offer such functionality because each Job runs in a separate process as opposed to the cmdlets mentioned before which run in different threads all in the same process. I would also personally not consider it as a parallelism alternative, it is pretty slow and in most cases a linear loop will be faster than it. Serialization and deserialization can be a problem in some cases too.
How to limit the number of running threads?
Both cmdlets offer the -ThrottleLimit parameter for this.
https://learn.microsoft.com/en-us/powershell/module/threadjob/start-threadjob?view=powershell-7.2#-throttlelimit
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.2#-throttlelimit
How would the code look?
$dir = (New-Item "000" -ItemType Directory -Force).FullName
# ForEach-Object -Parallel
$zipfiles | ForEach-Object -Parallel {
$name = [IO.Path]::GetFileNameWithoutExtension($_)
7z.exe x -o $name .\$name
Move-Item $_ $using:dir -Force
7z.exe a $_ .\$name\*.*
} -ThrottleLimit 5
# Start-ThreadJob
$jobs = foreach ($i in $zipfiles) {
Start-ThreadJob {
$name = [IO.Path]::GetFileNameWithoutExtension($using:i)
7z.exe x -o $name .\$name
Move-Item $using:i $using:dir -Force
7z.exe a $using:i .\$name\*.*
} -ThrottleLimit 5
}
$jobs | Receive-Job -Wait -AutoRemoveJob
How to achieve the same having only PowerShell 5.1 available and no ability to install new modules?
The RunspacePool offer this same functionality, either with it's .SetMaxRunspaces(Int32) Method or by targeting one of the RunspaceFactory.CreateRunspacePool overloads offering a maxRunspaces limit as argument.
How would the code look?
$dir = (New-Item "000" -ItemType Directory -Force).FullName
$limit = 5
$iss = [initialsessionstate]::CreateDefault2()
$pool = [runspacefactory]::CreateRunspacePool(1, $limit, $iss, $Host)
$pool.ThreadOptions = [Management.Automation.Runspaces.PSThreadOptions]::ReuseThread
$pool.Open()
$tasks = foreach ($i in $zipfiles) {
$ps = [powershell]::Create().AddScript({
param($path, $dir)
$name = [IO.Path]::GetFileNameWithoutExtension($path)
7z.exe x -o $name .\$name
Move-Item $path $dir -Force
7z.exe a $path .\$name\*.*
}).AddParameters(#{ path = $i; dir = $dir })
$ps.RunspacePool = $pool
#{ Instance = $ps; AsyncResult = $ps.BeginInvoke() }
}
foreach($task in $tasks) {
$task['Instance'].EndInvoke($task['AsyncResult'])
$task['Instance'].Dispose()
}
$pool.Dispose()
Note that for all examples, it's unclear if the 7zip code is correct or not, this answer attempts to demonstrate how async is done in PowerShell not how to zip files / folders.
Below is a helper function that can simplify the process of parallel invocations, tries to emulate ForEach-Object -Parallel and is compatible with PowerShell 5.1, though shouldn't be taken as a robust solution:
using namespace System.Management.Automation
using namespace System.Management.Automation.Runspaces
using namespace System.Collections.Generic
function Invoke-Parallel {
[CmdletBinding()]
param(
[Parameter(Mandatory, ValueFromPipeline, DontShow)]
[object] $InputObject,
[Parameter(Mandatory, Position = 0)]
[scriptblock] $ScriptBlock,
[Parameter()]
[int] $ThrottleLimit = 5,
[Parameter()]
[hashtable] $ArgumentList
)
begin {
$iss = [initialsessionstate]::CreateDefault2()
if($PSBoundParameters.ContainsKey('ArgumentList')) {
foreach($argument in $ArgumentList.GetEnumerator()) {
$iss.Variables.Add([SessionStateVariableEntry]::new($argument.Key, $argument.Value, ''))
}
}
$pool = [runspacefactory]::CreateRunspacePool(1, $ThrottleLimit, $iss, $Host)
$tasks = [List[hashtable]]::new()
$pool.ThreadOptions = [PSThreadOptions]::ReuseThread
$pool.Open()
}
process {
try {
$ps = [powershell]::Create().AddScript({
$args[0].InvokeWithContext($null, [psvariable]::new("_", $args[1]))
}).AddArgument($ScriptBlock.Ast.GetScriptBlock()).AddArgument($InputObject)
$ps.RunspacePool = $pool
$invocationInput = [PSDataCollection[object]]::new(1)
$invocationInput.Add($InputObject)
$tasks.Add(#{
Instance = $ps
AsyncResult = $ps.BeginInvoke($invocationInput)
})
}
catch {
$PSCmdlet.WriteError($_)
}
}
end {
try {
foreach($task in $tasks) {
$task['Instance'].EndInvoke($task['AsyncResult'])
if($task['Instance'].HadErrors) {
$task['Instance'].Streams.Error
}
$task['Instance'].Dispose()
}
}
catch {
$PSCmdlet.WriteError($_)
}
finally {
if($pool) { $pool.Dispose() }
}
}
}
An example of how it works:
# Hashtable Key becomes the Variable Name inside the Runspace!
$outsideVariables = #{ Message = 'Hello from {0}' }
0..10 | Invoke-Parallel {
"[Item $_] - " + $message -f [runspace]::DefaultRunspace.InstanceId
Start-Sleep 5
} -ArgumentList $outsideVariables -ThrottleLimit 3
To add to Santiago Squarzon's helpful answer:
Below is helper function Measure-Parallel, which allows you to compare the speed of the following approaches to parallelism:
Start-Job:
Child-process-based: creates a child PowerShell process behind the scenes, which makes this approach both slow and resource-intensive.
Start-ThreadJob - ships with PowerShell (Core) (v6+) ; installable via Install-Module ThreadJob in Windows PowerShell v5.1:
Thread-based: Much lighter-weight than Start-Job while providing the same functionality; additionally avoids potential loss of type fidelity due to cross-process serialization / deserialization.
ForEach-Object -Parallel - available only in PowerShell (Core) 7.0+:
Thread-based: In essence a simplified wrapper around Start-ThreadJob with support for direct pipeline input and direct output, with invariably synchronous overall execution (all launched threads are waited for).
Start-Process
Child-process-based: Invokes an external program asynchronously by default, on Windows in a new window by default.
Note that this approach only makes sense if your parallel tasks only consist of a single call to an external program, as opposed to needing to execute a block of PowerShell code.
Notably, the only way to capture output with this approach is by redirection to a file, invariably as text only.
Note:
Given that the tests below wrap a single call to an external executable (such as 7z.exe in your case), the Start-Process approach will perform best, because it doesn't have the overhead of job management. However, as noted above, this approach has fundamental limitations.
Due to its complexity, the runspace-pool-based approach from Santiago's answer wasn't included; if Start-ThreadJob or ForEach-Object -Parallel are available to you, you won't need to resort to this approach.
Sample Measure-Parallelism call, which contrast the runtime performance of the approaches:
# Run 20 jobs / processes in parallel, 5 at a time, comparing
# all approaches.
# Note: Omit the -Approach argument to enter interactive mode.
Measure-Parallel -Approach All -BatchSize 5 -JobCount 20
Sample output from a macOS machine running PowerShell 7.2.6 (timings vary based on many factors, but the ratios should provide a sense of relative performance):
# ... output from the jobs
JobCount : 20
BatchSize : 5
BatchCount : 4
Start-Job (secs.) : 2.20
Start-ThreadJob (secs.) : 1.17
Start-Process (secs.) : 0.84
ForEach-Object -Parallel (secs.) : 0.94
Conclusions:
ForEach-Object -Parallel adds the least thread/job-management overhead, followed by Start-ThreadJob
Start-Job, due to needing an extra child process - for the hidden PowerShell instance running each task - is noticeably slower. It seems that on Windows the performance discrepancy is much more pronounced.
Measure-Parallel source code:
Important:
The function hard-codes sample input objects as well as what external program to invoke - you'll have to edit it yourself as needed; the hard-coded external program is the platform-native shell in this case (cmd.exe on Windows, /bin/sh on Unix-like platform), which is passed a command to simply echo each input object.
It wouldn't be too hard to modify the function to accept a script block as an argument, and to receive input objects for the jobs via the pipeline (though that would preclude the Start-Process approach, except if you explicitly call the block via the PowerShell CLI - but in that case Start-Job could just be used).
What the jobs / processes output goes directly to the display and cannot be captured.
The batch size, which defaults to 5, can be modified with -BatchSize; for the thread-based approaches, the batch size is also used as the -ThrottleLimit argument, i.e. the limit on how many threads are allowed to run at the same time. By default, a single batch is run, but you may request multiple batches indirectly by passing the total number of parallel runs to the -JobCount
You can select approaches via the array-valued -Approach parameter, which supports Job, ThreadJob, Process, ForEachParallel, and All, which combines all of the preceding.
If -Approach isn't specified, interactive mode is entered, where you're (repeatedly) prompted for the desired approach.
Except in interactive mode, a custom object with comparative timings is output.
function Measure-Parallel {
[CmdletBinding()]
param(
[ValidateRange(2, 2147483647)] [int] $BatchSize = 5,
[ValidateSet('Job', 'ThreadJob', 'Process', 'ForEachParallel', 'All')] [string[]] $Approach,
[ValidateRange(2, 2147483647)] [int] $JobCount = $BatchSize # pass a higher count to run multiple batches
)
$noForEachParallel = $PSVersionTable.PSVersion.Major -lt 7
$noStartThreadJob = -not (Get-Command -ErrorAction Ignore Start-ThreadJob)
$interactive = -not $Approach
if (-not $interactive) {
# Translate the approach arguments into their corresponding hashtable keys (see below).
if ('All' -eq $Approach) { $Approach = 'Job', 'ThreadJob', 'Process', 'ForEachParallel' }
$approaches = $Approach.ForEach({
if ($_ -eq 'ForEachParallel') { 'ForEach-Object -Parallel' }
else { $_ -replace '^', 'Start-' }
})
}
if ($noStartThreadJob) {
if ($interactive -or $approaches -contains 'Start-ThreadJob') {
Write-Warning "Start-ThreadJob is not installed, omitting its test; install it with ``Install-Module ThreadJob``"
$approaches = $approaches.Where({ $_ -ne 'Start-ThreadJob' })
}
}
if ($noForEachParallel) {
if ($interactive -or $approaches -contains 'ForEach-Object -Parallel') {
Write-Warning "ForEach-Object -Parallel is not available in this PowerShell version (requires v7+), omitting its test."
$approaches = $approaches.Where({ $_ -ne 'ForEach-Object -Parallel' })
}
}
# Simulated input: Create 'f0.zip', 'f1'.zip', ... file names.
$zipFiles = 0..($JobCount - 1) -replace '^', 'f' -replace '$', '.zip'
# Sample executables to run - here, the native shell is called to simply
# echo the argument given.
# The external program to invoke.
$exe = if ($env:OS -eq 'Windows_NT') { 'cmd.exe' } else { 'sh' }
# The list of its arguments *as a single string* - use '{0}' as the placeholder for where the input object should go.
$exeArgList = if ($env:OS -eq 'Windows_NT') { '/c "echo {0}"' } else { '-c "echo {0}"' }
# A hashtable with script blocks that implement the 3 approaches to parallelism.
$approachImpl = [ordered] #{}
$approachImpl['Start-Job'] = { # child-process-based job
param([array] $batch)
$batch |
ForEach-Object {
Start-Job { Invoke-Expression ($using:exe + ' ' + ($using:exeArgList -f $args[0])) } -ArgumentList $_
} |
Receive-Job -Wait -AutoRemoveJob # wait for all jobs, relay their output, then remove them.
}
if (-not $noStartThreadJob) {
# If Start-ThreadJob is available, add an approach for it.
$approachImpl['Start-ThreadJob'] = { # thread-based job - requires Install-Module ThreadJob in WinPS
param([array] $batch)
$batch |
ForEach-Object {
Start-ThreadJob -ThrottleLimit $BatchSize { Invoke-Expression ($using:exe + ' ' + ($using:exeArgList -f $args[0])) } -ArgumentList $_
} |
Receive-Job -Wait -AutoRemoveJob
}
}
if (-not $noForEachParallel) {
# If ForEach-Object -Parallel is supported (v7+), add an approach for it.
$approachImpl['ForEach-Object -Parallel'] = {
param([array] $batch)
$batch | ForEach-Object -ThrottleLimit $BatchSize -Parallel {
Invoke-Expression ($using:exe + ' ' + ($using:exeArgList -f $_))
}
}
}
$approachImpl['Start-Process'] = { # direct execution of an external program
param([array] $batch)
$batch |
ForEach-Object {
Start-Process -NoNewWindow -PassThru $exe -ArgumentList ($exeArgList -f $_)
} |
Wait-Process # wait for all processes to terminate.
}
# Partition the array of all indices into subarrays (batches)
$batches = #(
0..([math]::Ceiling($zipFiles.Count / $batchSize) - 1) | ForEach-Object {
, $zipFiles[($_ * $batchSize)..($_ * $batchSize + $batchSize - 1)]
}
)
# In interactive use, print verbose messages by default
if ($interactive) { $VerbosePreference = 'Continue' }
:menu while ($true) {
if ($interactive) {
# Prompt for the approach to use.
$choices = $approachImpl.Keys.ForEach({
if ($_ -eq 'ForEach-Object -Parallel') { '&' + $_ }
else { $_ -replace '-', '-&' }
}) + '&Quit'
$choice = $host.ui.PromptForChoice("Approach", "Select parallelism approach:", $choices, 0)
if ($choice -eq $approachImpl.Count) { break }
$approachKey = #($approachImpl.Keys)[$choice]
}
else {
# Use the given approach(es)
$approachKey = $approaches
}
$tsTotals = foreach ($appr in $approachKey) {
$i = 0; $tsTotal = [timespan] 0
$batches | ForEach-Object {
$ts = Measure-Command { & $approachImpl[$appr] $_ | Out-Host }
Write-Verbose "$batchSize-element '$appr' batch finished in $($ts.TotalSeconds.ToString('N2')) secs."
$tsTotal += $ts
if (++$i -eq $batches.Count) {
# last batch processed.
if ($batches.Count -gt 1) {
Write-Verbose "'$appr' processing of $JobCount items overall finished in $($tsTotal.TotalSeconds.ToString('N2')) secs."
}
$tsTotal # output the overall timing for this approach
}
elseif ($interactive) {
$choice = $host.ui.PromptForChoice("Continue?", "Select action", ('&Next batch', '&Return to Menu', '&Quit'), 0)
if ($choice -eq 1) { continue menu }
if ($choice -eq 2) { break menu }
}
}
}
if (-not $interactive) {
# Output a result object with the overall timings.
$oht = [ordered] #{}; $i = 0
$oht['JobCount'] = $JobCount
$oht['BatchSize'] = $BatchSize
$oht['BatchCount'] = $batches.Count
foreach ($appr in $approachKey) {
$oht[($appr + ' (secs.)')] = $tsTotals[$i++].TotalSeconds.ToString('N2')
}
[pscustomobject] $oht
break # break out of the infinite :menu loop
}
}
}
You could add a counter to your foreach loop and break if the counter reaches your desired value
$numjobs = 5
$counter = 0
foreach ($i in $zipfiles) {
$counter++
if ($counter -ge $numjobs) {
break
}
<your code>
}
or with Powershells Foreach-Object
$numjobs = 5
$zipfiles | select -first $numjobs | Foreach-Object {
<your code>
}
If you want to process the whole array in batches and wait for each batch to complete you have to save the object that is returned by Start-Job and pass it to Wait-Job like this:
$items = 1..100
$batchsize = 5
while ($true) {
$jobs = #()
$counter = 0
foreach ($i in $items) {
if ($counter -ge $batchsize) {
$items = $items[$batchsize..($items.Length)]
break
}
$jobs += Start-Job -ScriptBlock { Start-Sleep 10 }
$counter++
}
foreach ($job in $jobs) {
$job | Wait-Job | Out-Null
}
if (!$items) {
break
}
}
By design arrays have fixed lengths, that's why I'm rewriting the whole array with $items = $items[$batchsize..($items.Length)]
I have the following code:
$project.PropertyGroup | Foreach-Object {
if($_.GetAttribute('Condition').Trim() -eq $propertyGroupConditionName.Trim()) {
$a = $project.RemoveChild($_);
Write-Host $_.GetAttribute('Condition')"has been removed.";
}
};
Question #1: How do I exit from ForEach-Object? I tried using "break" and "continue", but it doesn't work.
Question #2: I found that I can alter the list within a foreach loop... We can't do it like that in C#... Why does PowerShell allow us to do that?
First of all, Foreach-Object is not an actual loop and calling break in it will cancel the whole script rather than skipping to the statement after it.
Conversely, break and continue will work as you expect in an actual foreach loop.
Item #1. Putting a break within the foreach loop does exit the loop, but it does not stop the pipeline. It sounds like you want something like this:
$todo=$project.PropertyGroup
foreach ($thing in $todo){
if ($thing -eq 'some_condition'){
break
}
}
Item #2. PowerShell lets you modify an array within a foreach loop over that array, but those changes do not take effect until you exit the loop. Try running the code below for an example.
$a=1,2,3
foreach ($value in $a){
Write-Host $value
}
Write-Host $a
I can't comment on why the authors of PowerShell allowed this, but most other scripting languages (Perl, Python and shell) allow similar constructs.
There are differences between foreach and foreach-object.
A very good description you can find here: MS-ScriptingGuy
For testing in PS, here you have scripts to show the difference.
ForEach-Object:
# Omit 5.
1..10 | ForEach-Object {
if ($_ -eq 5) {return}
# if ($_ -ge 5) {return} # Omit from 5.
Write-Host $_
}
write-host "after1"
# Cancels whole script at 15, "after2" not printed.
11..20 | ForEach-Object {
if ($_ -eq 15) {continue}
Write-Host $_
}
write-host "after2"
# Cancels whole script at 25, "after3" not printed.
21..30 | ForEach-Object {
if ($_ -eq 25) {break}
Write-Host $_
}
write-host "after3"
foreach
# Ends foreach at 5.
foreach ($number1 in (1..10)) {
if ($number1 -eq 5) {break}
Write-Host "$number1"
}
write-host "after1"
# Omit 15.
foreach ($number2 in (11..20)) {
if ($number2 -eq 15) {continue}
Write-Host "$number2"
}
write-host "after2"
# Cancels whole script at 25, "after3" not printed.
foreach ($number3 in (21..30)) {
if ($number3 -eq 25) {return}
Write-Host "$number3"
}
write-host "after3"
To stop the pipeline of which ForEach-Object is part just use the statement continue inside the script block under ForEach-Object. continue behaves differently when you use it in foreach(...) {...} and in ForEach-Object {...} and this is why it's possible. If you want to carry on producing objects in the pipeline discarding some of the original objects, then the best way to do it is to filter out using Where-Object.
Since ForEach-Object is a cmdlet, break and continue will behave differently here than with the foreach keyword. Both will stop the loop but will also terminate the entire script:
break:
0..3 | foreach {
if ($_ -eq 2) { break }
$_
}
echo "Never printed"
# OUTPUT:
# 0
# 1
continue:
0..3 | foreach {
if ($_ -eq 2) { continue }
$_
}
echo "Never printed"
# OUTPUT:
# 0
# 1
So far, I have not found a "good" way to break a foreach script block without breaking the script, except "abusing" exceptions, although powershell core uses this approach:
throw:
class CustomStopUpstreamException : Exception {}
try {
0..3 | foreach {
if ($_ -eq 2) { throw [CustomStopUpstreamException]::new() }
$_
}
} catch [CustomStopUpstreamException] { }
echo "End"
# OUTPUT:
# 0
# 1
# End
The alternative (which is not always possible) would be to use the foreach keyword:
foreach:
foreach ($_ in (0..3)) {
if ($_ -eq 2) { break }
$_
}
echo "End"
# OUTPUT:
# 0
# 1
# End
If you insist on using ForEach-Object, then I would suggest adding a "break condition" like this:
$Break = $False;
1,2,3,4 | Where-Object { $Break -Eq $False } | ForEach-Object {
$Break = $_ -Eq 3;
Write-Host "Current number is $_";
}
The above code must output 1,2,3 and then skip (break before) 4. Expected output:
Current number is 1
Current number is 2
Current number is 3
Below is a suggested approach to Question #1 which I use if I wish to use the ForEach-Object cmdlet.
It does not directly answer the question because it does not EXIT the pipeline.
However, it may achieve the desired effect in Q#1.
The only drawback an amateur like myself can see is when processing large pipeline iterations.
$zStop = $false
(97..122) | Where-Object {$zStop -eq $false} | ForEach-Object {
$zNumeric = $_
$zAlpha = [char]$zNumeric
Write-Host -ForegroundColor Yellow ("{0,4} = {1}" -f ($zNumeric, $zAlpha))
if ($zAlpha -eq "m") {$zStop = $true}
}
Write-Host -ForegroundColor Green "My PSVersion = 5.1.18362.145"
I hope this is of use.
Happy New Year to all.
There is a way to break from ForEach-Object without throwing an exception. It employs a lesser-known feature of Select-Object, using the -First parameter, which actually breaks the pipeline when the specified number of pipeline items have been processed.
Simplified example:
$null = 1..5 | ForEach-Object {
# Do something...
Write-Host $_
# Evaluate "break" condition -> output $true
if( $_ -eq 2 ) { $true }
} | Select-Object -First 1 # Actually breaks the pipeline
Output:
1
2
Note that the assignment to $null is there to hide the output of $true, which is produced by the break condition. The value $true could be replaced by 42, "skip", "foobar", you name it. We just need to pipe something to Select-Object so it breaks the pipeline.
I found this question while looking for a way to have fine grained flow control to break from a specific block of code. The solution I settled on wasn't mentioned...
Using labels with the break keyword
From: about_break
A Break statement can include a label that lets you exit embedded
loops. A label can specify any loop keyword, such as Foreach, For, or
While, in a script.
Here's a simple example
:myLabel for($i = 1; $i -le 2; $i++) {
Write-Host "Iteration: $i"
break myLabel
}
Write-Host "After for loop"
# Results:
# Iteration: 1
# After for loop
And then a more complicated example that shows the results with nested labels and breaking each one.
:outerLabel for($outer = 1; $outer -le 2; $outer++) {
:innerLabel for($inner = 1; $inner -le 2; $inner++) {
Write-Host "Outer: $outer / Inner: $inner"
#break innerLabel
#break outerLabel
}
Write-Host "After Inner Loop"
}
Write-Host "After Outer Loop"
# Both breaks commented out
# Outer: 1 / Inner: 1
# Outer: 1 / Inner: 2
# After Inner Loop
# Outer: 2 / Inner: 1
# Outer: 2 / Inner: 2
# After Inner Loop
# After Outer Loop
# break innerLabel Results
# Outer: 1 / Inner: 1
# After Inner Loop
# Outer: 2 / Inner: 1
# After Inner Loop
# After Outer Loop
# break outerLabel Results
# Outer: 1 / Inner: 1
# After Outer Loop
You can also adapt it to work in other situations by wrapping blocks of code in loops that will only execute once.
:myLabel do {
1..2 | % {
Write-Host "Iteration: $_"
break myLabel
}
} while ($false)
Write-Host "After do while loop"
# Results:
# Iteration: 1
# After do while loop
You have two options to abruptly exit out of ForEach-Object pipeline in PowerShell:
Apply exit logic in Where-Object first, then pass objects to Foreach-Object, or
(where possible) convert Foreach-Object into a standard Foreach looping construct.
Let's see examples: Following scripts exit out of Foreach-Object loop after 2nd iteration (i.e. pipeline iterates only 2 times)":
Solution-1: use Where-Object filter BEFORE Foreach-Object:
[boolean]$exit = $false;
1..10 | Where-Object {$exit -eq $false} | Foreach-Object {
if($_ -eq 2) {$exit = $true} #OR $exit = ($_ -eq 2);
$_;
}
OR
1..10 | Where-Object {$_ -le 2} | Foreach-Object {
$_;
}
Solution-2: Converted Foreach-Object into standard Foreach looping construct:
Foreach ($i in 1..10) {
if ($i -eq 3) {break;}
$i;
}
PowerShell should really provide a bit more straightforward way to exit or break out from within the body of a Foreach-Object pipeline. Note: return doesn't exit, it only skips specific iteration (similar to continue in most programming languages), here is an example of return:
Write-Host "Following will only skip one iteration (actually iterates all 10 times)";
1..10 | Foreach-Object {
if ($_ -eq 3) {return;} #skips only 3rd iteration.
$_;
}
HTH
Answer for Question #1 -
You could simply have your if statement stop being TRUE
$project.PropertyGroup | Foreach {
if(($_.GetAttribute('Condition').Trim() -eq $propertyGroupConditionName.Trim()) -and !$FinishLoop) {
$a = $project.RemoveChild($_);
Write-Host $_.GetAttribute('Condition')"has been removed.";
$FinishLoop = $true
}
};
I have a script which pings Systems until it is terminated. The main code block is a function that has a foreach loop nested into a do until loop.
## Start
function start-skript{
Write-Output 'Ping läuft...'
$Repetition_Counter = 0
#Start loop
do {
#Host loop
foreach($system in $Hosts_Array) {
#Ping Function
Ping-Host -HostToPing $system
#Write-Host 'Sleep for' $Pause 'sekunde'
Start-Sleep -Seconds $Pause
} $Repetition_Counter ++
} until($Repetition_Counter -eq $Repetition -or (stop-script))
}
start-skript
It works finely. However, by invoking the stop-script function, the script can only be stopped, only after the function ping-host has been applied to all the devices in the $Hosts_Array. I wan to be able to stop the script by invoking the stop-script function, even if the funtion Ping-host hasn't applied to all the devices in the $Hosts_Array array . I thought, I could do like below
## Start
function start-skript{
Write-Output 'Ping läuft...'
$Repetition_Counter = 0
#Start loop
do {
#Host loop
foreach($system in $Hosts_Array) {
#Ping Function
Ping-Host -HostToPing $system
#Write-Host 'Sleep for' $Pause 'sekunde'
**stop-script** ## I've just added the function here
Start-Sleep -Seconds $Pause
} $Repetition_Counter ++
} until($Repetition_Counter -eq $Repetition -or (stop-script))
}
start-skript
I didn't work out. It just comes out of the foreach loop but then starts again with the foreach loop, bcz the foreach loop is into the do until loop..
Any suggestions??
Use a statement label:
:outer
do
{
:inner
foreach($i in 1..10){
if($i -eq 2){
continue inner
}
if($i -gt 5){
break outer
}
$i
}
}while($true)
Output will be:
1
3
4
5
Since we continue'd the inner loop on $i -eq 2 and broke out of the outer loop after $i -eq 5
I have the following code:
$project.PropertyGroup | Foreach-Object {
if($_.GetAttribute('Condition').Trim() -eq $propertyGroupConditionName.Trim()) {
$a = $project.RemoveChild($_);
Write-Host $_.GetAttribute('Condition')"has been removed.";
}
};
Question #1: How do I exit from ForEach-Object? I tried using "break" and "continue", but it doesn't work.
Question #2: I found that I can alter the list within a foreach loop... We can't do it like that in C#... Why does PowerShell allow us to do that?
First of all, Foreach-Object is not an actual loop and calling break in it will cancel the whole script rather than skipping to the statement after it.
Conversely, break and continue will work as you expect in an actual foreach loop.
Item #1. Putting a break within the foreach loop does exit the loop, but it does not stop the pipeline. It sounds like you want something like this:
$todo=$project.PropertyGroup
foreach ($thing in $todo){
if ($thing -eq 'some_condition'){
break
}
}
Item #2. PowerShell lets you modify an array within a foreach loop over that array, but those changes do not take effect until you exit the loop. Try running the code below for an example.
$a=1,2,3
foreach ($value in $a){
Write-Host $value
}
Write-Host $a
I can't comment on why the authors of PowerShell allowed this, but most other scripting languages (Perl, Python and shell) allow similar constructs.
There are differences between foreach and foreach-object.
A very good description you can find here: MS-ScriptingGuy
For testing in PS, here you have scripts to show the difference.
ForEach-Object:
# Omit 5.
1..10 | ForEach-Object {
if ($_ -eq 5) {return}
# if ($_ -ge 5) {return} # Omit from 5.
Write-Host $_
}
write-host "after1"
# Cancels whole script at 15, "after2" not printed.
11..20 | ForEach-Object {
if ($_ -eq 15) {continue}
Write-Host $_
}
write-host "after2"
# Cancels whole script at 25, "after3" not printed.
21..30 | ForEach-Object {
if ($_ -eq 25) {break}
Write-Host $_
}
write-host "after3"
foreach
# Ends foreach at 5.
foreach ($number1 in (1..10)) {
if ($number1 -eq 5) {break}
Write-Host "$number1"
}
write-host "after1"
# Omit 15.
foreach ($number2 in (11..20)) {
if ($number2 -eq 15) {continue}
Write-Host "$number2"
}
write-host "after2"
# Cancels whole script at 25, "after3" not printed.
foreach ($number3 in (21..30)) {
if ($number3 -eq 25) {return}
Write-Host "$number3"
}
write-host "after3"
To stop the pipeline of which ForEach-Object is part just use the statement continue inside the script block under ForEach-Object. continue behaves differently when you use it in foreach(...) {...} and in ForEach-Object {...} and this is why it's possible. If you want to carry on producing objects in the pipeline discarding some of the original objects, then the best way to do it is to filter out using Where-Object.
Since ForEach-Object is a cmdlet, break and continue will behave differently here than with the foreach keyword. Both will stop the loop but will also terminate the entire script:
break:
0..3 | foreach {
if ($_ -eq 2) { break }
$_
}
echo "Never printed"
# OUTPUT:
# 0
# 1
continue:
0..3 | foreach {
if ($_ -eq 2) { continue }
$_
}
echo "Never printed"
# OUTPUT:
# 0
# 1
So far, I have not found a "good" way to break a foreach script block without breaking the script, except "abusing" exceptions, although powershell core uses this approach:
throw:
class CustomStopUpstreamException : Exception {}
try {
0..3 | foreach {
if ($_ -eq 2) { throw [CustomStopUpstreamException]::new() }
$_
}
} catch [CustomStopUpstreamException] { }
echo "End"
# OUTPUT:
# 0
# 1
# End
The alternative (which is not always possible) would be to use the foreach keyword:
foreach:
foreach ($_ in (0..3)) {
if ($_ -eq 2) { break }
$_
}
echo "End"
# OUTPUT:
# 0
# 1
# End
If you insist on using ForEach-Object, then I would suggest adding a "break condition" like this:
$Break = $False;
1,2,3,4 | Where-Object { $Break -Eq $False } | ForEach-Object {
$Break = $_ -Eq 3;
Write-Host "Current number is $_";
}
The above code must output 1,2,3 and then skip (break before) 4. Expected output:
Current number is 1
Current number is 2
Current number is 3
Below is a suggested approach to Question #1 which I use if I wish to use the ForEach-Object cmdlet.
It does not directly answer the question because it does not EXIT the pipeline.
However, it may achieve the desired effect in Q#1.
The only drawback an amateur like myself can see is when processing large pipeline iterations.
$zStop = $false
(97..122) | Where-Object {$zStop -eq $false} | ForEach-Object {
$zNumeric = $_
$zAlpha = [char]$zNumeric
Write-Host -ForegroundColor Yellow ("{0,4} = {1}" -f ($zNumeric, $zAlpha))
if ($zAlpha -eq "m") {$zStop = $true}
}
Write-Host -ForegroundColor Green "My PSVersion = 5.1.18362.145"
I hope this is of use.
Happy New Year to all.
There is a way to break from ForEach-Object without throwing an exception. It employs a lesser-known feature of Select-Object, using the -First parameter, which actually breaks the pipeline when the specified number of pipeline items have been processed.
Simplified example:
$null = 1..5 | ForEach-Object {
# Do something...
Write-Host $_
# Evaluate "break" condition -> output $true
if( $_ -eq 2 ) { $true }
} | Select-Object -First 1 # Actually breaks the pipeline
Output:
1
2
Note that the assignment to $null is there to hide the output of $true, which is produced by the break condition. The value $true could be replaced by 42, "skip", "foobar", you name it. We just need to pipe something to Select-Object so it breaks the pipeline.
I found this question while looking for a way to have fine grained flow control to break from a specific block of code. The solution I settled on wasn't mentioned...
Using labels with the break keyword
From: about_break
A Break statement can include a label that lets you exit embedded
loops. A label can specify any loop keyword, such as Foreach, For, or
While, in a script.
Here's a simple example
:myLabel for($i = 1; $i -le 2; $i++) {
Write-Host "Iteration: $i"
break myLabel
}
Write-Host "After for loop"
# Results:
# Iteration: 1
# After for loop
And then a more complicated example that shows the results with nested labels and breaking each one.
:outerLabel for($outer = 1; $outer -le 2; $outer++) {
:innerLabel for($inner = 1; $inner -le 2; $inner++) {
Write-Host "Outer: $outer / Inner: $inner"
#break innerLabel
#break outerLabel
}
Write-Host "After Inner Loop"
}
Write-Host "After Outer Loop"
# Both breaks commented out
# Outer: 1 / Inner: 1
# Outer: 1 / Inner: 2
# After Inner Loop
# Outer: 2 / Inner: 1
# Outer: 2 / Inner: 2
# After Inner Loop
# After Outer Loop
# break innerLabel Results
# Outer: 1 / Inner: 1
# After Inner Loop
# Outer: 2 / Inner: 1
# After Inner Loop
# After Outer Loop
# break outerLabel Results
# Outer: 1 / Inner: 1
# After Outer Loop
You can also adapt it to work in other situations by wrapping blocks of code in loops that will only execute once.
:myLabel do {
1..2 | % {
Write-Host "Iteration: $_"
break myLabel
}
} while ($false)
Write-Host "After do while loop"
# Results:
# Iteration: 1
# After do while loop
You have two options to abruptly exit out of ForEach-Object pipeline in PowerShell:
Apply exit logic in Where-Object first, then pass objects to Foreach-Object, or
(where possible) convert Foreach-Object into a standard Foreach looping construct.
Let's see examples: Following scripts exit out of Foreach-Object loop after 2nd iteration (i.e. pipeline iterates only 2 times)":
Solution-1: use Where-Object filter BEFORE Foreach-Object:
[boolean]$exit = $false;
1..10 | Where-Object {$exit -eq $false} | Foreach-Object {
if($_ -eq 2) {$exit = $true} #OR $exit = ($_ -eq 2);
$_;
}
OR
1..10 | Where-Object {$_ -le 2} | Foreach-Object {
$_;
}
Solution-2: Converted Foreach-Object into standard Foreach looping construct:
Foreach ($i in 1..10) {
if ($i -eq 3) {break;}
$i;
}
PowerShell should really provide a bit more straightforward way to exit or break out from within the body of a Foreach-Object pipeline. Note: return doesn't exit, it only skips specific iteration (similar to continue in most programming languages), here is an example of return:
Write-Host "Following will only skip one iteration (actually iterates all 10 times)";
1..10 | Foreach-Object {
if ($_ -eq 3) {return;} #skips only 3rd iteration.
$_;
}
HTH
Answer for Question #1 -
You could simply have your if statement stop being TRUE
$project.PropertyGroup | Foreach {
if(($_.GetAttribute('Condition').Trim() -eq $propertyGroupConditionName.Trim()) -and !$FinishLoop) {
$a = $project.RemoveChild($_);
Write-Host $_.GetAttribute('Condition')"has been removed.";
$FinishLoop = $true
}
};
I'm trying to get some information of an Remove-Item operation in PowerShell.
Since I don't want the loop to stop when one Item on Remove-Item failes, I can't use Try{} catch{} and -ErrorAction Stop
Is there a way to get error information i want without clearing the error variable before Remove-Item, and also without having to use a loop to iterate over the files?
$error.clear()
$Files | Remove-Item -Force
0..($error.count - 1) | % {
$x = $Error[$_].CategoryInfo
$y = "{0}, {1}, {2}" -f $x.Category, $x.Reason, $x.TargetName
$ResultLog += [PSCustomObject]#{Result="Error"; Path=$p.path; Message=$y}
}
I like #HAL9256's gusto but I think using $Error.count is a bad idea. The count only goes up to 256 items before it stops counting up and starts dropping off the oldest errors. Depending on the volume of files and errors, you could easily run out of room there.
https://devblogs.microsoft.com/scripting/powershell-error-handling-and-why-you-should-care/
Rather than using the pipeline, I think a foreach would be better suited.
$ResultLog = #()
foreach ($file in $files) {
try {
Remove-Item $file -Force -ErrorAction Stop
} catch {
$x = $_.CategoryInfo
$y = "{0}, {1}, {2}" -f $x.Category, $x.Reason, $x.TargetName
$ResultLog += [PSCustomObject]#{Result="Error"; Path=$p.path; Message=$y}
}
}
Use -ErrorAction Continue It won't halt the running of the script but will still add to the $Error variable.
To not have to clear the $Error variable before running, since the $Error variable is an array, simply store the Error count before running, and then use a For loop to only iterate through the new messages.
$ErrorsBefore = $Error.Count
$Files | Remove-Item -Force -ErrorAction Continue
$ResultLog = #()
For($i=0 ; $i -lt ($error.count - $ErrorsBefore); $i++) {
$x = $Error[$i].CategoryInfo
$y = "{0}, {1}, {2}" -f $x.Category, $x.Reason, $x.TargetName
$ResultLog += [PSCustomObject]#{Result="Error"; Path=$p.path; Message=$y}
}