How to chunk items from pipeline in PowerShell? - powershell

Question

in my PowerShell cmdlet I get an arbitrary number of items via pipeline and want to return chunks of a specified number of items.
When, for example, my script gets as input:
("A", "B", "C", "D", "E", "F", "G")
And I define, let's say 4 as chunk size, I'd like to return something like this:
(
("A", "B", "C", "D"),
("E", "F", "G")
)
Any help would be appreciated.

Answer 1

You can write a simple function that buffers N input objects before spitting out a new array, then output any buffered values you might have left over when you reach the end of the input sequence:
function chunk {
param(
[Parameter(Mandatory = $true, ValueFromPipeline = $true)]
[psobject]$InputObject,
[ValidateRange(1, 100000)]
[int]$ChunkSize = 4
)
begin {
$counter = 0
# Set up array that will act as buffer
$chunk = [object[]]::new($ChunkSize)
}
process {
# Add input object to next available slot in array
$chunk[$counter++] = $InputObject
if($counter -eq $ChunkSize){
# If we've filled the buffer, output it as a new chunk
Write-Output $chunk -NoEnumerate
# Reset counter and buffer
$counter = 0
$chunk = [object[]]::new($ChunkSize)
}
}
end {
if($counter){
# There's no more input but we have some data left over still, output it
Write-Output $chunk[0..($counter-1)] -NoEnumerate
}
}
}
Now you can do:
PS ~> $firstChunk,$nextChunk = "A", "B", "C", "D", "E", "F", "G" |chunk
PS ~> $firstChunk
A
B
C
D
PS ~> $nextChunk
E
F
G

Answer 2

If I could save it to a file first, it could work using get-content's -readcount parameter. I couldn't keep the 2 lists wrapped with regular arrays and +=, so I used an arraylist, hiding the output of arraylist.add(). I wish you could do named pipes on the fly like in zsh.
echo A B C D E F G | set-content file # PS7: 'A'..'G'
get-content file -ReadCount 4 |
% { [collections.arraylist]$list = #() } { $list.add($_) > $null }
$list[0]
A
B
C
D
$list[1]
E
F
G

Answer 3

As of powershell 7 and if you have .net 6 installed:
$arr = [string[]]#("A", "B", "C", "D", "E", "F", "G")
$chunks = [System.Linq.Enumerable]::Chunk($arr, 4)
May be used like so:
foreach ($chunk in $chunks) {
"hello " + $chunk
}
Which outputs as:
hello A B C D
hello E F G
The Chunk function takes any IEnumerable as its first argument.
EDIT: Meh had to do this with powershell 5 :(
So for powershell 5, the accepted answer is good, but if you want a type safe approach:
function chunk {
[CmdletBinding()]
param (
[Parameter(Mandatory = $true, ValueFromPipeline = $true)]
[System.Collections.IEnumerable]$Iter,
[Parameter(Mandatory = $true)][ValidateRange([uint32]::MinValue, [uint32]::MaxValue)]
[uint32]$ChunkSize
)
$type = $iter[0].GetType().Name
$numer = $Iter.GetEnumerator()
$chunks = new-object System.Collections.Generic.List[System.collections.Generic.List[$($type)]]
while ($numer.MoveNext()) {
$i = 1
$chunk = new-object System.collections.Generic.List[$($type)]
$chunk.Add($numer.Current)
while ($i % $ChunkSize -gt 0 -and $numer.MoveNext()) {
$i++
$chunk.Add($numer.Current)
}
$chunks.Add($chunk)
}
return , $chunks
}