For some reason, the object won't output until the sleep command is done.
[pscustomobject]#{message = 'hi'}; sleep 5
Here's another example. You won't see the output until the loop finishes.
foreach ($i in 1..60) {
if ($i -eq 1) { [pscustomobject]#{message = $i} }
sleep 1
}
I guess you have to output at least 2 objects to see anything? ¯\_(ツ)_/¯ After 15 seconds, you see both objects.
foreach ($i in 1..60) {
if ($i -eq 1 -or $i -eq 15) { [pscustomobject]#{message = $i} }
sleep 1
}
Or output enough properties (> 4) to implicitly call format-list instead of format-table. Format-table is the problem. This comes out right away.
[pscustomobject]#{a=1; b=2; c=3; d=4; e=5}; sleep 10
I wonder if a parameter to format-table could be added like -NoWait.
Known object types with format files containing column widths don't have this problem.
foreach ($i in 1..60) {
if ($i -eq 1) { get-process powershell }
sleep 1
}
Or objects that default to format-custom:
foreach ($i in 1..60) {
if ($i -eq 1) { get-date }
sleep 1
}
tl;dr
If a command's output results in automatic tabular display (implicit Format-Table), display output can situationally be delayed for up to 300 ms. (see below for why and when), which can have two unexpected effects:
As in the question, a subsequent Start-Sleep submitted before the delay has elapsed further delays output for (at least) the duration of the sleep - it effectively suspends completing the 300 ms. wait.
A subsequent Write-Host or Out-Host call can produce output that unexpectedly comes first.
You can force synchronous display output by piping the command to Out-Host or to Format-Table explicitly (or any of the other Format-* cmdlets).
However, doing so means producing for-display output only, which means you lose the ability to (meaningfully) capture or relay the command's output.
# The Out-Host forces instant display, before sleeping starts.
# However, use of Out-Host means you can't capture the output.
[pscustomobject] #{message = 'hi'} | Out-Host; sleep 5
The behavior is explained by the infamous PSv5+ asynchronous behavior of implicitly applied Format-Table output: For data types without predefined formatting data that have 4 or fewer properties (which is what auto-selects table display), it waits for up to 300 msecs. before displaying output, in an effort to determine suitable column widths.
If you use Start-Sleep before that period has elapsed, you suspend waiting for as long as you're sleeping.
Output objects that happen not to trigger implicit Format-Table formatting are not affected, however:
# Immediate output, before sleeping ends:
# Out-of-band formatting of a .NET primitive.
PS> 1; Start-Sleep 5
# Implicit Format-*List* formatting due to having 5+ properties.
PS> [pscustomobject]#{a=1; b=2; c=3; d=4; e=5}; sleep 10
By contrast, because your command's output is an object with just 1 property and its type ([pscustomobject]) has no predefined formatting data associated with it, it triggers implicit Format-Table formatting and therefore exhibits the problem.
In short: The following command outputs are affected, because they select implicit Format-Table output while lacking predefined column widths, necessitating the delay:
objects whose type happens to have 4 or fewer properties
if those types have no associated predefined formatting data (see about_Format.ps1xml), which is generally true for [pscustomobject] instances.
Additionally, but far less commonly, types with formatting data that default to table view but don't have column widths predefined, are also affected (e.g., the System.Guid type instances that New-Guid outputs).
Types without formatting data that have 5 or more properties default to implicitly applied Format-List, where, due to line-by-line output, there's no need to determine useful column widths, and therefore no delay.
Note that this is only a display problem, and that if the command is captured or sent to a pipeline the data is immediately output (though the command won't finish overall until the Start-Sleep period has elapsed):
# The ForEach-Object command's script block receives the [pscustomobject]
# instance right away (and itself prints it *immediately* to the display,
# due to outputting a *string* (which never triggers the asynchronous behavior).
& { [pscustomobject]#{message = 'hi'}; sleep 5 } | ForEach-Object { "[$_]" }
While there are several ways to force synchronous (immediate) display output, they all change the fundamental behavior of the command:
# Piping to Out-Host:
# Directly prints to the *display* (host).
# No way for a caller to capture the result or for processing
# the result in a pipeline.
[pscustomobject]#{message = 'hi'} | Out-Host; sleep 5
# Using Write-Host:
# Prints directly to the *display* (host) by default.
# While it *is* possible to capture the result via output stream 6.
# the information stream (6> file.txt), that output:
# * is invariably converted to *strings*
# * and the string representation does *not* use the friendly default
# output formatting; instead, the objects are stringified with simple
# [psobject.].ToString() calls, which results in a much less friendly
# representation.
Write-Host ([pscustomobject]#{message = 'hi'}); sleep 5
# Piping to a Format-* cmdlet explicitly:
# While this does write to the success-output stream (stream number 1),
# as the command would by default, what is written isn't the original
# objects, but *formatting instructions*, which are useless for further
# programmatic processing.
# However, for redirecting the output to a file with Out-File or >
# this makes no difference, because they convert the formatting instructions
# to the strings you would see on the screen by default.
# By contrast, using Set-Content or any other cmdlet that expects actual data
# would not work meaningfully.
[pscustomobject]#{message = 'hi'} | Format-Table; sleep 5
Pipe your custom object to the Out-Host cmdlet:
[pscustomobject]#{message = 'hi'} | Out-Host; sleep 5
When you use the Out-Host cmdlet, you are immediately displaying the results to the host. Without it, the object is output to the pipeline which is not returned until after the Start-Sleep cmdlet.
Output less than 5 properties, and format-table implicitly runs. Format-table will wait an indefinite amount of time for the second object, before displaying the first object. This is for object types (like pscustomobject) without an xml file that define a default table view.
# no output for 5 seconds
&{get-date
sleep 5
get-date} | format-table
DisplayHint DateTime Date Day DayOfWeek DayOfYear Hour Kind Millisecond Minute
----------- -------- ---- --- --------- --------- ---- ---- ----------- ------
DateTime Saturday, February 8, 2020 10:24:48 AM 2/8/2020 12:00:00 AM 8 Saturday 39 10 Local 618 24
DateTime Saturday, February 8, 2020 10:24:53 AM 2/8/2020 12:00:00 AM 8 Saturday 39 10 Local 892 24
Compare with format-list:
& {get-date
sleep 5
get-date} | format-list
DisplayHint : DateTime
Date : 2/8/2020 12:00:00 AM
Day : 8
DayOfWeek : Saturday
DayOfYear : 39
Hour : 20
Kind : Local
Millisecond : 408
Minute : 37
Month : 2
Second : 18
Ticks : 637167910384087860
TimeOfDay : 20:37:18.4087860
Year : 2020
DateTime : Saturday, February 8, 2020 8:37:18 PM
DisplayHint : DateTime
Date : 2/8/2020 12:00:00 AM
Day : 8
DayOfWeek : Saturday
DayOfYear : 39
Hour : 20
Kind : Local
Millisecond : 662
Minute : 37
Month : 2
Second : 23
Ticks : 637167910436622480
TimeOfDay : 20:37:23.6622480
Year : 2020
DateTime : Saturday, February 8, 2020 8:37:23 PM
Related
Let's say we have an array of objects $objects. Let's say these objects have a "Name" property.
This is what I want to do
$results = #()
$objects | %{ $results += $_.Name }
This works, but can it be done in a better way?
If I do something like:
$results = objects | select Name
$results is an array of objects having a Name property. I want $results to contain an array of Names.
Is there a better way?
I think you might be able to use the ExpandProperty parameter of Select-Object.
For example, to get the list of the current directory and just have the Name property displayed, one would do the following:
ls | select -Property Name
This is still returning DirectoryInfo or FileInfo objects. You can always inspect the type coming through the pipeline by piping to Get-Member (alias gm).
ls | select -Property Name | gm
So, to expand the object to be that of the type of property you're looking at, you can do the following:
ls | select -ExpandProperty Name
In your case, you can just do the following to have a variable be an array of strings, where the strings are the Name property:
$objects = ls | select -ExpandProperty Name
As an even easier solution, you could just use:
$results = $objects.Name
Which should fill $results with an array of all the 'Name' property values of the elements in $objects.
To complement the preexisting, helpful answers with guidance of when to use which approach and a performance comparison.
Outside of a pipeline[1], use (requires PSv3+):
$objects.Name # returns .Name property values from all objects in $objects
as demonstrated in rageandqq's answer, which is both syntactically simpler and much faster.
Accessing a property at the collection level to get its elements' values as an array (if there are 2 or more elements) is called member-access enumeration and is a PSv3+ feature.
Alternatively, in PSv2, use the foreach statement, whose output you can also assign directly to a variable: $results = foreach ($obj in $objects) { $obj.Name }
If collecting all output from a (pipeline) command in memory first is feasible, you can also combine pipelines with member-access enumeration; e.g.:
(Get-ChildItem -File | Where-Object Length -lt 1gb).Name
Tradeoffs:
Both the input collection and output array must fit into memory as a whole.
If the input collection is itself the result of a command (pipeline) (e.g., (Get-ChildItem).Name), that command must first run to completion before the resulting array's elements can be accessed.
In a pipeline, in case you must pass the results to another command, notably if the original input doesn't fit into memory as a whole, use: $objects | Select-Object -ExpandProperty Name
The need for -ExpandProperty is explained in Scott Saad's answer (you need it to get only the property value).
You get the usual pipeline benefits of the pipeline's streaming behavior, i.e. one-by-one object processing, which typically produces output right away and keeps memory use constant (unless you ultimately collect the results in memory anyway).
Tradeoff:
Use of the pipeline is comparatively slow.
For small input collections (arrays), you probably won't notice the difference, and, especially on the command line, sometimes being able to type the command easily is more important.
Here is an easy-to-type alternative, which, however is the slowest approach; it uses ForEach-Object via its built-in alias, %, with simplified syntax (again, PSv3+):
; e.g., the following PSv3+ solution is easy to append to an existing command:
$objects | % Name # short for: $objects | ForEach-Object -Process { $_.Name }
Note: Use of the pipeline is not the primary reason this approach is slow, it is the inefficient implementation of the ForEach-Object (and Where-Object) cmdlets, up to at least PowerShell 7.2. This excellent blog post explains the problem; it led to feature request GitHub issue #10982; the following workaround greatly speeds up the operation (only somewhat slower than a foreach statement, and still faster than .ForEach()):
# Speed-optimized version of the above.
# (Use `&` instead of `.` to run in a child scope)
$objects | . { process { $_.Name } }
The PSv4+ .ForEach() array method, more comprehensively discussed in this article, is yet another, well-performing alternative, but note that it requires collecting all input in memory first, just like member-access enumeration:
# By property name (string):
$objects.ForEach('Name')
# By script block (more flexibility; like ForEach-Object)
$objects.ForEach({ $_.Name })
This approach is similar to member-access enumeration, with the same tradeoffs, except that pipeline logic is not applied; it is marginally slower than member-access enumeration, though still noticeably faster than the pipeline.
For extracting a single property value by name (string argument), this solution is on par with member-access enumeration (though the latter is syntactically simpler).
The script-block variant ({ ... }) allows arbitrary transformations; it is a faster - all-in-memory-at-once - alternative to the pipeline-based ForEach-Object cmdlet (%).
Note: The .ForEach() array method, like its .Where() sibling (the in-memory equivalent of Where-Object), always returns a collection (an instance of [System.Collections.ObjectModel.Collection[psobject]]), even if only one output object is produced.
By contrast, member-access enumeration, Select-Object, ForEach-Object and Where-Object return a single output object as-is, without wrapping it in a collection (array).
Comparing the performance of the various approaches
Here are sample timings for the various approaches, based on an input collection of 10,000 objects, averaged across 10 runs; the absolute numbers aren't important and vary based on many factors, but it should give you a sense of relative performance (the timings come from a single-core Windows 10 VM:
Important
The relative performance varies based on whether the input objects are instances of regular .NET Types (e.g., as output by Get-ChildItem) or [pscustomobject] instances (e.g., as output by Convert-FromCsv).
The reason is that [pscustomobject] properties are dynamically managed by PowerShell, and it can access them more quickly than the regular properties of a (statically defined) regular .NET type. Both scenarios are covered below.
The tests use already-in-memory-in-full collections as input, so as to focus on the pure property extraction performance. With a streaming cmdlet / function call as the input, performance differences will generally be much less pronounced, as the time spent inside that call may account for the majority of the time spent.
For brevity, alias % is used for the ForEach-Object cmdlet.
General conclusions, applicable to both regular .NET type and [pscustomobject] input:
The member-enumeration ($collection.Name) and foreach ($obj in $collection) solutions are by far the fastest, by a factor of 10 or more faster than the fastest pipeline-based solution.
Surprisingly, % Name performs much worse than % { $_.Name } - see this GitHub issue.
PowerShell Core consistently outperforms Windows Powershell here.
Timings with regular .NET types:
PowerShell Core v7.0.0-preview.3
Factor Command Secs (10-run avg.)
------ ------- ------------------
1.00 $objects.Name 0.005
1.06 foreach($o in $objects) { $o.Name } 0.005
6.25 $objects.ForEach('Name') 0.028
10.22 $objects.ForEach({ $_.Name }) 0.046
17.52 $objects | % { $_.Name } 0.079
30.97 $objects | Select-Object -ExpandProperty Name 0.140
32.76 $objects | % Name 0.148
Windows PowerShell v5.1.18362.145
Factor Command Secs (10-run avg.)
------ ------- ------------------
1.00 $objects.Name 0.012
1.32 foreach($o in $objects) { $o.Name } 0.015
9.07 $objects.ForEach({ $_.Name }) 0.105
10.30 $objects.ForEach('Name') 0.119
12.70 $objects | % { $_.Name } 0.147
27.04 $objects | % Name 0.312
29.70 $objects | Select-Object -ExpandProperty Name 0.343
Conclusions:
In PowerShell Core, .ForEach('Name') clearly outperforms .ForEach({ $_.Name }). In Windows PowerShell, curiously, the latter is faster, albeit only marginally so.
Timings with [pscustomobject] instances:
PowerShell Core v7.0.0-preview.3
Factor Command Secs (10-run avg.)
------ ------- ------------------
1.00 $objects.Name 0.006
1.11 foreach($o in $objects) { $o.Name } 0.007
1.52 $objects.ForEach('Name') 0.009
6.11 $objects.ForEach({ $_.Name }) 0.038
9.47 $objects | Select-Object -ExpandProperty Name 0.058
10.29 $objects | % { $_.Name } 0.063
29.77 $objects | % Name 0.184
Windows PowerShell v5.1.18362.145
Factor Command Secs (10-run avg.)
------ ------- ------------------
1.00 $objects.Name 0.008
1.14 foreach($o in $objects) { $o.Name } 0.009
1.76 $objects.ForEach('Name') 0.015
10.36 $objects | Select-Object -ExpandProperty Name 0.085
11.18 $objects.ForEach({ $_.Name }) 0.092
16.79 $objects | % { $_.Name } 0.138
61.14 $objects | % Name 0.503
Conclusions:
Note how with [pscustomobject] input .ForEach('Name') by far outperforms the script-block based variant, .ForEach({ $_.Name }).
Similarly, [pscustomobject] input makes the pipeline-based Select-Object -ExpandProperty Name faster, in Windows PowerShell virtually on par with .ForEach({ $_.Name }), but in PowerShell Core still about 50% slower.
In short: With the odd exception of % Name, with [pscustomobject] the string-based methods of referencing the properties outperform the scriptblock-based ones.
Source code for the tests:
Note:
Download function Time-Command from this Gist to run these tests.
Assuming you have looked at the linked code to ensure that it is safe (which I can personally assure you of, but you should always check), you can install it directly as follows:
irm https://gist.github.com/mklement0/9e1f13978620b09ab2d15da5535d1b27/raw/Time-Command.ps1 | iex
Set $useCustomObjectInput to $true to measure with [pscustomobject] instances instead.
$count = 1e4 # max. input object count == 10,000
$runs = 10 # number of runs to average
# Note: Using [pscustomobject] instances rather than instances of
# regular .NET types changes the performance characteristics.
# Set this to $true to test with [pscustomobject] instances below.
$useCustomObjectInput = $false
# Create sample input objects.
if ($useCustomObjectInput) {
# Use [pscustomobject] instances.
$objects = 1..$count | % { [pscustomobject] #{ Name = "$foobar_$_"; Other1 = 1; Other2 = 2; Other3 = 3; Other4 = 4 } }
} else {
# Use instances of a regular .NET type.
# Note: The actual count of files and folders in your file-system
# may be less than $count
$objects = Get-ChildItem / -Recurse -ErrorAction Ignore | Select-Object -First $count
}
Write-Host "Comparing property-value extraction methods with $($objects.Count) input objects, averaged over $runs runs..."
# An array of script blocks with the various approaches.
$approaches = { $objects | Select-Object -ExpandProperty Name },
{ $objects | % Name },
{ $objects | % { $_.Name } },
{ $objects.ForEach('Name') },
{ $objects.ForEach({ $_.Name }) },
{ $objects.Name },
{ foreach($o in $objects) { $o.Name } }
# Time the approaches and sort them by execution time (fastest first):
Time-Command $approaches -Count $runs | Select Factor, Command, Secs*
[1] Technically, even a command without |, the pipeline operator, uses a pipeline behind the scenes, but for the purpose of this discussion using the pipeline refers only to commands that use |, the pipeline operator, and therefore by definition involve multiple commands.
Caution, member enumeration only works if the collection itself has no member of the same name. So if you had an array of FileInfo objects, you couldn't get an array of file lengths by using
$files.length # evaluates to array length
And before you say "well obviously", consider this. If you had an array of objects with a capacity property then
$objarr.capacity
would work fine UNLESS $objarr were actually not an [Array] but, for example, an [ArrayList]. So before using member enumeration you might have to look inside the black box containing your collection.
(Note to moderators: this should be a comment on rageandqq's answer but I don't yet have enough reputation.)
I learn something new every day! Thank you for this. I was trying to achieve the same. I was directly doing this:
$ListOfGGUIDs = $objects.{Object GUID}
Which basically made my variable an object again! I later realized I needed to define it first as an empty array,
$ListOfGGUIDs = #()
For some reason, the object won't output until the sleep command is done.
[pscustomobject]#{message = 'hi'}; sleep 5
Here's another example. You won't see the output until the loop finishes.
foreach ($i in 1..60) {
if ($i -eq 1) { [pscustomobject]#{message = $i} }
sleep 1
}
I guess you have to output at least 2 objects to see anything? ¯\_(ツ)_/¯ After 15 seconds, you see both objects.
foreach ($i in 1..60) {
if ($i -eq 1 -or $i -eq 15) { [pscustomobject]#{message = $i} }
sleep 1
}
Or output enough properties (> 4) to implicitly call format-list instead of format-table. Format-table is the problem. This comes out right away.
[pscustomobject]#{a=1; b=2; c=3; d=4; e=5}; sleep 10
I wonder if a parameter to format-table could be added like -NoWait.
Known object types with format files containing column widths don't have this problem.
foreach ($i in 1..60) {
if ($i -eq 1) { get-process powershell }
sleep 1
}
Or objects that default to format-custom:
foreach ($i in 1..60) {
if ($i -eq 1) { get-date }
sleep 1
}
tl;dr
If a command's output results in automatic tabular display (implicit Format-Table), display output can situationally be delayed for up to 300 ms. (see below for why and when), which can have two unexpected effects:
As in the question, a subsequent Start-Sleep submitted before the delay has elapsed further delays output for (at least) the duration of the sleep - it effectively suspends completing the 300 ms. wait.
A subsequent Write-Host or Out-Host call can produce output that unexpectedly comes first.
You can force synchronous display output by piping the command to Out-Host or to Format-Table explicitly (or any of the other Format-* cmdlets).
However, doing so means producing for-display output only, which means you lose the ability to (meaningfully) capture or relay the command's output.
# The Out-Host forces instant display, before sleeping starts.
# However, use of Out-Host means you can't capture the output.
[pscustomobject] #{message = 'hi'} | Out-Host; sleep 5
The behavior is explained by the infamous PSv5+ asynchronous behavior of implicitly applied Format-Table output: For data types without predefined formatting data that have 4 or fewer properties (which is what auto-selects table display), it waits for up to 300 msecs. before displaying output, in an effort to determine suitable column widths.
If you use Start-Sleep before that period has elapsed, you suspend waiting for as long as you're sleeping.
Output objects that happen not to trigger implicit Format-Table formatting are not affected, however:
# Immediate output, before sleeping ends:
# Out-of-band formatting of a .NET primitive.
PS> 1; Start-Sleep 5
# Implicit Format-*List* formatting due to having 5+ properties.
PS> [pscustomobject]#{a=1; b=2; c=3; d=4; e=5}; sleep 10
By contrast, because your command's output is an object with just 1 property and its type ([pscustomobject]) has no predefined formatting data associated with it, it triggers implicit Format-Table formatting and therefore exhibits the problem.
In short: The following command outputs are affected, because they select implicit Format-Table output while lacking predefined column widths, necessitating the delay:
objects whose type happens to have 4 or fewer properties
if those types have no associated predefined formatting data (see about_Format.ps1xml), which is generally true for [pscustomobject] instances.
Additionally, but far less commonly, types with formatting data that default to table view but don't have column widths predefined, are also affected (e.g., the System.Guid type instances that New-Guid outputs).
Types without formatting data that have 5 or more properties default to implicitly applied Format-List, where, due to line-by-line output, there's no need to determine useful column widths, and therefore no delay.
Note that this is only a display problem, and that if the command is captured or sent to a pipeline the data is immediately output (though the command won't finish overall until the Start-Sleep period has elapsed):
# The ForEach-Object command's script block receives the [pscustomobject]
# instance right away (and itself prints it *immediately* to the display,
# due to outputting a *string* (which never triggers the asynchronous behavior).
& { [pscustomobject]#{message = 'hi'}; sleep 5 } | ForEach-Object { "[$_]" }
While there are several ways to force synchronous (immediate) display output, they all change the fundamental behavior of the command:
# Piping to Out-Host:
# Directly prints to the *display* (host).
# No way for a caller to capture the result or for processing
# the result in a pipeline.
[pscustomobject]#{message = 'hi'} | Out-Host; sleep 5
# Using Write-Host:
# Prints directly to the *display* (host) by default.
# While it *is* possible to capture the result via output stream 6.
# the information stream (6> file.txt), that output:
# * is invariably converted to *strings*
# * and the string representation does *not* use the friendly default
# output formatting; instead, the objects are stringified with simple
# [psobject.].ToString() calls, which results in a much less friendly
# representation.
Write-Host ([pscustomobject]#{message = 'hi'}); sleep 5
# Piping to a Format-* cmdlet explicitly:
# While this does write to the success-output stream (stream number 1),
# as the command would by default, what is written isn't the original
# objects, but *formatting instructions*, which are useless for further
# programmatic processing.
# However, for redirecting the output to a file with Out-File or >
# this makes no difference, because they convert the formatting instructions
# to the strings you would see on the screen by default.
# By contrast, using Set-Content or any other cmdlet that expects actual data
# would not work meaningfully.
[pscustomobject]#{message = 'hi'} | Format-Table; sleep 5
Pipe your custom object to the Out-Host cmdlet:
[pscustomobject]#{message = 'hi'} | Out-Host; sleep 5
When you use the Out-Host cmdlet, you are immediately displaying the results to the host. Without it, the object is output to the pipeline which is not returned until after the Start-Sleep cmdlet.
Output less than 5 properties, and format-table implicitly runs. Format-table will wait an indefinite amount of time for the second object, before displaying the first object. This is for object types (like pscustomobject) without an xml file that define a default table view.
# no output for 5 seconds
&{get-date
sleep 5
get-date} | format-table
DisplayHint DateTime Date Day DayOfWeek DayOfYear Hour Kind Millisecond Minute
----------- -------- ---- --- --------- --------- ---- ---- ----------- ------
DateTime Saturday, February 8, 2020 10:24:48 AM 2/8/2020 12:00:00 AM 8 Saturday 39 10 Local 618 24
DateTime Saturday, February 8, 2020 10:24:53 AM 2/8/2020 12:00:00 AM 8 Saturday 39 10 Local 892 24
Compare with format-list:
& {get-date
sleep 5
get-date} | format-list
DisplayHint : DateTime
Date : 2/8/2020 12:00:00 AM
Day : 8
DayOfWeek : Saturday
DayOfYear : 39
Hour : 20
Kind : Local
Millisecond : 408
Minute : 37
Month : 2
Second : 18
Ticks : 637167910384087860
TimeOfDay : 20:37:18.4087860
Year : 2020
DateTime : Saturday, February 8, 2020 8:37:18 PM
DisplayHint : DateTime
Date : 2/8/2020 12:00:00 AM
Day : 8
DayOfWeek : Saturday
DayOfYear : 39
Hour : 20
Kind : Local
Millisecond : 662
Minute : 37
Month : 2
Second : 23
Ticks : 637167910436622480
TimeOfDay : 20:37:23.6622480
Year : 2020
DateTime : Saturday, February 8, 2020 8:37:23 PM
I'm trying to write a few scripts using calculated properties (love that feature), and I seem to have an issue when I assign variables in it, for example:
ls | select #{N='What';E={get-acl | Tee-Object -Variable something}},#{N="ok";E={$something}}
it seems that the output I'm getting is:
And I'm unsure if there's a way to save a variable so I can use it in a different calculated property or even just piped to the next command? ( I do know that you can use that variable in the same calculated property though.)
Thanks to anyone who helps.
If you need to construct many inter-related properties and Select-Object is too verbose or inefficient, the idiomatic approach is to pipe to ForEach-Object and construct the output object manually:
Get-ChildItem |ForEach-Object {
$acl = $_ |Get-Acl
# now we can just use this local variable directly
$isOK = Test-WhateverYouNeedToTest $acl
# construct new object
[pscustomobject]#{
FilePath = $_.FullName
ACL = $acl
RuleCount = $acl.Access.Count
IsOK = $isOK
}
}
I'm unsure if there's a way to save a variable so I can use it in a different calculated property
You can't "cross-reference" the resolved value of one calculated property expression from another in the same call, if that's what you mean.
Property expressions are executed in their own local scope, which is why variable assignments also don't persist.
You could (but probably shouldn't) assign to a variable in a parent scope:
ls | select Name,#{N='ACL';E={ ($global:something = $_ |Get-Acl) }},#{N='RuleCount';E={$something.Access.Count}}
I would strongly recommend against it - keeping calculated properties free of side effects will make your code easier to read and understand.
[...] or even just piped to the next command
Again, you could, (but probably shouldn't) assign to a variable in a parent scope and use that with the downstream command:
ls | select #{N='ACL';E={ ($global:something = $_ |Get-Acl) }} | select ACL,#{N='RuleCount';E={$something.Access.Count}}
To explain why this is a bad idea, let's take this example instead:
function TimesTen {
1..10 | select #{N='Base';E={$global:v10 = $_ * 10; $_}} | select Base,#{N='Tenfold';E={$v10}}
}
This might appear to work as intended:
PS ~> TimesTen
Base Tenfold
---- -------
1 10
2 20
3 30
4 40
5 50
6 60
7 70
8 80
9 90
10 100
But this only works as long as pipeline output from Select-Object isn't buffered:
PS ~> $PSDefaultParameterValues['Select-Object:OutBuffer'] = 4
PS ~> TimesTen
Base Tenfold
---- -------
1 50
2 50
3 50
4 50
5 50
6 100
7 100
8 100
9 100
10 100
Oops! Now you need to explicitly prevent that:
function TimesTen {
1..10 | select #{N='Base';E={$global:v10 = $_ * 10; $_}} -OutBuffer 0 | select Base,#{N='Tenfold';E={$v10}}
}
With ForEach-Object, you avoid all of this.
For some reason, the object won't output until the sleep command is done.
[pscustomobject]#{message = 'hi'}; sleep 5
Here's another example. You won't see the output until the loop finishes.
foreach ($i in 1..60) {
if ($i -eq 1) { [pscustomobject]#{message = $i} }
sleep 1
}
I guess you have to output at least 2 objects to see anything? ¯\_(ツ)_/¯ After 15 seconds, you see both objects.
foreach ($i in 1..60) {
if ($i -eq 1 -or $i -eq 15) { [pscustomobject]#{message = $i} }
sleep 1
}
Or output enough properties (> 4) to implicitly call format-list instead of format-table. Format-table is the problem. This comes out right away.
[pscustomobject]#{a=1; b=2; c=3; d=4; e=5}; sleep 10
I wonder if a parameter to format-table could be added like -NoWait.
Known object types with format files containing column widths don't have this problem.
foreach ($i in 1..60) {
if ($i -eq 1) { get-process powershell }
sleep 1
}
Or objects that default to format-custom:
foreach ($i in 1..60) {
if ($i -eq 1) { get-date }
sleep 1
}
tl;dr
If a command's output results in automatic tabular display (implicit Format-Table), display output can situationally be delayed for up to 300 ms. (see below for why and when), which can have two unexpected effects:
As in the question, a subsequent Start-Sleep submitted before the delay has elapsed further delays output for (at least) the duration of the sleep - it effectively suspends completing the 300 ms. wait.
A subsequent Write-Host or Out-Host call can produce output that unexpectedly comes first.
You can force synchronous display output by piping the command to Out-Host or to Format-Table explicitly (or any of the other Format-* cmdlets).
However, doing so means producing for-display output only, which means you lose the ability to (meaningfully) capture or relay the command's output.
# The Out-Host forces instant display, before sleeping starts.
# However, use of Out-Host means you can't capture the output.
[pscustomobject] #{message = 'hi'} | Out-Host; sleep 5
The behavior is explained by the infamous PSv5+ asynchronous behavior of implicitly applied Format-Table output: For data types without predefined formatting data that have 4 or fewer properties (which is what auto-selects table display), it waits for up to 300 msecs. before displaying output, in an effort to determine suitable column widths.
If you use Start-Sleep before that period has elapsed, you suspend waiting for as long as you're sleeping.
Output objects that happen not to trigger implicit Format-Table formatting are not affected, however:
# Immediate output, before sleeping ends:
# Out-of-band formatting of a .NET primitive.
PS> 1; Start-Sleep 5
# Implicit Format-*List* formatting due to having 5+ properties.
PS> [pscustomobject]#{a=1; b=2; c=3; d=4; e=5}; sleep 10
By contrast, because your command's output is an object with just 1 property and its type ([pscustomobject]) has no predefined formatting data associated with it, it triggers implicit Format-Table formatting and therefore exhibits the problem.
In short: The following command outputs are affected, because they select implicit Format-Table output while lacking predefined column widths, necessitating the delay:
objects whose type happens to have 4 or fewer properties
if those types have no associated predefined formatting data (see about_Format.ps1xml), which is generally true for [pscustomobject] instances.
Additionally, but far less commonly, types with formatting data that default to table view but don't have column widths predefined, are also affected (e.g., the System.Guid type instances that New-Guid outputs).
Types without formatting data that have 5 or more properties default to implicitly applied Format-List, where, due to line-by-line output, there's no need to determine useful column widths, and therefore no delay.
Note that this is only a display problem, and that if the command is captured or sent to a pipeline the data is immediately output (though the command won't finish overall until the Start-Sleep period has elapsed):
# The ForEach-Object command's script block receives the [pscustomobject]
# instance right away (and itself prints it *immediately* to the display,
# due to outputting a *string* (which never triggers the asynchronous behavior).
& { [pscustomobject]#{message = 'hi'}; sleep 5 } | ForEach-Object { "[$_]" }
While there are several ways to force synchronous (immediate) display output, they all change the fundamental behavior of the command:
# Piping to Out-Host:
# Directly prints to the *display* (host).
# No way for a caller to capture the result or for processing
# the result in a pipeline.
[pscustomobject]#{message = 'hi'} | Out-Host; sleep 5
# Using Write-Host:
# Prints directly to the *display* (host) by default.
# While it *is* possible to capture the result via output stream 6.
# the information stream (6> file.txt), that output:
# * is invariably converted to *strings*
# * and the string representation does *not* use the friendly default
# output formatting; instead, the objects are stringified with simple
# [psobject.].ToString() calls, which results in a much less friendly
# representation.
Write-Host ([pscustomobject]#{message = 'hi'}); sleep 5
# Piping to a Format-* cmdlet explicitly:
# While this does write to the success-output stream (stream number 1),
# as the command would by default, what is written isn't the original
# objects, but *formatting instructions*, which are useless for further
# programmatic processing.
# However, for redirecting the output to a file with Out-File or >
# this makes no difference, because they convert the formatting instructions
# to the strings you would see on the screen by default.
# By contrast, using Set-Content or any other cmdlet that expects actual data
# would not work meaningfully.
[pscustomobject]#{message = 'hi'} | Format-Table; sleep 5
Pipe your custom object to the Out-Host cmdlet:
[pscustomobject]#{message = 'hi'} | Out-Host; sleep 5
When you use the Out-Host cmdlet, you are immediately displaying the results to the host. Without it, the object is output to the pipeline which is not returned until after the Start-Sleep cmdlet.
Output less than 5 properties, and format-table implicitly runs. Format-table will wait an indefinite amount of time for the second object, before displaying the first object. This is for object types (like pscustomobject) without an xml file that define a default table view.
# no output for 5 seconds
&{get-date
sleep 5
get-date} | format-table
DisplayHint DateTime Date Day DayOfWeek DayOfYear Hour Kind Millisecond Minute
----------- -------- ---- --- --------- --------- ---- ---- ----------- ------
DateTime Saturday, February 8, 2020 10:24:48 AM 2/8/2020 12:00:00 AM 8 Saturday 39 10 Local 618 24
DateTime Saturday, February 8, 2020 10:24:53 AM 2/8/2020 12:00:00 AM 8 Saturday 39 10 Local 892 24
Compare with format-list:
& {get-date
sleep 5
get-date} | format-list
DisplayHint : DateTime
Date : 2/8/2020 12:00:00 AM
Day : 8
DayOfWeek : Saturday
DayOfYear : 39
Hour : 20
Kind : Local
Millisecond : 408
Minute : 37
Month : 2
Second : 18
Ticks : 637167910384087860
TimeOfDay : 20:37:18.4087860
Year : 2020
DateTime : Saturday, February 8, 2020 8:37:18 PM
DisplayHint : DateTime
Date : 2/8/2020 12:00:00 AM
Day : 8
DayOfWeek : Saturday
DayOfYear : 39
Hour : 20
Kind : Local
Millisecond : 662
Minute : 37
Month : 2
Second : 23
Ticks : 637167910436622480
TimeOfDay : 20:37:23.6622480
Year : 2020
DateTime : Saturday, February 8, 2020 8:37:23 PM
Powershell seems to tie select cmdlets to the default output stream, and not release it after a command finishes, affecting future command in unexpected ways. It is not consistent in how it does this, which makes generating predictable output impossible without explicitly directing output when writing the script.
This weirdness happens if you put the commands on separate lines, mix in different object generation cmdlets, etc. It doesn't happen if you run the commands on separate lines interactively, and it doesn't happen with objects automatially created on the command line, unless you mix in objects not automatically created on the command line.
I give command line interactive examples, but it happens if you put these commands into a script with each command on a spearate line and then run the script.
PS /home/dennis> get-date|select dayofweek ; get-date
DayOfWeek
---------
Monday
Monday
PS /home/dennis> "string1"|select length ; "string2"
Length
------
7
string2
And for fun, check out this one:
S /home/dennis> "string0" ;"string1"|select length ; get-host ;"string2" ;get-date; 567 ; get-host
string0
Length
------
7
1
string2
1
567
1
PS /home/dennis> cat test.ps1
"string0"
"string1"|select length
get-host
"string2"
get-date
567
(1..5)
get-host
PS /home/dennis> ./test.ps1
string0
Length
------
7
1
string2
1
567
1
2
3
4
5
1
...
This also affects objects which are not of the same type, and in fact, it affects objects which do not even have the properties in the select statement. Delaying is not an option, and explictly forcing the output with out-host or write-host will directly write to the powershell output device, making it useless to create a script that will be used to produce objects in a pipeline. It also messes up variables. Observe:
PS /home/dennis> $d = get-date | select dayofweek ; $e = get-date ; $d ; $e
DayOfWeek
---------
Monday
Monday
PS /home/dennis> $d
DayOfWeek
---------
Monday
PS /home/dennis> $e
Monday, August 5, 2019 12:33:47 PM
For those who are thinking, it is only a display issue, and the script can be written to display it correctly, I say again, this makes scripts useless as tools you can reuse in other scripts.
Observe how a pipeline inside a script affects commands in an independent interactive shell.
PS /home/dennis> cat test.ps1
"string0"
"string1"|select length
get-host
"string2"
get-date
567
get-host
PS /home/dennis> ./test.ps1|% {$_}
string0
Length
------
7
1
string2
1
567
1
PS /home/dennis> ./test.ps1|% {write-host $_}
string0
#{Length=7}
System.Management.Automation.Internal.Host.InternalHost
string2
8/5/19 12:50:54 PM
567
System.Management.Automation.Internal.Host.InternalHost
PS /home/dennis> ./test.ps1|% {$_|out-host}
string0
Length
------
7
Name : ConsoleHost
Version : 6.2.2
InstanceId : 4e46c643-1a9d-4c55-9151-b311f287a9cb
UI : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture : en-US
CurrentUICulture : en-US
PrivateData : Microsoft.PowerShell.ConsoleHost+ConsoleColorProxy
DebuggerEnabled : True
IsRunspacePushed : False
Runspace : System.Management.Automation.Runspaces.LocalRunspace
string2
Monday, August 5, 2019 1:20:24 PM
567
Name : ConsoleHost
Version : 6.2.2
InstanceId : 4e46c643-1a9d-4c55-9151-b311f287a9cb
UI : System.Management.Automation.Internal.Host.InternalHostUserInterface
CurrentCulture : en-US
CurrentUICulture : en-US
PrivateData : Microsoft.PowerShell.ConsoleHost+ConsoleColorProxy
DebuggerEnabled : True
IsRunspacePushed : False
Runspace : System.Management.Automation.Runspaces.LocalRunspace
In any shell script I expect that a command will execute independently of the previous command.
WhatTheFortran is the logic behind this behaviour? What is the official recommendation to avoid this unpredictability?
I know this is confusing. Any time select-object seems to output a table, format-table is actually running in the background. These format commands are kind of an illusion, and controlled by format files. But the object is still the same underneath.
$a = [pscustomobject]#{name='Joe'}
$b = [pscustomobject]#{address='here'}
$a,$b | format-table
name
----
Joe
$a,$b | format-list
name : Joe
address : here
Other workarounds:
$(get-date|select dayofweek ; get-date) | format-list
$("string1"|select length ; "string2") | format-list
$("string0" ;"string1"|select length ; get-host ;"string2" ;get-date; 567 ; get-host) | format-list
test.ps1 | format-list
The problem with the $d example, is the assignment to $d is only the first statement up to the first semicolon. This is a different issue.
$d = $(get-date | select dayofweek ; $e = get-date ; $d ; $e)
$d | format-list # 3 objects, unless you repeat the last line
An example like this works, because both commands output object types that have .format.ps1xml files. Once you use select-object, the output is a generic PSCustomObject. Actually, you can try putting get-date first in any script.
get-date;get-host
From Windows Powershell in Action:
Out-Default uses steppable pipelines to run the formatter
cmdlets [like format-table] to do its rendering and then
calls Out-Host to display the formatted output.