Warning: Powershell novice, terms/code may be incorrect
Say I have a Powershell array with data from Invoke-Sqlcmd, say 10,000 "rows" with a half dozen (Col1-Col6) columns each. Say I want to find every row with an empty Col5. Right now I'm doing
foreach ($row in $sqlimport.col1)
{ $col5data = $sqlimport.Where({$_.col1 -eq $row}) | Select-Object -ExpandProperty Col5
if ( '' -eq $col5data ) {$col5data+1} else {} $col5data
Returns 1, which it should in my test. Code seems wrong and slow. It takes several minutes to run through. When something like
$sqlimport.Where({$_.col5 -eq 'somedatathatisthere'})
takes milliseconds
However,
$sqlimport.Where({$_.col5 -eq ''})
Returns blank
($sqlimport.Where({$_.col5 -eq ''})).Count
Returns 0
Right now, you're asking PowerShell to create an array consisting of all the values in column col1, and then for each iteration you search the entire array again to find the corresponding col5 value. That's entirely unnecessary.
Simply loop over the array itself:
foreach($row in $sqlimport){
if($row.Col5 -like ''){
Write-Host "Col5 is empty in row with id $($row.Col1)"
}
}
This only iterates over the entire array once.
Mathias' answer explains the problem with the inefficiency of your first attempt well and offers a solution that ultimately performs best, due to use of a (single) foreach statement (as opposed to the much slower ForEach-Object cmdlet).
However, a solution using a (single) .Where() array method call (or the related .ForEach() method):
is only slightly slower[1]
while being more concise and arguably conceptually more elegant.
It is what you tried with $sqlimport.Where({ $_.col5 -eq '' }), which, based on your feedback, requires only one small tweak to make it work:
Instead of -eq '' use -like '', which is a somewhat obscure[2] shortcut to testing for an empty string ('') and also for a $null value (as also used in Mathias' answer) - it is, in essence, the equivalent of .NET's [string]::IsNullOrEmpty() method.
# -like matches both '' and $null
$sqlimport.Where({ $_.col5 -like '' })
Note:
If you wanted to test for $null values only, use the following; note how $null is placed on the LHS, which is generally advisable for robust comparisons[3]:
$sqlimport.Where({ $null -eq $_.col5 })
As an aside: GitHub issue #10656 proposes introducing a simplified test for $null with something like -is $null; unfortunately, the associated PR has been abandonded.
As an aside: [string]-type-constrained PowerShell variables never store $null values: [string] $str = $null causes $str to contain '', not $null. However, [string]-typed property values not populated in PowerShell code can contain $null values, as in your case.
For passing a true $null value to a string-typed .NET API in PowerShell code, a special singleton must be used (passing $null directly would again result in ''): [NullString]::Value`
See this answer for background information.
[1] For a performance comparison of PowerShell's various enumeration (iteration) features, see the bottom section of this answer.
[2] The real purpose of the -like operator is to perform wildcard-expression matching. The shortcut relies on -like - which operates on (non-null) strings only - auto-converting non-string operands to strings, which in the case of $null causes conversion to '' (the empty string).
[3] To reliably test for $null, place it on the LHS of an -eq / -ne operation; e.g., $null -eq $var. If you place $null on the RHS - $var -eq $null - and $var happens to be a collection (such as an array), the return value is the array of matching elements, i.e. the array of those elements in $var whose value is $null, which is a different operation - see about_Comparison_Operators.
I'm trying to return TRUE from searching Get-ComplianceSearch's output for 'Completed'. My code below is a simple wait loop. But I don't think I'm returning the value correctly because the loop never finishes. I'm fairly new to PowerShell. Please assist or direct.
I'm using Powershell Core 7.1. There are no errors but the Search-String condition never returns TRUE.
try {
$timer = [Diagnostics.Stopwatch]::StartNew()
while (($timer.Elapsed.TotalSeconds -lt $Timeout) -and (-not (Get-ComplianceSearch -
Identity $searchName | Select-String 'Completed' -SimpleMatch -Quiet))) {
Start-Sleep -Seconds $RetryInterval
$totalSecs = [math]::Round($timer.Elapsed.TotalSeconds, 0)
Write-Verbose -Message "Still waiting for action to complete after [$totalSecs]
seconds..."
}
$timer.Stop()
if ($timer.Elapsed.TotalSeconds -gt $Timeout) {
throw 'Action did not complete before timeout period.'
} else {
Write-Verbose -Message 'Action completed before timeout period.'
}
} catch {
Write-Error -Message $_.Exception.Message
}
(This is the expected output of the command Get-ComplianceSearch)
Okay, you don't want to use Select-String here (although you can, see #mklement0's helpful answer, looking at object properties is usually preferred). That is returning an object and you want to check the Status property for "Completed". Make the following change to the -not subexpression:
(-not (Get-ComplianceSearch -Identity $searchName | Where-Object {
$_.Status -eq 'Completed'
}))
The above can be on one line but I broke it up for readability.
Basically, Select-String looks for content in strings. If you are looking for a particular value of an object property however, you can use Where-Object to test for a condition and return any objects matching that condition. In this case, we want to return any object that have a Status of 'Completed', so we can negate that in the if statement.
You (or others) might be wondering how this works since Where-Object returns matching objects, but not booleans. The answer is "truthiness". PowerShell objects are "truthy", which means anything can be evaluated as a [bool].
The following values evaluate to $false in most cases. I've included some gotchas to watch out for when relying on "truthy" values:
A numeric value of 0
A string value of 0 evaluates as $true
Empty arrays
Empty strings
A whitespace-only string or strings consisting only of non-printable characters evaluates as $true
$false
A string value of False evaluates as $true
Most everything else will evaluate to $true. This is also why comparison operators are syntactically optional when checking whether a variable is $null or not. Although there are times when an explicit value check is a good idea as comparison operators compare the actual values instead of only whether the variable "is" or "isn't".
How does this apply to the expression above then? Simple. if statements, always treat the condition expression as a [bool], no conversion required. In addition, logical operators and conditional operators also imply a boolean comparison. For example, $var = $obj assigns $obj to $var, but$var = $obj -eq $obj2 or $var = $obj -and $obj2 will assign $true or $false.
So knowing the above, if Where-Object returns nothing, it's $false. If it returns a tangible object, it's $true.
Bender the Greatest's helpful answer shows a better alternative to using Select-String, because OO-based filtering that queries specific properties is always more robust than searching string representations.
That said, for quick-and-dirty interactive searches, being able to search through a command's formatted display output can be handy, and, unfortunately, Select-String does not do that by default.
As for what you tried:
To make your Select-String work, you need to insert Out-String -Stream before the Select-String call, so as to ensure that the for-display representation is sent through the pipeline, line by line.
# `oss` can be used in lieu of `Out-String -Stream` in PSv5+.
# `sls` can be used in lieu of `Select-String`.
Get-ComplianceSearch | Out-String -Stream | Select-String 'Completed' -SimpleMatch -Quiet
Note:
If you want to search a for-display representation other than the default one, you can insert a Format-* cmdlet call before the Out-String -Stream segment; e.g.
Get-Item / | Format-List * | Out-String -Stream | Select-String ... would search through a list representation of all properties of the object output by Get-Item.
Perhaps surprisingly, Select-String does not search an input object's for-display representation, as you would see it in the console, using the rich formatting provided by PowerShell's display-formatting system.
Instead, it performs simple .ToString() stringification, whose results are often unhelpful and cannot be relied upon to include the values of properties. (E.g.,
#{ foo = 'bar' } | Select-String foo does not work as intended; it is equivalent to
#{ foo = 'bar' }.ToString() | Select-String foo and therefore to
'System.Collections.Hashtable' | Select-String foo
Arguably, Select-String should always have defaulted to searching through the input objects' formatted string representations:
That there is demand for this behavior is evidenced by the fact that PowerShell versions 5 and above (both editions) ship with the oss convenience function, which is a wrapper for Out-String -Stream.
GitHub issue #10726 asks that the current behavior of Select-String be changed to search the for-display string representations by default.
Here is the code below I am using:
$ProcessesToCheckFor = (
'company_name_uat-Historian'
)
$FoundProcesses = Get-Process -Name $ProcessesToCheckFor -ErrorAction SilentlyContinue
foreach ($Item in $ProcessesToCheckFor)
{
if ($FoundProcesses.Name -contains $Item)
{
'{0} runn1ng' -f $Item
}
else
{
'{0} fai1ed' -f $Item
}
}
The code checks to see if company_name_uat-Historian process is running on a server, and if it is running, it will output runn1ng and fai1ed if not.
The problem is, it works when checking only one process just like the code above, but not when you try to check a list of processes.
I need to check a list of processes, so when I chain the rest as listed below:
$ProcessesToCheckFor = (
'company_name_uat-Historian',
'company_name_uat-KEReviewCollector',
'company_name_uat-lwm',
'company_name_uat-MQAck',
'company_name_uat-MQOutput',
'company_name_uat-SQAC',
'company_name_uat-Store',
'company_name_uat-Store_STS',
'company_name_uat-StoreLegacy',
'spotify'
)
$FoundProcesses = Get-Process -Name $ProcessesToCheckFor -ErrorAction SilentlyContinue
foreach ($Item in $ProcessesToCheckFor)
{
if ($FoundProcesses.Name -contains $Item)
{
'{0} runn1ng' -f $Item
}
else
{
'{0} fai1ed' -f $Item
}
They all output fai1ed.
If I do them one by one, each different process will return runn1ng, just all bunched up together they return fai1ed.
Side notes (if anyone is wondering):
runn1ng and fai1ed are 'code words' that get replaced by images using JavaScript. I am making an HTML dashboard that monitors my Windows servers for work, so think green checkmarks and red x marks and what not.
Using PowerShell version 2 is not my choice at all, some of the servers I am responsible for monitoring are Windows 2008 R2. I believe they are going to be updated sometime this year, but the project deliverable requires me to enact a monitoring solution immediately. There are also some servers which are 2012 that I love to write PowerShell scripts for.
spotify is listed as a process because I know it's a legit process but is not installed on our servers. When I was writing the script, I initially tested on my own personal machine and used spotify as a means to test runn1ng if I opened it or fai1ed if I closed it. If all my processes are runn1ng and spotify is fai1ed, then that's an indication that the PS code is working.
It appears to happen in PowerShell version 2.
Any ideas what is causing it and how I can rewrite it?
The Answer
if (($FoundProcesses | Select-Object -ExpandProperty Name) -contains $Item) {
Why
PowerShell 3 added something called member enumeration. If you have an array of objects in 2.0, you can't directly call properties of the array, because it looks for those properties on the array object itself. In 3.0+ if the member doesn't exist on the array, it will check the items for the member as well. Using Select-Object -ExpandProperty is a more explicit way of calling the members.
You could also just move the Get-Process call into the foreach loop.
foreach ($Item in $ProcessesToCheckFor)
{
if (Get-Process -Name $Item -ErrorAction SilentlyContinue)
{
'{0} runn1ng' -f $Item
}
else
{
'{0} fai1ed' -f $Item
}
}
Patrick Meinecke's answer is effective and helpful, but let me attempt a more detailed explanation and a more efficient solution:
First things first: Your code works just fine in PSv3+, which should provide an incentive to leave PSv2 behind.
PSv3 introduced unified handling of scalars and collections through member(-access) enumeration, which bridged the great scalar/array divide that plagued PSv2-.
Specifically, in PSv3 you needn't worry about whether Get-Process -Name $ProcessesToCheckFor happens to return a single or multiple items: in either case you can call .Count on the output to count the number of items returned and you can invoke a member (property) on the result, which, in the case of an array result (multiple items) means that the member is called on each element (e.g., .Name), and the result can be again be used as a scalar or array as needed.
In PSv2-, however, you need to be aware of whether a cmdlet - situationally - happens to return a single item or multiple ones, which means:
Use array-subexpression operator #(...) to ensure that even if the enclosed command happens to return a single item (scalar), the result is an array.
To collect the property values of the elements of a given array or of a scalar in a new array or scalar, use ... | Select-Object -ExpandProperty - again, apply #(...) to ensure that the result is an array.
Note that PSv2- had some unified scalar/array handling, but only with respect to pipeline processing: to send a scalar (single item) to the pipeline, you can use it as-is, without needing to wrap it in #(...):
'foo' | ... works just fine - no need for #('foo') | ... (even though that works too).
Applied to your code we get:
$ProcessesToCheckFor = (
'company_name_uat-Historian',
'company_name_uat-KEReviewCollector',
'company_name_uat-lwm',
'company_name_uat-MQAck',
'company_name_uat-MQOutput',
'company_name_uat-SQAC',
'company_name_uat-Store',
'company_name_uat-Store_STS',
'company_name_uat-StoreLegacy',
'spotify'
)
# Collect the names of all running processes and
# make sure the result is an array (`#(...)`).
$FoundProcesses = #(Get-Process -Name $ProcessesToCheckFor -ErrorAction SilentlyContinue |
Select-Object -ExpandProperty Name)
foreach ($Item in $ProcessesToCheckFor) {
if ($FoundProcesses -contains $Item) {
'{0} runn1ng' -f $Item
} else {
'{0} fai1ed' -f $Item
}
}
I have written a simple PowerShell filter that pushes the current object down the pipeline if its date is between the specified begin and end date. The objects coming down the pipeline are always in ascending date order so as soon as the date exceeds the specified end date I know my work is done and I would like to let tell the pipeline that the upstream commands can abandon their work so that the pipeline can finish its work. I am reading some very large log files and I will frequently want to examine just a portion of the log. I am pretty sure this is not possible but I wanted to ask to be sure.
It is possible to break a pipeline with anything that would otherwise break an outside loop or halt script execution altogether (like throwing an exception). The solution then is to wrap the pipeline in a loop that you can break if you need to stop the pipeline. For example, the below code will return the first item from the pipeline and then break the pipeline by breaking the outside do-while loop:
do {
Get-ChildItem|% { $_;break }
} while ($false)
This functionality can be wrapped into a function like this, where the last line accomplishes the same thing as above:
function Breakable-Pipeline([ScriptBlock]$ScriptBlock) {
do {
. $ScriptBlock
} while ($false)
}
Breakable-Pipeline { Get-ChildItem|% { $_;break } }
It is not possible to stop an upstream command from a downstream command.. it will continue to filter out objects that do not match your criteria, but the first command will process everything it was set to process.
The workaround will be to do more filtering on the upstream cmdlet or function/filter. Working with log files makes it a bit more comoplicated, but perhaps using Select-String and a regular expression to filter out the undesired dates might work for you.
Unless you know how many lines you want to take and from where, the whole file will be read to check for the pattern.
You can throw an exception when ending the pipeline.
gc demo.txt -ReadCount 1 | %{$num=0}{$num++; if($num -eq 5){throw "terminated pipeline!"}else{write-host $_}}
or
Look at this post about how to terminate a pipeline: https://web.archive.org/web/20160829015320/http://powershell.com/cs/blogs/tobias/archive/2010/01/01/cancelling-a-pipeline.aspx
Not sure about your exact needs, but it may be worth your time to look at Log Parser to see if you can't use a query to filter the data before it even hits the pipe.
If you're willing to use non-public members here is a way to stop the pipeline. It mimics what select-object does. invoke-method (alias im) is a function to invoke non-public methods. select-property (alias selp) is a function to select (similar to select-object) non-public properties - however it automatically acts like -ExpandProperty if only one matching property is found. (I wrote select-property and invoke-method at work, so can't share the source code of those).
# Get the system.management.automation assembly
$script:smaa=[appdomain]::currentdomain.getassemblies()|
? location -like "*system.management.automation*"
# Get the StopUpstreamCommandsException class
$script:upcet=$smaa.gettypes()| ? name -like "*StopUpstreamCommandsException *"
function stop-pipeline {
# Create a StopUpstreamCommandsException
$upce = [activator]::CreateInstance($upcet,#($pscmdlet))
$PipelineProcessor=$pscmdlet.CommandRuntime|select-property PipelineProcessor
$commands = $PipelineProcessor|select-property commands
$commandProcessor= $commands[0]
$ci = $commandProcessor|select-property commandinfo
$upce.RequestingCommandProcessor | im set_commandinfo #($ci)
$cr = $commandProcessor|select-property commandruntime
$upce.RequestingCommandProcessor| im set_commandruntime #($cr)
$null = $PipelineProcessor|
invoke-method recordfailure #($upce, $commandProcessor.command)
if ($commands.count -gt 1) {
$doCompletes = #()
1..($commands.count-1) | % {
write-debug "Stop-pipeline: added DoComplete for $($commands[$_])"
$doCompletes += $commands[$_] | invoke-method DoComplete -returnClosure
}
foreach ($DoComplete in $doCompletes) {
$null = & $DoComplete
}
}
throw $upce
}
EDIT: per mklement0's comment:
Here is a link to the Nivot ink blog on a script on the "poke" module which similarly gives access to non-public members.
As far as additional comments, I don't have meaningful ones at this point. This code just mimics what a decompilation of select-object reveals. The original MS comments (if any) are of course not in the decompilation. Frankly I don't know the purpose of the various types the function uses. Getting that level of understanding would likely require a considerable amount of effort.
My suggestion: get Oisin's poke module. Tweak the code to use that module. And then try it out. If you like the way it works, then use it and don't worry how it works (that's what I did).
Note: I haven't studied "poke" in any depth, but my guess is that it doesn't have anything like -returnClosure. However adding that should be easy as this:
if (-not $returnClosure) {
$methodInfo.Invoke($arguments)
} else {
{$methodInfo.Invoke($arguments)}.GetNewClosure()
}
Here's an - imperfect - implementation of a Stop-Pipeline cmdlet (requires PS v3+), gratefully adapted from this answer:
#requires -version 3
Filter Stop-Pipeline {
$sp = { Select-Object -First 1 }.GetSteppablePipeline($MyInvocation.CommandOrigin)
$sp.Begin($true)
$sp.Process(0)
}
# Example
1..5 | % { if ($_ -gt 2) { Stop-Pipeline }; $_ } # -> 1, 2
Caveat: I don't fully understand how it works, though fundamentally it takes advantage of Select -First's ability to stop the pipeline prematurely (PS v3+). However, in this case there is one crucial difference to how Select -First terminates the pipeline: downstream cmdlets (commands later in the pipeline) do not get a chance to run their end blocks.
Therefore, aggregating cmdlets (those that must receive all input before producing output, such as Sort-Object, Group-Object, and Measure-Object) will not produce output if placed later in the same pipeline; e.g.:
# !! NO output, because Sort-Object never finishes.
1..5 | % { if ($_ -gt 2) { Stop-Pipeline }; $_ } | Sort-Object
Background info that may lead to a better solution:
Thanks to PetSerAl, my answer here shows how to produce the same exception that Select-Object -First uses internally to stop upstream cmdlets.
However, there the exception is thrown from inside the cmdlet that is itself connected to the pipeline to stop, which is not the case here:
Stop-Pipeline, as used in the examples above, is not connected to the pipeline that should be stopped (only the enclosing ForEach-Object (%) block is), so the question is: How can the exception be thrown in the context of the target pipeline?
Try these filters, they'll force the pipeline to stop after the first object -or the first n elements- and store it -them- in a variable; you need to pass the name of the variable, if you don't the object(s) are pushed out but cannot be assigned to a variable.
filter FirstObject ([string]$vName = '') {
if ($vName) {sv $vName $_ -s 1} else {$_}
break
}
filter FirstElements ([int]$max = 2, [string]$vName = '') {
if ($max -le 0) {break} else {$_arr += ,$_}
if (!--$max) {
if ($vName) {sv $vName $_arr -s 1} else {$_arr}
break
}
}
# can't assign to a variable directly
$myLog = get-eventLog security | ... | firstObject
# pass the the varName
get-eventLog security | ... | firstObject myLog
$myLog
# can't assign to a variable directly
$myLogs = get-eventLog security | ... | firstElements 3
# pass the number of elements and the varName
get-eventLog security | ... | firstElements 3 myLogs
$myLogs
####################################
get-eventLog security | % {
if ($_.timegenerated -lt (date 11.09.08) -and`
$_.timegenerated -gt (date 11.01.08)) {$log1 = $_; break}
}
#
$log1
Another option would be to use the -file parameter on a switch statement. Using -file will read the file one line at a time, and you can use break to exit immediately without reading the rest of the file.
switch -file $someFile {
# Parse current line for later matches.
{ $script:line = [DateTime]$_ } { }
# If less than min date, keep looking.
{ $line -lt $minDate } { Write-Host "skipping: $line"; continue }
# If greater than max date, stop checking.
{ $line -gt $maxDate } { Write-Host "stopping: $line"; break }
# Otherwise, date is between min and max.
default { Write-Host "match: $line" }
}
What Powershell pitfalls you have fall into? :-)
Mine are:
# -----------------------------------
function foo()
{
#("text")
}
# Expected 1, actually 4.
(foo).length
# -----------------------------------
if(#($null, $null))
{
Write-Host "Expected to be here, and I am here."
}
if(#($null))
{
Write-Host "Expected to be here, BUT NEVER EVER."
}
# -----------------------------------
function foo($a)
{
# I thought this is right.
#if($a -eq $null)
#{
# throw "You can't pass $null as argument."
#}
# But actually it should be:
if($null -eq $a)
{
throw "You can't pass $null as argument."
}
}
foo #($null, $null)
# -----------------------------------
# There is try/catch, but no callstack reported.
function foo()
{
bar
}
function bar()
{
throw "test"
}
# Expected:
# At bar() line:XX
# At foo() line:XX
#
# Actually some like this:
# At bar() line:XX
foo
Would like to know yours to walk them around :-)
My personal favorite is
function foo() {
param ( $param1, $param2 = $(throw "Need a second parameter"))
...
}
foo (1,2)
For those unfamiliar with powershell that line throws because instead of passing 2 parameters it actually creates an array and passes one parameter. You have to call it as follows
foo 1 2
Another fun one. Not handling an expression by default writes it to the pipeline. Really annoying when you don't realize a particular function returns a value.
function example() {
param ( $p1 ) {
if ( $p1 ) {
42
}
"done"
}
PS> example $true
42
"done"
$files = Get-ChildItem . -inc *.extdoesntexist
foreach ($file in $files) {
"$($file.Fullname.substring(2))"
}
Fails with:
You cannot call a method on a null-valued expression.
At line:3 char:25
+ $file.Fullname.substring <<<< (2)
Fix it like so:
$files = #(Get-ChildItem . -inc *.extdoesntexist)
foreach ($file in $files) {
"$($file.Fullname.substring(2))"
}
Bottom line is that the foreach statement will loop on a scalar value even if that scalar value is $null. When Get-ChildItem in the first example returns nothing, $files gets assinged $null. If you are expecting an array of items to be returned by a command but there is a chance it will only return 1 item or zero items, put #() around the command. Then you will always get an array - be it of 0, 1 or N items. Note: If the item is already an array putting #() has no effect - it will still be the very same array (i.e. there is no extra array wrapper).
# The pipeline doesn't enumerate hashtables.
$ht = #{"foo" = 1; "bar" = 2}
$ht | measure
# Workaround: call GetEnumerator
$ht.GetEnumerator() | measure
Here are my top 5 PowerShell gotchas
Here is something Ive stumble upon lately (PowerShell 2.0 CTP):
$items = "item0", "item1", "item2"
$part = ($items | select-string "item0")
$items = ($items | where {$part -notcontains $_})
what do you think that $items be at the end of the script?
I was expecting "item1", "item2" but instead the value of $items is: "item0", "item1", "item2".
Say you've got the following XML file:
<Root>
<Child />
<Child />
</Root>
Run this:
PS > $myDoc = [xml](Get-Content $pathToMyDoc)
PS > #($myDoc.SelectNodes("/Root/Child")).Count
2
PS > #($myDoc.Root.Child).Count
2
Now edit the XML file so it has no Child nodes, just the Root node, and run those statements again:
PS > $myDoc = [xml](Get-Content $pathToMyDoc)
PS > #($myDoc.SelectNodes("/Root/Child")).Count
0
PS > #($myDoc.Root.Child).Count
1
That 1 is annoying when you want to iterate over a collection of nodes using foreach if and only if there actually are any. This is how I learned that you cannot use the XML handler's property (dot) notation as a simple shortcut. I believe what's happening is that SelectNodes returns a collection of 0. When #'ed, it is transformed from an XPathNodeList to an Object[] (check GetType()), but the length is preserved. The dynamically generated $myDoc.Root.Child property (which essentially does not exist) returns $null. When $null is #'ed, it becomes an array of length 1.
On Functions...
The subtleties of processing pipeline input in a function with respect to using $_ or $input and with respect to the begin, process, and end blocks.
How to handle the six principal equivalence classes of input delivered to a function (no input, null, empty string, scalar, list, list with null and/or empty) -- for both direct input and pipeline input -- and get what you expect.
The correct calling syntax for sending multiple arguments to a function.
I discuss these points and more at length in my Simple-Talk.com article Down the Rabbit Hole- A Study in PowerShell Pipelines, Functions, and Parameters and also provide an accompanying wallchart--here is a glimpse showing the various calling syntax pitfalls for a function taking 3 arguments:
On Modules...
These points are expounded upon in my Simple-Talk.com article Further Down the Rabbit Hole: PowerShell Modules and Encapsulation.
Dot-sourcing a file inside a script using a relative path is relative to your current directory -- not the directory where the script resides!
To be relative to the script use this function to locate your script directory: [Update for PowerShell V3+: Just use the builtin $PSScriptRoot variable!]
function Get-ScriptDirectory
{ Split-Path $script:MyInvocation.MyCommand.Path }
Modules must be stored as ...Modules\name\name.psm1 or ...\Modules\any_subpath\name\name.psm1. That is, you cannot just use ...Modules\name.psm1 -- the name of the immediate parent of the module must match the base name of the module. This chart shows the various failure modes when this rule is violated:
2015.06.25 A Pitfall Reference Chart
Simple-Talk.com just published the last of my triumvirate of in-depth articles on PowerShell pitfalls. The first two parts are in the form of a quiz that helps you appreciate a select group of pitfalls; the last part is a wallchart (albeit it would need a rather high-ceilinged room) containing 36 of the most common pitfalls (some adapted from answers on this page), giving concrete examples and workarounds for most. Read more here.
There are some tricks to building command lines for utilities that were not built with Powershell in mind:
To run an executable who's name starts with a number, preface it with an Ampersand (&).
& 7zip.exe
To run an executable with a space anywhere in the path, preface it with an Ampersand (&) and wrap it in quotes, as you would any string. This means that strings in a variable can be executed as well.
# Executing a string with a space.
& 'c:\path with spaces\command with spaces.exe'
# Executing a string with a space, after first saving it in a variable.
$a = 'c:\path with spaces\command with spaces.exe'
& $a
Parameters and arguments are passed to legacy utilities positionally. So it is important to quote them the way the utility expects to see them. In general, one would quote when it contains spaces or does not start with a letter, number or dash (-).
C:\Path\utility.exe '/parameter1' 'Value #1' 1234567890
Variables can be used to pass string values containing spaces or special characters.
$b = 'string with spaces and special characters (-/&)'
utility.exe $b
Alternatively array expansion can be used to pass values as well.
$c = #('Value #1', $Value2)
utility.exe $c
If you want Powershell to wait for an application to complete, you have to consume the output, either by piping the output to something or using Start-Process.
# Saving output as a string to a variable.
$output = ping.exe example.com | Out-String
# Piping the output.
ping stackoverflow.com | where { $_ -match '^reply' }
# Using Start-Process affords the most control.
Start-Process -Wait SomeExecutable.com
Because of the way they display their output, some command line utilities will appear to hang when ran inside of Powershell_ISE.exe, particularly when awaiting input from the user. These utilities will usually work fine when ran within Powershell.exe console.
PowerShell Gotchas
There are a few pitfall that repeatedly reappear on StackOverflow. It is recommend to do some research if you are not familiar with these PowerShell gotchas before asking a new question. It might even be a good idea to investigate in these PowerShell gotchas before answering a PowerShell question to make sure that you teach the questioner the right thing.
TLDR: In PowerShell:
the comparison equality operator is: -eq
(Stackoverflow example: Powershell simple syntax if condition not working)
parentheses and commas are not used with arguments
(Stackoverflow example: How do I pass multiple parameters into a function in PowerShell?)
output properties are based on the first object in the pipeline
(Stackoverflow example: Not all properties displayed)
the pipeline unrolls
(Stackoverflow example: Pipe complete array-objects instead of array items one at a time?)
a. single item collections
(Stackoverflow example: Powershell ArrayList turns a single array item back into a string)
b. embedded arrays
(Stackoverflow example: Return Multidimensional Array From Function)
c. output collections
(Stackoverflow example: Why does PowerShell flatten arrays automatically?)
$Null should be on the left side of the equality comparison operator
(Stackoverflow example: Should $null be on the left side of the equality comparison)
parentheses and assignments choke the pipeline
(Stackoverflow example: Importing 16MB CSV Into Variable Creates >600MB's Memory Usage)
the increase assignment operator (+=) might become expensive
Stackoverflow example: Improve the efficiency of my PowerShell scrip
The Get-Content cmdlet returns separate lines
Stackoverflow example: Multiline regex to match config block
Examples and explanations
Some of the gotchas might really feel counter-intuitive but often can be explained by some very nice PowerShell features along with the pipeline, expression/argument mode and type casting.
1. The comparison equality operator is: -eq
Unlike the Microsoft scripting language VBScript and some other programming languages, the comparison equality operator differs from the assignment operator (=) and is: -eq.
Note: assigning a value to a variable might pass through the value if needed:
$a = $b = 3 # The value 3 is assigned to both variables $a and $b.
This implies that following statement might be unexpectedly truthy or falsy:
If ($a = $b) {
# (assigns $b to $a and) returns a truthy if $b is e.g. 3
} else {
# (assigns $b to $a and) returns a falsy if $b is e.g. 0
}
2. Parentheses and commas are not used with arguments
Unlike a lot of other programming languages and the way a primitive PowerShell function is defined, calling a function doesn't require parentheses or commas for their related arguments. Use spaces to separate the parameter arguments:
MyFunction($Param1, $Param2 $Param3) {
# ...
}
MyFunction 'one' 'two' 'three' # assigns 'one' to $Param1, 'two' to $Param2, 'three' to $Param3
Parentheses and commas are used for calling (.Net) methods.
Commas are used to define arrays. MyFunction 'one', 'two', 'three' (or MyFunction('one', 'two', 'three')) will load the array #('one', 'two', 'three') into the first parameter ($Param1).
Parentheses will intepret the containing contents as a single collection into memory (and choke the PowerShell pipeline) and should only be used as such, e.g. to call an embedded function, e.g.:
MyFunction (MyOtherFunction) # passes the results MyOtherFunction to the first positional parameter of MyFunction ($Param1)
MyFunction One $Two (getThree) # assigns 'One' to $Param1, $Two to $Param2, the results of getThree to $Param3
Note: that quoting text arguments (as the word one in the later example) is only required when it contains spaces or special characters.
3. Output properties are based on the first object in the pipeline
In a PowerShell pipeline each object is processed and passed on by a cmdlet (that is implemented for the middle of a pipeline) similar to how objects are processed and passed on by workstations in an assembly line. Meaning each cmdlet processes one item at the time while the prior cmdlet (workstation) simultaneously processes the upcoming one. This way, the objects aren't loaded into memory at once (less memory usage) and could already be processed before the next one is supplied (or even exists). The disadvantage of this feature is that there is no oversight of what (or how many) objects are expected to follow.
Therefore most PowerShell cmdlets assume that all the objects in the pipeline correspond to the first one and have the same properties which is usually the case, but not always...
$List =
[pscustomobject]#{ one = 'a1'; two = 'a2' },
[pscustomobject]#{ one = 'b1'; two = 'b2'; three = 'b3' }
$List |Select-Object *
one two
--- ---
a1 a2
b1 b2
As you see, the third column three is missing from the results as it didn't exists in the first object and the PowerShell was already outputting the results prior it was aware of the exists of the second object.
On way to workaround this behavior is to explicitly define the properties (of all the following objects) at forehand:
$List |Select-Object one, two, three
one two three
--- --- -----
a1 a2
b1 b2 b3
See also proposal: #13906 Add -UnifyProperties parameter to Select-Object
4. The pipeline unrolls
This feature might come in handy if it complies with the straightforward expectation:
$Array = 'one', 'two', 'three'
$Array.Length
3
a. single item collections
But it might get confusing:
$Selection = $Array |Select-Object -First 2
$Selection.Length
2
$Selection[0]
one
when the collection is down to a single item:
$Selection = $Array |Select-Object -First 1
$Selection.Length
3
$Selection[0]
o
Explanation
When the pipeline outputs a single item which is assigned to a variable, it is not assigned as a collection (with 1 item, like: #('one')) but as a scalar item (the item itself, like: 'one').
Which means that the property .Length (which is in fact an alias for the property .Count for an array) is no longer applied on the array but on the string: 'one'.length which equals 3. And in case of the index $Selection[0] , the first character of the string 'one'[0] (which equals the character o) is returned .
Workaround
To workaround this behavior, you might force the scalar item to an array using the Array subexpression operator #( ):
$Selection = $Array |Select-Object -First 1
#($Selection).Length
1
#($Selection)[0]
one
Knowing that in the case the $Selection is already an array, it will will not be further increased in depth (#(#('one', 'two')), see the next section 4b. Embedded collections are flattened).
b. embedded arrays
When an array (or a collection) includes embedded arrays, like:
$Array = #(#('a', 'b'), #('c', 'd'))
$Array.Count
2
All the embedded items will be processed in the pipeline and consequently returns a flat array when displayed or assigned to a new variable:
$Processed = $Array |ForEach-Object { $_ }
$Processed.Count
4
$Processed
a
b
c
d
To iterate the embedded arrays, you might use the foreach statement:
foreach ($Item in $Array) { $Item.Count }
2
2
Or a simply for loop:
for ($i = 0; $i -lt $Array.Count; $i++) { $Array[$i].Count }
2
2
c. output collections
Collections are usually unrolled when they are placed on the pipeline:
function GetList {
[Collections.Generic.List[String]]#('a', 'b')
}
(GetList).GetType().Name
Object[]
To output the collection as a single item, use the comma operator ,:
function GetList {
,[Collections.Generic.List[String]]#('a', 'b')
}
(GetList).GetType().Name
List`1
5. $Null should be on the left side of the equality comparison operator
This gotcha is related to this comparison operators feature:
When the input of an operator is a scalar value, the operator returns a Boolean value. When the input is a collection, the operator returns the elements of the collection that match the right-hand value of the expression. If there are no matches in the collection, comparison operators return an empty array.
This means for scalars:
'a' -eq 'a' # returns $True
'a' -eq 'b' # returns $False
'a' -eq $Null # returns $False
$Null -eq $Null # returns $True
and for collections, the matching elements are returned which evaluates to either a truthy or falsy condition:
'a', 'b', 'c' -eq 'a' # returns 'a' (truthy)
'a', 'b', 'c' -eq 'd' # returns an empty array (falsy)
'a', 'b', 'c' -eq $Null # returns an empty array (falsy)
'a', $Null, 'c' -eq $Null # returns $Null (falsy)
'a', $Null, $Null -eq $Null # returns #($Null, $Null) (truthy!!!)
$Null, $Null, $Null -eq $Null # returns #($Null, $Null, $Null) (truthy!!!)
In other words, to check whether a variable is $Null (and exclude a collection containing multiple $Nulls), put $Null at the LHS (left hand side) of the equality comparison operator:
if ($Null -eq $MyVariable) { ...
6. Parentheses and assignments choke the pipeline
The PowerShell Pipeline is not just a series of commands connected by pipeline operators (|) (ASCII 124). It is a concept to simultaneously stream individual objects through a sequence of cmdlets. If a cmdlet (or function) is written according to the Strongly Encouraged Development Guidelines and implemented for the middle of a pipeline, it takes each single object from the pipeline, processes it and passes the results to the next cmdlet just before it takes and processes the next object in the pipeline. Meaning that for a simple pipeline as:
Import-Csv .\Input.csv |Select-Object -Property Column1, Column2 |Export-Csv .\Output.csv
As the last cmdlet writes an object to the .\Output.csv file, the Select-Object cmdlet selects the properties of the next object and the Import-Csv reads the next object from the .\input.csv file (see also: Pipeline in Powershell). This will keep the memory usage low (especially where there are lots of object/records to process) and therefore might result in a faster throughput. To facilitate the pipeline, the PowerShell objects are quiet fat as each individual object contains all the property information (along with e.g. the property name).
Therefore it is not a good practice to choke the pipeline for no reason. There are two senarios that choke the pipeline:
Parentheses, e.g.:
(Import-Csv .\Input.csv) |Select-Object -Property Column1, Column2 |Export-Csv .\Output.csv
Where all the .\Input.csv records are loaded as an array of PowerShell objects into memory before passing it on to the Select-Object cmdlet.
Assignments, e.g.:
$Objects = Import-Csv .\Input.csv
$Objects |Select-Object -Property Column1, Column2 |Export-Csv .\Output.csv
Where all the .\Input.csv records are loaded as an array of PowerShell objects into $Objects (memory as well) before passing it on to the Select-Object cmdlet.
7. the increase assignment operator (+=) might become expensive
The increase assignment operator (+=) is syntactic sugar to increase and assign primitives as .e.g. $a += $b where $a is assigned $b + 1. The increase assignment operator can also be used for adding new items to a collection (or to String types and hash tables) but might get pretty expensive as the costs increases with each iteration (the size of the collection). The reason for this is that objects as array collections are immutable and the right variable in not just appended but *appended and reassigned to the left variable. For details see also: avoid using the increase assignment operator (+=) to create a collection
8. The Get-Content cmdlet returns separate lines
There are probably quite some more cmdlet gotchas, knowing that there exist a lot of (internal and external) cmdlets. In contrast to engine related gotchas, these gotchas are often easier to highlight (with e.g. a warning) as happend with ConvertTo-Json (see: Unexpected ConvertTo-Json results? Answer: it has a default -Depth of 2) or "fix". But there is very clasic gotcha in Get-Content which tight into the PowerShell general concept of streaming objects (in this case lines) rather than passing everything (the whole contents of the file) in once:
Get-Content .\Input.txt -Match '\r?\n.*Test.*\r?\n'
Will never work because, by default, Get-Contents returns a stream of objects where each object contains a single string (a line without any line breaks).
(Get-Content .\Input.txt).GetType().Name
Object[]
(Get-Content .\Input.txt)[0].GetType().Name
String
In fact:
Get-Content .\Input.txt -Match 'Test'
Returns all the lines with the word Test in it as Get-Contents puts every single line on the pipeline and when the input is a collection, the operator returns the elements of the collection that match the right-hand value of the expression.
Note: since PowerShell version 3, Get-Contents has a -Raw parameter that reads all the content of the concerned file at once, Meaning that this: Get-Content -Raw .\Input.txt -Match '\r?\n.*Test.*\r?\n' will work as it loads the whole file into memory.
alex2k8, I think this example of yours is good to talk about:
# -----------------------------------
function foo($a){
# I thought this is right.
#if($a -eq $null)
#{
# throw "You can't pass $null as argument."
#}
# But actually it should be:
if($null -eq $a)
{
throw "You can't pass $null as argument."
}
}
foo #($null, $null)
PowerShell can use some of the comparators against arrays like this:
$array -eq $value
## Returns all values in $array that equal $value
With that in mind, the original example returns two items (the two $null values in the array), which evalutates to $true because you end up with a collection of more than one item. Reversing the order of the arguments stops the array comparison.
This functionality is very handy in certain situations, but it is something you need to be aware of (just like array handling in PowerShell).
Functions 'foo' and 'bar' looks equivalent.
function foo() { $null }
function bar() { }
E.g.
(foo) -eq $null
# True
(bar) -eq $null
# True
But:
foo | %{ "foo" }
# Prints: foo
bar | %{ "bar" }
# PRINTS NOTHING
Returning $null and returning nothing is not equivalent dealing with pipes.
This one is inspired by Keith Hill example...
function bar() {}
$list = #(foo)
$list.length
# Prints: 0
# Now let's try the same but with a temporal variable.
$tmp = foo
$list = #($tmp)
$list.length
# Prints: 1
Another one:
$x = 2
$y = 3
$a,$b = $x,$y*5
because of operators precedence there is not 25 in $b; the command is the same as ($x,$y)*5
the correct version is
$a,$b = $x,($y*5)
The logical and bitwise operators don't follow standard precedence rules. The operator -and should have a higher priority than -or yet they're evaluated strictly left-to-right.
For example, compare logical operators between PowerShell and Python (or virtually any other modern language):
# PowerShell
PS> $true -or $false -and $false
False
# Python
>>> True or False and False
True
...and bitwise operators:
# PowerShell
PS> 1 -bor 0 -band 0
0
# Python
>>> 1 | 0 & 0
1
This works. But almost certainly not in the way you think it's working.
PS> $a = 42;
PS> [scriptblock]$b = { $a }
PS> & $b
42
This one has tripped me up before, using $o.SomeProperty where it should be $($o.SomeProperty).
# $x is not defined
[70]: $x -lt 0
True
[71]: [int]$x -eq 0
True
So, what's $x..?
Another one I ran into recently: [string] parameters that accept pipeline input are not strongly typed in practice. You can pipe anything at all and PS will coerce it via ToString().
function Foo
{
[CmdletBinding()]
param (
[parameter(Mandatory=$True, ValueFromPipeline=$True)]
[string] $param
)
process { $param }
}
get-process svchost | Foo
Unfortunately there is no way to turn this off. Best workaround I could think of:
function Bar
{
[CmdletBinding()]
param (
[parameter(Mandatory=$True, ValueFromPipeline=$True)]
[object] $param
)
process
{
if ($param -isnot [string]) {
throw "Pass a string you fool!"
}
# rest of function goes here
}
}
edit - a better workaround I've started using...
Add this to your custom type XML -
<?xml version="1.0" encoding="utf-8" ?>
<Types>
<Type>
<Name>System.String</Name>
<Members>
<ScriptProperty>
<Name>StringValue</Name>
<GetScriptBlock>
$this
</GetScriptBlock>
</ScriptProperty>
</Members>
</Type>
</Types>
Then write functions like this:
function Bar
{
[CmdletBinding()]
param (
[parameter(Mandatory=$True, ValueFromPipelineByPropertyName=$True)]
[Alias("StringValue")]
[string] $param
)
process
{
# rest of function goes here
}
}
Forgetting that $_ gets overwritten in blocks made me scratch my head in confusion a couple times, and similarly for multiple reg-ex matches and the $matches array. >.<
Remembering to explicitly type pscustom objects from imported data tables as numeric so they can be sorted correctly:
$CVAP_WA=foreach ($i in $C){[PSCustomObject]#{ `
County=$i.county; `
TotalVote=[INT]$i.TotalBallots; `
RegVoters=[INT]$i.regvoters; `
Turnout_PCT=($i.TotalBallots/$i.regvoters)*100; `
CVAP=[INT]($B | ? {$_.GeoName -match $i.county}).CVAP_EST }}
PS C:\Politics> $CVAP_WA | sort -desc TotalVote |ft -auto -wrap
County TotalVote RegVoters Turnout_PCT CVAP CVAP_TV_PCT CVAP_RV_PCT
------ --------- --------- ----------- ---- ----------- -----------
King 973088 1170638 83.189 1299290 74.893 90.099
Pierce 349377 442985 78.86 554975 62.959 79.837
Snohomish 334354 415504 80.461 478440 69.832 86.81
Spokane 227007 282442 80.346 342060 66.398 82.555
Clark 193102 243155 79.453 284190 67.911 85.52
Mine are both related to file copying...
Square Brackets in File Names
I once had to move a very large/complicated folder structure using Move-Item -Path C:\Source -Destination C:\Dest. At the end of the process there were still a number of files in source directory. I noticed that every remaining file had square brackets in the name.
The problem was that the -Path parameter treats square brackets as wildcards.
EG. If you wanted to copy Log001 to Log200, you could use square brackets as follows:
Move-Item -Path C:\Source\Log[001-200].log.
In my case, to avoid square brackets being interpreted as wildcards, I should have used the -LiteralPath parameter.
ErrorActionPreference
The $ErrorActionPreference variable is ignored when using Move-Item and Copy-Item with the -Verbose parameter.
Treating the ExitCode of a Process as a Boolean.
eg, with this code:
$p = Start-Process foo.exe -NoNewWindow -Wait -PassThru
if ($p.ExitCode) {
# handle error
}
things are good, unless say foo.exe doesn't exist or otherwise fails to launch.
in that case $p will be $null, and [bool]($null.ExitCode) is False.
a simple fix is to replace the logic with if ($p.ExitCode -ne 0) {},
however for clarity of code imo the following is better: if (($p -eq $null) -or ($p.ExitCode -ne 0)) {}