Where-Object, Select-Object and ForEach-object - Differences and Usage - powershell

Where-Object, Select-Object and ForEach-Object
I am a PowerShell beginner. I don't understand too much. Can someone give examples to illustrate the differences and usage scenarios between them?

If you are at all familiar with either LINQ or SQL then it should be much easier to understand because it uses the same concepts for the same words with a slight tweak.
Where-Object
is used for filtering out objects from the pipeline and is similar to how SQL filters rows. Here, objects are compared against a condition, or optionally a ScriptBlock, to determine whether it should be passed on to the next cmdlet in the pipeline. To demonstrate:
# Approved Verbs
Get-Verb | Measure-Object # count of 98
Get-Verb | Where-Object Verb -Like G* | Measure-Object # 3
# Integers 1 to 100
1..100 | Measure-Object # count of 100
1..100 | Where-Object {$_ -LT 50} | Measure-Object # count of 49
This syntax is usually the most readable when not using a ScriptBlock, but is necessary if you want to refer to the object itself (not a property) or for more complicated boolean results. Note: many resources will recommend (as #Iftimie Tudor mentions) trying to filter sooner (more left) in the pipeline for performance benefits.
Select-Object
is used for filtering properties of an object and is similar to how SQL filters columns. Importantly, it transforms the pipeline object into a new PSCustomObject that only has the requested properties with the object's values copied. To demonstrate:
Get-Process
Get-Process | Select-Object Name,CPU
Note, though, that this is only the standard usage. Explore its parameter sets using Get-Help Select-Object where it has similar row-like filtering capabilities like only getting the first n objects from the pipeline (aka, Get-Process | Select-Object -First 3) that continue onto the next cmdlet.
ForEach-Object
is like your foreach loops in other languages, with its own important flavour. In fact, PowerShell also has a foreach loop of its own! These may be easily confused but are operationally quite different. The main visual difference is that the foreach loop cannot be used in a pipeline, but ForEach-Object can. The latter, ForEach-Object, is a cmdlet (foreach is not) and can be used for transforming the current pipeline or for running a segment of code against the pipeline. It is really the most flexible cmdlet there is.
The best way to think about it is that it is the body of a loop, where the current element, $_, is coming from the pipeline and any output is passed onto the next cmdlet. To demonstrate:
# Transform
Get-Verb | ForEach-Object {"$($_.Verb) comes from the group $($_.Group)"}
# Retrieve Property
Get-Verb | ForEach-Object Verb
# Call Method
Get-Verb | ForEach-Object GetType
# Run Code
1..100 | ForEach-Object {
$increment = $_ + 1
$multiplied = $increment * 3
Write-Output $multiplied
}
Edit (Feb, 2023): thanks to #IkemKrueger for a missing }.

You have two things in there: filtering and iterating through a collection.
Filtering:
principle: Always use filtering left as much as possible. These two commands do the same thing, but the second one won't transmit a huge chunk of data through the pipe (or network):
Get-Process | where-Object {$_.Name -like 'chrome'} | Export-Csv
'c:\temp\processes.csv'
Get-Process -Name chrome | Export-Csv c:\temp\processes.csv
This is great when working with huge lists of computers or big files.
Many commandlets have their own filtering capabilities. Run get Get-Help get-process -full to see what they offer before piping.
iterating through collections:
Here you have 3 possibilities:
batch cmdlets is commandlet built in capability of passing a collection to another commandlet:
Get-Service -name BITS,Spooler,W32Time | Set-Service -startuptype
Automatic
WMI methods - WMI uses it's own way of doing the first one (different syntax)
gwmi win32_networkadapterconfiguration -filter "description like
'%intel%'" | EnableDHCP()
enumerating objects - iterating through the list:
Get-WmiObject Win32_Service -filter "name = 'BITS'" | ForEach-Object
-process { $_.change($null,$null,$null,$null,$null,$null,$null,"P#ssw0rd") }
Credits:
I found explanations that cleared the mess in my head around all these things in a book called : Learn Powershell in a month of lunches (chapters 9 and 13 in this case)

Related

Which way is better in PowerShell and why

I am novice in powershell and using it very rarely for some little things.
I am using this one liner in order to extract emails
recursive
(Get-ChildItem -Include *.txt -Recurse | Get-Content | Select-String -Pattern "(?:[a-zA-Z0-9_\-\.]+)#(?:[a-zA-Z0-9_\-\.]+)\.(?:[a-zA-Z]{2,5})").Matches | Select-Object -ExpandProperty Value -Unique
In order to access Matches property I've added parentheses. Later I come to that way:
Get-ChildItem -Include *.txt -Recurse | Get-Content | Select-String -Pattern "(?:[a-zA-Z0-9_\-\.]+)#(?:[a-zA-Z0-9_\-\.]+)\.(?:[a-zA-Z]{2,5})" | Select-Object -ExpandProperty Matches -Unique | Select-Object -ExpandProperty Value
I want to to ask what parentheses do exactly in the first version.
Say you have an $output via some function (gci in your case) and you are interested in the field $output.Matches.
If you run $output | select Matches (example 1), you run a
Foreach-Object statement against every object in your array. This
pipeline will use some RAM (very limited, indeed) that are used in a
serial calculation, so every object of $output is processed one after
the other.
If you run $output.Matches (example 2), you select a field from an
array. This will use a lot of RAM at once, but the field will be
processed as one big object instead of many little objects.
As it comes to performance. As always, note that PowerShell is not the way to go if you need high performance. It was never designed to be a fast programming language.
When you're using small objects (like gci $env:userprofile\Desktop), the performance hit will be small. When using large objects or using a lot of nested pipes, the performance hit will be large.
I've just tested it with a gci Z:\ -recurse when Z:\ is a network drive. Performance is dropped with a factor of 20 in this specific case. (Use Measure-Command to test this.)

How does pipeline and $_ work in PowerShell?

Get-Process|Get-Member $_, why doesn't this work? If $_ represents current object in pipeline, shouldn't the above output be returning members of each process object?
Will a cmdlet pipeline all its output objects at once to the next cmdlet or as and when there is an object available?
As #1 does not work, when exactly can I use $_ variable? The conditions under which this variable gets created will be more helpful rather than just a cmdlet example to demonstrate the use of $_.
$_ is an automatic variable that is available for use within the scriptblock input of certain cmdlets to represent the current item in the pipeline.
Get-Process | Get-Member $_ doesn't work because you send the pipeline object in to Get-Member via the |, but you then don't have any way to access the internals of Get-Member.
You could do this:
Get-Process | ForEach-Object {
$_ | Get-Member
}
You would then get a Get-Member output for every item in the collection of objects output by Get-Process, although this would be redundant as each would be the same.
Cmdlets do send the objects down the pipeline one at a time. You can see that with this example:
Get-Process | ForEach-Object {
$_
Start-Sleep 1
}
You can see with the added delay that the results are arriving in the ForEach-Object one at a time as soon as they are available.
Other places you can use the $_ variable are in Where-Object and Select-Object. For example:
Get-Process | Where-Object { $_.name -like 'win*' }
Here the Where-Object cmdlet is taking each item of the pipeline and we're using $_ to access the name property of that item to see if it's like the string win. If it is, then it gets sent onwards (and so comes out to the console) if it's not, Where-Object discards it.
You can use $_ in a Select-Object when doing calculated properties. For example:
Get-Process | Select-Object name,#{N='WorkingSetGB';E={$_.WorkingSet / 1GB}}
Here we use $_ to get at the WorkingSet property of each item and then convert it to a GB value by using / 1GB.
Yes, $_ represents the current object in the pipeline but since Get-Member takes a pipeline input you just have to pipe the result to the cmdlet:
Get-Process | Get-Member
Another example is
Get-Process | Export-Csv MyFile.csv
Here again, $_ is not needed, because Export-Csv takes pipeline input, and receives the output of Get-Process, one process at a time, through the pipeline. There is a loop inside the implementation of Export-csv, but that need not concern you here.
You typically use $_ when you pipe an object to the ForEach-Object cmdlet:
Get-Process | ForEach-Object {
Write-Host $_.Name
}

Powershell, Using the results of a Invoke-SQLcmd as a variable in a file path [duplicate]

I have a script that creates several jobs and stores two simple values in the jobs.
Start-Job -ScriptBlock {param ([string]$compip) tnc $compip | select RemoteAddress,PingSucceeded -WarningAction SilentlyContinue} -ArgumentList $compip
This works fine. What I would like to know is how can I store the following code into a variable?
Get-Job | Receive-Job | sort RemoteAddress | FT
I have tried this, but it does not work as I thought it would:
$pcs = Get-Job | Receive-Job | sort RemoteAddress | FT
$pcs.RemoteAddress
Am I going at this the wrong way? I would like to store the data from the get-job command above to use the values later in the script. I assumed it would work because the output looks correct:
Command:
Get-Job | Receive-Job | sort RemoteAddress | FT
Output:
RemoteAddress PingSucceeded
------------- -------------
192.168.0.163 True
192.168.0.101 False
192.168.0.2 False
192.168.0.251 True
Problem with Format-cmdlets
The issue here is your use of FT which is an alias for Format-Table. These Format- cmdlets are designed for console/screen output only. There are many things you can do with them to tailor that output but in every case PowerShell needs to massage the data in order to be able to do so. This includes breaking down to the passed objects into groups of different objects...
Microsoft.PowerShell.Commands.Internal.Format.FormatEndData
Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData
Microsoft.PowerShell.Commands.Internal.Format.FormatStartData
Microsoft.PowerShell.Commands.Internal.Format.GroupEndData
Microsoft.PowerShell.Commands.Internal.Format.GroupStartData
The above data types were extract from running this code.
Get-ChildItem c:\temp | Format-Table | Get-Member
So you no longer have the System.IO.FileInfo and System.IO.DirectoryInfo objects that you would normally get from Get-ChildItem
Another large issue comes from Format-cmdlets nature to truncated data, like arrays with a large numbers of elements or long strings, to make as much fit on screen. In the case of arrays, this is due to the preference variable $FormatEnumerationLimit which is commonly defaulted to 4.
Ten Numbers
-----------
{1, 2, 3, 4...}
These, and other, limitations can all be mitigated with cmdlet switches like -AutoSize and -HideTableHeaders, out-string -width, etc. That does not matter however because...
Solution
Good news is the solution is very simple. Stop using them for anything other than console output. Using my earlier example:
Saving results in a variable?: $result = Get-ChildItem C:\temp
Exporting Data: Get-ChildItem C:\temp | Export-CSV $path -NoTypeInformation. Other Export-cmdlets could be preferred here like Export-CLIXml for complex objects when you want to store them for use elsewhere. If you are just looking for something pretty to include in your output then consider ConvertTo-HTML instead.
Extracting individual properties?: Just use Select-Object. $result | Select prop1, prop2. You can also expand your property selection to just get the strings or string array with -ExpandProperty: $result | Select -ExpandProperty prop1
Performing inline calculations with said properties?: Use calculated expression just as you would with the Format-Cmdlets. $result | Select prop1, #{Name="prop2";Expression={$_.prop2 * 3}
Potential Acceptable Use
Some prefer the output for use in emails and for recording statistics. While it is integral to keep data in its more easily used format for later use. However if you really need that data keep in mind that you are not working with the object your originally had anymore.
So if you needed your data in a table format but stored as a string then consider Out-String
$body = Get-ChildItem c:\temp | Format-Table | Out-String
but remember that Format-Table will play with object output in order to get it to display on screen (truncated array properties and long strings). Really.. if you wanted it nice and formatted then you should just use ConvertTo-HTML.
Point is you almost never need to keep the data from Format-Table. There is almost always a better way.

How can I store output from Format-Table for later use

I have a script that creates several jobs and stores two simple values in the jobs.
Start-Job -ScriptBlock {param ([string]$compip) tnc $compip | select RemoteAddress,PingSucceeded -WarningAction SilentlyContinue} -ArgumentList $compip
This works fine. What I would like to know is how can I store the following code into a variable?
Get-Job | Receive-Job | sort RemoteAddress | FT
I have tried this, but it does not work as I thought it would:
$pcs = Get-Job | Receive-Job | sort RemoteAddress | FT
$pcs.RemoteAddress
Am I going at this the wrong way? I would like to store the data from the get-job command above to use the values later in the script. I assumed it would work because the output looks correct:
Command:
Get-Job | Receive-Job | sort RemoteAddress | FT
Output:
RemoteAddress PingSucceeded
------------- -------------
192.168.0.163 True
192.168.0.101 False
192.168.0.2 False
192.168.0.251 True
Problem with Format-cmdlets
The issue here is your use of FT which is an alias for Format-Table. These Format- cmdlets are designed for console/screen output only. There are many things you can do with them to tailor that output but in every case PowerShell needs to massage the data in order to be able to do so. This includes breaking down to the passed objects into groups of different objects...
Microsoft.PowerShell.Commands.Internal.Format.FormatEndData
Microsoft.PowerShell.Commands.Internal.Format.FormatEntryData
Microsoft.PowerShell.Commands.Internal.Format.FormatStartData
Microsoft.PowerShell.Commands.Internal.Format.GroupEndData
Microsoft.PowerShell.Commands.Internal.Format.GroupStartData
The above data types were extract from running this code.
Get-ChildItem c:\temp | Format-Table | Get-Member
So you no longer have the System.IO.FileInfo and System.IO.DirectoryInfo objects that you would normally get from Get-ChildItem
Another large issue comes from Format-cmdlets nature to truncated data, like arrays with a large numbers of elements or long strings, to make as much fit on screen. In the case of arrays, this is due to the preference variable $FormatEnumerationLimit which is commonly defaulted to 4.
Ten Numbers
-----------
{1, 2, 3, 4...}
These, and other, limitations can all be mitigated with cmdlet switches like -AutoSize and -HideTableHeaders, out-string -width, etc. That does not matter however because...
Solution
Good news is the solution is very simple. Stop using them for anything other than console output. Using my earlier example:
Saving results in a variable?: $result = Get-ChildItem C:\temp
Exporting Data: Get-ChildItem C:\temp | Export-CSV $path -NoTypeInformation. Other Export-cmdlets could be preferred here like Export-CLIXml for complex objects when you want to store them for use elsewhere. If you are just looking for something pretty to include in your output then consider ConvertTo-HTML instead.
Extracting individual properties?: Just use Select-Object. $result | Select prop1, prop2. You can also expand your property selection to just get the strings or string array with -ExpandProperty: $result | Select -ExpandProperty prop1
Performing inline calculations with said properties?: Use calculated expression just as you would with the Format-Cmdlets. $result | Select prop1, #{Name="prop2";Expression={$_.prop2 * 3}
Potential Acceptable Use
Some prefer the output for use in emails and for recording statistics. While it is integral to keep data in its more easily used format for later use. However if you really need that data keep in mind that you are not working with the object your originally had anymore.
So if you needed your data in a table format but stored as a string then consider Out-String
$body = Get-ChildItem c:\temp | Format-Table | Out-String
but remember that Format-Table will play with object output in order to get it to display on screen (truncated array properties and long strings). Really.. if you wanted it nice and formatted then you should just use ConvertTo-HTML.
Point is you almost never need to keep the data from Format-Table. There is almost always a better way.

Filter the output of a command as if it was text

I have a simple question, but I am also a beginner in PowerShell. I think it has to do with the fact that the output of the Get-Process command (alias ps) is objects and not text.
I want to get a list of the services running that have the name "sql" in them.
This is what I tried so far, but every attempt returns nothing:
Get-Service | where {$_ -match 'sql'}
Get-Service | where {$_ -like 'sql'}
Get-Service | Select-String sql
I am looking for a pattern that lets me treat the output of every command as searchable text.
Just forget it :o)
Outputs are objects. You are right, and you are going to use this.
So mjolinor has the shortest answer, but for your knowledge just test:
Get-Service | Get-Member
So you will understand that
Get-Service | Where-Object {$_.name -match ".*sql.*" }
also works, and there you've got your text as a property of the object.
Most answers here focus on finding the service name with "sql" in the name, not on filtering the entire output as if it was text. Also, the accepted answer uses a non-PowerShell function, "findstr".
So, granted, what follows is not the most elegant solution, but for sake of completeness I would like to provide the 100% PowerShell solution that takes the question of the OP literally:
(get-Service | Out-String) -split "`r`n" | Select-String sql
We need Out-String, because using the solutions provided in other answers doesn't provide us the full text output of the Get-Service command, only the Name parameter.
We need to split on newlines, because Select-String seems to treat the entire text as one long string, and returns it as a whole, if "sql" is found in it.
I use Select-String instead of findstr, because findstr is not a PowerShell function.
This is a purist answer, and in practice, for this specific use-case, I would not recommend it. But for people coming here through Google Search based on the question title, this is a more accurate answer...
Get-Service | Select-String -Pattern "sql"
This works just like grep. And you can even sort:
Get-Service | Select-String -Pattern "sql" | sort
The other answers are right of course about your specific question of starting services that have "sql" in their name, but to answer the generic question:
You can do Get-Service | Out-String, and you will get the output as string, much like how Unix commands work.
Also when the output is piped to non-PowerShell commands, it does get converted to text, so for example: Get-Service | grep sql would work the way you wanted.
But again, like #JPBlanc says, it is good embrace the way PowerShell works, which is that the outputs are objects. It gives you way more control and keeps things simple and readable (the Unix commands with sed, awk and what not operating on text output of other command outputs can get very cryptic!).
You're working way too hard at it:
Get-Service *sql*
If anyone wants more information on logical operations, please see Using the Where-Object Cmdlet:
• -lt -- Less than
• -le -- Less than or equal to
• -gt -- Greater than
• -ge -- Greater than or equal to
• -eq -- Equal to
• -ne -- Not equal to
• -like - Like; uses wildcards for pattern matching
Get-Service | where {$_ -match 'sql'} would be Get-Service | where {$_ -eq "sql"}
Get-Service | where {$_ -like 'sql'} would be Get-Service | where {$_ -like "sql"}
And now an actual example.
PS C:\> Get-Service | where {$_.name -like "net*"}
Status Name DisplayName
------ ---- -----------
Running Net Driver HPZ12 Net Driver HPZ12
Running Netlogon Netlogon
That the text of the name is a property of the object is important to get your head around, and how to use the property values in a filter.
Another aspect of PowerShell you can leverage to solve this is selecting properties out of objects with Select-Object (alias select):
Get-Service | select -expand name
will get you a string array with the names of the servers, and two of your original three filters would work on that. The -like isn't going to work, because there's no wildcards in the test string. The only thing it will ever match is just 'sql'.
I still believe the first solution I posted is best. It's important to know how to do late filtering, but also how to use early filtering when you can.
If you want to list all services with "sql" in the service name, just use:
get-service -name *sql*
You probably want this:
Function Select-ObjectPropertyValues {
param(
[Parameter(Mandatory=$true, Position=0)]
[String]
$Pattern,
[Parameter(ValueFromPipeline)]
$input
)
$input | Where-Object {($_.PSObject.Properties | Where-Object {$_.Value -match $Pattern} | Measure-Object).count -gt 0} | Write-Output
}
Here we are going though each property of an object to see if it matches the given pattern. If the object contains one or more such properties, we write it out. End result: grep by all properties of an object.
Put it in your configuration files and grep to your heart's content.
how about:
Get-Service| Out-String -stream | Select-String sql
where the key point is that -stream option converts the Out-String output in separate lines of text.