I m learning PowerShell and one of the task I did is to filter a Csv file records.
Based on this link: https://4sysops.com/archives/create-sort-and-filter-csv-files-in-powershell/ I tried something similar to:
Import-Csv -Path '.\sample.csv' | Select-Object EmailAddress,UniqueName,LastLoginDate | ? EmailAddress -like *gmail.com -Or ? EmailAddress -like *outlook.com | Export-Csv -Path $fileOut -NoTypeInformation
But the above gives me the error mentioned in the title.
Based on this link: https://www.computerperformance.co.uk/powershell/match/ I addressed the error by using Where-Object instead after the Select-Object line as follows:
Where-Object {$_.EmailAddress -Like "*gmail.com" -Or $_.EmailAddress -Like "*outlook.com"}
Why does the first example give me error but not the second example?
tl;dr
Both your commands use the Where-Object cmdlet; ? is simply a built-in alias for it.
However, your commands use different syntax forms: your first command uses the simpler and more concise, but feature-limited individual argument-based simplified syntax, whereas your second one uses the verbose, but fully featured script-block syntax - see next section.
Because you need to combine multiple -like operations, you must use script-block syntax - simplified syntax limits you to a single operation.
Regular, script block-based syntax:
Example:
# You're free to add additional expressions inside { ... }
Where-Object { $_.EmailAddress -like '*gmail.com' }
uses a single argument that is a script block ({ ... }), inside of which the condition to test is formulated based on the automatic $_ variable that represents the input object at hand.
This syntax:
Places no constraints on the complexity of the expression - the whole PowerShell language is at your disposal inside a script block.
However, it is somewhat verbose.
Simplified, multi-argument syntax:
Example:
# Equivalent of the above.
# Note the absence of { ... }, $_, and "..."
Where-Object EmailAddress -like *gmail.com
Simplified syntax is an alternative syntax that may be used with Where-Object as well as ForEach-Object, which:
as the name implies, is simpler and less verbose.
but is limited to a single conditional / operation based on a single property, or, in the case of method calls with ForEach-Object, the input object itself.
With simplified syntax the parts that make up a conditional / method call are passed as separate arguments, which therefore bind to distinct parameters that are specifically designed to work with this syntax:
Because separate arguments are used, there is no { ... } enclosure (no script block is used).
$_ need not be referenced, because its use is implied; e.g. EmailAddress is the equivalent of $_.EmailAddress in the script block-syntax.
A notable limitation as of PowerShell 7.2.x is that with Where-Object you cannot operate on the input object itself - you must specify a property. GitHub issue #8357 discusses overcoming this limitation in the future, but there hasn't been any activity in a long time.
As usual in argument-mode parsing, quoting around string values is optional, assuming they don't contain metacharacters such as spaces; e.g., *.gmail.com - without "..." or '...' - works with simplified syntax, whereas the expression-mode parsing inside the equivalent script block requires quoting, e.g. '*gmail.com'
Related
After playing around with some powershell script for a while i was wondering if there is a version of this without using c#. It feels like i am missing some information on how to pipe things properly.
$packages = Get-ChildItem "C:\Users\A\Downloads" -Filter "*.nupkg" |
%{ $_.Name }
# Select-String -Pattern "(?<packageId>[^\d]+)\.(?<version>[\w\d\.-]+)(?=.nupkg)" |
# %{ #($_.Matches[0].Groups["packageId"].Value, $_.Matches[0].Groups["version"].Value) }
foreach ($package in $packages){
$match = [System.Text.RegularExpressions.Regex]::Match($package, "(?<packageId>[^\d]+)\.(?<version>[\w\d\.-]+)(?=.nupkg)")
Write-Host "$($match.Groups["packageId"].Value) - $($match.Groups["version"].Value)"
}
Originally i tried to do this with powershell only and thought that with #(1,2,3) you could create an array.
I ended up bypassing the issue by doing the regex with c# instead of powershell, which works, but i am curious how this would have been done with powershell only.
While there are 4 packages, doing just the powershell version produced 8 lines. So accessing my data like $packages[0][0] to get a package id never worked because the 8 lines were strings while i expected 4 arrays to be returned
Terminology note re without using c#: You mean without direct use of .NET APIs. By contrast, C# is just another .NET-based language that can make use of such APIs, just like PowerShell itself.
Note:
The next section answers the following question: How can I avoid direct calls to .NET APIs for my regex-matching code in favor of using PowerShell-native commands (operators, automatic variables)?
See the bottom section for the Select-String solution that was your true objective; the tl;dr is:
# Note the `, `, which ensures that the array is output *as a single object*
%{ , #($_.Matches[0].Groups["packageId"].Value, $_.Matches[0].Groups["version"].Value) }
The PowerShell-native (near-)equivalent of your code is (note tha the assumption is that $package contains the content of the input file):
# Caveat: -match is case-INSENSITIVE; use -cmatch for case-sensitive matching.
if ($package -match '(?<packageId>[^\d]+)\.(?<version>[\w\d\.-]+)(?=.nupkg)') {
"$($Matches['packageId']) - $($Matches['Version'])"
}
-match, the regular-expression matching operator, is the equivalent of [System.Text.RegularExpressions.Regex]::Match() (which you can shorten to [regex]::Match()) in that it only looks for (at most) one match.
Caveat re case-sensitivity: -match (and its rarely used alias -imatch) is case-insensitive by default, as all PowerShell operators are; for case-sensitive matching, use the c-prefixed variant, -cmatch.
By contrast, .NET APIs are case-sensitive by default; you'd have to pass the [System.Text.RegularExpressions.RegexOptions]::IgnoreCase flag to [regex]::Match() for case-insensitive matching (you may use 'IgnoreCase', which PowerShell auto-converts for you).
As of PowerShell 7.2.x, there is no operator that is the equivalent of the related return-ALL-matches .NET API, [regex]::Matches(). See GitHub issue #7867 for a green-lit but yet-to-be-implemented proposal to introduce one, named -matchall.
However, instead of directly returning an object describing what was (or wasn't) matched, -match returns a Boolean, i.e. $true or $false, to indicate whether matching succeeded.
Only if -match returns $true does information about a match become available, namely via the automatic $Matches variable, which is a hashtable reflecting the matching parts of the input string: entry 0 is always the full match, with optional additional entries reflecting what any capture groups ((...)) captured, either by index, if they're anonymous (starting with 1) or, as in your case, for named capture groups ((?<name>...)) by name.
Syntax note: Given that PowerShell allows use of dot notation (property-access syntax) even with hashtables, the above command could have used $Matches.packageId instead of $Matches['packageId'], for instance, which also works with the numeric (index-based) entries, e.g., $Matches.0 instead of $Matches[0]
Caveat: If an array (enumerable) is used as the LHS operand, -match' behavior changes:
$Matches is not populated.
filtering is performed; that is, instead of returning a Boolean indicating whether matching succeeded, the subarray of matching input strings is returned.
Note that the $Matches hashtable only provides the matched strings, not also metadata such as index and length, as found in [regex]::Match()'s return object, which is of type [System.Text.RegularExpressions.Match].
Select-String solution:
$packages |
Select-String '(?<packageId>[^\d]+)\.(?<version>[\w\d\.-]+)(?=.nupkg)' |
ForEach-Object {
"$($_.Matches[0].Groups['packageId'].Value) - $($_.Matches[0].Groups['version'].Value)"
}
Select-String outputs Microsoft.PowerShell.Commands.MatchInfo instances, whose .Matches collection contains one or more [System.Text.RegularExpressions.Match] instances, i.e. instances of the same type as returned by [regex]::Match()
Unless -AllMatches is also passed, .Matches only ever has one entry, hence the use of [0] to target that entry above.
As you can see, working with Select-Object's output objects requires you to ultimately work with the same .NET type as when you call [regex]::Match() directly.
However, no method calls are required, and discovering the properties of the output objects is made easy in PowerShell via the Get-Member cmdlet.
If you want to capture the matches in a jagged array:
$capturedStrings = #(
$packages |
Select-String '(?<packageId>[^\d]+)\.(?<version>[\w\d\.-]+)(?=.nupkg)' |
ForEach-Object {
# Output an array of all capture-group matches,
# *as a single object* (note the `, `)
, $_.Matches[0].Groups.Where({ $_.Name -ne '0' }).Value
}
)
This returns an array of arrays, each element of which is the array of capture-group matches for a given package, so that $capturedStrings[0][0] returns the packageId value for the first package, for instance.
Note:
$_.Matches[0].Groups.Where({ $_.Name -ne '0' }).Value programmatically enumerates all capture-group matches and returns an their .Value property values as an array, using member-access enumeration; note how name '0' must be excluded, as it represents the whole match.
With the capture groups in your specific regex, the above is equivalent to the following, as shown in a commented-out line in your question:
#($_.Matches[0].Groups['packageId'].Value, $_.Matches[0].Groups['version'].Value)
, ..., the unary form of the array-construction operator, is used as a shortcut for outputting the array (symbolized by ... here) as a whole, as a single object. By default, enumeration would occur and the elements would be emitted one by one. , ... is in effect a shortcut to the conceptually clearer Write-Output -NoEnumerate ... - see this answer for an explanation of the technique.
Additionally, #(...), the array subexpression operator is needed in order to ensure that a jagged array (nested array) is returned even in the event that only one array is returned across all $packages.
I'm trying to get the following command into a program I am developing in PowerShell studio, I'm having some issues figuring out the escaping sequence here, and testing takes forever since I have to import the PST just to view the data :(
Consider the following command which works correctly from the shell...
New-MailboxExportRequest -Mailbox user#domain.com -Name JobName -IncludeFolders "#Inbox#/*","#SentItems#" -ContentFilter {(Received -gt "06/10/2022") -and (Sent -gt "06/10/2022")} -FilePath "\\Server\Folder\MyPST.pst"
I am using variables inside program designer, so my actual code looks like this...
New-MailboxExportRequest -Mailbox $mailbox -Name $jobname -IncludeFolders $IncludeFolders -ContentFilter {(Received -gt "$RecDate") -and (Sent -gt "$SentDate")} -FilePath "$FilePath"
Basically I need to get the first code sample working using variables, however the -IncludeFolders and -ContentFilter parameters require some escaping I can't seem to figure out. Any and all help is much appreciated.
tl;dr:
You need to enclose your -ContentFilter argument in "..." overall in order to support embedding variable values, which you can themselves enclose in embedded '...' quoting:
-ContentFilter "(Received -gt '$RecDate') -and (Sent -gt '$SentDate')"
No special syntax considerations apply with respect to using a variable with -IncludeFolders: Just make sure that variable $IncludeFolders contains an array of strings identifying the target folders; applied to your example:
$IncludeFolders = '#Inbox#/*', '#SentItems#'
If the list of folders must be parsed from user input provided as a single string ($textbox5.text in this example) do the following to convert this single string to an array of names:
# If $textbox5.text contains string '#Inbox#/*, #SentItems#',
# the result is the same as above.
$IncludeFolders = ($textbox5.text -split ',').Trim()
Read on for background information and caveats re -ContentFilter.
Note:
The -ContentFilter information below also applies to other Exchange cmdlets that accept filters, such as Get-Recipient with its -Filter parameter.
It also applies analogously to the -Filter parameter used with AD (Active Directory) cmdlets such as Get-ADUser, although the situation there is complicated by the fact that the AD provider does support evaluation of PowerShell variables, but only in simple cases - see this answer.
New-MailboxExportRequest's -ContentFilter expects a string argument, not a script block ({ ... }), which you've used in your question.
While you can situationally get away with a script block, it is best avoided for two reasons:
When a script block is used as a string (converted to one), string interpolation (embedding the values of variables in the string) is not supported - a script block stringifies to its verbatim content (sans { and }).
Conceptually, use of a script block can give the mistaken impression that an arbitrary piece of PowerShell code may be passed, which is not the case: -ContentFilter supports a domain-specific syntax only that only emulates a limited set of PowerShell features, called OPath filter syntax.
Notably, evaluation of PowerShell variables is not supported - their values must be embedded in the string up front, by PowerShell, using string interpolation.
Therefore:
It's best to use a string argument with -ContentFilter to begin with.
Since you do need to embed variable values, you need string interpolation, via an expandable (double-quoted) string ("..."), inside of which you may quote values with '...' for syntactical convenience.
In case there is no need to embed variable values, use a verbatim (single-quoted) string ('...'), inside of which you may quote values with "...", as in your question. (In effect, this is the equivalent of the (ill-advised) use of { ... }).
Applied to your example: The following embeds the (stringified) values of variables $RecDate and $SentDate in your -ContentFilter argument:
# Note the use of "..." for the outer quoting, and
# '...' for the embedded quoting.
-ContentFilter "(Received -gt '$RecDate') -and (Sent -gt '$SentDate')"
Caveats:
If the values of the variables to embed themselves contain ', use escaped embedded "..." quoting instead; e.g., `"$var`"
If a value could contain ' and/or " and you don't know which, you'd have to use an extra layer of up-front string interpolation on the PowerShell side to escape one of them, depending on the embedded quoting character chosen, using $(...), the subexpression operator, using a -replace operation:
In this escaping you must technically satisfy the OPath filter syntax requirements; the linked help OPath topic suggests that, like in PowerShell, '' can be used to escape ' inside a '...' string; e.g.: -ContentFilter "Body -like '$($var -replace "'", "''")*'"
While, as stated, OPath filters do not support evaluating PowerShell variable references in general, the following automatic variables - which are conceptually constants - are recognized (the linked help topic calls them system values):
$true, $false
$null
Therefore, if you use "..." for the outer quoting (for string interpolation), the $ in these variables must be escaped with ` (the so-called backtick, PowerShell's escape character), to prevent PowerShell from replacing these variable references with their values up front:
`$true, `$false
`$null
The script below finds folders older than 30 days. I just want to add a simple IF statement to say "There are not folders older than 30 days" if there are not any. But I am not sure how to do that
Thank you
$Test = Get-ChildItem "\\Server\XFER\Cory" -Directory |
Sort LastWriteTime -Descending |
Where-Object {($_.LastWriteTime -lt (Get-Date).AddDays(-30))} |
Select-Object Name, LastWriteTime
Your question boils down to this:
In PowerShell, how can I determine if a command or expression produced any output?
In almost all cases, the following is sufficient, as Ash suggests:
$Test = ... # ... represents any command or expression
if ($null -eq $Test) { 'no output' }
If you know that the command or expression - when it does produce output - only ever emits non-numeric and non-Boolean objects, you can simplify to the following, as Santiago Squarzon suggests, relying on PowerShell's implicit to-Boolean coercion logic, summarized in the bottom section of this answer:
$Test = ... # ... represents any command or expression
if (-not $Test) { 'no output' }
If you're dealing with an (unusual) command that outputs a collection (array) as a single object (as opposed to enumerating the collection and outputting each element separately, which is the normal pipeline behavior) and you want to treat an empty collection object as lack of output too:
$Test = ... # ... represents any command or expression
# Caveat: Due to a bug, only works robustly with
# Set-StrictMode turned off (the default)
# or Set-StrictMode -Version 1 (not higher).
if ($Test.Count -eq 0) { 'no output' }
Note that this works even with $null output and a scalar output object (a single, non-collection object). In the interest of unified treatment of scalars and collections (arrays), PowerShell adds a .Count property even to scalars that themselves do not have it, so that scalars can be treated as if they were a single-element collection, also with respect to indexing; e.g. (42).Count is 1, and (42)[0] is 42; however, note that $null.Count is 0. Such PowerShell engine-supplied type members are called intrinsic members.
Caveat: Due to a long-standing bug - reported in GitHub issue #2798 and still present as of PowerShell 7.2 - accessing the intrinsic .Count property on objects that don't natively have it causes a statement-terminating error if Set-StrictMode -Version 2 or higher is in effect.
However, the tests above do not allow you to distinguish between no output at all and a (single) $null value, which requires the following approach - though do note that actual $null output is unusual in PowerShell:[1]
$Test = ... # ... represents any command or expression
if ($null -eq $Test -and #($Test).Length -eq 0) { 'no output' }
This obscure test is necessary, because no output in PowerShell is represented by the [System.Management.Automation.Internal.AutomationNull]::Value] singleton, which behaves like $null in expression contexts, but not in enumeration contexts such as in a pipeline, including with #(...), the array-subexpression operator, which returns an empty array for [System.Management.Automation.Internal.AutomationNull]::Value] (element count 0) and a single-element array for $null.
While making the distinction between $null and [System.Management.Automation.Internal.AutomationNull]::Value often isn't necessary, there are definitely cases when it is, given that their enumeration behavior differs. Being able to distinguish via a simple test such as $Test -is [AutomationNull] is the subject of GitHub proposal #13465.
[1] Returning $null from PowerShell commands (cmdlets, scripts, functions) is best avoided; instead, simply omit output commands. However, .NET API methods may still return $null and object properties may contain $null (even [string]-typed ones, and even in PowerShell class definitions - see GitHub issue #7294).
I was assisting a user with this question, linked to my answer here: Powershell script to add users to A/D group from .csv using email address only?
Initially I wrote the script as follows, using a bracket-based filter for Get-AdUser like follows:
Import-CSV "C:\users\Balbahagw\desktop\test1.csv" |
Foreach-Object {
# Here, $_.EmailAddress refused to resolve
$aduser = Get-ADUser -Filter { EmailAddress -eq $_.EmailAddress }
if( $aduser ) {
Write-Output "Adding user $($aduser.SamAccountName) to groupname"
Add-ADGroupMember -Identity groupname -Members $aduser
} else {
Write-Warning "Could not find user in AD with email address $($_.EmailAddress)"
}
}
However, $_.EmailAddress failed to populate a value. However, changing the Get-ADUser filter to a string-based filter worked as intended:
$aduser = Get-ADUser -Filter "EmailAddress -eq '$($_.EmailAddress)'"
What is the strangeness I'm experiencing, and why? Is it because when I'm using brackets, it's treated as a new scope and the $PSItem won't follow?
-Filter parameters are generally string parameters (verify with
Get-Help Get-AdUser -Parameter Filter)
They generally do not accept PowerShell code - filters are provider-specific and often have their own syntax, although it happens to be PowerShell-like in the case of the AD cmdlets.
Also, they generally have no knowledge of PowerShell variables (see below).
Thus, when a script block ({ ... }) is passed, it is converted to a string, which evaluates to its literal contents (everything between the opening { and the closing }):
{ EmailAddress -eq $_.EmailAddress }.ToString() yields the literal string EmailAddress -eq $_.EmailAddress - without any evaluation - and that's what Get-AdUser sees - no evaluation takes place.
In a presumably well-meaning but misguided effort to support the widespread, but ill-advised practice of passing script blocks to the -Filter parameter of AD cmdlets, it seems that these cmdlets actually explicitly expand simple variable references such as $_ in the string literal they receive, but that doesn't work with expressions, such as accessing a property of a variable ($_.EmailAddress)
Therefore, -Filter arguments should generally be passed as expandable strings ("..."); in the case at hand:
-Filter "EmailAddress -eq '$($_.EmailAddress)'"
That is, the only robust solution is to use strings with the variable parts baked in, up front, via string expansion, as shown above.
For values that are neither numbers nor strings, such as dates, you may have to use a literal string ('...') and rely on the AD provider's ability to evaluate simple references to PowerShell variables (e.g., $date) - see this answer of mine for details.
As stated, the syntax of AD filters is only PowerShell-like: it supports only a subset of the operators that PowerShell supports and those that are supported differ subtly in behavior - see Get-Help about_ActiveDirectory_Filter.
It is tempting to use script blocks, because the code inside requires no escaping of embedded quotes / no alternating of quote chars and no use of subexpression operator $(...). However, aside from using script blocks as strings being inefficient in general, the problem here is that the script block is making a promise that it cannot keep: it looks like you're passing a piece of PowerShell code, but you're not - and it works only in simple cases (and then only due to the misguided accommodation mentioned above); generally, it's hard to remember under what circumstances it doesn't work and how to make it work if it fails.
It is therefore really unfortunate that the official documentation uses script blocks in its examples.
For a more comprehensive discussion, see this answer of mine.
You're not wrong, it's the module's fault
The type of payload you have to use with the -Filter parameter differs depending on which provider you're working with, a design decision which can be pretty confusing!
The output of Get-Help Get-ADUser -Parameter Filter gives you some pretty detailed examples of the different syntax options you can use with the Active Directory Provider's implementation of Filter syntax.
Here's an example:
#To get all user objects that have an e-mail message attribute, use one of the following commands:
Get-ADUser -Filter {EmailAddress -like "*"}
It looks like the ActiveDirectory provider places the specific restriction that you must wrap the input in quotes. Here's what happens when I look for my account without putting quotes around my e-mail.
Get-ADUser -Filter {EmailAddress -eq stephen#foxdeploy.com}
Get-ADUser : Error parsing query: 'EmailAddress -eq stephen#foxdeploy.com'
Error Message: 'syntax error' at position: '18'.
But adding quotes? It works!
Get-ADUser -Filter {EmailAddress -eq "stephen#foxdeploy.com"}
DistinguishedName : CN=Stephen,CN=Users,DC=FoxDeploy,DC=local
Enabled : True
GivenName : Stephen
Name : Stephen
ObjectClass : user
ObjectGUID : 6428ac3f-8d17-45d6-b615-9965acd9675b
SamAccountName : Stephen
SID : S-1-5-21-3818945699-900446794-3716848007-1103
Surname :
UserPrincipalName : Stephen#FoxDeploy.local
How to make yours work
Now, because of this confusing filter implementation, you will need to change your user lookup on line 5 to the following:
$aduser = Get-ADUser -Filter "EmailAddress -eq `"$($_.EmailAddress)`""
We are providing the -Filter payload as a String. Next we want to use String Expansion to pull out the .EmailAddress property, so we wrap the string in $( ) to signal string expansion. Finally, the provider wants our filter comparison wrapped in quotes, so we put double quotes around it, and then escape the quotes using the backtick character.
And now it should work.
TLDR - blame the provider and blame the module, there are so many inconsistencies with the Active Directory module.
I'm new to powershell and I'm still trying to understand the syntax.
In a command like this
Get-ChildItem *.txt | Rename-Item -NewName { $_.name -Replace '\.txt$','.log' }
What does - actually do?
Sometimes it is there, sometimes not, and I just don't understand what purpose it has xD
The hyphen character in PowerShell has a variety of uses that are context specific. It is primarily used in cmdlet names, parameters, operators and of course as a character literal.
CMDLet names
It is commonly used as a verb-noun separator e.g. Get-ChildItem. This is an encouraged practice when making custom functions and cmdlets as well.
Parameters
Used to tell the parser that a word denotes a parameter e.g. Rename-Item -NewName. In the example it is -NewName
Operators
This is a broad section but you will see the hyphen denote operators like -replace in your code sample. It does not always have a keyword associated either which is the case with arithmetic operators (-) and assignment operators (-=). You will also see the hyphen with comparison, containment, pattern-matching/text and logical/bitwise operators.
In Powershell, All the parameters of a specific cmdlet has been defined to start with "-". It indicates the parameters are for the corresponding cmdlet.
All the cmdlets are functions that are written in Csharp or Powershell functions where they have defined the way to pass the argument of the parameters like:
Get-ChildItem *.txt | Rename-Item -NewName { $_.name -Replace '\.txt$','.log' }
Get-Childitem is the cmdlet which has a -include paramater and whose value is **.txt*. So even though you have not given the parameter name , powershell has the capability to identify certain parameters value by native.
So that is the reason it is having no issue.
Similarly when you are piping the output of the first cmdlet to the second one (which is Rename-item) , it has a parameter -NewName whose value has been passed as an output of the entire {$_.name -Replace '.txt$','.log' }
Hope it helps you.
-LiteralPath is what the meaning of using -. It's for paramters.
From your example, -NewName is treated as a parameter and -Replace is the operator
EDIT: As Peter rightly pointed out; amended the answer