Powershell Regex Multiline parsing - powershell

I am working on building a script that will analyze a configuration file (cisco switch config) and build a report based on certain findings. Sadly- the findings must be recorded on a specific form so this painful path is my only option outside of manual generation of each form.
What I'm trying to do:
Using the following I am attempting to pull the following multi-line expression into PS for evaluation
interface vlan1
no ip address
shutdown
!
I have found multiple sources that point towards one of two options- the first (and simplest) being to load the file into Get-content using the "-raw" switch in order to evaluate the entire file as a single string and then use the "select-string" command to output the specific information that I am looking for.
My basic code looks something like this
if (get-content -path U:\Testing\Test.txt -Raw | select-string -Pattern "(?ms)interface vlan1.*no ip address.*(?!no shutdown)shutdown.*\!" -Quiet)
{
write-host('pass')
}
else
{
write-host('fail')
}
Expected outcome: if the string is true- I will append the finding to a file (that part I have already)
If the configuration does not contain "shutdown" exclusively (without the word no) then it will be annotated as such (again I have that process as well)
Thank you in advance for your assistance- hopefully this is clear and concise.
Further clarity: the script returns false positives/negatives. when running the get-content + select-string outside of the if command- I basically get the -raw output but no match on the string itself, leading me to believe that I am having an issue with the start of line (interface vlan1) and the end line (!)
I have played with the structure of the regex string to try and tease out a solution but I am still a bit vague as to the usage of multi-line output while using select-string.

Since you need to look at the file in full, there's no reason to use the Select-String cmdlet, given that -match, the regular-expression matching operator, works more effectively on strings that are already in memory.
Note: -match only every finds one match (if any); if this is not sufficient, use the [regex]::Matches() .NET method; it is unfortunate that there's no operator for multiple matches; GitHub issue #7867 proposes introducing one, named -matchall.
Your regex is too permissive (greedy) due to use of .* across lines due to the (?s) matching option, so matching happens across multiple blocks.
The following uses a regex without .*, and instead explicitly matches the lines in full, including explicit matching of intervening newlines (\r?\n).[1]
This works with your sample input, but you may need to tweak the regex (omitting the (?s) option makes .* match only intra-line; expressions can be made non-greedy by modifying a duplication symbol with ? (e.g. .*?)).
$re = '(?m)^interface vlan1\r?\nno ip address\r?\n(?!no shutdown)shutdown\r?\n!'
if ((Get-Content U:\Testing\Test.txt -Raw) -match $re) {
# ...
}
Note: The assumption is that there's no need to validate that the trailing ! is the only character on its line; if that is needed, append (?:\r?\n|\z).[2]
[1] This regex matches both common newline formats: CRLF (\r\n, Windows) and LF (\n, Unix).
[2] Unfortunately, use of $ to assert the end of a line (with the (?m) option in effect) may not work if the input uses CRLF (\r\n) newlines, because the $ matches the position of a LF character (\n) only, which means that $ does not match immediately after !, due to the intervening \r.

Related

Is there a way to set the default EOL separator for Out-File in PowerShell 5.1?

I am looking for a way to set the default EOL marker to 0x0A when writing text with Out-File.
On the internet, I found tons of examples that either replace 0x0D 0x0A after a file is already written, or -join the lines on 0x0A and then write the concatenated text into the file.
I find both approaches a bit clumsy as I'd just like to write the files with the redirection operator >.
So, is there a way to set the EOL style in PowerShell 5.1?
No, unfortunately, as of PowerShell 7.2, there is no way to make PowerShell use a different newline (EOL, line break) format.
It is the platform-native newline character or sequence (as reflected in [Environment]::NewLine) that is invariably used - both for separating multiple input objects and for the trailing newline by default.
To control the newline format, you need to:
Join the input objects explicitly with the newline character (sequence) of choice, using the -join operator, followed by another instance if a trailing newline is desired ...
... and use the -NoNewLine switch of Set-Content / Out-File (in lieu of >) so as to prevent appending of a trailing platform-native newline.
As for potential future enhancements:
GitHub feature request #2872 suggests adding a parameter to Set-Content, specifically, to allow specifying the newline format; the request has been green-lighted (a long time ago), but has yet to be implemented - however, I think it isn't comprehensive enough, and it wouldn't help with Out-File / >; see next point.
GitHub feature request #3855 more generally asked for a -Delimiter parameter (to mirror Get-Content's existing parameter by that name) to be added to Set-Content / Add-Content, Out-File and Out-String
Unfortunately, the proposal was rejected; if it hadn't, you would have been able to configure > - a virtual alias of Out-File - to use LF-only newlines, for instance, as follows:
# WISHFUL THINKING
$PSDefaultParameterValues['Out-File:Delimiter'] = "`n"

Split text after each end of line [duplicate]

This question already has answers here:
Powershell: split string with newline, then use -contains
(3 answers)
Closed 1 year ago.
I have a script that works perfectly fine with Powershell 5.x, but does not work anymore on Powershell Core (7.2.1)
The problem happens when I try to split a text (copy&past from an email)..
It all comes down to this part of the code:
$test="blue
green
yellow
"
#$test.Split([Environment]::NewLine)
$x = $test.Split([Environment]::NewLine)
$x[0]
$x[1]
In Powershell 5 the value for $x[0]==blue and $x[1]==green
But in Powershell Core the split doesn't do anything and $x[1] is "non existent".
In Powershell 7 the line breaks are handled differently (that's at least what I assume), but I couldn't find a solution to it..
I tried it with changing the code to
$rows = $path.split([Environment]::NewLine) and $path.Split([System.Environment]::NewLine, [System.StringSplitOptions]::RemoveEmptyEntries) but that doesn't change anything..
Also, when I use a "here-string"
$test = #'
green
yellow
blue
white
'#
$x= $test -split "`r`n", 5, "multiline"
Everything excepts $x[0] is empty (i.e $x[2])
I was already looking here: https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_split?view=powershell-7.2
And here: powershell -split('') specify a new line
And here: WT: Paste multiple lines to Windows Terminal without executing
So far I have not found a solution to my problem.
Any help is appreciated.
EDIT: I found a hint about that problem, but don't understand the implications of it yet: https://n-v-o.github.io/2021-06-10-String-Method-in-Powershell-7/
EDIT 2:
Thanks everyone for participating in answering my question.
First I thought I'm going to write a long explanation why my question is different then the duplicated answer from #SantiagoSquarzon. But while reading the answers to my question and the other question I noticed I was doing something differently..
Apparently there is something differnt when I use
$splits = $test -split '`r?`n' # doesn't work in 5.1 and 7.2.1
$splits = $test -split '\r?\n' # works in 5.1 and 7.2.1 as suggested from Santiago and others
BUT
$splits = $test.Split("\r?\n") # doesn't work in 5.1 and 7.2.1
$splits = $test.Split("`r?`n") # doesn't work in 5.1 and 7.2.1
$splits = $test.Split([char[]]"\r\n") # doesnt' work in 7.2.1
$splits = $test.Split([char[]]"`r`n") # works in 7.2.1
tl;dr:
Use -split '\r?\n to split multiline text into lines irrespective of whether Windows-format CRLF or Unix-format LF newlines are used (it even handles a mix of these formats in a single string).
If you additionally want to handle CR-only newlines (which would be unusual, but appears to be the case for you), use -split '\r?\n|\r'
On Windows, with CRLF newlines only, .Split([Environment]::NewLine) only works as intended in PowerShell (Core) 7+, not in Windows PowerShell (and, accidentally, in Windows PowerShell only with CR-only newlines, as in your case.) To explicitly split by CR only, .Split("`r") would happen to work as intended in both editions, due to splitting by a single character only.
# Works on both Unix and Windows, in both PowerShell editions.
# Input string contains a mix of CRLF and LF and CR newlines.
"one`r`ntwo`nthree`rfour" -split '\r?\n|\r' | ForEach-Object { "[$_]" }
Output:
[one]
[two]
[three]
[four]
This is the most robust approach, as you generally can not rely on input text to use the platform-native newline format, [Environment]::NewLine; see the bottom section for details.
Note:
The above uses PowerShell's -split operator, which operates on regexes (regular expressions), which enables the flexible matching logic shown above.
This regex101.com page explains the \r?\n|\r regex and allows you to experiment with it.
By contrast, the System.String.Split() .NET method only splits by literal strings, which, while faster, limits you to finding verbatim separators.
The syntax implications are:
Regex constructs such as escape sequences \r (CR) and \n (LF) are only supported by the .NET regex engine and therefore only by -split (and other PowerShell contexts where regexes are being used); ditto for regex metacharacters ? (match the preceding subexpression zero or one time) and | (alternation; match the subexpression on either side).
Inside strings (which is how regexes must be represented in PowerShell, preferably inside '...'), these sequences and characters have no special meaning, neither to PowerShell itself nor to the .Split() method, which treats them all verbatim.
By contrast, the analogous escape sequences "`r" (CR) and "`n" (LF) are PowerShell features, available in expandable strings, i.e. they work only inside "..." - not also inside verbatim strings, '...' - and are expanded to the characters they represent before the target operator, method, or command sees the resulting string.
This answer discusses -split vs. .Split() in more depth and recommends routine use of -split.
As for what you tried:
Use [Environment]::NewLine only if you are certain that the input string uses the platform-native newline format. Notably, multiline string literals entered interactively at the PowerShell prompt use Unix-format LF newlines even on Windows (the only exception is the obsolescent Windows-only ISE, which uses CRLF).
String literals in script files (*.ps1) use the same newline format that the script is saved in - which may or may not be the platform's format.
Additionally, as you allude to in your own answer, the addition of a string parameter overload in the System.String.Split() method in .NET Core / .NET 5+ - and therefore PowerShell (Core) v6+ - implicitly caused a breaking change relative to Windows PowerShell: specifically, .Split('ab') splits by 'a' or 'b' - i.e. by any of the individual characters that make up the string - in Windows PowerShell, whereas it splits by the whole string, 'ab', in PowerShell (Core) v6+.
Such implicit breaking changes are rare, but they do happen, and they're outside PowerShell's control.
For that reason, you should always prefer PowerShell-native features for long-term stability, which in this case means preferring the -split operator to the .Split() .NET method.
That said, sometimes .NET methods are preferable for performance reasons; you can make them work robustly, but only if carefully match the exact data types of the method overloads of interest, which may require cast; see below.
See this answer for more information, including a more detailed explanation of the implicit breaking change.
Your feedback on -split '\r?\n' not working for you and the solutions in your own answer suggest that your input string - unusually - uses CR-only newlines.
Your answer's solutions would not work as expected with Windows-format CRLF-format text, because splitting would happen for each CR and LF in isolation, which would result in extra, empty elements in the output array (each representing the empty string "between" a CRLF sequence).
If you did want to split by [Environment]::NewLine on Windows - i.e. by CRLF - and you wanted to stick with the .Split() method, in order to make it work in Windows PowerShell too, you'd need to call the overload that expects a [string[]] argument, indicating that each string (even if only one) is to be used as a whole as the separator - as opposed to splitting by any of its individual characters:
# On Windows, split by CRLF only.
# (Would also work on Unix with LF-only text.)
# In PowerShell (Core) 7+ only, .Split([Environment]::NewLine) would be enough.
"one`r`ntwo`r`nthree".Split([string[]] [Environment]::NewLine, [StringSplitOptions]::None) |
ForEach-Object { "[$_]" }
Output:
[one]
[two]
[three]
While this is obviously more ceremony than using -split '\r?\n', it does have the advantage of performing better - although that will rarely matter. See the next section for a generalization of this approach.
Using an unambiguous .Split() call for improved performance:
Note:
This is only necessary if -split '\r?\n' or -split '\r?\n|\r' turns out to be too slow in practice, which won't happen often.
To make this work robustly, in both PowerShell editions as well as long-term, you must carefully match the exact data types of the .Split() overload of interest.
The command below is the equivalent of -split '\r?\n|\r', i.e. it matches CRLF, LF, and CR newlines. Adapt the array of strings for more restrictive matching.
# Works on both Unix and Windows, in both PowerShell editions
"one`r`ntwo`nthree`rfour".Split(
[string[]] ("`r`n", "`n", "`r"),
[StringSplitOptions]::None
) | ForEach-Object { "[$_]" }
The reason: When pasting text into the terminal, it matters which terminal you are using. The default powershell 5.1, ISE terminals, and most other Windows software separates new lines with both carriage return \r and newline \n characters. We can check by converting to bytes:
# 5.1 Desktop
$test = "a
b
c"
[byte[]][char[]]$test -join ','
97,13,10,98,13,10,99
#a,\r,\n, b,\r,\n, c
Powershell Core separates new lines with only a newline \n character
# 7.2 Core
$test = "a
b
c"
[byte[]][char[]]$test -join ','
97,10,98,10,99
On Windows OS, [Environment]::NewLine is \r\n no matter which console. On Linux, it is \n.
The solution: split multiline strings on either \r\n or \n (but not on only \r). The easy way here is with regex like #Santiago-squarzon suggests:
$splits = $test -split '\r?\n'
$splits[0]
a
$splits[1]
b
Thanks to this site I found a solution:
https://n-v-o.github.io/2021-06-10-String-Method-in-Powershell-7/
In .NET 4, the string class only had methods that took characters as
parameter types. PowerShell sees this and automagically converts it,
to make life a little easier on you. Note there’s an implied ‘OR’ (|)
here as it’s an array of characters.
Why is PowerShell 7 behaving differently? In .NET 5, the string class
has some additional parameters that accept strings. PowerShell 7 does
not take any automatic action.
In order to fix my problem, I had to use this:
$test.Split("`r").Split("`n") #or
$test.Split([char[]]"`r`n")

Remove all text after line with [info]

I am trying to remove all lines of text after a single line of text "[info]" Here is an example:
Top=1266
[info]
name=tod
space=456
number=221,441,111,0
[version]
version=1
I only need the top, the other text will be replaced later on in the script. Here is all that I have tried
$Content -replace '\[Info\]*',''
Only removes the Info line and not anything past that. I have tried to loop, but I can't seem to find the line with a where object search.
What is a quick and easy way to remove all lines of code after a single line of set text?
To make the -replace operator treat it as one string, add (?s) to the pattern.
$Content -replace '(?s)\[Info\].*'
You also needed to match any character so .* works in this case. The second part is optional. Since you're replacing it with nothing you can simply omit it.
Read more about regular expression in powershell
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_regular_expressions?view=powershell-7.1
and operators
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_operators?view=powershell-7.1

Design Powershell script for find the Numbers which contain file

Everyone help to design the script to find the Numbers which contain file..
For example:
20200514_EE#998501_12.
I need numbers 12 then write to the txt file
the contain will generated different sequence numbers..
For example: #20200514_EE#998501_123.#
so, I need numbers 123 then write to the txt file
How to write the script in Powershell or bat file ?
Very appreciate!
Thanks
Tony
You can do the following as a start. You have not provided enough information/examples to work through any issues you are experiencing.
'#20200514_EE#998501_123.#' -replace '^.*?(\d+)\D*$','$1'
'#20200514_EE#998501_123' -replace '^.*?(\d+)\D*$','$1'
-replace uses regex matching and then replaces with a string and/or matched substitute. ^ is the start of the string. .*? lazily matches all characters. \d+ matches one or more digits in a capture group due to the encapsulating (). \D* matches zero or more non-digits. $ matches the end of the string. For the replacement, $1 is capture group 1, which is what was captured by (\d+).
You can use the .Split() method also in combination with -replace.
'#20200514_EE#998501_123.#'.Split('_')[-1] -replace '\D+$'

Extracting substring from inside bracketed string, where the substring may have spaces

I've got an application that has no useful api implemented, and the only way to get certain information is to parse string output. This is proving to be very painful...
I'm trying to achieve this in bash on SLES12.
Given I have the following strings:
QMNAME(QMTKGW01) STATUS(Running)
QMNAME(QMTKGW01) STATUS(Ended normally)
I want to extract the STATUS value, ie "Ended normally" or "Running".
Note that the line structure can move around, so I can't count on the "STATUS" being the second field.
The closest I have managed to get so far is to extract a single word from inside STATUS like so
echo "QMNAME(QMTKGW01) STATUS(Running)" | sed "s/^.*STATUS(\(\S*\)).*/\1/"
This works for "Running" but not for "Ended normally"
I've tried switching the \S* for [\S\s]* in both "grep -o" and "sed" but it seems to corrupt the entire regex.
This is purely a regex issue, by doing \S you requested to match non-white space characters within (..) but the failing case has a space between which does not comply with the grammar defined. Make it simple by explicitly calling out the characters to match inside (..) as [a-zA-Z ]* i.e. zero or more upper & lower case characters and spaces.
sed 's/^.*STATUS(\([a-zA-Z ]*\)).*/\1/'
Or use character classes [:alnum:] if you want numbers too
sed 's/^.*STATUS(\([[:alnum:] ]*\)).*/\1/'
sed 's/.*STATUS(\([^)]*\)).*/\1/' file
Output:
Running
Ended normally
Extracting a substring matching a given pattern is a job for grep, not sed. We should use sed when we must edit the input string. (A lot of people use sed and even awk just to extract substrings, but that's wasteful in my opinion.)
So, here is a grep solution. We need to make some assumptions (in any solution) about your input - some are easy to relax, others are not. In your example the word STATUS is always capitalized, and it is immediately followed by the opening parenthesis (no space, no colon etc.). These assumptions can be relaxed easily. More importantly, and not easy to work around: there are no nested parentheses. You will want the longest substring of non-closing-parenthesis characters following the opening parenthesis, no mater what they are.
With these assumptions:
$ grep -oP '\bSTATUS\(\K[^)]*(?=\))' << EOF
> QMNAME(QMTKGW01) STATUS(Running)
> QMNAME(QMTKGW01) STATUS(Ended normally)
> EOF
Running
Ended normally
Explanation:
Command options: o to return only the matched substring; P to use Perl extensions (the \K marker and the lookahead). The regexp: we look for a word boundary (\b) - so the word STATUS is a complete word, not part of a longer word like SUBSTATUS; then the word STATUS and opening parenthesis. This is required for a match, but \K instructs that this part of the matched string will not be returned in the output. Then we seek zero or more non-closing-parenthesis characters ([^)]*) and we require that this be followed by a closing parenthesis - but the closing parenthesis is also not included in the returned string. That's a "lookahead" (the (?= ... ) construct).