How to Parse a logfile in powershell and write out desired output - powershell

I have a script which uses robocopy to transfer files and write logs to a file "Logfile.txt" after that, i parse the file "LogFile.txt" further and skim the necessary data and write it to other text file called "LogFile_Parsed.Txt".My issue is over here.Initially i calculate no of lines and parse each and every line ; whats my goal is when i reach a line which matches the word skipped , if the line number is x; i will append out the lines (x-5) to (x+1) to the new log file "LogFile_Parsed.Txt". The line what i am talking about is below;
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 1 0 0 0 0
Now , whwere i am stuck is ; i only want to append these lines to parsed log fiel, when the digit below the line skipped or failed is greater than 0; i.e like following ;
Total Copied Skipped Mismatch FAILED Extras
Dirs : 1 1 1 0 1 0
How can it be done? the above 2 lines i mentioned are consistent throughout the log file.How can i know the exact position of digit under skipped or failed and read it? Please let me know your valuable suggestions.

If I understand correctly, you want to find any line with the word "Skipped" followed by a line with the number 1 in the column below "Skipped", and append those two lines and the five preceding lines to a new file?
Read LogFile.txt into an array
Iterate through the array searching for lines with "Skipped"
Whenever you find one, use a regex match to see if the next line (i.e., next element of the array) has a 1 in the corresponding position
Use an array slice to get the elements from 5 preceding to 1 following the current one, and append it to the new file
The following will work if all the matching lines are formatted as in your example:
$logfile = gc '<path>\Logfile.txt'
for ($i = 0; $i -lt $logfile.count; $i++) {
if ($logfile[$i] -match 'Skipped') {
if ($logfile[$i + 1] -match '(?<=Dirs :(\s+[0-9]+){2}\s+)1') {
$logfile[($i - 5)..($i + 1)] | Out-File -Append '<path>\Logfile_Parsed.txt'
}
}
}
If the columns can vary in number and order, you'll need to use capture groups to find the ordinal position of "Skipped" and check if there is a 1 in the corresponding position on the next line. That's a little more complicated, so I won't get into that if this is sufficient.

Related

Extract log lines from a starting string to the first time stamp after with PowerShell

It is my first time that I am reaching back to you as I am stuck on something and been scratching my head for over a week now. It is worth saying that I just started with PowerShell a few months ago and I love using it for my scripts, but apparently my skills still need improving. I am unable to find a simple and elegant solution that would extract a log from clearly defined start line until the first empty line CF\LF or time stamp that follows.
I am attaching the log I am trying to extract the data from. To specify the problem and give some more details about the log lines - they can vary in number, the end line of each log can also vary and the time stamp is different for each log depending on the time the test was executed.
cls
# Grab the profile system path
$userProfilePath = $env:LOCALAPPDATA
# Define log path
$logPath = "$userProfilePath\DWIO\logs\IOClient.txt"
# Define the START log line matching string
# This includes the the tests that PASS and FAIL
$logStartLine = " TEST "
# Find all START log lines matching the string and grab their line number
$StartLine = (Get-Content $logPath | select-string $logStartLine)
#Get content from file
foreach ($start in $StartLine) {
# Extract the date time stamp from every starting line
$dateStamp = ($start -split ' ')[0]
#Regex pattern to compare two strings
$pattern = "(.*)$dateStamp"
#Perform the opperation
$result = [regex]::Match($file,$pattern).Groups[1].Value
Write-Host $result
}
The log format is like:
08-31 16:32:20 INFO - [IOBridgeThread - mPerformAndComputeIntegrityCheck] - BridgeAsyncCall - mPerformAndComputeIntegrityCheck Result = TEST PASSED
Average Camera Temperature :40.11911°C
Module 0
Nb Points: 50673 pts (>32500)
Noise:
AMD: 0.00449238 mm (<0.027)
STD DEV: 0.006961088 mm
Dead camera: false
Module 1
Nb Points: 53809 pts (>40000)
Noise:
AMD: 0.0055302843 mm (<0.027)
STD DEV: 0.00869096 mm
Dead camera: false
Module consistency
Weak module: false
M0 to M1
Distance: 0.007857603 mm (<0.015)
Angle: 0.022567615 degrees (<0.07)
Target
Position: 0.009392071 mm (<5.0)
Angle: 0.54686683 degrees (<5.0)
Intensity: 120.35959
08-31 16:32:20 INFO - [cIOScannerService RUNNING] - Scanner State is now Scan-Ready
The issue is that the line at the end of every log would be different as well as the log lines would differ so it is the only logical way to achieve the correct extraction is to match the first line which would always contain: " TEST " and then grab the log to the first timestamp appearance after or the empty line which also shows every time at the end of the log.
Just not sure how to achieve that and the code I have is returning no/empty matches, however if I echo $StartLine - it shows correctly the log starting lines.
You can match the first line that starts with a date time like format and contains TEST in the line. Then capture in group 1 all the content that does not start with a date time like format.
(?m)^\d{2}-\d{2} \d{2}:\d{2}:\d{2}.*\bTEST\b.*\r?\n((?:(?!\d{2}-\d{2} \d{2}:\d{2}:\d{2}).*(?:\r?\n|$))*)
Explanation
(?m) Inline modifier for multiline
^ Start of line
\d{2}-\d{2} \d{2}:\d{2}:\d{2}.*\bTEST\b.* Match a date time like pattern followed by TEST in the line
\r?\n Match a newline
( Capture group 1
(?: Non capture group
(?!\d{2}-\d{2} \d{2}:\d{2}:\d{2}).*(?:\r?\n|$) If the line does not start with a date time like pattern, match the whole line followed by either a newline or the end of the line
)* Close non capture group and repeat 0+ times
) Close group 1
See a regex101 demo and a .NET regex demo (click on the Table tab) and a powershell demo
You can use Get-Content -Raw to get the contents of a file as one string.
$textIOClient = Get-Content -Raw "$userProfilePath\DWIO\logs\IOClient.txt"
$pattern = "(?m)^\d{2}-\d{2} \d{2}:\d{2}:\d{2}.*\bTEST\b.*\r?\n((?:(?!\d{2}-\d{2} \d{2}:\d{2}:\d{2}).*(?:\r?\n|$))*)"
Select-String $pattern -input $textIOClient -AllMatches | Foreach-Object {$_.Matches} | Foreach-Object {$_.Groups[1].Value}
I found an approach I really loved in this answer elsewhere on the site:
PowerShell - Search String in text file and display until the next delimeter
Using that, I wrote a little code around it in the following to show you how to use the results:
$itemCount = 1
$Server = ""
$Data = #()
$Collection = #()
Switch(GC C:\temp\stackTestlog.txt){
{[String]::IsNullOrEmpty($Server) -and !([String]::IsNullOrWhiteSpace($_))}{$Server = $_;Continue}
{!([String]::IsNullOrEmpty($Server)) -and !([String]::IsNullOrEmpty($_))}{$Data+="`n$_";Continue}
{[String]::IsNullOrEmpty($_)}{$Collection+=[PSCustomObject]#{Server=$Server;Data=$Data};Remove-Variable Server; $Data=#()}
}
If(!([String]::IsNullOrEmpty($Server))){$Collection+=[PSCustomObject]#{Server=$Server;Data=$Data};Remove-Variable Server; $Data=#()}
if(($null -eq $collection) -or ($Collection.Count -eq 0)){
Write-Warning "Could not parse file"
}
else{
Write-Output "Found $($collection.Count) members"
ForEach($item in $Collection){
#add additional code here if you need to do something with each parsed log entry
Write-Output "Item # $itemCount $($item.Server) records"
Write-Host $item.Data -ForegroundColor Cyan
$itemCount++
}
}
You can extend this in the line with a comment, and then remove the Write-output and Write-Host lines too.
Here's what it looks like in action.
Found 2 members
Item #1 08-31 16:32:20 INFO - [IOBridgeThread - mPerformAndComputeIntegrityCheck] - BridgeAsyncCall - mPerformAndCompu
teIntegrityCheck Result = TEST PASSED records
Average Camera Temperature :40.11911°C
#abridged...
Item #2 blahblahblah

Combining Multiple String Commands Into One Line

I'm using PowerShell and running a tool to extract Lenovo hardware RAID controller info to identify the controller number for use later on in another command line (this is part of a SCCM Server Build Task Sequence). The tool outputs a lot of data and I'm trying to isolate just what I need from the output.
I've been able to isolate what I need, but I'm thinking there has to be a more efficient way so looking for optimizations. I'm still learning when it comes to working with strings.
The line output from the tool that I'm looking for looks like this:
0 0 0 252:0 17 DRIVE Onln N 557.861 GB dsbl N N dflt -
I'm trying to get the 3 characters to the left of the :0 (the 252 but on other models this could be 65 or some other 2 or 3 digit number)
My existing code is:
$ControllerInfo = cmd /c '<path>\storcli64.exe /c0 show'
$forEach ($line in $ControllerInfo) {
if ($line -like '*:0 *') {
$ControllerNum = $line.split(':')[0] # Get everything left of :
$ControllerNum = $ControllerNum.Substring($ControllerNum.Length -3) # Get last 3 chars of string
$ControllerNum = $ControllerNum.Replace(' ', '') # Remove blanks
Write-Host $ControllerNum
break #stop looping through output
}
}
The above works but I'm wondering if there's a way to combine the three lines that start with $ControllerNum = so I can have just have a single $ControllerNum = (commands) line to set the variable instead of doing it in 3 lines. Basically want to combine the Split, Substring and Replace commands into a single line.
Thanks!
Here's another option:
$ControllerNum = ([regex]'(\d{2,3}):0').Match($line).Groups[1].Value
Used on your sample 0 0 0 252:0 17 DRIVE Onln N 557.861 GB dsbl N N dflt -
the result in $ControllerNum wil be 252
If you want just the last digits before the first :, without any whitespace, you can do that with one or two regex expressions:
$line -replace '^.*\b(\d+):.*$','$1'
Regex explanation:
^ # start of string
.* # any number of any characters
\b # word boundary
( # start capture group
\d+ # 1 or more strings
) # end capture group
: # a literal colon (:)
.* # any number of any characters
$ # end of string
replacement:
$1 # Value captured in the capture group above

powershell extracting data from strings or other suggestions

I have a script I am writing that essentially reads data from an excel document that is generated from another tool. It lists file ages in the format listed below. My issue is I would like to process each cell value and change the cell color based on that value. So anything older than 1 year gets changed to RED, 90+ days gets yellow\orange.
So after a bit of research, I elected to use an if statement to determine when it is greater than 0 years which seems to work fine, however when I reach the days portion I'm not sure how to extract JUST the digits portion to the left of d in each cell when you get to the y if its there just stop OR possibly just read the left digits only if the $_ contains d then I could further process if that value is -gt 90? I am unsure of how to extract variable length strings only if they are digits left of a character. I considered using a combination of the below method of finding a character and returning up to y or something else.
Find character position and update file name
Possible Age Formats:
13y170d
3y249d
8h7m
1y109d
1y109d
1y109d
5d22h
3y281d
3y184d
11y263d
7m25s
1h14m
[regex]$years = "\d{1,3}[0-9]y"
[regex]$days_90 = "\d{0,3}[0-9]d"
conditionally formatting/coloring row based on age (years)
if ( $( A$_ -match "$years") -eq $True ) {
$($test_home).$("Last Accessed") | ForEach-Object { $( $($_.Contains("y") -eq $True ) { New-ConditionalText -Text Red } }
conditionally formatting/coloring row based on age (90+ days)
if ( $( A$_ -match "$days_90") -eq $True ) { New-ConditionalText -Text Yellow }
What you are after is a positive lookahead and lookbehind. Effectivly it gets the text between two characters or sets. Really handy if you have a consistently formatted set of data to work with.
[regex]$days_90 = '(?<=y).*?(?=d)'
. Matches any characters without line breaks.
* Matches 0 or more of the preceding token.
? Makes the regex lazy and try to match as few as possible.

PowerShell : Spliting text into two (or more) lines

I just finished working on menu generator, but I need to add warning message or popup like this one :
the warning message, like the other elements of menu, should have same width of 80 (inner width is 78, because border takes two). I built a function with one parameter which can easy generate that message for the text that will be displayed. The problem is when I put text longer than 78 characters, I get errors. I want to split it into two (or more depending on how many lines we would get) parameters cause no one will count to 78 on each parameter/line. I'm looking for a possibility to split text into two or more lines to fit the inner width of 78.
Since this time I decided to split text with " "(space) separator
$textsplit = $text.Split(" ")
Then I decided to add each element of $textsplit to an array using a Do-Until loop
$warningmsgline1.Add($textsplit[$i])
$warningmsgline1.Add(" ")
to make a new variable that will contain a sentence (words) that contains less then 78 characters.
I hope you are keeping up :)
How can I create such a condition? Nested loops? What kind of loops? For? Do-Until?
Feel free to ask if something is unclear.
I am a little fuzzy on understanding your question. But if I understand you correctly, you want to handle line wrapping at a max of 78 characters? This is not complete, but should set you in the right direction.
$message = "This is a really long message that is longer than 80 characters. It will need to be wrapped onto a second line."
$wordArray = $message.Split(' ')
$output = #()
$currentLine = ""
$wordArray | ForEach-Object {
if ($currentLine.Length + 1 + $_.Length -le 78) {
$currentLine = "$currentLine $_"
}
else {
$output += $currentLine
$currentLine = $_
}
}
## Add remainder to output
$output += $currentLine
$output

Brainfuck challenge

I have a any challenge. I must write brainfuck-code.
For a given number n appoint its last digit .
entrance
Input will consist of only one line in which there is only one integer n ( 1 < = n < = 2,000,000,000 ) , followed by a newline ' \ n' (ASCII 10).
exit
On the output has to find exactly one integer denoting the last digit of n .
example I
entrance: 32
exit: 2
example II:
entrance: 231231132
exit: 2
This is what I tried, but it didn't work:
+[>,]<.>++++++++++.
The last input is the newline. So you have to go two memory positions back to get the last digit of the number. And maybe you don't have to return a newline character, so the code is
,[>,]<<.
Nope sorry, real answer is
,[>,]<.
because your answer was getting one too far ;)
Depending on the interpreter, you might have to escape the return key by yourself. considering the return key is ASCII: 10, your code should look like this :
>,----- -----[+++++ +++++>,----- -----]<.
broken down :
> | //first operation (just in case your interpreter does not
support a negative pointer index)
,----- ----- | //first entry if it's a return; you don't even get in the loop
[
+++++ +++++ | //if the value was not ASCII 10; you want the original value back
>, | //every next entry
----- ----- | //check again for the the return,
you exit the loop only if the last entered value is 10
]
<. | //your current pointer is 0; you go back to the last valid entry
and you display it
Your issue is that a loop continues for forever until at the end of the loop the cell the pointer is currently on in equal to 0. Your code never prints in the loop, and never subtracts, so your loop will never end, and all that your code does is take an ASCII character as input, move one forward, take an ASCII character as input, and so on. All of your code after the end of the loop is useless, because that your loop will never end.