Finding entries in a log file greater than a defined time - powershell

My company needs to analyse log files of a Tomcat to check for specific errors and uses Powershell. Those errors will be stored in an array and checked against 1:1. This happens every 30 minutes by using Windows Task Scheduler. In case such an error is found in the log file, a generated text file will be sent to the administrators.
However it is only of interest to check for errors during the last 30 minutes, not beforehand.
So I have defined first a variable for the time:
$logTimeStart = (Get-Date).AddMinutes(-30).ToString("yyyy-MM-dd HH:mm:ss")
Later on I check for the existence of such an error:
$request = Get-Content ($logFile) | select -last 100 | where { $_ -match $errorName -and $_ -gt $logTimeStart }
Unfortunately this does not work; it also sends errors happened before this interval of 30 minutes.
Here is an extract of the Tomcat log:
2016-05-25 14:21:30,669 FATAL [http-apr-8080-exec-4] admins#company.de de.abc.def.business.service.ExplorerService GH00000476:
de.abc.def.business.VisjBusinessException: Invalid InstanceId
at de.abc.def.business.service.ExplorerService$ExplorerServiceStatic.getExplorer(ExplorerService.java:721)
at de.abc.def.business.service.ExplorerService$ExplorerServiceStatic.getTreeItemList(ExplorerService.java:823)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at de.abc.def.business.provider.ServiceProvider.callServiceMethod(ServiceProvider.java:258)
at de.abc.def.business.communication.web.client.ServiceDirectWrapperDelegate.callServiceMethod(ServiceDirectWrapperDelegate.java:119)
at de.abc.def.business.communication.web.client.ServiceWrapperBase.callServiceMethod(ServiceWrapperBase.java:196)
at de.abc.def.business.communication.web.client.ServiceDirectWrapper.callServiceMethod(ServiceDirectWrapper.java:24)
at de.abc.def.web.app.service.stub.AbstractBaseStub.callServiceMethodDirect(AbstractBaseStub.java:72)
at de.abc.def.web.app.service.stub.AbstractBaseStub.callServiceMethod(AbstractBaseStub.java:183)
at de.abc.def.web.app.service.stub.StubSrvExplorer.getTreeItemList(StubSrvExplorer.java:135)
at de.abc.def.web.app.resource.servlet.ExplorerServlet.createXml(ExplorerServlet.java:350)
at de.abc.def.web.app.resource.servlet.ExplorerServlet.callExplorerServlet(ExplorerServlet.java:101)
at de.abc.def.web.app.resource.servlet.VisServlet.handleServlets(VisServlet.java:244)
at de.abc.def.web.app.FlowControlAction.isPing(FlowControlAction.java:148)
at de.abc.def.web.app.FlowControlAction.execute(FlowControlAction.java:101)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
Unfortunately one cannot say how many lines of such an error will show up. Therefore 100 is just an estimate (which works well).
So how to change the related line
$request = Get-Content ($logFile) | select -last 100 |
where { $_ -match $errorName -and $_ -gt $logTimeStart }
to a correct one?

Use Select-String and a regular expression to parse your log file. Basically a log entry consists of a timestamp, the severity, and a message (which may span several lines). A regular expression for that might look like this:
(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\s+(\w+)\s+(.*(?:\n\D.*)*(?:\n\t.*)*)
(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) matches the timestamp.
(\w+) matches the severity.
(.*(?:\n\D.*)*) matches the log message (the current line followed by zero or more lines not beginning with a number).
The parentheses around each subexpression capture the submatch as a group that can then be used for populating the properties of custom objects.
$datefmt = 'yyyy-MM-dd HH:mm:ss,FFF'
$culture = [Globalization.CultureInfo]::InvariantCulture
$pattern = '(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\s+(\w+)\s+(.*(?:\r\n\D.*)*)'
$file = 'C:\path\to\your.log'
Get-Content $file -Raw | Select-String $pattern -AllMatches | ForEach-Object {
$_.Matches | ForEach-Object {
New-Object -Type PSObject -Property #{
Timestamp = [DateTime]::ParseExact($_.Groups[1].Value, $datefmt, $culture)
Severity = $_.Groups[2].Value
Message = $_.Groups[3].Value
}
}
}
Parsing the date substring into a DateTime value isn't actually required (since date strings in ISO format can be sorted properly even with string comparisons), but it's nice to have so you don't have to convert your reference timestamp to a formatted string.
Note that you need to read the entire log file as a single string for this to work. In PowerShell v3 and newer this can be achieved by calling Get-Content with the parameter -Raw. On earlier versions you can pipe the output of Get-Content through the Out-String cmdlet to get the same result:
Get-Content $file | Out-String | ...

Related

Powershell Files fetch

Am looking for some help to create a PowerShell script.
I have a folder where I have lots of files, I need only those file that has below two content inside it:
must have any matching string pattern as same as in file file1 (the content of file 1 is -IND 23042528525 or INDE 573626236 or DSE3523623 it can be more strings like this)
also have date inside the file in between 03152022 and 03312022 in the format mmddyyyy.
file could be old so nothing to do with creation time.
then save the result in csv containing the path of the file which fulfill above to conditions.
Currently am using the below command that only gives me the file which fulfilling the 1 condition.
$table = Get-Content C:\Users\username\Downloads\ISIN.txt
Get-ChildItem `
-Path E:\data\PROD\server\InOut\Backup\*.txt `
-Recurse |
Select-String -Pattern ($table)|
Export-Csv C:\Users\username\Downloads\File_Name.csv -NoTypeInformation
To test if a file contains a certain keyword from a range of keywords, you can use regex for that. If you also want to find at least one valid date in format 'MMddyyyy' in that file, you need to do some extra work.
Try below:
# read the keywords from the file. Ensure special characters are escaped and join them with '|' (regex 'OR')
$keywords = (Get-Content -Path 'C:\Users\username\Downloads\ISIN.txt' | ForEach-Object {[regex]::Escape($_)}) -join '|'
# create a regex to capture the date pattern (8 consecutive digits)
$dateRegex = [regex]'\b(\d{8})\b' # \b means word boundary
# and a datetime variable to test if a found date is valid
$testDate = Get-Date
# set two variables to the start and end date of your range (dates only, times set to 00:00:00)
$rangeStart = (Get-Date).AddDays(1).Date # tomorrow
$rangeEnd = [DateTime]::new($rangeStart.Year, $rangeStart.Month, 1).AddMonths(1).AddDays(-1) # end of the month
# find all .txt files and loop through. Capture the output in variable $result
$result = Get-ChildItem -Path 'E:\data\PROD\server\InOut\Backup'-Filter '*.txt'-File -Recurse |
ForEach-Object {
$content = Get-Content -Path $_.FullName -Raw
# first check if any of the keywords can be found
if ($content -match $keywords) {
# now check if a valid date pattern 'MMddyyyy' can be found as well
$dateFound = $false
$match = $dateRegex.Match($content)
while ($match.Success -and !$dateFound) {
# we found a matching pattern. Test if this is a valid date and if so
# set the $dateFound flag to $true and exit the while loop
if ([datetime]::TryParseExact($match.Groups[1].Value,
'MMddyyyy',[CultureInfo]::InvariantCulture,
[System.Globalization.DateTimeStyles]::None,
[ref]$testDate)) {
# check if the found date is in the set range
# this tests INCLUDING the start and end dates
$dateFound = ($testDate -ge $rangeStart -and $testDate -le $rangeEnd)
}
$match = $match.NextMatch()
}
# finally, if we also successfully found a date pattern, output the file
if ($dateFound) { $_.FullName }
elseif ($content -match '\bUNKNOWN\b') {
# here you output again, because unknown was found instead of a valid date in range
$_.FullName
}
}
}
# result is now either empty or a list of file fullnames
$result | set-content -Path 'C:\Users\username\Downloads\MatchedFiles.txt'

Powershell Select-String

I need your help with PowerShell.
I need Select-String with fixed Date (in variable). & Set-Content to result.txt
Example: $Date = "01.07.2020"
But also i need select string with date which lower than i written in variable.
My code: Get-Content -Path log.txt | Select-String "?????" | Set-Content $result.txt
In log.txt i have many strings like " Creation date 01.07.2020 " ; " Creation date 01.06.2020 "
123.txt
Creation date 01.07.2020
Creation date 02.05.2020
Creation date 01.06.2020
Creation date 28.08.2020
Example script
$file = "C:\Users\userprofile\Desktop\test\123.txt"
$regexpattern = "\d{2}\.\d{2}\.\d{4}"
$content = Get-Content $file | Where-object { $_ -match $regexpattern}
foreach($line in $content){
$line.Substring(13,11)
}
I used regex to find the lines you are wanting to output. We get the content only if it matches our regex, then for each line we found, I'm using substring to pull the date out. You could also put together a regex for this if you wanted to. Since we know the lines have the same number of characters it's safe to use the substring function.
If you want that output to a file, simply find $line.Substring(13,11) and then add this after it | Out-file "C:\Users\userprofile\desktop\test\output.txt" -append.

PS1 Replace or Remove a single character without rewriting the whole file

Is it possible to remove or replace the last character on the last non-whitespace line of a file using PowerShell 1?
I'm trying to get an Uptime log that is precise to within 5 minutes.
I've found that there are built logs and commands that can be accessed through command prompt that would tell me when the last time a computer was booted up, or when it shut down correctly, but the native uptime log only records once every 24 hrs, so if there is a power failure, I won't know how long the system has been offline with any precision more refined than 24 hours.
So I have created the following script:
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$uptime = (Get-Date)-$formBootTime
"$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00}" -f $uptime.Days,$uptime.Hours,$uptime.Minutes,$uptime.Seconds >> C:\UptimeTracker.csv
However, this gets tediously long to scroll through when I want to evaluate how long my machine has been running over the last X days.
So I thought I would add a marker to identify the current or most recent Uptime log per any given Boot.
But in order for that to work I would need to be able to remove said marker as soon as the previous record is no longer the relevant record.
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$file = (Get-Content c:\UptimeTracker.csv | Measure-Object)
$numberOfLines = $file.Count
$numberOfWords = (Get-Content c:\UptimeTracker.csv | Select -Index ($numberOfLines -1) | Measure-Object -word)
$Line = Get-Content c:\UptimeTracker.csv | Select -Index ($numberOfLines -2)
$wordArray = $Line.Split(",")
$LastLineBT = $wordArray[0]
if($LastLineBT -eq $formBootTime) {
$unmark = "true"
}
else
{$unmark = "false"}
if($unmark == "true"){ <remove last character of file> }
$uptime = (Get-Date)-$formBootTime
"$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00},X" -f $uptime.Days,$uptime.Hours,$uptime.Minutes, $uptime.Seconds >> C:\UptimeTracker.csv
Some of the above is borrowed and modified from: https://stackoverflow.com/a/16210970/11035837
I have seen several methods that receive the file as the input file and write to a different output file, and from there it would be an easy thing to do to script renaming the new and old files to switch their positions (new, old, standby - and rotate) the reason I'm trying not to rewrite the whole file is to reduce those instances where the command/script is interrupted and the action doesn't complete. Ideally the only time the action doesn't complete would be on a power failure. However, I have already seen in a previous version, it would skip 5 minute intervals occasionally for up to 15 minutes without any change in the last reported boot time. I suspect this has to do with other higher priority processes preventing the task scheduler from running the script. If this is the case, then a complete rewrite of the file failing part way through the script would lose some percentage of the existing log data, and I would rather miss the latest record than all the data.
Nothing I have found indicates any ability to remove/replace the last character (or two since one is a newline char), neither have I found anything that explicitly declares this is not possible - I have found declarations that it is not possible to elective replace inner or beginning content without a complete rewrite.
Barring any solution definitive answer, or if the definitive answer is no this cannot be done, then I will attempt something like the following:
if($unmark == "true"){
$input = "C:\UptimeTracker_CUR.csv"
$output = "C:\UptimeTracker_NEW.csv"
$content = Get-Content $input
$content[-2] = $content[-2] -replace 'X', ' '
$content | Set-Content $output
Rename-Item -Path "C:\UptimeTracker_CUR.csv" -NewName "C:\UptimeTracker_SBY.csv"
Rename-Item -Path "C:\UptimeTracker_NEW.csv" -NewName "C:\UptimeTracker_CUR.csv"
}
EDIT - due to multi-read comment by TheMadTechnician
...
$file = Get-Content c:\UptimeTracker.csv
$fileMeasure = ($file | Measure-Object)
$numberOfLines = $fileMeasure.Count
$numberOfWords = ($file | Select -Index ($numberOfLines -1) | Measure-Object -word)
$Line = $file | Select -Index ($numberOfLines -2)
...
...
if($unmark == "true"){
$output = "C:\UptimeTracker_NEW.csv"
$file[-2] = $file[-2] -replace 'X', ' '
$file | Set-Content $output
Rename-Item -Path "C:\UptimeTracker.csv" -NewName "C:\UptimeTracker_SBY.csv"
Rename-Item -Path "C:\UptimeTracker_NEW.csv" -NewName "C:\UptimeTracker.csv"
}
You read the whole file in several times, which has got to be slowing the whole script down. I would suggest reading the whole file in, determining if you need to clear your flag, then do so when you output, adding your new line to the file. Assuming you aren't still running PowerShell v2, you can do this:
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$uptime = (Get-Date)-$formBootTime
$File = Get-Content c:\UptimeTracker.csv -raw
if($File.trim().split("`n")[-1].Split(',')[0] -eq $formBootTime){
$File.trim() -replace 'X(?=\s*$)',' '
}else{
$File.Trim()
},("$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00},X" -f $uptime.Days,$uptime.Hours,$uptime.Minutes, $uptime.Seconds)|Set-Content c:\UptimeTracker.csv
If you are running an old version you will not have the -raw option for Get-Content. As a work around you can do this instead, and the same solution should still work.
$bootTime = (Get-WmiObject Win32_OperatingSystem).LastBootUpTime
$formBootTime = [Management.ManagementDateTimeConverter]::ToDateTime($bootTime)
$uptime = (Get-Date)-$formBootTime
$File = (Get-Content c:\UptimeTracker.csv) -Join "`n"
if($File.trim().split("`n")[-1].Split(',')[0] -eq $formBootTime){
$File.trim() -replace 'X(?=\s*$)',' '
}else{
$File.Trim()
},("$formBootTime,$(Get-Date),{0:00},{1:00},{2:00},{3:00},X" -f $uptime.Days,$uptime.Hours,$uptime.Minutes, $uptime.Seconds)|Set-Content c:\UptimeTracker.csv
This is going to be slower, so should be considered a secondary option since you'll have to read the whole file in as an array of strings, and convert it to a single multi-line string.

Linking two array variables in one line

Right now, I have an array called $vlog that shows the most recent instance of each log file in a certain folder $logpathfor each version of a particular version of a service $verno running on a machine. It displays the filename and the last write time.
$VLog = foreach ($log in $verno) {
Get-ChildItem $LogPath |
Where-Object {$_.Name -like "*$log-*"} |
Select-Object -last 1 | Select-Object Name, LastWriteTime
}
Furthermore, I'm using a mix of Trim() and -replace to cut most of the name down to ONLY show the corresponding version number contained within the log file name.
$vname = $vlog.name.Trim('service.logfile.prefix-') -replace ".{9}$"
Basically, it's cutting the end of the file (containing a date and the extension .log), as well as a specific preceding bit of text that's a constant at the beginning of each file. This all works great.
Now here's the trouble.
I need to get two items from this logfile data ($vname, which is the version number), and the corresponding LastWriteTime for the logfile for THAT specific version. I need that as an array, which will be further put into a variable for an email
What I need it to say is Service instance version #VERSION# was last logged at #VERSION'S LAST WRITE TIME#.
This is what I have right now:
$mailvar = foreach ($version in $vname) {
"Service instance version $version was last logged at $PLEASEHELP."
}
I can get it to display the correct number of instances of that sentence, with the correct version number per sentence, but I can't get the time to display the last write time corresponding to that version number.
I believe what you're looking for is the FileInfo property LastWriteTime. Here's your example simplified:
foreach ($log in $verno) {
$file = (Get-ChildItem -Path $LogPath -Filter "*$log-*")[-1]
$version = $file.Name.Trim('service.logfile.prefix-') -replace '.{9}$'
$stamp = $file.LastWriteTime | Get-Date -UFormat '%Y-%m-%d %H:%M:%S'
'Service instance version {0} was last logged at {1}' -f $version, $stamp
}

Export-CSV cmdlet rewriting entire CSV during each iteration of a FOREACH statement

I'm working with some code that is going to take a series of performance counters, and then put the counters in a .csv file that rolls over every time it hits 1MB.
$Folder="C:\Perflogs\BBCRMLogs" # Change the bit in the quotation marks to whatever directory you want the log file stored in
$Computer = $env:COMPUTERNAME
$1GBInBytes = 1GB
$p = LOTS OF COUNTERS;
# If you want to change the performance counters, change the above list. However, these are the recommended counters for a client machine.
$num = 0
$file = "$Folder\SQL_log_${num}.csv"
if( !(test-path $folder)) {New-Item $Folder -type directory}
Get-Counter -counter $p -SampleInterval 2 -Continuous | Foreach {
if ((Get-Item $file -ErrorAction SilentlyContinue ).Length -gt 1mb)
{
$num +=1
$file = "$Folder\SQL_log_${num}.csv"
}
$_
} | Foreach-Object { $_ | Export-Csv $file -Force -Append}
Right now, it's working quite well. The iteration works fine, and it does create a new file each time the .csv reaches 1MB. However, each .CSV after the first is being created after 2 minutes already at 1MB, causing a new file to be created. I'm not quite sure why this is occurring, although I believe it's because Powershell is just rewriting the entirety of the .csv each time it creates it.
[I'm posting this as a new answer rather than editing the original because it's completely different. Replacing or appending to the original answer would make the ensuing discussion confusing.]
What you need to do is use a regex to extract the values from the Readings property of the output of Get-Counter, and manually construct CSV output from the timestamp and those values. Change the last line to this (format according to your preferred style):
| %{'"' + (Get-Date $_.Timestamp -f 's') + '","' + (([regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') | select -ExpandProperty Value) -join '","') + '"'} | Out-File $file -Append -Encoding ASCII
To break that down:
(Get-Date $_.Timestamp -f 's') This part is not strictly necessary, though I think it will make your results easier to follow. The 's' format puts the date in an ISO 8601 sortable pattern. You could substitute 'u' for another sortable format, or use your favorite custom format string. Or just replace it with $_.Timestamp to retain the original format.
[regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') The regex matches the contents of any line that is preceded by a line that begins with \\ and ends with : (those pesky counter names you wanted to get rid of). Note that I'm using [regex]::matches, which performs a global match, as opposed to [regex]::match or -match, which will just give you the first match for each string (the Readings property is a single string, so only the first counter reading would be returned).
| select -ExpandProperty Value Produces an array of all the matches, which you can then join with "," and surround with "'s to produce CSV output.
Since you're not using a conversion function, you also need to construct a header row. Add this line right above the pipeline:
`'"Timestamp","' + ($p -join '","') + '"' | Out-File $file -Append -Encoding ASCII`
That's assuming that $p is an array (which it should be). If it's a string, then depending on the format you can either use it as-is, or -split it and rejoin it in CSV format.
Change the last line to this, to convert each line to CSV format and then append it to the output file:
} | Foreach-Object {($_ | ConvertTo-Csv -NoTypeInformation)[1] | Out-File $file -Append -Encoding ASCII}.
A few notes:
The -Encoding ASCII is not strictly necessary, but you might have trouble with a Unicode CSV file in some applications (Excel, for example, won't open it as a CSV file by default, and everything will be in Column A)
The reason for the index in ($_ | ConvertTo-Csv -NoTypeInformation)[1] is that ConvertTo-Csv -NoTypeInformation still outputs the header row each time, so you want to grab the second line of the two-line output (($_ | ConvertTo-Csv -NoTypeInformation)[0] is the header row)
Since you're not outputting a header row, you'll need to output one to $file before the loop