Finding entries in a log file greater than a defined time - powershell

Question

My company needs to analyse log files of a Tomcat to check for specific errors and uses Powershell. Those errors will be stored in an array and checked against 1:1. This happens every 30 minutes by using Windows Task Scheduler. In case such an error is found in the log file, a generated text file will be sent to the administrators.
However it is only of interest to check for errors during the last 30 minutes, not beforehand.
So I have defined first a variable for the time:
$logTimeStart = (Get-Date).AddMinutes(-30).ToString("yyyy-MM-dd HH:mm:ss")
Later on I check for the existence of such an error:
$request = Get-Content ($logFile) | select -last 100 | where { $_ -match $errorName -and $_ -gt $logTimeStart }
Unfortunately this does not work; it also sends errors happened before this interval of 30 minutes.
Here is an extract of the Tomcat log:
2016-05-25 14:21:30,669 FATAL [http-apr-8080-exec-4] admins#company.de de.abc.def.business.service.ExplorerService GH00000476:
de.abc.def.business.VisjBusinessException: Invalid InstanceId
at de.abc.def.business.service.ExplorerService$ExplorerServiceStatic.getExplorer(ExplorerService.java:721)
at de.abc.def.business.service.ExplorerService$ExplorerServiceStatic.getTreeItemList(ExplorerService.java:823)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at de.abc.def.business.provider.ServiceProvider.callServiceMethod(ServiceProvider.java:258)
at de.abc.def.business.communication.web.client.ServiceDirectWrapperDelegate.callServiceMethod(ServiceDirectWrapperDelegate.java:119)
at de.abc.def.business.communication.web.client.ServiceWrapperBase.callServiceMethod(ServiceWrapperBase.java:196)
at de.abc.def.business.communication.web.client.ServiceDirectWrapper.callServiceMethod(ServiceDirectWrapper.java:24)
at de.abc.def.web.app.service.stub.AbstractBaseStub.callServiceMethodDirect(AbstractBaseStub.java:72)
at de.abc.def.web.app.service.stub.AbstractBaseStub.callServiceMethod(AbstractBaseStub.java:183)
at de.abc.def.web.app.service.stub.StubSrvExplorer.getTreeItemList(StubSrvExplorer.java:135)
at de.abc.def.web.app.resource.servlet.ExplorerServlet.createXml(ExplorerServlet.java:350)
at de.abc.def.web.app.resource.servlet.ExplorerServlet.callExplorerServlet(ExplorerServlet.java:101)
at de.abc.def.web.app.resource.servlet.VisServlet.handleServlets(VisServlet.java:244)
at de.abc.def.web.app.FlowControlAction.isPing(FlowControlAction.java:148)
at de.abc.def.web.app.FlowControlAction.execute(FlowControlAction.java:101)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:484)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
Unfortunately one cannot say how many lines of such an error will show up. Therefore 100 is just an estimate (which works well).
So how to change the related line
$request = Get-Content ($logFile) | select -last 100 |
where { $_ -match $errorName -and $_ -gt $logTimeStart }
to a correct one?

Answer 1

Use Select-String and a regular expression to parse your log file. Basically a log entry consists of a timestamp, the severity, and a message (which may span several lines). A regular expression for that might look like this:
(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\s+(\w+)\s+(.*(?:\n\D.*)*(?:\n\t.*)*)
(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) matches the timestamp.
(\w+) matches the severity.
(.*(?:\n\D.*)*) matches the log message (the current line followed by zero or more lines not beginning with a number).
The parentheses around each subexpression capture the submatch as a group that can then be used for populating the properties of custom objects.
$datefmt = 'yyyy-MM-dd HH:mm:ss,FFF'
$culture = [Globalization.CultureInfo]::InvariantCulture
$pattern = '(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})\s+(\w+)\s+(.*(?:\r\n\D.*)*)'
$file = 'C:\path\to\your.log'
Get-Content $file -Raw | Select-String $pattern -AllMatches | ForEach-Object {
$_.Matches | ForEach-Object {
New-Object -Type PSObject -Property #{
Timestamp = [DateTime]::ParseExact($_.Groups[1].Value, $datefmt, $culture)
Severity = $_.Groups[2].Value
Message = $_.Groups[3].Value
}
}
}
Parsing the date substring into a DateTime value isn't actually required (since date strings in ISO format can be sorted properly even with string comparisons), but it's nice to have so you don't have to convert your reference timestamp to a formatted string.
Note that you need to read the entire log file as a single string for this to work. In PowerShell v3 and newer this can be achieved by calling Get-Content with the parameter -Raw. On earlier versions you can pipe the output of Get-Content through the Out-String cmdlet to get the same result:
Get-Content $file | Out-String | ...