I have numerous .txt files that are output from a handle.exe run of several days. I need to reorganize the data to get it into a relational database. The first thing I need to do is get the dates re-formatted.
Each file has in excess of 800 dates, disbursed unevenly throughout the file. The dates are formatted:
June 29, 2016 12:05:45 PM and I need 06-29-16 12:05:45.
I'm just working on a single file for now, to get things dialed in. I've tried to replace the dates in situ (using an array for the original dates) with Get-Date and got nowhere. Then I tried -replace and that didn't work.
I've spent 3 or 4 days on this and I think I've broken my head. I've tried so many permutations of stuff that I don't know even where I am anymore.
The last thing I tried was below. An attempt to use a hashtable, with the old date and new date in the table.
##To set "|" as separator for arrays
$OFS = '|'
##To get original dates into array
$a = #(sls .\hp.txt -pattern '(june 29|june 30|july 1|july 2|july 3|july 4)' | select -ExpandProperty line)
##To get dates with corrected format into array
$b = #($a | foreach {$_ | Get-Date -Format "MM-dd-yy hh:mm:ss"})
##To get old and new dates into hash table
$dates = #{$a = $b}
##To bring in content from file
$file = (Get-Content C:\hp.txt)
##To replace "NAME" with "VALUE" from hash table into file
foreach ($d in $dates) {
$file = $file -replace $d.Name, $d.Value
}
##To save corrected file with new file name
Set-Content -Path C:\hpnew.txt -Value $file
The $a array contains (in small part):
June 29, 2016 12:04:51 PM
June 29, 2016 12:05:58 PM
June 29, 2016 12:07:00 PM
[NOTE: LOTS MORE DATES HERE]
June 30, 2016 12:01:17 AM
June 30, 2016 12:02:19 AM
June 30, 2016 12:04:22 AM
[NOTE:CONTINUING TO END]
The $b array contains:
06-29-16 12:04:51
06-29-16 12:05:58
06-29-16 12:07:00
[NOTE: LOTS MORE DATES ]
06-30-16 12:01:17
06-30-16 12:02:19
06-30-16 12:04:22
[NOTE: CONTINUING TO END]
There is probably a MUCH simpler, more elegant solution. But any help/direction would be great.
Use a regular expression to extract the date strings from your text, then pass the matches to a callback function where you parse them to actual DateTime values and format those according to your requirements:
$re = '((?:january|february|...|december) \d{1,2}, \d{4} \d{1,2}:\d{2}:\d{2} [ap]m)'
$input_fmt = 'MMMM d, yyyy h:mm:ss tt'
$output_fmt = 'MM-dd-yy HH:mm:ss'
$culture = [Globalization.CultureInfo]::InvariantCulture
$options = [Text.RegularExpressions.RegexOptions]::IgnoreCase
$callback = {
[DateTime]::ParseExact($args[0].Groups[1].Value, $input_fmt, $culture).ToString($output_fmt)
}
$txt = Get-Content '.\hp.txt' -Raw
[regex]::Replace($txt, $re, $callback, $options) | Set-Content '.\hpnew.txt'
Related
I would like to get date from the log file text.
Text in log file.
Error code. 200105. Simple text and so on -------------> it should get date as 2020 Jan 05
Error code. 2000207. Simple text and so on -------------> it should get date as 2020 Feb 07
I try this but it doesnt work.
Get-Date "200105" -format "y-m-d" but it doesnt work.
I also try "200105" | Date but still same issue
This does work [datetime]::ParseExact("120105", "y.m.d", $null) but how do I get just the date but ignore all of the other text
If you want a shorter version you can do that by piping the output as follows
$date = [datetime]::ParseExact($text2, "y.M.d", $null)
$date | Get-Date -Format dd-MMMM-yyyy
Or
$date.ToString("yyyy MMMM dd")
Your second example 2000207 is invalid because of the extra 0 in there.
I would use the TryParseExact method here to see if what you have got is actually a parsable datetime string.
$logLine = 'Error code. 200105. Simple text and so on'
if ($logLine -match '^Error code\s*\.?\s*(\d{6})') {
$date = Get-Date # any valid DateTime object will do
if ([datetime]::TryParseExact($Matches[1], 'yyMMdd', [cultureinfo]::InvariantCulture, 0, [ref]$date)) {
# do something with the date found. For demo, just output in the console
"Found a date: $date"
}
}
You are probably reading the log file line-by-line, something like:
Get-Content -Path 'TheLogFile' | ForEach-Object {
if ($_ -match '^Error code\s*\.?\s*(\d{6})') {
$date = Get-Date # any valid DateTime object will do
if ([datetime]::TryParseExact($Matches[1], 'yyMMdd', [cultureinfo]::InvariantCulture, 0, [ref]$date)) {
# do something with the date found. For demo, just output in the console
"Found a date: $date"
}
}
}
According to the documentation, Get-Date converts a string to a date if it recognises the date format from the locale settings.
For instance, in UK it recognises Get-Date "2020/03/21" but not Get-Date "20200321"
The format string is only used for formatting the current date.
This works: the number of characters in the format string represents the size of the input (it matches the number of digits in the day and year - it is more complicated for months) and M represents months (m represents minutes).
PS /home/alistair> [datetime]::ParseExact("200321","yyMMdd",$null)
Saturday, 21 March 2020 00:00:00
I have a log file that I have to scan every 5 minutes for specific keywords - specifically "Order Failed" then capture the lines around that. I have that part programmed with no issues. The rest of the log entries don't matter. I'm able to use the command:
[string] $readFile = get-content c:\test\friday.log | select-String -pattern '(order.*failed)' -context 1,7
which outputs:
Message: Order submission failed.
Timestamp: 4/1/2016 4:05:09 PM
Severity: Error
Message: Modifier not authorized for parent item
Message: Order submission failed.
Timestamp: 4/1/2016 4:18:15 PM
Severity: Error
Message: Modifier not authorized for parent item
Which is exactly what I want. My problem is trying to read through the above output and store the "Timestamp" into a variable that I can manipulate.
The first challenge is the "Timestamp" time is written in UTC time and we are located in the Pacific Time Zone.
The second challenge is I need to compare the "Timestamp" time to the current time and store that as an integer. We only want to report errors that are within 5 minutes of the current time.
My current code only grabs the first "Timestamp" entry since I hard-coded it:
[string] $readFile = get-content c:\test\friday.log | select-String -pattern '(order.*failed)' -context 1,7
$fileArray = $readFile.Split(“`n”)
[dateTime] $TrimTime = $fileArray[3].trim("Timestamp: ")
[dateTime] $currentTime = get-date
[int32] $getMinutes = (new-timespan -start $TrimTime -end $currentTime).minutes
I don't know how to loop through the output of the Get-content - to check for all of the Timestamps - we only want to report errors that are within 5 minutes of the current time.
Don't cast your matches to a string right away.
$readFile = get-content c:\test\friday.log | select-String -pattern '(order.*failed)' -context 1,7
If you leave the MatchInfo objects intact for the time being you can extract the timestamps from the aftercontexts like this:
$readFile | ForEach-Object {
$_.Context.AfterContext | Where-Object {
$_ -match 'timestamp: (\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{2}:\d{2} [ap]m)'
} | ForEach-Object {
$timestring = $matches[1]
}
}
Use the ParseExact() method to convert the matched substring to a DateTime value:
$fmt = 'M\/d\/yyyy h:mm:ss ttK'
$culture = [Globalization.CultureInfo]::InvariantCulture
$timestamp = [DateTime]::ParseExact("${timestring}Z", $fmt, $culture)
The Z that is appended to $timestring indicates the UTC timezone and the trailing K in the format string makes the method recognize it as such. The result is automatically converted to your local time. If you need the time in UTC: append .ToUniversalTime() to the method call.
I want yesterday's date as "MMdd", like "0418" today's date is "0419".
I tried this:
$d = (Get-Date).AddDays(-1) | select -Property Month,Day
$e = $d.Day
$d = $d.Month
$total = "$d" + "$e"
Write-Host $total
But the output was "418".
As you can see, this is a three-letter string, but I want a four-letter one. Because I want to search files they have that format and created a day before.
The date object has a method ToString, which allows you to specify the format:
$d = (Get-Date).AddDays(-1);
$d.ToString("MMdd");
I am calling this on April the 20th, and the output is
0419
See this link for a description of the available date format strings.
I have a log file where dates and corresponding information are listed in strings with a format of:
Tuesday, July 14, 2015 4:44:03 PM
Order Number 001006
Credit Card Type: AX
MessageType: 0100
Bit 2 [Primary_Account_Number..................] 3797*******1000
Bit 3 [Processing_Code.........................] 200000
Bit 4 [Amount_of_Transaction...................] 2.40
Bit 11 [Debit_Reg_E_Receipt_Number..............] 000083
Bit 14 [Expiration_Date.........................] 1704
Bit 18 [Merchant_Category_Code..................] 5812
Bit 22 [POS_Entry_Mode_PIN_Capability...........] 011
Bit 24 [Network_International_ID_NII............] 001
Bit 25 [POS_Condition_Code......................] 00
Bit 31 [Acquirer_Reference_Data.................] 2
Bit 37 [Retrieval_Reference_Number..............] 000000000283
Bit 41 [Terminal_ID.............................] 04765035
Bit 42 [Merchant_ID.............................] 000425249820993
Bit 59 [Merchant_ZIP_Postal_Code................] 19004
Tuesday, July 14, 2015 4:44:07 PM : Response:
Order Number
Credit Card Type:
MessageType: 0110
Bit 3 [Processing_Code.........................] 200000
Bit 4 [Amount_of_Transaction...................] 000000000240
Bit 7 [Transmission_Date_Time..................] 0714234410
Bit 11 [Debit_Reg_E_Receipt_Number..............] 000083
Bit 24 [Network_International_ID_NII............] 0001
Bit 25 [POS_Condition_Code......................] 00
Bit 37 [Retrieval_Reference_Number..............] 000000000283
Bit 39 [Response_Code...........................] 00
Bit 41 [Terminal_ID.............................] 04765035
Table 14
N 000000000000000000000240
Table 22
APPROVAL
This log is constantly updated, but I would like to get the current date/time and display only the information in the log from 3 hours ago, to present. I need all the information in the log, just not anything prior to Tuesday, July 14, 2015 1:44:03 (as an example). Is there some way I can get the current date/time and parse the log file for 3 hours prior to get the information I need? Maybe using Powershell or some sort of windows port of unix commands? The goal is to create a script that essentially truncates a log to display the past 3 hours of activity, and output that into a new log. I hope I stated this clearly enough. Any help is greatly appreciated.
I updated my answer as you updated your post :
$path = 'C:\list.txt'
$file = Get-Content $path
$i = 0
$before = (Get-Date).AddHours(-3)
foreach ($line in $file)
{
$i++
if ($line -match ', 2015 ')
{
$date = [regex]::match($line,'(?<=day, )(.*\n?)(?<=M)').value | Get-Date
if ($date -ge $before)
{
(Get-Content $path) | Select -Skip $($i-1)
Break
}
}
}
Ok, so this will set a boolean variable $NewLogs to $false. Then it reads your log file and for each line it checks to see if that $NewLogs has been set to $true, or if a date string is found, and if that date string is less than 3 hours old. If either the $NewLogs condition or the date string condition resolve to true, then it sets $NewLogs to $true (to make everything go faster for all lines past the current), and it passes that line through the pipeline. Then since $NewLogs is now $true all lines past that one will be passed through the pipeline.
$NewLogs = $false
Get-Content C:\Path\To\LogFile.txt -ReadCount 1000 |
ForEach{
If($NewLogs -or ($_ -match "(\S+day, \w+ \d{1,2}, \d{4} \d{1,2}:\d{1,2}:\d{2} \w{2})" -and ([datetime]::ParseExact($Matches[1],"dddd, MMMM dd, yyyy h:mm:ss tt",$null)) -gt [datetime]::Now.AddHours(-3))){$NewLogs=$true;$_}
}
Then you can just pipe that to a Set-Content command to output to a new file, or leave it as is to display it on the screen. If it were me I'd probably change that very last line to:
} | Set-Content (join-path 'C:\Path\To\' ([datetime]::now.tostring('MMddyyhhmmtt.\tx\t')))
So if I did that right now it would save the last 3 hours of logs to the file:
C:\Path\To\0715150359PM.txt
I get the same error about null array on Powershell v4 with the solution of TheMadTechnician, but doing the following works :
Remember to change the dates to
Thursday, July 16, 2015 8:44:03 PM
Thursday, July 16, 2015 8:44:04 PM
The updated script :
$NewLogs = $false
$file = Get-Content C:\list.txt -ReadCount 1000
foreach ($line in $file)
{
If($NewLogs -or ($line -match '(\S+day, \w+ \d{1,2}, \d{4} \d{1,2}:\d{1,2}:\d{2} \w{2})' -and ([datetime]::ParseExact($Matches[1],'dddd, MMMM dd, yyyy h:mm:ss tt',$null)) -gt [datetime]::Now.AddHours(-3)))
{
$NewLogs = $true
$line
}
}
Novice at PS here, but I have a log file where each entry is written on a separate line with this date format: 2013-04-29 08:55:09,261
I am trying to use PowerShell to delete all lines older than 30 days. I have been trying to plug get-date -format "yyyy-MM-dd hh" output with some sort of greater than code, but at this point I’m just guessing. Also been trying forfiles with a batch file but would like to stick with PS if I can.
Any help would be much appreciated.
Assuming a file with the contents:
2013-04-29 08:55:09,261 line1
2013-01-29 08:55:09,261 line2
2013-03-31 08:55:09,261 line3
For me your dates are in the international standard date notation so you can just use :
$a = [datetime]"2013-04-29 08:55:09"
Then $a will be a date, so nothing to do with the culture.
You can just write the following to filter all lines from a date (here "2013-03-31")
get-content "C:\temp\date.txt" | where { [datetime]($_.split(','))[0] -ge "2013-03-31"}
I just split the line on the coma, take the first part and convert it to a date before compararing it.
For your 30 days (get-date).date give the date without hours and (get-date).date.adddays(-30) give the date 30 days before today.
get-content C:\temp\date.txt | where { [datetime]($_.split(','))[0] -ge (get-date).date.adddays(-30)}
You can pipe the result in a new file | set-content "C:\temp\newdate.txt"
Try this:
gc .\logfile.txt | %{if ([datetime][regex]::match($_, '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}').Value -gt (Get-Date).AddDays(-30)) {$_}} > purgedlogfile.txt
Breakdown and explanation of the components:
.\logfile.txt is the path to the log file
^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3} is a regular expression that matches your log file's timestamp format at the beginning of each line
[regex]::match($_, '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}').Value returns the timestamp matched by the regular expression
Preceding it with the [datetime] typecast operator converts it to a DateTime object (in order to compare it to another DateTime)
(Get-Date) returns a DateTime object representing the current date and time, and .AddDays(-30) subtracts 30 days from it, to return a DateTime object representing 30 days ago
For each line in the logfile, the if block prints the line if the DateTime representing that line's timestamp is greater than the DateTime representing 30 days ago (i.e., the timestamp is more recent than 30 days), otherwise it ignores the line
Assuming a file with the contents:
2013-04-29 08:55:09,261 line1
2013-01-29 08:55:09,261 line2
2013-03-31 08:55:09,261 line3
You could get your desired output with:
$culture = [System.Globalization.CultureInfo]::InvariantCulture
$format = 'yyyy-MM-dd HH:mm:ss,fff'
Get-Content .\test.txt | ? { [datetime]::ParseExact(([string]$_).Substring(0,23), $format, $culture) -ge (Get-Date).AddDays(-30) }
Adi Inbar's answer is good, but if you'd prefer to avoid regular expressions, here's another way:
Get-Content e:\pathto\logfile.txt | Where-Object { ( (Get-Date) - (Get-Date $_.Substring(0,10)) ).Days -le 30} | Add-Content e:\pathto\NewLog.txt