Split a logfile - powershell

I have a log file with this:
Wed Oct 17 05:39:27 2018 : Resource = 'test04' cstep= 'titi04' time =18.751s
Wed Oct 17 05:40:31 2018 : Resource = 'test05' cstep= 'titi05' time =58.407s
Wed Oct 17 05:41:31 2018 : Resource = 'test06' cstep= 'titi06' time =3.400s
Wed Oct 17 05:42:31 2018 : Resource = 'test07' cstep= 'titi07' time =4.402s
I want split and want only the values greater than 5:
18.751
58.407
My script is in PowerShell and collects all values, not just values greater than 5:
$list = Get-Content "C:\Users\Desktop\slow_trans\log_file.txt"
$results = foreach ($line in $list) {
$line.Split('=')[3].Trim().TrimEnd('s')
}
$results
Results are
18.751
58.407
3.400
4.402
I want only
3.400
4.402

Changing the requirements on the fly is normally a no go,
so you don't deserve it.
Also the wording Superior 5 reminds me at a previous question from another user account..
Nevertheless here a script with a single pipe and datetime conversion.
## Q:\Test\2018\11\06\SO_53170145.ps1
Get-Content .\logfile.txt |
Where-Object {$_ -match '^(.*?) : .*time =([0-9\.]+)s'}|
Select-Object #{n='DT';e={([datetime]::ParseExact($Matches[1],'ddd MMM dd HH:mm:ss yyyy',[cultureinfo]::InvariantCulture).ToString('yyyy-MM-dd HH:mm:ss'))}},
#{n='val';e={[double]$Matches[2]}} |
Where-Object val -le 5
Sample output (decimal comma due to my German locale)
DT val
-- ---
2018-10-17 05:41:31 3,4
2018-10-17 05:42:31 4,402

the following casts the selected string as double and then returns only those which are less than 5
$results = Foreach ($line in $list) {
$val = [double]$line.Split('=')[3].Trim().TrimEnd('s')
if($val -lt 5) {
$val
}
}

Select-String is one option:
(Select-String -Path "TargetLog.txt" -Pattern ".*(?<time>\d+\.\d+)s").Matches |
ForEach-Object {
if([double]$_.Groups['time'].Value -lt 5.0) {$_.Value}
}
This will output the entire matching line:
Wed Oct 17 05:41:31 2018 : Resource = 'test06' cstep= 'titi06' time =3.400s
Wed Oct 17 05:42:31 2018 : Resource = 'test07' cstep= 'titi07' time =4.402s
If you only want the number from each line, change the if block to this:
{$_.Groups['time'].Value}

Related

Joining every two lines in Powershell output

I'm attempting to combine every two lines of the output of this Powershell command:
(((Invoke-WebRequest -Uri "https://www.timeanddate.com/holidays/us/$(Get-Date -Format yyyy)").Content | sls 'id=holidays') -split '<th class="nw" >' | Out-String -Stream) -replace '<|>',',' | ForEach-Object {$_.Split(',')[10,0];}
As you can see if you run it it outputs holidays and their date for the current year like so:
New Year's Day
Jan 1
World Braille Day
Jan 4
Epiphany
Jan 6
Orthodox Christmas Day
Jan 7
International Programmers' Day
Jan 7
etc.
My goal is for the output to be:
New Year's Day Jan 1
World Braille Day Jan 4
Epiphany Jan 6
Orthodox Christmas Day Jan 7
International Programmers' Day Jan 7
etc.
Any suggestions are welcome (I would like to do this without writing output to a file during the process). Or if there is a more efficient way of doing this I'm open to that as well.
Use a simple for loop with a counter that increments by 2 every time:
$splitLines = (((Invoke-WebRequest -Uri "https://www.timeanddate.com/holidays/us/$(Get-Date -Format yyyy)").Content | sls 'id=holidays') -split '<th class="nw" >' | Out-String -Stream) -replace '<|>',',' | ForEach-Object {$_.Split(',')[10,0];}
for($i = 0; $i -lt $splitLines.Count; $i += 2){
$splitLines[$i,($i+1)] -join ' '
}
You can do something along those lines. For some reason I have to use modulus 4 here and not 2 because when I split like that it makes every second line empty.
$inputData = #"
New Year's Day
Jan 1
World Braille Day
Jan 4
Epiphany
Jan 6
Orthodox Christmas Day
Jan 7
International Programmers' Day
Jan 7
"#
$splitData = $inputData.Split([Environment]::NewLine)
# use a dot here to make the output availabe after the iteration is completed
$splitData | . {
begin {
$output = ""
$i = 0
}
process {
if(0 -eq $i % 4) {
$output += $_ + [Environment]::NewLine
}
$i = $i + 1
}
}
Write-Output $output
# New Year's Day
# World Braille Day
# Epiphany
# Orthodox Christmas Day
# International Programmers' Day
It's not perfect but it works for the given problem I would suggest tinkering with it and make it little more nice. Haven't used PowerShell in a while.

Powershell - change file Date Created and Date Modified based on Filename

I have lots of file scanned PDF documents that have the file named with an included date. For example:
FileA_2017-10-15.pdf
FileB_2016-04-08.pdf
FileC_2018-01-30.pdf
some files also are formatted with an underscore at the end as well such as...
FileD_2018-01-30_1.pdf
FileE_2018-01-30_2.pdf
there are even a few that have two underscores before the date such as...
FileF_Example_2018-01-30_1.pdf
FileG_Example_2018-01-30_2.pdf
Unfortunately, the date they were scanned in is different than the actual date of the document. So the "Date Created" and "Date Modified" attributes are different than what is shown in the file name.
I would like a script that I could run to change the "Date Created" and "Date Modified" to match that of the filename.
I attempted this using someone else's script but I don't know enough about PowerShell to make it actually work. Note that I do not want to change the name of the file, only the time stamp.
$Directory = "C:\TestFolder"
$DateFormat = "yyyy-MM-dd"
foreach ($file in (Get-ChildItem $Directory)) {
$date_from_file=GetFileName::[datetime])
$file.CreationTime = $date_from_file
$file.LastAccessTime = $date_from_file
$file.LastWriteTime = $date_from_file
Write-Host ($file.Name + " - " + $date_from_file)
}
The code above can be scraped if something else has already been written since what I have doesn't work.
Edit
Wondering if it would also be possible to add to the script so that it could include files in sub-folders as well. Maybe it could be scripted in a way that would only consider the files in a folder if the Date Modified on the folder is today. I would like to run this on a parent folder that could potentially have many sub-folders and if those folders don't have a "Date Modified" of today, then it should skip the files in that folder. I was thinking that could speed up the process. Open to thoughts and thanks for the help!
You are quite near, you need
split the date part from filename and convert it to a [datetime]
I use a RegEx with a capture group anchored at the end $ of the BaseName
## Q:\Test\2019\05\19\SO_56211626.ps1
$Directory = "C:\TestFolder"
foreach ($file in (Get-ChildItem -Path $Directory -Filter *.pdf)){
if($File.BaseName -match '_(\d{4}-\d{2}-\d{2})(_\d)?$'){
$date_from_file= (Get-Date $Matches[1])
$file.CreationTime = $date_from_file
$file.LastAccessTime = $date_from_file
$file.LastWriteTime = $date_from_file
$file | Select-Object Name,CreationTime,LastAccessTime,LastWriteTime
}
}
Sample output:
> Q:\Test\2019\05\19\SO_56211626.ps1
Name CreationTime LastAccessTime LastWriteTime
---- ------------ -------------- -------------
FileA_2017-10-15.pdf 2017-10-15 00:00:00 2017-10-15 00:00:00 2017-10-15 00:00:00
FileB_2016-04-08.pdf 2016-04-08 00:00:00 2016-04-08 00:00:00 2016-04-08 00:00:00
FileC_2018-01-30.pdf 2018-01-30 00:00:00 2018-01-30 00:00:00 2018-01-30 00:00:00
An English locale (en-US) produces:
Name CreationTime LastAccessTime LastWriteTime
---- ------------ -------------- -------------
FileA_2017-10-15.pdf 10/15/2017 12:00:00 AM 10/15/2017 12:00:00 AM 10/15/2017 12:00:00 AM
FileB_2016-04-08.pdf 4/8/2016 12:00:00 AM 4/8/2016 12:00:00 AM 4/8/2016 12:00:00 AM
FileC_2018-01-30.pdf 1/30/2018 12:00:00 AM 1/30/2018 12:00:00 AM 1/30/2018 12:00:00 AM
[
edit - since the OP is getting very strange errors with my suggested fix - errors that i cannot reproduce with the sample data - i've changed this answer to the full suggested code.
edit 2 - added new file name variants and code to deal with them.
edit 3 - changed from splitting to a regex match since the sample data has changed yet again. [*sigh ...*]
]
you are not actually creating the datetime object that you need. the $date_from_file= line doesn't actually do anything other than create red error msgs ... [grin]
replace this line ...
$date_from_file=GetFileName::[datetime])
... with this line ...
$date_from_file = [datetime]::ParseExact($File.BaseName.Split('_')[-1], $DateFormat, $Null)
... and your $date_from_file variable will contain a proper [datetime] object that will work in your assignments.
i would likely change the sequence of those assignments to put the $file.LastAccessTime = $date_from_file LAST so that it doesn't get changed by the next line.
also, that value will change any time that the file is accessed, so it may not be worth changing. [grin]
here is the full script along with what it does -
what it does ...
sets the location & the date format to use
creates a set of test files from the OPs sample file names
gets the files from the source
converts the .BaseName into a [datetime] object
assigns the .CreationTime, .LastWriteTime, & .LastAccessTime values to the datetime from the file name
displays the changed values
here is the code ...
$Directory = $env:TEMP
$DateFormat = "yyyy-MM-dd"
# create some test files
$TestFileList = #(
'FileA_2017-10-15.pdf'
'FileB_2016-04-08.pdf'
'FileC_2018-01-30.pdf'
'FileD_2019-09-09_1.pdf'
'FileE_2015-05-05_2.pdf'
)
foreach ($TFL_Item in $TestFileList)
{
$Null = New-Item -Path $Directory -Name $TFL_Item -ItemType File -Force
}
$FileList = Get-ChildItem -LiteralPath $Directory -Filter '*.pdf' -File
foreach ($FL_Item in $FileList) {
# removed split, added regex match to work with ever-growing list of variant file names
$Null = $FL_Item.BaseName -match '_(?<DateString>\d{4}-\d{2}-\d{2})'
$DateString = $Matches.DateString
$date_from_file = [datetime]::ParseExact($DateString, $DateFormat, $Null)
$FL_Item.CreationTime = $date_from_file
$FL_Item.LastWriteTime = $date_from_file
$FL_Item.LastAccessTime = $date_from_file
# show the resulting datetime info
'=' * 20
$CurrentFileInfo = Get-Item -LiteralPath $FL_Item.FullName
$CurrentFileInfo.FullName
$CurrentFileInfo.CreationTime
$CurrentFileInfo.LastWriteTime
$CurrentFileInfo.LastAccessTime
}
screen output ...
====================
C:\Temp\FileA_2017-10-15.pdf
2017 October 15, Sunday 12:00:00 AM
2017 October 15, Sunday 12:00:00 AM
2017 October 15, Sunday 12:00:00 AM
====================
C:\Temp\FileB_2016-04-08.pdf
2016 April 08, Friday 12:00:00 AM
2016 April 08, Friday 12:00:00 AM
2016 April 08, Friday 12:00:00 AM
====================
C:\Temp\FileC_2018-01-30.pdf
2018 January 30, Tuesday 12:00:00 AM
2018 January 30, Tuesday 12:00:00 AM
2018 January 30, Tuesday 12:00:00 AM
====================
C:\Temp\FileD_2019-09-09_1.pdf
2019 September 09, Monday 12:00:00 AM
2019 September 09, Monday 12:00:00 AM
2019 September 09, Monday 12:00:00 AM
====================
C:\Temp\FileE_2015-05-05_2.pdf
2015 May 05, Tuesday 12:00:00 AM
2015 May 05, Tuesday 12:00:00 AM
2015 May 05, Tuesday 12:00:00 AM
i checked the files directly in explorer & they match the displayed values.
Thanks. I was stuck without this thread. I ended up with a variation that matched any filename with a correctly formatted date, thus:
# Call like:
# powershell -NoLogo -ExecutionPolicy Unrestricted -Sta -NonInteractive -WindowStyle Normal -File ".\Rename_files_selected_folders_ModifyDateStamps.ps1" -Folder "T:\files"
# 1. capture a commandline parameter 1 as a mandatory "Folder string" with a default value
param ( [Parameter(Mandatory=$true)] [string]$Folder = "T:\HDTV\autoTVS-mpg\Converted" )
[console]::BufferWidth = 512
$DateFormat = "yyyy-MM-dd"
write-output "Processing Folder: ",$Folder
# 2. Iterate the files
$FileList = Get-ChildItem -Recurse $Folder -Include '*.mp4','*.bprj','*.ts' -File
foreach ($FL_Item in $FileList) {
$ixxx = $FL_Item.BaseName -match '(?<DateString>\d{4}-\d{2}-\d{2})'
if($ixxx){
#write-output $FL_Item.FullName
$DateString = $Matches.DateString
$date_from_file = [datetime]::ParseExact($DateString, $DateFormat, $Null)
$FL_Item.CreationTime = $date_from_file
$FL_Item.LastWriteTime = $date_from_file
$FL_Item | Select-Object FullName,CreationTime,LastWriteTime
}
}
# https://stackoverflow.com/questions/56211626/powershell-change-file-date-created-and-date-modified-based-on-filename

Change values in an array using foreach

I have a CSV file I import, and I need to change the format of a column (backed_up_ts) to remove the original data in the column and replace it with the updated format.
The original data for each row in the column looks like this:
Tue Sep 13 07:46:26 MDT 2016
The first part of my code formats every row so that it can be compared to a date:
$csv = Import-Csv -Path "C:\Users\user\Desktop\NewReport.csv" | Select display_client_name, backed_up_ts
$csv = foreach ($budate in $csv.backed_up_ts) {
$budate = $budate.Substring(3)
$budate = $budate.Replace("MDT", "")
$budate = $budate.Replace("MST", "")
$budate = $budate.Split(" ")
$budate = $budate[1] + " " + $budate[2] + " " + $budate[5]
$budate = ($budate | Get-Date -Format d)
From here I need to take the updated format (in the $budate value) and replace the original data in the $csv.backed_up_ts array.
However, the value of $budate is only the last value of the object in the foreach loop (3/30/16).
The value of $csv.backed_up_ts remains in the array in its original format (Wednesday, August 30th, EST, 2016).
I need the original array $csv.backed_up_ts to be replaced by the coordinating values created by the $budate.
Thanks.
Updated answer based on clarification of desired transformation of data.
Code
# demonstrate changing columnn data in csv input
$input = #"
header1, header2
value11, "Tue Sep 13 07:46:26 MDT 2016"
value21, "Tue Oct 19 07:46:26 MDT 2017"
"#
$csv = ConvertFrom-Csv $input
"original"
$csv
foreach ($item in $csv)
{
# modify the header2 column in our csv input
$parts = $item.header2.Split(' ')
$newDate = $parts[1] + " " + $parts[2] + " " + $parts[5]
$item.header2 = [DateTime]::Parse($newDate) | Get-Date -Format d
}
"new"
$csv
Output
original
header1 header2
------- -------
value11 Tue Sep 13 07:46:26 MDT 2016
value21 Tue Oct 19 07:46:26 MDT 2017
new
value11 9/13/2016
value21 10/19/2017

Powershell - Search and output line by line

UPDATE
I have got a log file of 1000 lines containing some reference.
Time Reference Date of start Date of end
12:00 AT001 13 November 2011 15 November 2011
13:00 AT038 15 December 2012 17 December 2012
14:00 AT076 17 January 2013 19 January 2013
$ref1 = AT038
Basically, I want to parse the log file and have an output (line by line) for $ref1 such as :
Time : 13h
Reference : AT038
Date of start : 15 December 2012
Date of end : 17 December 2012
Thanks in advance
try:
$ref1 = "AT038"
$csv = Import-Csv .\myfile.txt -Delimiter ' '#Import file as CSV with space as delimiter
$csv | ? { $_.reference -EQ $ref1 } | FL #Piping each line of CSV to where-object cmdlet, filtering only line where value of column reference is equal to $ref1 variable value. Piping the result of the filtering to file-list to have output as requested in OP.
Code added after requisite are changed in OP:
$ref1 = "AT038"
$txt = gc .\myfile.txt
$txt2 = $txt | % { $b = $_ -split ' '; "$($b[0]) $($b[1]) $($b[2])_$($b[3])_$($b[4]) $($b[5])_$($b[6])_$($b[7])" }
$csv = convertfrom-csv -InputObject $txt2 -Delimiter ' '
$csv | ? { $_.reference -EQ $ref1 } | FL
How about this:
Get-Content SourceFileName.txt |
% { ($_ -Replace '(\d{2}):\d{2} (\w{2}\d{3})', 'Time : $1h|Reference : $2').Split('|')} |
Out-File TargetFileName.txt
Here is my revised version:
$regex = '(\d{2}):\d{2} (\w{2}\d{3}) (\d{2} \b\w+\b \d{4}) (\d{2} \b\w+\b \d{4})'
$replace = 'Time : $1h|Reference : $2|Date of start : $3|Date of end : $4'
Get-Content SourceFileName.txt |
% { ($_ -Replace $regex, $replace).Split('|')} |
Out-File TargetFileName.txt

PowerShell: Comparing dates

I am querying a data source for dates. Depending on the item I am searching for, it may have more than date associated with it.
get-date ($Output | Select-Object -ExpandProperty "Date")
An example of the output looks like:
Monday, April 08, 2013 12:00:00 AM
Friday, April 08, 2011 12:00:00 AM
I would like to compare these dates and return which one is set further out into the future.
As Get-Date returns a DateTime object you are able to compare them directly. An example:
(get-date 2010-01-02) -lt (get-date 2010-01-01)
will return false.
I wanted to show how powerful it can be aside from just checking "-lt".
Example: I used it to calculate time differences take from Windows event view Application log:
Get the difference between the two date times:
PS> $Obj = ((get-date "10/22/2020 12:51:1") - (get-date "10/22/2020 12:20:1 "))
Object created:
PS> $Obj
Days : 0
Hours : 0
Minutes : 31
Seconds : 0
Milliseconds : 0
Ticks : 18600000000
TotalDays : 0.0215277777777778
TotalHours : 0.516666666666667
TotalMinutes : 31
TotalSeconds : 1860
TotalMilliseconds : 1860000
Access an item directly:
PS> $Obj.Minutes
31
Late but more complete answer in point of getting the most advanced date from $Output
## Q:\test\2011\02\SO_5097125.ps1
## simulate object input with a here string
$Output = #"
"Date"
"Monday, April 08, 2013 12:00:00 AM"
"Friday, April 08, 2011 12:00:00 AM"
"# -split '\r?\n' | ConvertFrom-Csv
## use Get-Date and calculated property in a pipeline
$Output | Select-Object #{n='Date';e={Get-Date $_.Date}} |
Sort-Object Date | Select-Object -Last 1 -Expand Date
## use Get-Date in a ForEach-Object
$Output.Date | ForEach-Object{Get-Date $_} |
Sort-Object | Select-Object -Last 1
## use [datetime]::ParseExact
## the following will only work if your locale is English for day, month day abbrev.
$Output.Date | ForEach-Object{
[datetime]::ParseExact($_,'dddd, MMMM dd, yyyy hh:mm:ss tt',$Null)
} | Sort-Object | Select-Object -Last 1
## for non English locales
$Output.Date | ForEach-Object{
[datetime]::ParseExact($_,'dddd, MMMM dd, yyyy hh:mm:ss tt',[cultureinfo]::InvariantCulture)
} | Sort-Object | Select-Object -Last 1
## in case the day month abbreviations are in other languages, here German
## simulate object input with a here string
$Output = #"
"Date"
"Montag, April 08, 2013 00:00:00"
"Freidag, April 08, 2011 00:00:00"
"# -split '\r?\n' | ConvertFrom-Csv
$CIDE = New-Object System.Globalization.CultureInfo("de-DE")
$Output.Date | ForEach-Object{
[datetime]::ParseExact($_,'dddd, MMMM dd, yyyy HH:mm:ss',$CIDE)
} | Sort-Object | Select-Object -Last 1
Considering you want to include time also, I have included sample. I am putting datetime in the ISO8601, so it works in locale agnostic manner.
Monday, April 08, 2013 12:00:00 AM
Friday, April 08, 2011 12:00:00 AM
(Get-date "2013-04-08T00:00:00") -lt (Get-Date "2011-04-08T00:00:00")
False