Scenario: Working PS script traverses multiple folders containing logs (folder name is the name of the originating server), extracting the last line from the last completed log - this works, but in case the last log line is part of a multiline message, I get useless data from this line.
Question: How can I implement in the working script a cleanup that puts multiline messages all in one line?
Working script (original post)
# List all folders on the Initial Path
$folders = Get-ChildItem D:\Logs | Select-Object -ExpandProperty FullName
$export = [system.collections.generic.list[pscustomobject]]::new()
foreach($folder in $folders)
{
# Get the newest file in each folder
$file = Get-ChildItem $folder -File | Sort-Object LastWriteTime -Descending | Select -First 1
# Read the content of the file and get the last line
#$content = (Get-Content $file)[-1]
$content = (Get-Content $file.FullName)[-1]
# Here you can create a new object that will be used to export to Csv
# As an example, i'll create an object with the File Name,
# Last Write Time and Last Line of the Content (This will be the CSV columns)
$export.Add(
[PSCustomObject]#{
DirName=$file.FullName
# FileName = $file.Name
LastWriteTime = $file.LastWriteTime
LastLine = $content
})
}
# After the loop finishes you can export the results to a Csv
$export | Export-Csv -NoTypeInformation D:\export\MyFilelist_LastLogLines.csv
Related
I want to save in my computer "C:\logFiles" a specific date for logfile generated by program in another PC,
path that i will get from it the log file is "C:\Sut\Stat\03-2021.log"
Example : this file "C:\Sut\Stat\03-2021.Sutwin.log" contenant all the log of Mars month but i just want to get the log of last 7 Days from 19-03-2021 to 26-03-2021
I found this script in the internet but i doesn't work for me i need some help:
Example of the file .log in the photo attached:
Rest of image for the first screenshot :
my PC name : c01234
name of PC contenant log file : c06789
file that i will get from it the infos : 03-2021.Sutwin.log (exist in pc c06789)
i want to transfer the contents of just last 7 days in a folder in my PC c01234 with name Week11_LogFile
$log = "2015-05-09T06:39:34 Some information here
2015-05-09T06:40:34 Some information here
" -split "`n" | Where {$_.trim()}
#using max and min value for the example so all correct dates will comply
$upperLimit = [datetime]::MaxValue #replace with your own date
$lowerLimit = [datetime]::MinValue #replace with your own date
$log | foreach {
$dateAsText = ($_ -split '\s',2)[0]
try
{
$date = [datetime]::Parse($dateAsText)
if (($lowerLimit -lt $date) -and ($date -lt $upperLimit))
{
$_ #output the current item because it belongs to the requested time frame
}
}
catch [InvalidOperationException]
{
#date is malformed (maybe the line is empty or there is a typo), skip it
}
}
Based on your images, your log files look like simple tab-delimited files.
Assuming that's the case, this should work:
# Import the data as a tab-delimited file and add a DateTime column with a parsed value
$LogData = Import-Csv $Log -Delimiter "`t" |
Select-Object -Property *, #{n='DateTime';e={[datetime]::ParseExact($_.Date + $_.Time, 'dd. MMM yyHH:mm:ss', $null)}}
# Filter the data, drop the DateTime column, and write the output to a new tab-delimited file
$LogData | Where-Object { ($lowerLimit -lt $_.DateTime) -and ($_.DateTime -lt $upperLimit) } |
Select-Object -ExcludeProperty DateTime |
Export-Csv $OutputFile -Delimiter "`t"
The primary drawback here is that on Windows Powershell (v5.1 and below) you can only export the data quoted. On Powershell 7 and higher you can use -UseQuotes Never to prevent the fields from being double quote identified if that's important.
The only other drawback is that if these log files are huge then it will take a long time to import and process them. You may be able to improve performance by making the above a one-liner like so:
Import-Csv $Log -Delimiter "`t" |
Select-Object -Property *, #{n='DateTime';e={[datetime]::ParseExact($_.Date + $_.Time, 'dd. MMM yyHH:mm:ss', $null)}} |
Where-Object { ($lowerLimit -lt $_.DateTime) -and ($_.DateTime -lt $upperLimit) } |
Select-Object -ExcludeProperty DateTime |
Export-Csv $OutputFile -Delimiter "`t"
But if the log files are extremely large then you may run into unavoidable performance problems.
It's a shame your example of a line in the log file does not reveal the exact date format.
2015-05-09 could be yyyy-MM-dd or yyyy-dd-MM, so I'm guessing it's yyyy-MM-dd in below code..
# this is the UNC path where the log file is to be found
# you need permissions of course to read that file from the remote computer
$remotePath = '\\c06789\C$\Sut\Stat\03-2021.log' # or use the computers IP address instead of its name
$localPath = 'C:\logFiles\Week11_LogFile.log' # the output file
# set the start date for the week you are interested in
$startDate = Get-Date -Year 2021 -Month 3 -Day 19
# build an array of formatted dates for an entire week
$dates = for ($i = 0; $i -lt 7; $i++) { '{0:yyyy-MM-dd}' -f $startDate.AddDays($i) }
# create a regex string from that using an anchor '^' and the dates joined with regex OR '|'
$regex = '^({0})' -f ($dates -join '|')
# read the log file and select all lines starting with any of the dates in the regex
((Get-Content -Path $remotePath) | Select-String -Pattern $regex).Line | Set-Content -Path $localPath
Sorry in advance for the probably trivial question, I'm a powershell noob, please bear with me and give me advice on how to get better.
I want to achieve a file index index.txt that contains the list of all files in current dir and subdirs in this format:
./dir1/file1.txt 07.05.2020 16:16 1959281
where
dirs listed are relative (i.e. this will be run remotely and to save space, the relative path is good enough)
the delimiter is a tab \t
the date format is day.month.fullyear hours:minutes:seconds, last written (this is the case for me, but I'm guessing this would be different on system setting and should be enforced)
(the last number is the size in bytes)
I almost get there using this command in powershell (maybe that's useful to someone else as well):
get-childitem . -recurse | select fullname,LastWriteTime,Length | Out-File index.txt
with this result
FullName LastWriteTime Length
-------- ------------- ------
C:\Users\user1\Downloads\test\asdf.txt 07.05.2020 16:19:29 1490
C:\Users\user1\Downloads\test\dirtree.txt 07.05.2020 16:08:44 0
C:\Users\user1\Downloads\test\index.txt 07.05.2020 16:29:01 0
C:\Users\user1\Downloads\test\test.txt 07.05.2020 16:01:23 814
C:\Users\user1\Downloads\test\text2.txt 07.05.2020 15:55:45 1346
So the questions that remain are: How to...
get rid of the headers?
enforce this date format?
tab delimit everything?
get control of what newline character is used (\n or \r or both)?
Another approach could be this:
$StartDirectory = Get-Location
Get-ChildItem -Path $StartDirectory -recurse |
Select-Object -Property #{Name='RelPath';Expression={$_.FullName.toString() -replace [REGEX]::Escape($StartDirectory.ToString()),'.'}},
#{Name='LastWriteTime';Expression={$_.LastWriteTime.toString('dd.MM.yyyy HH:mm:ss')}},
Length |
Export-Csv -Path Result.csv -NoTypeInformation -Delimiter "`t"
I recommend to use proper CSV files if you have structured data like this. The resulting CSV file will be saved in the current working directory.
If the path you are running this from is NOT the current scrip path, do:
$path = 'D:\Downloads' # 'X:\SomeFolder\SomeWhere'
Set-Location $path
first.
Next, this ought to do it:
Get-ChildItem . -Recurse -File | ForEach-Object {
"{0}`t{1:dd.MM.yyyy HH:mm}`t{2}" -f ($_ | Resolve-Path -Relative), $_.LastWriteTime, $_.Length
} | Out-File 'index.txt'
On Windows the newline will be \r\n (CRLF)
If you want control over that, this should do:
$newline = "`n" # for example
# capture the lines as string array in variable $lines
$lines = Get-ChildItem . -Recurse -File | ForEach-Object {
"{0}`t{1:dd.MM.yyyy HH:mm}`t{2}" -f ($_ | Resolve-Path -Relative), $_.LastWriteTime, $_.Length
}
# join the array with the chosen newline and save to file
$lines -join $newline | Out-File 'index.txt' -NoNewline
Because your requirement is to NOT have column headers in the output file, I'm using Out-File here instead of Export-Csv
I have a directory on a server called 'servername'. In that directory, I have subdirectories whose name is a date. In those date directories, I have about 150 .csv file audit logs.
I have a partially working script that starts from inside the date directory, enumerates and loops through the .csv's and searches for a string in a column. Im trying to get it to export the row for each match then go on to the next file.
$files = Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\20180525'
ForEach ($file in $files) {
$Result = If (import-csv $file.FullName | Where {$_.'path/from' -like "*01May18.xlsx*"})
{
$result | Export-CSV -Path c:\temp\output.csv -Append}
}
What I am doing is searching the 'path\from' column for a string - like a file name. The column contains data that is always some form of \folder\folder\folder\filename.xls. I am searching for a specific filename and for all instances of that file name in that column in that file.
My issue is getting that row exported - export.csv is always empty. Id also like to start a directory 'up' and go through each date directory, parse, export, then go on to the next directory and files.
If I break it down to just one file and get it out of the IF it seems to give me a result so I think im getting something wrong in the IF or For-each but apparently thats above my paygrade - cant figure it out....
Thanks in advance for any assistance,
RichardX
The issue is your If block, when you say $Result = If () {$Result | ...} you are saying that the new $Result is equal what's returned from the if statement. Since $Result hasn't been defined yet, this is $Result = If () {$null | ...} which is why you are getting a blank line.
The If block isn't even needed. you filter your csv with Where-Object already, just keep passing those objects down the pipeline to the export.
Since it sounds like you are just running this against all the child folders of the parent, sounds like you could just use the -Recurse parameter of Get-ChildItem
Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\' -Recurse |
ForEach-Object {
Import-csv $_.FullName |
Where-Object {$_.'path/from' -like "*01May18.xlsx*"}
} | Export-CSV -Path c:\temp\output.csv
(I used a ForEach-Object loop rather than foreach just demonstrate objects being passed down the pipeline in another way)
Edit: Removed append per Bill_Stewart's suggestion. Will write out all entries for the the recursed folders in the run. Will overwrite on next run.
I don't see a need for appending the CSV file? How about:
Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\20180525' | ForEach-Object {
Import-Csv $_.FullName | Where-Object { $_.'path/from' -like '*01May18.xlsx*' }
} | Export-Csv 'C:\Temp\Output.csv' -NoTypeInformation
Assuming your CSVs are in the same format and that your search text is not likely to be present in any other columns you could use a Select-String instead of Import-Csv. So instead of converting string to object and back to string again, you can just process as strings. You would need to add an additional line to fake the header row, something like this:
$files = Get-ChildItem '\\servername\volume\dir1\audit\serverbeingaudited\20180525'
$result = #()
$result += Get-Content $files[0] -TotalCount 1
$result += ($files | Select-String -Pattern '01May18\.xlsx').Line
$result | Out-File 'c:\temp\output.csv'
I have a PowerShell script that almost does what I want.
Basically there are CSV file feeds that are written to a specific location and stored by year and month. I have to compare the number of rows between the two newest CSV files, as a large discrepancy indicates an issue.
Currently my script fetches the newest CSV file and returns the row count with no problems, but I can't work out how to get it to return the row count for the 2 newest files. It is likely due to the way I've structured the script:
$datemonth = (Get-Date).Month
$dateyear = (Get-Date).Year
## get latest csv files
$dir = "\\160.1.1.98\c$\Scheduled Task Software\ScheduledTask\Application Files\ScheduledTask_1_0_0_9\Files\$dateyear\$datemonth\SentFeedFiles"
$latest = Get-ChildItem -Path $dir |
Sort-Object LastAccessTime -Descending |
Select-Object -First 1
## get path to csv files, add headers and count number of rows.
$filepath = $dir + '\' + $latest
$CSVCOUNT = (Import-Csv $filepath -Header 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28).Count
If I change to -First 2 then I get the following error:
Import-Csv : Could not find file '\16.1.1.18\c$\Scheduled Task Software\ScheduledTask\Application Files\ScheduledTask_1_0_0_9\Files\2017\3\SentFeedFiles\lkrlkr200317.csv lkrlkr19017.csv'.
I know why I'm getting this error - its trying to join the two file names into one path. However, I'm at a loss of how to get around this. I'm thinking a loop may be required but I'm not sure where.
Chucked 3 CSV files in f:\tmp locally to test:
$dir = "F:\tmp"
$files = Get-ChildItem -Path $dir | Sort-Object LastAccessTime -Descending | Select-Object -First 2
($files | Get-Content).Count
Import-Csv only deals with a single file as far as I remember - so you can't pass two file paths to it.
If you want to use Import-CSV (for ignoring headers etc), you can foreach file, but you have to pass the full path into it:
($files.FullName | % { Import-Csv -Path $_ }).Count
To get two separate results, do the following:
Include headers:
($files[0] | Get-Content).count
($files[1] | Get-Content).count
Exclude headers:
(Import-Csv -Path $files[0].FullName).Count
(Import-Csv -Path $files[1].FullName).Count
I want to change the creation date on a various folders/files recursively. I have managed to get a simple powershell script to do this. However, the log file that is created only says true on several lines, depending on how many changes where made. What I would like is for the log file to list the file path and name of the file that was actually changed.
Below is the simple script I have that does the change but no details log file:
Get-ChildItem -recurse G:\ | % {$_.CreationTime = '10/10/2014 15:00'} | Out-File "c:\pslog.txt"
Please help as I am very new to powershell so the simpler the code the better.
Regards,
Mark
When you execute $_.CreationTime = '10/10/2014 15:00', the status of the operation is returned, so a bunch of Trues just means that the new CreationTime assignment succeeded
To get the file path, hide the assignment and drop $_.FullName into the pipeline:
Get-ChildItem -recurse G:\ | % {($_.CreationTime = '10/10/2014 15:00')|Out-Null; $_.FullName } | Out-File "C:\pslog.txt"
However, it might be useful to get the FileName along with the result, so that you can assess whether some of the files failed the assignment
You say you'd like some simple code, but compact one-liners does not equal "simple", and you might find that you learn a lot more from verbose code.
Let's split it into multiple statements for a better overview:
# Get the files
$gFiles = Get-ChildItem -recurse G:\
# Loop through them all
$gFiles | ForEach-Object {
# Set the creation date without returning any output
($_.CreationTime = '10/10/2014 15:00') |Out-Null
# Test if the previous operation was successful:
if($?)
{
# Success, create an object containing the Path and status
New-Object PSObject -Property #{
"FilePath" = $_.FullName
"FileSize" = $_.Length
"Result" = "Success"
}
}
else
{
# Success, create an object containing the Path and status
New-Object PSObject -Property #{
"FilePath" = $_.FullName
"FileSize" = $_.Length
"Result" = "Failed"
}
}
# Export the objects containing the result to a .CSV file
} |Export-Csv -LiteralPath "C:\pslog.csv" -Delimiter ";" -NoTypeInformation -Force
Now, C:\pslog.csv contains two semicolon-separated columns with appropriate headers for "Result" and "FilePath"
The FileSize property will be in number of bytes, but you could change it to KB or MB with:
"FileSize" = $_.Length / 1KB
"FileSize" = $_.Length / 1MB