PowerShell script efficiency advice - powershell

Question

I have a telephony .csv with compiled data from January 2020 and some days of February, each row has the date and time spent on each status, since someone uses different status over the day the file has one row for each status, my script is supposed to go through the file, find the minimum date and then start saving on new files all the data for the same day, so I'll end with one file for 01-01-2020, 02-01-2020 and so on, but it has 15 hours running and it's still at 1/22.
The column I'm using for the dates is called "DateFull" and this is the script
write-host "opening file"
$AT= import-csv “C:\Users\xxxxxx\Desktop\SignOnOff_20200101_20200204.csv”
write-host "parsing and sorting file"
$go= $AT| ForEach-Object {
$_.DateFull= (Get-Date $_.DateFull).ToString("M/d/yyyy")
$_
}
Write-Host "prep day"
$min = $AT | Measure-Object -Property Datefull -Minimum
Write-Host $min
$dateString = [datetime] $min.Minimum
Write-host $datestring
write-host "Setup dates"
$start = $DateString - $today
$start = $start.Days
For ($i=$start; $i -lt 0; $i++) {
$date = get-date
$loaddate = $date.AddDays($i)
$DateStr = $loadDate.ToString("M/d/yyyy")
$now = Get-Date -Format HH:mm:ss
write-host $datestr " " $now
#Install-Module ImportExcel #optional import if you dont have the module already
$Check = $at | where {$_.'DateFull' -eq $datestr}
write-host $check.count
if ($check.count -eq 0 ){}
else {$AT | where {$_.'DateFull' -eq $datestr} | Export-Csv "C:\Users\xxxxx\Desktop\signonoff\SignOnOff_$(get-date (get-date).addDays($i) -f yyyyMMdd).csv" -NoTypeInformation}
}
$at = ''

Answer 1

The first loop doesn't make much sense. It loops through CSV contents and converts each row's date into different a format. Afterwards, $go is never used.
$go= $AT| ForEach-Object {
$_.DateFull= (Get-Date $_.DateFull).ToString("M/d/yyyy")
$_
}
Later, there is an attempt to calculate a value from uninitialized a variable. $today is never defined.
$start = $DateString - $today
It looks, however, like you'd like to calculate, in days, how old eldest record is.
Then there's a loop that counts from negative days to zero. During each iteration, the whole CSV is searched:
$Check = $at | where {$_.'DateFull' -eq $datestr}
If there are 30 days and 15 000 rows, there are 30*15000 = 450 000 iterations. This has complexity of O(n^2), which means runtime will go sky high for even relative small number of days and rows.
The next part is that the same array is processed again:
else {$AT | where {$_.'DateFull' -eq $datestr
Well, the search condition is exactly the same, but now results are sent to a file. This has a side effect of doubling your work. Still, O(2n^2) => O(n^2), so at least the runtime isn't growing in cubic or worse.
As for how to fix this, there are a few things. If you sort the CSV based on date, it can be processed afterwards in just a single run.
$at = $at | sort -Property datefull
Then, iterate each row. Since the rows are in ascending order, the first is the oldest. For each row, check if date has changed. If not, add it to buffer. If it has, save the old buffer and create a new one.
The sample doesn't convert file names in yyyyMMdd format, and it assumes there are only two columns foo and datefull like so,
$sb = new-object text.stringbuilder
# What's the first date?
$current = $at[0]
# Loop through sorted data
for($i = 0; $i -lt $at.Count; ++$i) {
# Are we on next date?
if ($at[$i].DateFull -gt $current.datefull) {
# Save the buffer
$file = $("c:\temp\OnOff_{0}.csv" -f ($current.datefull -replace '/', '.') )
set-content $file $sb.tostring()
# Pick the current date
$current = $at[$i]
# Create new buffer and save data there
$sb = new-object text.stringbuilder
[void]$sb.AppendLine(("{0},{1}" -f $at[$i].foo, $at[$i].datefull))
} else {
[void]$sb.AppendLine(("{0},{1}" -f $at[$i].foo, $at[$i].datefull))
}
}
# Save the final buffer
$file = $("c:\temp\OnOff_{0}.csv" -f ($current.datefull -replace '/', '.') )
set-content $file $sb.tostring()