Parsing multiple entries in multiple CSV files - powershell

Question

Hope someone can offer a suggestion to help me speed up a Powershell script. What I am doing is reading in hundreds of CSV files, parsing the information to get data about missing entries, and then writing that output to a HTML file. Here is the loop that I am using to process the files:
ForEach ($Filename in $FileList) {
$CustTemp = import-csv "$FilePath\$Filename"
$CustName = $CustTemp[0].CustName
Write-Host "Reading data for $CustName"`r
For ($counter=0;$counter -lt 31;$counter++){
$CheckDate = (get-date).AddDays(-$counter)
$CheckShortDate = $CheckDate.ToShortDateString()
$TempData = import-csv "$FilePath\$Filename" | Select FileName,FileDate | where {$_.FileDate -eq $CheckShortDate}
If ($TempData -eq $null) {
$row = "No file found for $CheckShortDate for $CustName"
$HTMLReportItems += $row
}
$HTMLReportItems = $HTMLReportItems | ConvertTo-Html -Fragment
}
}
This loop worked fine when I was testing with a few CSV files but when running it against a large number of files (300+) the loop is taking an extremely long time to complete for each file (30s-1m). I'm pretty sure the reason why is that the CSV file is being accessed 30 times per iteration. What I am hoping is that someone will have a better suggestion on how I can process the data.

Answer 1

You're reading $FilePath\$Filename multiple times. Read it outside the for loop and only do the filtering inside. Move the HTML generation outside the loop as well.
$HTMLReportItems = foreach ($Filename in $FileList) {
$csv = Import-Csv (Join-Path $FilePath $Filename)
$CustName = $csv[0].CustName
$data = $csv | select FileName,FileDate
Write-Host "Reading data for $CustName"
for ($counter=0;$counter -lt 31;$counter++){
$CheckShortDate = (Get-Date).AddDays(-$counter).ToShortDateString()
$TempData = $data | ? {$_.FileDate -eq $CheckShortDate}
if ($TempData -eq $null) {
"No file found for $CheckShortDate for $CustName"
}
}
}
$HTMLReportItems = $HTMLReportItems | ConvertTo-Html -Fragment