Powershell script been running for days when doing comparison - powershell

Question

I got a powershell query, it works fine for smaller amount of data but i am trying to run my CSV against a folder which has multiple folders and files within. Folder size is nearly 800GB and 180 folders within.
I want to see if the file exists in the folder, I can manually search the files within Windows and does not take to long to return a result but my CSV has 3000 rows and i do not wish to do this for 3000 rows. My script works fine for a smaller amount of data.
The script has been running for 6 days and it has not generated a file with data as of yet. it is 0KB and I am running it via task scheduler.
Script is below.
$myFolder = Get-ChildItem 'C:\Test\TestData' -Recurse -ErrorAction
SilentlyContinue -Force
$myCSV = Import-Csv -Path 'C:\Test\differences.csv' | % {$_.'name' -replace "\\", ""}
$compare = Compare-Object -ReferenceObject $myCSV -DifferenceObject $myFolder
Write-Output "`n_____MISSING FILES_____`n"
$compare
Write-Output "`n_____MISSING FILES DETAILS____`n"
foreach($y in $compare){
if($y.SideIndicator -eq "<="){
write-output "$($y.InputObject) Is present in the CSV but not in Missing folder."
}
}
I then created another script which runs the above script and contains an out file command and runs with Task scheduler.
C:\test\test.ps1 | Out-File 'C:\test\Results.csv'
is there a better way of doing this?
Thanks

Answer 1

is there a better way of doing this?
Yes!
Add each file name on disk to a HashSet[string]
the HashSet type is SUPER FAST at determining whether it contains a
specific value or not, much faster than Compare-Object
Loop over your CSV records, check if each file name exists in the set from step 1
# 1. Build our file name index using a HashSet
$fileNames = [System.Collections.Generic.HashSet[string]]::new()
Get-ChildItem 'C:\Test\TestData' -Recurse -ErrorAction
SilentlyContinue -Force |ForEach-Object {
[void]$fileNames.Add($_.Name)
}
# 2. Check each CSV record against the file name index
Import-Csv -Path 'C:\Test\differences.csv' |ForEach-Object {
$referenceName = $_.name -replace '\\'
if(-not $fileNames.Contains($referenceName)){
"${referenceName} is present in CSV but not on disk"
}
}
Another option is to use the hash set from step 1 in a Where-Object filter:
$csvRecordsMissingFromDisk = Import-Csv -Path 'C:\Test\differences.csv' |Where-Object { -not $fileNames.Contains($_) }