Get-ChildItem script hangs due to large object? - powershell

Question

Okay - I am brand new to PowerShell. I only started using it two weeks ago. I've scoured the web to create some scripts and now I'm trying something that seems a bit advanced and I'm uncertain how I should solve this.
I'm creating an audit script to determine what files are different between two backup repositories to ensure they've been properly synchronized (the synchronization scripts use robocopy and they've failed more than once without producing an error). The folders are quite extensive and upon occasion, I'm finding that the script just hangs on certain folders (always on the largest of them) and it will never complete due to this.
At first, I was using Get-ChildItem on the full source path, but that created a memory problem and the script would never complete. So, I thought I'd enumerate the child directories and perform a compare on each child directory... but depending on the folder, that goes bad as well.
Here is the script (using Powershell 2):
$serverArray=#("Server1","Server2","Server3")
for ($i=0; $i -lt 8; $i++) {
$server = $serverArray[$i]
$source="\\$server\Share\"
$destination = "D:\BackupRepository\$server"
# Copy to removable drive
$remoteDestination = "T:\BackupRepository\" + $server
$log = $server + "ShareBackup.log"
$remoteLog = "Remote_" + $server + "ShareBackup.log"
$logDestination = $localLogPath + $log
$logUNCDestination = $uncLogPath + $log
$logRemoteDestination = $localLogPath + $remoteLog
$logUNCRemoteDestination = $uncLogPath + $remoteLog
## This file is used for the process of checking
## whether or not the backup was successful
$backupReport = $localReportPath + $server + "ShareBackupReport.txt"
$remoteBackupReport = $localReportPath + "Remote_" + $server + "ShareBackupReport.txt"
## Variables for the failure emails
$failEmailSubject = "AUDIT REPORT for " + $server
$failRemoteEmailSubject = "AUDIT REPORT for " + $server
$failEmailBody = "The Audit for " + $server + " has found a file mismatch. Please consult the attached Backup Report."
$failRemoteEmailBody = "The Audit of the Remote Backups for " + $server + " has found a file mismatch. Please consult the attached Backup Report."
$sourceFolderArray = Get-ChildItem $source | ?{ $_.PSIsContainer }
$sourceFolderCount = $sourceFolderArray.Count
$mismatchCount = 0
$remoteMismatchCount = 0
for ($s1=0; $s1 -lt $sourceFolderCount; $s1++) {
$sourceFolder = $sourceFolderArray[$s1].FullName
$sourceFolderName = $sourceFolderArray[$s1].Name
$destFolder = $destination + "\" + $sourceFolderName
$remoteDestFolder = $remoteDestination + "\" + $sourceFolderName
Write-Host "Currently working on: " $sourceFolderName
$shot1 = Get-ChildItem -recurse -path $sourceFolder
$shot2 = Get-ChildItem -recurse -path $destFolder
$shot3 = Get-ChildItem -recurse -path $remoteDestFolder
$auditReportDest = "C:\BackupReports\Audits\"
$auditReportOutput = $auditReportDest + $server + "_" + $sourceFolderName + ".txt"
$auditReportRemoteOutput = $auditReportDest + $server + "_Remote_" + $sourceFolderName + ".txt"
$auditMismatchReport = $auditReportDest + "MismatchReport_" + $numericDate + ".txt"
Compare-Object $shot1 $shot2 -PassThru > $auditReportOutput
Compare-Object $shot2 $shot3 -PassTHru > $auditReportRemoteOutput
$auditCompare = Get-ChildItem $auditReportOutput
$auditRemoteCompare = Get-ChildItem $auditReportRemoteOutput
if ($auditCompare.Length -gt 0) {
$content = Get-ChildItem -Recurse $auditReportOutput
Add-Content $auditMismatchReport $content
Write-Host "Mismatch FOUND: " $sourceFolderName
$mismatchCount = $mismatchCount + 1
}
if ($auditRemoteCompare.Length -gt 0) {
$remoteContent = Get-ChilItem -Recurse $auditReportRemoteOutput
Add-Content $auditMismatchReport $remoteContent
Write-Host "Remote Mismatch FOUND: " $sourceFolderName
$remoteMismatchCount = $remoteMismatchCount + 1
}
}
send-mailmessage -from $emailFrom -to $emailTo -subject "AUDIT REPORT: Backups" -body "The full mismatch report is attached. There were $mismatchCount mismatched folders found and $remoteMismatchCount remote mismatched folders found. Please review to ensure backups are current." -Attachments "$auditMismatchReport" -priority High -dno onSuccess, onFailure -smtpServer $emailServer
}
What I've discovered when run interactively is that I'll get a "Currently working on FolderName" and if that object is "too large" (whatever that is), the script will just sit there at that point giving no indication of any error, but it will not continue (I've waited hours). Sometimes I can hit Ctrl-C interactively and rather than quitting the script, it takes the interrupt as a cancel for the current process and moves to the next item.
The rub is, I need to schedule this to happen daily to ensure the backups remain synchronized. Any help or insight is appreciated. And, yes, this is probably raw and inelegant, but right now I'm just trying to solve how I can get around the script hanging on me.

Answer 1

Not sure what version of PS you're using, but Get-Childitem has known problems scaling to large directories:
http://blogs.msdn.com/b/powershell/archive/2009/11/04/why-is-get-childitem-so-slow.aspx
If you're just comparing file names, you can get much better results in large directory structures using the legacy dir command. The /b (bare) switch returns just the fullname strings that can be readily used with Powershell's comparison operators.
$sourcedir = 'c:\testfiles'
$source_regex = [regex]::escape($sourcedir)
(cmd /c dir $Sourcedir /b /s) -replace "$source_regex\\(.+)$",'$1'
This uses a regular expression and the -replace operator to trim the soruce directory off of the fullnames returned by dir. The -replace operator will work with arrays, so you can do all of them in one operation without a foreach loop.