Compare multiple folders for file differences - powershell

Question

I began to compare 2 folder structures to find files that did not match by date and size, but the requirment has been changed to 4 folders and I am stuck.
So here is what I am trying to do:
We upload several hundred folders\files to 4 different servers. The files must all match. Sometimes a file will not copy properly. So I need a script to read all four directories and compare all the files to make sure they match by size and date.
Output should only be a simple list that shows me the files that didn't match.
Any ideas?
Thanks.
I can do two folders but am lost on four. Also, this output is confusing. Not sure how to only list those that don't match.
$path1 = "\\path\folder
$path2 = "\\path\folder1
$dif = Compare-Object -ReferenceObject $path1 -DifferenceObject $path2 -Property FullName, Length, LastWriteTime
$dif | ft -AutoSize

Answer 1

I'd go about it with a hash based approach, and possibly use a database table somehwere to help yourself out. BTW, PSCX has the Get-Hash commandlet which will help you do this.
Basic approach
Traverse each server's desired folder-tree (you want to do this on the servers involved for performance reasons, not over a network share!) and generate a hash on each file you find. Store the hash and the full path and server name somewhere, preferrably a database table accessible from all four servers--it'll make processing much easier.
Then, if you've used a database table, write a few simple queries:
find any hash where there are fewer than 4 instances of the hash.
find any file path (you may have to process the path string to get it to the same relative root for each server ) where there are differing hashes (although this might be covered by 1. above).
All of this can be done from within PS, of course.
Why this way of doing things may be helpful
You don't have to run a four-way Compare-Object. The hashes serve as your point of comparison.
Your Powershell code to generate the hashes is one identical function that gets run on each server.
It scales. You could easily do this for 100 folders.
You end up with something easily manipulated and "distributed",i.e. accesible to the servers involved--the database table.
Downside
PSCX Get-Hash isn't very fast. This can easily be remedied by having PS fire some faster hash generating command, such as this one, md5sums.
How to do without using a database table
1. Write the hashes, file paths, severnames to files on each server as you are processing folders for hashes, and bring those files back when done.
2. Process the files into a hash table that keys on the hashcodes and counts each hash code.
3. You can have a parallel hash table (built at that same time as 2. while you pass throug the result files) that keys on each hash code to an array of paths/servers for that hash code.
4. Look for hash codes in hash table 1 with a count of less than 4. Use parallel hash table 2 to look up hash codes found with a count less 4, to find out what the file path(s) and server(s) were.

Answer 2

Try this:
Remember that the PrimaryPath has to be a masterlocation(contents are correct). Also, be consistent with how you write the paths(if you include the \ or not). Ex. Either use c:\folders\folder1\ for all paths or c:\folders\folder1.
Compare.ps1
Param(
[parameter(Mandatory=$true)] [alias("p")] [string]$PrimaryPath,
[parameter(Mandatory=$true)] [alias("c")] [string[]]$ComparePath
)
#Get filelist with relativepath property
function Get-FilesWithRelativePath ($Path) {
Get-ChildItem $Path -Recurse | ? { !$_.PSIsContainer } | % {
Add-Member -InputObject $_ -MemberType NoteProperty -Name RelativePath -Value $_.FullName.Substring($Path.Length)
$_
}
}
#If path exists and is folder
if (Test-Path $PrimaryPath -PathType Container) {
#Get master fileslist
$Masterfiles = Get-FilesWithRelativePath (Resolve-Path $PrimaryPath).Path
#Compare folders
foreach ($Folder in $ComparePath) {
if (Test-Path $Folder -PathType Container) {
#Getting filelist and adding relative-path property to files
$ResolvedFolder = (Resolve-Path $Folder).Path
$Files = Get-FilesWithRelativePath $ResolvedFolder
#Compare and output filepath to missing or old file
Compare-Object -ReferenceObject $Masterfiles -DifferenceObject $Files -Property RelativePath, Length, LastWriteTime | ? { $_.SideIndicator -eq "<=" } | Select #{n="FilePath";e={Join-Path $ResolvedFolder $_.RelativePath}}
} else { Write-Error "$Folder is not a valid foldername. Foldertype: Compare" }
}
} else { Write-Error "$PrimaryPath is not a valid foldername. Foldertype: Master" }