Looping through directories from CSV file - powershell

I have a working PS script that's used to compare the contents of two directories and report any missing files or files that have different contents. This is what it currently does:
Takes two specific directories.
Pulls all of the files from each directory (minus any excluded paths/files).
Checks to see if any files are missing from one or the other.
For each file that is in both directories, it does a hash comparison of that file's contents.
Places the results in variables based on files in the source but not the destination and vice-versa, and files that were in both the source and destination but had different contents.
Below is the main chunk of code that does all this. What I need to change/add is the ability to list ONLY the specific paths I want to be compared between the two servers. For example, say I want the following paths to be used for the comparison:
D:\Files\Stuff
D:\Files\Contents\Folders\MoreStuff
D:\Files\Archive
I would want each of those directories compared between their counterpart on the other server. Meaning it would compare the D:\Files\Stuff path between server 1 and server 2. It would NOT compare the paths with each other, though. Meaning, I DON'T want it to compare D:\Files\Stuff with D:\Files\Archive, regardless of the server.
What's the best way I can achieve this?
$SourceDir = "\\12345-serverP1\D$\Files";
$DestDir = "\\54321-serverP2\D$\Files";
#The excluded array holds all of the specific paths, files, and file types you don't want to be compared.
$ExcludedPaths = #(Import-Csv -LiteralPath 'D:\ExclusionList.csv') |Select-Object -Expand ExcludedPaths;
$ExcludedFiles = #(Import-Csv -LiteralPath 'D:\ExclusionList.csv') |Select-Object -Expand ExcludedFiles;
#Script block which stores the filter for the Where-Object used to exclude chosen paths.
#This script is called with the -FilterScript parameter below.
$Filter = {
$FullName = $_.FullName
-not($ExcludedPaths | Where-Object {$FullName -like "$_*";})
}
#Grabs all files from each directory, minus the excluded paths and files, and assigns them to variables based on Source and Destination.
try {$SourceFiles = Get-ChildItem -Recurse -Path $SourceDir -Exclude $ExcludedFiles -Force -ErrorAction Stop | Where-Object -FilterScript $Filter;}
catch {Write-Output "$(Get-Date) The following Source path was not found: $SourceDir" | Out-File $ErrorLog -Append;}
try {$DestFiles = Get-ChildItem -Recurse -Path $DestDir -Exclude $ExcludedFiles -Force -ErrorAction Stop | Where-Object -FilterScript $Filter;}
catch {Write-Output "$(Get-Date) The following Destination path was not found: $DestDir" | Out-File $ErrorLog -Append;}
#Pulls the name of each file and assigns it to a variable.
$SourceFileNames = $SourceFiles | % { $_.Name };
$DestFileNames = $DestFiles | % { $_.Name };
#Empty variables to be used in the loops below.
$MissingFromDestination = #();
$MissingFromSource = #();
$DifferentFiles = #();
$IdenticalFiles = #();
#Iterates through each file in the Source directory and compares the name against each file in the Destination directory.
#If the file is missing from the Destination, it is added to the MissingFromDestination variable.
#If the file appears in both directories, it compares the hash of both files.
#If the hash is the same, it adds it to the IdenticalFiles variable. If the hash is different, it adds the Source file to the DifferentFiles variable.
try {
foreach ($f in $SourceFiles) {
if (!$DestFileNames.Contains($f.Name)) {$MissingFromDestination += $f;}
elseif ($DestFileNames.Contains($f.Name)) {$IdenticalFiles += $f;}
else {
$t = $DestFiles | Where { $_.Name -eq $f.Name };
if ((Get-FileHash $f.FullName).hash -ne (Get-FileHash $t.FullName).hash) {$DifferentFiles += $f;}
}
}
}
catch {Write-Output "$(Get-Date) The Destination variable is null due to an incorrect path." | Out-File $ErrorLog -Append;}
#Iterates through each file in the Destination directory and compares the name against each file in the Source directory.
#If the file is missing from the Source, it is added to the MissingFromSource variable.
try {
foreach ($f in $DestFiles) {
if (!$SourceFileNames.Contains($f.Name)) {$MissingFromSource += $f;}
}
}
catch {Write-Output "$(Get-Date) The Source variable is null due to an incorrect path." | Out-File $ErrorLog -Append;}

Related

Merge CSV files in subfolders

I have found a script which does everything that I need, but it's only useful if you run it in a single folder. What I'd like is:
Script is located in c:/temp/. Upon running the script, it would go into each subfolder and execute. Each subfolder would then have a separate Final.csv.
Somebody mentioned just add -Recurse, but it doesn't complete the job as described. With -Recurse added, it goes into each subfolder and creates a Final.csv final in the root dir (C:/temp/) instead of creating a Final.csv in each subfolder.
$getFirstLine = $true
get-childItem *.csv | foreach {
$filePath = $_
$lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 2}
}
$getFirstLine = $false
Add-Content Final.csv $linesToWrite
}
If you are certain the csv files combined this way will leave you a valid 'Final.csv', you need to use Group-Object in order to create a combined file in each of the directories where the csv files to combine are found.
Suppose you have a folder with subfolders 'Folder1' and 'Folder2', both having csv files in them like these:
first.csv
Lorem,Ipsum,Dolor,Sic,Amet
data1-1,data1-2,data1-3,data1-4,data1-5
data2-1,data2-2,data2-3,data2-4,data2-5
second.csv
Lorem,Ipsum,Dolor,Sic,Amet
something,blah,whatever,very important,here's more..
Then this should do it for you:
$targetFileName = 'Final.csv'
# loop over the CSV files, but exclude the Final.csv file
# Group the files by their DirectoryNames
Get-ChildItem -Path 'D:\Test' -Filter '*.csv' -File -Recurse -Exclude $targetFileName | Group-Object DirectoryName | ForEach-Object {
# reset the $getFirstLine variable for each group
$getFirstLine = $true
# create the target path for the combined csv inside this folder.
# ($_.Name is the name of the group, which is the Directory name of the files inside the group)
$target = Join-Path -Path $_.Name -ChildPath $targetFileName
foreach ($file in $_.Group) {
if ($getFirstLine) {
# copy the first CSV as a whole
Get-Content -Path $file.FullName | Set-Content -Path $target
$getFirstLine = $false
}
else {
# add the content of the next file(s) without the header line
Get-Content -Path $file.FullName | Select-Object -Skip 1 | Add-Content -Path $target
}
}
}
The end result is that each subfolder will have a new 'Final.csv' file containing
Lorem,Ipsum,Dolor,Sic,Amet
data1-1,data1-2,data1-3,data1-4,data1-5
data2-1,data2-2,data2-3,data2-4,data2-5
something,blah,whatever,very important,here's more..
Of course I'm just showing an example for one of the subfolders.. Other subfolders will contain different 'Final.csv' content

copy matching files from a drive to a folder based on fodlername in powershell

I have a a bunch of language folders present in a directory under E:\Data\ like hu-hu, de-de etc.. on the other hand i have a bunch of file names in G:\ that contain the part of folder name for e.g.
amd64.de-de_OCR.cab,amd64.handwriting.de-de.cab
I need to copy all matching file names based on the foldername
for e.g. de-de should copy all matching files in G:\ i.e. both amd64.de-de_OCR.cab,amd64.handwriting.de-de.cab
This is the code i have so far but it is not copying over the files, and i am not sure how to proceed next, any help is appreciated.
$listfoldername = Get-ChildItem -Path "E:\Data" -Recurse -Directory -Force -ErrorAction SilentlyContinue | Select-Object Name
$destfolder = Get-ChildItem -Path "E:\Data" -Recurse -Directory -Force -ErrorAction SilentlyContinue | Select-Object FullName
$filename = Get-ChildItem -file G:\
if($filename -like $listfoldername)
{
Copy-Item -Path $filename -Destination $destfolder
}
There's a few issues with your code
The main issue with your code is that you are trying to use the -like operator to compare two objects (your object containing the directories you wish to move files to, and the object containing the files.
What you need to do is loop through each file and directory, one by one, to determine if the directory name (e.g. "hu-hu" is found in the filename (e.g. amd64.hu-hu_OCR.cab)
You'll want to use the wildcard indicator "*" with the -like operator (e.g. "*hu-hu*")
This below code snippet should do the trick. I tested using the file and folder names you've provided.
"G:" contains the folders:
de-de
hu-hu
us-us (note, I added this to make sure the code did not match this directory)
"E:\Data" contains the files
amd64.de-de_OCR.cab
amd64.handwriting.de-de.cab
amd64.handwritinghu-hu.cab
amd64.hu-hu_OCR.cab
$FileDirectory = "G:" # Change to "G:\", the trailing slash breaks syntax highlight on SO
$DataDirectory = "E:\Data"
$listfoldername = Get-ChildItem -Path "$DataDirectory" -Recurse -Directory -Force -ErrorAction SilentlyContinue | Select-Object Name
$filename = Get-ChildItem -file "$FileDirectory"
#Loop through each file one at a time
foreach ($file in $filename) {
# Then, loop through each folder one at a time
foreach ($folder in $listfoldername) {
# Set the current filename and listfoldername to variables for later -like operator
$FileString = $file.Name
$FolderString = $folder.Name
# If the current file "is like" the current folder name
if($FileString -like "*$FolderString*")
{
# Set the name of the current folder to a variable
$DataFolder = $folder.Name
Copy-Item -Path "$FileDirectory\$FileString" -Destination "$DataDirectory\$DataFolder"
} else {
Write-Output ("$FolderString pattern not found in $FileString")
}
}
}
I think you should start off by getting a list of possible language target folders. Then loop over the path where the files are, filtering their names to have at least the dash in it and next test if any of the language target folders matches the filename.
Something like this:
$langFolder = 'E:\Data'
$fileFolder = 'G:\' #'# dummy comment to fix syntax highlighting in SO
# get a list of the language folders
# if the languages folder has multiple subdirectories to include, add -Recurse here
$targetFolders = Get-ChildItem -Path $langFolder -Directory
# get a list of FileInfo objects for the files in the G:\ path
# if you need to search subdirectories aswell, add -Recurse here
$files = Get-ChildItem -Path $fileFolder -File -Filter '*-*.*'
foreach($file in $files) {
# check if a language name matches the file name
foreach($folder in $targetFolders) {
if ($file.BaseName -like "*$($folder.Name)*") {
# we have found a matching language target directory
$file | Copy-Item -Destination $folder.FullName
break # exit this folder foreach loop and get on with the next file
}
}
}
P.S. If all the files are .cab files you could speed up by setting the Filter to '*-*.cab' in line $files = Get-ChildItem ...

How do I create a new files automatically depend on an existing variable in PowerShell?

I have many file in a folder, I would like to check the existing and matching of the file with variable that I initialize. Then, if the file exit and match, I want to get some information from the files (many file), then create a new file depend on how many file exist and match.
I tried this code, I can check the matching and existing file. I can create a new file and get the information from the file, but I only can create 1 file.
The information that I get from the file, each file is different.
$ID = "123"
$Pre = "ABC"
$Path = "C:\Folder"
$PO = Get-ChildItem -Path $Path
foreach ($File in $PO) {
if (($File.Name -match $ID) -and ($File.Name -match $Pre)) {
Write-Host ">>POfile Found: $File"
} else {
Write-Host ">>Check Again!"
}
}
# CREATE FILE
$Jb_Path = "C:\Folder\Jb"
## GET INFORMATION
$count = 1
$Get_PO = Get-ChildItem -Path $Path\$File -File -Recurse
$POfile = Get-Random -InputObject $Get_PO -Count $count
Write-Host ">>Selected POfile= $POfile"
$FilteredContents = Get-Content $POfile | Where-Object {$_ -like "*;INFO*"}
$Get_INFO = $FilteredContents.Substring(5,2)
## NEW FILE
New-Item -Path $Jb_Path\NEW_$Pre$ID-$Get_INFO.txt -Force
In the section # CREATE FILE you are referencing the variable $File which has the last value iterated in the previous foreach (even if it didn't match the if condition).
Asuming the $Pre is for prefix and comes first in a file name simply do a
Get-ChildItem "$Path\*$Pre*$ID*"
to only get file names for your criteria.
As $File contains only one file name a Get-Random doesn't make sense, especially as it might not contain a line with ;INFO
Assuming the two characters to extract are in front of ;INFO this untested script might do:
$Pre = "ABC"
$ID = "123"
$Path = "C:\Folder"
$Jb_Path= "C:\Folder\Jb"
Get-ChildItem "$Path\*$Pre*$ID*" | Get-Content |
Select-String -Pattern '^.....(..).*;INFO' |
Get-Random | ForEach-Object {
$NewFile = Join-Path $Jb_Path ('NEW_{0}{1}-{2}.txt' -f $Pre,
$ID,$_.Matches.Groups[1].Value)
New-Item -Path $NewFile -ItemType File -Force -WhatIf
}
It will only output what it would do without the -WhatIf parameter.
If no file matching the criteria and RegEx pattern is found it will silently continue.
If my assumptions led me wrong, enhance your question be editing it with more details.

How to copy a file then save the destination path to CSV

I'm trying to copy a batch of files (those whose filename begins with 6 digits) from a temp folder to a permanent location, excluding those that already exist in the new location.
Once the copy is done, I want to export the filename and new path of the copied file into a CSV.
It's pretty easy to get the old file location and export to CSV, I'm just not quite sure how to get the new file location.
My script looks like this:
# Prompt for file origin
$file_location = Read-Host -Prompt "Where do you want your files to come from?"
# Prompt for file destination
$file_destination = Read-Host -Prompt "Where do you want your files to go? `n(They won't be copied if they're already there)"
# Save contents of file destination - used to check for duplicates
$dest_contents = Get-ChildItem $file_destination
<# For all of the files of interest (those whose names begin with 6 digits) in $file_location,
determine whether that filename already exists in the target directory ($file_destination)
If it doesn't, copy the file to the destination
Then save the filename and the new** filepath to a CSV #>
Get-ChildItem -Path $file_location |
Where-Object { $_.Name -match '^\d{6}' -And !($dest_contents -Match $_.Name ) } |
Copy-Item -Destination $file_destination -PassThru |
Select-Object -Property Name, FullName | # **Note: This saves the old filepath, not the new one
Export-CSV -Path "$file_location\files_copied.csv" -NoClobber
You can do this with a few changes to the code
# Prompt for file origin
$file_location = Read-Host -Prompt "Where do you want your files to come from?"
# Prompt for file destination
$file_destination = Read-Host -Prompt "Where do you want your files to go? `n(They won't be copied if they're already there)"
# Save contents of file destination - used to check for duplicates
$dest_contents = Get-ChildItem $file_destination | Select-Object -ExpandProperty Name
<# For all of the files of interest (those whose names begin with 6 digits) in $file_location,
determine whether that filename already exists in the target directory ($file_destination)
If it doesn't, copy the file to the destination
Then save the filename and the new** filepath to a CSV #>
Get-ChildItem -Path $file_location |
Where-Object { $_.Name -match '^\d{6}' -and ($dest_contents -notcontains $_.Name ) } |
ForEach-Object {
Copy-Item -Path $_.FullName -Destination $file_destination
# emit a PSObject storing the Name and (destination) FullName of the file that has been copied
# This will be used to generate the output in the 'files_copied.csv'
New-Object -TypeName PSObject -Property ([ordered]#{ Name = $_.Name; FullName = (Join-Path $file_destination $_.Name)})
} |
Export-CSV -Path "$file_location\files_copied.csv" -NoTypeInformation -Force
Note that i only gather the Names of the files already in the destination path instead of the fileInfo objects. This makes it a lot 'leaner' since the only reason for gathering is to have a collection of file names to compare with.
As it is now, you have a fixed name for the 'files_copied.csv' and personally i think it would be a good idea to make that more generic by adding the current date to it for instance like
Export-CSV -Path ("{0}\files_copied_{1}.csv" -f $file_location, (Get-Date).ToString("yyyy-MM-dd")) -NoTypeInformation -Force
P.s. I'm using the [ordered] here so the output will always have the properties in the same order. This however requires PowerShell v3 or better.
Also, i suggest looking at the -File or -Attributes switches on the Get-ChildItem command if you need to make sure the code only copies Files, not Directories. If your version of PowerShell is 2.0, you can use the Where-Object { !$_.PSIsContainer } construct to filter out files only.
I'd use a different approach testing if destination exist while iterating source files.
To append to a present csv (log) file you need to remove the header of the new one.
As Get-ChildItem allows ranges [0-9] you can directly select leading numbers
## Q:\Test\2018\08\08\SO_51738853.ps1
# Prompt for file origin
$SrcDir = Read-Host -Prompt "Where do you want your files to come from?"
# Prompt for file destination
$DstDir = Read-Host -Prompt "Where do you want your files to go? `n(They won't be copied if they're already there)"
$SrcFiles = Join-Path (Get-Item $SrcDir).Fullname [0-9][0-9][0-9][0-9][0-9][0-9]*
$CopiedFiles = ForEach ($SrcFile in (Get-ChildItem -Path $SrcFiles)){
$DstFile = Join-Path $DstDir $SrcFile.Name
If (!(Test-Path $DstFile)){
$SrcFile | Copy-Item -Destination $DstFile -PassThru |
Select-Object -Property Name, FullName
}
}
# Check if log file exists, if yes append, otherwise export.
$LogFile = Join-Path $SrcDir "files_copied.csv"
If ($CopiedFiles){
If (!(Test-Path $LogFile)){
Export-CSV -Path $LogFile -InputObject $CopiedFiles -NoTypeInformation
} else {
# We need to remove the Header appending to the present csv
$CopiedFiles | ConvertTo-Csv -NoType | Select-Object -Skip 1 | Add-Content -Path $LogFile
}
}

Trying to iterate through folders and take recursive actions on each file

I'm trying to build a script that I can use to delete old files based on Last Accessed date. As part of the script I want to interrogate each sub folder, find files not accessed in the last X days, create a log in the same folder of the files found and record file details in the log then delete the files.
What I think I need is a nested loop, loop 1 will get each subfolder (Get-ChildItem -Directory -Recurse) then for each folder found a second loop checks all files for last accessed date and if outside the limit will append the file details to a logfile in the folder (for user reference) and also to a master logfile (for IT Admin)
loop 1 is working as expected and getting the subfolders, but I cannot get the inner loop to recurse through the objects in the folder, I'm trying to use Get-ChildItem inside the first loop, is this the correct approach?
Code sample below, I have added pseudo to demo the logic, its really the loops I need help with:
# Set variables
$FolderPath = "E:TEST_G"
$ArchiveLimit = 7
$ArchiveDate = (Get-Date).AddDays(-$ArchiveLimit)
$MasterLogFile = "C:\Temp\ArchiveLog $(Get-Date -f yyyy-MM-dd).csv"
# Loop 1 - Iterate through each subfolder of $FolderPath
Get-ChildItem -Path $FolderPath -Directory -Recurse | ForEach-Object {
# Loop 2 - Check each file in the Subfolder and if Last Access is past
# $ArchiveDate take Action
Get-ChildItem -Path $_.DirectoryName | where {
$_.LastAccessTime -le $ArchiveDate
} | ForEach-Object {
# Check if FolderLogFile Exists, if not create it
# Append file details to folder Log
# Append File & Folder Details to Master Log
}
}
I think you're overcomplicating a bit:
#Set Variables
$FolderPath = "E:\TEST_G"
$ArchiveLimit = 7
$ArchiveDate = (Get-Date).AddDays(-$ArchiveLimit)
$MasterLogFile = "C:\Temp\ArchiveLog $(get-date -f yyyy-MM-dd).csv"
If (!(Test-Path $MasterLogFile)) {New-Item $MasterLogFile -Force}
Get-ChildItem -Path $FolderPath -File -Recurse |
Where-Object { $_.LastAccessTime -lt $ArchiveDate -and
$_.Extension -ne '.log' } |
ForEach-Object {
$FolderLogFile = Join-Path $_.DirectoryName 'name.log'
Add-Content -Value "details" -Path $FolderLogFile,$MasterLogFile
Try {
Remove-Item $_ -Force -EA Stop
} Catch {
Add-Content -Value "Unable to delete item! [$($_.Exception.GetType().FullName)] $($_.Exception.Message)"`
-Path $FolderLogFile,$MasterLogFile
}
}
Edit:
Multiple recursive loops are unnecessary since you're already taking a recursive action in the pipeline. It's powerful enough to do the processing without having to take extra action. Add-Content from the other answer is an excellent solution over Out-File as well, so I replaced mine.
One note, though, Add-Content's -Force flag does not create the folder structure like New-Item's will. That is the reason for the line under the $MasterLogFile declaration.
Your nested loop doesn't need recursion (the outer loop already takes care of that). Just process the files in each folder (make sure you exclude the folder log):
Get-ChildItem -Path $FolderPath -Directory -Recurse | ForEach-Object {
$FolderLogFile = Join-Path $_.DirectoryName 'FolderLog.log'
Get-ChildItem -Path $_.DirectoryName -File | Where-Object {
$_.LastAccessTime -le $ArchiveDate -and
$_.FullName -ne $FolderLogFile
} | ForEach-Object {
'file details' | Add-Content $FolderLogFile
'file and folder details' | Add-Content $MasterLogFile
Remove-Item $_.FullName -Force
}
}
You don't need to test for the existence of the folder log file, because Add-Content will automatically create it if it's missing.