Using powershell to copy new and updated files - powershell

I am a complete novice when it comes to powershell, but I have been given a script that I need to improve so that we can move updated or new files from one server to another. I've managed to get to grips with the current script but am struggling to find the right cmdlets and paramters to achieve the desired behaviour.
The script I have is successful at detecting changed files and moving them to a location ready for transfer to another server, but it doesn't detect any new files.
Can anyone give me some guidance as to how I would be able to achieve both behaviours?
$CurrentLocation = "C:\current"
$PreviousLocation = "C:\prev"
$DeltaLocation = "C:\delta"
$source = #{}
#
# Get the Current Location file information
#
Get-ChildItem -recurse $CurrentLocation | Foreach-Object {
if ($_.PSIsContainer) { return }
$source.Add($_.FullName.Replace($CurrentLocation, ""), $_.LastWriteTime.ToString())
}
Write-Host "Content of Source"
$source
$changesDelta = #{}
$changesPrevious = #{}
#
# Get the Previous Directory contents and compare the dates against the Current Directory contents
#
Get-ChildItem -recurse $PreviousLocation | Foreach-Object {
if ($_.PSIsContainer) { return }
$File = $_.FullName.Replace($PreviousLocation, "")
if ($source.ContainsKey($File)) {
if ($source.Get_Item($File) -ne $_.LastWriteTime.ToString()) {
$changesDelta.Add($CurrentLocation+$File, $DeltaLocation+$File)
$changesPrevious.Add($CurrentLocation+$File, $PreviousLocation+$File)
}
}
}
Write-Host "Content of changesDelta:"
$changesDelta
Write-Host "Content of changesPrevious:"
$changesPrevious
#
# Copy the files into a temporary directory
#
foreach ($key in $changesDelta.Keys) {
New-Item -ItemType File -Path $changesDelta.Get_Item($key) -Force
Copy-Item $key $changesDelta.Get_Item($key) -Force
}
Write-Host $changesDelta.Count "Files copied to" $DeltaLocation
#
# Copy the files into the Previous Location to match the Current Location
#
foreach ($key in $changesPrevious.Keys) {
Copy-Item $key $changesDelta.Get_Item($key) -Force
}

Here's a simplified approach to your needs. One thing to note is that some of the constructs I've used require PSv3+. This does not copy directory structure, just the files. Additionally, it compares the basenames (ignoring extensions) which may or may not do what you want. It can be expanded to include extensions by using .Name instead of .BaseName
#requires -Version 3
$CurrentLocation = 'C:\current'
$PreviousLocation = 'C:\prev'
$DeltaLocation = 'C:\delta'
$Current = Get-ChildItem -LiteralPath $CurrentLocation -Recurse -File
$Previous = Get-ChildItem -LiteralPath $PreviousLocation -Recurse -File
ForEach ($File in $Current)
{
If ($File.BaseName -in $Previous.BaseName)
{
If ($File.LastWriteTime -gt ($Previous | Where-Object { $_.BaseName -eq $File.BaseName }).LastWriteTime)
{
Write-Output "File has been updated: $($File.FullName)"
Copy-Item -LiteralPath $File.FullName -Destination $DeltaLocation
}
}
Else
{
Write-Output "New file detected: $($File.FullName)"
Copy-Item -LiteralPath $File.FullName -Destination $DeltaLocation
}
}
Copy-Item -Path "$DeltaLocation\*" -Destination $PreviousLocation -Force

Related

Moving files not woking in powershell

I have a code:
cls
$folderPath = 'Y:\lit\Retail\abc\abc\FTP-FromClient\AMS\2022-03-23'
$files = Get-ChildItem $folderPath
$destination='Y:\lit\Retail\abc\abc\FTP-FromClient\AMS\2022-03-23\bz2'
$leftfiles=('20220322054419',
'20220322083959',
'20220322084715',
'20220322085207',
'20220322085250',
'20220322085858',
'20220322090236',
'20220322090410',
'20220322090450'
'20220322170306')
foreach($j in $leftfiles)
{
foreach($f in $files)
{
if ($f -contains $j)
{
Move-Item $f.FullName $destination
}
}
}
In this $folderPath I have some files:
$folderPath = 'Y:\lit\Retail\abc\abc\FTP-FromClient\AMS\2022-03-23'
File naming has some naming convention as which matches with the numbers inside the aray:
Colly_20220322054419.dat.bz2
Colly_1_20220322084715.dat.bz2
Colly_3_20220322085207.dat
I only need to move all the files to the destination ,if filename contains the number inside array elements. So I tried using -contains to move the file but it's not working:
foreach($j in $leftfiles)
{
foreach($f in $files)
{
if ($f -contains $j)
{
Move-Item $f.FullName $destination
}
}
}
Your code could be simplified if you use a Where-Object clause that filters the files to move using the regex -match operator
$folderPath = 'Y:\lit\Retail\abc\abc\FTP-FromClient\AMS\2022-03-23'
$destination = 'Y:\lit\Retail\abc\abc\FTP-FromClient\AMS\2022-03-23\bz2'
$leftfiles = '20220322054419', '20220322083959','20220322084715','20220322085207','20220322085250',
'20220322085858','20220322090236','20220322090410','20220322090450','20220322170306'
# first make sure the destination folder exists
$null = New-Item -Path $destination -ItemType Directory -Force
# join the items from $leftfiles with a pipe symbol (=the regex 'OR')
$regex = $leftfiles -join '|'
Get-ChildItem -Path $folderPath -File |
Where-Object { $_.BaseName -match $regex } |
Move-Item -Destination $destination
If you want to test this out first, append switch -WhatIf to the Move-Item line. THat way, nothing actually gets moved. Only in the console, it is displayed what would be moved

Get-ChildItem is using current Dir instead of Dir specified in variable

Ouline -
I have 300,000+ folders containing subfolders and files.
I am trying to flatten each directory so that subfolders are removed and all files are brought to their respective parent directory.
Unfortunately, the Get-ChildItem cmdlet runs in the location of the .ps1 file and not those specified in the .txt file.
I have been trying to troubleshoot this for hours, any help would be greatly apprecieated.
Process -
First, I run a .ps1 file that retrieves the parent folder locations from a text file and calls a custom module:
[System.IO.File]::ReadLines("C:\Users\ccugnet\Desktop\test.txt") | ForEach-Object {
Import-Module MyFunctions
fcopy -SourceDir $line -DestinationDir $line
Remove-Module MyFunctions
}
Second, the custom module moves the child items to the parent folder, appending an incrementing digit to the end of the file name for duplicate files:
function fcopy ($SourceDir,$DestinationDir)
{
Get-ChildItem $SourceDir -Recurse | Where-Object { $_.PSIsContainer -eq $false } | ForEach-Object {
$SourceFile = $_.FullName
$DestinationFile = $DestinationDir + $_
if (Test-Path $DestinationFile) {
$i = 0
while (Test-Path $DestinationFile) {
$i += 1
$DestinationFile = $DestinationDir + $_.basename + $i + $_.extension
}
} else {
Move-Item -Path $SourceFile -Destination $DestinationFile -Verbose -Force -WhatIf
}
Move-Item -Path $SourceFile -Destination $DestinationFile -Verbose -Force -WhatIf
}
}
Text file contents:
"C:\Users\ccugnet\Desktop\Dir_Flatten\Fox, Hound & Hunter"
"C:\Users\ccugnet\Desktop\Dir_Flatten\Cat, Hat"
"C:\Users\ccugnet\Desktop\Dir_Flatten\Alice"
"C:\Users\ccugnet\Desktop\Dir_Flatten\Beetle | Juice"
Your $line is empty
PS P:\> [System.IO.File]::ReadLines("C:\Users\ccugnet\Desktop\test.txt") | ForEach-Object {
$line
}
Try $_
PS P:\> [System.IO.File]::ReadLines("C:\Users\ccugnet\Desktop\test.txt") | ForEach-Object {
$_
}
"C:\Users\ccugnet\Desktop\Dir_Flatten\Fox, Hound & Hunter"
"C:\Users\ccugnet\Desktop\Dir_Flatten\Cat, Hat"
"C:\Users\ccugnet\Desktop\Dir_Flatten\Alice"
"C:\Users\ccugnet\Desktop\Dir_Flatten\Beetle | Juice"

Powershell - Copy files only from the current day and if they don't exist in the target directory

I'm trying to develop a script that copy .txt files from two source folders to one destination folder.
I would like as well to add 2 additional conditions:
If a file already exists in the target directory, skip it
The copied files must not be older than the current day
This is what I made, but it doesn't work as expected:
$SourceDirItemsMITEMAvail = "\\wipfs02\Data\webshop\test\ecomcloud_import\carhartt\product\*"
$SourceDirCaproBPRIC = "\\wipfs02\Data\webshop\test\hybris_import\carhartt\product\*"
$TargetDirB2B = "\\wipfs02\Data\webshop\test\ecomcloud_import\carhartt\b2b\product\"
$ExcludeSubDir = "archive", "IDISC Test", "save", "archive_B2B", "archive_tmp", "Errors"
# Copy only the specific files
$SpecificTypesEcomCloudImport = "avail*", "items*", "MITEM*"
$SpecificTypesHybrisImport = "BPRIC*", "CAPRO*"
# Get the current date
$CurrentDateFiles = (Get-Date).Date
# Copy items from source directories and transfer them to the B2B Folder
For ($i=0; $i -lt $SourceDirItemsMITEMAvail.Count -and $SourceDirCaproBPRIC.Count; $i++) {
# If the copied files don't already exist into the target directory, then copy them
if (-not(Test-Path -Path $SourceDirItemsMITEMAvail, $SourceDirCaproBPRIC)) {
try {
# The copied files must not be older that the current day
$CopyFilesEcomCloudImportDir = Copy-Item $SourceDirItemsMITEMAvail -Exclude $ExcludeSubDir -Destination $TargetDirB2B -PassThru -Verbose -Include $SpecificTypesEcomCloudImport | Where-Object { $_.LastWriteTime -eq $CurrentDateFiles}
Write-Host ""$CopyFilesEcomCloudImportDir.Count" file(s) have been copied from $SourceDirItemsMITEMAvail to $TargetDirB2B."
# The copied files must not be older that the current day
$CopyFilesHybrisImportDir = Copy-Item $SourceDirCaproBPRIC -Exclude $ExcludeSubDir -Destination $TargetDirB2B -PassThru -Verbose -Include $SpecificTypesHybrisImport | Where-Object { $_.LastWriteTime -eq $CurrentDateFiles}
Write-Host ""$CopyFilesHybrisImportDir.Count" file(s) have been copied from $SourceDirCaproBPRIC to $TargetDirB2B."
}
catch {
"Error: $($_.Exception.Message)"
}
}else {
Write-Host "Cannot copy $CopyFilesEcomCloudImportDir because a file with that name already exists in the destination folder."
Write-Host "Cannot copy $CopyFilesHybrisImportDir because a file with that name already exists in the destination folder."
}
}
Either it copies all the files from the sources directories, or it goes immediately to the else condition.
If I remove the if (Test-Path) condition, it copies all the files from the sources, even those who are older than the current day.
If I let the if (Test-Path) condition, it goes immediately to the else statement.
Could you help me to resolve this? Thanks!
Using just PowerShell, you could do this:
$SourceDirItemsMITEMAvail = "\\wipfs02\Data\webshop\test\ecomcloud_import\carhartt\product"
$SourceDirCaproBPRIC = "\\wipfs02\Data\webshop\test\hybris_import\carhartt\product"
$TargetDirB2B = "\\wipfs02\Data\webshop\test\ecomcloud_import\carhartt\b2b\product"
$ExcludeSubDir = "archive", "IDISC Test", "save", "archive_B2B", "archive_tmp", "Errors"
# create a regex of the folders to exclude
# each folder will be Regex Escaped and joined together with the OR symbol '|'
$notThese = ($ExcludeSubDir | ForEach-Object { [Regex]::Escape($_) }) -join '|'
# Copy only the specific files
##############################
# if all files are .txt files, change the patterns below to include that:
# like "avail*.txt", "items*.txt", "MITEM*.txt","BPRIC*.txt", "CAPRO*.txt"
##############################
$FileNamePattern = "avail*", "items*", "MITEM*","BPRIC*", "CAPRO*"
# Get the current date
$today = (Get-Date).Date
Get-ChildItem -Path $SourceDirItemsMITEMAvail, $SourceDirCaproBPRIC -Include $FileNamePattern -File -Recurse |
Where-Object { $_.DirectoryName -notmatch $notThese -and $_.LastWriteTime.Date -eq $today } |
ForEach-Object {
$targetFile = Join-Path -Path $TargetDirB2B -ChildPath $_.Name
if (Test-Path -Path $targetFile -PathType Leaf) {
Write-Host "File '$targetFile' already exists. Skipping"
}
else {
$_ | Copy-Item -Destination $TargetDirB2B -Force
}
}

Fix script to allow a scanning of all fields for every row to find if that field contains a target folder name

This is an deepening extension to solve a previously question
Since the input csv file is inconsistent, I cannot use that in this way.
The folder to move to is not always in the same column, so any code trying to use that as input will hit the problem of the value not corresponding to the folder I want to move to. It simply reads garbage ("spam") where it expects the folder name.
The only way to do it is by examining all fields for every row to find if that field contains a target folder name you can use. That means a LOT of Test-Path lines
This is the incriminated code part
Foreach ($fileName in $_.Group.FileName) {
$ValidFileName = $filename -replace $invalid
$targetFile = Join-Path -Path $TargetFolder -ChildPath $fileName
Error is explicitly telling it to move those folders when I iterate through the FileName column. Foreach ($fileName in $_.Group.FileName) {.. this is reading: (1959 10) Showcase Presents n 22,(1959 12) Showcase Presents n 23,alfa, da definire.
My request: Since that examination of all fields for every row is required so I ask for a code editing of this script. Also if this means a LOT of Test-Path lines I suppose that there is no alternative.
However this script below don't create folders and move anything so my request is to try to fix it
$csvpath = 'C:\temp\temp.csv'
$invalid = "[{0}]" -f [RegEx]::Escape(([IO.Path]::GetInvalidFileNameChars() -join ''))
$sourcePath = 'C:\temp\'
Import-Csv C:\temp\temp.csv -Header Title,FileName,Link -Delimiter ';' |
Group-Object Title |
Foreach {
# I prefer to add a trailing slash to folder names
$TargetFolder = Join-Path -Path $sourcePath -ChildPath (($_.Name -replace $invalid)+'\')
# We don't have to create the new folders, because -Force will create them for us
Foreach ($fileName in $_.Group.FileName) {
$ValidFileName = $filename -replace $invalid
$targetFile = Join-Path -Path $TargetFolder -ChildPath $fileName
# Write your values to the console - Make sure the folder is what it should be
Write-Output "Moving '$targetFile' to '$TargetFolder'"
Move-Item $targetFile $TargetFolder -Force -WhatIf
}
}
I was hesitant to even comment on this since you're just not taking any of our advise on all the previoous posts, and just repeating what we say in a new question. We are here to help with your code, and not write it for you but, learning isn't a crime so I will just assume you're just unfamiliar with powershell in general.
$csvpath = 'C:\tomp\temp.csv'
#$invalid = "[{0}]" -f [RegEx]::Escape(([IO.Path]::GetInvalidFileNameChars() -join ''))
$sourcePath = 'C:\tomp\'
Import-Csv $csvpath -Header Title,FileName,Link -Delimiter ',' |
Group-Object Title |
ForEach-Object `
-Begin {
$FoldersLocation = Get-ChildItem -Path C:\tomp -Directory
$Source_Folder = ($FoldersLocation.Parent.FullName | Select-Object -Unique)
#Create array list to hold both columns
[System.Collections.ArrayList]$CombinedRows = #()
} `
-Process {
# Create folder if it doesn't exist
$Destination = Join-Path -Path $Source_Folder -ChildPath $_.Name
if ((Test-Path -Path $Destination) -eq $false) {
"Created: $Destination"
#$null = New-Item -Path ($FoldersLocation.Parent.FullName | Select -Unique) -Name $_.Name -ItemType Directory
}
#region Combine two columns
Foreach ($fileName in $_.Group.FileName) {
$null = $CombinedRows.Add($fileName)
}
Foreach ($link in $_.Group.Link) {
$null = $CombinedRows.Add($link)
}
#endregion
} `
-End {
Foreach ($name in $FoldersLocation) {
if ($name.Name -in $CombinedRows) {
if ($name.Name -match "Showcase"){
# No need for output when supplying the -verbose switch
"Moving: '$($Name.FullName)' to 'C:\tomp\Lanterna Verde - Le Prime Storie'"
# Move-Item -Path $Name.FullName -Destination "C:\tomp\Lanterna Verde - Le Prime Storie"
}
else {
# No need for output when supplying the -verbose switch
"Moving: '$($Name.FullName)]' to 'C:\tomp\Batman DC'"
# Move-Item -Path $Name.FullName -Destination "C:\tomp\Batman DC"
}
}
}
}
All it comes down to is control of flow logic. If a certain condition is met, then do this, or do something else if its not. Added very few inline comments as most of it pretty self explanatory.
I recommend reading up on some powershell instead of hoping others could do everything for you.
Powershell

How to limited copy file using Powershell?

I want to check .jpg file in the 2nd folder. 2nd folder has some subfolder. if .jpg exist in the subfolder of 2nd folder, I will copy a file from 1st folder to subfolder of 2nd folder based on the base name. I can do this part refer to this answer
How to copy file based on matching file name using PowerShell?
But I want to do limitation while I copy a file from 1st folder. I will NOT copy the file, if I already copy the same file from 1st folder to 2nd folder 3 times.
This is the code from the reference.
$Job_Path = "D:\Initial"
$JobError = "D:\Process"
Get-ChildItem -Path "$OpJob_Path\*\*.jpg" | ForEach-Object {
$basename = $_.BaseName.Substring(15)
$job = "$Job_Path\${basename}.png"
if (Test-Path $job) {
$timestamp = Get-Date -Format 'yyyyMMddhhmmss'
$dst = Join-Path $_.DirectoryName "${timestamp}_${basename}.gif"
Copy-Item $job $dst -Force
}
Anyone can help me to solve this problem please. Thank you.
Updated
$Job_Path = "D:\Initial"
$JobError = "D:\Process"
Get-ChildItem -Path "$OpJob_Path\*\*.jpg" | ForEach-Object {
$basename = $_.BaseName.Substring(15)
$job = "$Job_Path\${basename}.png"
if (Test-Path $job) {
$timestamp = Get-Date -Format 'yyyyMMddhhmmss'
$dst = Join-Path $_.DirectoryName "${timestamp}_${basename}.gif"
$Get = (Get-ChildItem -Name "$OpJob_Path\*\*$basename.jpg*" | Measure-Object).Count
$Get
if ($Get -eq "3") {
Write-Host "Continue Other Process"
NEXT_PROCESS
} else {
Write-Host "Less than 3"
}
Copy-Item $job $dst -Force
}
$Get is not a hashtable, and it's also not keeping track of what's been copied already. You need to define the hashtable outside the loop
$copy_count = #{}
Get-ChildItem -Path "$OpJob_Path\*\*.jpg" | ForEach-Object {
...
}
and then update it whenever you copy a file
if ($copy_count[$_.Name] -le 3) {
Copy-Item ...
$copy_count[$_.Name]++
} else {
...
}