Our requirement is to copy files from source to the destination folder.
The clause is during the first run of the script everything should be copied but in the subsequent runs it should copy only those files which have not been copied till yet and are new ones.
The issue is from the destination folder we have a script that works on the files and remove them once executed. So we dont want duplicate files from source copied to destination.
Example
source-> abc.txt,def.xt
after 1strun
dest->abc.txt,def.txt
subsequent runs
source->abc.txt,def.xt, ghi.txt
dest->abc.txt,def.xt, ghi.txt
Now when another script has worked on dest folder and removed abc.txt and ghi.txt then the logic should be
source->abc.txt,def.xt, ghi.txt,jkl.txt
Now when the script runs it should only copy the new files
dest->ghi.txt, jkl.txt
I was thinking if we can log the output after the script is run for the first time to a txt file and then put a condition to check in that log file if the text file is there before copying anything from the source folder to the destination .
Hope was able to explain.
Thx
You could copy those files which are not in your history like this:
$sourceFolder = "..."
$destFolder = "..."
$historyFile = "history.txt"
$recurse = $false
$history = Get-Content $historyFile
Get-ChildItem $sourceFolder -Recurse:$recurse | ? {
-not $_.PSIsContainer -and $history -notcontains $_.FullName
} | % {
Copy-Item $_.FullName $destFolder
$_.FullName >> $historyFile
}
The line $_.FullName >> $historyFile appends the copied files to the history file.
Related
Here is a section of code from a larger script. The goal is to recurse through a source directory, then copy all the files it finds into a destination directory, sorted into subdirectories by file extension. It works great the first time I run it. If I run it again, instead of overwriting existing files, it fails with this error on each file that already exists in the destination:
Copy-Item : Cannot overwrite the item with itself
I try, whenever possible, to write scripts that are idempotent but I havn't been able to figure this one out. I would prefer not to add a timestamp to the destination file's name; I'd hate to end up with thirty versions of the exact same file. Is there a way to do this without extra logic to check for a file's existance and delete it if it's already there?
## Parameters for source and destination directories.
$Source = "C:\Temp"
$Destination = "C:\Temp\Sorted"
# Build list of files to sort.
$Files = Get-ChildItem -Path $Source -Recurse | Where-Object { !$_.PSIsContainer }
# Copy the files in the list to destination folder, sorted in subfolders by extension.
foreach ($File in $Files) {
$Extension = $File.Extension.Replace(".","")
$ExtDestDir = "$Destination\$Extension"
# Check to see if the folder exists, if not create it
$Exists = Test-Path $ExtDestDir
if (!$Exists) {
# Create the directory because it doesn't exist
New-Item -Path $ExtDestDir -ItemType "Directory" | Out-Null
}
# Copy the file
Write-Host "Copying $File to $ExtDestDir"
Copy-Item -Path $File.FullName -Destination $ExtDestDir -Force
}
$Source = "C:\Temp"
$Destination = "C:\Temp\Sorted"
You are trying to copy files from a source directory to a sub directory of that source directory. The first time it works because that directory is empty. The second time it doesn't because you are enumerating files of that sub directory too and thus attempt to copy files over themselves.
If you really need to copy the files into a sub directory of the source directory, you have to exclude the destination directory from enumeration like this:
$Files = Get-ChildItem -Path $Source -Directory |
Where-Object { $_.FullName -ne $Destination } |
Get-ChildItem -File -Recurse
Using a second Get-ChildItem call at the beginning, which only enumerates first-level directories, is much faster than filtering the output of the Get-ChildItem -Recurse call, which would needlessly process each file of the destination directory.
I have the following script block built into my script:
$changedItems = (Get-ChildItem $srcdirectory -Recurse | Where-Object { $_.LastWriteTime -gt (Get-Date).AddHours(-24)} | where { ! $_.PSIsContainer }).FullName
foreach ($changedItem in $changeditems){
$archivePath = Join-Path -Path $dstDirectory -ChildPath ($archiveName + "_" + $fileDate + ".zip")
$7zipArgs = #(
"a";
"-mx=9";
"-tzip";
$archivePath;
$changedItem;
)
7zip #7zipArgs
}
It is simply supposed to identify items, that have changed in the last 24 hours, and archive them into a zip directory.
The source directory, that I am querying, has a lot of data in it, so the following part of the block allows, so that only changed files are copied:
| where { ! $_.PSIsContainer }
This eliminates the following problem that happens with the -Recurse part of Get-Childitem:
If there was a folder, which had been changed recently, but no files inside of the folder were changed, the whole folder and it's complete content would be archived.
HOWEVER:
Now I only have the changed files that I want, but I don't have any hierarchy in my new zip file. All of the files are just directly in the zip file, with no parent directory whatsoever.
Example:
Changed files from source directory:
SourceDirectory\A\B\file1
SourceDirectory\1\file2
SourceDirectory\x\y\z\file3
Come into the new zipped archive.zip like this:
file1
file2
file3
But I want to keep the parent directories in the archive.zip like this:
A\B\file1
1\file2
x\y\z\file3
Is there any way to do this?
I hope the explanation is not too complex, please feel free to ask for more information if it is not clear. I needed to add that much context because it is a very specific problem.
Thanks for all help and tips.
I want to merge many CSV-files into one (a few hundred files) removing the header row of the added CSVs.
As the files sit in several subfolders I need to start from the root traversing all the subfolders and process all CSVs in there. Before merging I want to archive them with zip deleting old CSVs. The new merged CSV-file and the zip-archive should be named like their parent folder.
In case the Script is started again for the same folder none of already processed files should be damaged or removed accidentally.
I am not a Powershell guy so I have been copying pasting from several resources in the web and came up with the following solution (Sorry don't remember the resources feel free to put references in the comment if you know).
This patch-work code does the job but it doesn't feel very bulletproof. For now it is processing the CSV files in the subfolders only. Processing the files within the given $targDir as well would also be nice.
I am wondering if it could be more compact. Suggestions for improvement are appreciated.
$targDir = "\\Servername\folder\"; #path
Get-ChildItem "$targDir" -Recurse -Directory |
ForEach-Object { #walkinthrough all subfolder-paths
#
Set-Location -Path $_.FullName
#remove existing AllInOne.csv (targed name for a merged file) in case it has been left over from a previous execution.
$FileName = ".\AllInOne.csv"
if (Test-Path $FileName) {
Remove-Item $FileName
}
#remove existing AllInOne.csv (targed name for archived files) in case it has been left over from a previous execution.
$FileName = ".\AllInOne.zip"
if (Test-Path $FileName) {
Remove-Item $FileName
}
#compressing all csv files in the current path, temporarily named AllInOne.zip. Doing that for each file adding it to the archive (with -Update)
# I wonder if there is a more efficient way to do that.
dir $_.FullName | where { $_.Extension -eq ".csv"} | foreach { Compress-Archive $_.FullName -DestinationPath "AllInOne.zip" -Update}
##########################################################
# This code is basically merging all the CSV files
# skipping the header of added files
##########################################################
$getFirstLine = $true
get-childItem ".\*.csv" | foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content ".\AllInOne.csv" $linesToWrite
# Output file is named AllInOne.csv temporarily - this is not a requirement
# It was simply easier for me to come up with this temp file in the first place (symptomatic for copy&paste).
}
#########################################################
#deleting old csv files
dir $_.FullName | where { $_.Extension -eq ".csv" -and $_ -notlike "AllInOne.csv"} | foreach { Remove-Item $_.FullName}
# Temporarily rename AllinOne files with parent folder name
Get-ChildItem -Path $_.FullName -Filter *.csv | Rename-Item -NewName {$_.Basename.Replace("AllInOne",$_.Directory.Name) + $_.extension}
Get-ChildItem -Path $_.FullName -Filter *.zip | Rename-Item -NewName {$_.Basename.Replace("AllInOne",$_.Directory.Name) + $_.extension}
}
I have been executing it in the Powershell ISE. The Script is for a house keeping only, executed casually and not on a regular base - so performance doesn't matter so much.
I prefer to stick with a script that doesn't depend on additional libraries if possible (e.g. for Zip).
It may not be bulletproof, but I have seen worse cobbled together scripts. It'll definitely do the job you want it to, but here are some small changes that will make it a bit shorter and harder to break.
Since all your files are CSVs and all would have the same headers, you can use Import-CSV to compile all of the files into an array. You won't have to worry about stripping the headers or accidentally removing a row.
Get-ChildItem "*.csv" | Foreach-Object {
$csvArray += Import-CSV $_
}
Then you can just use Export-CSV -Path $_.FullName -NoTypeInformation to output it all in to a new CSV file.
To have it check the root folder and all the subfolders, I would throw all of the lines in the main ForEach loop into a function and then call it once for the root folder and keep the existing loop for all the subfolders.
function CompileCompressCSV {
param (
[string] $Path
)
# Code from inside the ForEach Loop
}
# Main Script
CompileCompressCSV -Path $targetDir
Get-ChildItem -Path $targetDir -Recurse -Directory | ForEach-Object {
CompileCompressCSV -Path $_.FullName
}
This is more of a stylistic choice, but I would do the steps of this script in a slightly different order:
Get Parent Folder Name
Remove old compiled CSVs and ZIPs
Compile CSVs into an array and output with Parent Folder Name
ZIP together CSVs into a file with the Parent Folder Name
Remove all CSV files
Personally, I'd rather name the created files properly the first time instead of having to go back and rename them unless there is absolutely no way around it. That doesn't seem the case for your situation so you should be able to create them with the right name on the first go.
I am currently in the process of restructuring a Windows server file directory and need to bulk rename over 10,000 files all in different folders but most have duplicate filenames (e.g 0001.pdf, 0002.pdf)
I have written a Powershell script to help me do this but I am having some trouble with the execution.
The script reads a csv file (export of all the files in the folder structure - export.csv) then creates the variables $OldFilename & $NewFileName. The variables are then used to locate the old file location and copy it to a new destination using the Copy-Item command for each row in the CSV.
Code
$csv = import-csv c:\tmp\test.csv
$csv | foreach-object {
$OldFileName =$_.OldFileName
$NewFileName = $_.NewFileName
}
$csv | ForEach-Object {
$OldFileName =$_.OldFileName
$NewFileName = $_.NewFileName
Copy-Item $OldFileName -Destination $NewFileName
}
CSV is formatted as such
OldFileName,NewFileName
"X:\Folder1\0001.pdf","C:\temp\File0001.pdf"
"X:\Folder2\0001.pdf","C:\temp\File0002.pdf"
"X:\Folder3\0001.pdf","C:\temp\File0003.pdf"
Code does not error when run but files do not appear.
A Write-Output displays the correct file path but a Write-Host((Get-Item $file).length/1MB) does not retrieve the files size which leads me to believe there is an issue with the variable!?
Something I need to add?
Many Thanks
Jonny
I have a text file with a list of user names separated by semi colon, users names in the text file are: user1; user2; user3, etc.. The user names all have a network folder located at \testserver\users\user1, \testserver\users\user2, and so on.
I am trying to have PowerShell script read the text file and copy the folder and all data in each folder for each user from one location to another location called \testserver\newusers\users. However when I launch the script I have written so far, it just creates a folder with a user name from the text file I have. Below is what I have so far:
$File = Get-Content .\MyFile.txt
$File | ForEach-Object {
$_.Split(';') | ForEach-Object {
Copy-Item -Path "$_" -Destination '\\testserver\newusers\users'
}
}
I am launching my PowerShell .ps1 file from a location that has the myfile.txt file in it.
How do I get this to work properly?
Call Copy-Item with the parameter -Recurse if you want to copy the folders' content as well. Otherwise just the folder itself would be copied (without content). You also need to provide the full path to the source folders unless you run the script from \\testserver\users.
Something like this should work:
$server = 'testserver'
$src = "\\$server\users"
$dst = "\\$server\newusers"
(Get-Content .\MyFile.txt) -split ';' | % {
Copy-Item -Path "$src\$_" -Destination "$dst\" -Recurse
}