Copy files in PowerShell too slow - powershell

Good day, all. New member here and relatively new to PowerShell so I'm having trouble figuring this one out. I have searched for 2 days now but haven't found anything that quite suits my needs.
I need to copy folders created on the current date to another location using mapped drives. These folders live under 5 other folders, based on language.
Folder1\Folder2\Folder3\Folder4\chs, enu, jpn, kor, tha
The folders to be copied all start with the same letters followed by numbers - abc123456789_111. With the following script, I don't need to worry about folder names because only the folder I need will have the current date.
The folders that the abc* folders live in have about 35k files and over 1500 folders each.
I have gotten all of this to work using Get-ChildItem but it is so slow that the developer could manually copy the files by the time the script completes. Here is my script:
GCI -Path $SrcPath -Recurse |
Where {$_.LastWriteTime -ge (Get-Date).Date} |
Copy -Destination {
if ($_.PSIsContainer) {
Join-Path $DestPath $_.Parent.FullName.Substring($SrcPath.length)
} else {
Join-Path $DestPath $_.FullName.Substring($SrcPath.length)
}
} -Force -Recurse
(This only copies to one destination folder at the moment.)
I have also been looking into using cmd /c dir and cmd /c forfiles but haven't been able to work it out. Dir will list the folders but not by date. Forfiles has turned out to be pretty slow, too.
I'm not a developer but I'm trying to learn as much as possible. Any help/suggestions are greatly appreciated.

#BaconBits is right, you have a recurse on your copy-item as well as your getchild-item. This will cause a lot of extra pointless copies which are just overwrites due to your force parameter. Change your script to do a foreach loop and drop the recurse parameter from copy-item
GCI -Path $SrcPath -Recurse |
Where {$_.LastWriteTime -ge (Get-Date).Date} | % {
Copy -Destination {
if ($_.PSIsContainer) {
Join-Path $DestPath $_.Parent.FullName.Substring($SrcPath.length)
} else {
Join-Path $DestPath $_.FullName.Substring($SrcPath.length)
}
} -Force
}

Related

How to speed up Get-ChildItem in Powershell

Just wandering how could I speed up the Get-ChildItem in Powershell?
I have the following script to search for a file that created by today and copy it over to another folder.
$fromDirectory = "test_file_*.txt"
$fromDirectory = "c:\sour\"
$toDirectory = "c:\test\"
Get-ChildItem $fromDirectory -Include $fileName -Recurse | Where {$_.LastWriteTime -gt (Get-Date).Date} | Copy-Item -Destination $toDirectory
Due to the folder that I search have 124,553 history files, it's take me age for the search. Does any know how could I improve my script to speed up my search and copy?
Here are some things to try:
First, use Measure-Command {} to get the actual performance:
Measure-Command { Get-ChildItem $fromDirectory -Include $fileName -Recurse | Where {$_.LastWriteTime -gt (Get-Date).Date} | Copy-Item -Destination $toDirectory }
Then, consider removing the '-Recurse' flag, because this is actually going inside every directory and child and child of child. If your target log files are really that scattered, then...
Try using robocopy to match a pattern in the filename and lastwritetime, then use powershell to copy over. You could even use robocopy to do the copying.
It's possible that you just have a huge, slow problem to solve, but try these to see if you can break it down.
This is a well-known feature of NTFS. Microsoft's docs say that the limit for decreasing performance is about 50 000 files in a directory.
If the file names are of very similar, creation of 8dot3 legacy names will start to slow down when there are about 300 000 files. Though you have "only" 120 k files, it's the same order of magnitude.
Some previous questions discuss this issue. Sadly, there is no a single good solution but better hierarchy in directories. The usual tricks are to disable 8dot3 with fsutil and last access date via registry, but those will help only so much.
Can you redesign the directory structure? Moving old files into, say, year-quarter subdirs might keep the main directory clean enough. To find out file's year-quarter, a quick way is like so,
gci | % {
$("{2} => {1}\Q{0:00}" -f [Math]::ceiling( ($_.LastAccessTime.toString('MM'))/3),
$_.LastAccessTime.ToString('yyyy'),
$_.Name
)
}
I would try putting the Get-ChildItem $fromDirectory -Include $fileName -Recurse | Where {$_.LastWriteTime -gt (Get-Date).Date} in an Array and then copying results from the Array.
$GrabFiles =#()
$GrabFiles =#( Get-ChildItem $fromDirectory -Include $fileName -Recurse | Where {$_.LastWriteTime -gt (Get-Date).Date} )
Copy-Item -Path $GrabFiles -Destination $toDirectory }

PowerShell script isn't copying like I want

Right in the beginning I should note that I am a bloody beginner because I can't attend it classes in my grade.
I want to create a PowerShell script which will copy everything from
C:\Users\Robert\Desktop\test(lots of folders)
to
C:\Users\Robert\Desktop\neu(lots of folders with the exact same names as above)\price
As an absolute beginner I thought that it will be ok to replace the variable folder name with $_Name because it is the same name in both but I am obviously wrong and don't know why.
Here is my attempt
Copy-Item "C:\Users\Robert\Desktop\test\$_name\*" -Destination "C:\Users\Robert\Desktop\neu\$_Name\price" -Recurse
It is copying something but in one package in a new folder in "neu".
I can't avoid creating this script because it would take me at least two or three days to do it by hand.
I am also sorry for my poor English skills
Thank you
the $_ represents the current pipeline item. i don't see a pipeline in there ... [grin]
the following works by grabbing every file in the source dir & its subdirs, and copying that structure to the destination dir. it uses Splatting to structure the parameters neatly.
$SourceDir = "$env:TEMP\Apps - Copy"
$DestDir = "$env:TEMP\Apps - Copy - Two"
$CI_Params = #{
LiteralPath = $SourceDir
Destination = $DestDir
Force = $True
Recurse = $True
}
Copy-Item #CI_Params
If my understanding is correct:
$src = 'C:\Users\Robert\Desktop\test'
$dst = 'C:\Users\Robert\Desktop\neu\{0}\price'
Get-ChildItem $src -Directory | ForEach-Object {
Copy-Item -Path "$($_.FullName)\*" -Destination ($dst -f $_.BaseName) -Recurse -Force -WhatIf
}
Remove -WhatIf to actually do it.

PowerShell to copy files to destination's subfolders while excluding certain folders in the destination

So I have danced with this off and on throughout the day and the timeless phrase "There's more than one way to skin a cat" keeps coming to mind so I decided to take to the community.
Scenario:
Source folder "C:\Updates" has 100 files of various extensions. All need to be copied to the sub-folders only of "C:\Prod\" overwriting any duplicates that it may find.
The Caveats:
The sub-folder names (destinations) in "C:\Prod" are quite dynamic and change frequently.
A naming convention is used to determine which sub-folders in the destination need to be excluded when the source files are being copied (to retain the original versions). For ease of explanation lets say any folder names starting with "!stop" should be excluded from the copy process. (!stop* if wildcards considered)
So, here I am wanting the input of those greater than I to tackle this in PS if I'm lucky. I've tinkered with Copy-Item and xcopy today so I'm excited to hear other's input.
Thanks!
-Chris
Give this a shot:
Get-ChildItem -Path C:\Prod -Exclude !stop* -Directory `
| ForEach-Object { Copy-Item -Path C:\Updates\* -Destination $_ -Force }
This grabs each folder (the -Directory switch ensures we only grab folders) in C:\Prod that does not match the filter and pipes it to the ForEach-Object command where we are running the Copy-Item command to copy the files to the directory.
The -Directory switch is not available in every version of PowerShell; I do not know which version it was introduced in off the top of my head. If you have an older version of PowerShell that does not support -Directory then you can use this script:
Get-ChildItem -Path C:\Prod -Exclude !stop* `
| Where-Object { $_.PSIsContainer } `
| ForEach-Object { Copy-Item -Path C:\Updates\* -Destination $_ -Force }
To select only sub folders which do not begin with "!stop" do this
$Source = "C:\Updates\*"
$Dest = "C:\Prod"
$Stop = "^!stop"
$Destinations = GCI -Path $Dest |?{$_.PSIsContainer -and $_.Name -notmatch $Stop }
ForEach ($Destination in $Destinations) {
Copy-Item -Path $Source -Destination $Destination.FullName -Force
}
Edited Now copies all files from Update to subs of Source not beginning with "!stop" The -whatif switch shows what would happen, to arm the script remove the -whatif.
Edit2 Streamlined the script. If also Sub/sub-folders of C:\Prod shall receive copies include a -rec option to the gci just in front of he pipe.

Copy files with certain extension created within last half an hour with powershell

I'm trying to copy files (.zip) that has been created in the last half an hour and using the below script for that but for some reason all the zip files in the source directory get copied over. Can you please help me in correcting it?
copy-item C:\ABC\\*.zip -filter (Get-ChildItem | Where{$_.CreationTime -ge (Get-Date).AddMinutes(-30)}) -destination C:\
Try the following:
Get-ChildItem C:\ABC\*.zip | Where { $_.CreationTime -ge (Get-Date).AddMinutes(-30) } | % { Copy-Item $_.FullName -destination C:\ }
I am not sure why filter is not working, but this gets a list of all the zips, filters that down to the ones with the correct times, and then issues the copy statement.

Need a script to publish build output to a staging server

I am trying to write a PowerShell script that will copy a subset of files from a source folder and place them into a target folder. I've been playing with "copy-item" and "remove-item" for half a day and cannot get the desired or consistent results.
For example, when I run the following cmdlet multiple times, the files end up in different locations?!?!:
copy-item -Path $sourcePath -Destination $destinationPath -Include *.dll -Container -Force -Recurse
I've been trying every combination of options and commands I can think of but can't find the right solution. Since I'm sure that I'm not doing anything atypical, I'm hoping someone can ease my pain and provide me with the proper syntax to use.
The source folder will contain a large number of files with various extensions. For example, all of the following are possible:
.dll
.dll.config
.exe
.exe.config
.lastcodeanalysisissucceeded
.pdb
.Test.dll
.vshost.exe
.xml
and so on
The script needs to only copy .exe, .dll and .exe.config files excluding any .test.dll and .vshost.exe files. I also need the script to create the target folders if they don't already exist.
Any help getting me going is appreciated.
try:
$source = "C:\a\*"
$dest = "C:\b"
dir $source -include *.exe,*.dll,*.exe.config -exclude *.test.dll,*.vshost.exe -Recurse |
% {
$sp = $_.fullName.replace($sourcePath.replace('\*',''), $destPath)
if (!(Test-Path -path (split-path $sp)))
{
New-Item (split-path $sp) -Type Directory
}
copy-item $_.fullname $sp -force
}
As long as the files are in one directory, the following should work fine. It might be a bit more verbose than needed, but it should be a good starting point.
$sourcePath = "c:\sourcePath"
$destPath = "c:\destPath"
$items = Get-ChildItem $sourcePath | Where-Object {($_.FullName -like "*.exe") -or ($_.FullName -like "*.exe.config") -or ($_.FullName -like "*.dll")}
$items | % {
Copy-Item $_.Fullname ($_.FullName.Replace($sourcePath,$destPath))
}