Powershell with Robocopy and Arguments Passing - powershell

I'm trying to write a script that uses robocopy. If I were just doing this manually, my command would be:
robocopy c:\hold\test1 c:\hold\test2 test.txt /NJH /NJS
BUT, when I do this from powershell, like:
$source = "C:\hold\first test"
$destination = "C:\hold\second test"
$robocopyOptions = " /NJH /NJS "
$fileList = "test.txt"
robocopy $source $destination $fileLIst $robocopyOptions
I get:
-------------------------------------------------------------------------------
ROBOCOPY :: Robust File Copy for Windows
-------------------------------------------------------------------------------
Started : Fri Apr 10 09:20:03 2015
Source - C:\hold\first test\
Dest - C:\hold\second test\
Files : test.txt
Options : /COPY:DAT /R:1000000 /W:30
------------------------------------------------------------------------------
ERROR : Invalid Parameter #4 : " /NJH /NJS "
However, if I change the robocopy command to
robocopy $source $destination $fileLIst /NJH /NJS
everything runs successfully.
So, my question is, how can I pass a string as my robocopy command options (and, in a larger sense, do the same for any given external command)

Start robocopy -args "$source $destination $fileLIst $robocopyOptions"
or
robocopy $source $destination $fileLIst $robocopyOptions.split(' ')

Use the arrays, Luke. If you specify an array of values, PowerShell will automatically expand them into separate parameters. In my experience, this is the most reliable method. And it doesn't require you to mess with the Start-Process cmdlet, which is in my opinion is overkill for such tasks.
This trick is from the best article I've seen on the PowerShell behavior towards external executables: PowerShell and external commands done right.
Example:
$source = 'C:\hold\first test'
$destination = 'C:\hold\second test'
$robocopyOptions = #('/NJH', '/NJS')
$fileList = 'test.txt'
$CmdLine = #($source, $destination, $fileList) + $robocopyOptions
& 'robocopy.exe' $CmdLine

You can't use a string to pass options in that way because when you write
robocopy $source $destination $fileList $robocopyOptions
PowerShell will evaluate the last variable ($robocopyOptions) as a single string and it will quote it. This means robocopy will get "/NJH /NHS" (single string, quoted) on its command line. (Obviously not the intent.)
For details on how to work around these kinds of issues, see here:
http://windowsitpro.com/powershell/running-executables-powershell
The article includes the following function:
function Start-Executable {
param(
[String] $FilePath,
[String[]] $ArgumentList
)
$OFS = " "
$process = New-Object System.Diagnostics.Process
$process.StartInfo.FileName = $FilePath
$process.StartInfo.Arguments = $ArgumentList
$process.StartInfo.UseShellExecute = $false
$process.StartInfo.RedirectStandardOutput = $true
if ( $process.Start() ) {
$output = $process.StandardOutput.ReadToEnd() `
-replace "\r\n$",""
if ( $output ) {
if ( $output.Contains("`r`n") ) {
$output -split "`r`n"
}
elseif ( $output.Contains("`n") ) {
$output -split "`n"
}
else {
$output
}
}
$process.WaitForExit()
& "$Env:SystemRoot\system32\cmd.exe" `
/c exit $process.ExitCode
}
}
This function will let you run an executable in the current console window and also let you build an array of string parameters to pass to it.
So in your case you could use this function something like this:
Start-Executable robocopy.exe $source,$destination,$fileList,$robocopyOptions

Putting the options in separate arguments worked for me. Using Robocopy for copying excluding any CSV files.
$roboCopyPath = $env:ROBOCOPY_PATH
$otherLogsPath = [System.IO.Path]::Combine($basePath, "Logs-Other")
$atrTestResults = [System.IO.Path]::Combine($Release, $BuildNumber)
$ResultsSummary = [System.IO.Path]::Combine($basePath, "Result")
$robocopyOptions = #("/log:$otherLogsPath\robocopy.log", '/xf', '*.csv')
$CmdLine = #($atrTestResults, $ResultsSummary) + $robocopyOptions
&$roboCopyPath $CmdLine

Related

xcopy show only when files have been copied

I have made a script using xcopy that generates a csv log file with the date to check that my copies are done.
I would like to be able to display only when it has copied files, and not display anything when there are 0 files copied.
How can I do this?
if I didn't make myself clear let me know
Thanks :)
Here is my code:
$Logfile = "C:\Users\Name\Documents\Power\"
Function LogWrite
{
Param ([string]$logstring)
Add-content $Logfile -value $logstring and
}
function Get-TimeStamp
{
return "[{0:dd/MM/yy} {0:HH:mm:ss}]" -f (Get-Date)
}
xcopy /s /f /y /b /d C:\Users\Name\Documents\SourceBack C:\Users\Name\Documents\DestBack
>> xcopy.csv
Write-Output "last copied file(s) on $(Get-TimeStamp)" | Out-file
C:\Users\Name\Documents\Power\xcopy.csv -append
I think this is what you're looking for.
The output of xcopy is redirected to a temporary file. If that file contains only a single line (e.g. 0 File(s) copied), no further action is taken.
Otherwise, the output and an additional line with the timestamp is added to your CSV file.
At the end, regardless of the outcome, the temporary file is removed.
$Logfile = "C:\Users\Name\Documents\Power\xcopy.csv"
$TempFile = New-TemporaryFile
try {
Start-Process xcopy `
-ArgumentList '/s','/f','/y','/b','/d','C:\Users\Name\Documents\SourceBack','C:\Users\Name\Documents\DestBack' `
-RedirectStandardOutput $TempFile.FullName `
-Wait
$ProcessOutput = #(Get-Content $TempFile)
if ($ProcessOutput.Length -gt 1) {
($ProcessOutput + "last copied file(s) on $(Get-Date -Format '[dd/MM/yy HH:mm:ss]')") -join "`n" | Add-Content $Logfile
}
} finally {
Remove-Item $TempFile -Force
}
Remarks:
I removed the Get-TimeStamp function, as it was only used once and doesn't add a lot of benefit in that case.
I also removed the LogWrite function as it isn't used in your sample code, and it contains a syntax error (stray and).
You're appending a line to a CSV file (last copied file(s) …), which is bound to cause issues when you try to parse that file later on.
Update
You're never too old to learn, and it seems that Start-Process isn't necessary at all. For more information, see here and here.
$Logfile = "C:\Users\Name\Documents\Power\xcopy.csv"
[string[]] $Output = & xcopy /s /f /y /b /d C:\Users\Name\Documents\SourceBack C:\Users\Name\Documents\DestBack 2>&1
if ($ProcessOutput.Length -gt 1) {
($ProcessOutput + "last copied file(s) on $(Get-Date -Format '[dd/MM/yy HH:mm:ss]')") -join "`n" | Add-Content $Logfile
}

Converting a line of cmd to powershell

EDIT2: Final code below
I need help on converting some codes as I am very new to mkvmerge, powershell and command prompt.
The CMD code is from https://github.com/Serede/mkvtoolnix-batch/blob/master/mkvtoolnix-batch.bat
for %%f in (*.mkv) do %mkvmerge% #options.json -o "mkvmerge_out/%%f" "%%f"
What I've managed so far
$SourceFolder = "C:\tmp" #In my actual code, this is done using folder browser
$SourceFiles = Get-ChildItem -LiteralPath $SourceFolder -File -Include *.mkv
$SourceFiles | foreach
{
start-process "F:\Desktop\#progs\mkvtoolnix\mkvmerge.exe"
}
I'd be grateful for any help as I'm having trouble understanding and converting while learning both sides. Thank you very much.
**EDIT 2:**Here's my final working code.
Function Get-Folder($initialDirectory) {
#Prompt to choose source folder
[void] [System.Reflection.Assembly]::LoadWithPartialName('System.Windows.Forms')
$FolderBrowserDialog = New-Object System.Windows.Forms.FolderBrowserDialog
$FolderBrowserDialog.Description = 'Choose the video folder'
$FolderBrowserDialog.RootFolder = 'MyComputer'
if ($initialDirectory) { $FolderBrowserDialog.SelectedPath = $initialDirectory }
[void] $FolderBrowserDialog.ShowDialog()
return $FolderBrowserDialog.SelectedPath
}
Function ExitMessage
{
#endregion Function output
Write-Host "`nOperation complete";
Write-Host -NoNewLine 'Press any key to continue...';
$null = $Host.UI.RawUI.ReadKey('NoEcho,IncludeKeyDown');
Exit;
}
($SourceFolder = Get-Folder | select )
#Check for output folder and create if unavailable
$TestFile = "$SourceFolder" + "\mkvmerge_out"
if ((Test-Path -LiteralPath $TestFile) -like "False")
{
new-item -Path $SourceFolder -name "mkvmerge_out" -type directory
Write-Host 'Folder created';
}
#Checking for the presence of a Json file
$TestFile = (Get-ChildItem -LiteralPath $SourceFolder -File -Filter *.json)
if ($TestFile.count -eq 0)
{
Write-Host 'json file not found';
ExitMessage;
}
$TestFile = "$SourceFolder" + "\$TestFile"
#Getting the total number of files and start timer.
[Int] $TotalFiles = 0;
[Int] $FilesDone = 0;
$TotalFiles = (Get-ChildItem -LiteralPath $SourceFolder -File -Filter *.mkv).count
$PercentFiles = 0;
$Time = [System.Diagnostics.Stopwatch]::StartNew()
#Start mkvmerge process with progress bar
$mkvmergeExe = 'F:\Desktop\#progs\mkvtoolnix\mkvmerge.exe'
$JsonFile = "$TestFile" # alternatively, use Join-Path
Get-ChildItem -LiteralPath $SourceFolder -File -Filter *.mkv | ForEach-Object {
$PercentFiles = [math]::truncate(($FilesDone/$TotalFiles)*100)
Write-Progress -Activity mkvmerge -Status ("{0}% Completed; {1}/{2} done; Time Elapsed: {3:d2}:{4:d2}:{5:d2}" -f $PercentFiles, $FilesDone, $TotalFiles, $Time.Elapsed.Hours, $Time.Elapsed.minutes, $Time.Elapsed.seconds) -PercentComplete $PercentFiles;
Write-Host "Processing $_"
$f = $_.FullName
$of = "$SourceFolder\mkvmerge_out\$($_.Name)"
& $mkvmergeExe -q `#$JsonFile -o $of $f
$FilesDone++
}
Remove-Item -LiteralPath $JsonFile #Remove this line if you want to keep the Json file
$PercentFiles = [math]::truncate(($FilesDone/$TotalFiles)*100)
Write-Progress -Activity mkvmerge -Status ("{0}% Completed; {1}/{2} done; Time Elapsed: {3:d2}:{4:d2}:{5:d2}" -f $PercentFiles, $FilesDone, $TotalFiles, $Time.Elapsed.Hours, $Time.Elapsed.minutes, $Time.Elapsed.seconds) -PercentComplete $PercentFiles;
ExitMessage;
$mkvmergeExe = 'F:\Desktop\#progs\mkvtoolnix\mkvmerge.exe'
$optionsFile = "$SourceFolder\options.json" # alternatively, use Join-Path
Get-ChildItem -LiteralPath $SourceFolder -File -Filter *.mkv | ForEach-Object {
$f = $_.FullName
$of = "$SourceFolder\mkvmerge_out\$($_.Name)"
& $mkvmergeExe `#$optionsFile -o $of $f
}
Note that your cmd code assumes that it's operating in the current directory, while your PowerShell code passes a directory explicitly via $SourceFolder; therefore, the options.json file must be looked for in $SourceFolder and too, and the output file path passed to -o must be prefixed with $SourceFolder too which is achieved via expandable strings ("...") .
The main points to consider:
for %%f in (*.mkv) has no direct counterpart in PowerShell; you correctly used Get-ChildItem instead, to get a list of matching files, which are returned as System.IO.FileInfo instances.
However, -Include won't work as intended in the absence of -Recurse (unless you append \* - see this GitHub issue; -Filter does, and is also the faster method, but it has its limitations and legacy quirks (see this answer).
While PowerShell too allows you to execute commands whose names or paths are stored in a variable (or specified as a quoted string literal), you then need &, the call operator, to invoke it, for syntactic reasons.
Inside a script block ({ ... }) passed to the ForEach-Object cmdlet, automatic variable $_ represents the pipeline input object at hand.
$_.FullName ensures that the System.IO.FileInfo input instances are represented by their full path when used in a string context.
This extra step is no longer necessary in PowerShell [Core] 6+, where System.IO.FileInfo instances thankfully always stringify as their full paths.
The # character is preceded by ` (backtick), PowerShell's escape character, because # - unlike in cmd - is a metacharacter, i.e. a character with special syntactic meaning. `# ensures that the # is treated verbatim, and therefore passed through to mkvmerge.
Alternatively, you could have quoted the argument instead of escaping just the #: "#$optionsFile"
See this answer for background information.
You generally do not need to enclose arguments in "..." in PowerShell, even if they contain spaces or other metacharacters.

PowerShell and Robocopy - filename incorrect when trying to pass variable folder as destination

I'm trying to use a PowerShell script running Robocopy* to back some files up to a newly-made directory:
$Timestamp = Get-Date -format ddMMyyyy
$DestFolder = "`"\\NASBOX\Archives\$Timestamp\`""
$SourceFolder = "`"\\DESKTOP\d$`""
ROBOCOPY $SourceFolder $DestFolder /COPYALL /B /R:10 /W:90 /LOG:$Timestamp.txt /FP /TEE
This gives me the following error:
2018/01/23 16:26:20 ERROR 123 (0x0000007B) Accessing Destination Directory \\NASBOX\Archives\23012018" \COPYALL \B \R:10 \W:90 \LOG:23012018.txt \FP \TEE\
The filename, directory name, or volume label syntax is incorrect.
I've tried a few different methods, including passing the arguments as an array. Every single thing I've tried results in the exact same error.
I roughly understand why this is happening, but despite ~two hours spent online I can't find a solution that works in my specific context.
Where am I going wrong?
* I tried using Copy-Item but there are some super long directory paths on this desktop's "D" drive.
The issue is the trailing slash in the path you are building:
"\\NASBOX\Archives\23012018\"
This slash is escaping the double quote for robocopy, it is seeing this path as including a quote symbol at the end :
\\NASBOX\Archives\23012018"
The error message shows this, but isn't very helpful! To fix the issue, simply remove the trailing slash from your path:
$DestFolder = "`"\\NASBOX\Archives\$Timestamp`""
You don't need to try so hard with escaping of quotes in your variables. PowerShell handles most of this for you. This should be all you need to do:
$Timestamp = Get-Date -Format ddMMyyyy
$SourceFolder = "\\DESKTOP\d$"
$DestFolder = "\\NASBOX\Archives\$Timestamp"
ROBOCOPY $SourceFolder $DestFolder /COPYALL /B /R:10 /W:90 /LOG:$Timestamp.txt /FP /TEE
Note that the destination folder shouldn't include a trailing \.
TL;DR - It is not necessary to create strings with embedded " characters to pass to robocopy. Just put the variables on the robocopy command line and PowerShell will quote automatically when necessary.
Function Copy-File {
[CmdletBinding()]
Param(
[Parameter(Position=0)]
[string]$source,
[Parameter(Position=1)]
[string]$dest,
[Parameter(Position=2)]
[string]$sourcefile,
[Parameter(Position=3)]
[ref]$RoboError
)
Write-Log -message "Copying $sourcefile from $source to $dest"
$robotoday=(Get-Date).ToString('yyyyMMdd')
$logfile = -join($env:systemdrive, '\logs\', $robotoday, '_robocopy.log')
$what = #("$sourcefile",'/COPY:DAT', '/Z', '/E')
$options = #("/R:1","/W:1","/TEE","/ETA","/LOG+:$logfile")
$cmdArgs = #($source,$dest,$what,$options)
robocopy #cmdArgs
if ($lastexitcode -gt 7) {
$RoboError.value=$TRUE
Write-Log -level 'warn' -message "Robocopy function failed with error: $lastexitcode"
}
} # End Copy-File
[bool]$RoboError=$FALSE
Copy-File -source $copysource -dest $copydestination -sourcefile '*' -RoboError([ref]$RoboError)

Re-acl from text file

This one is stumping me. I am trying to do this with PowerShell, but it does not have to be...
Basically, I have a text file containing ACLs information for a large directory structure that has been migrated (I realize there is much to be desired from the contents, but we no longer can access the original filesystem, so this is what we have to work with).
I need to generate a batch file to re-acl the new (copied) file system.
So in a nutshell, I need to convert text like this (sorry for the scroll, but I am trying to preserve the line spacing):
Path: \\Share.Domain.com\Directory01$\Subdirectory01
AccessToString : DOMAIN\Group01-Knuckleheads Allow ReadAndExecute, Synchronize
BUILTIN\Administrators
Path: \\Share.Domain.com\Directory02$\Subdirectory01
AccessToString : DOMAIN\Different-Group02 Allow FullControl
BUILTIN\Administrators Allow FullControl
Into new files (or the same files, doesn't matter really) with the content like this:
ICacls "\\Share.Domain.com\Directory01$\Subdirectory01" /Grant "DOMAIN\Group01-Knuckleheads":(OI)(CI)R,X,S /t /c /l /q /inheritance:r
ICacls "\\Share.Domain.com\Directory01$\Subdirectory01" /Grant "BUILTIN\Administrators":(OI)(CI)F /t /c /l /q /inheritance:r
ICacls "\\Share.Domain.com\Directory02$\Subdirectory01" /Grant "DOMAIN\Different-Group02":(OI)(CI)F /c /l /q /inheritance:r
ICacls "\\Share.Domain.com\Directory02$\Subdirectory01" /Grant "BUILTIN\Administrators":(OI)(CI)F /c /l /q /inheritance:r
My semi-pathetic attempt to work this out are still progressing, but I realize that this foe is greater than my mojo:
#$ACLs = Get-Content C:\Scripts\Test\AndTest.txt
#ForEach-Object ($ACL in $ACLs)
#{
#Figure out how to break strings into variables....
#}
#Declare Var - Need to populate from the imported text file
$FilePath = "\\Share.Domain.com\Directory01$\Subdirectory01"
$GroupName = "DOMAIN\Domain Admins"
$TestPerm = "ReadAndExecute"
If ($TestPerm = "FullControl"){$Perms = "F"}
Elseif ($TestPerm = "ReadAndExecute"){$Perms = "RX"}
Elseif ($TestPerm = "Modify"){$Perms = "M"}
Elseif ($TestPerm = "Deny"){$Perms = "D"}
Elseif ($TestPerm = "Read"){$Perms = "R"}
Elseif ($TestPerm = "Write"){$Perms = "W"}
cls
#Build icacls string -Test Output
Write-Host "ICacls ""$FilePath"" /Grant ""$GroupName"":(OI)(CI)$Perms /t /c /l /q /inheritance:r"
#Write icacls batch file
#Out-File "C:\Scripts\Test\re-acl.cmd" "ICacls ""$FilePath"" /Grant ""$GroupName"":(OI)(CI)$Perms /t /c /l /q /inheritance:r"
I realize that there is a bunch of work to do here & I am just starting to figure this out. For example, I am thinking I need to list the permissions into an array and build the string as such. There are also bound to be special permissions etc...
But for now, I am trying to figure out how to import the text file and then break it down into variables... Like I said, I am working in Powershell, but it really can be anything... VB or Python perhaps?
Big thanks in advance!
Ok, I think most of your issue is parsing the text, so that's mainly what I helped with here. I did build the strings, and if you have more permissions that you want to include you should be able to figure out how to add them to the switch. For this we will want to read the entire text file in as a large multi-line string. Then we split it up into chunks based on the keyword "Path:". Then for each record I get the path out as a string, grab the permissions, and parse out accounts, allow/deny, and individual accesses. Then I convert the accesses to short versions accepted by icacls, and build the arguments out as a formatted string.
I output to the host, but once you are satisfied that it looks right to you, you can remove the Write-Host " and the trailing " and it will just execute it (assuming icacls is in the same folder that you're in, or in the PATH environment variable).
$Text = Get-Content C:\Scripts\Test\AndTest.txt -Raw
$Records = $Text -split "(?s)(Path:.*?)(?=Path:|$)"|?{$_}
#Loop through each file/ACL
ForEach($Record in $Records){
$FilePath = ($Record -split "[\r\n]+")[0].Substring(6)
$PermRecords = ($Record -split "(?s)AccessToString : "|Select -skip 1) -split "[\r\n]+"|?{$_}
$Perms = $PermRecords|?{$_ -match "(.*\\.*?)\s+(Allow|Deny)(.*)$"}|%{[pscustomobject]#{'Account'=$Matches[1].Trim();'Type'=$Matches[2];'Perms'=$Matches[3].Trim()}}
#Loop through each perm for the current file
$Perms | %{
#Convert friendly names to abbreviations
$ShortPerms = ''
Switch -regex ($_.Perms){
"FullControl" {$ShortPerms = "F";Continue}
"ReadAndExecute" {$ShortPerms += "RX,"}
"Synchronize" {$ShortPerms += "S,"}
"Modify" {$ShortPerms += "M,"}
"Read(?=,|$)" {$ShortPerms += "R,"}
"Write" {$ShortPerms += "W,"}
}
$ShortPerms = $ShortPerms.TrimEnd(',')
$Arguments = '"{0}" /{4} "{1}":(OI)(CI)({2}) /t /c /l /q /inheritance:r' -f $FilePath, $_.Account, $ShortPerms,$(If($_.Type -eq 'Allow'){'Grant'}else{'Deny'})
write-host "& ICAcls $Arguments"
}
}
If the -Raw argument doesn't work for you, you can work around that with
(Get-Content C:\Scripts\Test\AndTest.txt) -join "`r`n"
Let me know if you have questions or issues.

How to speed up Powershell Get-Childitem over UNC

DIR or GCI is slow in Powershell, but fast in CMD. Is there any way to speed this up?
In CMD.exe, after a sub-second delay, this responds as fast as the CMD window can keep up
dir \\remote-server.domain.com\share\folder\file*.*
In Powershell (v2), after a 40+ second delay, this responds with a noticable slowness (maybe 3-4 lines per second)
gci \\remote-server.domain.com\share\folder\file*.*
I'm trying to scan logs on a remote server, so maybe there's a faster approach.
get-childitem \\$s\logs -include $filemask -recurse | select-string -pattern $regex
Okay, this is how I'm doing it, and it seems to work.
$files = cmd /c "$GETFILESBAT \\$server\logs\$filemask"
foreach( $f in $files ) {
if( $f.length -gt 0 ) {
select-string -Path $f -pattern $regex | foreach-object { $_ }
}
}
Then $GETFILESBAT points to this:
#dir /a-d /b /s %1
#exit
I'm writing and deleting this BAT file from the PowerShell script, so I guess it's a PowerShell-only solution, but it doesn't use only PowerShell.
My preliminary performance metrics show this to be eleventy-thousand times faster.
I tested gci vs. cmd dir vs. FileIO.FileSystem.GetFiles from #Shawn Melton's referenced link.
The bottom line is that, for daily use on local drives, GetFiles is the fastest. By far. CMD DIR is respectable. Once you introduce a slower network connection with many files, CMD DIR is slightly faster than GetFiles. Then Get-ChildItem... wow, this ranges from not too bad to horrible, depending on the number of files involved and the speed of the connection.
Some test runs. I've moved GCI around in the tests to make sure the results were consistent.
10 iterations of scanning c:\windows\temp for *.tmp files
.\test.ps1 "c:\windows\temp" "*.tmp" 10
GetFiles ... 00:00:00.0570057
CMD dir ... 00:00:00.5360536
GCI ... 00:00:01.1391139
GetFiles is 10x faster than CMD dir, which itself is more than 2x faster than GCI.
10 iterations of scanning c:\windows\temp for *.tmp files with recursion
.\test.ps1 "c:\windows\temp" "*.tmp" 10 -recurse
GetFiles ... 00:00:00.7020180
CMD dir ... 00:00:00.7644196
GCI ... 00:00:04.7737224
GetFiles is a little faster than CMD dir, and both are almost 7x faster than GCI.
10 iterations of scanning an on-site server on another domain for application log files
.\test.ps1 "\\closeserver\logs\subdir" "appname*.*" 10
GetFiles ... 00:00:00.3590359
CMD dir ... 00:00:00.6270627
GCI ... 00:00:06.0796079
GetFiles is about 2x faster than CMD dir, itself 10x faster than GCI.
One iteration of scanning a distant server on another domain for application log files, with many files involved
.\test.ps1 "\\distantserver.company.com\logs\subdir" "appname.2011082*.*"
CMD dir ... 00:00:00.3340334
GetFiles ... 00:00:00.4360436
GCI ... 00:11:09.5525579
CMD dir is fastest going to the distant server with many files, but GetFiles is respectably close. GCI on the other hand is a couple of thousand times slower.
Two iterations of scanning a distant server on another domain for application log files, with many files
.\test.ps1 "\\distantserver.company.com\logs\subdir" "appname.20110822*.*" 2
CMD dir ... 00:00:00.9360240
GetFiles ... 00:00:01.4976384
GCI ... 00:22:17.3068616
More or less linear increase as test iterations increase.
One iteration of scanning a distant server on another domain for application log files, with fewer files
.\test.ps1 "\\distantserver.company.com\logs\othersubdir" "appname.2011082*.*" 10
GetFiles ... 00:00:00.5304170
CMD dir ... 00:00:00.6240200
GCI ... 00:00:01.9656630
Here GCI is not too bad, GetFiles is 3x faster, and CMD dir is close behind.
Conclusion
GCI needs a -raw or -fast option that does not try to do so much. In the meantime, GetFiles is a healthy alternative that is only occasionally a little slower than CMD dir, and usually faster (due to spawning CMD.exe?).
For reference, here's the test.ps1 code.
param ( [string]$path, [string]$filemask, [switch]$recurse=$false, [int]$n=1 )
[reflection.assembly]::loadwithpartialname("Microsoft.VisualBasic") | Out-Null
write-host "GetFiles... " -nonewline
$dt = get-date;
for($i=0;$i -lt $n;$i++){
if( $recurse ){ [Microsoft.VisualBasic.FileIO.FileSystem]::GetFiles( $path,
[Microsoft.VisualBasic.FileIO.SearchOption]::SearchAllSubDirectories,$filemask
) | out-file ".\testfiles1.txt"}
else{ [Microsoft.VisualBasic.FileIO.FileSystem]::GetFiles( $path,
[Microsoft.VisualBasic.FileIO.SearchOption]::SearchTopLevelOnly,$filemask
) | out-file ".\testfiles1.txt" }}
$dt2=get-date;
write-host $dt2.subtract($dt)
write-host "CMD dir... " -nonewline
$dt = get-date;
for($i=0;$i -lt $n;$i++){
if($recurse){
cmd /c "dir /a-d /b /s $path\$filemask" | out-file ".\testfiles2.txt"}
else{ cmd /c "dir /a-d /b $path\$filemask" | out-file ".\testfiles2.txt"}}
$dt2=get-date;
write-host $dt2.subtract($dt)
write-host "GCI... " -nonewline
$dt = get-date;
for($i=0;$i -lt $n;$i++){
if( $recurse ) {
get-childitem "$path\*" -include $filemask -recurse | out-file ".\testfiles0.txt"}
else {get-childitem "$path\*" -include $filemask | out-file ".\testfiles0.txt"}}
$dt2=get-date;
write-host $dt2.subtract($dt)
Here is a good explanation on why Get-ChildItem is slow by Lee Holmes. If you take note of the comment from "Anon 11 Mar 2010 11:11 AM" at the bottom of the page his solution might work for you.
Anon's Code:
# SCOPE: SEARCH A DIRECTORY FOR FILES (W/WILDCARDS IF NECESSARY)
# Usage:
# $directory = "\\SERVER\SHARE"
# $searchterms = "filname[*].ext"
# PS> $Results = Search $directory $searchterms
[reflection.assembly]::loadwithpartialname("Microsoft.VisualBasic") | Out-Null
Function Search {
# Parameters $Path and $SearchString
param ([Parameter(Mandatory=$true, ValueFromPipeline = $true)][string]$Path,
[Parameter(Mandatory=$true)][string]$SearchString
)
try {
#.NET FindInFiles Method to Look for file
# BENEFITS : Possibly running as background job (haven't looked into it yet)
[Microsoft.VisualBasic.FileIO.FileSystem]::GetFiles(
$Path,
[Microsoft.VisualBasic.FileIO.SearchOption]::SearchAllSubDirectories,
$SearchString
)
} catch { $_ }
}
I tried some of the suggested methods with a large amount of files (~190.000). As mentioned in Kyle's comment, GetFiles isn't very useful here, because it needs nearly forever.
cmd dir was better than Get-ChildItems at my first tests, but it seems, GCI speeds up a lot if you use the -Force parameter. With this the needed time was about the same as for cmd dir.
P.S.: In my case I had to exclude most of the files because of their extension. This was made with -Exclude in gci and with a | where in the other commands. So the results for just searching files might slightly differ.
Here's an interactive reader that parses cmd /c dir (which can handle unc paths), and will collect the 3 most important properties for most people: full path, size, timestamp
usage would be something like $files_with_details = $faster_get_files.GetFileList($unc_compatible_folder)
and there's a helper function to check combined size $faster_get_files.GetSize($files_with_details)
$faster_get_files = New-Module -AsCustomObject -ScriptBlock {
#$DebugPreference = 'Continue' #verbose, this will take figuratively forever
#$DebugPreference = 'SilentlyContinue'
$directory_filter = "Directory of (.+)"
$file_filter = "(\d+/\d+/\d+)\s+(\d+:\d+ \w{2})\s+([\d,]+)\s+(.+)" # [1] is day, [2] is time (AM/PM), [3] is size, [4] is filename
$extension_filter = "(.+)[\.](\w{3,4})" # [1] is leaf, [2] is extension
$directory = ""
function GetFileList ($directory = $this.directory) {
if ([System.IO.Directory]::Exists($directory)) {
# Gather raw file list
write-Information "Gathering files..."
$files_raw = cmd /c dir $directory \*.* /s/a-d
# Parse file list
Write-Information "Parsing file list..."
$files_with_details = foreach ($line in $files_raw) {
Write-Debug "starting line {$($line)}"
Switch -regex ($line) {
$this.directory_filter{
$directory = $matches[1]
break
}
$this.file_filter {
Write-Debug "parsing matches {$($matches.value -join ";")}"
$date = $matches[1]
$time = $matches[2] # am/pm style
$size = $matches[3]
$filename = $matches[4]
# we do a second match here so as to not append a fake period to files without an extension, otherwise we could do a single match up above
Write-Debug "parsing extension from {$($filename)}"
if ($filename -match $this.extension_filter) {
$file_leaf = $matches[1]
$file_extension = $matches[2]
} else {
$file_leaf = $filename
$file_extension = ""
}
[pscustomobject][ordered]#{
"fullname" = [string]"$($directory)\$($filename)"
"filename" = [string]$filename
"folder" = [string]$directory
"file_leaf" = [string]$file_leaf
"extension" = [string]$file_extension
"date" = get-date "$($date) $($time)"
"size" = [int]$size
}
break
}
} # finish directory/file test
} # finish all files
return $files_with_details
} #finish directory exists test
else #directory doesn't exist {throw("Directory not found")}
}
function GetSize($files_with_details) {
$combined_size = ($files_with_details|measure -Property size -sum).sum
$pretty_size_gb = "$([math]::Round($combined_size / 1GB, 4)) GB"
return $pretty_size_gb
}
Export-ModuleMember -Function * -Variable *
}