I had an excel script to search for files in a command.
I found this example on the forum, the statement says that to search for a file by name, you need to write down the name and send (*) but when requested, it does not find anything
Get-ChildItem -Path "C:\\Folder\\test\*"
What can I do to simplify the code and make it much faster. Wait 10 minutes to find a file out of 10000. this is very long
I have a folder with 10,000 files, and excel searches through VBA for a script in almost 2-3 seconds.
When to script in PowerShell via
$find = Get-ChildItem -Path "C:\\Folder"
for ($f=0; $f -lt $find.Count; $f++){
$path_name = $find\[$f\].Name
if($path_name-eq 'test'){
Write Host 'success'
}
}
ut it turns out sooooooo long, the script hangs for 10 minutes and does not respond, and maybe he will be lucky to answer.
How can I find a file by filter using
Get-ChildItem
To make your search faster you can use Get-ChildItem filter.
$fileName = "test.txt"
$filter = "*.txt"
$status = Get-ChildItem -Path "C:\PS\" -Recurse -Filter $filter | Where-Object {$_.Name -match $fileName}
if ($status) {
Write-Host "$($status.Name) is found"
} else {
Write-Host "No such file is available"
}
You could also compare the speed of searching by using Measure-Command
If the disk the data is on is slow then it'll be slow no matter what you do.
If the folder is full of files then it'll also be slow depending on the amount of RAM in the system.
Less files per folder equals more performance so try to split them up into several folders if possible.
Doing that may also mean you can run several Get-ChildItems at once (disk permitting) using PSJobs.
Using several loops to take take care of a related problem usually makes the whole thing run "number of loops" times as long. That's what Where-Object is for (in addition to the -Filter, -Include and -Exclude flags to Get-ChildItem`).
Console I/O takes A LOT of time. Do NOT output ANYTHING unless you have to, especially not inside loops (or cmdlets that act like loops).
For example, including basic statistics:
$startTime = Get-Date
$FileList = Get-ChildItem -Path "C:\Folder" -File -Filter 'test'
$EndTime = Get-Date
$FileList
$AfterOutputTime = Get-Date
'Seconds taken for listing:'
(EndTime - $startTime).TotalSeconds
'Seconds taken including output:'
($AfterOutputTime - $StartTime).TotalSeconds
Related
This is currently what I am trying to execute.
$folderPath = 'M:\abc\WORKFORCE\Media\Attachments'
Write-Host "Executing Script..."
foreach ($file in Get-ChildItem $folderPath -file)
{
# execute code
}
However when I execute the powershell script it freezes on me. It's been this way for an hour now. I'm assuming it might be because the directory has over 8 million items in it. Is there a more efficient way to move these items? Is waiting my only option? Or is it not possible to do this at all with powershell because of how large the directory is?
When you do not need any information except file name, you should use [System.IO.Directory]::EnumerateFiles($folderPath, '*')
EnumerateFiles returns IEnumerable[String].
IEnumerable is a special type that can be used in foreach statements. It does not loads information into memory, but instead it gets next item only when requested. It works almost immediately.
So, your code will be
$filesIEnumerable = [System.IO.Directory]::EnumerateFiles($folderPath,'*')
foreach ($fullName in $filesIEnumerable) {
# code here
$fileName = [System.IO.Path]::GetFileName($fullName)
# more code here
}
In case you want to keep in-memory all list of files instead of iterating once ( for example you need to iterate several times ), EnumerateFiles is still a faster and requires less memory than Get-ChildItem because it does not get any extended file attributes:
$files = #([System.IO.Directory]::EnumerateFiles($folderPath,'*'))
Look further about EnumerateFiles at learn.microsoft.com
Without further explanation of what the end-goal of the script is; there can not really be a solution to this question.
However, a tip on performance, can be given.
Original script:
$folderPath = 'M:\abc\WORKFORCE\Media\Attachments'
Write-Host "Executing Script..."
foreach ($file in Get-ChildItem $folderPath -file)
{
# execute code
}
Suggested approach:
$files = Get-ChildItem 'M:\abc\WORKFORCE\Media\Attachments' -file
$DestinationPath = 'F:\DestinationFolder'
Write-Host "Executing Script..."
$Files | ForEach-Object {
# execute code
# Write-Verbose "Moving $_.Name"
# Move-Item -Destination $DestinationPath
}
That being said, it looks like filimonic's take on an answer has a superior speed to its execution, than my suggestion.
( To expand on that, check this thread)
I'm finishing a script in PowerShell and this is what I must do:
Find and retrieve all .txt files inside a folder
Read their contents (there is a number inside that must be less than 50)
If any of these files has a number greater than 50, change a flag which will allow me to send a crit message to a monitoring server.
The piece of code below is what I already have, but it's probably wrong because I haven't given any argument to Get-Content, it's probably something very simple, but I'm still getting used to PowerShell. Any suggestions? Thanks a lot.
Get-ChildItem -Path C:\temp_erase\PID -Directory -Filter *.txt |
ForEach-Object{
$warning_counter = Get-Content
if ($warning_counter -gt '50')
{
$crit_counter = 1
Write-Host "CRITICAL: Failed to kill service more than 50 times!"
}
}
but it's probably wrong because I haven't given any argument to Get-Content
Yes. That is the first issue. Have a look at Get-Help <command> and or docs like TechNet when you are lost. For the core cmdlets you will always see examples.
Second, Get-Content, returns string arrays (by default), so if you are doing a numerical comparison you need to treat the value as such.
Thirdly you have a line break between foreach-object cmdlet and its opening brace. That will land you a parsing problem and PS will prompt for the missing process block. So changing just those mentioned ....
Get-ChildItem -Path C:\temp_erase\PID -Directory -Filter *.txt | ForEach-Object{
[int]$warning_counter = Get-Content $_.FullName
if ($warning_counter -gt '50')
{
$crit_counter = 1
Write-Host "CRITICAL: Failed to kill service more than 50 times!"
}
}
One obvious thing missing from this is you do not show which file triggered the message. You should update your notification/output process. You also have no logic validating file contents. The could easily fail, either procedural or programically, on files with non numerical contents.
I have what I'm assuming to be a simple question. I have a large block of code that I have written. In short, it downloads a dynamic list of file names and extensions, imports a csv with the roots of every one of our shares, and compares the files to find matches.
$Directory = Get-ChildItem -Path $path -Recurse -Depth 2 -Include $ExtList -Force
Super simple, right? Except the depth parameter isn't working. It recursively searches through every single level. If I do this:
$Directory = Get-ChildItem -Path $path -Depth 2 -Force
The Depth parameter works perfectly and it only searches through the two levels. If I don't include recurse or depth, it works as expected by searching only the top level. The only difference is that I'm removing the -Include parameter.
$Path is a variable like \server\root\
$ExtList is an array of filenames
Again, they both work individually, but not together.
I need to have both the depth and include parameters in here. Does anybody know what I am doing wrong, or if it's a glitch?
Edit ---------------------------
Doing "Where-Object" I tried this:
$Directory = Get-ChildItem -Path $path -Recurse -Depth 2 -Force | Where-Object {$_.Extension -like $ExtList}
And... nothing happens. For testing, this script takes about 10 minutes to successfully run on a good day, it finished in less than 1 second. (715ms to be exact)
When I go into debugging, it's like there is nothing being piped into the where-object.
EDIT------
The $ExtList setup looks like this:
#((Invoke-WebRequest -Uri "https://fsrm.experiant.ca/api/v1/get").content | convertfrom-json | % {$_.filters})
That will get you the exact list and format that I'm using.
$Path pulls from a csv file that looks like this:
This CSV has over 3000 different shares in it. I know it's weird, yes I have to do it this way for our infrastructure.
After switching to run on a Windows Server 2012 R2 box, the depth parameter works with include. It seems to have been a bug with the version that I was on.
My question is pretty much the same as the one posted on metafilter.
I need to use a PowerShell script to scan through a large amount of files. The issue is that it seems that the "Get-ChildItem" function insists on shoving the whole folder and file structure in memory. Since the drive has over a million files in over 30,000 folders, the script needs a lot of memory.
http://ask.metafilter.com/134940/PowerShell-recursive-processing-of-all-files-and-folders-without-OutOfMemory-exception
All what I need is the name, size and location of the files.
What I do since now is:
$filesToIndex = Get-ChildItem -Path $path -Recurse | Where-Object { !$_.PSIsContainer }
It works but I don't want to punish my memory :-)
Best regards,
greenhoorn
If you want to optimize the script to use less memory, you need to properly utilize the pipeline. What you are doing is saving the result of Get-ChildItem -recurse into memory, all of it! What you could do is something like this:
Get-ChildItem -Path $Path -Recurse | Foreach-Object {
if (-not($_.PSIsContainer)) {
# do stuff / get info you need here
}
}
This way you are always streaming the data through the pipeline and you will see that PowerShell will consume less memory (if done correctly).
One thing you can do to help is to reduce the size of the objects your saving by paring them down to just the properties you're interested in.
$filesToIndex = Get-ChildItem -Path $path -Recurse |
Where-Object { !$_.PSIsContainer } |
Select Name,Fullname,Length
My Powershell script seems slow, when I run the below code in ISE, it keeps running, doesn't stop.
I am trying to write the list of subfolders in a folder(the folder path is in $scratchpart) to a text file. There are >30k subfolders
$limit = (Get-Date).AddDays(-15)
$path = "E:\Data\PathToScratch.txt"
$scratchpath = Get-Content $path -TotalCount 1
Get-ChildItem -Path $scratchpath -Recurse -Force | Where-Object { $_.PSIsContainer -and $_.CreationTime -lt $limit } | Add-Content C:\Data\eProposal\POC\ScratchContents.txt
Let me know if my approach is not optimal. Ultimately, I will read the text file, zip the subfolders for archival and delete them.
Thanks for your help in advance. I am new to PS, watched few videos on MVA
Add-Content, Set-Content, and even Out-File are notoriously slow in PowerShell. This is because each call opens the file, writes to it, and closes the handle. It never does anything more intelligently than that.
That doesn't sound bad until you consider how pipelines work with Get-ChildItem (and Where-Object and Select-Object). It doesn't wait until it's completed before it begins passing objects into the pipeline. It starts passes objects as soon as the provider returns them. For a large result set, this means that the objects are still feeding in the pipeline long after several have finished processing. Generally speaking, this is great! It means the system will function more efficiently, and it's why stuff like this:
$x = Get-ChildItem;
$x | ForEach-Object { [...] };
Is significantly slower than stuff like this:
Get-ChildItem | ForEach-Object { [...] };
And it's why stuff like this appears to stall:
Get-ChildItem | Sort-Object Name | ForEach-Object { [...] };
The Sort-Object cmdlet needs to waits until it's received all pipeline objects before it sorts. It kind of has to to be able to sort. The sort itself is nearly instantaneous; it's just the cmdlet waiting until it has the full results.
The issue with Add-Content is that, well, it experiences the pipeline not as, "Here's a giant string to write once," but instead as, "Here's a string to write. Here's a string to write. Here's a string to write. Here's a string to write." You'll be sending content to Add-Content here line by line. Each line will instantiate a new call to Add-Content, requiring the file to open, write, and close. You'll likely see better performance if you assign the result of Get-ChildItem [...] | Where-Object [...] to a variable, and then write the entire variable to the file at once:
$limit = (Get-Date).AddDays(-15);
$path = "E:\Data\PathToScratch.txt";
$scratchpath = Get-Content $path -TotalCount 1;
$Results = Get-ChildItem -Path $scratchpath -Recurse -Force -Directory | `
Where-Object{$_.CreationTime -lt $limit } | `
Select-Object -ExpandPropery FullName;
Add-Content C:\Data\eProposal\POC\ScratchContents.txt -Value $Results;
However, you might be concerned about memory usage if your results are actually going to be extremely large. You can actually use System.IO.StreamWriter for this purpose, too. My process improved in speed by nearly two orders of magnitude (from 12 hours to 20 minutes) by switching to StreamWriter and also only calling StreamWriter when I had about 250 lines to write (that seemed to be the break-even point for StreamWriter's overhead). But I was parsing all ACLs for user home and group shares for about 10,000 users and nearly 10 TB of data. Your task might not be as large.
Here's a good blog explaining the issue.
Do you have at least PowerShell 3.0? If you do you should be able to reduce the time by filtering out the files since you are returning those as well.
Get-ChildItem -Path $scratchpath -Recurse -Force -Directory | ...
Currently you are returning all files and folders then filtering out the files with $_.PSIsContainer which would be slower. So should end up with something like this
Get-ChildItem -Path $scratchpath -Recurse -Force -Directory |
Where-Object{$_.CreationTime -lt $limit } |
Select-Object -ExpandPropery FullName |
Add-Content C:\Data\eProposal\POC\ScratchContents.txt