If i run the following:
Measure-Command -Expression {gci -Path C:\ -Recurse -ea SilentlyContinue | where Extension -eq ".txt"}
Measure-Command -Expression {gci -Path C:\ -Filter *.txt -Recurse -ea SilentlyContinue}
The second expression is always faster that the first one, im guessing its because it doesnt have to use the pipeline.
I thought maybe in the Pipeline method PowerShell recursed my drive and passed a collection of objects to the where clause, that would have to iterate through the items again, but i ruled that out, because if you run the first expression you can see it return output as it is recursing. So why is the Pipeline method slower?
Using Where-Object is always slower than using the built in parameters of the left hand side command. You first bring ALL objects to your shell and only then starts filtering them (client side filtering).
With regard to the -Filter parameter, it works faster because it performs on the provider level (server side filtering), objects are checked once accessed and you get back only the ones that match your criteria.
Shay's answer is totally correct. I wanted to touch on your secondary question a bit, though. Here's what's happening under the hood in the pipeline:
gci -Path C:\ -Recurse -ea SilentlyContinue | where Extension -eq ".txt"
gci will start searching for files and directories at or under c:\, any extension. As it finds each one, that one result is passed on to Where-Object, which will discard it if the extension is not .txt. If the extension is .txt, that object is passed on in the pipeline, and out to the console (or to a variable, or whatever). Then gci will continue its search, when it finds the next file, it will pass it on, etc. So although it might take a couple minutes to search the entire c:\ drive, you get partial results streamed back to you almost immediately, since the pipeline operates one object at a time.
What is not happening is that gci does the full disk search all at once, then hands the complete results set to Where-Object only when it's complete.
Related
I'm working on a project where I have to apply a file to multiple folders every so often. I'm trying to learn some PowerShell commands to make this a little easier. I came up with the following script, which works, but I feel that this is too verbose and could be distilled down with a better script:
[string]$sourceDirectory = "C:\Setup\App Folder Files\*"
# Create an array of folders
$destinationDirectories = #(
'C:\Users\GG_RCB1\Documents\',
'C:\Users\GG_RCB2\Documents\',
'C:\Users\LA_RCB1\Documents\',
'C:\Users\PR_RCB1\Documents\',
'C:\Users\PQ_RCB1\Documents\',
'C:\Users\PQ_RCB2\Documents\',
'C:\Users\XC_RCB1\Documents\',
'C:\Users\XC_RCB2\Documents\',
'C:\Users\XC_RCB3\Documents\',
'C:\Users\XC_RCB4\Documents\',
'C:\Users\XC_RCB5\Documents\',
'C:\Users\XC_RCB6\Documents\',
'C:\Users\XC_RCB7\Documents\',
'C:\Users\XC_RCB8\Documents\')
# Perform iteration to create the same file in each folder
foreach ($i in $destinationDirectories) {
Copy-item -Force -Recurse -Verbose $sourceDirectory -Destination $i
}
I go into this process knowing that every folder in the User folder area is going to have the same format: _RCB<#>\Documents\
I know that I can loop through those files using this code:
Get-ChildItem -Path 'C:\Users'| where-object {$_.Name -match "^[A-Z][A-Z]_RCB"}
What I'm not sure how to do is to how, within that loop, drill down to the Documents folder and do the copy. I want to avoid having to keep updating the array from the first code sample, particularly when I know the naming convention of the subfolders in the Users folder. I'm just looking for a cleaner way to do this.
Thanks for any suggestions!
Ehh, I'll go ahead and post what I had in mind as well. Not to take away from #Mathias suggestion in the comments, but to offer my solution, here's my take:
Get-ChildItem -Path "C:\users\[A-Z][A-Z]_RCB*\documents" |
Copy-Item -Path $sourceDirectory -Destination { $_.FullName } -Recurse -WhatIf
Since everyone loves the "One-Liners" that can accomplish your needs. Get-ChildItem accepts wildcard-expressions in it's path which let's us accomplish this in one go. Given that your directories are...
consistent with the same naming pattern,
[A-Z][A-Z]_*
and the folder destination is the same.
Documents
Luckily, Copy-Item also has some cool features on it's own such as being able to use a script block that will allow the passing of $_.FullName property as it's destination, while they are passed down the pipeline one at a time.
Remove the -WhatIf common parameter when you've dictated the results are what you're after.
I'm trying to batch-convert heic images to png images using Powershell. What I have tried:
Get-ChildItem -Include ('*.HEIC', '*.heic') -File | & .\bin\vips.exe copy $_.Name "$(_.BaseName).png"
Pause
and
Get-ChildItem -Include ('*.HEIC', '*.heic') -File | & .\bin\vips.exe copy $_.Name ($_.BaseName + '.png')
Pause
Both times I'm getting an error VipsForeignLoad: file ".png" does not exist which tells me it treats ".png" as the first (and only) argument and ignores the object Name and Basename properties.
You're missing a $. "$($_.BaseName).png" I would use -Filter vs -Include as it is more efficient.
Edited: Try this approach bypassing the Pipe and see if you get a different result. I've also added some additional code to insure everything is fully evaluated. If this approach works you can experiment with reducing come of the $() evaluation levels.
Also are you using Linux? Some of my googling led me to believe you might be. If so you should specify this in your tags for clarity.
Clear-Host
$x = Get-ChildItem -Filter "*.HEIC" -File
ForEach ($File in $x) {
& .\bin\vips.exe copy "$($File.FullName)" $("$($File.BaseName)" + ".pdf")
}
Also note the file extension in the Filter is case insensitive so no need to repeat.
I'd also recommend adding the -Path parameter for clarity rather than assuming the default directory but that's just me.
HTH
If I execute:
Get-ChildItem *.ext -recurse
the output consists of a series of Directory sections followed by one or more columns of info for each matching file separated by said directory sections. Is there something like the Unix find command? In which each matching file name appears on a single line with its full relative path?
Get-Childitem by default outputs a view for format-table defined in a format xml file somewhere.
get-childitem | format-table
get-childitem | format-list *
shows you the actual properties in the objects being output. See also How to list all properties of a PowerShell object . Then you can pick and choose the ones you want. This would give the full pathname:
get-childitem | select fullname
If you want an output to be just a string and not an object:
get-childitem | select -expand fullname
get-childitem | foreach fullname
Resolve-Path with the -Relative switch can be used to display the relative paths of a set of paths. You can collect the full path names (FullName property) from the Get-ChildItem command and use the member access operator . to grab the path values only.
Resolve-Path -Path (Get-ChildItem -Filter *.ext -Recurse).FullName -Relative
Note: The relative paths here only accurately reflect files found within the current directory (Get-ChildItem -Path .), i.e. Get-ChildItem -Path NotCurrentDirectory could have undesirable results.
Get-ChildItem's -Name switch does what you want:
It outputs the relative paths (possibly including subdir. components) of matching files as strings (type [string]).
# Lists file / dir. paths as *relative paths* (strings).
# (relative to the input dir, which is implicitly the current one here).
Get-ChildItem -Filter *.ext -Recurse -Name
Note that I've used -Filter, which significantly speeds up the traversal.
Caveat: As of PowerShell 7.0, -Name suffers from performance problems and behavioral quirks; see these GitHub issues:
https://github.com/PowerShell/PowerShell/issues/9014
https://github.com/PowerShell/PowerShell/issues/9119
https://github.com/PowerShell/PowerShell/issues/9126
https://github.com/PowerShell/PowerShell/issues/9122
https://github.com/PowerShell/PowerShell/issues/9120
I am having some problem passing the path plus filename to a parser. There are about 90 files of 1 GB each involved in my task. Each of the file is contained in a folder of its own. All of the folders are contained under a parent folder.
Goal: Ideally, I would like to parse 20 files simultaneously for multitasking and continue to the next 20 until all 90 files are done.
This would mean that I would like to spawn some concurrent parsing of 20 files in a batch at any one given time. In carrying out the parsing, I would like to use measure-command to time the work from beginning to finish.
Script I have used:
Get-ChildItem –Path "E:\\OoonaFTP\\input\\Videos3\\" -Filter *.mp4 -recurse | select -expand fullname
Foreach-Object {
Measure-Command { "E:\OoonaFTP\Ooona_x64_ver_2.5.13\OoonaParser.exe -encode -dat -drm $_.FullName" } | Select-Object -Property TotalSeconds
}
===============================
I have this working batch script with a for statement but doing each iteration one after another. This is not what is the ideal case though. I would really like to accomplish this in PowerShell and with simultaneous tasks.
Could someone please suggest some ways by which I could accomplish this?
Thank you very much!
Thanks for the various suggestions. I'm curious that some of them lead to empty output in my Powershell (PSVersion: 5.1.18362.145).
I tried a number of these and, inspired by some of them, found the best answer for my case at the moment:
Get-ChildItem *.ext -recurse | Select-Object -property fullname
(When I made the window wide enough I got all the info I needed; in general I suppose I might need to do more to get the formatting I want.)
I'm finishing a script in PowerShell and this is what I must do:
Find and retrieve all .txt files inside a folder
Read their contents (there is a number inside that must be less than 50)
If any of these files has a number greater than 50, change a flag which will allow me to send a crit message to a monitoring server.
The piece of code below is what I already have, but it's probably wrong because I haven't given any argument to Get-Content, it's probably something very simple, but I'm still getting used to PowerShell. Any suggestions? Thanks a lot.
Get-ChildItem -Path C:\temp_erase\PID -Directory -Filter *.txt |
ForEach-Object{
$warning_counter = Get-Content
if ($warning_counter -gt '50')
{
$crit_counter = 1
Write-Host "CRITICAL: Failed to kill service more than 50 times!"
}
}
but it's probably wrong because I haven't given any argument to Get-Content
Yes. That is the first issue. Have a look at Get-Help <command> and or docs like TechNet when you are lost. For the core cmdlets you will always see examples.
Second, Get-Content, returns string arrays (by default), so if you are doing a numerical comparison you need to treat the value as such.
Thirdly you have a line break between foreach-object cmdlet and its opening brace. That will land you a parsing problem and PS will prompt for the missing process block. So changing just those mentioned ....
Get-ChildItem -Path C:\temp_erase\PID -Directory -Filter *.txt | ForEach-Object{
[int]$warning_counter = Get-Content $_.FullName
if ($warning_counter -gt '50')
{
$crit_counter = 1
Write-Host "CRITICAL: Failed to kill service more than 50 times!"
}
}
One obvious thing missing from this is you do not show which file triggered the message. You should update your notification/output process. You also have no logic validating file contents. The could easily fail, either procedural or programically, on files with non numerical contents.
EDIT Hoping here to clarify my convoluted and misleading question... based on my mistaken assumption that -file accepts inputs. Thanks for setting me straight and pointing out that it's just a switch parameter; the inputs in my example actually get passed to -path. Sounds like that may be the fastest purely powershell way to search for multiple file types, since -filter accepts only a single input and -include is slower.
The get-childItem documentation says "Filters are more efficient than other parameters, because the provider applies them when retrieving the objects, rather than having Windows PowerShell filter the objects after they are retrieved."
v3 has a new parameter set with a -file parameter, probably meant for excluding directories, to match cmd.exe's dir /a:-d
Like -include and unlike -filter, -file accepts multiple, as in gci -file "*.ldf","*.bak"
So i'm wondering, and have thus far failed to reliably test, if -file is like -filter from a performance perspective, ie "more efficient", or more like the "other parameters" like -include. If -file is a filter, that's nice because afaict -filter only handles one filter at a time, so if you want multiple filters (like *.ldf and *.bak) then you need to either run gci -filter twice or use -include instead. So I'm wondering if -file lets us reap the efficiency benefits of a filter, for multiple filters.
I stumbled on some error text that's got me optimistic. The -file parameter wants -path to be the current directory, so this gci -path $path -file "*.bak","*.ldf" gives an error. Push-location seems like a viable workaround, but here I'm more interested in the content of the error text:
Get-ChildItem : Cannot convert 'System.Object[]' to the type 'System.String' required by parameter 'Filter'. Specified method is not supported.
I called -file but the error complains about "parameter 'Filter'". So maybe -file is efficient like a filter? OTOH, -filter doesn't need -path to be the current directory, so in that respect -file is more like -include.
just one precision:
gci -path $path -file "*.bak","*.ldf"
-file is a switch parameter (as -directory is) and doesn't accept values (To get only files, use the File parameter and omit the Directory parameter. To exclude files, use the Directory parameter and omit the File parameter); then the "*.bak","*.ldf" are implicitly passed to -filter as value, and filter accept only string and not string[]. That's where your error came from.
Regarding performance: using -file or -directory is faster than using a where-object psiscontainer or ? { !$_.psiscontainer} because it's done at provider (filesystem in this case) level.