I'm looking for a PowerShell version of the following Linux command for CI with GitHub actions:
find . -name "*.py" -not -path "./exclude_dir/*" | xargs pylint
here is where I'm now:
get-childitem -path $pwd -include *.py -recurse -name
at the moment I have no idea how to exclude "exclude_dir" and apply pylint to the selected python files.
Any help would be very appreciated!
Thanks in advance!
Best,
Alexey
While Get-ChildItem does have an -Exclude parameter, it only operates on the file-name part, not on the full path.
Therefore, you must perform the exclusion filtering after the fact, using the negated form of -like, the wildcard matching operator
pylint ((Get-ChildItem -Recurse -Name -Filter *.py) -notlike 'exclude_dir/*')
Note the use of -Filter rather than -Include, which speeds up the operation, because filtering happens at the source rather than being applied by PowerShell after the fact.
However, given that you're seemingly only excluding a single top-level folder, you could try:
pylint (Get-ChildItem -Recurse -Path * -Filter *.py -Exclude exclude_dir)
Note that I've omitted -Name in this case, because it wouldn't work properly in this scenario. As a result, the matching files are implicitly passed as full paths to pylint.
As of PowerShell 7.0, -Name exhibits several problematic behaviors, which are summarized in this answer.
Related
I'm a bit confused by the -Depth flag for Get-ChildItem. The following works great (finds all files and folders only one deep under "C:\Program Files"):
dir 'C:\Program Files\' -Depth 1
But if I then want to extend it to find only *.txt type files, I cannot find how to do that (following just give weirdly unexpected output where -Depth 1 is ignored and it instead does the equivalent of a -Recurse to all subfolders no matter how deep):
dir 'C:\Program Files\*.txt' -Depth 1
dir 'C:\Program Files\' -Include *.txt -Depth 1
dir 'C:\Program Files\*' -Include *.txt -Depth 1
How do we use -Depth to a specific depth for Get-ChildItem and a required file-pattern?
The behavior you're seeing is a bug in Windows PowerShell, that has since been fixed in PowerShell [Core] 6+ - see this GitHub issue.
Given that Windows PowerShell is no longer actively developed, it is unlikely that the bug will be fixed.
To spell it out, Windows PowerShell ignores -Depth's depth constraint in the following cases:
with -Include or -Exclude
if the (implied) -Path argument contains wildcard characters.
While recursion is still performed, no depth limit is imposed; in effect, -Depth behaves like -Recurse (alone) in these cases.
Workarounds:
For -Include and wildcard-based -Path arguments where the wildcards are limited to the last path component:
Use -Filter instead, as shown in Wasif Hasan's answer.
Caveat: -Filter is usually preferable anyway for its superior performance, but its wildcard language is less powerful than PowerShell's and has legacy quirks - notably, character sets and ranges ([...]) are not supported and in Windows PowerShell a filter such as *.xls also matches *.xlsx files, for instance - see this answer.
For -Exclude:
Use only -Depth and perform filtering after the fact with a Where-Object call; e.g.,
Get-ChildItem -File 'C:\Program Files\' -Depth 1 | Where-Object Name -NotLike *.txt
[Probably rarely needed] For wildcard-based -Path arguments with wildcard characters (also) in a component other than the last one (e.g., C:\foo*\bar)
Use -Recurse and perform filtering after the fact with Where-Object; in this case, you'll also have to weed out too-deep paths by counting the number of their components.
The issue gets solved when you use Filter instead of Include. Filter parameter will return file in correct pattern with depth. (TESTED)
dir 'C:\Program Files\' -Filter *.txt -Depth 1
In older versions of PowerShell there was no depth, in that case the above can also be
Get-ChildItem -Path "C:\DIRECTORY\*","C:\DIRECTORY\*\*"
If it is pure for filenames then
(Get-ChildItem -Path "c:\program files" -file -Depth 3 -Force -erroraction SilentlyContinue).FullName
Is identical to the ancient kind of tricks, i.e.
(cmd.exe /c dir "c:\program files" /b /a-d /s)|foreach {if ($_.split("\").length -le 5){$_}}
It's amazing that PowerShell is even faster than the above line! I remember that a few years ago that was not the case, but I just tested it and it was 3-4 times faster
To further clarify the answer by Wasif Hasan
As I was going through the official documentation for the Get-ChildItem, it is stated there
When using the -Include parameter, if you do not include an asterisk
in the path the command returns no output.
Which means that the Depth will be ignored automatically as the behavior required for the Include is recursive. Further some details of the -Include reveals these points.
If the Recurse parameter is added to the command, the trailing
asterisk (*) in the Path parameter is optional. The Recurse parameter
gets items from the Path directory and its subdirectories. For
example, -Path C:\Test\ -Recurse -Include *.txt
So the behavior you are looking for is in the Filter flag for the Get-ChildItem which do not requires any wild cards
For me the Depth flag with any other flag that accepts wild cards in the path do not make sense as the purpose of the Depth flag is to restrict the depth of search in the Items where as specifying a wild card excludes that particular purpose.
You can try this by simply using this command and you will see that the Depth parameter is not effective if you specify a wild card in the path for example
Get-ChildItem -Path C:\DIRECTORY\* -Depth 1
and
Get-ChildItem -Path C:\DIRECTORY\* -Depth 2
are going to return the same results.
Hope this helps clarify some issues
This question already has answers here:
How to select files that have no extension using powershell
(3 answers)
Closed 4 years ago.
I want to get a list of files that don't have a filename extension. Consider the content of my directory to be:
folder
file1
file2.mp4
My goal would be to get file1 only.
Running Get-ChildItem -Exclude *.* -File returned nothing.
Running Get-ChildItem -Exclude *.* returned folder and file1.
Running Get-ChildItem -File returned file1 and file2.mp4.
Any idea if there is any way of using Get-ChildItem to only return file1?
In PSv3+, but doesn't work in PowerShell Core v6.x, fixed in v7 (see this GitHub issue):
Get-ChildItem -File -Filter *.
-File limits output to just files (as opposed to directories).
-Filter *. selects only those files that have no extension.
-Filter is generally preferable to -Include / -Exclude for performance reasons, because it filters at the source, rather than returning all objects and letting PowerShell do the filtering.
In PSv2, where the -File switch isn't available, you need an additional Where-Object call to limit the results to files, as TheIncorrigible1 points out:
Get-ChildItem -Filter *. | Where-Object { -not $_.PSIsContainer }
Slower PowerShell Core solution:
Get-ChildItem -File | Where-Object -Not Extension
Optional background information:
That a -Filter argument is processed by the underlying provider, not by PowerShell, means that its behavior may differ from PowerShell's, which is indeed the case here: the FileSystem provider uses the Windows API's wildcard-expression matching, which has fewer features than PowerShell's as well as some historical quirks; also, it is limited to a single wildcard expression, whereas -Include / -Exclude support multiple ones (separated with ,).
Here, however, -Filter offers something that PowerShell's wildcard matching doesn't: using *. to match files / directories without extension.
-Include / -Exclude generally offer functional advantages at the expense of performance, but they have their own limitations and quirks:
*. isn't supported to match items without extension in PowerShell's wildcard language, and there is no wildcard-based way to achieve this that I know of.
-Include / -Exclude operate on the last component of the specified or implied path, so if you're implicitly targeting the current directory, they apply to that directory path, not to the individual files inside.
Specifying -Recurse changes that, but that searches the entire directory subtree.
While you should be able to add -Depth 0 to limit matches to the immediate child items while still being able to apply -Include / -Exclude, this is broken as of Windows PowerShell v5.1: The -Depth argument is ignored in this case.
This problem has been fixed in PowerShell Core, however.
In short: -Include / -Exclude offer no solution here.
When using "-filter":
Get-ChildItem -file -filter "*.txt" | foreach-object { write-host $_.FullName }
I get a listing of the 4 .txt files that's in the current folder.
I tried using "-include"
Get-ChildItem -file -include *.txt | foreach-object { write-host $_.FullName }
Get-ChildItem -file -include *txt | foreach-object { write-host $_.FullName }
and I get nothing. I tried with and without the "-file" parameter and it makes no difference.
I've looked at various guides/examples (ss64.com/TechNet and etc) and supposedly I am doing it right.
Any ideas what I could be doing wrong? Thanks!
From the Get-Help page for Get-ChildItem:
The -Include parameter is effective only when the command includes the -Recurse parameter or the path leads to the contents of a directory, such as C:\Windows*, where the "*" wildcard character specifies the contents of the C:\Windows directory.
You'll note that you don't get a syntax error if you specify -include and don't specify -recurse in spite of the fact that whatever it does is literally undefined. You'll also note that C:\Windows* is not a normal wildcard expression for "all files in the C:\Windows directory". It's a wildcard expression for "all items that start with 'Windows' in the C:\ directory and may or may not have an extension". I have no idea what the authors of Get-ChildItem think this parameter is supposed to do. They've done a fantastically poor job of documenting it and implementing it.
Consequently, I avoid the -Include parameter as broken/badly documented. I don't know what it's supposed to do that -Filter doesn't. I've read articles about what it does exactly. It "passes the value to the underlying provider to filter at that level" in some manner. I don't know why they assume that a sysadmin will know what that really means. My understanding is that it's the difference between calling DirectoryInfo.GetFiles() on each directory item and calling DirectoryInfo.GetFiles('*.txt') on each directory item, but most sysadmins aren't going to know what that means. However, it's so oddly behaved that I don't trust it, so even though I am about 95% sure of what it does... I still never use it.
Instead, I just pipe to Where-Object:
Get-ChildItem -file | Where-Object Extension -eq '.txt' | [...]
Also note that Get-ChildItem is broken with -LiteralPath, -Recurse and -Include in some versions of PowerShell, and will instead return all items.
Compare:
Get-ChildItem -LiteralPath $PSHOME *.exe -Recurse # works
Get-ChildItem -Path $PSHOME -Include *.exe -Recurse # works
Get-ChildItem -LiteralPath $PSHOME -Include *.exe -Recurse # does NOT work
Issue reported here for v6.
These work for me without recursion:
Get-ChildItem -Path "C:\Users\Athom\Desktop\*.txt"
Get-ChildItem -Path ".\*.txt"
Or Just add the recursion parameter:
Get-ChildItem -Include *.txt -Recurse
Let me start by saying that I've looked at Unable to exclude directory using Get-ChildItem -Exclude parameter in Powershell and How can I exclude multiple folders using Get-ChildItem -exclude?. Neither of these has an answer that solves my problem.
I need to search a directory recursively for files with a certain extension. For simplicity, let's just say I need to find *.txt. Normally, this command would suffice:
Get-ChildItem -Path 'C:\mysearchdir\' -Filter '*.txt' -Recurse
But I have a major problem. There's a node_modules directory buried somewhere inside C:\mysearchdir\, and NPM creates extremely deep nested directories. (The detail of it being an NPM managed directory is only important because this means the depth is beyond my control.) This results in the following error:
Get-ChildItem : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
I believe this error bubbles up from the limitations in the .NET IO libraries.
I can't search in the other directories around it very easily. It's not at the top of the directory; it's deeper in, say at C:\mysearchdir\dir1\dir2\dir3\node_modules, and there are directories I need to search at all those levels. So just searching the other directories around it is going to be cumbersome and not very maintainable as more files and directories are added.
I've tried to -Exclude parameter without any success. That isn't surprising since I just read that -Exclude is only applied after the results are fetched. I can't find any real info on using -Filter (as is noted in this answer).
Is there any way I can get Get-ChildItem to work, or am I stuck writing my own recursive traversal?
Oh, man, I feel dumb. I was facing the same problem as you. I was working with #DarkLite1's answer, trying to parse it, when I got to the "-EA SilentlyContinue" part.
FACEPALM!
That's all you need!
This worked for me, try it out:
Get-ChildItem -Path 'C:\mysearchdir\' -Filter '*.txt' -Recurse -ErrorAction SilentlyContinue
Note: This will not exclude node_modules from a search, just hide any errors generated by traversing the long paths. If you need to exclude it entirely, you're going to need a more complicated solution.
Maybe you could try something like this:
$Source = 'S:\Prod'
$Exclude = #('S:\Prod\Dir 1', 'S:\Prod\Dir 2')
Get-ChildItem -LiteralPath $Source -Directory -Recurse -PipelineVariable Dir -EV e -EA SilentlyContinue |
Where {($Exclude | Where {($Dir.FullName -eq "$_") -or ($Dir.FullName -like "$_\*")}).count -eq 0}
I would like to search for a folder in a specific directory and subdirectorys.
I tried googling it, but didn't really find any usefull examples.
Get-ChildItem C:\test -recurse | Where-Object {$_.PSIsContainer -eq $true -and $_.Name -match "keyword"}
I believe there's no dedicated cmdlet for searching files.
Edit in response to #Notorious comment:
Since Powershell 3.0 this is much easier, since switches -Directory and -File were added to Get-ChildItem. So if you want it short you've got:
ls c:\test *key* -Recurse -Directory
With command alias and tab-completion for switches it's a snap. I just missed that the first time.
Here is my version, which is only slightly different:
gci -Recurse -Filter "your_folder_name" -Directory -ErrorAction SilentlyContinue -Path "C:\"
some more info:
-Filter "your_folder_name"
From documentation: Filters are more efficient than other parameters. The provider applies filter when the cmdlet gets the objects rather than having PowerShell filter the objects after they're retrieved. The filter string is passed to the .NET API to enumerate files. The API only supports * and ? wildcards.
-Directory
Only examine directories, also could be -File
-ErrorAction SilentlyContinue
Silences any warnings
-Path "C:\"
Specifies a path to start searching from
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.management/get-childitem?view=powershell-7