How do I prevent Get-ChildItem from traversing a particular directory? - powershell

Let me start by saying that I've looked at Unable to exclude directory using Get-ChildItem -Exclude parameter in Powershell and How can I exclude multiple folders using Get-ChildItem -exclude?. Neither of these has an answer that solves my problem.
I need to search a directory recursively for files with a certain extension. For simplicity, let's just say I need to find *.txt. Normally, this command would suffice:
Get-ChildItem -Path 'C:\mysearchdir\' -Filter '*.txt' -Recurse
But I have a major problem. There's a node_modules directory buried somewhere inside C:\mysearchdir\, and NPM creates extremely deep nested directories. (The detail of it being an NPM managed directory is only important because this means the depth is beyond my control.) This results in the following error:
Get-ChildItem : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
I believe this error bubbles up from the limitations in the .NET IO libraries.
I can't search in the other directories around it very easily. It's not at the top of the directory; it's deeper in, say at C:\mysearchdir\dir1\dir2\dir3\node_modules, and there are directories I need to search at all those levels. So just searching the other directories around it is going to be cumbersome and not very maintainable as more files and directories are added.
I've tried to -Exclude parameter without any success. That isn't surprising since I just read that -Exclude is only applied after the results are fetched. I can't find any real info on using -Filter (as is noted in this answer).
Is there any way I can get Get-ChildItem to work, or am I stuck writing my own recursive traversal?

Oh, man, I feel dumb. I was facing the same problem as you. I was working with #DarkLite1's answer, trying to parse it, when I got to the "-EA SilentlyContinue" part.
FACEPALM!
That's all you need!
This worked for me, try it out:
Get-ChildItem -Path 'C:\mysearchdir\' -Filter '*.txt' -Recurse -ErrorAction SilentlyContinue
Note: This will not exclude node_modules from a search, just hide any errors generated by traversing the long paths. If you need to exclude it entirely, you're going to need a more complicated solution.

Maybe you could try something like this:
$Source = 'S:\Prod'
$Exclude = #('S:\Prod\Dir 1', 'S:\Prod\Dir 2')
Get-ChildItem -LiteralPath $Source -Directory -Recurse -PipelineVariable Dir -EV e -EA SilentlyContinue |
Where {($Exclude | Where {($Dir.FullName -eq "$_") -or ($Dir.FullName -like "$_\*")}).count -eq 0}

Related

Powershell: The specified path, or filename, or both, are too long - Directory Excludes

We’ve been running this script fine previously, but recently we’ve been getting some issues (See below error) with our config transform step from our TFS Build.
$serviceFiles = Get-ChildItem $localWorkspace -Recurse | where {$_.Extension -eq ".exe"}
We recently switched to using Gulp to compile our CSS and JS which has given us a “node_modules” folder. It looks like it’s trying to crawl these folders and actually gets to the directory length limit. I’ve tried various suggestions I’ve found from googling and other related questions on SO, but none seem to be working for me, it’s still hitting the same error (and I assume isn’t actually excluding these folders)
This is an example of a modified version I've tried using to exclude the folders
$excludedDirectories = #("*node_modules*", "*packages*", "*Common*");
Write-Verbose $localWorkspace
# get services for config types (service configs named same as assmebly .exe)
$serviceFiles = Get-ChildItem $localWorkspace -Recurse -Exclude $excludedDirectories | ?{$_.Extension -eq ".exe" -and $_.FullName -notmatch '*node_modules*' }
I've tried some variations on this based on other SO questions and answers, but the solution evades me. I've read from a few sources that the -Exclude doesn't work for a lot of people in most situations, so i tried the solution of a where clause to exclude a folder (I want to exclude multiple, but I tried just node_modules to see if I could get it past that, the other folders aren't too deep)
I want to exclude the node_modules directory, along with a couple of others that don't need to be checked for a transform. Any help would be greatly appreciated, thank you.
Unfortunately, the -Exclude parameter won't exclude the directories before enumerating all the files. So it still errors out. You need to exclude them earlier than that.
If those directories only exist at the top level, you can enumerate the top level directories, exclude the ones you don't want, and then examine the contents inside the remaining directories. Like this:
$excludedDirectories = #("node_modules", "packages", "Common")
$serviceFiles = Get-ChildItem $localWorkspace -Exclude $excludedDirectories | % { Get-ChildItem $_ -Recurse } | ? {$_.Extension -eq ".exe" }
Also, you can make this a bit simpler by using -Include:
$serviceFiles = Get-ChildItem $localWorkspace -Exclude $excludedDirectories | % { Get-ChildItem $_ -Recurse -Include "*.exe" }
Notice what I did. I removed the top level -Recurse and filtered out those directories. Then I used -Recurse on the remaining children of the top-most parent, giving us the files we're looking for.
If the directories you need to filter out appear deep in the hierarchy or at multiple levels, you'll have to write your own recursive traversal function:
function Get-ChildItemRecursiveExclude(
[Parameter(Mandatory=$true)][string[]]$Path,
[Parameter(Mandatory=$true)][string[]]$ExcludedDirNames
) {
$immediateChildren = Get-ChildItem $Path -Exclude $ExcludedDirNames
foreach ($c in $immediateChildren) {
# Uncaptured output is returned
$c
if (Test-Path $c -PathType Container) {
Get-ChildItemRecursiveExclude $c $ExcludedDirNames
}
}
}
$serviceFiles = Get-ChildItemRecursiveExclude $localWorkspace #("node_modules", "packages", "Common") | ? { $_.Extension -eq ".exe" }
Overall, the basic idea is that you have to keep PowerShell from traversing down into node_modules in the first place. npm creates very deep hierarchies that surpass the old limits on path lengths. (I'm not really clear why .NET still enforces them, but it does even though some of the underlying Windows API no longer does. for example, robocopy and several third party runtimes, like Node, don't bother with them.)

Powershell get-childitem exclude directory several levels deep

I have a folder containing changed files to deploy to another server. I am building a list of files below the root, recursively, and I want to skip an entire folder. This is not working:
gci -path \\myfolder -recurse -exclude "*\excludeme\*" | where{! $_PSIsContainer}
It is going ahead and listing all the files in excludeme as well. Even this didn't work:
gci -path \\myfolder -recurse -exclude "\\myfolder\excludeme\*" | where etc.
I should note that "excludeme" is a folder that contains a number of software project folders, i.e. a whole mess of subfolders in itself. Help!
You should be able to use -Exclude if you simple enter the name of the folder:
GCI \\MyFolder -Recurse -Exclude "ExcludeMe"
With that said, if it doesn't work for you (and it wouldn't surprise me, because the File System Provider is horrible at filtering IMHO), you can always filter after you get the results, it will just be a little slower:
GCI \\MyFolder -Recurse | Where{$_.FullName -NotMatch "\\ExcludeMe(\\|$)"}

Where can I download copyforbuild.bat?

I am interested in a power shell script that will do a recursive search for files using a filename pattern.
I thought I would find it with Copyforbuild.bat, but I cannot find where I can download it or copy the source code.
Any help is appreciated.
I am not sure what copyforbuild.bat is, if that is from an application, but that is not a powershell script. If you are looking for a file recursively you can use Get-ChildItem (or GCI for short).
gci -Path C:\ -Recursive -Filter somefile.ext
That would search the whole C:\ drive for a file named "somefile.ext". If that doesn't solve what you are trying to do, could you be more specific as to what you are looking for?
If you need to match patterns then you probably want to use a Regular Expression (RegEx). I'm not sure how powershell will deal with "too long" folders (over 256 character path length), but in general you could do:
gci -Path C:\ -Recursive | ? {$_.Name -match "<RegEx Pattern>"}
For more information on RegEx see http://www.regular-expressions.info/powershell.html
gci -Path "C:\tasks" -recurse -Filter "rspc.doc*" -ErrorAction SilentlyContinue | Out-File c:\temp\rspc9.txt
This command when entered into PowerShell will look for a file with the pattern rspc.doc* that is in C:\Tasks and all folders underneath it. It redirects the output to a file.
If there is a folder with a long name or there is some other error, then it will continue to search, ignoring the error.

How to limiting files searched by Get-ChildItem (or limiting depth of recursion)?

Background
There is a directory that is automatically populated with MSI files throughout the day. I plan on leveraging Task Scheduler to run the script shown below every 15 minutes. The script will search the directory and copy any new MSIs that have been created in the last 15 minutes to a network share.
Within this folder C:\ProgramData\flx\Output\<APP-NAME>\_<TIME_STAMP>\<APP-NAME>\ there are two other folders: Repackaged and MSI Package. The Repackaged folder does not need to be searched as it does not contain any MSIs. Also I have found that it needs to be excluded in some way to prevent this error:
Get-ChildItem : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
At line:14 char:32
+$listofFiles=(Get-ChildItem <<<< -Recurse -Path $outputPath -Include "*.msi" -Exclude "*.Context.msi" | where {$_.LastAccessTime -gt $time.AddMinutes($minutes)})
+ CategoryInfo : ReadError: C:\ProgramData\...xcellence\Leg 1:String) [Get-ChildItem], PathTooLongException
+ FullyQualifiedErrorId : DirIOError,Microsoft.PowerShell.Commands.GetChildItemCommand
Limitations
I am stuck using Powershell v1.0
I have no control over the directory structure of the source location
Updated:
I don't know the app name or the what the time stamp will be. That is something else that is out of my control.
Current plans
I have read about using -Filter and I am aware of filters that are similar to functions but I wasn't able to come up with any ideas of how to use them. My only thought at the moment would be to do something like:
$searchList=Get-ChildItem "all instances of the MSI Package folder"
foreach($folder in $searchList){
$listofFiles=Get-ChildItem "search for *.msi"
foreach($file in $listofFiles){"Logic to copy MSI from source to destination"}
}
However...I thought that there might be a more efficient way of doing this.
Questions
How can I limit depth that Get-ChildItem searches?
How can I limit the Get-ChildItem search to C:\ProgramData\flx\Output\<APP-NAME>_<TIME_STAMP>\<APP-NAME>\MSI Package
How can I only search folders that have been accessed in the last 15 minutes? I don't want to waste time drilling down into folders when I know MSI has already been copied.
Any additional advice on how to make this script more efficient overall would also be greatly appreciated.
Script
My current script can be found here. I kept getting: "Your post appears to contain code that is not properly formatted as code" and gave up after the fourth time trying to reformat it.
You can try this
dir C:\ProgramData\flx\Output\*\*\*\*\* -filter *.msi
this search all .msi files at this level
C:\ProgramData\flx\Output\<APP-NAME>\_<TIME_STAMP>\<APP-NAME>\Repackaged or 'MSI Package' or whatever else present folder
without recursion, this avoid too deep folder that give you error.
Pipe the result to:
Where {$_.LastAccessTime -gt (Get-Date).AddMinutes(-15)} #be sure no action on file is taken before the dir command
or
Where {$_.LastWriteTime -gt (Get-Date).AddMinutes(-15)} #some file can be re-copied maybe
With help from C.B. this is my new search which eliminates the issues I was having.
Changed -Path to C:\ProgramData\flx\Output\*\*\*\* to help limit the depth that was searched.
Used -Filter instead of -Include and put the -Exclude logic into the where clause.
Get-ChildItem -Path C:\ProgramData\flx\Output\*\*\*\* -Filter "*.msi" | where {$_.Name -notlike "*.Context.msi" -and $_.LastAccessTime -gt (Get-Date).AddMinutes(-15)}
You can't limit the recursion depth of Get-ChildItem except to not use -Recurse i.e. Get-ChildItem is either depth = 0 or N.
Set up variables for app name and timestamp e.g.:
$appName = "foo"
$timestamp = Get-date -Format HHmmss
Get-ChildItem "C:\ProgramData\flx\Output\${appName}_$timestamp\$appName\MSI Package" -force -r
You can filter the results like so:
Get-ChildItem <path> -R | Where {$_.LastWriteTime -gt (Get-Date).AddMinutes(-15)}

Get powershell to display all paths where a certain file can be found on a drive

I'm trying to build a function that will show me all path's where a certain filename is located. The function would take one parameter, that being the file name.
The result would be either a list of all paths, or a message saying there's no such file on the system.
I'm new to Powershell, and I'm not getting the syntax just yet.
I've tried this:
Get-ChildItem -Path -Include notepad.exe
But that threw an error message. I'm currently trying:
$i="notepad.exe"
Foreach ($i in Get-ChildItem c:\ -Recurse){echo -Path}
Started that now, it's still running, don't know what'll happen, really.
EDIT: echo'd an enormous amount of lines that just say "-Path"...
Can anybody help with this problem? I'm running Powershell 1.0 by the way.
So, to explain what I wish to see when executing this command, here is an example of what I expect after looking for *.txt:
C:/foo.txt
C:/A/foobar.txt
C:/A1/foo.txt
And so on, listing the path to all .txt files on my harddrive. Only the paths, one per line, no extra info needed.
EDIT2:
I've done it. I'm gonna leave this question up for those who make look for this in the future.
The function I used was this(this specific example will hand you a list of all .zip files on your harddrive, edit where needed):
Get-ChildItem -Path c:\ -Include "*.zip" -Recurse -Force -Name > c:\listOfPaths.txt
This created a file called listOfPaths.txt on my C:\ folder and this contained a list of all occurences of any file ending with .zip in all subfolders of my harddrive.
The "c:\" bit isn't mentioned, but I don't mind.
EDIT3:
thanks capar for a more complete version.
Here is capar's code(or how I got it to work, since Get-Children doesn't work in 1.0)
Get-ChildItem -Path c:\ -Recurse *.txt | Select-Object -Property FullName
Since it's Friday night, I decided to play with Power Shell to see if I can help :)
This comes pretty close to what you are asking for I think:
Get-ChildItem -Path c:\ -Recurse *.txt | Select-Object -Property FullName
If it helps, this command will list the properties of any object that will be returned by Get-ChildItem:
Get-ChildItem | Get-Member
ls c:\ -r | ? {$_.name -eq "notepad.exe"}
Get-Children is not recognized in Powershell V3 either. It would be great if someone removed that bad example.
As a warning to anyone searching for files: C:\ on today's hard drives will take a long time to run. You are well advised to narrow your search as much as you can. Since your folder structure might include spaces or special characters, use the typewriter quote (") or apostrophe (') delimeters.
$mylistoffiles = Get-ChildItem -Path 'C:\Windows\Setup\Scripts' -Recurse *.cmd | Select-Object -Property FullName
$mylistoffiles