Powershell script to list large files on multiple servers - powershell

I have been trying to create/modify a Powershell script that allows me to automate finding all files across multiple servers larger than 1GB and excluding .ldf and mdf.
I have found a script, but it only looks into the current C Drive and although I've been trying to modify this, I have been unsuccessful.
I'm unsure how to modify this to fit finding multiple servers.
gci -r|sort -descending -property length | select -first 10 name, #{Name="Gigabytes";Expression={[Math]::round($_.length / 1GB, 2)}}
Please help.

Complete Script:
$size=1GB
$path="C:\"
$omit="*.mdf,*.ldf"
Get-ChildItem -Path $path -Exclude $omit -Recurse -OutBuffer 1000|
where {($_.Length -gt $size)}|Select Name, Directory, Length
Sample Output:
Name Directory Length
---- --------- ------
CAP2015-07-29 21-07-08-71.avi C:\ 1216624984
CAP2015-07-29 21-08-17-48.avi C:\Movies 1205696024
Explination of Script:
Variable for controlling search size. Can be KB, MB, GB
$size=1GB
Variable to set base path to search from
$path="C:\"
Variable to set list of excluded extensions
$omit="*.mdf,*.ldf"
Searches through all items from the $Path recursively and returns only files that are over the set size controlled by $size, and omits files listed in $omit.
Get-ChildItem -Path $path -Exclude $omit -Recurse -OutBuffer 1000|
where {($_.Length -gt $size)}|Select Name, Directory, Length
NOTE: The -OutBuffer parameter controls how many items are gathered before continuing. Managing this parameter correctly can greatly increase the speed with which a command completes. This is from a group of parameter called "CommonParameters". Knowing what these are, and how they work is invaluable.Microsoft Docs about_CommonParameters

Related

powershell ls errors certain folders / files

I'm looking at doing a recursive Get-ChildItem -r to get lastWriteTime, length and group for count by extension.
I get a bunch of errors, e.g., 'Get-ChildItem : Could not find item C:\Pics & Videos\Thumbs.db'.
I was thinking some folders or filenames had special characters in the name of the folder or file. I was able to encapsulate in quotes to correct some of the erroring files, but not all.
[System.IO.File]::Exists("C:\Pics & Videos\Thumbs.db") gave me a True, but
Get-ChildItem "C:\Pics & Videos\Thumbs.db" gave me the error.
I'm going to look at [System.IO.Fileinfo], but wonder if anyone can answer why I get these errors using Get-ChildItem aka ls?
Thanks
I may have found what I was looking for. With $Path a full path to the starting folder I want to recursively get file info from.
[IO.Directory]::EnumerateFileSystemEntries($path,"*.*","AllDirectories") |
ForEach { [System.IO.FileInfo]"$_" }
Other suggestions are welcome that might be faster. I'm looking at millions of files over 4500 folders. get-childitem only was able to get 60% of the files with 40% being errors without values. This is for one department and there are several.
Tested: get-childItem vs EnumerateFiles vs Explorer vs TreeSize
$path = "P:\Proxy Server Files\Dept1\sxs\"
First choice was slow. I get errors; so I added the error count as a guess.
$error.clear()
(get-ChildItem $path -r -ErrorAction SilentlyContinue).count
1333
$error.count
256
Second choice was much faster but gave less numbers.
$error.clear()
([IO.Directory]::EnumerateFileSystemEntries($path,"*.*","AllDirectories")).count
1229
$error.count
0
Trying to only look at files recursively again I get errors; so I added the error count as a guess.
$error.clear()
(get-ChildItem $path -r -file).count
558
$error.count
256
Looking at just files I get a much lower number that expected.
([IO.Directory]::EnumerateFileSystemEntries($path,"*.*","AllDirectories") | ForEach { [System.IO.FileInfo]"$_" }| Where Mode -NotMatch "d").count
108
Tried another method but same result.
([IO.Directory]::EnumerateFiles($path,"*.*","AllDirectories")| ForEach { [System.IO.FileInfo]"$_" }| Where Mode -NotMatch "d").count
108
From Windows Eplorer I see 37 files and 80 folders.
TreeSize.exe shows 1175 files on 775 folders.
I'm not sure what count to believe. Admin rights used to get all counts.
Any ideas why so many different results?
Thumbs.db is (typically) a hidden file. By default Get-ChildItem doesn't look for hidden files. Pass -Force (-Hidden shows only hidden items):
PS> get-childitem .\Thumbs.db
Get-ChildItem: Could not find item C:\[...]\Thumbs.db.
PS> get-childitem .\Thumbs.db -Force
Directory: C:\[...]
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a-h- 24/12/2016 11:17 13824 Thumbs.db
This was answered in question 9508375.
get-childitem -literalpath
This handles the problem with special characters in the name.

Comparing Desktop and documents with their backup and produce an output.txt file that highlights the different folders and files between them

can someone please help me. Still new to powershell, I am comparing my documents and desktop mirror to check my backup solutions using several different codes. The one below is meant to check both the documents/desktop with its mirror folders and tell me exactly what files are 'different' between source and destination and output it into the same output.txt file (not sure if it overwrites it). When I do this for my documents alone, it works when I want try the code for my desktop it doesn't output anything at all. Any advice?
function Get-Directories ($path)
{
$PathLength = $path.length
Get-ChildItem $path -exclude *.pst,*.ost,*.iso,*.lnk | % {
Add-Member -InputObject $_ -MemberType NoteProperty -Name RelativePath -Value $_.FullName.substring($PathLength+1)
$_
}
}
Compare-Object (Get-Directories $Folder3) (Get-Directories $Folder4) -Property RelativePath | Sort RelativePath, Name -desc | Out-File C:\Users\desktop\output.txt
Judging by earlier revisions of your question, your problem wasn't that your code didn't output anything at all, but that the output had empty Name property values.
Thus, the only thing missing from your code was Compare-Object's -PassThru switch:
Compare-Object -PassThru (Get-Directories $Folder3) (Get-Directories $Folder4) -Property RelativePath |
Sort RelativePath, Name -desc |
Out-File C:\Users\desktop\output.txt
Without -PassThru, Compare-Object outputs [pscustomobject] instances that have a .SideIndicator property (to indicate which side a difference object is exclusive to) and only the comparison properties passed to -Property.
That is, in your original attempt the Compare-Object output objects had only .SideIndicator and .RelativePath properties, and none of the other properties of the original [System.IO.FileInfo] instances originating from Get-ChildItem, such as .Name, .LastWriteTime, ...
With -PassThru, the original objects are passed through, decorated with an ETS (Extended Type System) .SideIndicator property (decorated in the same way you added the .RelativePath property), accessing the .Name property later works as intended.
Note:
Since Out-File then receives the full (and decorated) [System.IO.FileInfo] instances, you may want to limit what properties get written via a Select-Object call beforehand.
Additionally you may choose a structured output format, via Export-Csv for instance, given that the formatting that Out-File applies is meant only for the human observer, not for programmatic processing.
$_.FullName.substring($PathLength+1) in your Get-Directories should be $_.FullName.substring($PathLength), otherwise you'll cut off the 1st char.
since you're not using -Recurse when listing files, you could just use the file name instead of adding on a relative path property:
$folder1 = gci 'C:\Users\username\desktop' -exclude *.pst,*.ost,*.iso,*.lnk
$folder2 = gci 'D:\desktop' -exclude *.pst,*.ost,*.iso,*.lnk
Compare-Object $folder1 $folder2 -Property Name,Length
Name Length SideIndicator
---- ------ -------------
test.txt 174 =>
test.csv 174 <=
# Alternatively, use -PassThru to keep the whole object:
Compare-Object $folder1 $folder2 -Property Name,Length -PassThru | select SideIndicator,FullName,Length,LastWriteTime
SideIndicator FullName Length LastWriteTime
------------- -------- ------ -------------
=> D:\desktop 174 7/14/2021 2:47:09 PM
<= C:\Users\username\desktop 174 7/14/2021 2:47:09 PM
Use Out-File -Append to append output to a file.
As for troubleshooting your current script, try manually checking whether the RelativePath property looks like it's getting set correctly for you:
(Get-Directories $Folder3).RelativePath
(Get-Directories $Folder4).RelativePath
Finally, I recommend using robocopy over powershell for backup stuff like this, since it can use backup privileges (for locked files) and can copy multiple files at a time, but it's personal preference:
robocopy source destination /b /mir /mt /r:0 /w:0
/b - Runs robocopy in backup mode. Will copy everything as long as you are an Administrator
/mir - Mirrors everything from the source to the destination
/mt - Copies up to 8 files at a time
/r:0 - Sets it to not retry a file, default is like a million retries
/w:0 - Sets the time to 0 seconds between retries - default is like 30 seconds
source: https://community.spiceworks.com/topic/286190-transfer-user-profiles-using-robocopy

powershell and windows explorer (properties) have different count results

I'm recursively counting total number of objects (files, folders, etc) to check folders vs their Amazon S3 backups.
When I use windows explorer on a folder (right click --> properties), I get a smaller number of total objects than what the following powershell code generates. Why?
Amazon S3 matches the count from Windows Explorer 100% of the time. Why is powershell giving a higher total number, and what is the likely difference (system files, hidden files, etc)? The total number of objects in these folders is routinely 77,000+.
folder_name; Get-ChildItem -Recurse | Measure-Object | %{$_.count}
I was unable to replicate.
When in file explorer, right-click the folder in question -> properties
Under the General tab, there's a section called Contains.
This lists both the Files and Folders as separate numbers.
In my example I have 19,267 Files, 1,163 Folders which is a total of 20,430 objects
When I run
Get-ChildItem -Path C:\folder -Recurse | measure | % Count
it returns 20430
When I run
Get-ChildItem -Path C:\folder -Recurse | ?{$_.PSiscontainer -eq $false} | measure | % count
it returns 19267
When I run
Get-ChildItem -Path C:\folder -Recurse | ?{$_.PSiscontainer -eq $true} | measure | % count
it returns 1163
Are you certain that you're counting both the files and folders when manually viewing the properties?
The discrepancy comes from Windows Explorer counting files, and separately, folders. Powershell (version 2.0 Build 6.1) is counting everything together. It seems -File and -Directory don't work in PowerShell V2.0.
I really want to be able to get a list as a .cvs or .txt output of just the number of files (recursively) from a large number of folders. Going through windows explorer is one by one, and I don't get this as an output that I can copy/paste.
To count the number of files and folders in separate variables, you can do
# create two variables for the count
[int64]$totalFolders, [int64]$totalFiles = 0
# loop over the folders in the path
Get-ChildItem -Path 'ThePath' -Recurse -Force -ErrorAction SilentlyContinue | ForEach-Object {
if ($_.PSIsContainer) { $totalFolders++ } else { $totalFiles++ }
}
# output the results
"Folders: $totalFolders`r`nFiles: $totalFiles"
The -Force switch makes sure also hidden and system files are counted.
A probably faster alternative is to use robocopy:
$roboCount = robocopy 'ThePath' 'NoDestination' /L /E /BYTES
$totalFolders = #($roboCount -match 'New Dir').Count - 1 # the rootfolder is also counted
$totalFiles = #($roboCount -match 'New File').Count
# output the results
"Folders: $totalFolders`r`nFiles: $totalFiles"

Get-ChildItem -Exclude returning Fullname instead of short name

I'm having a strange issue when I run a Get-ChildItem -Exclude.
I've been using the following command at both source and destination locations to create an array of files after I've run a copy and then comparing the 2 directories. It has been working just fine. (I was going to post images, but don't have a high enough reputation)
$sourcefiles = #(Get-ChildItem -Recurse -path e:\Home\o365.test)
When I run this command, the array populates with the short name of the files.
I excluded PST files from the file copy and I needed to also exclude them in the compare, so I added an exclude to the GCI command.
$sourcefiles = #(Get-ChildItem -Recurse -path e:\Home\o365.test -Exclude *.pst)
This returns all of the correct files (*.pst is in fact excluded), however it is returning the fullname of the files, rather than the shortname that I always got returned before.
This is causing the compare-object to fail, since the destination GCI doesn't have the exclude (Not needed since *.pst was excluded from the copy). Besides, it is a UNC path and the fullname wouldn't match anyway.
I realize I could use split-path -leaf (which I've tried - and seems to work)
$sourcefiles = #(Get-ChildItem -Recurse -path e:\Home\o365.test -Exclude *.pst | Split-Path -Leaf)
But I still don't understand why adding the -exclude param to GCI causes it to change the format of what it returns.
Does anyone have any insight into why this would be happening?
Thanks in advance for any help!
Well, I can't replicate what you're seeing here. Placing an at sign before an operation in PowerShell is in fact the array operator, put simply putting it before a parenthesis with an operation doesn't return single properties like you're describing.
Now, you can make this happen if you place a property name on the back of the parenthesis to select just one property. For example:
$files = #(Get-ChildItem -path c:\temp).BaseName
>LZelda.csv
>msvcr100.dll
>names-0.txt
>names-1.txt
>NOBGW00017.log
>nslookup.exe
>PolicySpy.exe
Adding the -Exclude parenthesis doesn't change the output at all.
$files = #(Get-ChildItem -path c:\temp -exclude *.png)
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 3/18/2015 2:34 PM Android
da---- 9/10/2013 7:17 PM Bootfiles
d----- 12/13/2014 4:29 PM camera
d----- 1/6/2015 11:49 AM CFS
d----- 12/11/2014 1:08 PM ebay
d----- 2/2/2014 1:55 PM Finance
My reason for showing you this is that the things we get back are still FileSystem objects. The only way to reduce $files down to containing just one property, like the FullName, BaseName or whatever is in the following two methods:
$files = #(Get-ChildItem -path c:\temp -exclude *.png).BaseName
$files = #(Get-ChildItem -path c:\temp -exclude *.png | Select -Expand BaseName)
Both will give the same output.
The problem here is a deliberate "bug" (as I call it) introduced right from the start of Powershell (or at least from PS2.0). In order to ease the transition from cmd to Powershell, Get-ChildItem was given the alias dir and was then made to work like the cmd version of dir (as far as necessary to give a similar looking output). As with cmd,
PS > dir <directory path>
produces a listing of <directory path> (when simply displayed). In both cases (cmd and Powershell), an absent directory path uses . (the current directory). However, if the cmd version is invoked using a file name or wildcard pattern after the directory path (again, implicit . if no directory path) then the output is the matching file(s) (and directories for the wildcard case) in that directory (if any). Get-ChildItem does the same so that the alias, dir, gives similar output. It might have been expected that Get-ChildItem, when provided a wildcard path, would return a list of the children of any directories matching that wildcard path and ignore any matching files. If the argument were a plain file name, Get-ChildItem would complain that the argument provided cannot have children (as opposed to an empty directory which can have children but just doesn't). After all, if you want to match files or directories themselves (rather than their contents), you'd use Get-Item. Indeed, the results returned by Get-ChildItem in the case of file arguments and wildcards are identical to the equivalent Get-Item. However, the results returned by Get-ChildItem for actual children of a specified directory path differ slightly from those of Get-Item. Specifically, they have different ToString() methods (even though they both return FileInfo or DirectoryInfo objects). This means that converting the object to a string either by explicitly invoking ToString() or by using the returned object in an expression that requires a string, e.g.
PS > dir | foreach { "$_" }
gives different results.
Demonstration,
PS > gci C:\Windows\Web\Screen | foreach { "$_" }
img100.jpg
img101.png
img102.jpg
img103.png
img104.jpg
img105.jpg
PS > gci C:\Windows\web\screen\* | foreach { "$_" }
C:\Windows\web\screen\img100.jpg # note: uses argument path case.
C:\Windows\web\screen\img101.png # actual directory path case is
C:\Windows\web\screen\img102.jpg # C:\WINDOWS\Web\Screen
C:\Windows\web\screen\img103.png
C:\Windows\web\screen\img104.jpg
C:\Windows\web\screen\img105.jpg
PS > gi C:\Windows\Web\screen\* | foreach { "$_" }
C:\Windows\Web\screen\img100.jpg
C:\Windows\Web\screen\img101.png
C:\Windows\Web\screen\img102.jpg
C:\Windows\Web\screen\img103.png
C:\Windows\Web\screen\img104.jpg
C:\Windows\Web\screen\img105.jpg
PS > gci C:\Windows\Web\screen\img100.jpg | foreach { "$_" }
C:\Windows\Web\screen\img100.jpg
PS > gi C:\Windows\Web\screen\img100.jpg | foreach { "$_" }
C:\Windows\Web\screen\img100.jpg
Thus, I suspect that the OP is using implicit string conversion to get the "name" of the file and running foul of this "bug" which apparently also manifests when -Exclude is used (independent of -Recurse which has no effect). The solution is to not rely on string conversion and use the actual string properties, either name or fullname (whichever is needed).
PS > gci C:\Windows\Web\Screen | foreach { "$($_.fullname)" }
C:\Windows\Web\screen\img100.jpg
C:\Windows\Web\screen\img101.png
C:\Windows\Web\screen\img102.jpg
C:\Windows\Web\screen\img103.png
C:\Windows\Web\screen\img104.jpg
C:\Windows\Web\screen\img105.jpg
PS > gci C:\Windows\web\screen\* | foreach { "$($_.name)" }
img100.jpg
img101.png
img102.jpg
img103.png
img104.jpg
img105.jpg
An alternate (long term) solution could be to fix Get-ChildItem so that ToString() always returned the same result. This might be considered a breaking change but I suspect this "bug" has tripped up many users, even those who are already aware of it. I know I keep forgetting. Could even have Get-Item and Get-ChildItem use the same ToString(), whichever that is.
For a radical (and likely inferno inducing) solution, "fix" Get-ChildItem to only return children (or errors) and make dir a function that invoked Get-ChildItem or Get-Item as required. This would then restore the Powershell ethos of "cmdlets do what their name says they do" instead of having Get-ChildItem sometimes return children and sometimes items. (Way too much of a breaking change so not holding my breath for that one.)
Test performed using PS 5.1.18362.145.

How to limiting files searched by Get-ChildItem (or limiting depth of recursion)?

Background
There is a directory that is automatically populated with MSI files throughout the day. I plan on leveraging Task Scheduler to run the script shown below every 15 minutes. The script will search the directory and copy any new MSIs that have been created in the last 15 minutes to a network share.
Within this folder C:\ProgramData\flx\Output\<APP-NAME>\_<TIME_STAMP>\<APP-NAME>\ there are two other folders: Repackaged and MSI Package. The Repackaged folder does not need to be searched as it does not contain any MSIs. Also I have found that it needs to be excluded in some way to prevent this error:
Get-ChildItem : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
At line:14 char:32
+$listofFiles=(Get-ChildItem <<<< -Recurse -Path $outputPath -Include "*.msi" -Exclude "*.Context.msi" | where {$_.LastAccessTime -gt $time.AddMinutes($minutes)})
+ CategoryInfo : ReadError: C:\ProgramData\...xcellence\Leg 1:String) [Get-ChildItem], PathTooLongException
+ FullyQualifiedErrorId : DirIOError,Microsoft.PowerShell.Commands.GetChildItemCommand
Limitations
I am stuck using Powershell v1.0
I have no control over the directory structure of the source location
Updated:
I don't know the app name or the what the time stamp will be. That is something else that is out of my control.
Current plans
I have read about using -Filter and I am aware of filters that are similar to functions but I wasn't able to come up with any ideas of how to use them. My only thought at the moment would be to do something like:
$searchList=Get-ChildItem "all instances of the MSI Package folder"
foreach($folder in $searchList){
$listofFiles=Get-ChildItem "search for *.msi"
foreach($file in $listofFiles){"Logic to copy MSI from source to destination"}
}
However...I thought that there might be a more efficient way of doing this.
Questions
How can I limit depth that Get-ChildItem searches?
How can I limit the Get-ChildItem search to C:\ProgramData\flx\Output\<APP-NAME>_<TIME_STAMP>\<APP-NAME>\MSI Package
How can I only search folders that have been accessed in the last 15 minutes? I don't want to waste time drilling down into folders when I know MSI has already been copied.
Any additional advice on how to make this script more efficient overall would also be greatly appreciated.
Script
My current script can be found here. I kept getting: "Your post appears to contain code that is not properly formatted as code" and gave up after the fourth time trying to reformat it.
You can try this
dir C:\ProgramData\flx\Output\*\*\*\*\* -filter *.msi
this search all .msi files at this level
C:\ProgramData\flx\Output\<APP-NAME>\_<TIME_STAMP>\<APP-NAME>\Repackaged or 'MSI Package' or whatever else present folder
without recursion, this avoid too deep folder that give you error.
Pipe the result to:
Where {$_.LastAccessTime -gt (Get-Date).AddMinutes(-15)} #be sure no action on file is taken before the dir command
or
Where {$_.LastWriteTime -gt (Get-Date).AddMinutes(-15)} #some file can be re-copied maybe
With help from C.B. this is my new search which eliminates the issues I was having.
Changed -Path to C:\ProgramData\flx\Output\*\*\*\* to help limit the depth that was searched.
Used -Filter instead of -Include and put the -Exclude logic into the where clause.
Get-ChildItem -Path C:\ProgramData\flx\Output\*\*\*\* -Filter "*.msi" | where {$_.Name -notlike "*.Context.msi" -and $_.LastAccessTime -gt (Get-Date).AddMinutes(-15)}
You can't limit the recursion depth of Get-ChildItem except to not use -Recurse i.e. Get-ChildItem is either depth = 0 or N.
Set up variables for app name and timestamp e.g.:
$appName = "foo"
$timestamp = Get-date -Format HHmmss
Get-ChildItem "C:\ProgramData\flx\Output\${appName}_$timestamp\$appName\MSI Package" -force -r
You can filter the results like so:
Get-ChildItem <path> -R | Where {$_.LastWriteTime -gt (Get-Date).AddMinutes(-15)}