The command
( Get-ChildItem c:\MyFolder | Measure-Object ).Count
sometimes gives the error
CategoryInfo :ObjectNotFound: (:\MyFolder\myfile.xml:String) [Get-ChildItem], IOException
FullyQualifiedErrorId : ItemNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand
It is correct that the object that cannot be found has been moved to the folder by another process but how to make not checking more than once?
To be clear, the problem is not the missing object but that the count files if the content change between Time1 and Time2 - and I dont get what Time2 is.
EDIT
I have simplified it and added some switches - partly at random
This exactly what I do
$i = 0
ForEach ($file in $files) {
ProcessOneFile ($file)
start-sleep -Milliseconds 500
$i++
if ($i -eq 100) {
write-host '100'
while((Get-ChildItem -force -file -name $Myfolder'\*xml').Count -gt 200)
{
write-host 'too many, wait 30 sec'
start-sleep -Milliseconds 30000
}
$i = 0
}
}
It seems it works if -name is used. I suppose it disconnects the collection from the actual file system. Surprising to me but I lack the indepth knowledge (clearly)
So for count only
(Get-ChildItem -name -force -file $MyDir'\*xml').Count
Is the safer bet
Related
I found a PowerShell script on TechNet to help locate duplicate files in folders. However, when I run it, I am getting an error on what appears to be every folder\file. Not sure what switch is supposed to be used in this.
$Path = '\\servername\Share\Folders' #define path to folders to find duplicate files
$Files=gci -File -Recurse -path $Path | Select-Object -property FullName,Length
$Count=1
$TotalFiles=$Files.Count
$MatchedSourceFiles=#()
ForEach ($SourceFile in $Files)
{
Write-Progress -Activity "Processing Files" -status "Processing File $Count / $TotalFiles" -PercentComplete ($Count / $TotalFiles * 100)
$MatchingFiles=#()
$MatchingFiles=$Files |Where-Object {$_.Length -eq $SourceFile.Length}
Foreach ($TargetFile in $MatchingFiles)
{
if (($SourceFile.FullName -ne $TargetFile.FullName) -and !(($MatchedSourceFiles |
Select-Object -ExpandProperty File) -contains $TargetFile.FullName))
{
Write-Verbose "Matching $($SourceFile.FullName) and $($TargetFile.FullName)"
Write-Verbose "File sizes match."
if ((fc.exe /A $SourceFile.FullName $TargetFile.FullName) -contains "FC: no differences encountered")
{
Write-Verbose "Match found."
$MatchingFiles+=$TargetFile.FullName
}
}
}
if ($MatchingFiles.Count -gt 0)
{
$NewObject=[pscustomobject][ordered]#{
File=$SourceFile.FullName
MatchingFiles=$MatchingFiles
}
$MatchedSourceFiles+=$NewObject
}
$Count+=1
}
$MatchedSourceFiles
Errors
FC: Insufficient number of file specifications
fc.exe : FC: Invalid Switch
At line:18 char:12
gci : Could not find a part of the path
At line:2 char:8
fc.exe : FC: Invalid Switch
At line:18 char:12
To fix your fc.exe error and optimize tour script, I also recommend #rich-moss 's solution.
But if you only want to find duplicates, you can easily accomplish so by checking their hashes.
Example:
$Duplicates = Get-ChildItem -File -Recurse | Get-FileHash | Group-Object -Property Hash | Where-Object Count -gt 1
If ($duplicates.count -lt 1) {
$null # 'No duplicates found. Do stuff ...'
} else {
$result = foreach ($d in $duplicates) {
$d.Group | Select-Object -Property Path, Hash
}
The script you provided is very inefficient and provides false positives in my tests. It's inefficient because it compares every file twice (Source->Target and Target->Source) and because it iterates through all files regardless of size. Here's a quicker version that gathers the files into groups of similarly sized files and only executes FC.EXE once per pair of files:
$Path = 'C:\Temp'
$SameSizeFiles = gci -Path $Path -File -Recurse | Select FullName, Length | Group-Object Length | ? {$_.Count -gt 1} #the list of files with same size
$MatchingFiles=#()
$GroupNdx=1
Foreach($SizeGroup in ($SameSizeFiles | Select Group)){
For($FromNdx = 0; $FromNdx -lt $SizeGroup.Group.Count - 1; $FromNdx++){
For($ToNdx = $FromNdx + 1; $ToNdx -lt $SizeGroup.Group.Count; $ToNdx++){
If( (fc.exe /A $SizeGroup.Group[$FromNdx].FullName $SizeGroup.Group[$ToNdx].FullName) -contains "FC: no differences encountered"){
$MatchingFiles += [pscustomobject]#{File=$SizeGroup.Group[$FromNdx].FullName; Match = $SizeGroup.Group[$ToNdx].FullName }
}
}
}
Write-Progress -Activity "Finding Duplicates" -status "Processing group $GroupNdx of $($SameSizeFiles.Count)" -PercentComplete ($GroupNdx / $SameSizeFiles.Count * 100)
$GroupNdx += 1
}
$MatchingFiles
Efficiency will be even more important if you're running it over the network. You may find it quicker to execute the script on the server itself, rather than from a share. There is some discussion here about the fastest way to compare files in .Net.
I have a script that is supposed to look at a logfile that starts filling with several hundred lines. While the log is filling, I need to look for two different conditions at the same time. If I find "Completed Successfully" then it will break the while loop and continue with the rest of the script. If it finds "Error" in the script, it will restart the server. Unfortunately this isn't working for me. It just sits until the timeout of the stopwatch. I'm not quite sure what is wrong here. Is there a better way to do this?
$Comp="ServerA.domain"
$Logfile="\\$Comp\Logs\logfile1.txt"
$pattern1 = "Completed Successfully"
$Pattern2 = "Error"
$timeout = new-timespan -Minutes 5
$stopwatch = [diagnostics.stopwatch]::StartNew()
while ($stopwatch.elapsed -lt $timeout){
try{
$logContent2 = Get-Content -Path $Logfile -Tail 1000 | Select-String -Pattern $Pattern2 | Measure-Object | Select-Object -ExpandProperty Count
$logContent1 = Get-Content -Path $Logfile -Tail 1000 | select-string -pattern $pattern1 | Measure-Object | Select-Object -ExpandProperty Count
If ($logContent1 -gt 0) {
Write-Host -Object "Compiled Successfully on $Comp"
#break;
}
ElseIf ($logContent2 -gt 0) {
Write-Host -Object "Error Found on $Comp! Restart required. Rebooting..."
Restart-Computer -ComputerName $Comp -Wait -For PowerShell -Force
}
}
Catch{
$Error[0] > \\$Comp\Logs\ErrorLog.txt
}
}
If ($stopwatch.elapsed -ge $timeout){
Write-Error -Message "$pattern1 did not appear" -ErrorAction Stop
exit;
}
Yes! I accidentally placed a break in the wrong place. This is resolved. The above code works but you need to break the loop (I had a break in my code in the wrong place) and I don't include the break in the code in this example.
There is COTS app we have that creates reports and never deletes it. So we need to start cleaning it up. I started doing a foreach and would run out of memory on the server (36GB) when it got up to 50ish million files. After searching it seemed you could change it like so
Get-ChildItem -path $Path -recurse | foreach {
and it won't go through memory but process each item at a time. I can get to 140 million files before I run out of memory.
Clear-Host
#Set Age to look for
$TimeLimit = (Get-Date).AddMonths(-4)
$Path = "D:\CC\LocalStorage"
$TotalFileCount = 0
$TotalDeletedCount = 0
Get-ChildItem -Path $Path -Recurse | foreach {
if ($_.LastWriteTime -le $TimeLimit) {
$TotalDeletedCount += 1
$_.Delete()
}
$TotalFileCount += 1
$FileDiv = $TotalFileCount % 10000
if ($FileDiv -eq 0 -and $TotalFileCount -ne 0) {
$TF = [string]::Format('{0:N0}', $TotalFileCount)
$TD = [string]::Format('{0:N0}', $TotalDeletedCount)
Write-Host "Files Scanned : " -ForegroundColor Green -NoNewline
Write-Host "$TF" -ForegroundColor Yellow -NoNewline
Write-Host " Deleted: " -ForegroundColor Green -NoNewline
Write-Host "$TD" -ForegroundColor Yellow
}
Is there a better way to do this? My only next thought was not to use the -Recurse command but make my own function that calls itself for each directory.
EDIT:
I used the code provided in the first answer and it does not solve the issue. Memory is still growing.
$limit = (Get-Date).Date.AddMonths(-3)
$totalcount = 0
$deletecount = 0
$Path = "D:\CC\"
Get-ChildItem -Path $Path -Recurse -File | Where-Object { $_.LastWriteTime -lt $limit } | Remove-Item -Force
Using the ForEach-Object and the pipeline should actually prevent the code from running out of memory. If you're still getting OOM exceptions I suspect that you're doing something in your code that counters this effect, which you didn't tell us about.
With that said, you should be able to clean up your data directory with something like this:
$limit = (Get-Date).Date.AddMonths(-4)
Get-ChildItem -Path $Path -Recurse -File |
Where-Object { $_.LastWriteTime -lt $limit } |
Remove-Item -Force -WhatIf
Remove the -WhatIf switch after you verified that everything is working.
If you need the total file count and the number of deleted files, add counters like this:
$totalcount = 0
$deletecount = 0
Get-ChildItem -Path $Path -Recurse -File |
ForEach-Object { $totalcount++; $_ } |
Where-Object { $_.LastWriteTime -lt $limit } |
ForEach-Object { $deletecount++; $_ } |
Remove-Item -Force -WhatIf
I don't recommend printing status information to the console when you're bulk-processing large numbers of files. The output could significantly slow down the processing. If you must have that information, write it to a log file and tail that file separately.
I have a workflow that I created that recursively scans hundreds of file shares for specific file names/extensions. Everything works great, but I'm wondering if there is a way to improve it's speed.
In the foreach -parallel, I'm getting child items and ACL recursively from a list of shares, 20 at a time. What I'm wondering is if there is a way to also make it process in parallel within the GCI.
In other words, is foreach -parallel nesting possible/doable, and if so can you offer suggestions such as syntax, the dangers of doing it, etc. I've found very little documentation on it and am hoping for some expert advice
If that doesn't make sense, here's a different way to explain it. I have 5 shares that I'm searching recursively, all with different sizes and a different number of files:
\share1\folder1 Size: 5GB
\share2\folder1 Size: 79GB
\share3\folder1 Size: 2GB
\share4\folder1 Size: 8GB
\share5\folder1 Size: 103GB
The GCI will process all 5 at the same time, however share2 and share5 are going to take a LOT more time than the others due to their size. Is there a way to to make a script process all 5 shares in parallel, and also process multiple files at once within the GCI? So it would start searching all 20 shares at once, and get information on say 5 files within the share at a time?
Here's a piece of what I'm doing, shortened down a bit for length reasons with variable names/input changed. I'm sure there are things that I can improve upon in here as well, but overall it works.
workflow Scan-Shares
{
[cmdletbinding()]
param(
[int]$ThrottleLimit = 20
)
$File1 = Import-csv C:\Drives.csv
$ExtList = #((Invoke-WebRequest -Uri "www.websitewithfilenames.com").content | convertfrom-json | % {$_.filters})
foreach -parallel -throttlelimit $throttlelimit ($Line in $File1)
{
inlinescript
{
$Line=$using:Line
$Extlist=$using:Extlist
$Path = $Line.Path + '\*'
$Directory = Get-ChildItem -Path $path -Recurse -Force -ErrorAction SilentlyContinue -WarningAction SilentlyContinue | select-object name, Extension, FullName, LastWriteTime, Directory -ExpandProperty Name
$singleRegex = ($extlist | %{ '^' + $_ + '$' }) -join '|'
[array]$Files = $Directory -match $singleRegex
if($files.count -gt 0 )
{
foreach($SingleFile in $Files)
{
$Owner = (Get-Acl $FullName -Erroraction SilentlyContinue -WarningAction SilentlyContinue).Owner
New-Object -TypeName PSCustomObject -Property #{
Path = $File.Fullname
Filename = $File.Name
LastWriteTime = $File.LastWriteTime
Owner = $Owner
Directory = $File.Directory } |
Export-Csv -Force C:\Share-Results.CSV -Append
}
}
}
}
}
I wanted to make a PowerShell script that deleted any type of file in a certain folder when they were 7 days old. The problem I am having is creating a log file with the date, time, and names of all the files that were deleted or were previously deleted when the script ran.
I am wondering if there is a way of modifying the answer found on this page: https://stackoverflow.com/questions/12326171/powershell-script-to-delete-sub-folders-and-files-if-creation-date-is-7-days-bu?lq=1
I would like it so that instead of sending an email with the report it will create one log file of all the files deleted with the names, time, and dates in the folder that I am storing the script (Please note I'd like it to append the log file each time not overwrite it). I am sorry because I do not know how to code so I have been stuck trying to do this myself for a long time now and I have been asking questions and such but still can't seem to get it to work. So yea if anyone can modify that script it would be very appreciated! Thank you!
Here is what I did with the script (if it helps) , but it doesn't work (Again I do not know how to code):
$report_items = #()
# set folder path
$dump_path = "C:FileDeleter\AutoDeleteFilesInThisFolder"
# set min age of files
$max_days = "-7"
# get the current date
$curr_date = Get-Date
# determine how far back we go based on current date
$del_date = $curr_date.AddDays($max_days)
# get the sub directories
$sub_dirs = Get-ChildItem $dump_path | Where-Object {!$_.PSIsContainer }
$sub_dirs | % {
if (Get-ChildItem $_.FullName -Recurse | Where-Object { $_.LastWriteTime -gt $del_date } ) {
$report_items += New-Object -Type PSObject -Property #{
Time = Get-Date -f "hh:mm:ss"
Message = "Skipping " + $_.FullName + " because it contains items newer than " + $del_date
}
}
else {
Remove-Item $_.FullName -Recurse
$report_items += New-Object -Type PSObject -Property #{
Time = Get-Date -f "hh:mm:ss"
Message = "Deleting " + $_.FullName + " because it contains no items newer than " + $del_date
}
}
}
$report_items | out-file "C:\FileDeleter\PruningReport.txt"
This should do it:
Get-ChildItem -Path "C:\FileDeleter\AutoDeleteFilesInThisFolder" -Filter *.torrent -Recurse | Where { $_.CreationTime -lt (Get-Date).AddDays(-7) } | %{Remove-Item -Force; if ($?) {"$(Get-Date -Format 'MM-dd-yy HH:mm:ss.ff') $_.Name" | Add-Content 'C:\FileDeleter\PruningReport.txt'}}
(The reason for the if ($?) block is to only write a log entry if the file deletion was successful - you don't want to assume that it was.)
Not as succinct, but I'm a fan of not over-pipelining in PowerShell.
[String] $strBaseDir = "c:\temp";
[String] $strFileFilter = "*.txt";
[Int] $intDayThreshold = 7;
[System.IO.FileSystemInfo] $objFile = $null;
[Int] $intLastWriteDays = 0;
[String] $strLogFileName = "d:\data\yourlogfile.log";
Get-ChildItem -LiteralPath $strBaseDir -Filter $strFileFilter -Recurse | Where-Object{ ! $_.PSIsContainer } |
Foreach-Object {
$objFile = $_;
$intLastWriteDays = [Math]::Round((New-TimeSpan -Start $objFile.LastWriteTime -End (Get-Date)).TotalDays);
if ( $intLastWriteDays -ge $intDayThreshold ) {
Write-Output -InputObject ( "{0:G} : Deleting file [{1}] with last write time of [{2:G}], which is [{3}] day(s) ago." -f (Get-Date), $objFile.FullName, $objFile.LastWriteTime, $intLastWriteDays ) | Add-Content -Path $strLogFileName -PassThru;
$objFile | Remove-Item -Force -ErrorAction silentlyContinue;
if ( ! $? ) {
Write-Output -InputObject "ERROR : Failed to remove file." | Add-Content -Path $strLogFileName -PassThru;
} #if
} else {
Write-Output -InputObject ( "{0:G} : Skipping file [{1}] with last write time of [{2:G}], which is [{3}] day(s) ago." -f (Get-Date), $objFile.FullName, $objFile.LastWriteTime, $intLastWriteDays ) | Add-Content -Path $strLogFileName -PassThru;
} #else-if
} #Foreach-Object