Powershell copy and log items based on folder and .txt content comparison - powershell

Current situation
I've been working on a script that should copy .txt files containing specific words and with an age of 7 days maximum into a folder, once. So far I've only been able to get code to copy files to the destination folder if the file doesn't exist already.
Code
$path = "c:\PS1\*.txt"
$Destination = "c:\PS2\"
$filter = "thisisatest"
$logfile = "C:\PS2\testlog_$(get-date -format `"MMyyyy`").txt"
#Picking files with certain words and modified within the last 7 days
Foreach($file in (Get-ChildItem $path | Where-Object {$_.name -Match $filter}))
{
If($file.LastWriteTime -gt (Get-Date).adddays(-7).date)
#Logging
{
function log($string, $color)
{
if ($Color -eq $null) {$color = "white"}
write-host $string -foregroundcolor $color
$string | out-file -Filepath $logfile -append
}
#Comparing folder content and copying those not present in destination folder
Compare-Object $path $Destination -Property Name, Length | Where-Object {$_.SideIndicator -eq "=>"} | ForEach-Object {Copy-Item -Path $file.fullname -Destination $Destination -Force}
}
log $file.fullname
}
In conclusion
I have tried finding code which would make it possible to do the following:
Compare path folder content to .txt log for reoccurring names and only copy those not present in list, if a file name is present in the list, move on to next file to copy.
Log only files that have just been copied in a .txt file, if log doesn't exist, create.
Delete log content older than 30 days
Some of my code is probably obsolete or lacking parts, it is made up from bits and pieces I have found while looking for examples.
I know most of it is probably doable with Robocopy, but I hope it can be done in powershell,
Hope you can help me

Ok, moved parameters inside function, made the $string parameter mandatory, gave the option of specifying a logfile (it will default to the global variable $logfile, and gave the color validation so people can IntelliSense it or tab complete it. I also moved it to the beginning of the script, since that's where you usually find functions and it just made more sense to me. Also made sure that the log file isn't a folder, and that it has an extension (it adds .log if it doesn't and isn't an existing file that you're just adding to). I think I may have overdone it on the function. It's very functional, and versatile in case it's needed for another script, but its kind of overkill.
Then I looked at how files were being selected, and then filtered, and revamped things a bit. Now the Get-ChildItem (alias GCI) filters the file name against $Filter, the last write time against Get-Date -7 days, it makes sure the FullName is not in the log file, and it makes sure that it's a file, not a folder.
I also added a line to clean up old logs.
function log{
Param(
[string]$string=$(Throw 'You must provide a string to enter into the log'),
$LogFile = $Global:LogFile,
[ConsoleColor]
$color = "white")
If((gci $logfile -ea SilentlyContinue).psiscontainer){"Specified log file is a folder, please specify a file name.";break}
If($logfile -notmatch "\.[^\\]*$" -and !(test-path $LogFile)){$LogFile = $LogFile + ".log"}
write-host $string -foregroundcolor $color
$string | out-file -Filepath $logfile -append
}
$path = "c:\PS1\*.txt"
$Destination = "c:\PS2\"
$filter = "thisisatest"
$logfile = $Destination+"testlog_$(get-date -format `"MMyyyy`").txt"
gci $Destination -filter "testlog_*.txt" |?{$_.LastWriteTime -le (get-date).AddDays(-42)}| Remove-Item -Force
$AllLogs = gc "$($Destination)testlog_*.txt"
#Picking files with certain words, was modified within the last 7 days, and not already in a logfile
Foreach($file in (Get-ChildItem $path | Where-Object {!$_.PSIsContainer -and $_.LastWriteTime -gt (Get-Date).adddays(-7) -and $AllLogs -notcontains $_.FullName -and $_.Name -inotcontains $filter}))
{
$File|?{!(Test-Path $Destination+$File.Name)}|Copy-Item -Destination $Destination
log $file.fullname
}

Related

PowerShell script speed standalone vs within Visual Code

I have a simple PowerShell script that writes to a file a list of recently added files.
$fpath = "\\DS1821\video\movies"
$file1 = "C:\Users\brian.w.williams\Desktop\RecentMovie.txt"
if (Test-Path $file1) {Remove-Item -Path $file1}
Get-ChildItem -Path "$fpath" -File -Recurse -Include "*.eng.srt" |
ForEach-Object {
$movie = $_.BaseName -replace ".eng",""
if ( ($_.LastWriteTime.Month -ge 7) -and ($_.LastWriteTime.Year -ge 2021) ) {
Write-Host $movie " = " $_.LastWriteTime
Write-Output $movie | Out-file $file1 -append;
}
}
The script works fine. But I noticed that the script runs much faster (a couple of minutes) when run within Visual Code (i.e. "Run without debugging"). When I run the script standalone (i.e. "Run with PowerShell") the script can take hours to complete. Why the difference? Is there anything I can do to speed it up? I have tried mapping the network folder, but that made no difference.
There is improvement to be made to help the code speed up.
First of all, using Write-Host and Write-Output to append to a file inside the ForEach loop is very time consuming.
Then you're using parameter -Include on Get-ChildItem, where you really want to use -Filter. A Filter is MUCH faster than Include, because the latter will only filter the filenames afterwards. -Filter however can only work on one single filename pattern, but this is exactly what you are doing here.
The if(..) can also be improved to have it compare only one value (a datetime) instead of two separate properties from LastWriteTime.
Try:
$sourcePath = '\\DS1821\video\movies'
# set a variable to an output file on your desktop
$outputFile = Join-Path -Path ([Environment]::GetFolderPath("Desktop")) -ChildPath 'RecentMovies.txt'
# set the reference date in a variable for faster comparison later
$refdate = (Get-Date -Year 2021 -Month 7 -Day 1).Date
# get the files, filter on the name ending in '*.eng.srt'
# filter more with Where-Object so you'll get only files newer than or equal to the reference date
# output objects with one calculated property ('MovieName') and the LastWriteTime
$movies = Get-ChildItem -Path $sourcePath -File -Recurse -Filter '*.eng.srt' |
Where-Object { $_.LastWriteTime -ge $refdate } |
Select-Object #{Name = 'MovieName'; Expression = {$_.BaseName -replace '\.eng\.srt$', '.srt'}},
LastWriteTime
# now output what you have collected to file.
# Set-Content creates a new file or overwrites an existing file
$movies.MovieName | Set-Content -Path $outputFile
# and to screen
$movies | ForEach-Object { Write-Host ('{0} = {1}' -f $_.MovieName, $_.LastWriteTime.ToString()) }
# or to Out-GridView if you prefer
$movies | Out-GridView -Title 'My Movies'
1. I've changed some variable names to make the code better readable
2. Since -replace uses regex, you have to escape the dots with a backslash. The regex also anchors the string to the end of the BaseName with the dollar sign ($) at the end

Filter and delete files and folders(and files inside of folders) older than x days in powershell

this is my first post on this forum. Im a beginner in coding and I need help with one of my very first self coded tools.
I made a small script, which deletes files based on if they are older than date x (lastwritetime). Now to my problem: I want the script also to check for files inside of folders inside of a directory and only delete a folder afterwards if it is truly empty. I cant figure out how to solve the recursion in this problem, seems like the script deletes just the entire folder in relation to the date x. Could anyone tell me please what I missed in this code and help me to create a own recursion to solve the problem or fix the code? Thanks to you all, guys! Here is my code:
I would be glad if someone knows how to make the code work by using a function
$path = Read-Host "please enter your path"
"
"
$timedel = Read-Host "Enter days in the past (e.g -12)"
$dateedit = (Get-Date).AddDays($timedel)
"
"
Get-ChildItem $path -File -Recurse | foreach{ if ($_.LastWriteTime -and !$_.LastAccessTimeUtc -le $dateedit) {
Write-Output "older as $timedel days: ($_)" } }
"
"
pause
Get-ChildItem -Path $path -Force -Recurse | Where-Object { $_.PsisContainer -and $_.LastWriteTime -le $dateedit } | Remove-Item -Force -Recurse
""
Write-Output "Files deleted"
param(
[IO.DirectoryInfo]$targetTolder = "d:\tmp",
[DateTime]$dateTimeX = "2020-11-15 00:00:00"
)
Get-ChildItem $targetTolder -Directory -Recurse | Sort-Object {$_.FullName} -Descending | ForEach-Object {
Get-ChildItem $_ -File | Where-Object {$_.LastWriteTime -lt $dateTimeX} | Remove-Item -Force
if ((Get-ChildItem $_).Count -eq 0){Remove-Item $_ -Force}
}
remove -WhatIf after test
To also remove folders that are older than the set days in the past if they are empty leaves you with the problem that as soon as a file is removed from such a folder, the LastWriteTime of the folder is set to that moment in time.
This means you should get a list of older folders first, before you start deleting older files and use that list afterwards to also remove these folders if they are empty.
Also, a minimal check on user input from Read-Host should be done. (i.e. the path must exist and the number of days must be convertable to an integer number. For the latter I chose to simply cast it to [int] because if that fails, the code would generate an execption anyway.
Try something like
$path = Read-Host "please enter your path"
# test the user input
if (-not (Test-Path -Path $path -PathType Container)) {
Write-Error "The path $path does not exist!"
}
else {
$timedel = Read-Host "Enter days in the past (e.g -12)"
# convert to int and make sure it is a negative value
$timedel = -[Math]::Abs([int]$timedel)
$dateedit = (Get-Date).AddDays($timedel).Date # .Date sets this date to midnight (00:00:00)
# get a list of all folders (FullNames only)that have a LastWriteTime older than the set date.
# we check this list later to see if any of the folders are empty and if so, delete them.
$folders = (Get-ChildItem -Path $path -Directory -Recurse | Where-Object { $_.LastWriteTime -le $dateedit }).FullName
# get a list of files to remove
Get-ChildItem -Path $path -File -Recurse | Where-Object { $_.LastWriteTime -le $dateedit} | ForEach-Object {
Write-Host "older as $timedel days: $($_.FullName)"
$_ | Remove-Item -Force -WhatIf # see below about the -WhatIf safety switch
}
# now that old files are gone, test the folder list we got earlier and remove any if empty
$folders | ForEach-Object {
if ((Get-ChildItem -Path $_ -Force).Count -eq 0) {
Write-Host "Deleting empty folder: $_"
$_ | Remove-Item -Force -WhatIf # see below about the -WhatIf safety switch
}
}
Write-Host "All Done!" -ForegroundColor Green
}
The -WhatIf switch used on Remove-Item is there for your own safety. With that, no file or folder is actually deleted, instead in the console it is written what would be deleted. If you are satisfied that this is all good, remove the -WhatIf and run the code again to really delete the files and folders
try something like this:
$timedel=-12
#remove old files
Get-ChildItem "C:\temp" -Recurse -File | Where LastWriteTime -lt (Get-Date).AddDays($timedel) | Remove-Item -Force
#remove directory without file
Get-ChildItem "C:\temp\" -Recurse -Directory | where {(Get-ChildItem $_.FullName -Recurse -File).count -eq 0} | Remove-Item -Force -recurse

Trying to iterate through folders and take recursive actions on each file

I'm trying to build a script that I can use to delete old files based on Last Accessed date. As part of the script I want to interrogate each sub folder, find files not accessed in the last X days, create a log in the same folder of the files found and record file details in the log then delete the files.
What I think I need is a nested loop, loop 1 will get each subfolder (Get-ChildItem -Directory -Recurse) then for each folder found a second loop checks all files for last accessed date and if outside the limit will append the file details to a logfile in the folder (for user reference) and also to a master logfile (for IT Admin)
loop 1 is working as expected and getting the subfolders, but I cannot get the inner loop to recurse through the objects in the folder, I'm trying to use Get-ChildItem inside the first loop, is this the correct approach?
Code sample below, I have added pseudo to demo the logic, its really the loops I need help with:
# Set variables
$FolderPath = "E:TEST_G"
$ArchiveLimit = 7
$ArchiveDate = (Get-Date).AddDays(-$ArchiveLimit)
$MasterLogFile = "C:\Temp\ArchiveLog $(Get-Date -f yyyy-MM-dd).csv"
# Loop 1 - Iterate through each subfolder of $FolderPath
Get-ChildItem -Path $FolderPath -Directory -Recurse | ForEach-Object {
# Loop 2 - Check each file in the Subfolder and if Last Access is past
# $ArchiveDate take Action
Get-ChildItem -Path $_.DirectoryName | where {
$_.LastAccessTime -le $ArchiveDate
} | ForEach-Object {
# Check if FolderLogFile Exists, if not create it
# Append file details to folder Log
# Append File & Folder Details to Master Log
}
}
I think you're overcomplicating a bit:
#Set Variables
$FolderPath = "E:\TEST_G"
$ArchiveLimit = 7
$ArchiveDate = (Get-Date).AddDays(-$ArchiveLimit)
$MasterLogFile = "C:\Temp\ArchiveLog $(get-date -f yyyy-MM-dd).csv"
If (!(Test-Path $MasterLogFile)) {New-Item $MasterLogFile -Force}
Get-ChildItem -Path $FolderPath -File -Recurse |
Where-Object { $_.LastAccessTime -lt $ArchiveDate -and
$_.Extension -ne '.log' } |
ForEach-Object {
$FolderLogFile = Join-Path $_.DirectoryName 'name.log'
Add-Content -Value "details" -Path $FolderLogFile,$MasterLogFile
Try {
Remove-Item $_ -Force -EA Stop
} Catch {
Add-Content -Value "Unable to delete item! [$($_.Exception.GetType().FullName)] $($_.Exception.Message)"`
-Path $FolderLogFile,$MasterLogFile
}
}
Edit:
Multiple recursive loops are unnecessary since you're already taking a recursive action in the pipeline. It's powerful enough to do the processing without having to take extra action. Add-Content from the other answer is an excellent solution over Out-File as well, so I replaced mine.
One note, though, Add-Content's -Force flag does not create the folder structure like New-Item's will. That is the reason for the line under the $MasterLogFile declaration.
Your nested loop doesn't need recursion (the outer loop already takes care of that). Just process the files in each folder (make sure you exclude the folder log):
Get-ChildItem -Path $FolderPath -Directory -Recurse | ForEach-Object {
$FolderLogFile = Join-Path $_.DirectoryName 'FolderLog.log'
Get-ChildItem -Path $_.DirectoryName -File | Where-Object {
$_.LastAccessTime -le $ArchiveDate -and
$_.FullName -ne $FolderLogFile
} | ForEach-Object {
'file details' | Add-Content $FolderLogFile
'file and folder details' | Add-Content $MasterLogFile
Remove-Item $_.FullName -Force
}
}
You don't need to test for the existence of the folder log file, because Add-Content will automatically create it if it's missing.

Powershell copy file after a date has passed with file structure

I am trying to copy a file off a server and onto another, I want to keep the structure of the file like so C:\folder\folder\file! If the folder is there copy the file into it, if it is not then create the folders and then copy into it!
I would like it also to filter out the files that are still needed so I want to keep files for 30 days and then move them!
Blockquote
`[int]$Count = 0
$filter = (Get-Date).AddDays(-15).ToString("MM/dd/yyyy")
Get-WMIObject Win32_LogicalDisk | ForEach-Object{
$SearchFolders = Get-Childitem ($_.DeviceID + "\crams") -recurse
$FileList = $SearchFolders |
Where-Object {$_.name -like "Stdout_*" -and $_.lastwritetime -le $filter}
[int]$Totalfiles = ($FileList | Measure-object).count
write-host "There are a total of $Totalfiles found."
echo $FileList
start-sleep 30
[int]
ForEach ($Item in $FileList)
{$Count++
$File = $Item
Write-Host "Now Moving $File"
$destination ="C:\StdLogFiles\"
$path = test-Path (get-childitem $destination -Exclude "Stdout_*")
if ($path -eq $true) {
write-Host "Directory Already exists"
copy-item $File -destination $destination
}
elseif ($path -eq $false) {
cd $destination
mkdir $File
copy-Item $File -destination $destination
}
}
}`
Is what I have so far it has changed a lot due to trying to get it to work but the search works and so does the date part I can not get it to keep the structure of the file!
Okay I took out the bottom part and put in
ForEach ($Item in Get-ChildItem $FileList)
also tried get-content but path is null
{$Count++
$destination = "C:\StdLogFiles"
$File = $Item
Write-Host "Now Moving $File to $destination"
Copy-Item -Path $file.fullname -Destination $destination -force}}
it is copying everything that is in c into that folder but not the files I do not understand what it is doing now! I had it copying the files even wen back to an older version and can't get it to work again! I am going to leave it before I break it more!
Any help or thoughts would be appreciated
I think RoboCopy is probably a simpler solution for you to be honest. But, if you insist on using PowerShell you are going to need to setup your destination better if you want to keep your file structure. You also want to leave your filter date as a [DateTime] object instead of converting it to a string since what you are comparing it to (lastwritetime) is a [DateTime] object. You'll need to do something like:
$filter = (Get-Date).AddDays(-15)
$FileList = Get-WMIObject Win32_LogicalDisk | ForEach-Object{
Get-Childitem ($_.DeviceID + "\crams") -recurse | Where-Object {$_.name -like "Stdout_*" -and $_.lastwritetime -le $filter}
}
$Totalfiles = $FileList.count
For($i = 1;$i -le $TotalFiles; $i++)
{
$File = $FileList[($i-1)]
Write-Progress -Activity "Backing up old files" -CurrentOperation ("Copying file: " + $file.Name) -Status "$i of $Totalfiles files" -PercentComplete ($i*100/$Totalfiles)
$Destination = (Split-Path $file.fullname) -replace "^.*?\\crams", "C:\StdLogFiles"
If(!(Test-Path $Destination)){
New-Item -Path $Destination -ItemType Directory | Out-Null
}
Copy-Item $File -Destination $Destination
}
Write-Progress -Completed
That gathers all the files you need to move from all disks. Takes a count of them, and then enters a loop that will cycle as many times as you have files. In the loop is assigns the current item to a variable, then updates a progress bar based on progress. It then parses the destination by replacing the beginning of the file's full path (minus file name) with your target destination of 'C:\StdLogFiles'. So D:\Crams\HolyPregnantNunsBatman\Stdout04122015.log becomes C:\StdLogFiles\HolyPregnantNunsBatman. Then it tests the path, and if it's not valid it creates it (piped to out-null to avoid spam). Then we copy the file to the destination and move on to the next item. After the files are done we close out the progress bar.

Trying to remove-item but it's not getting removed

I'm thinning out my backup files with a powershell script, and I know I have the correct filenames, but for some reason when I use remove-item, the item doesn't get removed and no exception is thrown. This is what it looks like:
try{
$Drive = "E:\temp\"
$deleteTime = -42
$limit = (Get-Date).AddDays($deleteTime) #files older than 6 weeks
#get files in folder older than deleteTime and with signature of *junk.vhd* (to be changed later)
$temp1 = Get-ChildItem -Path $Drive -filter "*junk.vhd*" | Where-Object {$_.LastWriteTime -lt $limit} | Select -Expand Name #this has 5 files in list
#will delete every other file
for($i=$temp1.GetLowerBound(0);$i -le $temp1.GetUpperBound(0);$i+=2) {
$name = $temp1[$i]
Write-Host "removing $name" #prints correct file names to screen
Get-ChildItem -Path $Drive -include $name | Remove-Item -recurse -force #this is handling correct files but they aren't deleted for some reason
}
}#try
Catch [Exception] {
#nothing is caught
Write-Host "here"
}
Does anyone have any ideas why it's finding and Write-Host the correct filenames to remove, but the Remove-Item isn't removing them?
I was looking at removal a little different example, but everything looks good.
try to replace this line:
Get-ChildItem -Path $Drive -include $name | Remove-Item -recurse -force #this is handling correct files but they aren't deleted for some reason
with this:
Remove-Item $temp1[i].FullName -force -recurse
Since you already got the full patrh to the file, I don't feel it's necessary to call Get-ChilItem again and pass it through the pipeline, instead you just feed Remove-Item with the FUllName property which is the full path to thew file.