Copy-Item by header & timestamp - powershell

I am using the code below to filter out files depending on the headers in the file.
It works like a charm, but I have a problem with that it takes all the files in the $InputDirectory.
I would like to limit it so it only takes files that are 1-2 hours old.
There are two ways where I can get the date for this process.
Filename contains timestamp = XXXXXXXXXXX_XXXXXXXX_valuereport_YYYYMMDDhhmmss.csv
The timestamp the file was created (please note we are talking about 800K-1M files in the directory and more is added every hour, so the fastest way would be appreciated.
So how do I insert something in my code, so it besides the header, only takes files that are <1-2 hours old?
Sorry about the code example, I am new to this site and not sure how to get it in the right order.
Nothing yet.
foreach ($FilePath in (Get-ChildItem $InputDirectory -File) | Select-Object -ExpandProperty FullName) {
$Header = Get-Content $FilePath -First 1
# test for a string in the header line that distincts it from the other files
if ($Header -match ';energy,Wh,') {
# the substring ';energy,Wh,' defines this file as a 'HeatMeter' file
Copy-Item -Path $FilePath -Destination $OutputPathHeat
} elseif ($Header -match ';fabrication-no,,inst-value,0,0,0;datetime,,inst-value,0,0,0;volume,m3') {
# the substring ';datetime,,inst-value,0,0,0;volume,m3' defines this file as a 'WaterMeter' file
Copy-Item -Path $FilePath -Destination $OutputPathWater
} else {
# if all key substrings above did not match, move to the 'Other' directory
Copy-Item -Path $FilePath -Destination $OutputPathOther
}

There are several ways to filter a directory listing. The easiest way is to pipe the result of Get-ChildItem through Where-Object like:
Get-ChildItem -Path $InputDirectory -File |
Where-Object { $_.CreationTime -gt (Get-Date).AddHours(-2) } |
Select-Object -ExpandProperty FullName |
ForEach-Object {
$FilePath = $_
$Header = Get-Content $FilePath -First 1
# test for a string in the header line that distincts it from the other files
if ($Header -match ';energy,Wh,') {
# the substring ';energy,Wh,' defines this file as a 'HeatMeter' file
Copy-Item -Path $FilePath -Destination $OutputPathHeat
}
elseif ($Header -match ';fabrication-no,,inst-value,0,0,0;datetime,,inst-value,0,0,0;volume,m3') {
# the substring ';datetime,,inst-value,0,0,0;volume,m3' defines this file as a 'WaterMeter' file
Copy-Item -Path $FilePath -Destination $OutputPathWater
}
else {
# if all key substrings above did not match, move to the 'Other' directory
Copy-Item -Path $FilePath -Destination $OutputPathOther
}
}
It checks that the CreationTime is greater than now - 2h. Note that the last modified (LastWriteTime) timestamp may also be suitable for your use case.

Related

Find similarly-named files, and if present, remove the files without a specific string using PowerShell

In a directory, there are files with the following filenames:
ExampleFile.mp3
ExampleFile_pn.mp3
ExampleFile2.mp3
ExampleFile2_pn.mp3
ExampleFile3.mp3
I want to iterate through the directory, and IF there is a filename that contains the string '_pn.mp3', I want to test if there is a similarly named file without the '_pn.mp3' in the same directory. If that file exists, I want to remove it.
In the above example, I'd want to remove:
ExampleFile.mp3
ExampleFile2.mp3
and I'd want to keep ExampleFile3.mp3
Here's what I have so far:
$pattern = "_pn.mp3"
$files = Get-ChildItem -Path '$path' | Where-Object {! $_.PSIsContainer}
Foreach ($file in $files) {
If($file.Name -match $pattern){
# filename with _pn.mp3 exists
Write-Host $file.Name
# search in the current directory for the same filename without _pn
<# If(Test-Path $currentdir $filename without _pn.mp3) {
Remove-Item -Force}
#>
}
enter code here
You could use Group-Object to group all files by their BaseName (with the pattern removed), and then loop over the groups where there are more than one file. The result of grouping the files and filtering by count would look like this:
$files | Group-Object { $_.BaseName.Replace($pattern,'') } |
Where-Object Count -GT 1
Count Name Group
----- ---- -----
2 ExampleFile {ExampleFile.mp3, ExampleFile_pn.mp3}
2 ExampleFile2 {ExampleFile2.mp3, ExampleFile2_pn.mp3}
Then if we loop over these groups we can search for the files that do not end with the $pattern:
#'
ExampleFile.mp3
ExampleFile_pn.mp3
ExampleFile2.mp3
ExampleFile2_pn.mp3
ExampleFile3.mp3
'# -split '\r?\n' -as [System.IO.FileInfo[]] | Set-Variable files
$pattern = "_pn"
$files | Group-Object { $_.BaseName.Replace($pattern,'') } |
Where-Object Count -GT 1 | ForEach-Object {
$_.Group.Where({-not $_.BaseName.Endswith($pattern)})
}
This is how your code would look like, remove the -WhatIf switch if you consider the code is doing what you wanted.
$pattern = "_pn.mp3"
$files = Get-ChildItem -Path -Filter *.mp3 -File
$files | Group-Object { $_.BaseName.Replace($pattern,'') } |
Where-Object Count -GT 1 | ForEach-Object {
$toRemove = $_.Group.Where({-not $_.BaseName.Endswith($pattern)})
Remove-Item $toRemove -WhatIf
}
I think you can get by here by adding file names into a hash map as you go. If you encounter a file with the ending you are interested in, check if a similar file name was added. If so, remove both the file and the similar match.
$ending = "_pn.mp3"
$files = Get-ChildItem -Path $path -File | Where-Object { ! $_.PSIsContainer }
$hash = #{}
Foreach ($file in $files) {
# Check if file has an ending we are interested in
If ($file.Name.EndsWith($ending)) {
$similar = $file.Name.Split($ending)[0] + ".mp3"
# Check if we have seen the similar file in the hashmap
If ($hash.Contains($similar)) {
Write-Host $file.Name
Write-Host $similar
Remove-Item -Force $file
Remove-Item -Force $hash[$similar]
# Remove similar from hashmap as it is removed and no longer of interest
$hash.Remove($similar)
}
}
else {
# Add entry for file name and reference to the file
$hash.Add($file.Name, $file)
}
}
Just get a list of the files with the _pn then process against the rest.
$pattern = "*_pn.mp3"
$files = Get-ChildItem -Path "$path" -File -filter "$pattern"
Foreach ($file in $files) {
$TestFN = $file.name -replace("_pn","")
If (Test-Path -Path $(Join-Path -Path $Path -ChildPath $TestFN)) {
$file | Remove-Item -force
}
} #End Foreach

If testfile1.log exists, create testfile2.log, and so on.. - PowerShell

My PowerShell script creates a log file, but when I run the script for the second time, it tells me that the testfile1.log file already exists.
How do I make the script if it finds testfile1.log, it creates testfile2.log, and if this also exists, it creates testfile3.log, and so on..
New-Item -Path $path -Name "testfile1.log" -ItemType "file"
You could do it this way, first get all the files in the desired path and sort them by the ending digits on their name. If no files are found create the testfile1.log, if there were files found, get the last sorted file (the one with the highest ending digit) extract the ending digits and add +1 to the count and use it to create the new file.
$files = Get-ChildItem $path -Filter testfile*.log | Sort-Object {
$_.BaseName -replace '\D' -as [int]
}
if(-not $files)
{
New-Item -Path $path -Name "testfile1.log" -ItemType File
}
else
{
[int]$number = $files[-1].BaseName -replace '\D'
$number++
New-Item -Path $path -Name "testfile$number.log" -ItemType File
}
An alternative method, based on this answer could be
$path = 'D:\Test'
$log = 'testfile'
$index = ((Get-ChildItem -Path $path -Filter "$log*.log" -File |
Where-Object { $_.BaseName -match "$log\d+$" } |
Select-Object #{Name = 'index'; Expression = {[int]($_.BaseName -replace '\D')}}).index |
Measure-Object -Maximum).Maximum + 1
# create the new file
New-Item -Path (Join-Path -Path $path -ChildPath "$log${index}.log") -ItemType File
A concise solution that also builds on this answer (see there for an explanation of the core technique):
$path = '.' # Output dir.
$nameTemplate = 'testfile{0}.log' # {0} is the sequence-number placeholder
New-Item -ItemType File -Path $path -Name (
$nameTemplate -f (1 + (
# Find all existing log files
Get-ChildItem (Join-Path $path $nameTemplate.Replace('{0}', '*')) |
Measure-Object -Maximum {
# Extract the embedded sequence number.
$_.Name -replace [regex]::Escape($nameTemplate).Replace('\{0}', '(\d+)'), '$1'
}
).Maximum)
) -WhatIf
Note: The -WhatIf common parameter in the command above previews the operation. Remove -WhatIf once you're sure the operation will do what you want.
Note:
The above uses a complex -replace operation to reliably extract the sequence number from existing file names; if you know that that only one number is present in each given file name, $_.BaseName -replace '\D' (removing all non-digit characters) will do in the Measure-Object call above.
If you wanted to use zero-padded, fixed-width sequence numbers, you can adjust (all occurrences of) the {0} placeholder accordingly; e.g, to create sequence numbers 01, 02, ... 99, use {0:00} - see the Composite formatting help topic, which describes the string formatting language also used by PowerShell's -foperator

Powershell to remove files from CSV older than x days

I am trying to read a CSV with a list of files including folder path and then delete them if they are older than x days.
I can do this is I list folders in the csv but cannot get it to work for just files.
CSV
FullName,,,,,,,,,,,,,
E:\$RECYCLE.BIN\S-1-5-21-352280589-691296097-1232828436-9414\$RCUCS3H.txt,,,,,,,,,,,,,
E:\$RECYCLE.BIN\S-1-5-21-352280589-691296097-1232828436-9414\$RWF5KKJ.txt,,,,,,,,,,,,,
E:\Account Lockout Files\Alockout.zip,,,,,,,,,,,,,
E:\Account Lockout Files\AlockoutXP.zip,,,,,,,,,,,,,
PowerShell contains.
$DatetoDelete = (Get-Date).AddDays(-3650)
Get-Content 'C:\delete\1.csv' | ForEach-Object { $_.Trim() } | Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item -Force
When I use the script I did above it just deletes the files even if they are older, please can any one assist? thanks.
You are almost there with your code, but you are missing the actual check for the last write time, if you use Get-Item to get the properties of the file, you will then have the LastWriteTime property for you to use in your Where-Object
The below should do what you need with just one extra command and a pipe
Get-Content 'C:\delete\1.csv' | ForEach-Object {
$_.Trim()
Get-Item -Path $_ |Where-Object { $_.LastWriteTime -lt $DatetoDelete } | Remove-Item -Force
}
Thanks for sharing an example of the input csv file. We now can be sure it actually is csv with headers and therefore, using Get-Content is the wrong cmdlet.
Try with Import-Csv instead:
$DatetoDelete = (Get-Date).AddDays(-3650).Date # set to midnight
Import-Csv -Path 'C:\delete\1.csv' | ForEach-Object {
$file = Get-Item -Path $_.FullName -ErrorAction SilentlyContinue
if (($file) -and $file.LastWriteTime -lt $DatetoDelete) {
$file | Remove-Item -Force
}
}
For some files in the csv you may not have permissions to delete, like perhaps the ones inside the E:\$RECYCLE.BIN\S-1-5-21-352280589-691296097-1232828436-9414..

How to copy files using a txt list to define beginning of file names

Hello awesome community :)
I have a list containing a bunch of SKU's. All the filenames of the files, that I need to copy to a new location, starts with the corresponding SKU like so
B6BC004-022_10_300_f.jpg
In this case "B6BC004" is the SKU and my txt list contains "B6BC004" along with many other SKU's.
Somewhere in the code below I know I have to define that it should search for files beginning with the SKU's from the txt file but I have no idea how to define it.
Get-Content .\photostocopy.txt | Foreach-Object { copy-item -Path $_ -Destination "Z:\Photosdestination\"}
Thanks in advance :)
If all files start with one of the SKU's, followed by a dash like in your example, this should work:
$sourceFolder = 'ENTER THE PATH WHERE THE FILES TO COPY ARE'
$destination = 'Z:\Photosdestination'
# get an array of all SKU's
$sku = Get-Content .\photostocopy.txt | Select-Object -Unique
# loop through the list of files in the source folder and copy all that have a name beginning with one of the SKU's
Get-ChildItem -Path $sourceFolder -File -Recurse |
Where-Object { $sku -contains ($_.Name -split '\s*-')[0] } |
ForEach-Object { $_ | Copy-Item -Destination $destination }
I haven't tested this so please proceed with caution!
What is does it loops through all the items in your photostocopy.txt file, searches the $source location for a file(s) with a name like the current item from your file. It then checks if any were found before outputting something to the console and possibly moving the file(s).
$source = '#PATH_TO_SOURCE'
$destination = '#PATH_TO_DESTINATION'
$photosToCopy = Get-Content -Path '#PATH_TO_TXT_FILE'
$photosToCopy | ForEach-Object{
$filesToCopy = Get-ChildItem -Path $source -File | Where-Object {$_.Name -like "$_*"}
if ($fileToCopy.Count -le 0){
Write-Host "No files could be found for: " $_
}else{
$filesToCopy | ForEach-Object{
Write-Host "Moving: " $_.Name
Copy-Item -Path $_.FullName -Destination $destination
}
}
}
Let me know how if this helps you :)

Copy content form all text files from a directory and write to single text file

The code below will copy content from text file to a file in directory but it will overwrite the existing file content:
Get-ChildItem $FilePath -Recurse -Include *.docx,*folder\abc\source.txt | ForEach-Object {Copy-Item $_.FullName -Destination C:\destination.txt}
I want to append all the content to a single file.
Use Get-Content and Add-Content instead of Copy-Item:
Get-ChildItem $FilePath -Recurse -Include *.txt | ? {
$_.FullName -like '*folder\abc\source.txt'
} | Get-Content | Add-Content 'C:\destination.txt'
Edit: As #BaconBits correctly pointed out in the comments to your question you cannot simply concatenate Word documents and text files, because the former are binary files (zip archives containing a bunch of XML files, actually). However, if you want just the (text) content of the document appended, you could do something like this:
$wd = New-Object -COM 'Word.Application'
Get-ChildItem $FilePath -Recurse | ? {
$_.Extension -eq '.docx' -or
$_.FullName -like '*folder\abc\source.txt'
} | % {
$path = $_.FullName
switch ($_.Extension) {
'.docx' {
$doc = $wd.Documents.Open($path)
$doc.Content.Text
$doc.Close()
}
'.txt' { Get-Content $path }
}
} | Add-Content 'C:\destination.txt'
$wd.Quit()
Note: I was unable to get Get-ChildItem to work with partial paths in the -Include parameter, so I moved those conditions to a Where-Object filter.