Powershell - Remove Duplicate lines in TXT based on ID - powershell

Question

I have a TXT-File with thousands of lines. The number after the first Slash is the image ID.
I want to delete all lines so that only one line remains for every ID. Which of the lines is getting killed doesn't matter.
I tried to pipe the TXT to a CSV with Powershell and work with the unique parameter. But it didnt work. Any ideas how I can iterate through the TXT and kill all lines, so that always only one line per unique ID remains? :/
Status Today
thumbnails/4000896042746/2021-08-17_4000896042746_small.jpg
thumbnails/4000896042746/2021-08-17_4000896042746_smallX.jpg
thumbnails/4000896042333/2021-08-17_4000896042746_medium.jpg
thumbnails/4000896042444/2021-08-17_4000896042746_hugex.jpg
thumbnails/4000896042333/2021-08-17_4000896042746_tiny.jpg
After the script
thumbnails/4000896042746/2021-08-17_4000896042746_small.jpg
thumbnails/4000896042333/2021-08-17_4000896042746_medium.jpg
thumbnails/4000896042444/2021-08-17_4000896042746_hugex.jpg

Answer 1

If it concerns "TXT-File with thousands of lines", I would use the PowerShell pipeline for this because (if correctly setup) it will perform the same but uses far less memory.
Performance improvements might actually be leveraged from using a HashTable (or a HashSet) which is based on a binary search (and therefore much faster then e.g. grouping).
(I am pleading to get an accelerated HashSet #16003 into PowerShell)
$Unique = [System.Collections.Generic.HashSet[string]]::new()
Get-Content .\InFile.txt |ForEach-Object {
if ($Unique.Add(($_.Split('/'))[-2])) { $_ }
} | Set-Content .\OutFile.txt

Answer 2

To add to iRons great answer, I've done a speed comparison on 5 different ways to do it using 250k lines of the OPs' example.
Using the Get-Content -raw read and write using Set-Content method is the fastest way to do it. At least in these examples, as it is nearly 3x faster than using Get-Content and Set-Content.
I was curious to see how the HashSet method stacked up against the System.Collections.ArrayList one. And as you can see from the result below for that it's not too dissimilar.
Edit note Got the -raw switch to work as it needed splitting by a new line.
$fileIn = "C:\Users\user\Desktop\infile.txt"
$fileOut = "C:\Users\user\Desktop\outfile.txt"
# All examples below tested with 250,000 lines
# In order from fastest to slowest
#
# EXAMPLE 1 (Fastest)
#
# [Finished in 2.4s]
# Using the -raw switch only with Get-Content
$Unique = [System.Collections.Generic.HashSet[string]]::new()
$fileInSplit = (Get-Content -raw $fileIn).Split([Environment]::NewLine,[StringSplitOptions]::None)
$fileInSplit |ForEach-Object {
if ($Unique.Add(($_.Split('/'))[-2])) { $_ }
} | Set-Content $fileOut
#
# EXAMPLE 2 (2nd fastest)
#
# [Finished in 2.5s]
# Using the -raw switch with Get-Content
# Using [IO.File] for write only
$Unique = [System.Collections.Generic.HashSet[string]]::new()
$fileInSplit = (Get-Content -raw $fileIn).Split([Environment]::NewLine,[StringSplitOptions]::None)
$contentToWriteArr = New-Object System.Collections.ArrayList
$fileInSplit |ForEach-Object {
if ($Unique.Add(($_.Split('/'))[-2])) { [void]$contentToWriteArr.Add($_) }
}
[IO.File]::WriteAllLines($fileOut, $contentToWriteArr)
# #
# EXAMPLE 3 (3rd fastest example)
#
# [Finished in 2.7s]
# Using [IO.File] for the read and write
$Unique = [System.Collections.Generic.HashSet[string]]::new()
$fileInSplit = ([IO.File]::ReadAllLines($fileIn)).Split([Environment]::NewLine,[StringSplitOptions]::None)
$contentToWriteArr = [Collections.Generic.HashSet[string]]::new()
$fileInSplit |ForEach-Object {
if ($Unique.Add(($_.Split('/'))[-2])) { $contentToWriteArr.Add($_) | out-null }
}
[IO.File]::WriteAllLines($fileOut, $contentToWriteArr)
#
# EXAMPLE 4 (4th fastest example)
#
# [Finished in 2.8s]
# Using [IO.File] for the read only
$Unique = [System.Collections.Generic.HashSet[string]]::new()
$fileInSplit = ([IO.File]::ReadAllLines($fileIn)).Split([Environment]::NewLine,[StringSplitOptions]::None)
$fileInSplit |ForEach-Object {
if ($Unique.Add(($_.Split('/'))[-2])) { $_ }
} | Set-Content $fileOut
#
# EXAMPLE 5 (5th fastest example)
#
# [Finished in 2.9s]
# Using [IO.File] for the read and write
# This is using a System.Collections.ArrayList instead of a HashSet
$Unique = [System.Collections.Generic.HashSet[string]]::new()
$fileInSplit = ([IO.File]::ReadAllLines($fileIn)).Split([Environment]::NewLine,[StringSplitOptions]::None)
$contentToWriteArr = New-Object System.Collections.ArrayList
$fileInSplit |ForEach-Object {
if ($Unique.Add(($_.Split('/'))[-2])) { $contentToWriteArr.Add($_) | out-null }
}
[IO.File]::WriteAllLines($fileOut, $contentToWriteArr)
#
# EXAMPLE 6 (Slowest example) - As per iRons answer
#
# [Finished in 7.2s]
$Unique = [System.Collections.Generic.HashSet[string]]::new()
$fileInSplit = Get-Content $fileIn
$fileInSplit |ForEach-Object {
if ($Unique.Add(($_.Split('/'))[-2])) { $_ }
} | Set-Content $fileOut

Answer 3

Here a solution that uses a calculated property to create an object that contains the ID and the FileName. Then I group the result based on the ID, iterate over each group and select the first FileName:
$yourFileList = #(
'thumbnails/4000896042746/2021-08-17_4000896042746_small.jpg',
'thumbnails/4000896042746/2021-08-17_4000896042746_smallX.jpg',
'thumbnails/4000896042333/2021-08-17_4000896042746_medium.jpg',
'thumbnails/4000896042444/2021-08-17_4000896042746_hugex.jpg',
'thumbnails/4000896042333/2021-08-17_4000896042746_tiny.jpg'
)
$yourFileList |
Select-Object #{Name = "Id"; Expression = { ($_ -split '/')[1] } }, #{ Name = 'FileName'; Expression = { $_ } } |
Group Id |
ForEach-Object { $_.Group[0].FileName }

Answer 4

You can group by custom property. So if you know what's your ID then you just have to group by that and then take the first element from the group:
$content = Get-Content "path_to_your_file";
$content = ($content | group { ($_ -split "/")[1] } | % { $_.Group[0] });
$content | Out-File "path_to_your_result_file"