How to merge and remove duplicates of CSV files using Powershell - powershell

Question

I would like to remove duplicates in a CSV file using PowerShell. I know that there are posts about this already but I can't seem to find one that helps.
I'm trying to merge 2 CSV Files that have the same header and then remove the duplicates of the resulting file based on the IDs listed in the first column and then put it to the same CSV file.
The properties of the file are as follows:
And when I try to use the sort and unique method, I get the following (not a table:
Here is my code so far:
####
#MERGE
$getFirstLine = $true
get-childItem "C:\IGHandover\Raw\IG_INC*.csv"| foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "C:\IGHandover\new.csv" $linesToWrite
}
####
#REMOVE DUPLICATES
Import-Csv "C:\IGHandover\new.csv" | Sort inc_number -Unique |
Set-Content "C:\IGHandover\new.csv"

Answer 1

Don't use Get-Content or Set-Content to import or export csv file
Import-Csv (Get-ChildItem 'C:\IGHandover\Raw\IG_INC*.csv') |
Sort-Object -Unique inc_number |
Export-Csv 'C:\IGHandover\new.csv' -NoClobber -NoTypeInformation

Answer 2

I guess you want to update a table (HandoverINC.csv) with records from a new table (New.csv), replacing any records in the HandoverINC.csv with the same primary key (inc_number) from the New.csv in the HandoverINC.csv. And add any new records in the New.csv to the HandoverINC.csv (Basically what is called a Full Join in SQL).
Using the Join-Object described at: https://stackoverflow.com/a/45483110/1701026
Import-CSV .\HandoverINC.csv | FullJoin (Import-CSV .\New.csv) inc_number {$Right.$_} | Export-CSV .\HandoverINC.csv

Answer 3

As suggested by Lieven Keersmaekers and Vivek Kumar, I've made a few changes in my code:
Put the merged contents to a temporary file
Import the csv file with the merge contents
Sort the column of reference and use the unique parameter
Export the results to a new csv file
I found that my code was similar to Vincent K's:
#MERGE
$getFirstLine = $true
get-childItem "C:\IGHandover\Raw\IG_INC*.csv"|
foreach {
$filePath = $_
$lines = $lines = Get-Content $filePath
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}}
$getFirstLine = $false
Add-Content "C:\IGHandover\HandoverINCtemp.csv" $linesToWrite }
#REMOVE DUPLICATES
Import-Csv "C:\IGHandover\HandoverINCtemp.csv" | Sort inc_number -Unique |
Export-Csv "C:\IGHandover\HandoverINC.csv" -NoClobber -NoTypeInformation -Force
Remove-Item "C:\IGHandover\HandoverINCtemp.csv"
To simplify (merging and removing duplicates with the same header), as suggested by Vincent:
Import-Csv (Get-ChildItem "C:\IGHandover\Raw\IG_INC*.csv") | Sort inc_number -Unique |
Export-Csv "C:\IGHandover\HandoverINC.csv" -NoClobber -NoTypeInformation -Force
I hope this helps anyone who'd like to do the same with their files