Remove rows present in one .csv from another .csv (windows, powershell, notepad++) - powershell

Question

I have notepad++, powershell, and excel 2007. I have two .csv files named
database.csv and import.csv . Import.csv contains new entries that I want to put
into my database online. Database.csv contains the current records in that database.
Both files contain a simple comma-newline delimited list of unique values.
However, the database may already contain some entries in the new file. And, the new
file contains entries that are not in the database. And, the database file contains
entries that are still retained for recording purposes, but are not in the input file.
Simply combining them results in duplicates of any record that has an ongoing existence.
It also results in single copies of records only present in the database and records only
present in the input file.
What I want is a file that only contains records that are only present in the input file.
Any advice?

Answer 1

Assuming your csv files have the columns a, b, & c:
$db = Import-Csv database.csv
$import = Import-Csv import.csv
$new = Compare-Object -ReferenceObject $db -DifferenceObject $import -Property a,b,c -PassThru | ? { $_.SideIndicator -eq "=>" } | Select a,b,c
Just replace a, b, and c with the names of the columns you want to compare

Answer 2

Powershell:
Get-Content <database file> -TotalCount 1 |
Set-Content C:\somedir\ToUpload.csv
$import = #{}
Get-Content <import file> |
select -Skip 1
foreach {
$import[$_] = $true
}
Get-Content <Database file> |
select -Skip 1 |
foreach {
if ($import[$_])
{
$import[$_].remove()
}
}
$import.Keys |
Add-Content C:\Somedir\ToUpload.csv
Alternatively, reading both files into memory:
Get-Content <database file> -TotalCount 1 |
Set-Content C:\somedir\ToUpload.csv
$import = Get-Content <import file>
select -Skip 1
$database = Get-Content <database file>
select -Skip 1
$import |
where {$database -notcontains $_} |
Add-Content C:\somedir\ToUpload.csv
The solutions using import / export csv will work but impose additional memory and process overhead compared to dealing with the files as text data. The difference may be trivial or substantial, depending on the size of the files and the number of columns there are in the csv files. IMHO.

Answer 3

Compare-Object struggles sometimes with customobject imported from csv if you don't have any specific properties to match.
If you want performance(for large csv files), you could try this:
$i = #{}
[IO.File]::ReadAllLines("C:\input.csv") | % { $i[$_] = $true }
$reader = New-Object System.IO.StreamReader "C:\db.csv"
#Skip header. This way the output file(new.csv) will get input.csv's header
$reader.ReadLine() | Out-Null
while (($line = $reader.ReadLine()) -ne $null) {
#Remove row if it exists in db.csv
if ($i.ContainsKey($line)) {
$i.Remove($line)
}
}
$reader.Close()
$i.Keys | Add-Content c:\new.csv