PowerShell Keeping track of updated and rejected rows while cleaning up files - powershell

Question

Add-Member, hashtables, arrays and such confuse me a bit so I'm not sure the best way to approach this. My goal is to take an input.CSV, perform clean up and send those cleaned rows to Fixed.CSV, and send any 'reject rows' that couldn't be handled to reject.CSV with an explanation of why they were rejected.
My original script was splitting the 'good' from the 'bad' based on a single characteristic (e.g. a missing Account ID), but as a I get into the clean-up, there are other things that would cause a row to error out and I don't want to read the data into memory with .Where() and continually 'split' it - especially considering I'd like to finish with only 3 files total (OG-input.CSV, Fixed.CSV, Junk-reject.CSV).
$data, $rejectData = (Import-CSV $CSV).Where({![string]::IsNullOrEmpty($_."Account ID")}, 'Split')
If($rejectData){
$rejectData | Add-Member -NotePropertyName "Reject Reason" -NotePropertyValue "Account ID missing"
$rejectData | Export-CSV -LiteralPath "$($CSV.DirectoryName)\$($CSV.BaseName)_reject.csv" -NoTypeInformation
My output file was basically created after I had performed a bunch of steps on each row of $data above.
$outputFile = New-Object System.Collections.ArrayList
Foreach($row in $data){
# Do stuff, check using If, make updates, etc.
[void]$outputFile.Add($row)
}
$outputFile | Export-CSV -LiteralPath "$($CSV.DirectoryName)\$($CSV.BaseName)Fixed.csv" -NoTypeInformation
What I'm thinking at this point is instead of splitting the data initially, I should just iterate through all rows and if I can update them; I will and send to $outputFixed. If there is an error that can't be corrected, I'll send them to $outputReject - but here's the caveat, I want to add a new column for "Reject Reason" and then update that as I go. What I mean by this is, there could be multiple reasons a row gets rejected and I'd like to track each one. I've gotten it somewhat close, but creating the new column is giving me trouble. I was originally going to use Add-Member for the first time I add the column, and then just update the value in that column for each $row; something like $row."Reject Reason" = "$($row."Reject Reason")|New Reason" as this gets me a pipe-delimited list of reasons a row rejected. Then I found Powershell add-member. Add a member that's an ArrayList? that got me thinking maybe I could have the reasons within Reject Reason be a list themselves rather than just delimited. However, I'm not sure I quite understand the nuances of the answers proposed and can't figure out what might work best for me.

Answer 1

Nested arrays/lists are great, but you'll have to consider how you want to store and display your data.
A CSV file, like a table, doesn't properly handle nested objects like lists or arrays. This can be fine if you know your data, and don't mind converting your RejectReason field from/to a delimited string when you read it. For example, you could use Where-Object's filter block to find all the entries in $outputRejected with a specific reason:
# similar to what you had before
$csv = Import-Csv $path
$report = foreach ($row in $csv) {
$row | Add-Member -NotePropertyName 'RejectCode' -NotePropertyValue ''
if ($row.id -lt 5) { $row.RejectCode = $row.RejectCode+'Too Low|' }
if ($row.id -gt 3) { $row.RejectCode = $row.RejectCode+'Too High|' }
# Output the finalized row
$row
}
# Example: filter by reason code
$OutputRejected | Where-Object {($_.Reason -split '\|') -contains 'Too High'}
ID RejectCode
-- ----------
4 Too Low|Too High|
5 Too High|
For what you are doing, this usually works just fine. You have to be careful of your additional separator characters, but since you're defining the RejectCode yourself, it shouldn't be an issue.
For anything more complicated, I tend to create a PSCustomObject from each $row and set each property to what I need. This tends to work a little better for me than using Add-Member:
$report = foreach ($row in $csv) {
# custom object with manually defined properties
$reportRow = [PSCustomObject][Ordered]#{
ID = $row.ID
Name = $row.Name
Data = # run some commands to fix bad data
Reasons = #() # list object
}
# can edit properties as normal
if ($row.id -lt 5) { $reportRow.Reasons += $row.RejectCode+'Too Low|' }
if ($row.id -gt 3) { $reportRow.Reasons += $row.RejectCode+'Too High|' }
$reportrow
}
Just be aware that powershell's CSV commands tend to squish properties into the unhelpful system.object[] text when your properties aren't simple values like strings or ints. A better option for saving nested objects like this is a structured format like JSON. e.g.: $report | ConvertTo-Json | Out-File $path.

Answer 2

Without seeing any of your CSV, You could do something like this:
$csvPath = 'X:\Temp'
$original = Import-CSV -Path (Join-Path -Path $csvPath -ChildPath 'OG-input.CSV')
# create a List object to collect the rejected items
$rejects = [System.Collections.Generic.List[object]]::new()
$correct = foreach ($item in $original) {
$reason = $null
if ([string]::IsNullOrWhiteSpace($_.'Account ID')) { $reason = "Empty 'Account ID' field" }
elseif ($_.'Account ID'.Length -gt 20) { $reason = "'Account ID' field exceeds maximum length" }
# more elseif checks go here
# after all checks are done
if (!$reason) {
# all OK for this row; just output so it gets collected in $correct
$item
}
else {
# it's a rejected item, add an object to the $rejects list
$obj = $item | Select-Object *, #{Name = 'Reason'; Expression = {$reason}}
$rejects.Add($obj)
}
}
# save both files
$correct | Export-Csv -Path (Join-Path -Path $csvPath -ChildPath 'Fixed.CSV') -NoTypeInformation
$rejects | Export-Csv -Path (Join-Path -Path $csvPath -ChildPath 'Junk-reject.CSV') -NoTypeInformation
You need to fill in the rest of the checks and reasons for rejection of course

Answer 3

Here's what I ended up going with. I think it works, as the output looks about like I expected.
Foreach($row in $data){
#Process all reject reasons first and reject those rows
If([string]::IsNullOrEmpty($row."Account ID")){
$row | Add-Member -NotePropertyName "Reject Reason" -NotePropertyValue ("$($row."Reject Reason")", "Missing Account ID" -Join "|").TrimStart("|") -Force
}
If([string]::IsNullOrEmpty($row."Service Start Dates") -And ([string]::IsNullOrEmpty($row."Service End Dates"))){
$row | Add-Member -NotePropertyName "Reject Reason" -NotePropertyValue ("$($row."Reject Reason")", "Missing Both Service Dates" -Join "|").TrimStart("|") -Force
}
If(Get-Member -InputObject $row "Reject Reason"){
[void]$outputReject.Add($row)
Continue
}
If([string]::IsNullOrEmpty($row."Birth Date")){
$row."Birth Date" = $dte
}
If([string]::IsNullOrEmpty($row."Gender")){
$row."Gender" = "Female"
}
If( [string]::IsNullOrEmpty($row."Service Start Dates") -And !( [string]::IsNullOrEmpty($row."Service End Dates"))){
$row."Service Start Dates" = $row."Service End Dates"
}
[void]$outputFixed.Add($row)
}
$outputFixed | Export-CSV -LiteralPath "$($inputFile.DirectoryName)\$($inputFile.BaseName)Fixed.csv" -NoTypeInformation
If($outputReject){
$outputReject | Export-CSV -LiteralPath "$($inputFile.DirectoryName)\$($inputFile.BaseName)RejectedRows.csv" -NoTypeInformation
}
Basically I'm still collecting each row in an ArrayList that will be output once the entire file has been processed. I'm using Add-Member with -Force to 'overwrite' the reject reason(s) and a -Join of the text with a .TrimStart("|") to get rid of the leading pipe. This will definitely work for me (plus was easy to implement with what I already had written)