Get unique values - powershell

Question

We're trying to optimize some code that removes duplicates from an Array as fast as possible. Normally this can be easily done by piping the input to Group-Object and then using only the Name property. But we would like to avoid the pipeline, as it is slower.
However, we tried the following code:
[System.Collections.ArrayList]$uniqueFrom = #()
$From = #('A', 'A', 'B')
$From.Where({-not ($uniqueFrom.Contains($_))}).ForEach({
$uniqueFrom.Add($_)
})
$uniqueFrom
In theory, this should work. But for one reason or another the output is not the expected #('A', 'B'). Why is it not reevaluating the ArrayList in the .where clause?

Answer 1

In my experience reducing the 'pipe filtering' to get the unique values can be achieve by using DataView. If you are processing an array you need to convert this to a DataTable first before you get the values using the DataView.
e.g.
$arr = #('val1','val1','val1','val2','val1','val3'....)
$newDatatable = New-Object System.Data.Datatable
[void]$newDatatable.Columns.Add("FetchUniqueColumn")
foreach($e in $arr)
{
$row = $newDatatable.NewRow()
$row.Item('FetchUniqueColumn') = $e
$newDatatable.Rows.Add($row)
}
$filterDataView = New-Object System.Data.Dataview($newDatatable)
$UniqueDT = $filterDataView.ToTable($true,'FetchUniqueColumn')
$UniqueValues_array = $UniqueDT.Rows.FetchUniqueColumn
Note this is a whole lot faster if your input is a DataTable since you don't have to convert it anymore prior to setting the DataView filter for unique values to $true in creating the $UniqueDT datatable from the dataview:
$UniqueDT = $filterDataView.ToTable($true,'FetchUniqueColumn')
Tested by querying 1 column with 3000 rows datatable from SQL.
My results are as follows:
**With 1 column Data Table as input
Select -Unique - 300 ms
Using DataView - 21 ms
**With #() array as input (converted SQL results to array prior to benchmarking)
Select Unique - 262 ms
Using DataView - 106 ms

Answer 2

Disclaimer: in this answer I'm just explaining why the current code isn't working, not attempting to give alternative solution. For solution check the accepted answer.
Why is it not reevaluating the ArrayList in the .where clause?
It's not supposed to do this. What it is actually doing is filtering here:
$From.Where({-not ($uniqueFrom.Contains($_))})
and then executing
$uniqueFrom.Add($_)
for each element. As you did
[System.Collections.ArrayList]$uniqueFrom = #()
this array is empty and therefore will return $false for any $uniqueFrom.Contains($_)
Proof:
To verify that what I've written above is true you can do the following:
[System.Collections.ArrayList]$uniqueFrom = #()
$uniqueFrom.add("A")
$From.Where({-not ($uniqueFrom.Contains($_))}).ForEach({
$uniqueFrom.Add($_)
})
Output is A, B (A was added manually, two A were skipped as this entry already exists in $uniqueFrom, B was added inside ForEach) as expected.