Powershell - Search every element of a large array against every element of another large array - powershell

Question

I have two large arrays. One is an array (call it Array1) of 100,000 PSCustomObjects, each of which has a property called "Token". And the other array is simply an array of strings, the size of this second array being 2500 elements.
The challenge is that EVERY element of Array1 needs to be checked against all the elements in Array2 and tagged accordingly. i.e., if the the Token value from Array1 matches any of the elements from Array2, label it as "Match found!"
Looping through would actually make it extremely slow. Is there a better way to do this?
P.S.: The items in Array1 have an ordinal number property as well, and the array is sorted in that order.
Here is the code:
$Array1 = #()
$Array2 = #()
#Sample object:
$obj = New-Object -TypeName PSCustomObject
$obj | Add-Member -MemberType NoteProperty -Name Token -Value "SOMEVALUEHERE"
$obj | Add-Member -MemberType NoteProperty -Name TokenOrdinalNum -Value 1
$Array1 += $obj # This array has 100K such objects
$Array2 = #("VAL1", "SOMEVALUEHERE", ......) #Array2 has 2500 such strings.
The output of this would need to be a new array of objects, say 'ArrayFinal', that has an additional noteproperty called 'MatchFound'.
Please help.

Answer 1

I would create a Hashtable for fast lookups from the values in your $Array2.
For clarity, I have renamed $Array1 and $Array2 into $objects and $tokens.
# the object array
$objects = [PsCustomObject]#{ Token = 'SOMEVALUEHERE'; TokenOrdinalNum = 1 },
[PsCustomObject]#{ Token = 'VAL1'; TokenOrdinalNum = 123 },
[PsCustomObject]#{ Token = 'SomeOtherValue'; TokenOrdinalNum = 555 } # etcetera
# the array with token keywords to check
$tokens = 'VAL1', 'SOMEVALUEHERE', 'ShouldNotFindThis' # etcetera
# create a lookup Hashtable from the array of token values for FAST lookup
# you can also use a HashSet ([System.Collections.Generic.HashSet[string]]::new())
# see https://learn.microsoft.com/en-us/dotnet/api/system.collections.generic.hashset-1
$lookup = #{}
$tokens | ForEach-Object { $lookup[$_] = $true } # it's only the Keys that matter, the value is not important
# now loop over the objects in the first array and check their 'Token' values
$ArrayFinal = foreach ($obj in $objects) {
$obj | Select-Object *, #{Name = 'MatchFound'; Expression = { $lookup.ContainsKey($obj.Token) }}
}
# output on screen
$ArrayFinal | Format-Table -AutoSize
# write to Csv ?
$ArrayFinal | Export-Csv -Path 'Path\To\MatchedObjects.csv' -NoTypeInformation
Output:
Token TokenOrdinalNum MatchFound
----- --------------- ----------
SOMEVALUEHERE 1 True
VAL1 123 True
SomeOtherValue 555 False

Answer 2

100kb objects isn't too big. Here's an example using compare-object. By default it checks every object against every other object (919 ms). EDIT: Ok, if I change the order of $b, it takes much longer (13 min). Sorting both lists first should work well, if most of the positions end up the same.(1.99 s with measure-command). If every item were off by 1 position it will still take a long time ($b = 1,$b).
$a = foreach ($i in 1..100kb) { [pscustomobject]#{token = get-random} }
$a = $a | sort-object token
$b = $a.token | sort-object
compare-object $a.token $b -IncludeEqual
InputObject SideIndicator
----------- -------------
1507400001 ==
120471924 ==
28523825 ==
...