Append dictionary to a dictionary in powershell (hashtable) - powershell

I have two dictionaries like this:
$first = #{}
$first.Add('John', 'Doe')
$first.Add('Johnny', 'Doe')
$second = #{}
$second.Add('Jack', 'Test')
$second.Add('Jacky', 'Test')
And I have a general $all = #{} dictionary, that stands for all dictionaries combined.
Ex. when I want to see all keys that $all contains:
foreach($key in $all){
Write-Host $key
}
It will show this:
John
Johnny
Jack
Jacky
p.s. I have this one:
$all = #{}
$all_dict = #{}
$all_dict += $first
$all_dict += $second
foreach($dict in $all_dict){
foreach($key in $dict.Key){
$all.Add($key, $dict[$key])
}
}
But I was wondering if there is another way to do it without the need to add all dictionaries to an array and then iterate through them

I wouldn't do the += addition to hashtables, but instead use a ForEach-Object on the hashes .Keys. That way, the code can be shortened, but also it will leave you an easier choice whether you would want the possible duplicates from Hashtable 1 ($first) to be overwritten by the values from the second Hashtable ($second).
Something like this:
$first = #{}
$first.Add('John', 'Doe')
$first.Add('Johnny', 'Doe')
$second = #{}
$second.Add('Jack', 'Test')
$second.Add('Jacky', 'Test')
$second.Add('Johnny', 'Depp') # Duplicate key: same first name, different lastname
$all = #{}
# copy all keys and values from the $first Hashtable into $all
$first.Keys | ForEach-Object { $all[$_] = $first[$_] }
For the next part, you'll have to decide what to do with duplicate keys:
Method 1
# add the stuff from Hashtable $second to it:
# this will overwrite the value if the key already exists (i.e. $second value 'wins')
$second.Keys | ForEach-Object { $all[$_] = $second[$_] }
OR use Method 2
# make sure the value of the $first hashtable is NOT overwritten (i.e. $first value 'wins')
$second.Keys | ForEach-Object { if (!($all.ContainsKey($_))) { $all[$_] = $second[$_] }}
In case you choose to overwrite (method 1), the $all hash will contain
Name Value
---- -----
John Doe
Jacky Test
Johnny Depp
Jack Test
If you choose NOT to overwrite (method 2), $all will be
Name Value
---- -----
John Doe
Jacky Test
Johnny Doe
Jack Test
Edit
There is another approach where you rely on the fact that an exception is thrown if you try to add an entry that already exists. In that case, use the .Add(key, value) method and wrap it inside a try{..} catch{..} block.
Without that catch, the error prevents the $all Hashtable to be filled, as it stops at the first duplicate key you try to add.
$second.Keys | ForEach-Object {
try {
$all.Add($_, $second[$_])
}
catch {
# catch the exception in order to carry on adding items
# the effect will be that the values from $first will not be overwritten
# just like with method 2
Write-Warning $_.Exception.Message
}
}

I think your $all_dict already contains what you want (i.e. a hashtable with all 4 entries), but your foreach( $dict in $all_dict ) isn't enumerating the hashtable entries like you expect it to.
The quick answer is to iterate over the Keys collection instead:
foreach( $key in $all.Keys )
{
write-host $key
}
The longer answer is that in your example PowerShell is doing some "helper" things for you with enumeration - foreach($key in $all) is only enumerating over a single object ($all), but write-host $all is evaluating an array of all of the entries in $all and serializing them into a single string:
Compare the behaviour of these two lines and you can see the difference:
PS> foreach($item in #{ "aaa"="bbb"; "ccc"="ddd" }) { write-host $item }
System.Collections.DictionaryEntry System.Collections.DictionaryEntry
PS> foreach($item in #{ "aaa"="bbb"; "ccc"="ddd" }.Keys) { write-host $item }
ccc
aaa
By the way, watch out for if your keys collide - if you try #{ "aaa"="bbb"; "ccc"="ddd" } + #{ "aaa"="eee" } for example, you'll get an error Item has already been added. Key in dictionary: 'aaa' Key being added: 'aaa'. so you might want to find a better way to merge your hashtables rather than just using +.

Am I not understanding the question? You can add them.
$first = #{John = 'Doe'; Johnny = 'Doe'} # hashtables
$second = #{Jacky = 'Test'; Jack = 'Test'}
$all = $first + $second # merge two hashtables
foreach ($i in $all.getenumerator()) { $i } # loops 4 times

Related

Check if a condition is met by a line within a TXT but "in an advanced way"

I have a TXT file with 1300 megabytes (huge thing). I want to build code that does two things:
Every line contains a unique ID at the beginning. I want to check for all lines with the same unique ID if the conditions is met for that "group" of IDs. (This answers me: For how many lines with the unique ID X have all conditions been met)
If the script is finished I want to remove all lines from the TXT where the condition was met (see 2). So I can rerun the script with another condition set to "narrow down" the whole document.
After few cycles I finally have a set of conditions that applies to all lines in the document.
It seems that my current approach is very slow.( one cycle needs hours). My final result is a set of conditions that apply to all lines of code.
If you find an easier way to do that, feel free to recommend.
Help is welcome :)
Code so far (does not fullfill everything from 1&2)
foreach ($item in $liste)
{
# Check Conditions
if ( ($item -like "*XXX*") -and ($item -like "*YYY*") -and ($item -notlike "*ZZZ*")) {
# Add a line to a document to see which lines match condition
Add-Content "C:\Desktop\it_seems_to_match.txt" "$item"
# Retrieve the unique ID from the line and feed array.
$array += $item.Split("/")[1]
# Remove the line from final document
$liste = $liste -replace $item, ""
}
}
# Pipe the "new cleaned" list somewhere
$liste | Set-Content -Path "C:\NewListToWorkWith.txt"
# Show me the counts
$array | group | % { $h = #{} } { $h[$_.Name] = $_.Count } { $h } | Out-File "C:\Desktop\count.txt"
Demo Lines:
images/STRINGA/2XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGA/3XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGB/4XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGB/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg images/STRINGC/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
performance considerations:
Add-Content "C:\Desktop\it_seems_to_match.txt" "$item"
try to avoid wrapping cmdlet pipelines
See also: Mastering the (steppable) pipeline
$array += $item.Split("/")[1]
Try to avoid using the increase assignment operator (+=) to create a collection
See also: Why should I avoid using the increase assignment operator (+=) to create a collection
$liste = $liste -replace $item, ""
This is a very expensive operation considering that you are reassigning (copying) a long list ($liste) with each iteration.
Besides it is a bad practice to change an array that you are currently iterating.
$array | group | ...
Group-Object is a rather slow cmdlet, you better collect (or count) the items on-the-fly (where you do $array += $item.Split("/")[1]) using a hashtable, something like:
$Name = $item.Split("/")[1]
if (!$HashTable.Contains($Name)) { $HashTable[$Name] = [Collections.Generic.List[String]]::new() }
$HashTable[$Name].Add($Item)
To minimize memory usage it may be better to read one line at a time and check if it already exists. Below code I used StringReader and you can replace with StreamReader for reading from a file. I'm checking if the entire string exists, but you may want to split the line. Notice I have duplicaes in the input but not in the dictionary. See code below :
$rows= #"
images/STRINGA/2XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGA/3XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/4XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGC/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGA/2XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGA/3XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/4XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGB/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
images/STRINGC/5XXXXXXXX_rTTTTw_GGGG1_Top_MMM1_YY02_ZZZ30_AAAA5.jpg
"#
$dict = [System.Collections.Generic.Dictionary[int, System.Collections.Generic.List[string]]]::new();
$reader = [System.IO.StringReader]::new($rows)
while(($row = $reader.ReadLine()) -ne $null)
{
$hash = $row.GetHashCode()
if($dict.ContainsKey($hash))
{
#check if list contains the string
if($dict[$hash].Contains($row))
{
#string is a duplicate
}
else
{
#add string to dictionary value if it is not in list
$list = $dict[$hash].Value
$list.Add($row)
}
}
else
{
#add new hash value to dictionary
$list = [System.Collections.Generic.List[string]]::new();
$list.Add($row)
$dict.Add($hash, $list)
}
}
$dict

Powershell sorting hash table

I am seeing some seemingly very weird behavior with a hash table I am sorting and then trying to review the results. I build the hash table, then I need to sort that table based on values, and I see two bits of weirdness.
This works fine outside of a class
$hash = [hashtable]::New()
$type = 'conformset'
$hash.Add($type, 1)
$type = 'applyset'
$hash.Add($type , 1)
$type = 'conformset'
$hash.$type ++
$hash.$type ++
$hash
Write-Host
$hash = $hash.GetEnumerator() | Sort-Object -property:Value
$hash
I see the contents of the hash twice, unsorted and then sorted.
However, when using a class it does nothing.
class Test {
# Constructor (abstract class)
Test () {
$hash = [hashtable]::New()
$type = 'conformset'
$hash.Add($type, 1)
$type = 'applyset'
$hash.Add($type , 1)
$type = 'conformset'
$hash.$type ++
$hash.$type ++
$hash
Write-Host
$hash = $hash.GetEnumerator() | Sort-Object -property:Value
$hash
}
}
[Test]::New()
This just echos Test to the console, with nothing related to the hash table. My assumption here is that it relates somehow to how the pipeline is interrupted, which lets be honest, is a great reason to move to classes, given how common polluted pipeline errors are. So, moving to a loop based approach, this fails to show the second, sorted, sorted hash table in a class or not.
$hash = [hashtable]::New()
$type = 'conformset'
$hash.Add($type, 1)
$type = 'applyset'
$hash.Add($type , 1)
$type = 'conformset'
$hash.$type ++
$hash.$type ++
foreach ($key in $hash.Keys) {
Write-Host "$key $($hash.$key)!"
}
Write-Host
$hash = ($hash.GetEnumerator() | Sort-Object -property:Value)
foreach ($key in $hash.Keys) {
Write-Host "$key $($hash.$key)!!"
}
But, very weirdly, this shows only the first loop based output, but BOTH of the direct dumps.
$hash = [hashtable]::New()
$type = 'conformset'
$hash.Add($type, 1)
$type = 'applyset'
$hash.Add($type , 1)
$type = 'conformset'
$hash.$type ++
$hash.$type ++
foreach ($key in $hash.Keys) {
Write-Host "$key $($hash.$key)!"
}
$hash
Write-Host
$hash = ($hash.GetEnumerator() | Sort-Object -property:Value)
foreach ($key in $hash.Keys) {
Write-Host "$key $($hash.$key)!!"
}
$hash
The output now is
conformset 3!
applyset 1!
Name Value
---- -----
conformset 3
applyset 1
applyset 1
conformset 3
So obviously $hash is being sorted. But the loop won't show it? Huh? Is this buggy behavior, or intended behavior I just don't understand the reason for, and therefor the way around?
Vasil Svilenov Nikolov's helpful answer explains the fundamental problem with your approach:
You fundamentally cannot sort a hash table ([hashtable] instance) by its keys: the ordering of keys in a hash table is not guaranteed and cannot be changed.
What $hash = $hash.GetEnumerator() | Sort-Object -property:Value does is to instead create an array of [System.Collections.DictionaryEntry] instances; the resulting array has no .Keys property, so your second foreach ($key in $hash.Keys) loop is never entered.
An unrelated problem is that you generally cannot implicitly write to the output stream from PowerShell classes:
Writing to the output stream from a class method requires explicit use of return; similarly, errors must be reported via Throw statements.
In your case, the code is in a constructor for class Test, and constructors implicitly return the newly constructed instance - you're not allowed to return anything from them.
To solve your problem, you need a specialized data type that combines the features of a hash table with maintaining the entry keys in sort order.[1]
.NET type System.Collections.SortedList provides this functionality (there's also a generic version, as Lee Dailey notes):
You can use that type to begin with:
# Create a SortedList instance, which will maintain
# the keys in sorted order, as entries are being added.
$sortedHash = [System.Collections.SortedList]::new()
$type = 'conformset'
$sortedHash.Add($type, 1) # Or: $sortedHash[$type] = 1 or: $sortedHash.$type = 1
$type = 'applyset'
$sortedHash.Add($type , 1)
$type = 'conformset'
$sortedHash.$type++
$sortedHash.$type++
Or even convert from (and to) an existing hash table:
# Construct the hash table as before...
$hash = [hashtable]::new() # Or: $hash = #{}
$type = 'conformset'
$hash.Add($type, 1)
$type = 'applyset'
$hash.Add($type , 1)
$type = 'conformset'
$hash.$type++
$hash.$type++
# ... and then convert it to a SortedList instance with sorted keys.
$hash = [System.Collections.SortedList] $hash
[1] Note that this is different from an ordered dictionary, which PowerShell offers with literal syntax [ordered] #{ ... }: an ordered dictionary maintains the keys in the order in which they are inserted, not based on sorting. Ordered dictionaries are of type System.Collections.Specialized.OrderedDictionary
When you do $hash = $hash.GetEnumerator() | Sort-Object -property:Value , you are re-assigning the hashtable in to an array, try $hash.GetType() , and that will of course behave differently than a hashtable, you can check out the methods etc. Get-Member -InputObject $hash
I dont think you can sort a hashtable, and you do not need to. You might instead try Ordered Dictionary $hash = [Ordered]#{}
Ordered dictionaries differ from hash tables in that the keys always
appear in the order in which you list them. The order of keys in a
hash table is not determined.
One of the best use of a hashtable that I like is the speed of search.
For example, you can instantly get the value of a name in the hashtable the following way $hash['applyset']
If you want to know more about how hashtable work, and how/when to use it, I think this article is a good start :
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_hash_tables?view=powershell-6

Casting Object to String Array Powershell

I want to create an array of strings instead of a variable object so that I can use the "contains" keyword on each index of the array.
$myArray = Get-ADDomain
The above creates an object, which is not what I want. I also tried
[string[]] $myArray = Get-ADDomain
But after that, $myArray only contains one string and it is the first non-empty property of Get-ADDomain, in my case "ComputersContainer". What should I do to receive an array of strings where each string is a different property, such as
$myArray[0] = "AllowedDNSSuffixes = {}"
PowerShell will always return objects by design of course, and specifying that [string[]], does not really change that.
For what you are trying to use, you have to force the array creation. The below is just one way, but I am sure others will have more elegant ways of doing this as well. Though I am curious why one would want to do this, this way. But, hey, that's just me.
# Create an empty array
$DomainData = #()
# Get all the data points for the utilized cmdlet, split on a common delimiter for the array
[string[]]$DomainData = (Get-ADDomain | Select *) -split ';'
# Display the array count
$DomainData.Count
34
# validate getting a value from the array by using an index number
$Item = $DomainData[17]
NetBIOSName=CONTOSO
[array]::IndexOf($DomainData, $Item)
17
# Use that element number to validate the use of the contains comparison operator
0..($DomainData.Count - 1) | %{ If($DomainData[$_] -contains $item){"Index key is $_ contains a value of $Item"} }
Index key is 17 contains a value of NetBIOSName=CONTOSO
# Use the previous with a partial string for a comparison, -contains cannot be used, like or match has to be used
# From the documentation:
# -Contains
# Description: Containment operator. Tells whether a collection of reference values includes a single test value.
$Item = '*domain*'
0..($DomainData.Count - 1) | %{ If($DomainData[$_] -like $item){"Index key is $_ like a value of $Item"} }
Index key is 1 like a value of *domain*
Index key is 6 like a value of *domain*
Index key is 7 like a value of *domain*
Index key is 8 like a value of *domain*
Index key is 18 like a value of *domain*
Index key is 20 like a value of *domain*
You cannot cast a PSObject directly to a string array like that.
However, this can be accomplished rather easily.
To get an array of string from the object
$myArray = Get-ADDomain
# You can use a standard array #() but these tends to be slower for bigger amount of data
$outArray = New-Object -TypeName System.Collections.Generic.List[String]
#To add just the value
$myArray.psobject.properties | Foreach { $outArray.Add($_.Value) }
# To add Name = {Value} instead
$myArray.psobject.properties | Foreach { $outArray.Add("$($_.Name) = {$($_.Value)}") }
Using an hasthable instead:
$myArray = Get-ADDomain
$hashtable = #{}
$myArray.psobject.properties | Foreach { $hashtable[$_.Name] = $_.Value }
# If you need to do something with the key
Foreach ($key in $hashtable.Keys) {
$Value = $hashtable[$key]
if ($value -like '*prod*') {
Write-Host $key
}
}

Unable to remove item from hash table

In Powershell, I have a hash table that contains data similar to this -
Name Value
---- -----
1-update.bat 1
2-update.bat 2
3-update.bat 3
3.1-update.bat 3.1
4-update.bat 4
I also have an variable that contians a number, for example 3
What I would like to do is loop through the array and remove any entry where the value is less than or equal to 3
I'm thinking that this will be easy, especially as the docs say that has tables contain a .remove method. However, the code I have below fails, yeilding this error -
Exception calling "Remove" with "1" argument(s): "Collection was of a
fixed size."
Here is the code that I used -
$versions = #{}
$updateFiles | ForEach-Object {
$versions.Add($_.name, [decimal]($_.name -split '-')[0])
}
[decimal]$lastUpdate = Get-Content $directory\$updatesFile
$versions | ForEach-Object {
if ( $_.Value -le $lastUpdate ) {
$versions.Remove($version.Name)
}
}
I first tried to loop $versions in a different manner, trying both the foreach and for approaches, but both failed in the same manner.
I also tried to create a temporary array to hold the name of the versions to remove, and then looping that array to remove them, but that also failed.
Next I hit Google, and while I can find several similar questions, none that answer my specific question. Mostly they suggest using a list (New-Object System.Collections.Generic.List[System.Object]), whcih from what I can tell is of no help to me here.
Is anyone able to suggest a fix?
Here you go, you can use .Remove(), you just need a clone of the hashtable so that it will let you remove items as you enumerate.
[hashtable]$ht = #{ '1-update.bat'=1;'2-update.bat'=2;'3-update.bat'=3;'3.1-update.bat'=3.1; '4-update.bat'=4 }
#Clone so that we can remove items as we're enumerating
$ht2 = $ht.Clone()
foreach($k in $ht.GetEnumerator()){
if([decimal]$k.Value -le 3){
#notice, deleting from clone, then return clone at the end
$ht2.Remove($k.Key)
}
}
$ht2
Notice I've cast the original variable too so that it's explicitly a hash table, may not be required, but I like to do it to at least keep things clear.
It looks like you just confused ForEach-Object with foreach but only halfway (maybe it was foreach before and you converted it).
You can't send a [hashtable] directly to ForEach-Object; the $_ in that case will just refer to the single [hashtable] you sent in. You can do:
foreach ($version in $versions.GetEnumerator()) {
$version.Name
$version.Value
}
or you can do something like this:
$versions.Keys | ForEach-Object {
$_ # the name/key
$versions[$_] # the value
$versions.$_ # also the value
}
$ht.Keys # list all keys
$ht[$_] # take an element of hastable
$ht.Remove($_) # remove an element of hastable by his key
what you want:
$ht.Keys | ? { $ht[$_] -le 3 } | %{$ht.Remove($_) }
You need to create a temporary array to hold the name/key of the versions to remove, and then looping that array to remove them from hash table:
$versionKeysToRemove = $versions.Keys | Where-Object { $versions[$_] -le $lastUpdate }
$versionKeysToRemove | ForEach-Object { $versions.Remove($_) }
Or shorter:
($versions.Keys | ? { $versions[$_] -le $lastUpdate }) | % { $versions.Remove($_) }
Please note the parentheses.

Compare objects based on subset of properties

Say I have 2 powershell hashtables one big and one small and, for a specific purpose I want to say they are equal if for the keys in the small one, the keys on the big hastable are the same.
Also I don't know the names of the keys in advance. I can use the following function that uses Invoke-Expression but I am looking for nicer solutions, that don't rely on this.
Function Compare-Subset {
Param(
[hashtable] $big,
[hashtable] $small
)
$keys = $small.keys
Foreach($k in $keys) {
$expression = '$val = $big.' + "$k" + ' -eq ' + '$small.' + "$k"
Invoke-Expression $expression
If(-not $val) {return $False}
}
return $True
}
$big = #{name='Jon'; car='Honda'; age='30'}
$small = #{name = 'Jon'; car='Honda'}
Compare-Subset $big $small
A simple $true/$false can easily be gotten. This will return $true if there are no differences:
[string]::IsNullOrWhiteSpace($($small|Select -Expand Keys|Where{$Small[$_] -ne $big[$_]}))
It checks for all keys in $small to see if the value of that key in $small is the same of the value for that key in $big. It will only output any values that are different. It's wrapped in a IsNullOrWhitespace() method from the [String] type, so if any differences are found it returns false. If you want to list differences just remove that method.
This could be the start of something. Not sure what output you are looking for but this will output the differences between the two groups. Using the same sample data that you provided:
$results = Compare-Object ($big.GetEnumerator() | % { $_.Name }) ($small.GetEnumerator() | % { $_.Name })
$results | ForEach-Object{
$key = $_.InputObject
Switch($_.SideIndicator){
"<="{"Only reference object has the key: '$key'"}
"=>"{"Only difference object has the key: '$key'"}
}
}
In primetime you would want something different but just to show you the above would yield the following output:
Only reference object has the key: 'age'