How to remove item from an array in PowerShell (v5.1) - powershell

I have an array of elements in PowerShell (v5.1) as below.
I want to remove the item with matching condition EmpId=102
$testData = #(
#{ Name='ABC'; EmpId=100 },
#{ Name='EFG'; EmpId=101 },
#{ Name='XYZ'; EmpId=102 }
);
Is there any simple way to achieve this?
maybe something like $testData.Remove($testData.Find("EmpId == 102")) ?

if your array is really a collection of one-item hashtables, then the following otta work ... [grin] it pipes the collection thru the Where-Object cmdlet & filters out anything item that has the value-to-exclude in the EmpId key value.
$TestData = #(
#{ Name='ABC'; EmpId=100 }
#{ Name='EFG'; EmpId=101 }
#{ Name='XYZ'; EmpId=102 }
)
$EmpId_ToExclude = 102
$TestData |
Where-Object {
$_['EmpId'] -ne $EmpId_ToExclude
}
output ...
Name Value
---- -----
EmpId 100
Name ABC
EmpId 101
Name EFG
note that the items are not required to be in order ... that is how hashtables work if you don't specify [ordered].

I hope this info is some use. Maybe you can use an arraylist. You don't need #( ) to make an array.
[collections.arraylist]$testData = #{ Name='ABC'; EmpId=100 },
#{ Name='EFG'; EmpId=101 },
#{ Name='XYZ'; EmpId=102 }
$a = $testdata[1]
$testdata.remove($a)
Or
$testdata.removeat(1)
Unfortunely, this doesn't work. I guess you would need a pointer to the hashtable:
$testdata.remove(#{ Name='XYZ'; EmpId=102 })
Actually, it would be easy with a single hashtable:
$testData = #{ 'ABC'=100
'EFG'=101
'XYZ'=102 }
$testdata.remove('xyz')
$testdata
Name Value
---- -----
ABC 100
EFG 101

Related

How do I convert a powershell hashtable to an object?

Some hashtables in PowerShell, such as those imported with Import-PowerShellDataFile, would be much easier to navigate if being a PSCustomObject instead.
#{
AllNodes = #(
#{
NodeName = 'SRV1'
Role = 'Application'
RunCentralAdmin = $true
},
#{
NodeName = 'SRV2'
Role = 'DistributedCache'
RunCentralAdmin = $true
},
#{
NodeName = 'SRV3'
Role = 'WebFrontEnd'
PSDscAllowDomainUser = $true
PSDscAllowPlainTextPassword = $true
CertificateFolder = '\\mediasrv\Media'
},
#{
NodeName = 'SRV4'
Role = 'Search'
},
#{
NodeName = '*'
DatabaseServer = 'sql1'
FarmConfigDatabaseName = '__FarmConfig'
FarmContentDatabaseName = '__FarmContent'
CentralAdministrationPort = 1234
RunCentralAdmin = $false
}
);
NonNodeData = #{
Comment = 'No comment'
}
}
When imported it will become a hashtable of hashtables
$psdnode = Import-PowerShellDataFile .\nodefile.psd1
$psdnode
Name Value
---- -----
AllNodes {System.Collections.Hashtable, System.Collect...
NonNodeData {Comment}
$psdnode.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Hashtable System.Object
and the data structure will be just weird when navigating by property name.
There's good information in the existing answers, but given your question's generic title, let me try a systematic overview:
You do not need to convert a hashtable to a [pscustomobject] instance in order to use dot notation to drill down into its entries (properties), as discussed in the comments and demonstrated in iRon's answer.
A simple example:
#{ top = #{ nested = 'foo' } }.top.nested # -> 'foo'
See this answer for more information.
In fact, when possible, use of hashtables is preferable to [pscustomobject]s, because:
they are lighter-weight than [pscustomobject] instances (use less memory)
it is easier to construct them iteratively and modify them.
Note:
The above doesn't just apply to the [hashtable] type, but more generally to instances of types that implement the [System.Collections.IDictionary] interface or its generic counterpart, System.Collections.Generic.IDictionary[TKey, TValue]], notably including ordered hashtables (which are instances of type System.String, which PowerShell allows you to construct with syntactic sugar [ordered] #{ ... }).
Unless noted, hashtable in the following section refers to all such types.
In cases where you do need to convert a [hasthable] to a [pscustomobject]:
While many standard cmdlets accept [hasthable]s interchangeably with [pscustomobjects]s, some do not, notably ConvertTo-Csv and Export-Csv (see GitHub issue #10999 for a feature request to change that); in such cases, conversion to [pscustomobject] is a must.
Caveat: Hasthables can have keys of any type, whereas conversion to [pscustomobject] invariably requires using string "keys", i.e. property names. Thus, not all hashtables can be faithfully or meaningfully converted to [pscustomobject]s.
Converting non-nested hashtables to [pscustomobject]:
The syntactic sugar PowerShell offers for [pscustomobject] literals (e.g., [pscustomobject] #{ foo = 'bar'; baz = 42 }) also works via preexisting hash; e.g.:
$hash = #{ foo = 'bar'; baz = 42 }
$custObj = [pscustomobject] $hash # Simply cast to [pscustomobject]
Converting nested hashtables, i.e. an object graph, to a [pscustomobject] graph:
A simple, though limited and potentially expensive solution is the one shown in your own answer: Convert the hashtable to JSON with ConvertTo-Json, then reconvert the resulting JSON into a [pscustomobject] graph with ConvertFrom-Json.
Performance aside, the fundamental limitation of this approach is that type fidelity may be lost, given that JSON supports only a few data types. While not a concern with a hashtable read via Import-PowerShellDataFile, a given hashtable may contain instances of types that have no meaningful representation in JSON.
You can overcome this limitation with a custom conversion function, ConvertFrom-HashTable (source code below); e.g. (inspect the result with Format-Custom -InputObject $custObj):
$hash = #{ foo = 'bar'; baz = #{ quux = 42 } } # nested hashtable
$custObj = $hash | ConvertFrom-HashTable # convert to [pscustomobject] graph
ConvertFrom-HashTable source code:
Note: Despite the name, the function generally supports instance of types that implement IDictionary as input.
function ConvertFrom-HashTable {
param(
[Parameter(Mandatory, ValueFromPipeline)]
[System.Collections.IDictionary] $HashTable
)
process {
$oht = [ordered] #{} # Aux. ordered hashtable for collecting property values.
foreach ($entry in $HashTable.GetEnumerator()) {
if ($entry.Value -is [System.Collections.IDictionary]) { # Nested dictionary? Recurse.
$oht[$entry.Key] = ConvertFrom-HashTable -HashTable $entry.Value
} else { # Copy value as-is.
$oht[$entry.Key] = $entry.Value
}
}
[pscustomobject] $oht # Convert to [pscustomobject] and output.
}
}
What is the issue/question?
#'
#{
AllNodes = #(
#{
NodeName = 'SRV1'
Role = 'Application'
RunCentralAdmin = $true
},
#{
NodeName = 'SRV2'
Role = 'DistributedCache'
RunCentralAdmin = $true
},
#{
NodeName = 'SRV3'
Role = 'WebFrontEnd'
PSDscAllowDomainUser = $true
PSDscAllowPlainTextPassword = $true
CertificateFolder = '\\mediasrv\Media'
},
#{
NodeName = 'SRV4'
Role = 'Search'
},
#{
NodeName = '*'
DatabaseServer = 'sql1'
FarmConfigDatabaseName = '__FarmConfig'
FarmContentDatabaseName = '__FarmContent'
CentralAdministrationPort = 1234
RunCentralAdmin = $false
}
);
NonNodeData = #{
Comment = 'No comment'
}
}
'# |Set-Content .\nodes.psd1
$psdnode = Import-PowerShellDataFile .\nodefile.psd1
$psdnode
Name Value
---- -----
NonNodeData {Comment}
AllNodes {SRV1, SRV2, SRV3, SRV4…}
$psdnode.AllNodes.where{ $_.NodeName -eq 'SRV3' }.Role
WebFrontEnd
A very simple way, that I discovered just yesterday, is to do a "double-convert" over JSON.
$nodes = Import-PowerShellDataFile .\nodes.psd1 | ConvertTo-Json | ConvertFrom-Json
$nodes
AllNodes
--------
{#{NodeName=SRV1; RunCentralAdmin=True; Role=Application}, #{NodeName=SRV2; RunCentralAdm...}
$nodes.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True False PSCustomObject System.Object

Generated string names in Calculated Properties aren't accepted by select cmdlet

I want to generate the following table:
AAA BBB CCC
--- --- ---
10 10 10
10 10 10
10 10 10
10 10 10
10 10 10
So I write the following code using a foreach loop to generate the column names:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = $_; expression = { 10 } }
}
#(1..5) | select -Property $property
But I get the following error saying the name is not a string:
select : The "name" key has a type, System.Management.Automation.PSObject, that is not valid; expected type is System.String.
At line:4 char:11
+ #(1..5) | select -Property $property
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Select-Object], NotSupportedException
+ FullyQualifiedErrorId : DictionaryKeyIllegalValue2,Microsoft.PowerShell.Commands.SelectObjectCommand
To get the code work, I have to convert the $_ to string like below:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = [string]$_; expression = { 10 } }
}
#(1..5) | select -Property $property
Or like below:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = $_; expression = { 10 } }
}
$property | foreach { $_.name = [string]$_.name }
#(1..5) | select -Property $property
The question is: the $_ is already a string. Why do I have to convert it to string again? And why select thinks that the name is PSObject?
To confirm that it's already a string, I write the following code to print the type of name:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = $_; expression = { 10 } }
}
$property | foreach { $_.name.GetType() }
The following result confirms that it's already a string:
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
True True String System.Object
True True String System.Object
I know that there are many other easier ways to generate the table. But I want to understand why I have to convert a string to string to make the code work, and why select doesn't think that the string is a string. For what it's worth, my $PSVersionTable.PSVersion is:
Major Minor Build Revision
----- ----- ----- --------
5 1 18362 1474
You're seeing the unfortunate effects of incidental, normally invisible [psobject] wrappers PowerShell uses behind the scenes.
In your case, because the input strings are supplied via the pipeline, they get wrapped in and stored as [psobject] instances in your hashtables, which is the cause of the problem.
The workaround - which is neither obvious nor should it be necessary - is to discard the wrapper by accessing .psobject.BaseObject:
$property = 'AAA', 'BBB', 'CCC' | ForEach-Object {
#{ name = $_.psobject.BaseObject; expression = { 10 } }
}
1..5 | select -Property $property
Note:
In your case, a simpler alternative to .psobject.BaseObject (see the conceptual about_Intrinsic Members help topic) would have been to call .ToString(), given that you want a string.
To test a given value / variable for the presence of such a wrapper, use -is [psobject]; with your original code, the following yields $true, for instance:
$property[0].name -is [psobject]
Note, however, that this test is meaningless for [pscustomobject] instances, where it is always $true (custom objects are in essence [psobject] instances without a .NET base objects - they only have dynamic properties).
That the normally invisible [psobject] wrappers situationally, obscurely result in behavioral differences is arguably a bug and the subject of GitHub issue #5579.
Simpler and faster alternative, using the .ForEach() array method:
$property = ('AAA', 'BBB', 'CCC').ForEach({
#{ name = $_; expression = { 10 } }
})
1..5 | select -Property $property
Unlike the pipeline, the .ForEach() method does not wrap $_ in [psobject], so the problem doesn't arise and no workaround is needed.
Using the method is also faster, although note that, unlike the pipeline, it must collect all its input in memory up front (clearly not a problem with an array literal).

Handling multiple CSVs in Powershell efficiently

I am retrieving two CSVs from an API, one called students.csv similar to:
StudentNo,PreferredFirstnames,PreferredSurname,UPN
111, john, smith, john#email.com
222, jane, doe, jane#email.com
one called rooms.csv:
roomName, roomNo, students
room1, 1, {#{StudentNo=111; StudentName=john smith; StartDate=2018-01-01T00:00:00; EndDate=2018-07-06T00:00:00},....
room2, 2,{#{StudentNo=222; StudentName=jane doe; StartDate=2018-01-01T00:00:00; EndDate=2018-07-06T00:00:00},...
The third column in rooms.csv is an array as provided by the API
What is the best way to consolidate the two into
StudentNo,PreferredFirstnames,PreferredSurname,UPN, roomName
111, john, smith, john#email.com, room1
222, jane, doe, jane#email.com, room2
Im thinking something like...
$rooms = Import-Csv rooms.csv
$students = Import-Csv students.csv
$combined = $students | select-object StudentNo,PreferredSurname,PreferredFirstnames,UPN,
#{Name="roomName";Expression={ ForEach ($r in $rooms) {
if ($r.Students.StudentNo.Contains($_.StudentNo) -eq "True")
{return $r.roomName}}}}
This works, but is the foreach the right way to go am i mixing things up or is there a more efficient way???
--- Original Post ---
With all of this information I need to compare the student data and update AzureAD and then compile a list of data including first name, last name, upn, room and others that are retrieved from AzureAD.
My issue is "efficiency". I have code that mostly works but it takes hours to run. Currently I am looping through students.csv and then for each student looping through rooms.csv to find the room they're in, and obviously waiting for multiple api calls in-between all this.
What is the most efficient way to find the room for each student? Is importing the CSV as a custom PSObject comparable to using hash tables?
I was able to get your proposed code to work but it requires some tweaks to the code and data:
There must be some additional step where you are deserializing the students column of rooms.csv to a collection of objects. It appears to be a ScriptBlock that evaluates to an array of HashTables, but some changes to the CSV input are still needed:
The StartDate and EndDate properties need to be quoted and cast to [DateTime].
At least for rooms that contain multiple students, the value must be quoted so Import-Csv doesn't interpret the , separating array elements as an additional column.
The downside of using CSV as an intermediate format is the original property types are lost; everything becomes a [String] upon import. Sometimes it's desirable to cast back to the original type for efficiency purposes, and sometimes it's absolutely necessary in order for certain operations to work. You could cast those properties every time you use them, but I prefer to cast them once immediately after import.
With those changes rooms.csv becomes...
roomName, roomNo, students
room1, 1, "{#{StudentNo=111; StudentName='john smith'; StartDate=[DateTime] '2018-01-01T00:00:00'; EndDate=[DateTime] '2018-07-06T00:00:00'}}"
room2, 2, "{#{StudentNo=222; StudentName='jane doe'; StartDate=[DateTime] '2018-01-01T00:00:00'; EndDate=[DateTime] '2018-07-06T00:00:00'}}"
...and the script becomes...
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock)
return $studentsArray
}
}
# Replace the [String] property "StudentNo" with an [Int32] property of the same name
$students = Import-Csv students.csv `
| Select-Object `
-ExcludeProperty 'StudentNo' `
-Property '*', #{
Name = 'StudentNo'
Expression = { [Int32] $_.StudentNo }
}
$combined = $students `
| Select-Object -Property `
'StudentNo', `
'PreferredSurname', `
'PreferredFirstnames', `
'UPN', `
#{
Name = "roomName";
Expression = {
foreach ($r in $rooms)
{
if ($r.Students.StudentNo -contains $_.StudentNo)
{
return $r.roomName
}
}
#TODO: Return text indicating room not found?
}
}
The reason this can be slow is because you are performing a linear search - two of them, in fact - for every student object; first through the collection of rooms (foreach), then through the collection of students in each room (-contains). This can quickly turn into a lot of iterations and equality comparisons because in every room to which the current student is not assigned you are iterating the entire collection of that room's students, on and on until you do find the room for that student.
One easy optimization you can make when performing a linear search is to sort the items you're searching (in this case, the Students property will be ordered by the StudentNo property of each student)...
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock) `
| Sort-Object -Property #{ Expression = { $_.StudentNo } }
return $studentsArray
}
}
...and then when you're searching that same collection if you come across an item that is greater than the one you're searching for you know the remainder of the collection can't possibly contain what you're searching for and you can immediately abort the search...
#{
Name = "roomName";
Expression = {
foreach ($r in $rooms)
{
# Requires $room.Students to be sorted by StudentNo
foreach ($roomStudentNo in $r.Students.StudentNo)
{
if ($roomStudentNo -eq $_.StudentNo)
{
# Return the matched room name and stop searching this and further rooms
return $r.roomName
}
elseif ($roomStudentNo -gt $_.StudentNo)
{
# Stop searching this room
break
}
# $roomStudentNo is less than $_.StudentNo; keep searching this room
}
}
#TODO: Return text indicating room not found?
}
}
Better yet, with a sorted collection you can also perform a binary search, which is faster than a linear search*. The Array class already provides a BinarySearch static method, so we can accomplish this in less code, too...
#{
Name = "roomName";
Expression = {
foreach ($r in $rooms)
{
# Requires $room.Students to be sorted by StudentNo
if ([Array]::BinarySearch($r.Students.StudentNo, $_.StudentNo) -ge 0)
{
return $r.roomName
}
}
#TODO: Return text indicating room not found?
}
}
The way I would approach this problem, however, is to use a [HashTable] mapping a StudentNo to a room. There is a little preprocessing required to build the [HashTable] but this will provide constant-time lookups when retrieving the room for a student.
function GetRoomsByStudentNoTable()
{
$table = #{ }
foreach ($room in $rooms)
{
foreach ($student in $room.Students)
{
#NOTE: It is assumed each student belongs to at most one room
$table[$student.StudentNo] = $room
}
}
return $table
}
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock)
return $studentsArray
}
}
# Replace the [String] property "StudentNo" with an [Int32] property of the same name
$students = Import-Csv students.csv `
| Select-Object `
-ExcludeProperty 'StudentNo' `
-Property '*', #{
Name = 'StudentNo'
Expression = { [Int32] $_.StudentNo }
}
$roomsByStudentNo = GetRoomsByStudentNoTable
$combined = $students `
| Select-Object -Property `
'StudentNo', `
'PreferredSurname', `
'PreferredFirstnames', `
'UPN', `
#{
Name = "roomName";
Expression = {
$room = $roomsByStudentNo[$_.StudentNo]
if ($room -ne $null)
{
return $room.roomName
}
#TODO: Return text indicating room not found?
}
}
You can ameliorate the hit of building $roomsByStudentNo by doing so at the same time as importing rooms.csv...
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock)
return $studentsArray
}
} `
| ForEach-Object -Begin {
$roomsByStudentNo = #{ }
} -Process {
foreach ($student in $_.Students)
{
#NOTE: It is assumed each student belongs to at most one room
$roomsByStudentNo[$student.StudentNo] = $_
}
return $_
}
*Except for on small arrays

Powershell v2.0 substitute null values from a Hash table

I have a hash table as below:
$Hash = #{
Team1=$Team1.count
Team2=$Team2.count
Team3=$Team3.count
}
$GroupByTeam = New-Object psobject -Property $Hash |
Select 'Team1','Team2','Team3' | ConvertTo-Html -Fragment
This is fine and each "team" returns their own value. However, teams may have a null value and I wish to substitute this for "0".
In an attempt to work this out, I have tried to select the null value first but can't seem to do this:
$Hash.values | select -property Values
Values
------
{1, 2}
But
$Hash.values | select -property Values | where {$_.Values is $null}
doesn't pull back anything. Also tried:
$Hash.values | select -expandproperty Values | where {$_.Values is $null}
Any ideas?
thanks
Your best option is to cast the values to int when creating the hashtable:
$Hash = #{
Team1 = [int]$Team1.Count
Team2 = [int]$Team2.Count
Team3 = [int]$Team3.Count
}
If that's not possible for some reason you could go with an enumerator:
($Hash.GetEnumerator()) | ForEach-Object {
if ($_.Value -eq $null) { $Hash[$_.Name] = 0 }
}
or (as Mathias suggested) use the Keys property to the same end:
($Hash.Keys) | ForEach-Object {
if ($Hash[$_] -eq $null) { $Hash[$_] = 0 }
}
Note that either way you need to use a subexpression (or assign the enumerated objects/keys to a variable) otherwise you'll get an error because you're modifying a data structure while it's being enumerated.
What you'll want to do is collect the keys that refer to null values, and then populate those with 0s:
# Create and populate hashtable
$HashTable = #{
Team1 = 123
Team2 = $null
Team3 = 456
}
# Find keys of `$null` values
$nullKeys = $HashTable.Keys |Where-Object { $HashTable[$_] -eq $null }
# Populate appropriate indices with 0
$nullKeys |ForEach-Object { $HashTable[$_] = 0 }

Find matches in two different Powershell objects based on one property

I am trying to find the matching names in two different types of Powershell objects
$Object1 has two properties - Name (string), ResourceID (uint32)
$object2 has one noteproperty - Name (system.string)
This gives me a list of the matching names but I also want the corresponding resourceID property from $object1.
$computers = Compare-Object $Object1.name $WSD_CM12 | where {$_.sideindicator -eq "=>"} | foreach {$_.inputobject}
These are big objects with over 10,000 items so I'm looking for the most efficient way to accomplish this.
If I'm understanding what you're after, I'd start by creating a hash table from your Object1 collection:
$object1_hash = #{}
Foreach ($object1 in $object1_coll)
{ $object1_hash[$object1.Name] = $object1.ResourceID }
Then you can find the ResourceID for any given Object2.name with:
$object1_hash[$Object2.Name]
Test bed for creating hash table:
$object1_coll = $(
New-Object PSObject -Property #{Name = 'Name1';ResourceID = 001}
New-Object PSObject -Property #{Name = 'Name2';ResourceID = 002}
)
$object1_hash = #{}
Foreach ($object1 in $object1_coll)
{ $object1_hash[$object1.Name] = $object1.ResourceID }
$object1_hash
Name Value
---- -----
Name2 2
Name1 1
Alternative method:
# Create sample list of objects with both Name and Serial
$obj1 = New-Object -Type PSCustomObject -Property:#{ Name = "Foo"; Serial = "1234" }
$obj2 = New-Object -Type PSCustomObject -Property:#{ Name = "Cow"; Serial = "4242" }
$collection1 = #($obj1, $obj2)
# Create subset of items with only Name
$objA = New-Object -Type PSCustomObject -Property:#{ Name = "Foo"; }
$collection2 = #($objA)
#Everything above this line is just to make sample data
# replace $collection1 and $collection2 with $Object1, $WSD_CM12
# Combine into one list
($collection1 + $collection2) |
# Group by name property
Group-Object -Property Name |
# I only want items that exist in both
Where { $_.Count -gt 1 } |
# Now give me the object
Select -Expand Group |
# And get the properties
Where { $_.Serial -ne $null }