I want to generate the following table:
AAA BBB CCC
--- --- ---
10 10 10
10 10 10
10 10 10
10 10 10
10 10 10
So I write the following code using a foreach loop to generate the column names:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = $_; expression = { 10 } }
}
#(1..5) | select -Property $property
But I get the following error saying the name is not a string:
select : The "name" key has a type, System.Management.Automation.PSObject, that is not valid; expected type is System.String.
At line:4 char:11
+ #(1..5) | select -Property $property
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [Select-Object], NotSupportedException
+ FullyQualifiedErrorId : DictionaryKeyIllegalValue2,Microsoft.PowerShell.Commands.SelectObjectCommand
To get the code work, I have to convert the $_ to string like below:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = [string]$_; expression = { 10 } }
}
#(1..5) | select -Property $property
Or like below:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = $_; expression = { 10 } }
}
$property | foreach { $_.name = [string]$_.name }
#(1..5) | select -Property $property
The question is: the $_ is already a string. Why do I have to convert it to string again? And why select thinks that the name is PSObject?
To confirm that it's already a string, I write the following code to print the type of name:
$property = #('AAA', 'BBB', 'CCC') | foreach {
#{ name = $_; expression = { 10 } }
}
$property | foreach { $_.name.GetType() }
The following result confirms that it's already a string:
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True String System.Object
True True String System.Object
True True String System.Object
I know that there are many other easier ways to generate the table. But I want to understand why I have to convert a string to string to make the code work, and why select doesn't think that the string is a string. For what it's worth, my $PSVersionTable.PSVersion is:
Major Minor Build Revision
----- ----- ----- --------
5 1 18362 1474
You're seeing the unfortunate effects of incidental, normally invisible [psobject] wrappers PowerShell uses behind the scenes.
In your case, because the input strings are supplied via the pipeline, they get wrapped in and stored as [psobject] instances in your hashtables, which is the cause of the problem.
The workaround - which is neither obvious nor should it be necessary - is to discard the wrapper by accessing .psobject.BaseObject:
$property = 'AAA', 'BBB', 'CCC' | ForEach-Object {
#{ name = $_.psobject.BaseObject; expression = { 10 } }
}
1..5 | select -Property $property
Note:
In your case, a simpler alternative to .psobject.BaseObject (see the conceptual about_Intrinsic Members help topic) would have been to call .ToString(), given that you want a string.
To test a given value / variable for the presence of such a wrapper, use -is [psobject]; with your original code, the following yields $true, for instance:
$property[0].name -is [psobject]
Note, however, that this test is meaningless for [pscustomobject] instances, where it is always $true (custom objects are in essence [psobject] instances without a .NET base objects - they only have dynamic properties).
That the normally invisible [psobject] wrappers situationally, obscurely result in behavioral differences is arguably a bug and the subject of GitHub issue #5579.
Simpler and faster alternative, using the .ForEach() array method:
$property = ('AAA', 'BBB', 'CCC').ForEach({
#{ name = $_; expression = { 10 } }
})
1..5 | select -Property $property
Unlike the pipeline, the .ForEach() method does not wrap $_ in [psobject], so the problem doesn't arise and no workaround is needed.
Using the method is also faster, although note that, unlike the pipeline, it must collect all its input in memory up front (clearly not a problem with an array literal).
Related
When we're trying to export data to other functions via the pipeline, we observe some strange behavior in PowerShell.
Example code:
$Array = #()
$Obj1 = [PSCustomObject]#{
Member1 = 'First'
Member2 = 'Second'
}
$Obj2 = [PSCustomObject]#{
Member1 = 'First'
Member2 = 'Second'
Member3 = 'Third'
}
$Array = $Obj1, $Obj2
$Array | Out-GridView -Title 'Not showing Member3'
$Array = $Obj2, $Obj1
$Array | Out-GridView -Title 'All members correctly displayed'
In the example above you can see that when the first object only contains 2 properties, the Out-GridView CmdLet (and others) only show 2 properties, even though the second object has 3 properties. However, when the first object in the array has 3 properties it does display them all correctly.
Is there a way around this? Because it's not possible to predict up front how many properties on an object there will be and if the object with the most properties will be the first one in the array.
I had the same experience once and created the following reusable 'Union' function:
# 2021-08-25 Removed Union function
Usage:
$Obj1, $Obj2 | Union | Out-GridView -Title 'Showing all members'
It is also supposed to work with complex objects. Some standard cmdlets output multiple object types at once and if you view them (e.g. Out-GridView) or dump them in a file (e.g. Export-Csv) you might miss a lot of properties. Take as another example:
Get-WmiObject -Namespace root/hp/instrumentedBIOS -Class hp_biosSetting | Union | Export-Csv ".\HPBIOS.csv"
Added 2014-09-19:
Maybe this is already between the lines in the comments $Array | Select * | … will not resolve the issue but specifically selecting the properties $Array | Select Member1, Member2, Member3 | … does.
Besides, although in most cases the Union function will work, there are some exceptions to that as it will only align the first object with the rest.
Consider the following object:
$List = #(
New-Object PSObject -Property #{Id = 2}
New-Object PSObject -Property #{Id = 1}
New-Object PSObject -Property #{Id = 3; Name = "Test"}
)
If you Union this object everything appears to be fine and if you e.g. ExportTo-CSV and work with the export .csv file from then on you will never have any issue.
$List | Union
Id Name
-- ----
2
1
3 Test
Still there is a catch as only the first object is aligned. If you e.g. sort the result on Id (Sort Id) or take just the last 2 (Select -Last 2) entries, the Name is not listed because the second object doesn’t contain the Name property:
$List | Union | Sort Id
Id
--
1
2
3
Therefor I have rewritten the Union-Object (Alias Union) function`):
Union-Object
# 2021-08-25 Removed Union-Object function
Syntax:
$Array | Union | Out-GridView -Title 'All members correctly displayed'
Update 2021-08-25
Based on az1d helpful feedback on an error caused by equal property names with different casing, I have created a new UnifyProperties function.
(I will no longer use the name UnionObject for his)
function UnifyProperties {
$Names = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase)
$InputCollected = #($Input)
$InputCollected.ForEach({
foreach ($Name in $_.psobject.Properties.Name) { $Null = $Names.Add($Name) }
})
$inputCollected | Select-Object #($Names)
}
Usage:
[pscustomobject] #{ one = 1; two = 2; three = 3 },
[pscustomobject] #{ ONE = 10; THREE = 30; FOUR = 4 } |
UnifyProperties
one two three FOUR
--- --- ----- ----
1 2 3
10 30 4
See also: #13906 Add -UnifyProperties parameter to Select-Object
The Goal
See if $SP.ip is in $NLIP.IpRanges and if it is, add $NLIP.IpRanges and $NLIP.DisplayName to the $SP array or all into a new array.
The Arrays
Array 1 is $SP, it's a CSV import and has the properties 'name' and 'ip', it looks like this:
name: bob
ip: 1.9.8.2
Array 2 is $NLIP and has the relevant properties 'IpRanges' and 'DisplayName'. It's fetched from: $NLIP = Get-AzureADMSNamedLocationPolicy | where-object {$_.OdataType -eq "#microsoft.graph.ipNamedLocation"}, it looks like this:
DisplayName : Named Location 1
IpRanges : {class IpRange {
CidrAddress: 16.29.28.9/28 #fictitious CIDR
}
, class IpRange {
CidrAddress: 1.9.8.3/28 #fictitious CIDR
}
}
The Code / the problem
I'm using IPInRange.ps1 function from https://github.com/omniomi/PSMailTools to find if the IP is in the range. It works like so:
> IPInRange 1.9.8.2 1.9.8.3/28
True
I also worked out that $NLTP.IpRanges.split() | Where-Object ($_ -like "*/*"} can return all the ranges, but $NLIP | Where-Object {$_.IpRanges.split() -like "*/*"} doesn't. I would naturally use the second to keep the variable in the pipe to return the DisplayName. So I'm struggling on how to pull the individual ranges out in such a way that I can then add the 'IpRange' and 'DisplayName' to an array.
Also, maybe it's because I haven't worked out the above issue, but I'm struggling to think how I would iterate through both arrays and combine them into one. I know I would probably enter into a foreach ($item in $SP) and create a temporary array, but after that it's getting hazy.
The result
What I'm hoping to have in the end is:
name: bob
ip: 1.9.8.2
IpRange: 1.9.8.3/28 #fictitious CIDR
DisplayName: Named Location 1
thanks in advance.
I believe this will work for you if I understood the NLIP construct correctly.
We will loop through all the SP objects and see if we can find any NLIP that match the IP range using the IPinRange function you linked. We will then add the 2 properties you want to the SP object if matched and finally pass thru to the pipeline or you can append | export-csv -path YourPath to the end if you would like to send to a csv file
$SP | ForEach-Object {
$target = $_
$matched = $NLIP | ForEach-Object {
$item = $_
# Using where to single out matching range using IPinRange function
$_.IpRanges.Where({ IPInRange -IPAddress $target.ip -Range $_.CidrAddress }) |
ForEach-Object {
# for matching range output custom object containing the displayname and iprange
[PSCustomObject]#{
DisplayName = $item.DisplayName
IpRange = $_.CidrAddress
}
}
}
# add the 2 properties (DisplayName and IpRange) from the match to the original $SP
# object and then pass thru
$target | Add-Member -NotePropertyName DisplayName -NotePropertyValue $matched.DisplayName
$target | Add-Member -NotePropertyName IpRange -NotePropertyValue $matched.IpRange -PassThru
}
By the way, this is how I envisioned the NLIP objects and what I tested with
$NLIP = #(
[pscustomobject]#{
DisplayName = 'Named location 1'
IpRanges = #(
[pscustomobject]#{
CidrAddress = '16.29.28.9/28'
},
[pscustomobject]#{
CidrAddress = '1.9.8.3/28'
}
)
},
[pscustomobject]#{
DisplayName = 'Named location 2'
IpRanges = #(
[pscustomobject]#{
CidrAddress = '16.29.28.25/28'
},
[pscustomobject]#{
CidrAddress = '1.9.8.25/28'
}
)
}
)
Let's to shed some lights in the hazy darkness by first creating a Minimal, Reproducible Example (mcve):
$SP = ConvertFrom-Csv #'
IP, Name
1.9.8.2, BOB
10.10.10.10, Apple
16.29.28.27, Pear
16.30.29.28, Banana
'#
$NLIP = ConvertFrom-Csv #'
IPRange, SubNet
16.29.28.9/28, NetA
1.9.8.3/28, NetB
'#
To tackle this, you need two loops where the second loop is inside the first loop. For the outer loop you might use the ForEach-Object cmdlet which lets you stream each object and with that actually use less memory (assuming that you import the data from a file and eventually export it to a new file). Within the inner loop you might than cross link each IP address with the IPRange using the function you refer to and in case the condition is true create a new PSCustomObject:
$SP |ForEach-Object { # | Import-Csv .\SP.csv |ForEach-Object { ...
ForEach($SubNet in $NLIP) {
if (IPInRange $_.IP $SubNet.IPRange) {
[PSCustomObject]#{
IP = $_.IP
Name = $_.Name
IPRange = $SubNet.IPRange
SubNet = $SubNet.SubNet
}
}
}
} # | Export-Csv .\Output.csv
Which results in:
IP Name IPRange SubNet
-- ---- ------- ------
1.9.8.2 BOB 1.9.8.3/28 NetB
16.29.28.27 Pear 16.29.28.9/8 NetA
16.30.29.28 Banana 16.29.28.9/8 NetA
But as you are considering 3rd party scripts anyways, you might as well use this Join-Object script/Join-Object Module (see also: In Powershell, what's the best way to join two tables into one?):
$SP |Join $NLIP -Using { IPInRange $Left.IP $Right.IPRange }
Which gives the same results.
I have two API endpoints to hit and retrieve user info and write the data into a SQL table. The endpoint uses employee ID in the URL so I am looping through each user to grab their data. Endpoint #2 contains "custom fields" for the user. I am trying to combine the returns of both endpoints and write them as a single row for each user in the SQL table.
They return PSCustomObject as a hash table.
Endpoint # 1: cat6 NoteProperty string cat6=NONE
Endpoint #2: CustomText94 NoteProperty System.Management.Automation.PSCustomObject CustomText94=#{description=; value=}
function Export-Feed {
Begin {
$serverInstance = ""
$database = ""
$tableName = ""
$employees = Get-Directory | Where-Object eestatus -eq A
}
Process {
$result = $employees | ForEach-Object -Parallel {
$params = #(
'firstname',
'middlename',
'lastname',
#{n='ADID';e={(Connect-Api -apiEndpoint "/api/v1/employee/$($_.eecode)/customfield").CustomText04.value}},
'hire_date',
'rehire_date',
'position_title',
'termination_date',
'work_email',
'employee_code',
'clocksequencenumber',
'department_code',
'department_description',
'employee_status',
'supervisor_primary',
'supervisor_primary_code',
'supervisor_secondary',
'supervisor_secondary_code',
'supervisor_tertiary',
'supervisor_tertiary_code',
'supervisor_quaternary',
'cat1',
'cat1desc',
'cat2',
'cat2desc',
'cat3',
'cat3desc',
'cat4',
'cat4desc',
'cat5',
'cat5desc',
'cat6',
'cat6desc',
'cat7',
'cat7desc',
'cat8',
'cat8desc',
'cat9',
'cat9desc'
)
Connect-Api -apiEndpoint "/api/v1/employee/$($_.eecode)" | Select-Object $params
}
}
End {
$result | Out-File c:\temp\test.txt
#Write-SqlTableData -DatabaseName $database -TableName $tableName -ServerInstance $serverInstance -SchemaName dbo -InputData $result -force
}
}
To merge two custom objects ([pscustomobject] instances, which are not the same as hashtables), use the following technique:
# Two sample input objects.
$o1 = [pscustomobject] #{ one = 1; two = 2; three = 3 }
$o2 = [pscustomobject] #{ four = 4; five = 5; six = 6 }
# Merge the two, by appending the properties of the 2nd to the 1st.
# Note: This assumes that there is no overlap between the property names.
foreach ($prop in $o2.psobject.Properties) {
$o1.psobject.Properties.Add($prop, $true)
}
# $o1 now contains the union of both property sets;
# Output it.
$o1 | Format-Table
The above yields:
one two three four five six
--- --- ----- ---- ---- ---
1 2 3 4 5 6
The above relies on every object in PowerShell having a hidden .psobject property that provides reflection information about the object, notably the collection of all properties returned by the .Properties property.
PowerShell allows properties to be dynamically added to any property, which is feature of its ETS (Extended Type System).
[pscustomobject] instances contain only such dynamic properties.
The .Add() method allows adding a copy of another object's property to an object; the $true indicates that no validation need be performed, which speeds up the operation.
I have an array of elements in PowerShell (v5.1) as below.
I want to remove the item with matching condition EmpId=102
$testData = #(
#{ Name='ABC'; EmpId=100 },
#{ Name='EFG'; EmpId=101 },
#{ Name='XYZ'; EmpId=102 }
);
Is there any simple way to achieve this?
maybe something like $testData.Remove($testData.Find("EmpId == 102")) ?
if your array is really a collection of one-item hashtables, then the following otta work ... [grin] it pipes the collection thru the Where-Object cmdlet & filters out anything item that has the value-to-exclude in the EmpId key value.
$TestData = #(
#{ Name='ABC'; EmpId=100 }
#{ Name='EFG'; EmpId=101 }
#{ Name='XYZ'; EmpId=102 }
)
$EmpId_ToExclude = 102
$TestData |
Where-Object {
$_['EmpId'] -ne $EmpId_ToExclude
}
output ...
Name Value
---- -----
EmpId 100
Name ABC
EmpId 101
Name EFG
note that the items are not required to be in order ... that is how hashtables work if you don't specify [ordered].
I hope this info is some use. Maybe you can use an arraylist. You don't need #( ) to make an array.
[collections.arraylist]$testData = #{ Name='ABC'; EmpId=100 },
#{ Name='EFG'; EmpId=101 },
#{ Name='XYZ'; EmpId=102 }
$a = $testdata[1]
$testdata.remove($a)
Or
$testdata.removeat(1)
Unfortunely, this doesn't work. I guess you would need a pointer to the hashtable:
$testdata.remove(#{ Name='XYZ'; EmpId=102 })
Actually, it would be easy with a single hashtable:
$testData = #{ 'ABC'=100
'EFG'=101
'XYZ'=102 }
$testdata.remove('xyz')
$testdata
Name Value
---- -----
ABC 100
EFG 101
How do we make Inner-Join or something a Cross-Join in PowerShell or PowerCLI?
Even though im new to PowerCLI/PowerShell , I do have a basic grasp on them, yet have practically spent 2 days trying to figure this, going through numerous documentations and blogs to no avail.
All I really want to know is if after typing my command
Get-Content File.txt
and getting:
Output1 or Table1 is
Name: Abc
Group: Bad
Policy: Great
Name: redi
Group: Good
Policy: MAD
etc. etc.
100s of these, and obviously more than just the 3 elements of Name, Group, Policy each.
Table2/Output2
Name: Abc
Limit: 10
used: 5
Name: redi
Limit: 20
used: 1
etc. etc.
100s of these.
and like 13 more of these text file tables, all with the "Name" as unique.
How can I combine it into one output at the end using Name with all the other elements?
My most obvious thought was something akin to joins, even if I had to do them 1 at a time, but even that I cant figure out how to do.
Is there anyway to do this in PowerShell itself without me having to go into Python or SQL?
If yes is there a method that is able to combine fields in spots where it's null?
If its not clear what type of result I am hoping for it will look something akin to this:
Name: Abc
Group: Bad
Policy: Great
Limit: 10
used: 5
Name: redi
Group: Good
Policy: MAD
Limit: 20
used: 1
Paweł Dyl provided you a solution
based on your two tables. However you probably need a generic solution where you don't have to specify each property by name yourself.
I would combine each table to a an array. Group the tables on the Name property using the Group-Object cmdlet. Iterate over each group and create a PsObject using the properties:
$table1 = [PSCustomObject]#{ Name = 'Abc'; Group = 'Bad'; Policy = 'Great'}, [PSCustomObject]#{ Name = 'redi'; Group = 'Good'; Policy = 'MAD'}
$table2 = [PSCustomObject]#{ Name = 'Abc'; Limit = '10'; used = '5'}, [PSCustomObject]#{ Name = 'redi'; Limit = '20'; used = '1'}
$allTables = $table1 + $table2
$allTables | group Name | Foreach {
$properties = #{}
$_.Group | Foreach {
$_.PsObject.Properties | Where Name -ne 'Name' | Foreach {
$properties += #{
"$($_.Name)" = "$($_.Value)"
}
}
}
$properties += #{Name = $_.Name}
New-Object PSObject –Property $properties
}
Output:
Group : Bad
Policy : Great
Name : Abc
Limit : 10
used : 5
Group : Good
Policy : MAD
Name : redi
Limit : 20
used : 1
You can use simple loop join as follows:
$table1 = [pscustomobject]#{Name='Abc';Group='Bad';Policy='Great'},[pscustomobject]#{Name='redi';Group='Good ';Policy='MAD'}
$table2 = [pscustomobject]#{Name='Abc';Limit=10;used=5},[pscustomobject]#{Name='redi';Limit=20;used=1}
$table1 | % {
foreach ($t2 in $table2) {
if ($_.Name -eq $t2.Name) {
[pscustomobject]#{Name=$_.Name;Group=$_.Group;Policy=$_.Policy;Limit=$t2.Limit;Used=$t2.Used}
}
}
}
Assuming uniqueness of keys you can also use faster, hashtable approach:
$hashed = $table1 | group Name -AsHashTable
$table2 | % {
$matched = $hashed[$_.Name]
if ($matched) {
[pscustomobject]#{Name=$matched.Name;Group=$matched.Group;Policy=$matched.Policy;Limit=$_.Limit;Used=$_.Used}
}
}
You can also use generic solution and wrap it in function. It matches records by their property names:
function Join-Records($tab1, $tab2){
$prop1 = $tab1 | select -First 1 | % {$_.PSObject.Properties.Name} #properties from t1
$prop2 = $tab2 | select -First 1 | % {$_.PSObject.Properties.Name} #properties from t2
$join = $prop1 | ? {$prop2 -Contains $_}
$unique1 = $prop1 | ?{ $join -notcontains $_}
$unique2 = $prop2 | ?{ $join -notcontains $_}
if ($join) {
$tab1 | % {
$t1 = $_
$tab2 | % {
$t2 = $_
foreach ($prop in $join) {
if (!$t1.$prop.Equals($t2.$prop)) { return; }
}
$result = #{}
$join | % { $result.Add($_,$t1.$_) }
$unique1 | % { $result.Add($_,$t1.$_) }
$unique2 | % { $result.Add($_,$t2.$_) }
[PSCustomObject]$result
}
}
}
}
$table1 = [pscustomobject]#{Name='Abc';Group='Bad';Policy='Great'},
[pscustomobject]#{Name='redi';Group='Good ';Policy='MAD'},
[pscustomobject]#{Name='Not joined';Group='Very bad';Policy='Great'}
$table2 = [pscustomobject]#{Name='Abc';Limit=10;used=5},
[pscustomobject]#{Name='redi';Limit=20;used=1},
[pscustomobject]#{Name='redi';Limit=20;used=2}
#name is only common property, records joined by name
Join-Records $table1 $table2
#example2
$test1 = [pscustomobject]#{A=1;B=1;C='R1'},
[pscustomobject]#{A=1;B=2;C='R2'},
[pscustomobject]#{A=2;B=2;C='R3'}
$test2 = [pscustomobject]#{A=1;B=1;D='R4'},
[pscustomobject]#{A=3;B=2;D='R5'},
[pscustomobject]#{A=4;B=2;D='R6'}
Join-Records $test1 $test2 #joined by two common columns - A and B
You can also cascade calls:
$test1 = [pscustomobject]#{A=1;B=1;C='R1'},
[pscustomobject]#{A=1;B=2;C='R2'},
[pscustomobject]#{A=2;B=2;C='R3'}
$test2 = [pscustomobject]#{A=1;B=1;D='R4'},
[pscustomobject]#{A=3;B=2;D='R5'},
[pscustomobject]#{A=4;B=2;D='R6'}
$test3 = [pscustomobject]#{B=1;E='R7'},
[pscustomobject]#{B=2;E='R8'},
[pscustomobject]#{B=3;E='R9'}
#first join by common A and B, then join result by common B
Join-Records (Join-Records $test1 $test2) $test3
So I found an Answer which was more suitable and it uses the join-Object function which was defined below:
you can access it at https://github.com/RamblingCookieMonster/PowerShell/blob/master/Join-Object.ps1
All I really had to do was Define my outputs as $A and $B and $C and so on, and just
$Join1= Join-Object -Left $A -Right $B -LeftJoinProperty Name - RightJoinProperty Name
made $Join2 then 3 so on until I got it all done
$Join2 = Join-Object -Left $Join1 -Right $C -LeftJoinProperty Name -RightJoinProperty Name
$Join3 = Join-Object -Left $Join2 -Right $D -LeftJoinProperty Name -RightJoinProperty Name
$Join4 = Join-Object -Left $Join3 -Right $E -LeftJoinProperty Name -RightJoinProperty Name
Until I got it all done
$Table1 | Join $Table2 -Using Name
$Table1 | Join $Table2 #Cross Join
See: In Powershell, what's the best way to join two tables into one?