Combine API results into single object in Powershell - powershell

I have two API endpoints to hit and retrieve user info and write the data into a SQL table. The endpoint uses employee ID in the URL so I am looping through each user to grab their data. Endpoint #2 contains "custom fields" for the user. I am trying to combine the returns of both endpoints and write them as a single row for each user in the SQL table.
They return PSCustomObject as a hash table.
Endpoint # 1: cat6 NoteProperty string cat6=NONE
Endpoint #2: CustomText94 NoteProperty System.Management.Automation.PSCustomObject CustomText94=#{description=; value=}
function Export-Feed {
Begin {
$serverInstance = ""
$database = ""
$tableName = ""
$employees = Get-Directory | Where-Object eestatus -eq A
}
Process {
$result = $employees | ForEach-Object -Parallel {
$params = #(
'firstname',
'middlename',
'lastname',
#{n='ADID';e={(Connect-Api -apiEndpoint "/api/v1/employee/$($_.eecode)/customfield").CustomText04.value}},
'hire_date',
'rehire_date',
'position_title',
'termination_date',
'work_email',
'employee_code',
'clocksequencenumber',
'department_code',
'department_description',
'employee_status',
'supervisor_primary',
'supervisor_primary_code',
'supervisor_secondary',
'supervisor_secondary_code',
'supervisor_tertiary',
'supervisor_tertiary_code',
'supervisor_quaternary',
'cat1',
'cat1desc',
'cat2',
'cat2desc',
'cat3',
'cat3desc',
'cat4',
'cat4desc',
'cat5',
'cat5desc',
'cat6',
'cat6desc',
'cat7',
'cat7desc',
'cat8',
'cat8desc',
'cat9',
'cat9desc'
)
Connect-Api -apiEndpoint "/api/v1/employee/$($_.eecode)" | Select-Object $params
}
}
End {
$result | Out-File c:\temp\test.txt
#Write-SqlTableData -DatabaseName $database -TableName $tableName -ServerInstance $serverInstance -SchemaName dbo -InputData $result -force
}
}

To merge two custom objects ([pscustomobject] instances, which are not the same as hashtables), use the following technique:
# Two sample input objects.
$o1 = [pscustomobject] #{ one = 1; two = 2; three = 3 }
$o2 = [pscustomobject] #{ four = 4; five = 5; six = 6 }
# Merge the two, by appending the properties of the 2nd to the 1st.
# Note: This assumes that there is no overlap between the property names.
foreach ($prop in $o2.psobject.Properties) {
$o1.psobject.Properties.Add($prop, $true)
}
# $o1 now contains the union of both property sets;
# Output it.
$o1 | Format-Table
The above yields:
one two three four five six
--- --- ----- ---- ---- ---
1 2 3 4 5 6
The above relies on every object in PowerShell having a hidden .psobject property that provides reflection information about the object, notably the collection of all properties returned by the .Properties property.
PowerShell allows properties to be dynamically added to any property, which is feature of its ETS (Extended Type System).
[pscustomobject] instances contain only such dynamic properties.
The .Add() method allows adding a copy of another object's property to an object; the $true indicates that no validation need be performed, which speeds up the operation.

Related

Convert multiple txt file to one single csv file [duplicate]

When we're trying to export data to other functions via the pipeline, we observe some strange behavior in PowerShell.
Example code:
$Array = #()
$Obj1 = [PSCustomObject]#{
Member1 = 'First'
Member2 = 'Second'
}
$Obj2 = [PSCustomObject]#{
Member1 = 'First'
Member2 = 'Second'
Member3 = 'Third'
}
$Array = $Obj1, $Obj2
$Array | Out-GridView -Title 'Not showing Member3'
$Array = $Obj2, $Obj1
$Array | Out-GridView -Title 'All members correctly displayed'
In the example above you can see that when the first object only contains 2 properties, the Out-GridView CmdLet (and others) only show 2 properties, even though the second object has 3 properties. However, when the first object in the array has 3 properties it does display them all correctly.
Is there a way around this? Because it's not possible to predict up front how many properties on an object there will be and if the object with the most properties will be the first one in the array.
I had the same experience once and created the following reusable 'Union' function:
# 2021-08-25 Removed Union function
Usage:
$Obj1, $Obj2 | Union | Out-GridView -Title 'Showing all members'
It is also supposed to work with complex objects. Some standard cmdlets output multiple object types at once and if you view them (e.g. Out-GridView) or dump them in a file (e.g. Export-Csv) you might miss a lot of properties. Take as another example:
Get-WmiObject -Namespace root/hp/instrumentedBIOS -Class hp_biosSetting | Union | Export-Csv ".\HPBIOS.csv"
Added 2014-09-19:
Maybe this is already between the lines in the comments $Array | Select * | … will not resolve the issue but specifically selecting the properties $Array | Select Member1, Member2, Member3 | … does.
Besides, although in most cases the Union function will work, there are some exceptions to that as it will only align the first object with the rest.
Consider the following object:
$List = #(
New-Object PSObject -Property #{Id = 2}
New-Object PSObject -Property #{Id = 1}
New-Object PSObject -Property #{Id = 3; Name = "Test"}
)
If you Union this object everything appears to be fine and if you e.g. ExportTo-CSV and work with the export .csv file from then on you will never have any issue.
$List | Union
Id Name
-- ----
2
1
3 Test
Still there is a catch as only the first object is aligned. If you e.g. sort the result on Id (Sort Id) or take just the last 2 (Select -Last 2) entries, the Name is not listed because the second object doesn’t contain the Name property:
$List | Union | Sort Id
Id
--
1
2
3
Therefor I have rewritten the Union-Object (Alias Union) function`):
Union-Object
# 2021-08-25 Removed Union-Object function
Syntax:
$Array | Union | Out-GridView -Title 'All members correctly displayed'
Update 2021-08-25
Based on az1d helpful feedback on an error caused by equal property names with different casing, I have created a new UnifyProperties function.
(I will no longer use the name UnionObject for his)
function UnifyProperties {
$Names = [System.Collections.Generic.HashSet[string]]::new([StringComparer]::OrdinalIgnoreCase)
$InputCollected = #($Input)
$InputCollected.ForEach({
foreach ($Name in $_.psobject.Properties.Name) { $Null = $Names.Add($Name) }
})
$inputCollected | Select-Object #($Names)
}
Usage:
[pscustomobject] #{ one = 1; two = 2; three = 3 },
[pscustomobject] #{ ONE = 10; THREE = 30; FOUR = 4 } |
UnifyProperties
one two three FOUR
--- --- ----- ----
1 2 3
10 30 4
See also: #13906 Add -UnifyProperties parameter to Select-Object

How to to have an array as a return type on a custom PSObject

I've got the following:
# Compare the 2 lists and return the ones that exist in the top and children (meaning they're redundant).
$redundantUsers = $usersAssignedToThisGroup | where { $usersAssignedToGroupsInThisGroup -contains $_ }
# Build the results to output
$results += New-Object PSObject -property #{
group = $group.name #group assigned out of scope
users = #($redundantUsers)
}
I'm expecting to be able to call my script like:
$users = ./MyScript -myParam "something" | Select users
Then I'm expecting to be able to type $users[1] in the console and get the first element of that array.
But instead, I'm having to do $users[0].users[1].
How do I change my data and or call to do what I want?
I'm essentially just trying to let the script return the data in a useful way.

Listing all user properties but excluding whitespace/empty fields in Powershell [duplicate]

How do I get properties that ONLY have populated values?
So for example if I run
Get-QADUser -Identity "SomeOne" -IncludeAllProperties
the output would of course include.. all properties, including those with and those without values. I want a listing of properties with values only. How is it done generally speaking?
This wouldn't be restricted to Quest Cmdlets, I only use Get-QADUser as an example.
You could try using the built-in (hidden) property of PowerShell objects called PSObject, which includes a property called Properties, i.e. a list of all properties on the parent object.
Maybe easier with an example. Take Get-Process... a process can have many attributes (properties) with or without values. In order to get just the ones with values you do this:
(Get-Process | Select -First 1).PSObject.Properties | ?{$_.Value -ne $null} | FT Name,Value
Note that I limited this to just the first process returned by Get-Process. We then get all the properties defined on that object, filtering where Value is not null and then displaying just the Name and Value for those properties.
To complement Charlie Joynt's helpful answer:
Below is convenience function Remove-NullProperties, which creates custom-object copies of its input objects populated with only the non-$null properties of the input objects.
Example use:
# Sample input collection, with 2 objects with different $null-valued
# properties.
$coll = [pscustomobject] #{ one = 'r1c1'; two = $null; three = 'r1c3' },
[pscustomobject] #{ one = 'r2c1'; two = 'r2c2'; three = $null }
# Output copies containing only non-$null-valued properties.
# NOTE: The `ForEach-Object { Out-String -InputObject $_ }` part is solely
# there to ensure that *all* resulting properties are shown.
# With the default output, only the properties found on the FIRST
# input object would be used in the output table.
$coll | Remove-NullProperties |
ForEach-Object { Out-String -InputObject $_ }
This yields the following - note how the respective null-valued properties were removed:
one three
--- -----
r1c1 r1c3
one two
--- ---
r2c1 r2c2
Remove-NullProperties source code:
<#
.SYNOPSIS
Removes properties with $null values from custom-object copies of
the input objects.
.DESCRIPTION
Note that output objects are custom objects that are copies of the input
objects with copies of only those input-object properties that are not $null.
CAVEAT: If you pipe multiple objects to this function, and these objects
differ in what properties are non-$null-valued, the default output
format will show only the non-$null-valued properties of the FIRST object.
Use ... | ForEach-Object { Out-String -InputObject $_ } to avoid
this problem.
.NOTES
Since the output objects are generally of a distinct type - [pscustomobject] -
and have only NoteProperty members, use of this function only makes sense
with plain-old data objects as input.
.EXAMPLE
> [pscustomobject] #{ one = 1; two = $null; three = 3 } | Remove-NullProperties
one three
--- -----
1 3
#>
function Remove-NullProperties {
param(
[parameter(Mandatory,ValueFromPipeline)]
[psobject] $InputObject
)
process {
# Create the initially empty output object
$obj = [pscustomobject]::new()
# Loop over all input-object properties.
foreach($prop in $InputObject.psobject.properties) {
# If a property is non-$null, add it to the output object.
if ($null -ne $InputObject.$($prop.Name)) {
Add-Member -InputObject $obj -NotePropertyName $prop.Name -NotePropertyValue $prop.Value
}
}
# Give the output object a type name that reflects the type of the input
# object prefixed with 'NonNull.' - note that this is purely informational, unless
# you define a custom output format for this type name.
$obj.pstypenames.Insert(0, 'NonNull.' + $InputObject.GetType().FullName)
# Output the output object.
$obj
}
}
These answers didn't work for me in the case of importing an object from an Infoblox csv file. Some values were the empty string, but not null. Testing whether a property is true or not, seems to work better for me. And the result is an object.
$a = [pscustomobject]#{one='hi';two='';three='there'}
$prop = $a.psobject.Properties | where value | foreach name
$a | select $prop
one three
--- -----
hi there
You first get its properties (since Get-QADUser depends on AD schema, the properties list is dynamic) with get-member -type property, then filter out those that don't have \{.*(get).*\} in its definition (that is, they are not "gettable"), then enumerate the resultant list by name and filter out nulls.
$someone=Get-QADUser -Identity "SomeOne" -IncludeAllProperties
$members=$someone|get-member -type property| where {$_.definition -match '\{.*(get).*\}'}
foreach ($member in $members) {
if ($someone[$member.name] -ne $null) {
write-host $member.name $someone[$member.name]
}
}

How do I get properties that ONLY have populated values?

How do I get properties that ONLY have populated values?
So for example if I run
Get-QADUser -Identity "SomeOne" -IncludeAllProperties
the output would of course include.. all properties, including those with and those without values. I want a listing of properties with values only. How is it done generally speaking?
This wouldn't be restricted to Quest Cmdlets, I only use Get-QADUser as an example.
You could try using the built-in (hidden) property of PowerShell objects called PSObject, which includes a property called Properties, i.e. a list of all properties on the parent object.
Maybe easier with an example. Take Get-Process... a process can have many attributes (properties) with or without values. In order to get just the ones with values you do this:
(Get-Process | Select -First 1).PSObject.Properties | ?{$_.Value -ne $null} | FT Name,Value
Note that I limited this to just the first process returned by Get-Process. We then get all the properties defined on that object, filtering where Value is not null and then displaying just the Name and Value for those properties.
To complement Charlie Joynt's helpful answer:
Below is convenience function Remove-NullProperties, which creates custom-object copies of its input objects populated with only the non-$null properties of the input objects.
Example use:
# Sample input collection, with 2 objects with different $null-valued
# properties.
$coll = [pscustomobject] #{ one = 'r1c1'; two = $null; three = 'r1c3' },
[pscustomobject] #{ one = 'r2c1'; two = 'r2c2'; three = $null }
# Output copies containing only non-$null-valued properties.
# NOTE: The `ForEach-Object { Out-String -InputObject $_ }` part is solely
# there to ensure that *all* resulting properties are shown.
# With the default output, only the properties found on the FIRST
# input object would be used in the output table.
$coll | Remove-NullProperties |
ForEach-Object { Out-String -InputObject $_ }
This yields the following - note how the respective null-valued properties were removed:
one three
--- -----
r1c1 r1c3
one two
--- ---
r2c1 r2c2
Remove-NullProperties source code:
<#
.SYNOPSIS
Removes properties with $null values from custom-object copies of
the input objects.
.DESCRIPTION
Note that output objects are custom objects that are copies of the input
objects with copies of only those input-object properties that are not $null.
CAVEAT: If you pipe multiple objects to this function, and these objects
differ in what properties are non-$null-valued, the default output
format will show only the non-$null-valued properties of the FIRST object.
Use ... | ForEach-Object { Out-String -InputObject $_ } to avoid
this problem.
.NOTES
Since the output objects are generally of a distinct type - [pscustomobject] -
and have only NoteProperty members, use of this function only makes sense
with plain-old data objects as input.
.EXAMPLE
> [pscustomobject] #{ one = 1; two = $null; three = 3 } | Remove-NullProperties
one three
--- -----
1 3
#>
function Remove-NullProperties {
param(
[parameter(Mandatory,ValueFromPipeline)]
[psobject] $InputObject
)
process {
# Create the initially empty output object
$obj = [pscustomobject]::new()
# Loop over all input-object properties.
foreach($prop in $InputObject.psobject.properties) {
# If a property is non-$null, add it to the output object.
if ($null -ne $InputObject.$($prop.Name)) {
Add-Member -InputObject $obj -NotePropertyName $prop.Name -NotePropertyValue $prop.Value
}
}
# Give the output object a type name that reflects the type of the input
# object prefixed with 'NonNull.' - note that this is purely informational, unless
# you define a custom output format for this type name.
$obj.pstypenames.Insert(0, 'NonNull.' + $InputObject.GetType().FullName)
# Output the output object.
$obj
}
}
These answers didn't work for me in the case of importing an object from an Infoblox csv file. Some values were the empty string, but not null. Testing whether a property is true or not, seems to work better for me. And the result is an object.
$a = [pscustomobject]#{one='hi';two='';three='there'}
$prop = $a.psobject.Properties | where value | foreach name
$a | select $prop
one three
--- -----
hi there
You first get its properties (since Get-QADUser depends on AD schema, the properties list is dynamic) with get-member -type property, then filter out those that don't have \{.*(get).*\} in its definition (that is, they are not "gettable"), then enumerate the resultant list by name and filter out nulls.
$someone=Get-QADUser -Identity "SomeOne" -IncludeAllProperties
$members=$someone|get-member -type property| where {$_.definition -match '\{.*(get).*\}'}
foreach ($member in $members) {
if ($someone[$member.name] -ne $null) {
write-host $member.name $someone[$member.name]
}
}

Difference between PSObject, Hashtable, and PSCustomObject

Can anybody explain the details? If I create an object using
$var = [PSObject]#{a=1;b=2;c=3}
and then I look for its type using getType() PowerShell tells me it's of type Hashtable.
When using Get-Member (alias gm) to inspect the object it's obvious that a hashtable has been created, since it has a keys and a values property. So what's the difference to a "normal" hashtable?
Also, what's the advantage of using a PSCustomObject? When creating one using something like this
$var = [PSCustomObject]#{a=1;b=2;c=3}
the only visible difference to me is the different datatype of PSCustomObject. Also instead of keys and value properties, a inspection with gm shows that now every key has been added as a NoteProperty object.
But what advantages do I have? I'm able to access my values by using its keys, just like in the hashtable. I can store more than simple key-value pairs (key-object pairs for example) in the PSCustomObject, JUST as in the hashtable. So what's the advantage? Are there any important differences?
One scenario where [PSCustomObject] is used instead of HashTable is when you need a collection of them. The following is to illustrate the difference in how they are handled:
$Hash = 1..10 | %{ #{Name="Object $_" ; Index=$_ ; Squared = $_*$_} }
$Custom = 1..10 | %{[PSCustomObject] #{Name="Object $_" ; Index=$_ ; Squared = $_*$_} }
$Hash | Format-Table -AutoSize
$Custom | Format-Table -AutoSize
$Hash | Export-Csv .\Hash.csv -NoTypeInformation
$Custom | Export-Csv .\CustomObject.csv -NoTypeInformation
Format-Table will result in the following for $Hash:
Name Value
---- -----
Name Object 1
Squared 1
Index 1
Name Object 2
Squared 4
Index 2
Name Object 3
Squared 9
...
And the following for $CustomObject:
Name Index Squared
---- ----- -------
Object 1 1 1
Object 2 2 4
Object 3 3 9
Object 4 4 16
Object 5 5 25
...
The same thing happens with Export-Csv, thus the reason to use [PSCustomObject] instead of just plain HashTable.
Say I want to create a folder. If I use a PSObject you can tell it is wrong by
looking at it
PS > [PSObject] #{Path='foo'; Type='directory'}
Name Value
---- -----
Path foo
Type directory
However the PSCustomObject looks correct
PS > [PSCustomObject] #{Path='foo'; Type='directory'}
Path Type
---- ----
foo directory
I can then pipe the object
[PSCustomObject] #{Path='foo'; Type='directory'} | New-Item
From the PSObject documentation:
Wraps an object providing alternate views of the available members and ways to extend them. Members can be methods, properties, parameterized properties, etc.
In other words, a PSObject is an object that you can add methods and properties to after you've created it.
From the "About Hash Tables" documentation:
A hash table, also known as a dictionary or associative array, is a compact data structure that stores one or more key/value pairs.
...
Hash tables are frequently used because they are very efficient for finding and retrieving data.
You can use a PSObject like a Hashtable because PowerShell allows you to add properties to PSObjects, but you shouldn't do this because you'll lose access to Hashtable specific functionality, such as the Keys and Values properties. Also, there may be performance costs and additional memory usage.
The PowerShell documentation has the following information about PSCustomObject:
Serves as a placeholder BaseObject when PSObject's constructor with no parameters is used.
This was unclear to me, but a post on a PowerShell forum from the co-author of a number of PowerShell books seems more clear:
[PSCustomObject] is a type accelerator. It constructs a PSObject, but does so in a way that results in hash table keys becoming properties. PSCustomObject isn't an object type per se – it's a process shortcut. ... PSCustomObject is a placeholder that's used when PSObject is called with no constructor parameters.
Regarding your code, #{a=1;b=2;c=3} is a Hashtable. [PSObject]#{a=1;b=2;c=3} doesn't convert the Hashtable to a PSObject or generate an error. The object remains a Hashtable. However, [PSCustomObject]#{a=1;b=2;c=3} converts the Hashtable into a PSObject. I wasn't able to find documentation stating why this happens.
If you want to convert a Hashtable into an object in order to use its keys as property names you can use one of the following lines of code:
[PSCustomObject]#{a=1;b=2;c=3}
# OR
New-Object PSObject -Property #{a=1;b=2;c=3}
# NOTE: Both have the type PSCustomObject
If you want to convert a number of Hashtables into an object where their keys are property names you can use the following code:
#{name='a';num=1},#{name='b';num=2} |
% { [PSCustomObject]$_ }
# OR
#{name='a';num=1},#{name='b';num=2} |
% { New-Object PSObject -Property $_ }
<#
Outputs:
name num
---- ---
a 1
b 2
#>
Finding documentation regarding NoteProperty was difficult. In the Add-Member documentation, there isn't any -MemberType that makes sense for adding object properties other than NoteProperty. The Windows PowerShell Cookbook (3rd Edition) defined the Noteproperty Membertype as:
A property defined by the initial value you provide
Lee, H. (2013). Windows PowerShell Cookbook. O'Reilly Media, Inc. p. 895.
One advantage I think for PSObject is that you can create custom methods with it.
For example,
$o = New-Object PSObject -Property #{
"value"=9
}
Add-Member -MemberType ScriptMethod -Name "Sqrt" -Value {
echo "the square root of $($this.value) is $([Math]::Round([Math]::Sqrt($this.value),2))"
} -inputObject $o
$o.Sqrt()
You can use this to control the sorting order of the PSObject properties (see PSObject sorting)
I think the biggest difference you'll see is the performance. Have a look at this blog post:
Combining Objects Efficiently – Use a Hash Table to Index a Collection of Objects
The author ran the following code:
$numberofobjects = 1000
$objects = (0..$numberofobjects) |% {
New-Object psobject -Property #{'Name'="object$_";'Path'="Path$_"}
}
$lookupobjects = (0..$numberofobjects) | % {
New-Object psobject -Property #{'Path'="Path$_";'Share'="Share$_"}
}
$method1 = {
foreach ($object in $objects) {
$object | Add-Member NoteProperty -Name Share -Value ($lookupobjects | ?{$_.Path -eq $object.Path} | select -First 1 -ExpandProperty share)
}
}
Measure-Command $method1 | select totalseconds
$objects = (0..$numberofobjects) | % {
New-Object psobject -Property #{'Name'="object$_";'Path'="Path$_"}
}
$lookupobjects = (0..$numberofobjects) | % {
New-Object psobject -Property #{'Path'="Path$_";'Share'="Share$_"}
}
$method2 = {
$hash = #{}
foreach ($obj in $lookupobjects) {
$hash.($obj.Path) = $obj.share
}
foreach ($object in $objects) {
$object |Add-Member NoteProperty -Name Share -Value ($hash.($object.path)).share
}
}
Measure-Command $method2 | select totalseconds
Blog author's output:
TotalSeconds
------------
167.8825285
0.7459279
His comment regarding the code results is:
You can see the difference in speed when you put it all together. The object method takes 167 seconds on my computer while the hash table method will take under a second to build the hash table and then do the lookup.
Here are some of the other, more-subtle benefits:
Custom objects default display in PowerShell 3.0
We have a bunch of templates in our Windows-PKI and we needed a script, that has to work with all active templates. We do not need to dynamically add templates or remove them.
What for me works perfect (since it is also so "natural" to read) is the following:
$templates = #(
[PSCustomObject]#{Name = 'template1'; Oid = '1.1.1.1.1'}
[PSCustomObject]#{Name = 'template2'; Oid = '2.2.2.2.2'}
[PSCustomObject]#{Name = 'template3'; Oid = '3.3.3.3.3'}
[PSCustomObject]#{Name = 'template4'; Oid = '4.4.4.4.4'}
[PSCustomObject]#{Name = 'template5'; Oid = '5.5.5.5.5'}
)
foreach ($template in $templates)
{
Write-Output $template.Name $template.Oid
}
Type-1: $PSCustomObject = [PSCustomObject] #{a=1;b=2;c=3;d=4;e=5;f=6}
Type-2: $PsObject = New-Object -TypeName PSObject -Property #{a=1;b=2;c=3;d=4;e=5;f=6}
The only difference between Type-1 & Type-2
Type-1 Property are displayed in same order as we added
Type-1 enumerates the data faster
Type-1 will not work with systems running PSv2.0 or earlier
Both Type-1 & Type-2 are of type “System.Management.Automation.PSCustomObject”
Difference between HashTable and PSCustomObject/PSObject is
You can add new methods and properties to PSCustomObject/PSObject
You can use PSCustomObject/PSObject for pipeline parameter binding using ValueFromPipelineByPropertyName as explained by Zombo
example: [PSCustomObject] #{Path='foo'; Type='directory'} | New-Item