I have created a PsCustomObject, when the variable is called is ISE, it reads a table of the relevant data. However, if I try to compare the PsCustomObject with another object, the PsCustomObject doesn't read correctly. I'd like to tell the script if any of the lines in the existing CSV match the PSCustomObject do not export the data to the CSV, in other words skip duplicate rows in the CSV file. The CSV may or may not have multiple rows.
$fileInfo = #(
[pscustomobject]#{
user_id = $user
studio = $studio
function = $Task
end_time_local = $creationTime
asin = $ASIN
variant = $variant
process_class_id = $processClass
}
)
$currentData = Import-Csv "$scansFolder\$fileName.csv"
if($fileInfo -ne $currentData){
$fileInfo | Export-Csv "$scansFolder\$fileName.csv" -Append -NoTypeInformation -Force
}
[pscustomobject] is a .NET reference type, so comparing two instances[1] with -eq will test for reference equality (identity), i.e. if the two instances are one and the same object[2] - which is obviously not the case in your scenario.
Assuming that the properties of your custom objects are instances of value types or strings (which appears to be the case), you can use Compare-Object to compare objects by their property values, with the ability to compare two collections:
$fileInfo = #(
[pscustomobject]#{
user_id = $user
studio = $studio
function = $Task
end_time_local = $creationTime
asin = $ASIN
variant = $variant
process_class_id = $processClass
}
)
# Get the property names.
# This assumes that the CSV data has (at least) the same
# set of properties (columns).
$propNames = $fileInfo[0].psobject.properties.Name
$currentData = Import-Csv "$scansFolder\$fileName.csv"
# Compare the $fileInfo custom object(s) to the custom objects read
# from the CSV file and only export those that are unique to the RHS ('=>')
# back to the file, i.e., those that don't match $fileInfo.
Compare-Object -Property $propNames $fileInfo $currentData |
Where-Object SideIndicator -eq '=>' | Select-Object InputObject |
Export-Csv "$scansFolder\$fileName.csv" -Append -NoTypeInformation -Force
[1] Import-Csv outputs [pscustomobject] instances too.
[2] See the Equality Comparison help topic (written for C#, but applies analogously to PowerShell's -eq operator).
Related
I have a script that periodically generates a list of all files in a directory, and then writes a text file of the results to a different directory.
I'd like to change this so it checks the newest text file in the output directory, and only makes a new one if there's differences. It seemed simple enough.
Here's what I tried:
First I get the most recent file in the directory, grab the hash, and write my variable values to the console:
$lastFile = gci C:\ReportOutputDir | sort LastWriteTime | select -last 1 | Select-Object -ExpandProperty FullName
$oldHash = Get-FileHash $lastFile | Select-Object Hash
Write-Host 'lastFile = '$lastFile
Write-Host 'oldHash = '$oldHash
Output:
lastFile = C:\ReportOutputDir\test1.txt
oldHash = #{Hash=E7787C54F5BAE236100A24A6F453A5FDF6E6C7333B60ED8624610EAFADF45521}
Then I do the exact same gci on the FileList dir, and create a new file (new_test.txt), then grab the hash of this file:
gci -Path C:\FileLists -File -Recurse -Name -Depth 2 | Sort-Object | out-file C:\ReportOutputDir\new_test.txt
$newFile = gci C:\ReportOutputDir | sort LastWriteTime | select -last 1 | Select-Object -ExpandProperty FullName
$newHash = Get-FileHash $newFile | Select-Object Hash
Write-Host 'newFile = '$newFile
Write-Host 'newHash = '$newHash
Output:
newFile = C:\ReportOutputDir\new_test.txt
newHash = #{Hash=E7787C54F5BAE236100A24A6F453A5FDF6E6C7333B60ED8624610EAFADF45521}
Finally, I attempt my -eq operator where I'd usually simply remove the newFile if it's equal. For now, I'm just doing a simple :
if ($newHash -eq $oldHash){
'files are equal'
}
else {'files are not equal'}
And somehow, I'm getting
files are not equal
What gives? Also, for the record I was originally trying to save the gci output to a variable and comparing the contents of the last file to the gci output, but was also having trouble with the -eq operator. Fairly new to powershell stuff so I'm sure I'm doing something wrong here.
Select-Object Hash creates an object with a .Hash property and it is that property that contains the hash string.
The object returned is of type [pscustomobject], and two instances of this type never compare as equal - even if all their property names and values are equal:
The reason is that reference equality is tested, because [pscustomobject] is a .NET reference type that doesn't define custom equality-testing logic.
Testing reference equality means that only two references to the very same instance compare as equal.
A quick example:
PS> [pscustomobject] #{ foo = 1 } -eq [pscustomobject] #{ foo = 1 }
False # !! Two distinct instances aren't equal, no matter what they contain.
You have two options:
Compare the .Hash property values, not the objects as a whole:
if ($newHash.Hash -eq $oldHash.Hash) { # ...
If you don't need a [pscustomobject] wrapper for the hash strings, use Select-Object's -ExpandProperty parameter instead of the (possibly positionally implied) -Property parameter:
Select-Object -ExpandProperty Hash
As for why the Write-Host output matched:
When you force objects to be converted to string representations - essentially, Write-Host calls .ToString() on its arguments - the string representations of distinct [pscustomobject] instances that have the same properties and values will be the same:
PS> "$([pscustomobject] #{ foo = 1 })" -eq "$([pscustomobject] #{ foo = 1 })"
True # Same as: '#{foo=1}' -eq '#{foo=1}'
However, you should not rely on these hashtable-like string representations to determine equality of [pscustomobject]s as a whole, because of the inherent limitations of these representations, which can easily yield false positives.
This answer shows how to compare [pscustomobject] instances as a whole, by comparing all of their property values, by passing all property names to Compare-Object -Property - but note that this assumes that all property values are either strings or instances of .NET value types or corresponding properties must again either reference the very same instance of a .NET reference type or be of a type that implements custom equality-comparison logic.
I have 2 PScustomObjects that i would like to combine into a single csv as follows:
$Output1 = [PSCustomObject]#{
Timestamp1 = (Get-Date)
InstanceName1 = $ProcessName1
Count1 = $Processes1.Count
Memory1 = $MEMLoad1
CPU1 = $CPULoad1
}
$Output2 = [PSCustomObject]#{
Timestamp2 = (Get-Date)
InstanceName2 = $ProcessName2
Count2 = $Processes2.Count
Memory2 = $MEMLoad2
CPU2 = $CPULoad2
}
The CSV should have the titles TimeStamp1, InstanceName1......TimeStamp2, InstanceName2....
This code runs in a loop and exports data each pass.
Is there a way to do this? Also is there a way to do this dynamically to avoid replicating large amounts of code if i am to export data on say 100 PsCustomObjects, maybe lookping through the input data for the object and the putting in in one object and passing that to the csv while dynamically changing titles?
I use the following line to export. I've tried -InputObject $Output1, $Output2 but that gives gibberish.
Export-Csv -InputObject $Output1 -path $Path1 -NoTypeInformation -Append -Force
The only solution i have so far is to export multiple CSV's but that gets bulky fast.
Any help is greatly appreciated!
The best solution is to ensure all the objects your script/function outputs use the same schema (that is, they should all share the same common property names).
If that's not possible, you can use Select-Object to ensure all possible properties are selected for all objects:
$Output1,$Output2 |Select-Object Count1,Count2,CPU1,CPU2,InstanceName1,InstanceName2,Memory1,Memory2,Timestamp1,Timestamp2 |Export-Csv ...
This is obviously not very practical, so if you have many different object schemas, you'll want to automate the discovery of all possible property names.
To do that, we can inspect the psobject hidden memberset:
$allTheObjectsToExport = $Output1,$Output2
$propertyNames = $allTheObjectsToExport |ForEach-Object {
# Discover all the object's properties
$_.psobject.Properties |ForEach-Object {
# Get each property name
$_.Name
}
} |Sort-Object -Unique
# Now the CSV will have a column for every single unique property name
$allTheObjectsToExport |Select-Object $propertyNames |Export-Csv -Path $Path1 -NoTypeInformation
How do I get properties that ONLY have populated values?
So for example if I run
Get-QADUser -Identity "SomeOne" -IncludeAllProperties
the output would of course include.. all properties, including those with and those without values. I want a listing of properties with values only. How is it done generally speaking?
This wouldn't be restricted to Quest Cmdlets, I only use Get-QADUser as an example.
You could try using the built-in (hidden) property of PowerShell objects called PSObject, which includes a property called Properties, i.e. a list of all properties on the parent object.
Maybe easier with an example. Take Get-Process... a process can have many attributes (properties) with or without values. In order to get just the ones with values you do this:
(Get-Process | Select -First 1).PSObject.Properties | ?{$_.Value -ne $null} | FT Name,Value
Note that I limited this to just the first process returned by Get-Process. We then get all the properties defined on that object, filtering where Value is not null and then displaying just the Name and Value for those properties.
To complement Charlie Joynt's helpful answer:
Below is convenience function Remove-NullProperties, which creates custom-object copies of its input objects populated with only the non-$null properties of the input objects.
Example use:
# Sample input collection, with 2 objects with different $null-valued
# properties.
$coll = [pscustomobject] #{ one = 'r1c1'; two = $null; three = 'r1c3' },
[pscustomobject] #{ one = 'r2c1'; two = 'r2c2'; three = $null }
# Output copies containing only non-$null-valued properties.
# NOTE: The `ForEach-Object { Out-String -InputObject $_ }` part is solely
# there to ensure that *all* resulting properties are shown.
# With the default output, only the properties found on the FIRST
# input object would be used in the output table.
$coll | Remove-NullProperties |
ForEach-Object { Out-String -InputObject $_ }
This yields the following - note how the respective null-valued properties were removed:
one three
--- -----
r1c1 r1c3
one two
--- ---
r2c1 r2c2
Remove-NullProperties source code:
<#
.SYNOPSIS
Removes properties with $null values from custom-object copies of
the input objects.
.DESCRIPTION
Note that output objects are custom objects that are copies of the input
objects with copies of only those input-object properties that are not $null.
CAVEAT: If you pipe multiple objects to this function, and these objects
differ in what properties are non-$null-valued, the default output
format will show only the non-$null-valued properties of the FIRST object.
Use ... | ForEach-Object { Out-String -InputObject $_ } to avoid
this problem.
.NOTES
Since the output objects are generally of a distinct type - [pscustomobject] -
and have only NoteProperty members, use of this function only makes sense
with plain-old data objects as input.
.EXAMPLE
> [pscustomobject] #{ one = 1; two = $null; three = 3 } | Remove-NullProperties
one three
--- -----
1 3
#>
function Remove-NullProperties {
param(
[parameter(Mandatory,ValueFromPipeline)]
[psobject] $InputObject
)
process {
# Create the initially empty output object
$obj = [pscustomobject]::new()
# Loop over all input-object properties.
foreach($prop in $InputObject.psobject.properties) {
# If a property is non-$null, add it to the output object.
if ($null -ne $InputObject.$($prop.Name)) {
Add-Member -InputObject $obj -NotePropertyName $prop.Name -NotePropertyValue $prop.Value
}
}
# Give the output object a type name that reflects the type of the input
# object prefixed with 'NonNull.' - note that this is purely informational, unless
# you define a custom output format for this type name.
$obj.pstypenames.Insert(0, 'NonNull.' + $InputObject.GetType().FullName)
# Output the output object.
$obj
}
}
These answers didn't work for me in the case of importing an object from an Infoblox csv file. Some values were the empty string, but not null. Testing whether a property is true or not, seems to work better for me. And the result is an object.
$a = [pscustomobject]#{one='hi';two='';three='there'}
$prop = $a.psobject.Properties | where value | foreach name
$a | select $prop
one three
--- -----
hi there
You first get its properties (since Get-QADUser depends on AD schema, the properties list is dynamic) with get-member -type property, then filter out those that don't have \{.*(get).*\} in its definition (that is, they are not "gettable"), then enumerate the resultant list by name and filter out nulls.
$someone=Get-QADUser -Identity "SomeOne" -IncludeAllProperties
$members=$someone|get-member -type property| where {$_.definition -match '\{.*(get).*\}'}
foreach ($member in $members) {
if ($someone[$member.name] -ne $null) {
write-host $member.name $someone[$member.name]
}
}
How do I get properties that ONLY have populated values?
So for example if I run
Get-QADUser -Identity "SomeOne" -IncludeAllProperties
the output would of course include.. all properties, including those with and those without values. I want a listing of properties with values only. How is it done generally speaking?
This wouldn't be restricted to Quest Cmdlets, I only use Get-QADUser as an example.
You could try using the built-in (hidden) property of PowerShell objects called PSObject, which includes a property called Properties, i.e. a list of all properties on the parent object.
Maybe easier with an example. Take Get-Process... a process can have many attributes (properties) with or without values. In order to get just the ones with values you do this:
(Get-Process | Select -First 1).PSObject.Properties | ?{$_.Value -ne $null} | FT Name,Value
Note that I limited this to just the first process returned by Get-Process. We then get all the properties defined on that object, filtering where Value is not null and then displaying just the Name and Value for those properties.
To complement Charlie Joynt's helpful answer:
Below is convenience function Remove-NullProperties, which creates custom-object copies of its input objects populated with only the non-$null properties of the input objects.
Example use:
# Sample input collection, with 2 objects with different $null-valued
# properties.
$coll = [pscustomobject] #{ one = 'r1c1'; two = $null; three = 'r1c3' },
[pscustomobject] #{ one = 'r2c1'; two = 'r2c2'; three = $null }
# Output copies containing only non-$null-valued properties.
# NOTE: The `ForEach-Object { Out-String -InputObject $_ }` part is solely
# there to ensure that *all* resulting properties are shown.
# With the default output, only the properties found on the FIRST
# input object would be used in the output table.
$coll | Remove-NullProperties |
ForEach-Object { Out-String -InputObject $_ }
This yields the following - note how the respective null-valued properties were removed:
one three
--- -----
r1c1 r1c3
one two
--- ---
r2c1 r2c2
Remove-NullProperties source code:
<#
.SYNOPSIS
Removes properties with $null values from custom-object copies of
the input objects.
.DESCRIPTION
Note that output objects are custom objects that are copies of the input
objects with copies of only those input-object properties that are not $null.
CAVEAT: If you pipe multiple objects to this function, and these objects
differ in what properties are non-$null-valued, the default output
format will show only the non-$null-valued properties of the FIRST object.
Use ... | ForEach-Object { Out-String -InputObject $_ } to avoid
this problem.
.NOTES
Since the output objects are generally of a distinct type - [pscustomobject] -
and have only NoteProperty members, use of this function only makes sense
with plain-old data objects as input.
.EXAMPLE
> [pscustomobject] #{ one = 1; two = $null; three = 3 } | Remove-NullProperties
one three
--- -----
1 3
#>
function Remove-NullProperties {
param(
[parameter(Mandatory,ValueFromPipeline)]
[psobject] $InputObject
)
process {
# Create the initially empty output object
$obj = [pscustomobject]::new()
# Loop over all input-object properties.
foreach($prop in $InputObject.psobject.properties) {
# If a property is non-$null, add it to the output object.
if ($null -ne $InputObject.$($prop.Name)) {
Add-Member -InputObject $obj -NotePropertyName $prop.Name -NotePropertyValue $prop.Value
}
}
# Give the output object a type name that reflects the type of the input
# object prefixed with 'NonNull.' - note that this is purely informational, unless
# you define a custom output format for this type name.
$obj.pstypenames.Insert(0, 'NonNull.' + $InputObject.GetType().FullName)
# Output the output object.
$obj
}
}
These answers didn't work for me in the case of importing an object from an Infoblox csv file. Some values were the empty string, but not null. Testing whether a property is true or not, seems to work better for me. And the result is an object.
$a = [pscustomobject]#{one='hi';two='';three='there'}
$prop = $a.psobject.Properties | where value | foreach name
$a | select $prop
one three
--- -----
hi there
You first get its properties (since Get-QADUser depends on AD schema, the properties list is dynamic) with get-member -type property, then filter out those that don't have \{.*(get).*\} in its definition (that is, they are not "gettable"), then enumerate the resultant list by name and filter out nulls.
$someone=Get-QADUser -Identity "SomeOne" -IncludeAllProperties
$members=$someone|get-member -type property| where {$_.definition -match '\{.*(get).*\}'}
foreach ($member in $members) {
if ($someone[$member.name] -ne $null) {
write-host $member.name $someone[$member.name]
}
}
I find myself continually faced with the need to store mixed-type data in some kind of a structure for later lookup.
For a recent example, I am performing data migration and I will store the old UUID, new UUID, source environment, target environment, and schema for an unknown number of entries.
I have been meeting this need by creating an array and inserting System.Objects with NoteProperty members for each of the columns of data.
This strikes me as a very clumsy approach but I feel like I may be limited by Powershell's functionality. If I need to, for example, locate all entries that used a particular schema, I write a foreach loop that sticks each entry with a matching schema name in a whole new array that I can return. I would really like the ability to more easily search for all objects that contain a member matching a particular value, modify existing members, etc.
Is there a better built-in data structure that will suit my needs, or is creating a custom object the right thing to do?
For reference, I'm doing something like this to create my structure:
$objectArray= #();
foreach(thing to process){
$tempObj = New-Object System.Object;
$tempObj | Add-Member -MemberType NoteProperty -Name "membername" -Value xxxxx
....repeat for each member...
$objectArray += $tempObj
}
If I need to find something in it, I then have to:
$matchingObjs = #()
foreach ($obj in $objectArray){
if($obj.thing -eq value){$matchingObjs += $obj}
}
This really sucks and I know there has to be a more elegant way. I'm still fairly new to powershell so I don't know what utilities it has to help me. I'm using v5.
With PowerShell 3.0 you could use a [PSCustomObject], here's an article on the different object creation methods.
Also setting the array equal to the output of the foreach loop will be more efficient than repeatedly recreating an array with +=.
$objectArray = foreach ($item in $collection) {
[pscustomobject]#{
"membername" = "xxxxx"
}
}
The Where-Object cmdlet or the .where() method looks like what you need in your second loop.
$matchingObjs = $objectArray | Where-Object {$_.thing -eq "value"}
It also sounds like you could use Where-Object/.where() to filter the initial data and just create an object which matches what you are looking for. For example:
$matchingObjs = $InputData |
Where-Object {$_.thing -eq "value"} |
ForEach-Object {
[pscustomobject]#{
"membername" = xxxxx
}
}
If your data can be expressed as key value pairs, then a hashtable will be the most efficient, see about_Hash_Tables for more info.
There is no built-in way to do what you are asking. One way is to segment your data into separate hashtables so you can do easy lookups by a common key, say the ID.
# Create a hastable for the IDs
$ids = #{};
foreach(thing to process){
$ids.Add($uid, 'Value')
}
# Find the $uid exists
$keyExists = $ids.Keys -Contains $uid
# Find value of stored for $uid
$keyValue = $ids[$uid]
As a side note, you don't have to create Syste.Object, you can simple do this:
$objectArray = #();
gci | % {
$objectArray += #{
'Key1' = 'Value 1'
'Key2' = 'Value 2'
}
}
If you need to compare complex objects, you can build them with #{} and then use Compare-Object on the two objects, just another idea.
For example, this will get a file listing of two different directories, and tell me what file exists or doesn't exist between the two directories:
$packages = (gci $boxStarterRepo -Recurse *.nuspec | Select-Object -ExpandProperty Name) -replace '.nuspec', ''
$packages += (gci $boxStarterPrivateRepo -Recurse *.nuspec | Select-Object -ExpandProperty Name) -replace '.nuspec', ''
$packages = $packages | Sort-Object
Compare-Object $packages $done