Exporting multiple PSCustomObject s to a single CSV - powershell

I have 2 PScustomObjects that i would like to combine into a single csv as follows:
$Output1 = [PSCustomObject]#{
Timestamp1 = (Get-Date)
InstanceName1 = $ProcessName1
Count1 = $Processes1.Count
Memory1 = $MEMLoad1
CPU1 = $CPULoad1
}
$Output2 = [PSCustomObject]#{
Timestamp2 = (Get-Date)
InstanceName2 = $ProcessName2
Count2 = $Processes2.Count
Memory2 = $MEMLoad2
CPU2 = $CPULoad2
}
The CSV should have the titles TimeStamp1, InstanceName1......TimeStamp2, InstanceName2....
This code runs in a loop and exports data each pass.
Is there a way to do this? Also is there a way to do this dynamically to avoid replicating large amounts of code if i am to export data on say 100 PsCustomObjects, maybe lookping through the input data for the object and the putting in in one object and passing that to the csv while dynamically changing titles?
I use the following line to export. I've tried -InputObject $Output1, $Output2 but that gives gibberish.
Export-Csv -InputObject $Output1 -path $Path1 -NoTypeInformation -Append -Force
The only solution i have so far is to export multiple CSV's but that gets bulky fast.
Any help is greatly appreciated!

The best solution is to ensure all the objects your script/function outputs use the same schema (that is, they should all share the same common property names).
If that's not possible, you can use Select-Object to ensure all possible properties are selected for all objects:
$Output1,$Output2 |Select-Object Count1,Count2,CPU1,CPU2,InstanceName1,InstanceName2,Memory1,Memory2,Timestamp1,Timestamp2 |Export-Csv ...
This is obviously not very practical, so if you have many different object schemas, you'll want to automate the discovery of all possible property names.
To do that, we can inspect the psobject hidden memberset:
$allTheObjectsToExport = $Output1,$Output2
$propertyNames = $allTheObjectsToExport |ForEach-Object {
# Discover all the object's properties
$_.psobject.Properties |ForEach-Object {
# Get each property name
$_.Name
}
} |Sort-Object -Unique
# Now the CSV will have a column for every single unique property name
$allTheObjectsToExport |Select-Object $propertyNames |Export-Csv -Path $Path1 -NoTypeInformation

Related

How to merge 2 x CSVs with the same column but overwrite not append?

I've got this one that has been baffling me all day, and I can't seem to find any search results that match exactly what I am trying to do.
I have 2 CSV files, both of which have the same columns and headers. They look like this (shortened for the purpose of this post):
"plate","labid","well"
"1013740016604537004556","none46","F006"
"1013740016604537004556","none47","G006"
"1013740016604537004556","none48","H006"
"1013740016604537004556","3835265","A007"
"1013740016604537004556","3835269","B007"
"1013740016604537004556","3835271","C007"
Each of the 2 CSVs only have some actual Lab IDs, and the 'nonexx' are just fillers for the importing software. There is no duplication ie each 'well' is only referenced once across the 2 files.
What I need to do is merge the 2 CSVs, for example the second CSV might have a Lab ID for well H006 but the first will not. I need the lab ID from the second CSV imported into the first, overwriting the 'nonexx' currently in that column.
Here is my current code:
$CSVB = Import-CSV "$RootDir\SymphonyOutputPending\$plateID`A_Header.csv"
Import-CSV "$RootDir\SymphonyOutputPending\$plateID`_Header.csv" | ForEach-Object {
$CSVData = [PSCustomObject]#{
labid = $_.labid
well = $_.well
}
If ($CSVB.well -match $CSVData.wellID) {
write-host "I MATCH"
($CSVB | Where-Object {$_.well -eq $CSVData.well}).labid = $CSVData.labid
}
$CSVB | Export-CSV "$RootDir\SymphonyOutputPending\$plateID`_final.csv" -NoTypeInformation
}
The code runs but doesn't 'merge' the data, the final CSV output is just a replication of the first input file. I am definitely getting a match as the string "I MATCH" appears several times when debugging as expected.
Based on the responses in the comments of your question, I believe this is what you are looking for. This assumes that the both CSVs contain the exact same data with labid being the only difference.
There is no need to modify csv2 if we are just grabbing the labid to overwrite the row in csv1.
$csv1 = Import-Csv C:\temp\LabCSV1.csv
$csv2 = Import-Csv C:\temp\LabCSV2.csv
# Loop through csv1 rows
Foreach($line in $csv1) {
# If Labid contains "none"
If($line.labid -like "none*") {
# Set rows labid to the labid from csv2 row that matches plate/well
# May be able to remove the plate section if well is a unique value
$line.labid = ($csv2 | Where {$_.well -eq $line.well -and $_.plate -eq $line.plate}).labid
}
}
# Export to CSV - not overwrite - to confirm results
$csv1 | export-csv C:\Temp\LabCSV1Adjusted.csv -NoTypeInformation
Since you need to do a bi-directional comparison of the 2 Csvs you could create a new array of both and then group the objects by their well property, for this you can use Group-Object, then filter each group if their Count is equal to 2 where their labid property does not start with none else return the object as-is.
Using the following Csvs for demonstration purposes:
Csv1
"plate","labid","well"
"1013740016604537004556","none46","F006"
"1013740016604537004556","none47","G006"
"1013740016604537004556","3835265","A007"
"newrowuniquecsv1","none123","X001"
Csv2
"plate","labid","well"
"1013740016604537004556","none48","A007"
"1013740016604537004556","3835269","F006"
"1013740016604537004556","3835271","G006"
"newrowuniquecsv2","none123","X002"
Code
Note that this code assumes there will be a maximum of 2 objects with the same well property and, if there are 2 objects with the same well, one of them must have a value not starting with none.
$mergedCsv = #(
Import-Csv pathtocsv1.csv
Import-Csv pathtocsv2.csv
)
$mergedCsv | Group-Object well | ForEach-Object {
if($_.Count -eq 2) {
return $_.Group.Where{ -not $_.labid.StartsWith('none') }
}
$_.Group
} | Export-Csv pathtomerged.csv -NoTypeInformation
Output
plate labid well
----- ----- ----
1013740016604537004556 3835265 A007
1013740016604537004556 3835269 F006
1013740016604537004556 3835271 G006
newrowuniquecsv1 none123 X001
newrowuniquecsv2 none123 X002
If the lists are large, performance might be an issue as Where-Object (or any other where method) and Group-Object do not perform very well for embedded loops.
By indexing the second csv file (aka creating a hashtable), you have quicker access to the required objects. Indexing upon two (or more) items (plate and well) is issued here: Does there exist a designated (sub)index delimiter? and resolved by #mklement0 and zett42 with a nice CaseInsensitiveArrayEqualityComparer class.
To apply this class on Drew's helpful answer:
$csv1 = Import-Csv C:\temp\LabCSV1.csv
$csv2 = Import-Csv C:\temp\LabCSV2.csv
$dict = [hashtable]::new([CaseInsensitiveArrayEqualityComparer]::new())
$csv2.ForEach{ $dict.($_.plate, $_.well) = $_ }
Foreach($line in $csv1) {
If($line.labid -like "none*") {
$line.labid = $dict.($line.plate, $line.well).labid
}
}
$csv1 | export-csv C:\Temp\LabCSV1Adjusted.csv -NoTypeInformation

Compare PSCustomObject to Object

I have created a PsCustomObject, when the variable is called is ISE, it reads a table of the relevant data. However, if I try to compare the PsCustomObject with another object, the PsCustomObject doesn't read correctly. I'd like to tell the script if any of the lines in the existing CSV match the PSCustomObject do not export the data to the CSV, in other words skip duplicate rows in the CSV file. The CSV may or may not have multiple rows.
$fileInfo = #(
[pscustomobject]#{
user_id = $user
studio = $studio
function = $Task
end_time_local = $creationTime
asin = $ASIN
variant = $variant
process_class_id = $processClass
}
)
$currentData = Import-Csv "$scansFolder\$fileName.csv"
if($fileInfo -ne $currentData){
$fileInfo | Export-Csv "$scansFolder\$fileName.csv" -Append -NoTypeInformation -Force
}
[pscustomobject] is a .NET reference type, so comparing two instances[1] with -eq will test for reference equality (identity), i.e. if the two instances are one and the same object[2] - which is obviously not the case in your scenario.
Assuming that the properties of your custom objects are instances of value types or strings (which appears to be the case), you can use Compare-Object to compare objects by their property values, with the ability to compare two collections:
$fileInfo = #(
[pscustomobject]#{
user_id = $user
studio = $studio
function = $Task
end_time_local = $creationTime
asin = $ASIN
variant = $variant
process_class_id = $processClass
}
)
# Get the property names.
# This assumes that the CSV data has (at least) the same
# set of properties (columns).
$propNames = $fileInfo[0].psobject.properties.Name
$currentData = Import-Csv "$scansFolder\$fileName.csv"
# Compare the $fileInfo custom object(s) to the custom objects read
# from the CSV file and only export those that are unique to the RHS ('=>')
# back to the file, i.e., those that don't match $fileInfo.
Compare-Object -Property $propNames $fileInfo $currentData |
Where-Object SideIndicator -eq '=>' | Select-Object InputObject |
Export-Csv "$scansFolder\$fileName.csv" -Append -NoTypeInformation -Force
[1] Import-Csv outputs [pscustomobject] instances too.
[2] See the Equality Comparison help topic (written for C#, but applies analogously to PowerShell's -eq operator).

Modifying Column Within Array

I'm reading in a CSV file which contains 25,000 records, and am reading each column into a psobject. Here is what I have so far:
$file = Import-CSV .\server.csv
$tempobj = New-Object psobject -Property #{
'Name' = $file.Name
'Group' = $file.Group
}
When this is ran, I get the correct results I want, being that $file.Name contains all the server names, and $file.Group contains the groups for servers. However, my issue is that I need to edit the names of each server without interfering with the .Group. Here is an example of what a server name look like as is.
WindowsAuthServer #{wdk9870WIN}
I need to remove WindowsAuthServer #{ and WIN} from each server name, leaving only the server name left, or for this example, wdk9870.
I tried using the -replace function ($tempobj.Name -replace "WindowsAuthServer #{",""), but it requires that I save the results to a new array, which then messes up or removes .Group entirely
Is there a different way to go about doing this? I'm lost.
Suppose your server.csv looks like this:
"Name","Group"
"WindowsAuthServer #{wdk9870WIN}","Group1"
"WindowsAuthServer #{wdk9880WIN}","Group2"
"WindowsAuthServer #{wdk9890WIN}","Group1"
"WindowsAuthServer #{wdk9900WIN}","Group1"
And you want to change the values in the Name column only, then something like this would probably do it:
Import-Csv .\server.csv | ForEach-Object {
New-Object psobject -Property #{
'Name' = ([regex]'#\{(\w+)WIN\}').Match($_.Name).Groups[1].Value
'Group' = $_.Group
}
}
This will output:
Name Group
---- -----
wdk9870 Group1
wdk9880 Group2
wdk9890 Group1
wdk9900 Group1
If you want, you can simply pipe this info to the Export-Csv cmdlet to save as a new CSV file. For that, just append | Export-Csv -Path .\server_updated.csv -NoTypeInformation to the code.
Hope that helps

Export results of (2) cmdlets to separate columns in the same CSV

I'm new to PS, so your patience is appreciated.
I'm trying to grab data from (2) separate CSV files and then dump them into a new CSV with (2) columns. Doing this for (1) is easy, but I don't know how to do it for more.
This works perfectly:
Import-CSV C:\File1.csv | Select "Employee" | Export-CSV -Path D:\Result.csv -NoTypeInformation
If I add another Import-CSV, then it simply overwrites the existing data:
Import-CSV C:\File2.csv | Select "Department" | Export-CSV -Path D:\Result.csv -NoTypeInformation
How can I get columns A and B populated with the info result from these two commands? Thanks for your help.
I would have choose this option:
$1 = Import-Csv -Path "C:\Users\user\Desktop\1.csv" | Select "Employee"
$2 = Import-Csv -Path "C:\Users\user\Desktop\2.csv" | Select "Department"
$marged = [pscustomobject]#()
$object = [pscustomobject]
for ($i=0 ; $i -lt $1.Count ; $i++){
$object = [pscustomobject]#{
Employees = $1[$i].Employee
Department = $2[$i].Department}
$marged += $object
}
$marged | ForEach-Object{ [pscustomobject]$_ } | Export-Csv -Path "C:\Users\user\Desktop\3.csv" -NoTypeInformation -Force
I'll explain how I would do this, but I do it this way because I'm more comfortable working with objects than with hastables. Someone else may offer an answer using hashtables which would probably work better.
First, I would define an array to hold your data, which can later be exported to CSV:
$report = #()
Then, I would import your CSV to an object that can be iterated through:
$firstSet = Import-CSV .\File1.csv
Then I would iterate through this, importing each row into an object that has the two properties I want. In your case these are Employee and Department (potentially more which you can add easily).
foreach($row in $firstSet)
{
$employeeName = $row.Employee
$employee = [PSCustomObject]#{
Employee = $employee
Department = ""
}
$report += $employee
}
And, as you can see in the example above, add this object to your report.
Then, import the second CSV file into a second object to iterate through (for good form I would actually do this at the begining of the script, when you import your first one):
$secondSet = Import-CSV .\File2.csv
Now here is where it gets interesting. Based on just the information you have provided, I am assuming that all employees in the one file are in the same order as the departments in the other files. So for example, if I work for the "Cake Tasting Department", and my name is on row 12 of File 1, row 12 of File 2 says "Cake Tasting Department".
In this case it's fairly easy. You would just roll through both lists and update the report:
$i = 0
foreach($row in $secondSet)
{
$dept = $row.Department
$report[i].Department = $dept
$i++
}
After this, your $report object will contain all of your employees in one row and departments in the other. Then you can export it to CSV:
$report | Export-CSV .\Result.csv -NoTypeInformation
This works if, as I said, your data aligns across both files. If not, then you need to get a little fancier:
foreach($row in $secondSet)
{
$emp = $row.Employee
$dept = $row.Department
$report | Where {$_.Employee -eq $emp} foreach {$_.Department = $dept
}
Technically you could just do it this way anyway, but it depends on a lot of things. First of all whether you have the data to match in that column across both files (which obviously in my example you don't otherwise you wouldn't need to do this in the first place, but you could match across other fields you may have, like EmployeeID or DoB). Second, on the sovereignty of individual records (e.g., if you have multiple matching records in your first file, you will have a problem; you would expect duplicates in the second as there are more than one person in each department).
Anyway, I hope this helps. As I said there is probably a 'better' way to do this, but this is how I would do it.

Adding columns and manipulating existing column values in csv file using powershell

I have a lot of csv files with values arranged like so:
X1,Y1
X2,Y2
...,...
Xn,Yn
I find it very tedious processing these with excel, so I want to setup a batch script to process these files such that they appear like this:
#where N is a specified value like 65536
X1,N-Y1,1
X2,N-Y2,2
...,...,...
Xn,N-Yn,n
I have only recently started using powershell for image processing (really simple scripts) and file name appending, so I am not certain how to go about this. A lot of the scripts I have encountered looking to answer this question use csv files with titles per column whereas my files are just arrays of values without object titles in the first row. I would like to avoid running multiple scripts to add titles.
My bonus question is something I have yet to find a good answer to at all, and is the most tedious part of processing. Using excels sort function, I usually change the order of the Yn values in Col2 such that they are sorted in the exported csv like so:
X1,N-Yn,n
...,...,...
Xn-1,N-Y2,2
Xn,N-Y1,1
Using the Col3 values as the sorting order (largest to smallest), then I delete this column so that the final saved csv only contains the first two columns (crucial step). Any help at all would be greatly appreciated, I apologize for the long-winded-ness of this question.
I have encountered looking to answer this question use csv files with titles per column whereas my files are just arrays of values without object titles in the first row.
The -Header parameter of Import-Csv is for adding column headers when the file does not contain them. It takes an array of strings, of however many columns there are.
I would like to avoid running multiple scripts to add titles.
If you couldn't use -Header, you could read the lines with Get-Content into memory, add a header in memory, and then use ConvertFrom-CSV all in one script.
That said, if I'm reading it rightly, you want:
No headers in the input file, and I imagine no headers in the output file
The whole point of adding the third column and sorting and removing it is just to reverse the lines?
The only column you keep is column 1?
I wouldn't use Import-Csv for this, it won't make it much nicer.
$n = 65536
# Read lines into a list, and reverse it
$lines = [Collections.Generic.List[String]](Get-Content -LiteralPath 'c:\test\test.csv')
$lines.Reverse()
# Split each line into two, create a new line with X and N-Y
# write new lines to an output file
$lines | ForEach-Object {
$x, $y = $_.split(',')
"$x,$($n - [int]$y)"
} | Set-Content -LiteralPath 'c:\test\output.csv' -Encoding Ascii
If you do want to use CSV handling, then:
$n = 65536
$counter = 1
Import-Csv -LiteralPath 'C:\test\test.csv' -Header 'ColX', 'ColY' |
Add-Member -MemberType ScriptProperty -Name 'ColN-Y' -Value {$n - $_.ColY} -PassThru |
Add-Member -MemberType ScriptProperty -Name 'N' -Value {$script:counter++} -PassThru |
Sort-Object -Property 'N' -Descending |
Select-Object -Property 'ColX', 'ColN-Y' |
Export-Csv -LiteralPath 'c:\test\output.csv' -NoTypeInformation
But the output will have CSV headers and double-quoted values.
I would try something like, by extending the original table with a calculatable script-property as a new column:
#Your N number
$N = 65536
# Import CSV file without header columns
$table = Import-Csv -Header #("colX","colY") `
-Delimiter ',' `
-Path './numbers.csv'
Write-Host "Original table"
$table | Format-Table
# Manipulate table
$newtable = $table |
Add-Member -MemberType ScriptProperty -Name colNX -Value { $N-$this.colX } - PassThru
Write-Host "New table"
$newtable | Format-Table