Export results of (2) cmdlets to separate columns in the same CSV - powershell

I'm new to PS, so your patience is appreciated.
I'm trying to grab data from (2) separate CSV files and then dump them into a new CSV with (2) columns. Doing this for (1) is easy, but I don't know how to do it for more.
This works perfectly:
Import-CSV C:\File1.csv | Select "Employee" | Export-CSV -Path D:\Result.csv -NoTypeInformation
If I add another Import-CSV, then it simply overwrites the existing data:
Import-CSV C:\File2.csv | Select "Department" | Export-CSV -Path D:\Result.csv -NoTypeInformation
How can I get columns A and B populated with the info result from these two commands? Thanks for your help.

I would have choose this option:
$1 = Import-Csv -Path "C:\Users\user\Desktop\1.csv" | Select "Employee"
$2 = Import-Csv -Path "C:\Users\user\Desktop\2.csv" | Select "Department"
$marged = [pscustomobject]#()
$object = [pscustomobject]
for ($i=0 ; $i -lt $1.Count ; $i++){
$object = [pscustomobject]#{
Employees = $1[$i].Employee
Department = $2[$i].Department}
$marged += $object
}
$marged | ForEach-Object{ [pscustomobject]$_ } | Export-Csv -Path "C:\Users\user\Desktop\3.csv" -NoTypeInformation -Force

I'll explain how I would do this, but I do it this way because I'm more comfortable working with objects than with hastables. Someone else may offer an answer using hashtables which would probably work better.
First, I would define an array to hold your data, which can later be exported to CSV:
$report = #()
Then, I would import your CSV to an object that can be iterated through:
$firstSet = Import-CSV .\File1.csv
Then I would iterate through this, importing each row into an object that has the two properties I want. In your case these are Employee and Department (potentially more which you can add easily).
foreach($row in $firstSet)
{
$employeeName = $row.Employee
$employee = [PSCustomObject]#{
Employee = $employee
Department = ""
}
$report += $employee
}
And, as you can see in the example above, add this object to your report.
Then, import the second CSV file into a second object to iterate through (for good form I would actually do this at the begining of the script, when you import your first one):
$secondSet = Import-CSV .\File2.csv
Now here is where it gets interesting. Based on just the information you have provided, I am assuming that all employees in the one file are in the same order as the departments in the other files. So for example, if I work for the "Cake Tasting Department", and my name is on row 12 of File 1, row 12 of File 2 says "Cake Tasting Department".
In this case it's fairly easy. You would just roll through both lists and update the report:
$i = 0
foreach($row in $secondSet)
{
$dept = $row.Department
$report[i].Department = $dept
$i++
}
After this, your $report object will contain all of your employees in one row and departments in the other. Then you can export it to CSV:
$report | Export-CSV .\Result.csv -NoTypeInformation
This works if, as I said, your data aligns across both files. If not, then you need to get a little fancier:
foreach($row in $secondSet)
{
$emp = $row.Employee
$dept = $row.Department
$report | Where {$_.Employee -eq $emp} foreach {$_.Department = $dept
}
Technically you could just do it this way anyway, but it depends on a lot of things. First of all whether you have the data to match in that column across both files (which obviously in my example you don't otherwise you wouldn't need to do this in the first place, but you could match across other fields you may have, like EmployeeID or DoB). Second, on the sovereignty of individual records (e.g., if you have multiple matching records in your first file, you will have a problem; you would expect duplicates in the second as there are more than one person in each department).
Anyway, I hope this helps. As I said there is probably a 'better' way to do this, but this is how I would do it.

Related

How to merge 2 x CSVs with the same column but overwrite not append?

I've got this one that has been baffling me all day, and I can't seem to find any search results that match exactly what I am trying to do.
I have 2 CSV files, both of which have the same columns and headers. They look like this (shortened for the purpose of this post):
"plate","labid","well"
"1013740016604537004556","none46","F006"
"1013740016604537004556","none47","G006"
"1013740016604537004556","none48","H006"
"1013740016604537004556","3835265","A007"
"1013740016604537004556","3835269","B007"
"1013740016604537004556","3835271","C007"
Each of the 2 CSVs only have some actual Lab IDs, and the 'nonexx' are just fillers for the importing software. There is no duplication ie each 'well' is only referenced once across the 2 files.
What I need to do is merge the 2 CSVs, for example the second CSV might have a Lab ID for well H006 but the first will not. I need the lab ID from the second CSV imported into the first, overwriting the 'nonexx' currently in that column.
Here is my current code:
$CSVB = Import-CSV "$RootDir\SymphonyOutputPending\$plateID`A_Header.csv"
Import-CSV "$RootDir\SymphonyOutputPending\$plateID`_Header.csv" | ForEach-Object {
$CSVData = [PSCustomObject]#{
labid = $_.labid
well = $_.well
}
If ($CSVB.well -match $CSVData.wellID) {
write-host "I MATCH"
($CSVB | Where-Object {$_.well -eq $CSVData.well}).labid = $CSVData.labid
}
$CSVB | Export-CSV "$RootDir\SymphonyOutputPending\$plateID`_final.csv" -NoTypeInformation
}
The code runs but doesn't 'merge' the data, the final CSV output is just a replication of the first input file. I am definitely getting a match as the string "I MATCH" appears several times when debugging as expected.
Based on the responses in the comments of your question, I believe this is what you are looking for. This assumes that the both CSVs contain the exact same data with labid being the only difference.
There is no need to modify csv2 if we are just grabbing the labid to overwrite the row in csv1.
$csv1 = Import-Csv C:\temp\LabCSV1.csv
$csv2 = Import-Csv C:\temp\LabCSV2.csv
# Loop through csv1 rows
Foreach($line in $csv1) {
# If Labid contains "none"
If($line.labid -like "none*") {
# Set rows labid to the labid from csv2 row that matches plate/well
# May be able to remove the plate section if well is a unique value
$line.labid = ($csv2 | Where {$_.well -eq $line.well -and $_.plate -eq $line.plate}).labid
}
}
# Export to CSV - not overwrite - to confirm results
$csv1 | export-csv C:\Temp\LabCSV1Adjusted.csv -NoTypeInformation
Since you need to do a bi-directional comparison of the 2 Csvs you could create a new array of both and then group the objects by their well property, for this you can use Group-Object, then filter each group if their Count is equal to 2 where their labid property does not start with none else return the object as-is.
Using the following Csvs for demonstration purposes:
Csv1
"plate","labid","well"
"1013740016604537004556","none46","F006"
"1013740016604537004556","none47","G006"
"1013740016604537004556","3835265","A007"
"newrowuniquecsv1","none123","X001"
Csv2
"plate","labid","well"
"1013740016604537004556","none48","A007"
"1013740016604537004556","3835269","F006"
"1013740016604537004556","3835271","G006"
"newrowuniquecsv2","none123","X002"
Code
Note that this code assumes there will be a maximum of 2 objects with the same well property and, if there are 2 objects with the same well, one of them must have a value not starting with none.
$mergedCsv = #(
Import-Csv pathtocsv1.csv
Import-Csv pathtocsv2.csv
)
$mergedCsv | Group-Object well | ForEach-Object {
if($_.Count -eq 2) {
return $_.Group.Where{ -not $_.labid.StartsWith('none') }
}
$_.Group
} | Export-Csv pathtomerged.csv -NoTypeInformation
Output
plate labid well
----- ----- ----
1013740016604537004556 3835265 A007
1013740016604537004556 3835269 F006
1013740016604537004556 3835271 G006
newrowuniquecsv1 none123 X001
newrowuniquecsv2 none123 X002
If the lists are large, performance might be an issue as Where-Object (or any other where method) and Group-Object do not perform very well for embedded loops.
By indexing the second csv file (aka creating a hashtable), you have quicker access to the required objects. Indexing upon two (or more) items (plate and well) is issued here: Does there exist a designated (sub)index delimiter? and resolved by #mklement0 and zett42 with a nice CaseInsensitiveArrayEqualityComparer class.
To apply this class on Drew's helpful answer:
$csv1 = Import-Csv C:\temp\LabCSV1.csv
$csv2 = Import-Csv C:\temp\LabCSV2.csv
$dict = [hashtable]::new([CaseInsensitiveArrayEqualityComparer]::new())
$csv2.ForEach{ $dict.($_.plate, $_.well) = $_ }
Foreach($line in $csv1) {
If($line.labid -like "none*") {
$line.labid = $dict.($line.plate, $line.well).labid
}
}
$csv1 | export-csv C:\Temp\LabCSV1Adjusted.csv -NoTypeInformation

Exporting multiple PSCustomObject s to a single CSV

I have 2 PScustomObjects that i would like to combine into a single csv as follows:
$Output1 = [PSCustomObject]#{
Timestamp1 = (Get-Date)
InstanceName1 = $ProcessName1
Count1 = $Processes1.Count
Memory1 = $MEMLoad1
CPU1 = $CPULoad1
}
$Output2 = [PSCustomObject]#{
Timestamp2 = (Get-Date)
InstanceName2 = $ProcessName2
Count2 = $Processes2.Count
Memory2 = $MEMLoad2
CPU2 = $CPULoad2
}
The CSV should have the titles TimeStamp1, InstanceName1......TimeStamp2, InstanceName2....
This code runs in a loop and exports data each pass.
Is there a way to do this? Also is there a way to do this dynamically to avoid replicating large amounts of code if i am to export data on say 100 PsCustomObjects, maybe lookping through the input data for the object and the putting in in one object and passing that to the csv while dynamically changing titles?
I use the following line to export. I've tried -InputObject $Output1, $Output2 but that gives gibberish.
Export-Csv -InputObject $Output1 -path $Path1 -NoTypeInformation -Append -Force
The only solution i have so far is to export multiple CSV's but that gets bulky fast.
Any help is greatly appreciated!
The best solution is to ensure all the objects your script/function outputs use the same schema (that is, they should all share the same common property names).
If that's not possible, you can use Select-Object to ensure all possible properties are selected for all objects:
$Output1,$Output2 |Select-Object Count1,Count2,CPU1,CPU2,InstanceName1,InstanceName2,Memory1,Memory2,Timestamp1,Timestamp2 |Export-Csv ...
This is obviously not very practical, so if you have many different object schemas, you'll want to automate the discovery of all possible property names.
To do that, we can inspect the psobject hidden memberset:
$allTheObjectsToExport = $Output1,$Output2
$propertyNames = $allTheObjectsToExport |ForEach-Object {
# Discover all the object's properties
$_.psobject.Properties |ForEach-Object {
# Get each property name
$_.Name
}
} |Sort-Object -Unique
# Now the CSV will have a column for every single unique property name
$allTheObjectsToExport |Select-Object $propertyNames |Export-Csv -Path $Path1 -NoTypeInformation

How can I concatenate csv colums and rename their header with Powershell?

I am attempting to merge two csv files together and select only two of their columns for use in a new csv. I don't understand why I cannot use the code I have already:
$Temp1 = (Import-csv "C:\path\APPcsv.csv" -header "APP") |
select-object APP
$Temp2 = (Import-csv "C:\path\ALLdb42APPs.csv"-header "NA1", "NA2", "Applications", "NA3", "Project") |
select-object Project
$CSV= #($temp1, $temp2) |
export-csv -path "C:\path\Why isn't this working.csv" -noTypeInformation
Here is an example line from each CSV:
CSV1 (ALLdb42APPs.csv)
"Current Application","Calculation","AdobeReaderDC-18.011.20036 V1 - Add Instalation Status: SUCCESSFUL","2018-05-16 08:54:17","DK ATM error main"
CSV2 (APPcsv.csv)
"DameWareService-10.0.0.0-x64 V2 - Add"
So your issue is because #($temp1,$temp2) doesn't combine the first element of $temp1 with the first element of $temp2, but instead makes a new collection which is all of $temp1's objects followed by all of $temp2.
Since $temp1 is objects with an APP property and $temp2 is objects with a Project, combining these into a collection doesn't make sense to export to a csv.
If $temp1 is a bag of apples and $temp2 is a bag of oranges, #($temp1,$temp2) isn't holding the bags together, it's dumping both into one bag on top of each other.
You could either join the two objects into one. Warren Frame has a well respected module Join-Object that could be used as James C pointed out, but your two csvs would need to share a column.
The other alternative is to use a for loop, then in each iteration take the value from each collection and create a new object with both values.
$Temp1 = (Import-csv "C:\path\APPcsv.csv" -header "APP") |
Select-Object -ExpandProperty APP
$Temp2 = (Import-csv "C:\path\ALLdb42APPs.csv"-header "NA1", "NA2", "Applications", "NA3", "Project") |
Select-Object -ExpandProperty Project
$LargestIndex = [math]::Max($temp1.count,$temp2.count)
$CombinedArray = For ($i=0; $i -le $LargestIndex; $i++) {
[pscustomobject]#{
APP = $temp1[$i]
Project = $temp2[$i]
}
}
$CombinedArray |
Export-Csv -Path "C:\path\Example.csv" -NoTypeInformation
Note: requires PowerShell 3+ for the pscustomobject way of creating objects.

Compare 2 csv files and match based on 1 column then export new file that contains fields from both

I have 2 csv files. Each have different headers and different number of columns, and have different number of entries.
Here are some examples of the first couple lines
CSV 1
ID,Last_Name,First_Name,Middle_Name,Email_Addr,Title,Gender
###1,smith,bill,p,smith#soso.com,boss,m
###2,smith2,billy,p,smith2#soso.com,someguy,m
CSV 2
ID,Name Id,Last Name,First Name,Middle Name,Gender
###2,ID1010,smith2,billy,p,M
I am trying to import them and compare the ID column. When a match is found I am wanting a new csv file with All info from CSV 1 and the matched Name ID from csv 2.
New CSV Example:
ID,Last_Name,First_Name,Middle_Name,Email_Addr,Title,Gender,Name Id
###1,smith,bill,p,smith#soso.com,boss,m,
###2,smith2,billy,p,smith2#soso.com,someguy,m,ID1010
Ive been looking and came across this Stackoverflow from about a year ago that seemed to be on the right track but I cant seem to get code modified for my needs. Here is what I have tried.
$csv1 = Import-Csv -Path C:\STAFF\test1sky.csv
$csv2 = Import-Csv -Path C:\STAFF\test1power.csv
ForEach($Record in $csv2){
$MatchedValue = (Compare-Object $csv1 $Record -Property "ID" -IncludeEqual -ExcludeDifferent -PassThru).value
$Record = Add-Member -InputObject $Record -Type NoteProperty -Name "Name Id" -Value $MatchedValue
}
$csv2|Export-Csv 'C:\STAFF\combined.csv' -NoTypeInformation
I get the correct header in the new file but I never get the Name ID values to come though.
Any idea where I went wrong? I maybe on the wrong path completely and there be a easier way, but I need to be able to do this nightly without user interaction. Any help is appreciated!!
Let's try to simplify this. Add the 'Name ID' field to all records in CSV1. Then loop through it, and get the matches, and update the field. Something like:
$CSV1 = C:\Path\To\File1.csv
$CSV2 = C:\Path\To\File2.csv
$CSV1|ForEach{$_|Add-Member 'Name ID' $Null}
ForEach($Record in $CSV1){
$Record.'Name ID' = $CSV2|Where{$_.ID -eq $Record.ID}|Select -Expand 'Name ID'
}
$CSV1 = import-csv C:\Path\To\File1.csv
$CSV2 = import-csv C:\Path\To\File2.csv
#adds a row named "Name ID" to the PS Object( the CSV Import)
$CSV1|ForEach{$_|Add-Member 'Name ID' $Null}
ForEach($Record in $CSV1){
#gets the value from CSV1 for comparing to CSV2
$NameValue=Record."Last_Name"
#gets the Power Shell Object from the CSV2 Import that matches the Name ID from $csv1
$Nameobject= $CSV2|Where-object "Last Name" -contains $Namevalue
#Sets the Field "Name ID" in the PS Object $CSV1 Record to the Name ID from $csv2
$record."Name ID" = $Nameobject."Name ID"
}
You can easily grab addtional fields by adding other references to the CSV1 File by manipulating the CSV2 PS Object.
$record."Middle Name" = $nameobject."Middle_Name"
Since you have the entire object in the for loop form $csv2 you can call any of its fields or manipulate them by using variables and " |select -Property "Value" Like this
$objlength = $nameobject |select "First_Name"
$objlength.length
but i prefer to call it directly from the object as the output looks cleaner like this
$nameobject."First_Name".length
The operation you are looking for is called a relational join. Sometimes it's called an inner join, and sometimes just a join. My knowledge of join comes from SQL, not from Powershell.
Here's a description of "Join-Object". It seems to be what you are looking for.
http://blogs.msdn.com/b/powershell/archive/2012/07/13/join-object.aspx

Add Column to CSV Windows PowerShell

I have a fairly standard csv file with headers I want to add a new column & set all the rows to the same data.
Original:
column1, column2
1,b
2,c
3,5
After
column1, column2, column3
1,b, setvalue
2,c, setvalue
3,5, setvalue
I can't find anything on this if anybody could point me in the right direction that would be great. Sorry very new to Power Shell.
Here's one way to do that using Calculated Properties:
Import-Csv file.csv |
Select-Object *,#{Name='column3';Expression={'setvalue'}} |
Export-Csv file.csv -NoTypeInformation
You can find more on calculated properties here: http://technet.microsoft.com/en-us/library/ff730948.aspx.
In a nutshell, you import the file, pipe the content to the Select-Object cmdlet, select all exiting properties (e.g '*') then add a new one.
The ShayLevy's answer also works for me!
If you don't want to provide a value for each object yet the code is even easier...
Import-Csv file.csv |
Select-Object *,"column3" |
Export-Csv file.csv -NoTypeInformation
None of the scripts I've seen are dynamic in nature, so they're fairly limited in their scope & what you can do with them.. that's probably because most PS Users & even Power Users aren't programmers. You very rarely see the use of arrays in Powershell. I took Shay Levy's answer & improved upon it.
Note here: The Import needs to be consistent (two columns for instance), but it would be fairly easy to modify this to dynamically count the columns & generate headers that way too. For this particular question, that wasn't asked. Or simply don't generate a header unless it's needed.
Needless to say the below will pull in as many CSV files that exist in the folder, add a header, and then later strip it. The reason I add the header is for consistency in the data, it makes manipulating the columns later down the line fairly straight forward too (if you choose to do so). You can modify this to your hearts content, feel free to use it for other purposes too. This is generally the format I stick with for just about any of my Powershell needs. The use of a counter basically allows you to manipulate individual files, so there's a lot of possibilities here.
$chargeFiles = 'C:\YOURFOLDER\BLAHBLAH\'
$existingReturns = Get-ChildItem $chargeFiles
for ($i = 0; $i -lt $existingReturns.count; $i++)
{
$CSV = Import-Csv -Path $existingReturns[$i].FullName -Header Header1,Header2
$csv | select *, #{Name='Header3';Expression={'Header3 Static'}}
| select *, #{Name='Header4';Expression={'Header4 Static Tet'}}
| select *, #{Name='Header5';Expression={'Header5 Static Text'}}|
CONVERTTO-CSV -DELIMITER "," -NoTypeInformation |
SELECT-OBJECT -SKIP 1 | % {$_ -replace '"', ""} |
OUT-FILE -FilePath $existingReturns[$i].FullName -FORCE -ENCODING ASCII
}
You could also use Add-Member:
$csv = Import-Csv 'input.csv'
foreach ($row in $csv)
{
$row | Add-Member -NotePropertyName 'MyNewColumn' -NotePropertyValue 'MyNewValue'
}
$csv | Export-Csv 'output.csv' -NoTypeInformation
For some applications, I found that producing a hashtable and using the .values as the column to be good (it would allow for cross reference validation against another object that was being enumerated).
In this case, #powershell on freenode brought my attention to an ordered hashtable (since the column header must be used).
Here is an example without any validation the .values
$newcolumnobj = [ordered]#{}
#input data into a hash table so that we can more easily reference the `.values` as an object to be inserted in the CSV
$newcolumnobj.add("volume name", $currenttime)
#enumerate $deltas [this will be the object that contains the volume information `$volumedeltas`)
# add just the new deltas to the newcolumn object
foreach ($item in $deltas){
$newcolumnobj.add($item.volume,$item.delta)
}
$originalcsv = #(import-csv $targetdeltacsv)
#thanks to pscookiemonster in #powershell on freenode
for($i=0; $i -lt $originalcsv.count; $i++){
$originalcsv[$i] | Select-Object *, #{l="$currenttime"; e={$newcolumnobj.item($i)}}
}
Example is related to How can I perform arithmetic to find differences of values in two CSVs?
create a csv file with nothin in it
$csv >> "$PSScriptRoot/dpg.csv"
define the csv file's path. here $psscriptroot is the root of the script
$csv = "$PSScriptRoot/dpg.csv"
now add columns to it
$csv | select vds, protgroup, vlan, ports | Export-Csv $csv