I have a .csv file which looks like
ID, Dept
1,x
1,y
1,z
2,a
2,b
2,c
output should be
ID, Dept
1, x;y;z
2, a;b;c
I tried the with below in the PowerShell but it is returning 0 for both columns
$csvValues = Get-Content "DeptDetails.csv"
$duplicates = $csvValues | group-object ID | ? Count -gt 1
$objs = New-Object System.Collections.ArrayList
ForEach ($duplicate in $duplicates){
$objs.Add([pscustomobject]#{ID = ($duplicate.Group.ID | select -Unique) -as [int];
GroupName=($duplicate.Group.Dept | ? Length -gt 0) -join ';'})
}
$objs | Sort ID
Use the Import-Csv cmdlet to load your csv. Group the entries by its ID using the Group-Object cmdlet. Then you can iterate over the group using the Foreach-Object cmdlet and create your desired Object for each ID. Finally export it back to csv using the Export-Csv cmdlet:
Import-Csv 'DeptDetails.csv' | Group-Object ID | ForEach-Object {
[PsCustomObject]#{
ID = $_.Name
Dept = $_.Group.Dept -join ';'
}
} | Export-Csv 'DeptDetails_output.csv' -NoTypeInformation
Related
I have a .csv file which I'm grouping on two properties 'DN', 'SKU' and then performing a sum on another property 'DeliQty'
This works fine and the sum is reflected back to the group.
However I then need to re group just on 'DN' and write out to separate files.
I've tried Select-Object -Expand Group but this reverts to the original contents without the summed lines.
Is there a way to un group preserving the summed lines and then group again?
$CSVFiles = Get-ChildItem -Path C:\Scripts\INVENTORY\ASN\IMPORT\ -Filter *.csv
foreach ($csv in $CSVFiles) {
$group = Import-Csv $csv.FullName | Group-Object DN, SKU
$group | Where Count -gt 1 | ForEach-Object {
$_.Group[0].'DeliQty' = ($_.Group | Measure-Object DeliQty -Sum).Sum
}
}
You may do the following:
$CSVFiles = Get-ChildItem -Path C:\Scripts\INVENTORY\ASN\IMPORT\ -Filter *.csv
foreach ($csv in $CSVFiles) {
$group = Import-Csv $csv.FullName | Group-Object DN, SKU | Foreach-Object {
if ($_.Count -gt 1) {
$_.Group[0].DeliQty = ($_.Group | Measure-Object DeliQty -Sum).Sum
}
$_.Group[0]
}
# outputs summed and single group objects
$group
}
I have a csv file where I am trying to export rows into another csv file only where the values in the id column have duplicates.
I have the following csv file...
"id","blablah"
"valOne","valTwo"
"valOne","asdfdsa"
"valThree","valFour"
"valFive","valSix"
"valFive","qwreweq"
"valSeven","valEight"
I need the output csv file to look like the following...
"valOne","valTwo"
"valOne","asdfdsa"
"valFive","valSix"
"valFive","qwreweq"
Here is the code I have so far:
$inputCsv = Import-CSV './test.csv' -delimiter ","
#$output = #()
$inputCsv | Group-Object -prop id, blablah | Where-Object {$_.id -gt 1} |
Select-Object
##{n='id';e={$_.Group[0].id}},
##{n='blablah';e={$_.Group[0].blablah}}
#Export-Csv 'C:\scripts\powershell\output.csv' -NoTypeInformation
#Write-Host $output
#$output | Export-Csv 'C:\scripts\powershell\output.csv' -NoTypeInformation
I've searched multiple how-to's but can't seem to find the write syntax. Can anyone help with this?
Just group on the ID property and if there is more than 1 count in the group then expand those and export.
$inputCsv = Import-CSV './test.csv' -delimiter ","
$inputCsv |
Group-Object -Property ID |
Where-Object count -gt 1 |
Select-Object -ExpandProperty group |
Export-Csv output.csv -NoTypeInformation
output.csv will contain
"id","blablah"
"valOne","valTwo"
"valOne","asdfdsa"
"valFive","valSix"
"valFive","qwreweq"
I got a excel sheet ,which got values from server . I want to remove columns which have no value..I did that but resulted excel sheet column names are in alphabetical order.But I want them in desired order. If use Select-object to get desired order , it will give the removed columns again.Below is the code i used to remove blank columns.
$x = Import-Csv YourFile.csv
$f = $x[0] | Get-Member -MemberType NoteProperty | Select name
$f | Add-Member -Name count -Type NoteProperty -Value 0
$f | %{
$n = $_.Name
$_.Count = #($x | Select $n -ExpandProperty $n | ? {$_ -ne ''}).count
}
$f = #($f | ? {$_.count -gt 0} | Select Name -expandproperty Name)
$x | Select $f | Export-Csv NewFile.csv -NoTypeInformation
As Axel Andersen points out, Get-Member invariably returns an object's members alphabetically sorted.
Try the following approach, which uses .psobject.Properties.Name to get the column (property) names in the original order:
$rows = Import-Csv YourFile.csv
$columnNames = $rows[0].psobject.Properties.Name
$nonEmptyColumnNames = $columnNames.Where({ (#($rows.$_) -ne '').Count -gt 0 })
$rows | Select-Object $nonEmptyColumnNames | Export-Csv NewFile.csv -NoTypeInformation
I have this CSV file that I kind of do a lot to. My most recent task is to add a summary sheet.
With that said I have a CSV file I pull from a website and send through lot of checks. Code Below:
$Dups = import-csv 'C:\Working\cylrpt.csv' | Group-Object -Property 'Device Name'| Where-Object {$_.count -ge 2} | ForEach-Object {$_.Group} | Select #{Name="Device Name"; Expression={$_."Device Name"}},#{Name="MAC"; Expression={$_."Mac Addresses"}},Zones,#{Name="Agent"; Expression={$_."Agent Version"}},#{Name="Status"; Expression={$_."Is online"}}
$Dups | Export-CSV $working\temp\01-Duplicates.csv -NoTypeInformation
$csvtmp = Import-CSV $working\cylrpt.csv | Select #{N='Device';E={$_."Device Name"}},#{N='OS';E={$_."OS Version"}},Zones,#{N='Agent';E={$_."Agent Version"}},#{N='Active';E={$_."Is Online"}},#{N='Checkin';E={[DateTime]$_."Online Date"}},#{N='Checked';E={[DateTime]$_."Offline Date"}},Policy
$csvtmp | %{
if ($_.Zones -eq ""){$_.Zones = "Unzoned"}
}
$csvtmp | Export-Csv $working\cy.csv -NoTypeInformation
import-csv $working\cy.csv | Select Device,policy,OS,Zones,Agent,Active,Checkin,Checked | % {
$_ | Export-CSV -path $working\temp\$($_.Zones).csv -notypeinformation -Append
}
The first check is for duplicates, I used separate lines of code for this because I wanted to create a CSV for duplicates.
The second check backfills all blank cells under the Zones column with "UnZoned"
The third thing is does is goes through the entire CSV file and creates a CSV file for each Zone
So this is my base. I need to add another CSV file for a Summary of the Zone information. The Zones are in the format of XXX-WS or XXX-SRV, where XXX can be between 3 and 17 letters.
I would like the Summary sheet to look like this
ABC ###
ABC-WS ##
ABC-SRV ##
DEF ###
DEF-WS ##
DEF-SRV ##
My thoughts are to either do the count from the original CSV file or to count the number of lines in each CSV file and subtract 1, for the header row.
Now the Zones are dynamic so I can't just say I want ZONE XYZ, because that zone may not exist.
So what I need is to be able to either count the like zone type in the original file and either output that to an array or file, that would be my preferred method to give the number of items with the same zone name. I just don't know how to write it to look for and count matching variables. Here is the code I'm trying to use to get the count:
import-csv C:\Working\cylrpt.csv | Group-Object -Property 'Zones'| ForEach-Object {$_.Group} | Select #{N='Device';E={$_."Device Name"}},Zones | % {
$Znum = ($_.Zones).Count
If ($Znum -eq $null) {
$Znum = 1
} else {
$Znum++
}
}
$Count = ($_.Zones),$Znum | Out-file C:\Working\Temp\test2.csv -Append
Here is the full code minus the report key:
$cylURL = "https://protect.cylance.com/Reports/ThreatDataReportV1/devices/"
$working = "C:\Working"
Remove-item -literalpath "\\?\C:\Working\Cylance Report.xlsx"
Invoke-WebRequest -Uri $cylURL -outfile $working\cylrpt.csv
$Dups = import-csv 'C:\Working\cylrpt.csv' | Group-Object -Property 'Device Name'| Where-Object {$_.count -ge 2} | ForEach-Object {$_.Group} | Select #{Name="Device Name"; Expression={$_."Device Name"}},#{Name="MAC"; Expression={$_."Mac Addresses"}},Zones,#{Name="Agent"; Expression={$_."Agent Version"}},#{Name="Status"; Expression={$_."Is online"}}
$Dups | Export-CSV $working\temp\01-Duplicates.csv -NoTypeInformation
$csvtmp = Import-CSV $working\cylrpt.csv | Select #{N='Device';E={$_."Device Name"}},#{N='OS';E={$_."OS Version"}},Zones,#{N='Agent';E={$_."Agent Version"}},#{N='Active';E={$_."Is Online"}},#{N='Checkin';E={[DateTime]$_."Online Date"}},#{N='Checked';E={[DateTime]$_."Offline Date"}},Policy
$csvtmp | %{
if ($_.Zones -eq ""){$_.Zones = "Unzoned"}
}
$csvtmp | Export-Csv $working\cy.csv -NoTypeInformation
import-csv $working\cy.csv | Select Device,policy,OS,Zones,Agent,Active,Checkin,Checked | % {
$_ | Export-CSV -path $working\temp\$($_.Zones).csv -notypeinformation -Append
}
cd $working\temp;
Rename-Item "Unzoned.csv" -NewName "02-Unzoned.csv"
Rename-Item "Systems-Removal.csv" -NewName "03-Systems-Removal.csv"
$CSVFiles = Get-ChildItem -path $working\temp -filter *.csv
$Excel = "$working\Cylance Report.xlsx"
$Num = $CSVFiles.Count
Write-Host "Found the following Files: ($Num)"
ForEach ($csv in $CSVFiles) {
Write-host "Merging $CSVFiles.Name"
}
$EXc1 = New-Object -ComObject Excel.Application
$Exc1.SheetsInNewWorkBook = $CSVFiles.Count
$XLS = $EXc1.Workbooks.Add()
$Sht = 1
ForEach ($csv in $CSVFiles) {
$Row = 1
$Column = 1
$WorkSHT = $XLS.WorkSheets.Item($Sht)
$WorkSHT.Name = $csv.Name -Replace ".csv",""
$File = (Get-Content $csv)
ForEach ($line in $File) {
$LineContents = $line -split ',(?!\s*\w+")'
ForEach ($Cell in $LineContents) {
$WorkSHT.Cells.Item($Row,$Column) = $Cell -Replace '"',''
$Column++
}
$Column = 1
$Row++
}
$Sht++
}
$Output = $Excel
$XLS.SaveAs($Output)
$EXc1.Quit()
Remove-Item *.csv
cd ..\
Found the solution
$Zcount = import-csv C:\Working\cylrpt.csv | where Zones -ne "$null" | select #{N='Device';E={$_."Device Name"}},Zones | group Zones | Select Name,Count
$Zcount | Export-Csv -path C:\Working\Temp\01-Summary.csv -NoTypeInformation
I am trying to compare 2 csv files using 'user Id' column which is common in both the files and want to append the missing entries in csv 1 from csv 2 using power shell
If the first CSV file has missing rows, and they need to be copied in from the second CSV file directly:
ipcsv 2.csv |? 'User Id' -notin (ipcsv 1.csv).'User Id' | epcsv 1.csv -Append
If you're also saying that User IDs are unique in your files, you could instead take the unique rows out of the set of both CSVs into the first CSV:
(ipcsv 1.csv, 2.csv) | sort -Unique 'User Id' | epcsv 1.csv -NoTypeInformation
Alternatively, if you're saying that the files have the same User Ids but don't have the same other columns, and the entries in the first CSV are present but have some empty columns, then:
$csv1 = Import-Csv d:\file1.csv
$csv2 = Import-Csv d:\file2.csv
foreach ($csv1item in $csv1)
{
$csv2item = $csv2.Where{$_.'User Id' -eq $csv1item.'User Id'}
$item1Properties = $csv1item | Get-Member -MemberType NoteProperty | select -ExpandProperty Name
$item2Properties = $csv2item | Get-Member -MemberType NoteProperty | select -ExpandProperty Name
$sharedProperties = $item1Properties.where{$_ -in $item2Properties}
$sharedProperties | ForEach-Object {
$value = $csv1item."$_"
if ($value -eq '') {
$csv1item."$_" = $csv2item."$_"
}
}
}
$csv1 | Export-Csv D:\file1.csv -NoTypeInformation
Alternatively again, if you're saying the two files have different columns which overlap at 'User Id' and also the entries are missing from the first CSV entirely, rather than being present with missing columns, then this will add them from the second CSV, taking only the properties which overlap between them (hopefully):
$csv1 = Import-Csv d:\file1.csv
$csv2 = Import-Csv d:\file2.csv
$item1Properties = ($csv1[0] | gm -M NoteProperty).Name
$newCsv1Items = foreach($csv2item in $csv2.Where{$csv1.'User Id' -notcontains $_.'User ID'}) {
$newcsv1Item = #{}
$item1Properties.ForEach{$newcsv1Item."$_" = $csv2item."$_"}
[PSCustomObject]$newcsv1Item
}
$newCsv1Items | Export-Csv -Append D:\file1.csv