How do I code this to output the columns in this order:
unique_sis_group_id, unique_sis_user_id, unique_sis_school_id, mm_admin
My code:
$objs = #();
$output = Import-Csv -Path "c:\users\patrick\desktop\RelayFiles\RelayMemberships\file2concatenate.csv" | ForEach {
$Object = New-Object PSObject -Property #{
unique_sis_group_id = [String]::Concat($_.unique_sis_group_id, $_.unique_sis_user_id)
unique_sis_user_id = $_.unique_sis_school_id
unique_sis_school_id = $_.mm_admin
mm_admin = 0
}
$objs += $Object;
}
$objs
$objs | Export-Csv -NoTypeInformation c:\users\patrick\desktop\RelayFiles\RelayMemberships\memberships.csv
You assign the ForEach-Object to $output, but nothing inside actually outputs something.
Using a [PSCustomObject] is much simpler:
## Q:\Test\2019\01\25\SO_54373962.ps1
$CsvIn = "c:\users\patrick\desktop\RelayFiles\RelayMemberships\file2concatenate.csv"
$CsvOut = "c:\users\patrick\desktop\RelayFiles\RelayMemberships\memberships.csv"
$output = Import-Csv -Path $CsvIn | ForEach-Object {
[PSCustomObject]#{
unique_sis_group_id = [String]::Concat($_.unique_sis_group_id, $_.unique_sis_user_id)
unique_sis_user_id = $_.unique_sis_school_id
unique_sis_school_id = $_.mm_admin
mm_admin = 0
}
}
$output
$output | Export-Csv $CsvOut -NoTypeInformation
This took care of it. Is there a better way?
$objs =#();
$output = Import-Csv -Path "c:\users\patrick\desktop\RelayFiles\RelayMemberships\file2concatenate.csv" | ForEach {
$Object = New-Object PSObject -Property #{
unique_sis_group_id = [String]::Concat($_.unique_sis_group_id, $_.unique_sis_user_id)
unique_sis_user_id = $_.unique_sis_school_id
unique_sis_school_id = $_.mm_admin
mm_admin = 0
}
$objs += $Object;
}
$objs |
Select-Object "unique_sis_group_id", "unique_sis_user_id",
"unique_sis_school_id", "mm_admin" |
Export-CSv -NoTypeInformation c:\users\patrick\desktop\RelayFiles\RelayMemberships\memberships.csv
I'm trying to create a PowerShell script that transpose a CSV file from column to rows.
I found examples of doing the opposite (converting row based CSV to column) but I found nothing on column to rows. My problem being that I don't know exactly how many column I'll have. I tried adapting the row to column to column to rows but unsuccessfully.
$a = Import-Csv "input.csv"
$a | FT -AutoSize
$b = #()
foreach ($Property in $a.Property | Select -Unique) {
$Props = [ordered]#{ Property = $Property }
foreach ($Server in $a.Server | Select -Unique){
$Value = ($a.where({ $_.Server -eq $Server -and
$_.Property -eq $Property })).Value
$Props += #{ $Server = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
$b | FT -AutoSize
$b | Out-GridView
$b | Export-Csv "output.csv" -NoTypeInformation
For example my CSV can look like this:
"ID","DATA1"
"12345","11111"
"54321","11111"
"23456","44444"
or this (number of column can vary):
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555",
and I would like the script to convert it like this:
"ID","DATA"
"12345","11111"
"12345","22222"
"12345","33333"
"54321","11111"
"23456","44444"
"23456","55555"
The trick is to query the members of the table to get the column names. Once you do that then the rest is straightforward:
function Flip-Table ($Table) {
Process {
$Row = $_
# Get all the columns names, excluding the ID field.
$Columns = ($Row | Get-Member -Type NoteProperty | Where-Object Name -ne ID).Name
foreach ($Column in $Columns) {
if ($Row.$Column) {
$Properties = [Ordered] #{
"ID" = $Row.ID
"DATA" = $Row.$Column
}
New-Object PSObject -Property $Properties
}
}
# Garbage collection won't kick in until the end of the script, so
# invoke it every 100 input rows.
$Count++;
if (($Count % 100) -eq 0) {
[System.GC]::GetTotalMemory('forceFullCollection') | out-null
}
}
}
Import-Csv input.csv | Flip-Table | Export-Csv -NoTypeInformation output.csv
Well, here is mine. I'm not as fancy as the rest:
$in = Get-Content input.csv | Select -Skip 1
$out = New-Object System.Collections.ArrayList
foreach($row in $in){
$parts = $row.Split(',')
$id = $parts[0]
foreach($data in $parts[1..$parts.Count]){
if($data -ne '' -AND $data -ne $null){
$temp = New-Object PSCustomObject -Property #{'ID' = $id;
'Data' = $data}
$out.Add($temp) | Out-Null
}
}
}
$out | Export-CSV output.csv -NoTypeInformation
You can do something like this
# Convert csv to object
$csv = ConvertFrom-Csv #"
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555"
"#
# Ignore common members and the ID property
$excludedMembers = #(
'GetHashCode',
'GetType',
'ToString',
'Equals',
'ID'
)
$results = #()
# Iterate around each csv row
foreach ($row in $csv) {
$members = $row | Get-Member
# Iterate around each member from the 'row object' apart from our
# exclusions and empty values
foreach ($member in $members |
Where { $excludedMembers -notcontains $_.Name -and $row.($_.Name)}) {
# add to array of objects
$results += #{ ID=$row.ID; DATA=$row.($member.Name)}
}
}
# Write the csv string
$outstring = "ID,DATA"
$results | foreach { $outstring += "`n$($_.ID),$($_.DATA)" }
# New csv object
$csv = $outstring | ConvertFrom-Csv
Probably not the most elegant solution, but should do what you need
I left some comments explaining what it does
If you only want to accept a limited number DATA columns (e.g. 5), you could do:
ForEach ($i in 1..5) {$CSV | ? {$_."Data$i"} | Select ID, #{N='Data'; E={$_."Data$i"}}}
And if you have a potential unlimited number of DATA columns:
ForEach ($Data in ($CSV | Select "Data*" -First 1).PSObject.Properties.Name) {
$CSV | ? {$_.$Data} | Select ID, #{N='Data'; E={$_.$Data}}
}
I have this codes below all working OK - however when i try to export to xls it does not export anything - i am getting blind now... any one can help?
$StoppedInstances = (Get-EC2Instance).instances | Where-Object {$_.State.Name -eq "stopped" -or $_.State.Name -eq "running"}
$VPCS = Get-EC2Vpc
foreach ($VPC in $VPCS) {
$StoppedInstances | Where-Object {$_.VpcId -eq $VPC.VpcId} | foreach {
New-Object -TypeName PSObject -Property #{
'InstanceId' = $_.InstanceId
'InstanceName' = ($_.Tags | Where-Object {$_.Key -eq 'Name'}).Value
'LaunchTime' = $_.LaunchTime
'State' = $_.State.Name
#'State1' = $_.State.GetType()
'Private IP' = $_.PrivateIpAddress
'Public IP' = $_.PublicIpAddress
'Public Dns' = $_.PublicDnsName
'loadbalace' = $_.AmiLaunchIndex
'vpcID' = $_.VpcId
'instancetype' = $_.InstanceType
'EBSDISK' = $_.BlockDeviceMappings.Count
'ELB' = $_.NetworkInterfaces
} | Format-Table -GroupBy date -Wrap | Export-Csv C:\temp\test4.csv
}
}
Because you're piping into Format-Table. Format-Table is used ONLY when you send data to the screen, like when you want to view something in the console. Remove your Format-Table statement and this will work as is.
In this example I use Tee-Object to snap out a variable containing the output of your command, and then send the main stream on to Format-Table for viewing.
Then, in the next step, we pipe that variable into the CSV file you want to export.
$StoppedInstances = (Get-EC2Instance).instances | Where-Object {$_.State.Name -eq "stopped" -or $_.State.Name -eq "running"}
$VPCS = Get-EC2Vpc
$export = foreach ($VPC in $VPCS) {
$StoppedInstances | Where-Object {$_.VpcId -eq $VPC.VpcId} | foreach {
New-Object -TypeName PSObject -Property #{
'InstanceId' = $_.InstanceId
'InstanceName' = ($_.Tags | Where-Object {$_.Key -eq 'Name'}).Value
'LaunchTime' = $_.LaunchTime
'State' = $_.State.Name
#'State1' = $_.State.GetType()
'Private IP' = $_.PrivateIpAddress
'Public IP' = $_.PublicIpAddress
'Public Dns' = $_.PublicDnsName
'loadbalace' = $_.AmiLaunchIndex
'vpcID' = $_.VpcId
'instancetype' = $_.InstanceType
'EBSDISK' = $_.BlockDeviceMappings.Count
'ELB' = $_.NetworkInterfaces
}
}
}
$export | Format-Table -GroupBy date -Wrap
$export | Export-Csv C:\temp\test4.csv
I created a similiar script like that:
$Groups = Get-QADGroup
$Result = #()
$Groups | ForEach-Object {
$Group = $_
$Members = Get-QADGroupMember $Group -Indirect | ? objectClass -eq "user"
$Obj = '' | Select-Object -Property Name, Members
$Obj.Name = $Group.Name
$Obj.Members = ($Members | % {$_.SamAccountName + "_" + 'Test'})
$Result += $Obj
}
$Result | Export-Csv -Path C:\Temp\groups.csv -NoTypeInformation -Encoding Unicode -Delimiter ";"
The output looks something like this (example data):
"Name";"Members"
"RootGroup01";"Subuser01_Test";"Subuser02_Test";"Subuser03_Test"
"RootGroup02";"Subuser02_Test"
"RootGroup03";"Subuser01_Test";"Subuser02_Test";"Subuser04_Test";
Is it possible to change the script, that I get something like this?:
"RootGroup01";"RootGroup02";"RootGroup03"
"Subuser01_Test";"Subuser02_Test";"Subuser01_Test"
"Subuser02_Test";;"Subuser02_Test"
"Subuser03_Test";;"Subuser04_Test"
The group names should be the header and the belonging users are in the right column. If there is no user, the column cell just stays empty.
You can write such function to transpose your data:
function Transpose-Data{
param(
[String[]]$Names,
[Object[][]]$Data
)
for($i = 0;; ++$i){
$Props = [ordered]#{}
for($j = 0; $j -lt $Data.Length; ++$j){
if($i -lt $Data[$j].Length){
$Props.Add($Names[$j], $Data[$j][$i])
}
}
if(!$Props.get_Count()){
break
}
[PSCustomObject]$Props
}
}
Then you invoke it in the following way:
$Groups = #(Get-QADGroup)
$Members = #(
$Groups | ForEach-Object {
,#(
Get-QADGroupMember $_ -Indirect |
? objectClass -eq user |
% {$_.SamAccountName + "_" + 'Test'}
)
}
)
Transpose-Data $Groups.Name $Members |
Export-Csv -Path C:\Temp\groups.csv -NoTypeInformation -Encoding Unicode -Delimiter ";"
What I am trying todo is filter a csv file (happens to be a weblog) with an array of hashtables (user information from a database).
#$data is from a database. (about 500 items) Type: System.Data.DataTable
$users = #()
foreach($row in $data)
{
$userItem = #{
LoginId = $row[0]
LastName = $row[3]
FirstName = $row[4]
LastAccess = $null
}
$users += $userItem
}
#Log files are about 14,000 lines long
$logfiles = Get-ChildItem $logFolder -Recurse | where {$_.Extension -eq ".log"} | Sort-Object BaseName -Descending
foreach($log in $logfiles)
{
$csvLog = Import-Csv $log.FullName -Header ("Blank","LoginId","Date")
$u = $users | Select {&_.LoginId}
$filteredcsvLog = $cvsLog | Where-Object { $u -contains $_.LoginId}
#This returns null
....
}
This does not seem to work, what am I missing. My guess is that I need to flatten the array into [string[]], however I can't seem todo that either.
Rather than do an array of hashtables, I would do a hashtable of custom objects e.g.:
$users = #{}
foreach($row in $data)
{
$userItem = new-object psobject -property #{
LoginId = $row[0]
LastName = $row[3]
FirstName = $row[4]
LastAccess = $null
}
$users[$userItem.LoginId] = $userItem
}
Then the filtering is easier and faster:
foreach($log in $logfiles)
{
$csvLog = Import-Csv $log.FullName -Header ("Blank","LoginId","Date")
$filteredcsvLog = $cvsLog | Where-Object { $users[$_.LoginId} }
....
}