Basically i have input.csv file with mutiple columns it has to convert to mutiple rows..
input.csv:
Month,Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec
and i want to add extra columns to csv file
i have a problem here..my entire input file changed,and now from jan to dec..i have these in 2nd row ,like
sample.csv
"Application","Entity","Project","Years","Version","Currency","Scenario","LineItem","Account","Period"
"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"
"App - NA","11002","Project - NA","FY22","Working","USD","Budget","No Line Item","640605",171.40,171.80,170.50,169.10,178.60,168.00,167.80,167.70,179.40,176.70,176.70,177.10
but i want output.csv..like
"Application","Entity","Project","Years","Version","Currency","Scenario","LineItem","Account","Period","Data"
"App - NA","11002","Project - NA","FY22","Working","USD","Budget","No Line Item","640605",Jan, 171.40 "App - NA","11002","Project - NA","FY22","Working","USD","Budget","No Line Item","640605",Feb,171.80
Seems like you need to unpivot some columns. This worked for me:
Function UnPivot-Object {
[CmdletBinding()][OutputType([Object[]])]Param (
[Parameter(ValueFromPipeLine = $True)]$InputObject
)
Process {
$Properties = $InputObject.PSObject.Properties
$Row = ($Properties | Select-Object -First 1).Value
$Properties | Select-Object -Skip 1 | ForEach-Object {
[PSCustomObject]#{Row = $Row; Column = $_.Name; Value = $_.Value}
}
}
}; Set-Alias UnPivot UnPivot-Object
As for adding new columns, try this:
foreach ($row in $csv)
{
$data = Get-DataFromSomewhere -Id $row.Id
$row | Add-Member -Name 'first_name' -Value $data.FirstName -MemberType NoteProperty
$row | Add-Member -Name 'last_name' -Value $data.LastName -MemberType NoteProperty
}
Hope this helped!
Related
I want to work with a CSV file of more than 300,000 lines. I need to verify information line by line and then display it in a .txt file in the form of a table to see which file was missing for all servers. For example
Name,Server
File1,Server1
File2,Server1
File3,Server1
File1,Server2
File2,Server2
...
File345,Server76
File346,Server32
I want to display in table form this result which corresponds to the example above:
Name Server1 Server2 ... Server 32 ....Server 76
File1 X X
File2 X X
File3 X
...
File345 X
File346 X
To do this actually, I have a function that creates objects where the members are the Server Name (The number of members object can change) and I use stream reader to split data (I have more than 2 columns in my csv so 0 is for the Server name and 5 for the file name)
$stream = [System.IO.StreamReader]::new($File)
$stream.ReadLine() | Out-Null
while ((-not $stream.EndOfStream)) {
$line = $stream.ReadLine()
$strTempo = $null
$strTempo = $line -split ","
$index = $listOfFile.Name.IndexOf($strTempo[5])
if ($index -ne -1) {
$property = $strTempo[0].Replace("-", "_")
$listOfFile[$index].$property = "X"
}
else {
$obj = CreateEmptyObject ($listOfConfiguration)
$obj.Name = $strTempo[5]
$listOfFile.Add($obj) | Out-Null
}
}
When I export this I have a pretty good result. But the script take so much time (between 20min to 1hour)
I didn't know how optimize actually the script. I'm beginner to PowerShell.
Thanks for the futures tips
You might use HashSets for this:
$Servers = [System.Collections.Generic.HashSet[String]]::New()
$Files = #{}
Import-Csv -Path $Path |ForEach-Object {
$Null = $Servers.Add($_.Server)
if ($Files.Contains($_.Name)) { $Null = $Files[$_.Name].Add($_.Server) }
else { $Files[$_.Name] = [System.Collections.Generic.HashSet[String]]$_.Server }
}
$Table = foreach($Name in $Files.get_Keys()) {
$Properties = [Ordered]#{ Name = $Name }
ForEach ($Server in $Servers) {
$Properties[$Server] = if ($Files[$Name].Contains($Server)) { 'X' }
}
[PSCustomObject]$Properties
}
$Table |Format-Table -Property #{ expression='*' }
Note that in contrast to PowerShell's usual behavior, the .Net HashSet class is case-sensitive by default. To create an case-insensitive HashSet use the following constructor:
[System.Collections.Generic.HashSet[String]]::New([StringComparer]::OrdinalIgnoreCase)
See if this works faster. Change filename as required
$Path = "C:\temp\test1.txt"
$table = Import-Csv -Path $Path
$columnNames = $table | Select-Object -Property Server -Unique| foreach{$_.Server} | Sort-Object
Write-Host "names = " $columnNames
$groups = $table | Group-Object {$_.Name}
$outputTable = [System.Collections.ArrayList]#()
foreach($group in $groups)
{
Write-Host "Group = " $group.Name
$newRow = New-Object -TypeName psobject
$newRow | Add-Member -NotePropertyName Name -NotePropertyValue $group.Name
$servers = $group.Group | Select-Object -Property Server | foreach{$_.Server}
Write-Host "servers = " $servers
foreach($item in $columnNames)
{
if($servers.Contains($item))
{
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue 'X'
}
else
{
#if you comment out next line code doesn't work
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue ''
}
}
$outputTable.Add($newRow) | Out-Null
}
$outputTable | Format-Table
I want to add headers to a CSV file initially. The reason I want to add the headers initially is for some rows, some column values might be empty.
As per Microsoft documentation, the export-csv only takes headers/columns which is present in first row.
When you submit multiple objects to Export-CSV, Export-CSV organizes the file >based on the properties of the first object that you submit. If the remaining >objects do not have one of the specified properties, the property value of >that object is null, as represented by two consecutive commas. If the >remaining objects have additional properties, those property values are not >included in the file.
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/export-csv?view=powershell-6#notes
What I have tried so far:
$csvContents =#()
foreach ($item in $items) {
$row = New-Object System.Object # Create an object to append to the array
$row | Add-Member -MemberType NoteProperty -Name "header1" -Value "value1"
$row | Add-Member -MemberType NoteProperty -Name "header2" -Value "value2"
$row | Add-Member -MemberType NoteProperty -Name "header3" -Value "value3"
$csvContents += $row # append the new data to the array
}
$csvContents | Export-CSV -NoTypeInformation -Path $ResultCsvPath
The problem is for example Item1 may have only header1 and header3, means these columns are dynamic. So after export the result csv will only contain header1 and header3 only and header2 will be missing.
What I expect is that I want to add header1, header2, header3 initially.
With large collections this may take up some time, but here's something that might work for you:
Suppose this is your collection
$obj = #(
[pscustomobject]#{
'header1' = 'value1'
'header3' = 'value3'
},
[pscustomobject]#{
'header1' = 'value1'
'header2' = 'value2'
},
[pscustomobject]#{
'header3' = 'value3'
},
[pscustomobject]#{
'header1' = 'value1'
'header2' = 'value2'
'header3' = 'value3'
}
)
Then you can add the missing properties like:
# Try and find all headers by looping over all items.
# You could change this to loop up to a maximum number of items if you like.
# The headers will be captured in the order in which they are found.
$headers = $obj | ForEach-Object {($_.PSObject.Properties).Name} | Select-Object -Unique
# Find the missing headers in the first item of the collection
# and add those with value $null to it.
$headers | Where-Object { ($obj[0].PSObject.Properties).Name -notcontains $_ } | ForEach-Object {
$obj[0] | Add-Member -MemberType NoteProperty -Name $_ -Value $null
}
# output on console
$obj
# output to csv file
$obj | Export-Csv -Path 'D:\test.csv' -NoTypeInformation
Output:
header1 header3 header2
------- ------- -------
value1 value3
value1 value2
value3
value1 value3 value2
Make sure if the data is missing for the column in a row that you use a $null value. I just tested this and get the output you are expecting.
$row = New-Object System.Object # Create an object to append to the array
$row | Add-Member -MemberType NoteProperty -Name "header1" -Value "value1"
$row | Add-Member -MemberType NoteProperty -Name "header2" -Value $null
$row | Add-Member -MemberType NoteProperty -Name "header3" -Value "value3"
$row | Export-Csv -NoTypeInformation test.csv
Output (from CSV)
"header1","header2","header3"
"value1",,"value3"
Depending on how complex you want to go, Ive done something similar to this in the past (I;ve changed it to use [pscustomobject] too):
$csvcontents = $items | foreach-object {
If (-not $_.Header1) { $value1 = '' } Else { $value1 = $_.Value1 }
If (-not $_.Header2) { $value2 = '' } Else { $value1 = $_.Value2 }
If (-not $_.Header3) { $value3 = '' } Else { $value1 = $_.Value3 }
[pscustomobject]#{header1 = $_.value1;header2=$value2;$header3=$value3}
}
Disclaimer, not tested the above, but it gives you the gist of the idea.
Please find here an example creating a CSV based of a generated directory output:
$basedir = "E:\20220205MassUpload1"
$picDir = "$($baseDir)\pics"
$res=ls $picDir
$outputArray = #()
foreach($line in $res) {
$name = $($line.name).Replace(".png","")
$outputArray += (
[pscustomobject]#{
Title = "$name"
Brand = 'Brand'
BulletPoint1='BulletPoint1'
BulletPoint2='BulletPoint2'
Description='Description'
Price='19.95'
Tags='Tags'
ImagePath="$($picDir)\$($line.name)"
});
}
$outputArray | Export-csv -NoTypeInformation "$basedir\upload.csv"
What it does it, take a list of all elements in a directory.
Create a "pscustomobject" for each result and create the values for the column, some are "fixed" and some "calculated".
Last the Array will be piped to a CSV.
I'm sure the other methods are the "right" way to do it, but I stumbled on a much easier way at some point in the past.
When I start a new CSV, I use something like this:
$vOutputFileName = ".\DDG-Info.csv"
"Name,DisplayName,RecipientFilter" | Out-File $vOutputFileName -encoding ASCII
Then in the script, I do the same thing:
$vOutput = "$vDDGName,$vDisplayName,$vRF"
"$vOutput" | Out-File $vOutputFileName -Append -encoding ASCII
Seems to work just fine, no messing around with explicitly creating objects etc.
Good Afternoon,
I searched through this forum and a few others combining ideas and trying different angles but haven't figured this out. If this has been answered, and I suck at searching, I am sorry and please let me know.
End Goal of script: have an excel file with columns for the AD properties Name, Office, Org, and most importantly a seperate column for each group a user is a member of.
The problem I am running into is creating a new column for each/every group that a user has. Not all users have the same amount of groups. Some have 10 some have 30 (yes 30, our AD is a mess).
Here is what I have done so far, and the spot that I am having difficulty with is towards the end:
$scoop = get-content C:\temp\SCOOP.txt ##This is a text file with a list of user id's, to search AD with
$outfile = 'C:\temp\SCOOP_ID.csv'
$ou = "OU=Humans,OU=Coupeville,DC=ISLANDS" #This is the searchbase, helps AD isolate the objects
Clear-Content $outfile #I clear content each time when I am testing
Foreach($ID in $scoop){
##AD Search filters##
$filtertype = 'SamAccountName'
$filter1 = $ID
##End AD Search filters##
##AD Search --MY MAIN ISSUE is getting the MemberOF property properly
$properties = get-aduser -SearchBase $ou -Filter {$filtertype -eq $filter1} -Properties Name,Office,Organization,MemberOf | select Name,Office,Organization,MemberOf
##AD Search ## Turns the MemberOf property to a string, I tried this during my testing not sure if objects or strings are easier to work with
#$properties = get-aduser -SearchBase $ou -Filter {$filtertype -eq $filter1} -Properties Name,Office,Organization,MemberOf | select Name,Office,Organization, #{n='MemberOf'; e= { $_.memberof | Out-String}}
#Attempt to specify each property for the output to csv
$name = $properties.Name
$office = $properties.Office
$org = $properties.Organization
$MemberOf = $properties.MemberOf
$membersArray = #()
foreach($mem in $MemberOf){ $membersArray += $mem }
###This is what I typically use to export to a CSV - I am sure there are other maybe better ways but this one I know.
$Focus = [PSCustomObject]#{}
$Focus | Add-Member -MemberType NoteProperty -Name 'Name' -Value $name -Force
$Focus | Add-Member -MemberType NoteProperty -Name 'Office' -Value $office -Force
$Focus | Add-Member -MemberType NoteProperty -Name 'Org' -Value $org -Force
$Focus | Add-Member -MemberType NoteProperty -Name 'Groups' -Value $MemberOf -Force
<########################
#
# Main ISSUE is getting the $memberOf variable, which contains all of the users groups, into seperate columns in the csv. To make it more plain, each column would be labeled 'Groups', then 'Groups1', and so on.
I have tried a few things but not sure if any were done properly or what I am messing up. I tried using $memberof.item($i) with a FOR loop but couldnt figure out how to send each
item out into its own Add-Member property.
#
#######################>
##Final Output of the $Focus Object
$Focus | Export-Csv $outfile -Append -NoTypeInformation
}
I came up with this solution, simply loop $MemberOf and add a unique name.
$MemberOf = #('a','b','c','d','e','f') #example
$Focus = [PSCustomObject]#{}
$Focus | Add-Member -MemberType NoteProperty -Name 'Name' -Value 'test' -Force
$i=0; $MemberOf | % {
$i++
$Focus | Add-Member -MemberType NoteProperty -Name "Group $i" -Value $_ -Force
}
$Focus | Export-Csv xmgtest1.csv -Append -Force -NoTypeInformation
However this doesn't work in your case, since you run this code once per user and there's no "awareness" of the entire set. So you would (in your existing architecture) need to ensure that the largest number of groups is added first.
You can fix this by creating the CSV all at once (only other option would be to constantly load/unload the csv file).
First, add $csv = #() to the top of your file.
Then, after you finish creating $Focus, add it to $csv.
Finally, you may remove -Append.
The slimmed down version looks like this:
$csv = #()
#Foreach($ID in $scoop){ etc..
$MemberOf = #('a','b','c','d','e','f') #example
$Focus = [PSCustomObject]#{}
$Focus | Add-Member -MemberType NoteProperty -Name 'Name' -Value 'test' -Force
#$Focus | Add-Member etc ...
$i=0; $MemberOf | % {
$i++
$Focus | Add-Member -MemberType NoteProperty -Name "Group $i" -Value $_ -Force
}
$csv += $Focus
# } end of Foreach($ID in $scoop)
$csv | Export-Csv xmgtest1.csv -NoTypeInformation
Hope this helps.
I'm trying to create a PowerShell script that transpose a CSV file from column to rows.
I found examples of doing the opposite (converting row based CSV to column) but I found nothing on column to rows. My problem being that I don't know exactly how many column I'll have. I tried adapting the row to column to column to rows but unsuccessfully.
$a = Import-Csv "input.csv"
$a | FT -AutoSize
$b = #()
foreach ($Property in $a.Property | Select -Unique) {
$Props = [ordered]#{ Property = $Property }
foreach ($Server in $a.Server | Select -Unique){
$Value = ($a.where({ $_.Server -eq $Server -and
$_.Property -eq $Property })).Value
$Props += #{ $Server = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
$b | FT -AutoSize
$b | Out-GridView
$b | Export-Csv "output.csv" -NoTypeInformation
For example my CSV can look like this:
"ID","DATA1"
"12345","11111"
"54321","11111"
"23456","44444"
or this (number of column can vary):
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555",
and I would like the script to convert it like this:
"ID","DATA"
"12345","11111"
"12345","22222"
"12345","33333"
"54321","11111"
"23456","44444"
"23456","55555"
The trick is to query the members of the table to get the column names. Once you do that then the rest is straightforward:
function Flip-Table ($Table) {
Process {
$Row = $_
# Get all the columns names, excluding the ID field.
$Columns = ($Row | Get-Member -Type NoteProperty | Where-Object Name -ne ID).Name
foreach ($Column in $Columns) {
if ($Row.$Column) {
$Properties = [Ordered] #{
"ID" = $Row.ID
"DATA" = $Row.$Column
}
New-Object PSObject -Property $Properties
}
}
# Garbage collection won't kick in until the end of the script, so
# invoke it every 100 input rows.
$Count++;
if (($Count % 100) -eq 0) {
[System.GC]::GetTotalMemory('forceFullCollection') | out-null
}
}
}
Import-Csv input.csv | Flip-Table | Export-Csv -NoTypeInformation output.csv
Well, here is mine. I'm not as fancy as the rest:
$in = Get-Content input.csv | Select -Skip 1
$out = New-Object System.Collections.ArrayList
foreach($row in $in){
$parts = $row.Split(',')
$id = $parts[0]
foreach($data in $parts[1..$parts.Count]){
if($data -ne '' -AND $data -ne $null){
$temp = New-Object PSCustomObject -Property #{'ID' = $id;
'Data' = $data}
$out.Add($temp) | Out-Null
}
}
}
$out | Export-CSV output.csv -NoTypeInformation
You can do something like this
# Convert csv to object
$csv = ConvertFrom-Csv #"
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555"
"#
# Ignore common members and the ID property
$excludedMembers = #(
'GetHashCode',
'GetType',
'ToString',
'Equals',
'ID'
)
$results = #()
# Iterate around each csv row
foreach ($row in $csv) {
$members = $row | Get-Member
# Iterate around each member from the 'row object' apart from our
# exclusions and empty values
foreach ($member in $members |
Where { $excludedMembers -notcontains $_.Name -and $row.($_.Name)}) {
# add to array of objects
$results += #{ ID=$row.ID; DATA=$row.($member.Name)}
}
}
# Write the csv string
$outstring = "ID,DATA"
$results | foreach { $outstring += "`n$($_.ID),$($_.DATA)" }
# New csv object
$csv = $outstring | ConvertFrom-Csv
Probably not the most elegant solution, but should do what you need
I left some comments explaining what it does
If you only want to accept a limited number DATA columns (e.g. 5), you could do:
ForEach ($i in 1..5) {$CSV | ? {$_."Data$i"} | Select ID, #{N='Data'; E={$_."Data$i"}}}
And if you have a potential unlimited number of DATA columns:
ForEach ($Data in ($CSV | Select "Data*" -First 1).PSObject.Properties.Name) {
$CSV | ? {$_.$Data} | Select ID, #{N='Data'; E={$_.$Data}}
}
I am trying to output different attributes of a Skype response group queue for documentation purpose.
I want to get Name, TimeoutThreshold, TimeoutAction , Timeouturi, OverflowThreshold, OverflowAction , OverflowCandidate as a .csv file header in row 1 and then the output to be entered in various columns from row 2.
I have tried below, but the formatting is really bad and the headers keep repeating. Can some one please help.
Also tried getting output in HTML, but no luck.
$p = Get-CsRgsQueue | Where-Object {$_.Name -like "IPL*"} | Select-Object Name
foreach ($Name in $p)
{
$q = Get-CsRgsQueue -Name "$Name"
$N = $q.Name
$TT = $q.TimeoutThreshold
$TA = $q.TimeoutAction.Action
$TAU = $q.TimeoutAction.uri
$OF = $q.OverflowThreshold
$OFA = $q.OverflowAction
$OFC = $q.OverflowCandidate
$out = New-Object PSObject
$out | Add-Member NoteProperty QueueName $N
$out | Add-Member NoteProperty Timeout $TT
$out | Add-Member NoteProperty TimeoutAction $TA
$out | Add-Member NoteProperty TransferURI $TAU
$out | Add-Member NoteProperty OverflowThreshhold $OF
$out | Add-Member NoteProperty OverflowAction $OFA
$out | Add-Member NoteProperty OverflowCandidate $OFC
$out | FT -AutoSize | Export-Csv C:\abc.csv -Append
}
I have tried below, but the formatting is really bad and the headers
keep repeating. Can some one please help.
That's because you pipe your objects through FT -AutoSize (Format-Table -AutoSize) - only ever use the Format-* cmdlets when you're about to show/present your data.
You can also save some time by only calling Get-CsRgsQueue once, piping it to ForEach-Object and finally construct a hashtable for the object properties:
Get-CsRgsQueue | Where-Object {$_.Name -like "IPL*"} | ForEach-Object {
New-object psobject -Property #{
QueueName = $_.Name
Timeout = $_.TimoutThreshold
TimeoutAction = $_.TimeoutAction.Action
TransferURI = $_.TimeoutAction.Uri
OverflowThreshhold = $_.OverflowThreshold
OverflowAction = $_.OverflowAction
OverflowCandidate = $_.OverflowCandicate
}
} |Export-Csv c:\abc.csv -NoTypeInformation
short solution of Mathias Jessen
Get-CsRgsQueue | where Name -like "IPL*" | %{
[pscustomobject] #{
QueueName = $_.Name
Timeout = $_.TimoutThreshold
TimeoutAction = $_.TimeoutAction.Action
TransferURI = $_.TimeoutAction.Uri
OverflowThreshhold = $_.OverflowThreshold
OverflowAction = $_.OverflowAction
OverflowCandidate = $_.OverflowCandicate
}
} | Export-Csv C:\result.csv -NoType