I want to add headers to a CSV file initially. The reason I want to add the headers initially is for some rows, some column values might be empty.
As per Microsoft documentation, the export-csv only takes headers/columns which is present in first row.
When you submit multiple objects to Export-CSV, Export-CSV organizes the file >based on the properties of the first object that you submit. If the remaining >objects do not have one of the specified properties, the property value of >that object is null, as represented by two consecutive commas. If the >remaining objects have additional properties, those property values are not >included in the file.
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/export-csv?view=powershell-6#notes
What I have tried so far:
$csvContents =#()
foreach ($item in $items) {
$row = New-Object System.Object # Create an object to append to the array
$row | Add-Member -MemberType NoteProperty -Name "header1" -Value "value1"
$row | Add-Member -MemberType NoteProperty -Name "header2" -Value "value2"
$row | Add-Member -MemberType NoteProperty -Name "header3" -Value "value3"
$csvContents += $row # append the new data to the array
}
$csvContents | Export-CSV -NoTypeInformation -Path $ResultCsvPath
The problem is for example Item1 may have only header1 and header3, means these columns are dynamic. So after export the result csv will only contain header1 and header3 only and header2 will be missing.
What I expect is that I want to add header1, header2, header3 initially.
With large collections this may take up some time, but here's something that might work for you:
Suppose this is your collection
$obj = #(
[pscustomobject]#{
'header1' = 'value1'
'header3' = 'value3'
},
[pscustomobject]#{
'header1' = 'value1'
'header2' = 'value2'
},
[pscustomobject]#{
'header3' = 'value3'
},
[pscustomobject]#{
'header1' = 'value1'
'header2' = 'value2'
'header3' = 'value3'
}
)
Then you can add the missing properties like:
# Try and find all headers by looping over all items.
# You could change this to loop up to a maximum number of items if you like.
# The headers will be captured in the order in which they are found.
$headers = $obj | ForEach-Object {($_.PSObject.Properties).Name} | Select-Object -Unique
# Find the missing headers in the first item of the collection
# and add those with value $null to it.
$headers | Where-Object { ($obj[0].PSObject.Properties).Name -notcontains $_ } | ForEach-Object {
$obj[0] | Add-Member -MemberType NoteProperty -Name $_ -Value $null
}
# output on console
$obj
# output to csv file
$obj | Export-Csv -Path 'D:\test.csv' -NoTypeInformation
Output:
header1 header3 header2
------- ------- -------
value1 value3
value1 value2
value3
value1 value3 value2
Make sure if the data is missing for the column in a row that you use a $null value. I just tested this and get the output you are expecting.
$row = New-Object System.Object # Create an object to append to the array
$row | Add-Member -MemberType NoteProperty -Name "header1" -Value "value1"
$row | Add-Member -MemberType NoteProperty -Name "header2" -Value $null
$row | Add-Member -MemberType NoteProperty -Name "header3" -Value "value3"
$row | Export-Csv -NoTypeInformation test.csv
Output (from CSV)
"header1","header2","header3"
"value1",,"value3"
Depending on how complex you want to go, Ive done something similar to this in the past (I;ve changed it to use [pscustomobject] too):
$csvcontents = $items | foreach-object {
If (-not $_.Header1) { $value1 = '' } Else { $value1 = $_.Value1 }
If (-not $_.Header2) { $value2 = '' } Else { $value1 = $_.Value2 }
If (-not $_.Header3) { $value3 = '' } Else { $value1 = $_.Value3 }
[pscustomobject]#{header1 = $_.value1;header2=$value2;$header3=$value3}
}
Disclaimer, not tested the above, but it gives you the gist of the idea.
Please find here an example creating a CSV based of a generated directory output:
$basedir = "E:\20220205MassUpload1"
$picDir = "$($baseDir)\pics"
$res=ls $picDir
$outputArray = #()
foreach($line in $res) {
$name = $($line.name).Replace(".png","")
$outputArray += (
[pscustomobject]#{
Title = "$name"
Brand = 'Brand'
BulletPoint1='BulletPoint1'
BulletPoint2='BulletPoint2'
Description='Description'
Price='19.95'
Tags='Tags'
ImagePath="$($picDir)\$($line.name)"
});
}
$outputArray | Export-csv -NoTypeInformation "$basedir\upload.csv"
What it does it, take a list of all elements in a directory.
Create a "pscustomobject" for each result and create the values for the column, some are "fixed" and some "calculated".
Last the Array will be piped to a CSV.
I'm sure the other methods are the "right" way to do it, but I stumbled on a much easier way at some point in the past.
When I start a new CSV, I use something like this:
$vOutputFileName = ".\DDG-Info.csv"
"Name,DisplayName,RecipientFilter" | Out-File $vOutputFileName -encoding ASCII
Then in the script, I do the same thing:
$vOutput = "$vDDGName,$vDisplayName,$vRF"
"$vOutput" | Out-File $vOutputFileName -Append -encoding ASCII
Seems to work just fine, no messing around with explicitly creating objects etc.
Related
I want to work with a CSV file of more than 300,000 lines. I need to verify information line by line and then display it in a .txt file in the form of a table to see which file was missing for all servers. For example
Name,Server
File1,Server1
File2,Server1
File3,Server1
File1,Server2
File2,Server2
...
File345,Server76
File346,Server32
I want to display in table form this result which corresponds to the example above:
Name Server1 Server2 ... Server 32 ....Server 76
File1 X X
File2 X X
File3 X
...
File345 X
File346 X
To do this actually, I have a function that creates objects where the members are the Server Name (The number of members object can change) and I use stream reader to split data (I have more than 2 columns in my csv so 0 is for the Server name and 5 for the file name)
$stream = [System.IO.StreamReader]::new($File)
$stream.ReadLine() | Out-Null
while ((-not $stream.EndOfStream)) {
$line = $stream.ReadLine()
$strTempo = $null
$strTempo = $line -split ","
$index = $listOfFile.Name.IndexOf($strTempo[5])
if ($index -ne -1) {
$property = $strTempo[0].Replace("-", "_")
$listOfFile[$index].$property = "X"
}
else {
$obj = CreateEmptyObject ($listOfConfiguration)
$obj.Name = $strTempo[5]
$listOfFile.Add($obj) | Out-Null
}
}
When I export this I have a pretty good result. But the script take so much time (between 20min to 1hour)
I didn't know how optimize actually the script. I'm beginner to PowerShell.
Thanks for the futures tips
You might use HashSets for this:
$Servers = [System.Collections.Generic.HashSet[String]]::New()
$Files = #{}
Import-Csv -Path $Path |ForEach-Object {
$Null = $Servers.Add($_.Server)
if ($Files.Contains($_.Name)) { $Null = $Files[$_.Name].Add($_.Server) }
else { $Files[$_.Name] = [System.Collections.Generic.HashSet[String]]$_.Server }
}
$Table = foreach($Name in $Files.get_Keys()) {
$Properties = [Ordered]#{ Name = $Name }
ForEach ($Server in $Servers) {
$Properties[$Server] = if ($Files[$Name].Contains($Server)) { 'X' }
}
[PSCustomObject]$Properties
}
$Table |Format-Table -Property #{ expression='*' }
Note that in contrast to PowerShell's usual behavior, the .Net HashSet class is case-sensitive by default. To create an case-insensitive HashSet use the following constructor:
[System.Collections.Generic.HashSet[String]]::New([StringComparer]::OrdinalIgnoreCase)
See if this works faster. Change filename as required
$Path = "C:\temp\test1.txt"
$table = Import-Csv -Path $Path
$columnNames = $table | Select-Object -Property Server -Unique| foreach{$_.Server} | Sort-Object
Write-Host "names = " $columnNames
$groups = $table | Group-Object {$_.Name}
$outputTable = [System.Collections.ArrayList]#()
foreach($group in $groups)
{
Write-Host "Group = " $group.Name
$newRow = New-Object -TypeName psobject
$newRow | Add-Member -NotePropertyName Name -NotePropertyValue $group.Name
$servers = $group.Group | Select-Object -Property Server | foreach{$_.Server}
Write-Host "servers = " $servers
foreach($item in $columnNames)
{
if($servers.Contains($item))
{
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue 'X'
}
else
{
#if you comment out next line code doesn't work
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue ''
}
}
$outputTable.Add($newRow) | Out-Null
}
$outputTable | Format-Table
How can I specify the column position when adding a new column to an existing csv file?
I want to add the new column as second column (Not at the end what the default is).
The first column is always the same, but the other columns can differ per file (so it is not known on beforehand which columns (names and order) there are (with the exception of the first column, which always contains the name)).
As far as I know there is no position parameter for Add-Member.
Sample code:
$csv = Import-Csv -Path "attendees.csv" -Delimiter ';'
foreach ($row in $csv)
{
$row | Add-Member -'GUID' -Value (New-Guid).Guid -MemberType NoteProperty
}
$csv | Export-Csv new_attendees.csv' -NoTypeInformation
In case you do not know the column names at forehand.
Using Select-Object with a calculated property for this:
$csv = Import-Csv -Path "attendees.csv" -Delimiter ';'
$Properties = [Collections.Generic.List[Object]]$csv[0].psobject.properties.name
$Properties.Insert(1, #{ n='Guid'; e={ New-Guid } }) # insert at column #1
$csv |Select-Object -Property $Properties |Export-Csv new_attendees.csv' -NoTypeInformation
Explanation: (Updated 2022-11-12)
Each object PowerShell has een hidden PSObject property where you can dynamically access information about the property as e.g. its name.
Using the PowerShell Member-Access enumeration feature will list all the psobject.properties.name as an array of scalar strings.
I am using just the first object $csv[0] to determine the property (column) names as I do not want to choke the PowerShell pipeline and continue to support one-at-a-time processing. In other words, I presume that the following objects have unified property names. Any well written PowerShell cmdlet follows the strongly encouraged development guideline to implement for the middle of a pipeline
Thanks to the impliciet .Net conversion, it is easy to type cast the PowerShell Array (of property names) to the Collections.Generic.List[Object] type
Which happens to have a List<T>.Insert(Int32, T) Method. This lets you insert a item (in this case an object) at a certain position (in this case: 1)
Note that this method is The zero-based
The -Property parameter of the Select-Object cmdlet, doesn't just support an ordered list of property names but also calculated properties which is used here to create complete property along with its name, value (expression) in the form of:
#{ n='Guid'; e={ New-Guid } }
I don't think there's an easy way to insert a property into an object in-place at a specific index, but here's a quick proof-of-concept I knocked out to create a new object based on the original...
It could do with some error handling and perhaps some parameter attributes to support the pipeline, but it basically works...
function Insert-NoteProperty
{
param(
[pscustomobject] $InputObject,
[string] $Name,
[object] $Value,
[int] $Index
)
$properties = #($InputObject.psobject.Properties);
$result = [ordered] #{};
# append the properties before the index
for( $i = 0; $i -lt $Index; $i++ )
{
$result.Add($properties[$i].Name, $properties[$i].Value);
}
# append the new property
$result.Add($Name, $Value);
# append the properties after the index
for( $i = $Index; $i -lt $properties.Length; $i++ )
{
$result.Add($properties[$i].Name, $properties[$i].Value);
}
return [pscustomobject] $result;
}
Example:
$original = [pscustomobject] [ordered] #{ "aaa"="bbb"; "ccc"="ddd" }
$original
# aaa ccc
# --- ---
# bbb ddd
$updated = Insert-NoteProperty `
-InputObject $original `
-Name "new" `
-Value "value" `
-Index 1;
$updated
# aaa new ccc
# --- --- ---
# bbb value ddd
You can use this with a csv file as follows:
$csv = #"
aaa,ccc
bbb,ddd
"#
$data = $csv | ConvertFrom-Csv;
$newdata = $data | foreach-object {
Insert-NoteProperty `
-InputObject $_ `
-Name "new" `
-Value "value" `
-Index 1
}
$newcsv = $newdata | ConvertTo-Csv
$newcsv
# "aaa","new","ccc"
# "bbb","value","ddd"
Add-Member does not let you specify at which position to add a property to an object. So you have to build a new object where you can control the position of the property, e.g.:
$newObjects = #(
foreach ($row in $csv){
$attrsHt = [ordered]#{
[propertyName]=[propertyValue]
Guid=([guid]::NewGuid()).guid
[propertyName]=[propertyValue]
[propertyName]=[propertyValue]
}
New-Object -TypeName psobject -Property $attrsht
}
)
Or you can use select-object to change the order of the output:
foreach ($row in $csv)
{
$row | Add-Member -Name 'GUID' -Value (New-Guid).Guid -MemberType NoteProperty
}
$csv | select-object [propertyName],Guid,[propertyName],[propertyName] | export-csv new_attendees.csv' -NoTypeInformation
Ok as the propertyNames are unknown/dynamic you could do:
#Get the csv header and split by delimiter to get array of property names
[System.Collections.ArrayList]$csvHeader = (get-content [path])[0] -split ","
#Insert Guid at 2nd Position
$csvHeader.insert(1,'GUID')
$csv | select-object $csvHeader | export-csv new_attendees.csv -NoTypeInformation
Basically i have input.csv file with mutiple columns it has to convert to mutiple rows..
input.csv:
Month,Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec
and i want to add extra columns to csv file
i have a problem here..my entire input file changed,and now from jan to dec..i have these in 2nd row ,like
sample.csv
"Application","Entity","Project","Years","Version","Currency","Scenario","LineItem","Account","Period"
"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"
"App - NA","11002","Project - NA","FY22","Working","USD","Budget","No Line Item","640605",171.40,171.80,170.50,169.10,178.60,168.00,167.80,167.70,179.40,176.70,176.70,177.10
but i want output.csv..like
"Application","Entity","Project","Years","Version","Currency","Scenario","LineItem","Account","Period","Data"
"App - NA","11002","Project - NA","FY22","Working","USD","Budget","No Line Item","640605",Jan, 171.40 "App - NA","11002","Project - NA","FY22","Working","USD","Budget","No Line Item","640605",Feb,171.80
Seems like you need to unpivot some columns. This worked for me:
Function UnPivot-Object {
[CmdletBinding()][OutputType([Object[]])]Param (
[Parameter(ValueFromPipeLine = $True)]$InputObject
)
Process {
$Properties = $InputObject.PSObject.Properties
$Row = ($Properties | Select-Object -First 1).Value
$Properties | Select-Object -Skip 1 | ForEach-Object {
[PSCustomObject]#{Row = $Row; Column = $_.Name; Value = $_.Value}
}
}
}; Set-Alias UnPivot UnPivot-Object
As for adding new columns, try this:
foreach ($row in $csv)
{
$data = Get-DataFromSomewhere -Id $row.Id
$row | Add-Member -Name 'first_name' -Value $data.FirstName -MemberType NoteProperty
$row | Add-Member -Name 'last_name' -Value $data.LastName -MemberType NoteProperty
}
Hope this helped!
This is odd behavior I have discovered due to the structure of some JSON I am trying to process. I am simply trying to return all the property names. This is my JSON:
$x = #"
[
{
"test": [
"item1"
]
},
{
"test2": [
"item2"
]
}
]
"# | ConvertFrom-Json
You may also create the objects like so, which results in the same issue:
$x= #()
$x += [pscustomobject]#{
'test' = 'item1'
}
$x += [pscustomobject]#{
'test2' = 'item2'
}
Notice that now, I can write $x | fl and get all this information as per usual.
$x | fl
test : item1
test2 : item2
However, when using Get-Member, only the first object is included.
$x | Get-Member -MemberType NoteProperty
Name MemberType Definition
---- ---------- ----------
test NoteProperty string test=item1
Does anyone know why this is? I cannot change the JSON structure.
Get-Member looks at the first item (of each distinct type) in the pipeline to determine the set of properties to use.
Out-GridView and other cmdlets like Export-CSV do the same thing.
It has nothing to do with the fact that these are PSCustomObjects. Here's an example with FileInfo objects:
$files = Get-ChildItem -Path C:\temp -File |
Select-Object -First 2
#Add a property to second item
$files[1] | Add-Member -MemberType NoteProperty -Name Test -Value 'hello world'
#doesn't show Test, because it wasn't in the first item
$files | Get-Member
$files2 = Get-ChildItem -Path C:\temp -File |
Select-Object -First 2
#add property to first item
$files2[0] | Add-Member -MemberType NoteProperty -Name Test -Value 'hello world'
#shows Test, because it was in the first item
$files2 | Get-Member
I am trying to output different attributes of a Skype response group queue for documentation purpose.
I want to get Name, TimeoutThreshold, TimeoutAction , Timeouturi, OverflowThreshold, OverflowAction , OverflowCandidate as a .csv file header in row 1 and then the output to be entered in various columns from row 2.
I have tried below, but the formatting is really bad and the headers keep repeating. Can some one please help.
Also tried getting output in HTML, but no luck.
$p = Get-CsRgsQueue | Where-Object {$_.Name -like "IPL*"} | Select-Object Name
foreach ($Name in $p)
{
$q = Get-CsRgsQueue -Name "$Name"
$N = $q.Name
$TT = $q.TimeoutThreshold
$TA = $q.TimeoutAction.Action
$TAU = $q.TimeoutAction.uri
$OF = $q.OverflowThreshold
$OFA = $q.OverflowAction
$OFC = $q.OverflowCandidate
$out = New-Object PSObject
$out | Add-Member NoteProperty QueueName $N
$out | Add-Member NoteProperty Timeout $TT
$out | Add-Member NoteProperty TimeoutAction $TA
$out | Add-Member NoteProperty TransferURI $TAU
$out | Add-Member NoteProperty OverflowThreshhold $OF
$out | Add-Member NoteProperty OverflowAction $OFA
$out | Add-Member NoteProperty OverflowCandidate $OFC
$out | FT -AutoSize | Export-Csv C:\abc.csv -Append
}
I have tried below, but the formatting is really bad and the headers
keep repeating. Can some one please help.
That's because you pipe your objects through FT -AutoSize (Format-Table -AutoSize) - only ever use the Format-* cmdlets when you're about to show/present your data.
You can also save some time by only calling Get-CsRgsQueue once, piping it to ForEach-Object and finally construct a hashtable for the object properties:
Get-CsRgsQueue | Where-Object {$_.Name -like "IPL*"} | ForEach-Object {
New-object psobject -Property #{
QueueName = $_.Name
Timeout = $_.TimoutThreshold
TimeoutAction = $_.TimeoutAction.Action
TransferURI = $_.TimeoutAction.Uri
OverflowThreshhold = $_.OverflowThreshold
OverflowAction = $_.OverflowAction
OverflowCandidate = $_.OverflowCandicate
}
} |Export-Csv c:\abc.csv -NoTypeInformation
short solution of Mathias Jessen
Get-CsRgsQueue | where Name -like "IPL*" | %{
[pscustomobject] #{
QueueName = $_.Name
Timeout = $_.TimoutThreshold
TimeoutAction = $_.TimeoutAction.Action
TransferURI = $_.TimeoutAction.Uri
OverflowThreshhold = $_.OverflowThreshold
OverflowAction = $_.OverflowAction
OverflowCandidate = $_.OverflowCandicate
}
} | Export-Csv C:\result.csv -NoType