Manipulate CSV value without knowing column names [duplicate] - powershell

I want to process a csv file in powershell, but I don't know what the column headings in the CSV file will be when it is processed.
For example:
$path = "d:\scratch\export.csv"
$csv = Import-csv -path $path
foreach($line in $csv)
{
foreach ($head in $line | get-member | where-object {$_.MemberType -eq "NoteProperty"} | select Definition)
{
#pseudocode...
doSomething($head.columnName, $head.value)
}
}
How do I loop through the line in the csv file, getting the name of the column and the value? Or is there another way I should be doing this (like not using Import-csv)?

Import-Csv $path | Foreach-Object {
foreach ($property in $_.PSObject.Properties)
{
doSomething $property.Name, $property.Value
}
}

A slightly other way of iterating through each column of each line of a CSV-file would be
$path = "d:\scratch\export.csv"
$csv = Import-Csv -path $path
foreach($line in $csv)
{
$properties = $line | Get-Member -MemberType Properties
for($i=0; $i -lt $properties.Count;$i++)
{
$column = $properties[$i]
$columnvalue = $line | Select -ExpandProperty $column.Name
# doSomething $column.Name $columnvalue
# doSomething $i $columnvalue
}
}
so you have the choice: you can use either $column.Name to get the name of the column, or $i to get the number of the column

$header3 = #("Field_1","Field_2","Field_3","Field_4","Field_5")
Import-Csv $fileName -Header $header3 -Delimiter "`t" | select -skip 3 | Foreach-Object {
$record = $indexName
foreach ($property in $_.PSObject.Properties){
#doSomething $property.Name, $property.Value
if($property.Name -like '*TextWrittenAsNumber*'){
$record = $record + "," + '"' + $property.Value + '"'
}
else{
$record = $record + "," + $property.Value
}
}
$array.add($record) | out-null
#write-host $record
}

Related

Import CSV with loop problem - The result contains only last item from the CSV

I need to import a few groups from a CSV file and then export members - but I need it like one row each, something like:
"Group_name1", "member1, member2,member3"
"Group_name2", "member1,member2,member3"
"Group_name3", "member1,member2,member3"
And my script is working fine for a single group but I'm having troubles with adding a for-each loop - the result contains only last item from the CSV..
#$DL='plum'
$DL_List = "C:\ps1\shared_mailboxes\groups.csv"
$DL_array = (Import-Csv -Path $DL_List).name
foreach ($DL in $DL_array)
{
$DL_Membership = (Get-DistributionGroupMember -identity $DL).displayName
if([string]$DL_Membership -ne "")
{
$Members = ""
foreach($DL_Membership in $DL_Membership)
{
if($Members -ne "")
{
$Members=$Members +","
}
$Members = $Members + $DL_Membership
}
}
}
$ExportCSV=".\group_members_$((Get-Date -format yyyy-MMM-dd-ddd` hh-mm` tt).ToString()).csv"
$Result = #{'Group'=$DL;'Users'=$Members}
$Results = New-Object PSObject -Property $Result
$Results |select-object 'Group','Users' | Export-Csv -Path $ExportCSV -Notype -Append
I googled it but I'm not sure what I should change in my script..
The script has been re-writed using -join operator and now it's working as excepted.
$DL_List = Import-Csv "C:\ps1\shared_mailboxes\groups.csv"
$CsvOut = foreach ( $dl in $DL_List ) {
$Group = $dl.name
$MemberArray = (Get-DistributionGroupMember -identity $Group).displayName
$MemberList = $MemberArray -join ','
# generate the output object
# that goes in $CsvOut
[PSCustomObject] #{
Group = $Group
Members = $MemberList
}
}
$Timestamp = Get-Date -Format 'yyyy-MM-dd-HHmm' # this is a string
$CsvOutPath = "c:\ps1\shared_mailboxes\memberlist.$Timestamp.csv"
$CsvOut | Export-Csv $CsvOutPath -NoTypeInfo
I think you are over-complicating this.
If your desired result is a CSV file, then output objects straight away like below:
$DL_List = 'C:\ps1\shared_mailboxes\groups.csv'
$DL_array = (Import-Csv -Path $DL_List).name
$outFile = '.\group_members_{0:yyyy-MMM-dd-ddd hh-mm}.csv' -f (Get-Date)
$result = foreach ($DL in $DL_array) {
# output an object with the name of the DL and its members
[PsCustomObject]#{
Group = $DL
Users = (($DL | Get-DistributionGroupMember).displayName |
Where-Object {$_ -match '\S'}) -join ','
}
}
$result | Export-Csv -Path $outFile -NoTypeInformation
Using calculated properties, you can also do this without a foreach loop:
$DL_List = 'C:\ps1\shared_mailboxes\groups.csv'
$DL_array = (Import-Csv -Path $DL_List).name
$outFile = '.\group_members_{0:yyyy-MMM-dd-ddd hh-mm}.csv' -f (Get-Date)
$DL_array | Select-Object #{Name = 'Group'; Expresssion = {$_}},
#{Name = 'Users'; Expresssion = {
(($_ | Get-DistributionGroupMember).displayName |
Where-Object {$_ -match '\S'}) -join ','
}} |
Export-Csv -Path $outFile -NoTypeInformation

How to update cells in csv one by one without knowing column names

I try to replace some texts inside a csv like following:
$csv = Import-Csv $csvFileName -Delimiter ';'
foreach ($line in $csv)
{
$properties = $line | Get-Member -MemberType Properties
for ($i = 0; $i -lt $properties.Count;$i++)
{
$column = $properties[$i]
$value = $line | Select -ExpandProperty $column.Name
# Convert numbers with , as separator to . separator
if ($value -match "^[+-]?(\d*\,)?\d+$")
{
$value = $value -replace ",", "."
# HOW TO update the CSV cell with the new value here???
# ???
}
}
}
$csv | Export-Csv $csvFileName -Delimiter ',' -NoTypeInformation -Encoding UTF8
As you can see, I miss the line where I want to update the csv line's cell value with the new value => can someone tell me how I can do that?
Assuming the regex you have in place will match the pattern you expect, you can simplify your code using 2 foreach loops, easier than a for. This method invokes the intrinsic PSObject member, available to all PowerShell objects.
$csv = Import-Csv $csvFileName -Delimiter ';'
foreach ($line in $csv) {
foreach($property in $line.PSObject.Properties) {
if ($property.Value -match "^[+-]?(\d*\,)?\d+$") {
# regex operator not needed in this case
$property.Value = $property.Value.Replace(",", ".")
}
}
}
$csv | Export-Csv ....
You can also do all process in pipeline (method above should be clearly faster, however this method is likely to be memory friendlier):
Import-Csv $csvFileName -Delimiter ';' | ForEach-Object {
foreach($property in $_.PSObject.Properties) {
if ($property.Value -match "^[+-]?(\d*\,)?\d+$") {
# regex operator not needed in this case
$property.Value = $property.Value.Replace(",", ".")
}
}
$_ # => output this object
} | Export-Csv myexport.csv -NoTypeInformation

separate one CSV row into multiple rows based on contents of a single cell

I based my own solution on How to separate one row into multiple rows based on a single cell containing multiple values in the same row in a CSV file using PowerShell
Trying to iterate over one row, retrieve the zips, split it by ' ' and create a new PSCustomObject for each using the zipcodes and place the zipcode in the column with new rows in the same csv.
Sample (yes, hood = no value)
City,State,hood,zip,lat,lng
Carmel,IN,,"46290 46032 46033 46280 46082",39.9783711,-86.1180435
My non working solution is a modification of the solution in the link sited
$x = Get-Content 'D:\Carmel-IN.csv'
$y = $x | ConvertFrom-Csv -Delimiter ' '
$y | Foreach {
$current = $_
$current.zip -split ' ' | foreach {
[PSCUstomObject]#{
zip = $_
}
}
} | ConvertTo-Csv -NoTypeInformation -Delimiter ' ' | % {
$_ -replace '"',''
}
The output is the same on ISE and console ... the one word zip and nothing is written to file.
I am very new to PS, using 5.1. I have a feeling the correction is very simple but I cant see it. Looking for a result file as ...
City,State,hood,zip,lat,lng
Carmel,IN,,"46290 46032 46033 46280 46082",39.9783711,-86.1180435
,,,46290,,
,,,46032,,
,,,46033,,
,,,46280,,
,,,46082,,
UPDATE:
I managed to rework the script to get result I can use
$x = Get-Content 'D:\temp\Carmel-IN.csv'
$exportlocation = 'D:\temp\Carmel-IN.csv'
$y = $x | ConvertFrom-Csv -Delimiter ','
$y | Foreach {
$current = $_
$current.zip -split ' ' | foreach {
[PSCUstomObject]#{
City = $null
State = $null
hood = $null
zip = $_
lat = $null
lng = $null
}
}
} | Export-CSV -Append -NoTypeInformation -Delimiter ',' -Force $exportlocation | % { $_ -replace '"',''}
and the result is
City,State,hood,zip,lat,lng
Carmel,IN,,"46290 46032 46033 46280 46082",39.9783711,-86.1180435
,,,"46290",,
,,,"46032",,
,,,"46033",,
,,,"46280",,
,,,"46082",,
I am going to care of the extra "'s in another script
The | % { $_ -replace '"',''} doesn't seem to be removing them
Thanks for all suggestions
Use Select-Object:
$x = Get-Content 'D:\Carmel-IN.csv'
$y = $x | ConvertFrom-Csv -Delimiter ' '
foreach($row in $y){
foreach($zip in -split $row.zip){
$row |select *,#{Name='zip';Expression={$zip}} -ExcludeProperty zip
}
}
Alternatively, if you want to preserve the order of columns, set up the property list of properties to select prior to the loop:
$columns = $y[0].psobject.Properties.Name |ForEach-Object {
if($_ -eq 'zip'){
$_ = #{Name = 'zip';Expression = { -split $zip }}
}
$_
}
foreach($row in $y){
foreach($zip in -split $row.Zip){
$row |select $columns
}
}

Powershell loop csv file export csv line one at a time

I'm trying to loop a pipe delimited file, check if the line has 28 columns and just export that line to the file. This is the primary question and answer I'm looking for (I have researched a lot, but I'm new to PS and so many different ways I need some help). The following works but there are two issues. The export file will not let me choose pipe delimited, AND the output goes in alphabetical format by field name, not by ordinal.
Also, is there a way I can make the output not "text" qualified?
$path = Get-ChildItem c:\temp\*.txt
$staticPath = '0859'
$year = Get-Date -format yy
$month = Get-Date -format MM
$day = Get-Date -format dd
$output = 'c:\temp\' + $year + $month + $day + $staticPath + '.TXT'
$outputbad = 'c:\temp\BAD' + $year + $month + $day + $staticPath + '.TXT'
$input = 'c:\temp\' + $path.Name
$input
$csv = Import-Csv -path $input -Delimiter "|"
foreach($line in $csv)
{
$properties = $line | Get-Member -MemberType Properties
$row = ''
$properties.Count
$obj = new-object PSObject
for($i=0; $i -lt $properties.Count;$i++)
{
$column = $properties[$i]
$columnvalue = $line | Select -ExpandProperty $column.Name
#$row += $columnvalue
$obj | add-member -membertype NoteProperty -name $column.Name -value $columnvalue
}
if($properties.Count -eq 28)
{
$obj | export-csv -Path $output -Append -NoTypeInformation
#$row | Export-Csv -Path $output -Append -Delimiter "|" -NoTypeInformation
}
else
{
$obj | Export-Csv -Path $outputbad -Append -Delimiter "|" -NoTypeInformation
}
}
If you want to avoid any chance of changing the formatting of these lines files, perhaps you don't want to use the -CSV commands. Export-csv can add quotation marks etc. Here's different way that might do what you want:
$path | ForEach-Object {
$good = #()
$bad = #()
Get-Content $_ | ForEach-Object {
if (($value = $_ -split '\|').length -eq 28) {
$good += $_
} else {
$bad += $_
}
}
if ($good) { Out-File -Append -InputObject $good $output }
if ($bad) { Out-File -Append -InputObject $bad $outputbad }
}
Note however that this will count quoted values containing a pipe differently than import-csv. Pipe separated values are sometimes generated without any quoting logic.
The $values variable will be an array of individual columns, so if you want to write some code to fix them up inside the if, you can use that then join them back up with $good += $values -join '|', or perhaps use another regex to fix errors.
silly me, here is the answer. But if somebody could answer why the PSObject or the properties loop is alphabetical it would be helpful. Soon i would like to add intelligence to this to check ordinal (not alphabetic field name order) a field if its a integer or not, then i know how to fix the BAD record.
$path = Get-ChildItem c:\temp\*.txt
$staticPath = '0859'
$year = Get-Date -format yy
$month = Get-Date -format MM
$day = Get-Date -format dd
$output = 'c:\temp\' + $year + $month + $day + $staticPath + '.TXT'
$outputbad = 'c:\temp\BAD' + $year + $month + $day + $staticPath + '.TXT'
$input = 'c:\temp\' + $path.Name
$csv = Import-Csv -path $input -Delimiter "|"
foreach($line in $csv)
{
$properties = $line | Get-Member -MemberType Properties
if($properties.Count -eq 28)
{
$line | export-csv -Path $output -Append -NoTypeInformation -Delimiter "|"
}
else
{
$line | Export-Csv -Path $outputbad -Append -Delimiter "|"
}
}

How to add columns into multiple CSV files in a single directory without adding quotation marks?

I have thousands of CSV files in a single directory. I'm looking for a way how to add 2 columns (including headers) to each file.
There are some conditions:
There is always 0 value in column #5
In column #6 I want to store file name without extension (ABC)
INPUT FILE EXAMPLE (FILENAME IS ABC.CSV)
HEADER1,HEADER2,HEADER3,HEADER4
04/22/2012,47.64,47.97,47.05
04/23/2012,47.6,48.2,47.4
04/24/2012,48.13,48.33,47.84
04/25/2012,47.81,48.14,47.59
04/26/2012,47.83,48.21,47.49
04/27/2012,47.2,47.31,46.84
04/28/2012,47.01,47.05,46.33
The code I've posted below has 1 problem,
It adds quotation marks ("04/22/2012","47.64","47.97","47.05","0","ABC") to every value in a new file.
OUTPUT FILE EXAMPLE I NEED
HEADER1,HEADER2,HEADER3,HEADER4,HEADER5,HEADER6
04/22/2012,47.64,47.97,47.05,0,ABC
04/23/2012,47.6,48.2,47.4,0,ABC
04/24/2012,48.13,48.33,47.84,0,ABC
04/25/2012,47.81,48.14,47.59,0,ABC
04/26/2012,47.83,48.21,47.49,0,ABC
04/27/2012,47.2,47.31,46.84,0,ABC
04/28/2012,47.01,47.05,46.33,0,ABC
$files = Get-ChildItem ".\" -filter "*.csv"
for ($i=0; $i -lt $files.Count; $i++) {
$outfile = $files[$i].FullName + "out"
$csv = Import-Csv $files[$i].FullName
$newcsv = #()
foreach ( $row in $csv ) {
$row | Add-Member -MemberType NoteProperty -Name 'HEADER 5' -Value '0'
$row | Add-Member -MemberType NoteProperty -Name 'HEADER 6' -Value $files[$i].BaseName
$newcsv += $row
}
$newcsv | Export-Csv $files[$i].FullName -NoTypeInformation
}
And one more question. Because I have thousands of files in a directory, is this code efficient enough to do a task as fast as possible?
Somebody has already suggested me to improve code by Instead of looping thru the rows consider building te members with select
$csv = $csv | select-object *,#{n="HEADER5";e={0}},#{n="HEADER6";e={$file.BaseName}}
But I don't know how to implement his suggestion into my code.
Not tested:
$InputFolder = 'c:\SomeFolder'
$OutputFolder = 'c:\SomeOtherFolder'
Get-ChildItem $InputFolder -Filter *.* |
where {-not $_.psiscontainer} |
foreach {
$FileName = $_.Name
$BaseName = $_.Basename
$data = Get-Content $_ -ReadCount 0
"$($data[0]),Header5,Header6" | Set-Content $OutputFolder\$FileName
$data[1..($data.Length -1)] -replace '$',",0,$BaseName" |
Add-Content $OutputFolder\$FileName
}
I think my original answer depended too much on v3/v4 stuff, how about something like the following:
$files = Get-ChildItem ".\" | Where-Object { $_.Extension -eq ".csv" }
for ($i=0; $i -lt $files.Count; $i++) {
$outfile = $files[$i].FullName + "out"
$csv = Import-Csv $files[$i].FullName
$newcsv = #()
foreach ( $row in $csv ) {
$row | Add-Member -MemberType NoteProperty -Name 'HEADER5' -Value '0'
$row | Add-Member -MemberType NoteProperty -Name 'HEADER6' -Value $files[$i].BaseName
$newcsv += $row
}
( $newcsv | ConvertTo-Csv -NoTypeInformation ) | Foreach-Object { $_ -replace '"', '' } | Out-File $outfile
}