How do I process multiple CSV files in Powershell and give them output names? - powershell

Question

So I'm trying to process CSV files, then giving the output new name. I can do it with one file by explicitly specifying the file name. But is there a way / wildcard I can use to make the script to process multiple files at the same time? Let's just say I want to process anything with .csv as an extension. Here's my script that's used to process a specific file
$objs =#();
$output = Import-csv -Path D:\TEP\FilesProcessing\Test\file1.csv | ForEach {
$Object = New-Object PSObject -Property #{
Time = $_.READ_DTTM
Value = $_.{VALUE(KWH)}
Tag = [String]::Concat($_.SUBSTATION,'_',$_.CIRCUITNAME,'_',$_.PHASE,'_',$_.METERID,'_KWH')
}
$objs += $Object;
}
$objs
$objs | Export-CSv -NoTypeInformation D:\TEP\FilesProcessing\Test\file1_out.csv

Answer 1

You can combine Get-ChildItem and Import-Csv.
Here's an example that specifies different input and output directories to avoid name collisions:
$inputPath = "D:\TEP\FilesProcessing\Test"
$outputPath = "D:\TEP\FilesProcessing\Output"
Get-ChildItem (Join-Path $inputPath "*.csv") | ForEach-Object {
$outputFilename = Join-Path $outputPath $_.Name
Import-Csv $_.FullName | ForEach-Object {
New-Object PSObject -Property #{
"Time" = $_.READ_DTTM
"Value" = $_.{VALUE(KWH)}
"Tag" = "{0}_{1}_{2}_{3}_KWH" -f $_.SUBSTATION,$_.CIRCUITNAME,$_.PHASE,$_.METERID
}
} | Export-Csv $outputFilename -NoTypeInformation
}
Note that there's no need for creating an array and repeatedly appending it. Just output the custom objects you want and export afterwards.

Answer 2

Use the Get-Childitem and cut out all the unnecessary intermediate variables so that you code it in a more Powershell type way. Something like this:
Get-CHhilditems 'D:\TEP\FilesProcessing\Test\*.csv' | % {
Import-csv $_.FullName | % {
New-Object PSObject -Property #{
Time = $_.READ_DTTM
Value = $_.{VALUE(KWH)}
Tag = '{0}_{1}_{2}_{3}_KWH' -f $_.SUBSTATION, $_.CIRCUITNAME, $_.PHASE, $_.METERID
}
} | Export-CSv ($_.FullName -replace '\.csv', '_out.csv') -NoTypeInformation
}

Answer 3

The Get-ChildItem is very useful for situations like this.
You can add wildcards directly into the path:
Get-ChildItem -Path D:\TEP\FilesProcessing\Test\*.csv
You can recurse a path and use the provider to filter files:
Get-ChildItem -Path D:\TEP\FilesProcessing\Test\ -recurse -include *.csv
This should get you what you need.
$Props = #{
Time = [datetime]::Parse($_.READ_DTTM)
Value = $_.{VALUE(KWH)}
Tag = $_.SUBSTATION,$_.CIRCUITNAME,$_.PHASE,$_.METERID,'KWH' -join "_"
}
$data = Get-ChildItem -Path D:\TEP\FilesProcessing\Test\*.csv | Foreach-Object {Import-CSV -Path $_.FullName}
$data | Select-Object -Property $Props | Export-CSv -NoTypeInformation D:\TEP\FilesProcessing\Test\file1_out.csv
Also when using Powershell avoid doing these things:
$objs =#();
$objs += $Object;