I'm trying (badly) to work through combining CSV files into one file and prepending a column that contains the file name. I'm new to PowerShell, so hopefully someone can help here.
I tried initially to do the well documented approach of using Import-Csv / Export-Csv, but I don't see any options to add columns.
Get-ChildItem -Filter *.csv | Select-Object -ExpandProperty FullName | Import-Csv | Export-Csv CombinedFile.txt -UseQuotes Never -NoTypeInformation -Append
Next I'm trying to loop through the files and append the name, which kind of works, but for some reason this stops after the first row is generated. Since it's not a CSV process, I have to use the switch to skip the first title row of each file.
$getFirstLine = $true
Get-ChildItem -Filter *.csv | Where-Object {$_.Name -NotMatch "Combined.csv"} | foreach {
$filePath = $_
$collection = Get-Content $filePath
foreach($lines in $collection) {
$lines = ($_.Basename + ";" + $lines)
}
$linesToWrite = switch($getFirstLine) {
$true {$lines}
$false {$lines | Select -Skip 1}
}
$getFirstLine = $false
Add-Content "Combined.csv" $linesToWrite
}
This is where the -PipelineVariable parameter comes in real handy. You can set a variable to represent the current iteration in the pipeline, so you can do things like this:
Get-ChildItem -Filter *.csv -PipelineVariable File | Where-Object {$_.Name -NotMatch "Combined.csv"} | ForEach-Object { Import-Csv $File.FullName } | Select *,#{l='OriginalFile';e={$File.Name}} | Export-Csv Combined.csv -Notypeinfo
Merging your CSVs into one and adding a column for the file's name can be done as follows, using a calculated property on Select-Object:
Get-ChildItem -Filter *.csv | ForEach-Object {
$fileName = $_.Name
Import-Csv $_.FullName | Select-Object #{
Name = 'FileName'
Expression = { $fileName }
}, *
} | Export-Csv path/to/merged.csv -NoTypeInformation
I am trying to Remove unnecessary commas in a column in the CSV file. For now, I know a few issues and hard-coded it, But I wanted the code to be dynamic. Any suggestions are greatly appreciated.
$FilePath = "C:\Test\"
Get-ChildItem $FilePath -Filter .csv | ForEach-Object {
(Get-Content $_.FullName -Raw) | Foreach-Object {
$_ -replace ',"Frederick, Fred",' , ',"Frederick Fred",' `
-replace ',"Brian, Josiah",' , ',"Brian Josiah",' `
-replace ',"Lisinopril ,Tablet / 20MG",' , ',"Lisinopril Tablet / 20MG",'
} | Set-Content $_.FullName
}
Try this, also note that I worked with the csv sample that you gave here.It might not work with other csv files.
also make sure that you change the path of %YOURCSVFILE% to the real path of your file
#import the csv
$csv = Import-Csv -Path %YOURCSVFILE% -Delimiter ','
#going each row and replacing commas
foreach ($desc in $csv){
$desc.Desc = $desc.Desc -replace ',',''
}
#exporting the csv
$csv | Export-csv -NoTypeInformation "noCommas.csv"
Here's a few more alteratives for you:
Method 1. Loop through the rows with foreach(..) and capture the output:
$result = foreach ($row in (Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv')) {
$row.Desc = $row.Desc -replace ','
$row # output the updated item
}
$result | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 2. Use ForEach-Object and the automatic variable $_. Pipe the results through:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' | ForEach-Object {
$_.Desc = $_.Desc -replace ','
$_ # output the updated item
} | Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
Method 3. Use a calculated property:
Import-Csv -Path 'D:\Test\FileWithCommasInDescription.csv' |
Select-Object ID, #{Name = 'Desc'; Expression = {$_.Desc -replace ','}}, Nbr -ExcludeProperty Desc |
Export-Csv -Path 'D:\Test\FileWithoutCommasInDescription.csv' -NoTypeInformation
All will result in a new CSV file
"ID","Desc","Nbr"
"12","Frederick Fred","11"
"21","Brian Josiah","31"
"13","Lisinopril Tablet / 20MG","17"
I have a simple text file that looks like this...
A,400000051115,null,null,null,null,null,null,null,20190312,090300,Answer Machine,2019,3,14,10,0
A,400000051117,null,null,null,null,null,null,null,20190312,090300,Confirmed,2019,3,14,10,30
A,400000051116,null,null,null,null,null,null,null,20190312,090300,Answer Machine,2019,3,14,11,0
A,400000051114,null,null,null,null,null,null,null,20190312,090300,Wants to Cancel,2019,3,14,9,0
A,400000051117,null,null,null,null,null,null,null,20190312,091800,SMS Sent,2019,3,14,10,30
A,400000051116,null,null,null,null,null,null,null,20190312,091800,SMS Sent,2019,3,14,11,0
A,400000051115,null,null,null,null,null,null,null,20190312,091800,SMS Sent,2019,3,14,10,0
A,400000051116,null,null,null,null,null,null,null,20190312,093000,Appointment Cancelled/Rescheduled Via SMS,2019,3,14,11,0
I need to save all the lines except those that have "SMS Sent" in them to a new file. I am using the following...
get-content $SourceFile.FullName | select-string -pattern 'SMS Sent' -notmatch | Out-File $targetFile
Why in the resulting file do I get a blank first line?
If you change Out-File $targetFile to Out-Host or even just omit that last segment in the pipeline, you will see a blank line in the console output, too.
The output analog of Get-Content is Set-Content, so if you change Out-File $targetFile to Set-Content $targetFile the first line is no longer blank.
Also, since you're working with a CSV file you could use Import-CSV to read the data and Where-Object to filter on that specific column, although a little extra work is required to specify the headers and omit them from the output file...
$csvHeaders = 1..17 | ForEach-Object -Process { "Column $_" }
$csvHeaders[11] = 'Status'
Import-Csv -Path $SourceFile.FullName -Header $csvHeaders `
| Where-Object -Property 'Status' -NE -Value 'SMS Sent' `
| ConvertTo-Csv -NoTypeInformation `
| Select-Object -Skip 1 `
| Set-Content $targetFile
...which writes...
"A","400000051115","null","null","null","null","null","null","null","20190312","090300","Answer Machine","2019","3","14","10","0"
"A","400000051117","null","null","null","null","null","null","null","20190312","090300","Confirmed","2019","3","14","10","30"
"A","400000051116","null","null","null","null","null","null","null","20190312","090300","Answer Machine","2019","3","14","11","0"
"A","400000051114","null","null","null","null","null","null","null","20190312","090300","Wants to Cancel","2019","3","14","9","0"
"A","400000051116","null","null","null","null","null","null","null","20190312","093000","Appointment Cancelled/Rescheduled Via SMS","2019","3","14","11","0"
...to $targetFile. Note that all of the values are quoted now. If your input file does have headers then you could use simply...
Import-Csv -Path $SourceFile.FullName `
| Where-Object -Property 'Status' -NE -Value 'SMS Sent' `
| Export-Csv -NoTypeInformation -LiteralPath $targetFile
In either case the output file will not contain a leading blank line.
Starting with a 500,000 line CSV, I need to split the files by day and hour (the second and third columns). I've tried the modify the group to include the hour and while I see the hour get added to my filename, I get no results in the exported file.
The foreach doing the work:
foreach ($group in $data | Group Day,hour) {
$data | Where-Object { $_.Day -and $_.Hour -eq $group.Name }
ConvertTo-Csv -NoTypeInformation |
foreach {$_.Replace('"','')} |
Out-File "$Path\Testfile_$($group.name -replace $regexA, '').csv"
Sample Data:
Bob,1/27/2012,8:00,Basic,Operations
Charlie,2/3/2012,9:00,Advanced,Production
Bill,3/7/2012,10:00,Advanced,Production
You could import the CSV, determine the output filename on the fly, and append each record to the matchning file:
Import-Csv 'C:\path\to\input.csv' | ForEach-Object {
$filename = ('output_{0}_{1}.csv' -f $_.Day, $_.Hour) -replace '[/:]'
$_ | Export-Csv "C:\path\to\$filename" -Append -NoType
}
Note that Export-Csv -Append requires PowerShell v3 or newer.