Export-Csv adding unwanted header double quotes - powershell
I have got a source CSV file (without a header, all columns delimited by a comma) which I am trying split out into separate CSV files based upon the value in the first column and using that column value as the output file name.
Input file:
S00000009,2016,M04 01/07/2016,0.00,0.00,0.00,0.00,0.00,0.00,750.00,0.00,0.00
S00000009,2016,M05 01/08/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000009,2016,M06 01/09/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000010,2015,W28 05/10/2015,2275.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
S00000010,2015,W41 04/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000010,2015,W42 11/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000012,2015,W10 01/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W11 08/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W12 15/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
My PowerShell script looks like this:
Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def |
Group-Object -Property "service_id" |
Foreach-Object {
$path = $_.Name + ".csv";
$_.group | Export-Csv -Path $path -NoTypeInformation
}
Output files:
S00000009.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000009","2016","M04 01/07/2016","0.00","0.00","0.00","0.00","0.00","0.00","750.00","0.00","0.00"
"S00000009","2016","M05 01/08/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
"S00000009","2016","M06 01/09/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
S00000010.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000010","2015","W28 05/10/2015","2275.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00"
"S00000010","2015","W41 04/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
"S00000010","2015","W42 11/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
It is generating the new files using the header value in column 1 (service_id).
There are 2 problems.
The output CSV file contains a header row which I don't need.
The columns are enclosed with double quotes which I don't need.
First of all the .csv file needs headers and the quote marks as a csv file structure. But if you don't want them then you can go on with a text file or...
$temp = Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def | Group-Object -Property "service_id" |
Foreach-Object {
$path=$_.name+".csv"
$temp0 = $_.group | ConvertTo-Csv -NoTypeInformation | Select-Object -Skip 1
$temp1 = $temp0.replace("""","")
$temp1 > $path
}
But this output is not a "real" csv file.
Hope that helps.
For your particular scenario you could probably use a simpler approach. Read the input file as a plain text file, group the lines by splitting off the first field, then write the groups to output files named after the groups:
Get-Content 'INPUT_FILE.csv' |
Group-Object { $_.Split(',')[0] } |
ForEach-Object { $_.Group | Set-Content ($_.Name + '.csv') }
Another solution,
using no named headers but simply numbers (as they aren't wanted in output anyway)
avoiding unneccessary temporary files.
removing only field delimiting double quotes.
Import-Csv INPUT_FILE.csv -Header (1..12) |
Group-Object -Property "1" | Foreach-Object {
($_.Group | ConvertTo-Csv -NoType | Select-Object -Skip 1).Trim('"') -replace '","',',' |
Set-Content -Path ("{0}.csv" -f $_.Name)
}
Related
PowerShell CSV, take a specific row from each line and combine it into one CSV
I have 300 CSV files all separated in a directory.
I want to get one specific criteria from each CSV and put it into another using PowerShell.
This is the line I have, but doesn't seem to work.
Get-ChildItem -Filter "*Results.csv" | Get-Content | Where-Object {$_.NAME -eq "Cage,Johnny"} | Add-Content "test.csv"
I filtered for the specific CSVs I wanted in my directory with gci, Got the content of each using Get-Content and Where the value is Johnny Cage in the NAME column, and Add-Content into a test.csv file but doesn't work.
Any help would be great!
You need to deserialize your CSV text into objects with properties that can be referenced. Then you can compare the Name property. You can do the following if all your csv files have the same headers.
Get-ChildItem -Filter "*Results.csv" | Foreach-Object {
Import-Csv $_.FullName |
Where-Object {$_.NAME -eq "Cage,Johnny"} } |
Export-Csv "test.csv"
If your CSV files contain different headers, then you have a couple of options. One, you could create your output CSV with all possible headers that exist across all files (or just the headers you want as long as they are the same across all files). Second, you could just output your data rows and have a broken CSV.
# Broken CSV Approach
Get-ChildItem -Filter "*Results.csv" | Foreach-Object {
Import-Csv $_.FullName |
Where-Object {$_.NAME -eq "Cage,Johnny"}} | Foreach-Object {
$_ | ConvertTo-Csv -Notype | Select-Object -Skip 1
} | Add-Content test.csv
I think I got it.
Get-ChildItem -Filter *Results.csv |
ForEach-Object{
Import-Csv $_.NAME | ? { $_.EMPLID -eq "Cage,Johnny"}
} | Export-Csv "test.csv"
Powershell adds quotes to splitted CSV file
I'm trying to split a csv file by the first digits of the longitude column. Here is a sample:
X,Y,TYPE,SPEED,DirType,Direction
-44.058251,-19.945982,1,30,1,339
-54.629503,-20.497509,1,30,1,263
-54.646202,-20.496151,1,30,1,86
I have no powershell knowledge but I found some script online and it did what I wanted:
Import-Csv maparadar.csv
| Group-Object -Property {($_.x)[0..2] -join ""}
| Foreach-Object {$path=$_.name+".csv" ; $_.group
| Export-Csv -Path $path -NoTypeInformation}
With this I get output files like -44.csv, -54.csv
But it adds unwanted quotes to every field in the output file like:
"X","Y","TYPE","SPEED","DirType","Direction"
"-46.521991","-23.690235","1","30","1","169"
"-46.670774","-23.756021","1","30","1","281"
"-46.549897","-23.120720","1","30","1","99"
Is there any way I can export the csv without adding those quotes?
The following should provide the desired output:
Import-Csv maparadar.csv |
Group-Object -Property {($_.x)[0..2] -join ""} |
Foreach-Object { $path=$_.name+".csv" ; ($_.group |
ConvertTo-Csv -NoTypeInformation) -Replace '"' |
Set-Content -Path $path }
Explanation:
We replaced your Export-Csv with ConvertTo-Csv, which provides the CSV output to the console/pipeline rather than outputting to the file. Those CSV formatted outputs are sent through the -Replace operator to replace the literal " characters. Finally the formatted output is sent to the desired file using Set-Content -Path $path.
Merges csv files from directory into a single csv file PowerShell
How can I run one single PowerShell script that does the following in series?
Adds a the filename of all csv files in a directory as a column in the end of each file using this script:
Get-ChildItem *.csv | ForEach-Object {
$CSV = Import-CSV -Path $_.FullName -Delimiter ","
$FileName = $_.Name
$CSV | Select-Object *,#{N='Filename';E={$FileName}} | Export-CSV $_.FullName -NTI -Delimiter ","}
Merges all csv files in the directory into a single csv file
Keeping only a header (first row) only from first csv and excluding all other first rows from files.
Similiar to what kemiller2002 has done here, except one script with csv inputs and a csv output.
Bill's answer allows you to combine CSVs, but doesn't tack file names onto the end of each row. I think the best way to do that would be to use the PipelineVariable common parameter to add that within the ForEach loop.
Get-ChildItem \inputCSVFiles\*.csv -PipelineVariable File |
ForEach-Object { Import-Csv $_ | Select *,#{l='FileName';e={$File.Name}}} |
Export-Csv \outputCSVFiles\newOutputFile.csv -NoTypeInformation
That should accomplish what you're looking for.
This is the general pattern:
Get-ChildItem \inputCSVFiles\*.csv |
ForEach-Object { Import-Csv $_ } |
Export-Csv \outputCSVFiles\newOutputFile.csv -NoTypeInformation
Make sure the output CSV file has a different filename pattern, or use a different directory name (like in this example).
If your csv files dont have always same header you can do it :
$Dir="c:\temp\"
#get header first csv file founded
$header=Get-ChildItem $Dir -file -Filter "*.csv" | select -First 1 | Get-Content -head 1
#header + all rows without header into new file
$header, (Get-ChildItem $Dir -file -Filter "*.csv" | %{Get-Content $_.fullname | select -skip 1}) | Out-File "c:\temp\result.csv"
Spliting a csv file based on the value of one column
I have a csv file that I have to split based on the value of a column.
I'm using the following script to do so:
Import-Csv test.csv | Group-Object -Property "Nr dep" |
Foreach-Object {$path=$_.name+".csv" ; $_.group |
Export-Csv -Path E:\PowerShell\script\$path -NoTypeInformation}
The file is split to files based on the Nr dep value column but with quotes and it works only with comma delimited csv files.
I tried to use -replace but still no result (maybe I'am writing it in a bad manner)
Import-Csv test.csv | Group-Object -Property "Nr dep" |
Foreach-Object {$path=$_.name+".csv" ; ($_.group |
ConvertTo-Csv -NoTypeInformation) -replace '"', "" | Out-File E:\PowerShell\script\$path -Force}
1) How can I make the delimitation a semi comma instead of a comma
2) how can I get rid of the quotes
3) is it possible to have an .xlsx outfile instead of a .csv file
1) You can specify the delimiter by using the -Delimiter parameter: Import-Csv test.csv -Delimiter ';' 2) By using -replace '"' which you already do. 3) You will need a framework // application for that.
Export-Csv emits Length and not values
I want to read a CSV file and output a CSV file with only one (1) field. I have tried to create a concise example.
PS C:\src\powershell> Get-Content .\t.csv
field1,field2,field3
1,2,3
4,55,6
7,888,9
PS C:\src\powershell> Import-Csv -Path .\t.csv | `
>> ForEach-Object {
>> $_.field2 `
>> } | `
>> Export-Csv -Path .\x.csv -NoTypeInformation
>>
The problem is that the Length of field2 is written to the exported CSV file. I want the field header to be "field2" and the values to be the value from the original CSV file. Also, I only want quotes where they are required; not everywhere.
I have read Export-CSV exports length but not name and Export to CSV only returning string length. But these do not seem to address producing an actual CSV file with a header and one field value.
PS C:\src\powershell> get-content .\x.csv
"Length"
"1"
"2"
"3"
CSV object uses note properties in each row to store its fields so we'll need to filter each row object and leave just the field(s) we want using Select-Object cmdlet (alias: select), which processes the entire CSV object at once:
Import-Csv 1.csv | select field2 | Export-Csv 2.csv -NoTypeInformation
Note, there's no need to escape the end of line if it ends with |, {, (, or ,.
It's possible to specify several fields: select field2, field3.
To strip unneeded doublequotes, general multi-field case:
Import-Csv 1.csv |
select field2 |
%{
$_.PSObject.Properties | %{ $_.value = $_.value -replace '"', [char]1 }
$_
} |
ConvertTo-Csv -NoTypeInformation |
%{ $_ -replace '"(\S*?)"', '$1' -replace '\x01', '""' } |
Out-File 2.csv -Encoding ascii
Simplified one-field case:
Import-Csv 1.csv |
select field2 |
%{
$_.field2 = $_.field2 -replace '"', [char]1
$_
} |
ConvertTo-Csv -NoTypeInformation |
%{ $_ -replace '"(\S*?)"', '$1' -replace '\x01', '""' } |
Out-File 2.csv -Encoding ascii
A tricky case of embedded quotes inside a field was solved by temporary replacing them with a control character code 01 (there are just a few that can be used in a typical non-broken text file: 09/tab, 0A/line feed, 0D/carriage return).
As per WOxxOm's response, Select-Object is best way to select only field from an input and pipe to output.
Regarding the quote marks, this is a known (and frustrating) issue with PowerShell. Specifying , as the delimiter did not help.
I have gotten round it by using ConvertTo-Csv and Foreach-Object replacements. THe replacements will need to be more complex if your data contains quote marks.
Import-Csv .\1.csv |
Select-Object field2 |
ConvertTo-Csv -NoTypeInformation |
ForEach-Object {$_ -replace '"',''} |
Out-File .\2.csv