Import-Csv - Member already present issue - powershell

I have to combine multiple CSV files into a single one. Each of the CSV has a header. One of the columns header is identical. Ideally, the end file (all_out.csv) has to have a single header.
I run the PowerShell code:
Import-Csv out_1_result.csv,out_2_result.csv,out_3_result.csv,out_4_result.csv,out_5_result.csv,out_6_result.csv,out_7_result.csv,out_8_result.csv,out_9_result.csv,out_10_result.csv,out_11_result.csv,out_12_result.csv,out_13_result.csv,out_14_result.csv,out_15_result.csv,out_16_result.csv,out_17_result.csv,out_18_result.csv,out_19_result.csv,out_20_result.csv,out_21_result.csv,out_22_result.csv,out_23_result.csv,out_24_result.csv,out_25_result.csv,out_26_result.csv,out_27_result.csv,out_28_result.csv |
Export-Csv all_out.csv -NoType
and I end up with an error
Import-Csv : The member "URL" is already present.
Is there a way to ignore/fix this?


One of the columns header is identical
That means each CSV has two columns header 'URL'? Import-Csv creates objects where each header becomes a property name, e.g. #{Id=10; Url='example.com'} and using the same name again will clash.
There is no way this will work cleanly without you changing the csv files, as there is no way to say "use different column names" and also "skip the header row" just with the Import-Csv cmdlet.
The easiest change I can think of is to drop the header line from each one, e.g.:
$CsvFiles = 'out_1_result.csv','out_2_result.csv','out_3_result.csv','out_4_result.csv','out_5_result.csv','out_6_result.csv','out_7_result.csv','out_8_result.csv','out_9_result.csv','out_10_result.csv','out_11_result.csv','out_12_result.csv','out_13_result.csv','out_14_result.csv','out_15_result.csv','out_16_result.csv','out_17_result.csv','out_18_result.csv','out_19_result.csv','out_20_result.csv','out_21_result.csv','out_22_result.csv','out_23_result.csv','out_24_result.csv','out_25_result.csv','out_26_result.csv','out_27_result.csv','out_28_result.csv'
$NewFileNames = $CsvFiles | ForEach-Object {
$NewFileName = $_ + "_noheader.csv"
Get-Content $_ | Select-Object -Skip 1 | Set-Content $NewFileName -Encoding UTF8
$NewFileName # write new name to output stream
}
And then, when they have no header line, import them all and specify the header line as a parameter
Import-Csv -Path $NewFileNames -Header 'Col1', 'Col2', 'Url1', 'Url2' | Export-Csv ...


Place -Encoding parameter at the end of the command
Import-Csv -Path $output -Encoding UTF8
Export-csv -NoTypeInformation -Path $output -Encoding UTF8

Related

PowerShell remove last column of pipe delimited text file

I have a folder of pipe delimited text files that I need to remove the last column on. I'm not seasoned in PS but I found enough through searches to help. I have two pieces of code. The first creates new text files in my destination path, keeps the pipe delimiter, but doesn't remove the last column. There are 11 columns. Here is that script:
$OutputFolder = "D:\DC_Costing\Vendor Domain\CostUpdate_Development_Load_To_IMS"
ForEach ($File in (Get-ChildItem "D:\DC_Costing\Vendor Domain\CostUpdate_Development_Stage_To_IMS\*.txt"))
{
(Get-Content $File) | Foreach-Object { $_.split()[0..9] -join '|' } | Out-File $OutputFolder\$($File.Name)
}
Then this second code I tried creates the new text files on my destination path, it DOES get rid of the last column, but it loses the pipe delimiter. Ugh.
$OutputFolder = "D:\DC_Costing\Vendor Domain\CostUpdate_Development_Load_To_IMS"
ForEach ($File in (Get-ChildItem "D:\DC_Costing\Vendor Domain\CostUpdate_Development_Stage_To_IMS\*.txt"))
{
Import-Csv $File -Header col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11 -Delimiter '|' |
Foreach-Object {"{0} {1} {2} {3} {4} {5} {6} {7} {8} {9}" -f $_.col1,$_.col2,$_.col3,$_.col4,$_.col5,$_.col6,$_.col7,$_.col8,$_.col9,$_.col10} | Out-File $destination\$($File.Name)
}
I have no clue on what I'm doing wrong. I have no preference in which way I get this done but I need to keep the delimiter and the have the last column removed. Any help would be greatly appreciated.

In your plain-text processing attempt with Get-Content, you simply need to split each line by | first (.Split('|')), before extracting the fields of interest with a range operation (..) and joining them back with |:
Get-Content $File |
Foreach-Object { $_.Split('|')[0..9] -join '|' } |
Out-File $OutputFolder\$($File.Name)
In your Import-Csv-based attempt, you can take advantage of the fact that it will only read as many columns as you supply column names for, via -Header:
# Pass only 10 column names to -Header
Import-Csv $File -Header (0..9).ForEach({ 'col' + $_ }) -Delimiter '|' |
ConvertTo-Csv -Delimiter '|' | # convert back to CSV with delimiter '|'
Select-Object -Skip 1 | # skip the header row
Out-File $destination\$($File.Name)
Note that ConvertTo-Csv, just like Export-Csv by default double-quotes each field in the resulting CSV data / file.
In Windows PowerShell, you cannot avoid this, but in PowerShell (Core) 7+ you can control this behavior with -UseQuotes Never, for instance.

You can give this a try, should be more efficient than using Import-Csv, however note, this should always exclude the last column of your files no matter how many columns they have and assuming they're pipe delimited:
$OutputFolder = "D:\DC_Costing\Vendor Domain\CostUpdate_Development_Load_To_IMS"
foreach ($File in (Get-ChildItem "D:\DC_Costing\Vendor Domain\CostUpdate_Development_Stage_To_IMS\*.txt")) {
[IO.File]::ReadAllLines($File.FullName) | & {
process{
-join ($_ -split '(?=\|)' | Select-Object -SkipLast 1)
}
} | Set-Content (Join-Path $OutputFolder -ChildPath $File.Name)
}

Using Powershell to add values to existing CSV

I have been battling this for way too long and I hope you can be of assistance.
I have a CSV for which I need to add some values, preferrably with Powershell. I need to add header row and one column with a fixed text value.
My CSV, before anything done to it, looks like this:
contact-email;contact-sms;contact-firstname
test#example.com;+3580000000;Mike
And I need it to look like this:
contact-email;contact-sms;contact-firstname;order-site
test#example.com;+3580000000;Mike;Finland
So the last column "order-site" needs to be added and every line in that column should have a value of "Finland".
So far I have written this Powershell script I got off a tutorial:
$file = Import-Csv E:\Raportit\SALES\SALES_TD_S01.csv -Delimiter "`t" -Encoding Default -Header "contact-email;contact-sms;contact-firstname"
foreach($c in $file) {
Add-Member -Input $c -MemberType NoteProperty -Name "order-site" -Value "Finland"
}
$file | Export-Csv -Path "C:\Temp\test.csv" -Encoding Default -NoTypeInformation
But unfortunately, this makes the CSV look like this:
"contact-email;contact-sms;contact-firstname","order-site"
"test#example.com;+3580000000;Mike","Finland"
For the use case I have for this file, it need to look like the first should-look-like example, without double quotes and columns separated by semicolon (;). The double quotes are OK as long as the output looks like this:
"contact-email;contact-sms;contact-firstname;order-site"
"test#example.com;+3580000000;Mike;Finland"
I thank you guys so much in advance, I know this is probably a super simple task but I just cannot wrap my head around it to save my life.

if the file HAS headers:
Import-Csv -Path 'E:\Raportit\SALES\SALES_TD_S01.csv' -Delimiter ';' |
Select-Object *, #{Name = 'order-site'; Expression = {'Finland'}} |
Export-Csv -Path "C:\Temp\test.csv" -Delimiter ';' -NoTypeInformation
if the file DOES NOT HAVE headers:
$headers = 'contact-email','contact-sms','contact-firstname'
Import-Csv -Path 'E:\Raportit\SALES\SALES_TD_S01.csv' -Delimiter ';' -Header $headers |
Select-Object *, #{Name = 'order-site'; Expression = {'Finland'}} |
Export-Csv -Path "C:\Temp\test.csv" -Delimiter ';' -NoTypeInformation
CSV Output:
"contact-email";"contact-sms";"contact-firstname";"order-site"
"test#example.com";"+3580000000";"Mike";"Finland"

Export-Csv adding unwanted header double quotes

I have got a source CSV file (without a header, all columns delimited by a comma) which I am trying split out into separate CSV files based upon the value in the first column and using that column value as the output file name.
Input file:
S00000009,2016,M04 01/07/2016,0.00,0.00,0.00,0.00,0.00,0.00,750.00,0.00,0.00
S00000009,2016,M05 01/08/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000009,2016,M06 01/09/2016,0.00,0.00,0.00,0.00,0.00,0.00,600.00,0.00,0.00
S00000010,2015,W28 05/10/2015,2275.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00
S00000010,2015,W41 04/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000010,2015,W42 11/01/2016,0.00,0.00,0.00,0.00,0.00,0.00,568.75,0.00,0.00
S00000012,2015,W10 01/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W11 08/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
S00000012,2015,W12 15/06/2015,0.00,0.00,0.00,0.00,0.00,0.00,650.00,0.00,0.00
My PowerShell script looks like this:
Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def |
Group-Object -Property "service_id" |
Foreach-Object {
$path = $_.Name + ".csv";
$_.group | Export-Csv -Path $path -NoTypeInformation
}
Output files:
S00000009.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000009","2016","M04 01/07/2016","0.00","0.00","0.00","0.00","0.00","0.00","750.00","0.00","0.00"
"S00000009","2016","M05 01/08/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
"S00000009","2016","M06 01/09/2016","0.00","0.00","0.00","0.00","0.00","0.00","600.00","0.00","0.00"
S00000010.csv:
"service_id","year","period","cash_exp","cash_inc","cash_def","act_exp","act_inc","act_def","comm_exp","comm_inc","comm_def"
"S00000010","2015","W28 05/10/2015","2275.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00","0.00"
"S00000010","2015","W41 04/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
"S00000010","2015","W42 11/01/2016","0.00","0.00","0.00","0.00","0.00","0.00","568.75","0.00","0.00"
It is generating the new files using the header value in column 1 (service_id).
There are 2 problems.
The output CSV file contains a header row which I don't need.
The columns are enclosed with double quotes which I don't need.

First of all the .csv file needs headers and the quote marks as a csv file structure. But if you don't want them then you can go on with a text file or...
$temp = Import-Csv INPUT_FILE.csv -Header service_id,year,period,cash_exp,cash_inc,cash_def,act_exp,act_inc,act_def,comm_exp,comm_inc,comm_def | Group-Object -Property "service_id" |
Foreach-Object {
$path=$_.name+".csv"
$temp0 = $_.group | ConvertTo-Csv -NoTypeInformation | Select-Object -Skip 1
$temp1 = $temp0.replace("""","")
$temp1 > $path
}
But this output is not a "real" csv file.
Hope that helps.

For your particular scenario you could probably use a simpler approach. Read the input file as a plain text file, group the lines by splitting off the first field, then write the groups to output files named after the groups:
Get-Content 'INPUT_FILE.csv' |
Group-Object { $_.Split(',')[0] } |
ForEach-Object { $_.Group | Set-Content ($_.Name + '.csv') }

Another solution,
using no named headers but simply numbers (as they aren't wanted in output anyway)
avoiding unneccessary temporary files.
removing only field delimiting double quotes.
Import-Csv INPUT_FILE.csv -Header (1..12) |
Group-Object -Property "1" | Foreach-Object {
($_.Group | ConvertTo-Csv -NoType | Select-Object -Skip 1).Trim('"') -replace '","',',' |
Set-Content -Path ("{0}.csv" -f $_.Name)
}

Merges csv files from directory into a single csv file PowerShell

How can I run one single PowerShell script that does the following in series?
Adds a the filename of all csv files in a directory as a column in the end of each file using this script:
Get-ChildItem *.csv | ForEach-Object {
$CSV = Import-CSV -Path $_.FullName -Delimiter ","
$FileName = $_.Name
$CSV | Select-Object *,#{N='Filename';E={$FileName}} | Export-CSV $_.FullName -NTI -Delimiter ","}
Merges all csv files in the directory into a single csv file
Keeping only a header (first row) only from first csv and excluding all other first rows from files.
Similiar to what kemiller2002 has done here, except one script with csv inputs and a csv output.

Bill's answer allows you to combine CSVs, but doesn't tack file names onto the end of each row. I think the best way to do that would be to use the PipelineVariable common parameter to add that within the ForEach loop.
Get-ChildItem \inputCSVFiles\*.csv -PipelineVariable File |
ForEach-Object { Import-Csv $_ | Select *,#{l='FileName';e={$File.Name}}} |
Export-Csv \outputCSVFiles\newOutputFile.csv -NoTypeInformation
That should accomplish what you're looking for.

This is the general pattern:
Get-ChildItem \inputCSVFiles\*.csv |
ForEach-Object { Import-Csv $_ } |
Export-Csv \outputCSVFiles\newOutputFile.csv -NoTypeInformation
Make sure the output CSV file has a different filename pattern, or use a different directory name (like in this example).

If your csv files dont have always same header you can do it :
$Dir="c:\temp\"
#get header first csv file founded
$header=Get-ChildItem $Dir -file -Filter "*.csv" | select -First 1 | Get-Content -head 1
#header + all rows without header into new file
$header, (Get-ChildItem $Dir -file -Filter "*.csv" | %{Get-Content $_.fullname | select -skip 1}) | Out-File "c:\temp\result.csv"

PowerShell write integer to file after x number of tabs

I'm sure this is ridiculously easy, but I'm a noob and trying to learn PowerShell.
I want to write an integer to each line of a tab delimited file, i.e. each line has 20 tabs; put a 1 after the nth tab.
No need to overwrite what's already there because in the current scenario there isn't anything.
Thanks!

If there is a header line then just import the file as a CSV, run it through a ForEach-Object loop and set that column to the integer that you want, then export the CSV again.
Import-CSV $File -Delimiter "`t" | ForEach{$_.ColumnName = $Integer} | Export-CSV $File -Delimiter "`t" -NoTypeInfo
If there is no header you could do the same thing and define your own headers. Except you would use ConvertTo-CSV instead of Export-CSV and then use Select to skip the header row, and use Set-Content to write the file. For my example I set the 7th column to $Integer.
$Headers = 1..20|ForEach{"Col$_"}
Import-CSV $File -Delimiter "`t" -Header $Headers | ForEach{$_.Col7 = $Integer} | ConvertTo-CSV -Delimiter "`t" -NoTypeInfo | Select -Skip 1 | Set-Content $File