literal newline/carriage found in data exported with pandas to csv - postgresql

Question

I have the following text file that has CRLF at the end of each line and that has a relatively small number of bad rows (b'Skipping line 55000: expected 14 fields, saw 15\n').
0.0;0.7;John;0.29
1.0;0.23;Mike;0.55
0.0;0.72;;Jane;;-3.4
0.0;0.98;Gil;0.68
0.0;0.48;;;0
1.0;0.34;Karl;0.73
0.0;0.44;James;0.06
1.0;0.4;Kiki;0.74
0.0;0.18;Albert;0.18
1.0;0.53;Mark;0.53
I import the file with pandas, python 3.5.2 for windows 10 as follows:
with open('E:\DATA\my_file.txt','rb') as f:
df = pd.read_csv(f, sep=';', encoding='CP1252', error_bad_lines=False) // skipping bad rows
df looks like this: // the bad rows seem to be empty now
0.0;0.7;John;0.29
1.0;0.23;Mike;0.55
0.0;0.98;Gil;0.68
1.0;0.34;Karl;0.73
0.0;0.44;James;0.06
1.0;0.4;Kiki;0.74
0.0;0.18;Albert;0.18
1.0;0.53;Mark;0.53
Then I export the table into csv as follows:
with open('E:\DATA\csv_file.csv','w',newline='\n') as outfile:
df.to_csv(outfile, sep=';',index = False, line_terminator = '\r')
csv_file.csv looks like this: // the empty rows seem to be removed
0.0;0.7;John;0.29
1.0;0.23;Mike;0.55
0.0;0.98;Gil;0.68
1.0;0.34;Karl;0.73
0.0;0.44;James;0.06
1.0;0.4;Kiki;0.74
0.0;0.18;Albert;0.18
1.0;0.53;Mark;0.53
Unfortunately when I import the file into postgres with the following code:
set client_encoding to 'WIN1252';
COPY my_table FROM 'E:\DATA\csv_file.csv' (DELIMITER(';'));
I get the following error:
ERROR: literal newline found in data
HINT: Use "\n" to represent newline.
CONTEXT: COPY my_table, line 25408: ""
When I open the csv_file in nopetad++, I see that it has "CR" at the end of each row up to line 25407, line 25408 and a few others have "CRLF" at the end of the line.
I tried a few things I read on this site like opnening the file in binary mode, but nothing helped.
Can anyone explain to me what is going on here and how I can solve this? Thanks

Answer 1

UPDATE2: it just works fine:
In [194]: pd.read_csv(r'D:\download\onetest.txt', sep=';')
Out[194]:
COL1 COL2 COL2.1 COL3 COL4 COL5 COL6 COL7 COL8 COL9 COL10
0 23 21.0 UP 15/08/1986 BOBO NaN 1071001 268-Z DON 1620 NaN
1 012R 65.0 UP 15/10/1986 ESTO NaN 15065108 066-B DON 8415 NaN
2 234 8.0 EIJFTERF 17/12/1989 KING NaN 15571508 0776-V UP 6329 NaN
UPDATE: if your file(s) are small enough to fit into memory you can try this:
import io
data = []
with open(r'E:\DATA\my_file.txt') as f:
for line in f:
data.append(line.rstrip())
df = pd.read_csv(io.StringIO(data), sep=';', encoding='CP1252', error_bad_lines=False)
df.to_csv(r'E:\DATA\csv_file.csv', sep=';', index = False)
OLD answer:
You are using '\r' as a line-break and PostgreSQL's COPY command expects '\n', so try the following:
df = pd.read_csv(r'E:\DATA\my_file.txt', sep=';', encoding='CP1252', error_bad_lines=False)
df.to_csv(r'E:\DATA\csv_file.csv', sep=';', index = False)
in PostgreSQL:
set client_encoding to 'WIN1252';
COPY my_table FROM 'E:\DATA\csv_file.csv' (DELIMITER(';'));