I have CSV file(10+ million records) like below format.
"1","abc","","test","","","100"
I have tried to load CSV file's data into greenplum table using gpload where I passed gpload parameter NULL_AS : "" but it failed
It loads file if I replace "" with blank and leaving NULL_AS blank, like this :
"1","abc",,"test",,,"100"
Related
I am trying to import the following csv file (named dummy.csv):
"F1", "F2", "F3"
1,2,"a"
3,4, "b"
3.4,2.4,"c"
to PostgreSQL using the pgAdmin GUI with the following command:
"C:\\Program Files\\PostgreSQL\\12\\bin\\psql.exe"
--command " "\\copy public.\"Result\"
FROM 'dummy.csv'
DELIMITER ','
CSV HEADER ENCODING 'UTF8'
QUOTE '\"'
ESCAPE '''';""
but keep getting the following error:
ERROR: extra data after the last expected column
CONTEXT: COPY Result, line 2: «1,2,"a"
What am I doing wrong?
The csv is not very 'clean':
"F1", "F2", "F3"
1,2,"a"
3,4, "b"
3.4,2.4,"c"
There are spaces between the separator and the quotes which could give problems with some parsers.
Maybe try with the following first and see what this gives:
"F1","F2","F3"
1,2,"a"
3,4,"b"
3.4,2.4,"c"
You could even remove the quotes in this example.
I have a csv file :
SrNo,TradeDate,TradeTime,SettlementNo
1,25-Apr-17,14:48:19,NSE-N-2017077
2,05-May-17,09:33:28,NSE-N-2017084
I am trying to upload this file using the command:
copy public.shares(SrNo,TradeDate,TradeTime,SettlementNo) from 'C:/Users/Dell/Downloads/report_jul2018.csv' with csv header;
It is giving the error:
ERROR: column "srno" of relation "shares" does not exist SQL state: 42703
Any solution?
I'm importing data from csv file into postgres db using pgadmin 4
Everything is ok but I get an issue when I try to import a file which contain some data like this
“‘t Zand, Vlotbrug”
“Dussen, `t Middeltje”
as you can see, the data contains
`
and
'
I also tried to import the file with utf-8 encoding but could not.
Anyone knows how to solve this issue?
Updated
Structure:
stop_id,stop_code,stop_name,stop_lat,stop_lon,location_type,parent_station,stop_timezone,wheelchair_boarding,platform_code,zone_id
Data:
stoparea:123953,,"De Zande, 'Koelucht'",52.5184475,5.956368,1,,,0,,
Error:
ERROR: unterminated CSV quoted field
CONTEXT: COPY stops, line 69400: "stoparea:123953,,"De Zande, 'Koelucht'",52.5184475,5.956368,1,,,0,,
stoparea:120536,,"Poortvliet, Zu..."
Updated 2
Command:
"/Applications/pgAdmin 4.app/Contents/SharedSupport/psql" --command "
"\copy transit.stops (stop_id, stop_code, stop_name, stop_lat,
stop_lon, location_type, parent_station, stop_timezone,
wheelchair_boarding, platform_code, zone_id) FROM
'/Users/tvtan/Desktop/gtfs-nl/stops.txt' DELIMITER ',' CSV HEADER
QUOTE '\"' ESCAPE '''';""
UI:
From the command line it looks like you have defined the ESCAPE character as a single quote. The single quote appears in your data but is not escaped.
The default ESCAPE character is the same as the QUOTE character.
More information here
Hi I did this command on my original DB :
\COPY (SELECT * FROM cms_title WHERE title = 'Migration-test') TO '/Users/JayCee/cms_title_dump.csv' WITH CSV HEADER DELIMITER ';' NULL AS '';
It worked fine.
But when I try to copy it to an other DB I have this :
\COPY cms_title FROM 'cms_title_dump.csv' DELIMITER ';' CSV;
ERROR: invalid input syntax for integer: "id"
CONTEXT: COPY cms_title, line 1, column id: "id"
I don't understand, here is what I have when I do a cat -e on my file :
id;language;title;page_title;menu_title;meta_description;slug;path;has_url_overwrite;redirect;creation_date;published;publisher_is_draft;publisher_state;page_id;publisher_public_id$
217;en;Migration-test;"";"";"";migration-test;migration-test;f;"";2015-11-24 13:01:52.184969+00;t;t;0;99;218$
wow I am so lost even after reading tons of subject.. But contrary to those, I don't see any mistake here!
Your \COPY command generates a header line which is being read by the import command as data.
Either you do not generate the header line or tell the import command to not read the first line.
To avoid the header line generation omit the HEADER option.
To ignore the header line insert the HEADER option in the import command.
I have a CSV file from which I am trying to use Postgres COPY command in order to populate a table from that CSV file. One of the table columns NEXT_VISIT is of a date data_type. Some of the corresponding fields in the CSV file which are supposed to go into this date column have null values.
The Copy command am running is like so:
COPY "VISIT_STAGING_TABLE" from E'C:\\Users\\Sir Codealot\\Desktop\\rufijihdss-2007-2010\\rufijihdss\\VISIT_TEST.CSV' CSV HEADER
When I run this command I get the error:
ERROR: invalid input syntax for type date: ""
CONTEXT: COPY VISIT_STAGING_TABLE, line 2, column NEXT_VISIT: ""
********** Error **********
ERROR: invalid input syntax for type date: ""
SQL state: 22007
Context: COPY VISIT_STAGING_TABLE, line 2, column NEXT_VISIT: ""
How can I run the copy command and get Postgres to accept that some of the fields in the CSV file corresponding to NEXT_VISIT have values ""?
Add WITH NULL AS '' to your command (COPY expects NULLs to be represented as "\N" (backslash-N) by default).
COPY "VISIT_STAGING_TABLE" from E'C:\\Users\\Sir Codealot\\Desktop\\rufijihdss-2007-2010\\rufijihdss\\VISIT_TEST.CSV' WITH CSV HEADER NULL AS ''
More details here: postgresql COPY
I was having the exact same problem, and what solved it for me was to use the statement WITH NULL ''. It is important not to have a space between the apostrophes.
I originally used the statement WITH NULL ' ' and got the same error message you did (ERROR: syntax error at or near "WITH NULL").
But when I eliminated the space between the apostrophes it worked.