I tried the below query to import data
COPY table_name(id, name, create_uid, create_date, write_uid, write_date)
FROM '/path/filename.csv'
DELIMITER ',' CSV HEADER ENCODING 'UTF8' QUOTE '"' ESCAPE '''';
The name field data type is JSONB. I tried the below formats in my CSV file.
These 2 formats both produced the same error:
"{"en_US": "Test record"}" or {"en_US": "Test record"}
Error: ERROR: invalid input syntax for type json
DETAIL: Token "en_US" is invalid.
and this format produced a different error:
'{"en_US": "Test record"}'
ERROR: invalid input syntax for type json
DETAIL: Token "'" is invalid.
How can I fix the above errors?
Postgres uses " as escaping characters. Since importing from CSV file, postgres will escape the quotes.
Try something like,
"{""en_us"": ""Test record""}"
or
"{\"en_us\": \"Test record\"}"
Related
I found one strange problem. With greenplum gpload tool, I tried to import data from a single text file to greenplum db.
The content of file t1.out is:
\N|24234243
\N|\N
12342|\N
and gpload version is:
gpload version 5.3.0 build
commit:2155c5a8cf8bb7f13f49c6e248fd967a74fed591
and the table t1 is as follow:
test=# \d t1;
Table "public.t1"
Column | Type | Modifiers
--------+--------+-----------
id1 | bigint |
id2 | bigint |
when I use gpload with t1.yaml, I got following error:
2019-04-09 20:12:18|WARN|Please use following query to access the detailed error
2019-04-09 20:12:18|WARN|select * from p_read_error_log('ext_gpload_reusable_b7ef1344_5ac0_11e9_b6fc_fa163e2d09a1') where cmdtime > to_timestamp('1554811937.76')
and when I ran this sql(made small change only selecting two key fields) in postgresql, I got following errors:
errmsg | rawdata
invalid input syntax for integer: "\N", column id1 | \N|24234243
invalid input syntax for integer: "\N", column id1 | \N|\N
invalid input syntax for integer: "\N", column id2 | 12342|\N
It showed all the 3 lines were not imported with 'invalid input syntax for integer: "\N"'.
but I can use COPY command to import above 3 items into t1 successfully.
I tried several ways to find why, but failed. Part of my t1.yaml as follows:
- FORMAT: text
- DELIMITER: '|'
- ESCAPE: 'OFF'
- NULL_AS: '\N'
- ERROR_LIMIT: 100
BTW:https://gpdb.docs.pivotal.io/530/utility_guide/admin_utilities/gpload.html#topic1__cfnullas has shown that the default NULL_AS is \N, and gpload has found \N, and why it failed to mark the two field as NULL?
Any help is appreciated!
What version of GPDB are you using?
There was a known issue in v5 that was fixed in 5.6+
https://gpdb.docs.pivotal.io/560/relnotes/GPDB_561_README.html
29197 - gpload/ gpfdist
When running a gpload operation, the gpfdist utility did not recognize a \N as the NULL character when the gpload configuration file specified a null character null_as: '\N'. When processing the configuration file, the gpload utility incorrectly escaped the backslash () with another backslash.
This issue has been resolved. The gpload utility has been improved to properly handle a backslash when processing the null_as property.
I'm importing data from csv file into postgres db using pgadmin 4
Everything is ok but I get an issue when I try to import a file which contain some data like this
“‘t Zand, Vlotbrug”
“Dussen, `t Middeltje”
as you can see, the data contains
`
and
'
I also tried to import the file with utf-8 encoding but could not.
Anyone knows how to solve this issue?
Updated
Structure:
stop_id,stop_code,stop_name,stop_lat,stop_lon,location_type,parent_station,stop_timezone,wheelchair_boarding,platform_code,zone_id
Data:
stoparea:123953,,"De Zande, 'Koelucht'",52.5184475,5.956368,1,,,0,,
Error:
ERROR: unterminated CSV quoted field
CONTEXT: COPY stops, line 69400: "stoparea:123953,,"De Zande, 'Koelucht'",52.5184475,5.956368,1,,,0,,
stoparea:120536,,"Poortvliet, Zu..."
Updated 2
Command:
"/Applications/pgAdmin 4.app/Contents/SharedSupport/psql" --command "
"\copy transit.stops (stop_id, stop_code, stop_name, stop_lat,
stop_lon, location_type, parent_station, stop_timezone,
wheelchair_boarding, platform_code, zone_id) FROM
'/Users/tvtan/Desktop/gtfs-nl/stops.txt' DELIMITER ',' CSV HEADER
QUOTE '\"' ESCAPE '''';""
UI:
From the command line it looks like you have defined the ESCAPE character as a single quote. The single quote appears in your data but is not escaped.
The default ESCAPE character is the same as the QUOTE character.
More information here
I perform the same sql scripts on two server with different postgres versions. The first one has postgres 9.4.4 (this works fine) and the other one 9.5 (this throws an exception) installed.
UPDATE archived_invoice SET encrypted_xml
= encrypt(xml::bytea, 'MySuperSecretKey'::bytea, 'aes-ecb/pad:pkcs')
The exception:
Caused by: org.postgresql.util.PSQLException: ERROR: invalid input syntax for type bytea
SQL Status:22P02
Are there any differences between these two postgresql versions?
PostgreSQL will throw an ERROR: invalid input syntax for type bytea at you if the text you're casting into a bytea contains an invalid escape sequence:
# select '\i is not a valid escape sequence'::bytea;
ERROR: invalid input syntax for type bytea
LINE 1: select '\i is not a valid escape sequence'::bytea;
I'm guessing your "xml" column contains or may contain some backslashes. This is fine for XML and text columns, but presents an issue when casting to bytea.
You will need to escape the backslashes:
UPDATE archived_invoice SET encrypted_xml
= encrypt(
replace(xml, '\', '\\')::bytea,
'MySuperSecretKey'::bytea,
'aes-ecb/pad:pkcs'
)
Hi I did this command on my original DB :
\COPY (SELECT * FROM cms_title WHERE title = 'Migration-test') TO '/Users/JayCee/cms_title_dump.csv' WITH CSV HEADER DELIMITER ';' NULL AS '';
It worked fine.
But when I try to copy it to an other DB I have this :
\COPY cms_title FROM 'cms_title_dump.csv' DELIMITER ';' CSV;
ERROR: invalid input syntax for integer: "id"
CONTEXT: COPY cms_title, line 1, column id: "id"
I don't understand, here is what I have when I do a cat -e on my file :
id;language;title;page_title;menu_title;meta_description;slug;path;has_url_overwrite;redirect;creation_date;published;publisher_is_draft;publisher_state;page_id;publisher_public_id$
217;en;Migration-test;"";"";"";migration-test;migration-test;f;"";2015-11-24 13:01:52.184969+00;t;t;0;99;218$
wow I am so lost even after reading tons of subject.. But contrary to those, I don't see any mistake here!
Your \COPY command generates a header line which is being read by the import command as data.
Either you do not generate the header line or tell the import command to not read the first line.
To avoid the header line generation omit the HEADER option.
To ignore the header line insert the HEADER option in the import command.
I have a CSV file from which I am trying to use Postgres COPY command in order to populate a table from that CSV file. One of the table columns NEXT_VISIT is of a date data_type. Some of the corresponding fields in the CSV file which are supposed to go into this date column have null values.
The Copy command am running is like so:
COPY "VISIT_STAGING_TABLE" from E'C:\\Users\\Sir Codealot\\Desktop\\rufijihdss-2007-2010\\rufijihdss\\VISIT_TEST.CSV' CSV HEADER
When I run this command I get the error:
ERROR: invalid input syntax for type date: ""
CONTEXT: COPY VISIT_STAGING_TABLE, line 2, column NEXT_VISIT: ""
********** Error **********
ERROR: invalid input syntax for type date: ""
SQL state: 22007
Context: COPY VISIT_STAGING_TABLE, line 2, column NEXT_VISIT: ""
How can I run the copy command and get Postgres to accept that some of the fields in the CSV file corresponding to NEXT_VISIT have values ""?
Add WITH NULL AS '' to your command (COPY expects NULLs to be represented as "\N" (backslash-N) by default).
COPY "VISIT_STAGING_TABLE" from E'C:\\Users\\Sir Codealot\\Desktop\\rufijihdss-2007-2010\\rufijihdss\\VISIT_TEST.CSV' WITH CSV HEADER NULL AS ''
More details here: postgresql COPY
I was having the exact same problem, and what solved it for me was to use the statement WITH NULL ''. It is important not to have a space between the apostrophes.
I originally used the statement WITH NULL ' ' and got the same error message you did (ERROR: syntax error at or near "WITH NULL").
But when I eliminated the space between the apostrophes it worked.