invalid input syntax for integer: "\N" when gpload a text file - copy

I found one strange problem. With greenplum gpload tool, I tried to import data from a single text file to greenplum db.
The content of file t1.out is:
\N|24234243
\N|\N
12342|\N
and gpload version is:
gpload version 5.3.0 build
commit:2155c5a8cf8bb7f13f49c6e248fd967a74fed591
and the table t1 is as follow:
test=# \d t1;
Table "public.t1"
Column | Type | Modifiers
--------+--------+-----------
id1 | bigint |
id2 | bigint |
when I use gpload with t1.yaml, I got following error:
2019-04-09 20:12:18|WARN|Please use following query to access the detailed error
2019-04-09 20:12:18|WARN|select * from p_read_error_log('ext_gpload_reusable_b7ef1344_5ac0_11e9_b6fc_fa163e2d09a1') where cmdtime > to_timestamp('1554811937.76')
and when I ran this sql(made small change only selecting two key fields) in postgresql, I got following errors:
errmsg | rawdata
invalid input syntax for integer: "\N", column id1 | \N|24234243
invalid input syntax for integer: "\N", column id1 | \N|\N
invalid input syntax for integer: "\N", column id2 | 12342|\N
It showed all the 3 lines were not imported with 'invalid input syntax for integer: "\N"'.
but I can use COPY command to import above 3 items into t1 successfully.
I tried several ways to find why, but failed. Part of my t1.yaml as follows:
- FORMAT: text
- DELIMITER: '|'
- ESCAPE: 'OFF'
- NULL_AS: '\N'
- ERROR_LIMIT: 100
BTW:https://gpdb.docs.pivotal.io/530/utility_guide/admin_utilities/gpload.html#topic1__cfnullas has shown that the default NULL_AS is \N, and gpload has found \N, and why it failed to mark the two field as NULL?
Any help is appreciated!

What version of GPDB are you using?
There was a known issue in v5 that was fixed in 5.6+
https://gpdb.docs.pivotal.io/560/relnotes/GPDB_561_README.html
29197 - gpload/ gpfdist
When running a gpload operation, the gpfdist utility did not recognize a \N as the NULL character when the gpload configuration file specified a null character null_as: '\N'. When processing the configuration file, the gpload utility incorrectly escaped the backslash () with another backslash.
This issue has been resolved. The gpload utility has been improved to properly handle a backslash when processing the null_as property.

Related

Can't import a CSV file into PostgreSQL

Summary
I failed to import CSV files into a table on PostgreSQL.
Even though it says that the import was successfully completed, there's no rows created.
How did this happen, and how can I fix this? Thank you.
Details
1. The CSV file I (failed to) imported, is like this 1. CSV file imported
| number | ticket | category | question | answer | url | note |
|--------|------------|-----------|--------------------------|-----------------------|----------------|----------|
| 1 | #0000000 | Temp>123 | *confirming* | Would you...? | https:///....a | - |
| 2 | #1234567 | AAA / BBB | "a" vs "b" | If A, "a". If B, "b". | https:///....b | #0000000 |
| 3 | #1234567-2 | AAA>abc | Can we do sth using "a"? | Yes, blah blah blah. | https:///....b | - |
And this is the table on PostgreSQL
numberr : numeric
ticketr : char
category : char[]
question : char
answer : char
url : char
note : char
2.\ The message after the import
Even though it says that the import was "successfully completed"
When I hit “More details” of the import pop up (3. Message - Completed)
--command " "\\copy public.test (\"number\", ticket, category, question, answer, url, note) FROM '/Users/alice/Desktop/test.csv' DELIMITER ',' CSV HEADER ENCODING 'UTF8' QUOTE '\"' ESCAPE '''';""
3. The message when I made sure that the file was actually imported
When I click "Count Rows", it says "Table rows counted: 0"
I tried the following script in Query Tool of the table, and it shows no rows created
SELECT * FROM (table name)
For references
Created Postgres Log, but only the header is created.
Screenshot
4. No row created / 1. CSV file imported / 2. Import Preference / 3. Message - Completed / 5. postgres_log
After changing the name of a column from "number" to "consecutive", the error message showed up in Query Tool (not in Import/Export)
Tried Query Tool instead of Import/Emport
--> the situation didn’t change
Changed the first column name from “number” to “constructive” in both csv and psql table
--> the situation didn’t change
Tried Query Tool
copy public.test (consecutive, ticket, category, question, answer, url, note) FROM '/Users/alice/Desktop/test5.csv' DELIMITER ',' CSV HEADER ENCODING 'UTF8' QUOTE '"' ESCAPE '''';""
--> the situation didn’t change
Tried Query Tool
copy public.test (consecutive, ticket, category, question, answer, url, note) FROM '/Users/alice/Desktop/test5.csv' DELIMITER ',' CSV HEADER ENCODING 'UTF8'
—>got error message
ERROR: could not open file "/Users/alice/Desktop/test5.csv" for reading: Permission denied
HINT: COPY FROM instructs the PostgreSQL server process to read a file. You may want a client-side facility such as psql's \copy.
SQL state: 42501
check columns settings in tab at 2. Import Preference image
right from options tab
there you should set columns order as in your file
also check more details at 3. Message - Cpmpleted
This is a file permission issue.
Open a shell terminal, go to the directory where the data file is stored, and run chmod +rx * and retry loading the data file into your DB.

Importing csv file into postgres db using pgadmin with special characters

I'm importing data from csv file into postgres db using pgadmin 4
Everything is ok but I get an issue when I try to import a file which contain some data like this
“‘t Zand, Vlotbrug”
“Dussen, `t Middeltje”
as you can see, the data contains
`
and
'
I also tried to import the file with utf-8 encoding but could not.
Anyone knows how to solve this issue?
Updated
Structure:
stop_id,stop_code,stop_name,stop_lat,stop_lon,location_type,parent_station,stop_timezone,wheelchair_boarding,platform_code,zone_id
Data:
stoparea:123953,,"De Zande, 'Koelucht'",52.5184475,5.956368,1,,,0,,
Error:
ERROR: unterminated CSV quoted field
CONTEXT: COPY stops, line 69400: "stoparea:123953,,"De Zande, 'Koelucht'",52.5184475,5.956368,1,,,0,,
stoparea:120536,,"Poortvliet, Zu..."
Updated 2
Command:
"/Applications/pgAdmin 4.app/Contents/SharedSupport/psql" --command "
"\copy transit.stops (stop_id, stop_code, stop_name, stop_lat,
stop_lon, location_type, parent_station, stop_timezone,
wheelchair_boarding, platform_code, zone_id) FROM
'/Users/tvtan/Desktop/gtfs-nl/stops.txt' DELIMITER ',' CSV HEADER
QUOTE '\"' ESCAPE '''';""
UI:
From the command line it looks like you have defined the ESCAPE character as a single quote. The single quote appears in your data but is not escaped.
The default ESCAPE character is the same as the QUOTE character.
More information here

Postgresql 9.5 Syntax error

I perform the same sql scripts on two server with different postgres versions. The first one has postgres 9.4.4 (this works fine) and the other one 9.5 (this throws an exception) installed.
UPDATE archived_invoice SET encrypted_xml
= encrypt(xml::bytea, 'MySuperSecretKey'::bytea, 'aes-ecb/pad:pkcs')
The exception:
Caused by: org.postgresql.util.PSQLException: ERROR: invalid input syntax for type bytea
SQL Status:22P02
Are there any differences between these two postgresql versions?
PostgreSQL will throw an ERROR: invalid input syntax for type bytea at you if the text you're casting into a bytea contains an invalid escape sequence:
# select '\i is not a valid escape sequence'::bytea;
ERROR: invalid input syntax for type bytea
LINE 1: select '\i is not a valid escape sequence'::bytea;
I'm guessing your "xml" column contains or may contain some backslashes. This is fine for XML and text columns, but presents an issue when casting to bytea.
You will need to escape the backslashes:
UPDATE archived_invoice SET encrypted_xml
= encrypt(
replace(xml, '\', '\\')::bytea,
'MySuperSecretKey'::bytea,
'aes-ecb/pad:pkcs'
)

postgresql Importing csv integer error but the id number is in the file

Hi I did this command on my original DB :
\COPY (SELECT * FROM cms_title WHERE title = 'Migration-test') TO '/Users/JayCee/cms_title_dump.csv' WITH CSV HEADER DELIMITER ';' NULL AS '';
It worked fine.
But when I try to copy it to an other DB I have this :
\COPY cms_title FROM 'cms_title_dump.csv' DELIMITER ';' CSV;
ERROR: invalid input syntax for integer: "id"
CONTEXT: COPY cms_title, line 1, column id: "id"
I don't understand, here is what I have when I do a cat -e on my file :
id;language;title;page_title;menu_title;meta_description;slug;path;has_url_overwrite;redirect;creation_date;published;publisher_is_draft;publisher_state;page_id;publisher_public_id$
217;en;Migration-test;"";"";"";migration-test;migration-test;f;"";2015-11-24 13:01:52.184969+00;t;t;0;99;218$
wow I am so lost even after reading tons of subject.. But contrary to those, I don't see any mistake here!
Your \COPY command generates a header line which is being read by the import command as data.
Either you do not generate the header line or tell the import command to not read the first line.
To avoid the header line generation omit the HEADER option.
To ignore the header line insert the HEADER option in the import command.

Error on using copy Command in Postgres (ERROR: invalid input syntax for type date: "")

I have a CSV file from which I am trying to use Postgres COPY command in order to populate a table from that CSV file. One of the table columns NEXT_VISIT is of a date data_type. Some of the corresponding fields in the CSV file which are supposed to go into this date column have null values.
The Copy command am running is like so:
COPY "VISIT_STAGING_TABLE" from E'C:\\Users\\Sir Codealot\\Desktop\\rufijihdss-2007-2010\\rufijihdss\\VISIT_TEST.CSV' CSV HEADER
When I run this command I get the error:
ERROR: invalid input syntax for type date: ""
CONTEXT: COPY VISIT_STAGING_TABLE, line 2, column NEXT_VISIT: ""
********** Error **********
ERROR: invalid input syntax for type date: ""
SQL state: 22007
Context: COPY VISIT_STAGING_TABLE, line 2, column NEXT_VISIT: ""
How can I run the copy command and get Postgres to accept that some of the fields in the CSV file corresponding to NEXT_VISIT have values ""?
Add WITH NULL AS '' to your command (COPY expects NULLs to be represented as "\N" (backslash-N) by default).
COPY "VISIT_STAGING_TABLE" from E'C:\\Users\\Sir Codealot\\Desktop\\rufijihdss-2007-2010\\rufijihdss\\VISIT_TEST.CSV' WITH CSV HEADER NULL AS ''
More details here: postgresql COPY
I was having the exact same problem, and what solved it for me was to use the statement WITH NULL ''. It is important not to have a space between the apostrophes.
I originally used the statement WITH NULL ' ' and got the same error message you did (ERROR: syntax error at or near "WITH NULL").
But when I eliminated the space between the apostrophes it worked.