Tried to insert Chinese character and failed. I used my MacOS to brew install the latest version 11.1 of psql, the locale was automatically set as zh_CN UTF8. Then when I tried to insert words like '店长', it showed 'invalid byte sequence for encoding "UTF8": 0xa9'
Can anyone help fix this T^T
Related
I am trying to upload a series of tables (.txt files) into a PostgreSQL database that runs on my Windows 10 desktop. I use psql upload the files. I have successfully uploaded a couple of tables but the largest one (5GB with over 20 million rows) is giving me trouble:
databasename=# \copy table1 FROM 'C:\Users\tablename.txt' DELIMITER ',' CSV HEADER;
ERROR: character with byte sequence 0x9d in encoding "WIN1252" has no equivalent in encoding "UTF8"
CONTEXT: COPY table1, line 581330
I found an answer here which suggested I check the client encoding...
databasename=# SHOW client_encoding;
client_encoding
-----------------
WIN1252
(1 row)
and then change it, which I tried:
databasename=# SET CLIENT_ENCODING TO 'utf8';
SET
I then try the same copy command again and get the following error:
ERROR: invalid byte sequence for encoding "UTF8": 0x92
CONTEXT: COPY table1, line 206051
I've read a little about 0x92 here. It sounds like there is a character in the file which cannot be encoded when I try and perform the \copy command.
Some background:
I was able to upload about 1 million rows into SQL Server 2019 (free version) using the SQL Server Import and Export Wizard. (I stopped the import because it was taking too long.) I was also able to view the file in R using read.csv. Not sure if any of this is helpful. Thank you all in advance.
I am trying to use the postgreSQL COPY function to insert a UTF 16 encoded csv into a table. However, when running the below query:
COPY temp
FROM 'C:\folder\some_file.csv'
WITH (
DELIMITER E'\t',
FORMAT csv,
HEADER);
I get the error below:
ERROR: invalid byte sequence for encoding "UTF8": 0xff
CONTEXT: COPY temp, line 1
SQL state: 22021
and when I run the same query, but adding the encoding settings Encoding 'UTF-16' or Encoding 'UTF 16' to the with block, I get the error below:
ERROR: argument to option "encoding" must be a valid encoding name
LINE 13: ENCODING 'UTF 16' );
^
SQL state: 22023
Character: 377
I've looked through the postgres documentation to try to find the correct encoding, but haven't managed to find anything. Is this because the copy function does support UTF 16 encoded files? I would have thought that this would almost certainly have been possible!
I'm running postgres 12, on windows 10 pro
Any help would be hugely appreciated!
No, you cannot do that.
UTF-16 is not in the list of supported encodings.
PostgreSQL will never support an encoding that is not an extension of ASCII.
You will have to convert the file to UTF-8.
I am working on PostgreSQL and getting below error during insert statement execution from batch script(command line).
ERROR: invalid byte sequence for encoding "UTF8": 0xeb 0x6e 0x74
I have checked client_encoding by show client_encoding command and it is showing UTF-8.
Also checked database properties by using command
select * from pg_database where datname='<mydbName>'
In Output:
datcollate = English_United States.1252
datctype = English_United States.1252
How to resolve this issue?
If the three bytes quoted by the error message are supposed to encode the string “ënt”, you can solve your problem by setting the correct client encoding, e.g.
SET client_encoding = WIN1252;
This happened because you have special characteres in the name of the columns.
Rename and delete special characters change the encoding for win-1252 and will work.
I get an error when trying to use copy utility to extract data from csv file with UCS-2 LE BOM encoding (as reported by notepad++).
COPY pub.calls (............ )
FROM 'c:\IMPORT\calls.csv'
WITH
DELIMITER ','
HEADER
CSV
ENCODING 'UCS2';
The error is something like this
SQL Error [22023]Error The argument of encoding parameter should be
acceptable encoding name.
UCS-2 gives the same error.
For the list of supported charsets:
https://www.postgresql.org/docs/current/static/multibyte.html
or in psql type \encoding and dbl tab for autocomplete:
postgres=# \encoding
BIG5 EUC_JP GB18030 ISO_8859_6 JOHAB LATIN1 LATIN3 LATIN6 LATIN9 SJIS UTF8 WIN1252 WIN1255 WIN1258
EUC_CN EUC_KR GBK ISO_8859_7 KOI8R LATIN10 LATIN4 LATIN7 MULE_INTERNAL SQL_ASCII WIN1250 WIN1253 WIN1256 WIN866
EUC_JIS_2004 EUC_TW ISO_8859_5 ISO_8859_8 KOI8U LATIN2 LATIN5 LATIN8 SHIFT_JIS_2004 UHC WIN1251 WIN1254 WIN1257 WIN874
I try to create a collation on a new database(firebird3.0.1 on windows. I follows the steps:
create database 'c:\tmp\ebizmis.fdb' user 'SYSDBA' password '123456' default character set utf8 collation unicode;
connect 'c:\tmp\ebizmis.fdb' user 'SYSDBA' password '123456';
create collation py for utf8 from UNICODE case insensitive 'LOCALE=zh';
in this step, prompting error:
Statement failed, SQLSTATE = HY000
unsuccessful metadata update
-CREATE COLLATION PY failed
-Invalid collation attributes
But on CentOS it is successfull on firebird 3.0.1.
When I change the uppercase LOCALE to lowercase:
create collation py for utf8 from UNICODE case insensitive 'locale=zh';
then it can be excuted on windows, but do not work for sorting the characters in chinese pinyin order.
I cannot build Firebird 3.0.1 under windows, but I can build 2.5.6. So I debug under 2.5.6. I found there is no bug in Firebird. The cause of the problem is that I haven't include zh locale in the icu. So I download the data from icu web site and replace the file in 3.0.1. It works!