Currently I use:
The utf8mb4 database character set.
The utf8mb4_unicode_520_ci database collation.
I understand that utf8mb4 supports up to four bytes per character. I also understand that Unicode is a standard that continues to get updates. In the past I thought utf8 was sufficient until I had some test data get corrupted, lesson learned. However I'm having difficulty understanding the upgrade path for both the character set and collations.
The utf8mb4_unicode_520_ci database collation is based off of Unicode Collation Algorithm version 5.2.0. If you navigate to the parent directory you'll see up to version 14.0 listed at the time of typing this. Now those are the Unicode standards, then there is the supported MariaDB character sets and collations.
Offhand I'm not sure when the need to go from four bytes per character gets superseded to go to eight bytes per character or even 16 so it's not as simple a measure of just updating the database collation. Additionally I'm not seeing anything that seems to be newer than version 5.2.0 on MariaDB's documentation.
So in short my three highly related questions are:
Are the newer collations such as version 14 still fully compatible with four byte characters or have they exhausted all combinations and now require up to eight or 16 bytes per character?
What is the latest database collation that MariaDB supports in regards to Unicode versions?
In regards to the second question, a newer version than 5.2.0 is supported by MariaDB then is utf8mb4 still sufficient for a character set or not?
I am not bound to or care about MySQL compatibility.
You can inspect the collations currently supported by your MariaDB instance:
SELECT * FROM INFORMATION_SCHEMA.COLLATIONS
WHERE CHARACTER_SET_NAME = 'utf8mb4';
As far as I know, MariaDB does not support any UTF-8 collation version more current than utf8_unicode_520ci. If you try to use the '900' version, for example importing metadata from MySQL to MariaDB, you get errors.
There is no such thing as an 8-byte or 16-byte UTF-8 encoding. UTF-8 is an encoding that uses between 1 and 4 bytes per character, no more than that.
MariaDB also supports utf16 and utf32, but neither of these uses more than 4 bytes per character. Utf16 is variable-length, using one or two 16-bit code units per character. Utf32 is fixed-width, always using 32-bits (4 bytes) per character.
Related
Looking for a reference that discusses PostgreSQL's support for the NATIONAL CHARACTER set of data types. e.g. this query runs without error:
select cast('foo' as national character varying(10))
yet the docs don't seem to discuss that type Postgres character data types
Does Postgres implement these differently from the CHARACTER data types? That is, how does the NATIONAL keyword affect how data is stored or represented?
Can someone share a link or two to any references I can't seem to find? (other than some mailing list correspondence from a while back)
If you request a national character varying in PostgresSQL, you'll get a regular character varying.
PostgreSQL uses the same encoding for normal and national characters.
“National character” is a leftover from the bad old days when people still used single-byte encodings like LATIN-1 and needed a different encoding for characters that didn't fit.
PostgreSQL has always supported UNICODE encodings, so this is not an issue. Just make sure that you don't specify an encoding other than the default UTF8.
NATIONAL CHARACTER has no real meaning in the SQL:92 standard (section 4.2.1), saying only that it means “a particular implementation-defined character repertoire”. If you are surprised, don’t be. There are many screwy aspects to the SQL standard.
As for text handling in Postgres, you would likely be interested in learning about:
character encoding
Unicode
UTF-8
collations
support for ICU in Postgres 10 and later.
See:
More robust collations with ICU support in PostgreSQL 10 by Peter Eisentraut, post, 2017-05.
Collations: Introduction, Features, Problems by Peter Eisentraut, video, 2019-07-12.
Unicode collation algorithm ( UCA )
ICU User Guide – Locale
List of locales with 209 languages, 501 regions & variants, as defined in ICU
I have a chat server that stores messages in MongoDB...emoticons (emoji specifically) are giving me grief.
Apparently emoticons/emoji are UTF8-mb4 encoded...can MongoDB store data in this encoding? If it can't stored UTF8-mb4 directly is there some kind of workaround?
MongoDB supports UTF8 which supports all characters (variable byte length).
The original implementation "utf8" by MySQL was only for up to 3 Bytes (like many other implementations). The MySQL implementation called 'utf8-mb4' is actually up to 4 Bytes long (like the official RFC recommends it).
So utf8-mb4 in MySQL is the same as UTF8 in mongoDB.
What I saw in my Tests:
Robomongo does not support chinese 4 Byte characters while for example MongoVUE has no problems.
Interesting article about the utf8 max byte size: https://stijndewitt.wordpress.com/2014/08/09/max-bytes-in-a-utf-8-char/
(Answer updated)
Does anyone know of a simple chart or list that would show all acceptable varchar characters? I cannot seem to find this in my googling.
What codepage? Collation? Varchar stores characters assuming a specific codepage. Only the lower 127 characters (the ASCII subset) is standard. Higher characters vary by codepage.
The default codepage used matches the collation of the column, whose defaults are inherited from the table,database,server. All of the defaults can be overriden.
In sort, there IS no "simple chart". You'll have to check the character chart for the specific codepage, eg. using the "Character Map" utility in Windows.
It's far, far better to use Unicode and nvarchar when storing to the database. If you store text data from the wrong codepage you can easily end up with mangled and unrecoverable data. The only way to ensure the correct codepage is used, is to enforce it all the way from the client (ie the desktop app) to the application server, down to the database.
Even if your client/application server uses Unicode, a difference in the locale between the server and the database can result in faulty codepage conversions and mangled data.
On the other hand, when you use Unicode no conversions are needed or made.
using jdbc (jt400) to insert data into an as400 table.
db table code page is 424. Host Code Page 424
the ebcdic 424 code page does not support many of the characters that may come from the client.
for example the sign → (Ascii 26 Hex 1A)
the result is an incorrect translation.
is there any built-in way in the toolbox to remove any of the unsupported characters?
You could try to create a logical file over your ccsid424 physical file with a different codepage. It is possible on the as/400 to create logical files with different codepages for individual columns, by adding the keyword CCSID(<num>). You can even set it to an unicode charset, e.g. CCSID(1200) for UTF-16. Of course your physical file will still only be able to store chars that are in the 424 codepage, and those will be replaced by some invalid character char, but the translation might be better that way.
There is no way to store chars, that are not in codepage 424 in a column with that codepage directly (the only way I can think of is encoding them somehow with multiple chars, but that is most likely not what you want to do, since it will bring more problems than it "solves").
If you have control over that system, and it is possible to do some bigger changes, you could do it the other way around: create a new unicode version of that physical file with a different name (I'd propose CCSID(1200), that's as close as you get to UTF-16 on as/400 afaik, and UTF-8 is not supported by all parts of the system in my experience. IBM does recommend 1200 for unicode). Than transfer all data from your old file to the new one, delete the old one (before that, backup it!), and than create a logical file over the new physical, with the name of the old physical file. In that logical file change all ccsid-bearing columns from 1200 to 424. That way, existing programs can still work on the data. Of course there will be invalid chars in the logical file now, once you insert data that is not in a subset of ccsid 424; so you will most likely have to take a look at all programs that use the new logical file.
I would like to have a collation which orders the UTF-8 encoding of 0x1234 below of 0x1235 regardless of the character mapping in the Unicode standard. MySQL uses utf8_bin for this. MSSQL apparently http://msdn.microsoft.com/en-us/library/ms143350.aspx have BIN and BIN2 collations. While finding these were easy, I can't even find a list of collations PostgreSQL supports much less answer to this specific question.
The C locale will do. UTF-8 is designed so that byte ordering is also codepoint ordering. This is not trivial but consider how UTF-8 works:
Number range Byte 1 Byte 2 Byte 3
0000-007F 0xxxxxxx
0080-07FF 110xxxxx 10xxxxxx
0800-FFFF 1110xxxx 10xxxxxx 10xxxxxx
When sorting binary data aka C locale, the first non-equal byte will determine the ordering. What we neeed to see that if two numbers encoded into UTF-8 differ then the first non-equal byte will be lower for the lower value. If the numbers are in different ranges then the first byte will indeed be lower for the lower number. Within the same range, the order is determined by literally the same bits as without encoding.
Sort order of text depends on lc_collate (not on the system locale!). The system locale only serves as a default when creating the db cluster if you don't provide another locale.
The behaviour you are expecting only works with locale C. Read all about it in the fine manual:
The C and POSIX collations both specify "traditional C" behavior, in
which only the ASCII letters "A" through "Z" are treated as letters,
and sorting is done strictly by character code byte values.
Emphasis mine. PostgreSQL 9.1 has a couple of new features for collation. Might be exactly what you are looking for.
Postgres uses the collation defined by the system locale on cluster creation.
You might try to ORDER BY encode(column,'hex')