I'm just migrating from Windows Phone to Android and I found that I cannot migrate my chat history of WhatsApp (at least not easily). Now I wanted to use the export-to-mail-feature WhatsApp offers. It works quite well, but (mainly) for old messages, some of the emoticons are not properly transmitted.
Looking into the UTF-8 bytes representation, it appears as if these emoticons that are not shown only consist of three bytes instead of the four normal ones.
Example:
08.09.2013 20:00:10: Name:
is exported instead of
08.09.2013 20:00:10: Name: 💩👽💀
Looking into the hexadecimal representation of the emoticons (only):
0 1 2 3 4 5 6 7 8 ... Dump
ee 81 9a ee 84 8c ee 84 9c î.šî„Œî„œ
instead of
0 1 2 3 4 5 6 7 8 9 a b ... Dump
f0 9f 92 a9 f0 9f 91 bd f0 9f 92 80 💩👽💀
So my question is: Is this just an error occurring during the WhatsApp export or is it some form of compressing to reduce the file size? If so, is there a decoding algorithm that converts the "compressed" version into the normal one?
WhatsApp seems to be using the Softbank encoding of emoji in Unicode's Private Use Area, which was used on iOS versions 2–4 before they were standardised in their own Unicode block. For example:
0xEE 0x84 0x8C is the UTF-8 encoding of U+E10C
U+E10C is a private use character that Softbank/Apple used to encode what would later be standardised as U+1F47D (👽).
There is no official list of these mappings, nor is an a algorithmic conversion possible (a mapping table must be built). However, you can easily find compilations of the Softbank/Unicode mappings to build a converter between them.
Related
After a long time, I come to review the contents of my HDD and see a weird file name.
I'm not sure what tool or program has changed it this way, but when I see the content of the file I could find its original name.
Anyway, I'm encountering a type of encoding and I want to find it. It's not complicated. Mostly for those who are familiar with unicode and utf8. Now I map them and you guess what has happened.
In the following, I give a table which maps the characters. In the second column, there's utf8 form and in the first column there's its equivalent character which is converted.
I need to know what happened and how is it converted to convert it back to utf8. that is, what I have is in the first column, and what I need to get is in the second column:
Hide Copy Code
638 2020 646
639 AF 6AF
637 A7 627
637 B1 631
637 B3 633
637 6BE 62A
20 20
638 67E 641
63A 152 6CC
For more description, consider the first row, utf8 form is 46 06 (type bytes) or 0x0646. The file name for this character is converted into two wide-characters, 0x0638 0x2020.
I found the solution myself.
In Notepad++:
Select "Encode in ANSI" from Encoding menu.
Paste the corrupted text.
Select "Encode in UTF-8" from Encoding menu.
That's it. The correct text will be displayed.
If so, how can I do the same with Perl?
I am not sure I've got my nomenclature right, so please correct me :)
I've received a text file representing a Pāli dictionary: a list of words separated by newline \n (0x0a) characters. Supposedly, some of the special letters are encoded using UTF-8, but I doubt that.
Loading this text file into any of my editors (vim, Notepad, TextEdit, ..) shows quite scrambled text, for example
mhiti
A closer look at the actual bytes then reveal the following (using hexdump -C)
0a 0a 1e 6d 68 69 74 69 0a 0a ...mhiti..
which seems to me the Unicode code point U+1E6D ("ṭ" or LATIN SMALL LETTER T WITH DOT BELOW). That particular letter has UTF-8 encoding e1 b9 ad.
My question: is there a tool which helps me convert this particular file into actual UTF-8 encoding? I tried iconv but without success; I looked briefly into a Python script but would think there's an easier way to get this done. It seems that this is a useful link for this problem, but isn't there a tool that can get this done? Am I missing something?
EDIT: Just to make things a little bit more entertaining, there seem to be actual UTF-8 encoded characters scattered throughout as well. For example, the word "ākiñcaññāyatana" has the following sequence of bytes
01 01 6b 69 c3 b1 63 61 c3 b1 c3 b1 01 01 79 61 74 61 6e 61
ā k i ñ c a ñ ñ ā y a t a n a
where the "ā" is encoded by its Unicode code point U-0101, and the "ñ" is encoded by the UTF-8 sequence \xc3b1 which has Unicode code point U-00F1.
EDIT: Here's one that I can't quite figure out what it's supposed to be:
01 1e 37 01 01 76 61 6b 61
? ā v a k a
I can only guess, but that too doesn't make sense. The Unicode code point U+011e is a "Ğ" (UTF-8 \xc49e) but that's not a Pāli character AFAIK; then a "7" follows which doesn't make sense in a word. Then the Unicode code point U+1E37 is a "ḷ" (UTF-8 \xe1b8b7) which is a valid Pāli character. But that would leave the first byte \x01 by itself. If I had to guess I would think this is the name "Jīvaka" but that would not match the bytes. LATER: According to the author, this is "Āḷāvaka" — so assuming the heuristics of character encoding from above, again a \x00 is missing. Adding it back in
01 00 1e 37 01 01 76 61 6b 61
Ā ḷ ā v a k a
Are there "compressions" that remove \x00 bytes from UTF-16 encoded Unicode files?
I'm assuming in this context that "ṭhiti" makes sense as the contents of that file.
From your description, it looks like that file encodes characters < U+0080 as a single byte and characters > U+0100 as two-byte big-endian. That's not decodable, in general; two linefeeds (U+000A, U+000A) would have the same encoding as GURMUKHI LETTER UU (U+0A0A).
There's no invocation of iconv that'll decode it for you; you'll either need to take the heuristics you know, either based on character ranges or ordering in the file, to write a custom decoder (or ask for another copy in a standard encoding).
I think in the end this was my own fault, somehow. Browsing to this file showed a very mangled and broken version of the original UTF-16 encoded file; the "Save as" menu from the browser then saved that broken file which created the initial question for this thread.
It seems that a web browser tries to display that UTF-16 encoded file, removes non-printable characters like \x00 and converts some others to UTF-8, thus completely mangling the original file.
Using wget to fetch the file fixed the problem, and I could convert it nicely into UTF-8 and use it further.
It looks like postgres upper/lower function does not handle select characters in Turkish character set.
select upper('Aaı'), lower('Aaİ') from mytable;
returns :
AAı, aaİ
instead of :
AAI, aai
Note that normal english characters are converted correctly, but not the Turkish I (lower or upper)
Postgres version: 9.2 32 bit
Database encoding (Same result in any of these): UTF-8, WIN1254, C
Client encoding:
UTF-8, WIN1254, C
OS: Windows 7 enterprise edition 64bit
SQL functions lower and upper return the following same bytes for ı and İ on UTF-8 encoded database
\xc4b1
\xc4b0
And the following on WIN1254 (Turkish) encoded database
\xfd
\xdd
I hope my investigation is wrong, and there is something I missed.
Your problem is 100% Windows. (Or rather Microsoft Visual Studio, which PostgreSQL was built with, to be more precise.)
For the record, SQL UPPER ends up calling Windows' LCMapStringW (via towupper via str_toupper) with almost all the right parameters (locale 1055 Turkish for a UTF-8-encoded, Turkish_Turkey database),
but
the Visual Studio Runtime (towupper) does not set the LCMAP_LINGUISTIC_CASING bit in LCMapStringW's dwMapFlags. (I can confirm that setting it does the trick.) This is not considered a bug at Microsoft; it is by design, and will probably not ever be "fixed" (oh the joys of legacy.)
You have three ways out of this:
implement #Sorrow's wrapper solution (or write your own native function replacement (DLL).)
run your PostgreSQL instance on e.g. Ubuntu which exhibits the right behaviour for Turkic locales (#Sorrow confirmed that it works for him); this is probably the simplest and cleanest way out.
drop in a patched 32-bit MSVCR100.DLL in your PostgreSQL bin directory (but although UPPER and LOWER would work, other things such as collation may continue to fail -- again, at the Windows level. YMMV.)
For completeness (and nostalgic fun) ONLY, here is the procedure to patch a Windows system (but remember, unless you'll be managing this PostgreSQL instance from cradle to grave you may cause a lot of grief to your successor(s); whenever deploying a new test or backup system from scratch you or your successor(s) would have to remember to apply the patch again -- and if let's say you one day upgrade to PostgreSQL 10, which say uses MSVCR120.DLL instead of MSVCR100.DLL, then you'll have to try your luck with patching the new DLL, too.) On a test system
use HxD to open C:\WINDOWS\SYSTEM32\MSVCR100.DLL
save the DLL right away with the same name under you PostgreSQL bin directory (do not attempt to copy the file using Explorer or the command line, they might copy the 64bit version)
with the file still open in HxD, go to Search > Replace, pick Datatype: Hexvalues, then
search for...... 4E 14 33 DB 3B CB 0F 84 41 12 00 00 B8 00 01 00 00
replace with... 4E 14 33 DB 3B CB 0F 84 41 12 00 00 B8 00 01 00 01
...then once more...
search for...... FC 51 6A 01 8D 4D 08 51 68 00 02 00 00 50 E8 E2
replace with... FC 51 6A 01 8D 4D 08 51 68 00 02 00 01 50 E8 E2
...and re-save under the PostgreSQL bin directory, then restart PostgreSQL and re-run your query.
if your query still does not work (make sure your database is UTF-8 encoded with Turkish_Turkey for both LC_CTYPE and LC_COLLATE) open postgres.exe in 32-bit Dependency Walker and make sure it indicates it loads MSVCR100.DLL from the PostgreSQL bin directory.
if all functions well copy the patched DLL to the production PostgreSQL bin directory and restart.
BUT REMEMBER, the moment you move the data off the Ubuntu system or off the patched Windows system to an unpatched Windows system you will have the problem again, and you may be unable to import this data back on Ubuntu if the Windows instance introduced duplicates in a citext field or in a UPPER/LOWER-based function index.
It seems to me that your problem is related to Windows. This is how it looks on Ubuntu (Postgres 8.4.14), database encoding UTF-8:
test=# select upper('Aaı'), lower('Aaİ');
upper | lower
-------+-------
AAI | aai
(1 row)
My recommendation would be - if you have to use Windows - to write a stored procedure that will do the conversion for you. Use built-in replace: replace('abcdefabcdef', 'cd', 'XX') returns abXXefabXXef. There might be a more optimal solution, I do not claim that this approach is the correct one.
This is indeed bug in PostgreSQL (still not fixed, even in current git tree).
Proof: https://github.com/postgres/postgres/blob/master/src/port/pgstrcasecmp.c
PostgreSQL developers even mention specifically those Turkish characters there:
SQL99 specifies Unicode-aware case normalization, which we don't yet
have the infrastructure for. Instead we use tolower() to provide a
locale-aware translation.
However, there are some locales where this is not right either (eg, Turkish may do strange things with 'i' and 'I').
Our current compromise is to use tolower() for characters with
the high bit set, and use an ASCII-only downcasing for 7-bit
characters.
pg_upper() implemented in this file is extremely simplistic (as its companion pg_tolower()):
unsigned char
pg_toupper(unsigned char ch)
{
if (ch >= 'a' && ch <= 'z')
ch += 'A' - 'a';
else if (IS_HIGHBIT_SET(ch) && islower(ch))
ch = toupper(ch);
return ch;
}
As you can see, this code does not treat its parameter as Unicode code point, and cannot possibly work 100% correctly, unless currently selected locale happens to be the one that we care for (like Turkish non-unicode locale) and OS-provided non-unicode toupper() is working correctly.
This is really sad, I just hope that this will be solved in upcoming PostgreSQL releases...
The source of the problem explained above. It seems the problem only occurs with the conversion of 'I' to 'ı' and 'i' to 'İ'. As a workaround just replace those characters directly as below before calling lower or upper functions:
SELECT lower(replace('IİĞ', 'I', 'ı')) -> ıiğ
SELECT upper(replace('ıiğ', 'i', 'İ')) -> IİĞ
In our application we use VerQueryValue() API call to fetch version info such as ProductName etc. For some applications running on a machine in Traditional Chinese (code page 950), the ProductName which has Unicode sequences that span multiple code pages, some characters are not translated properly. For instance,in the sequence below,
51 00 51 00 6F 8F F6 4E A1 7B 06 74
Some characters are returned as invalid Unicode 0x003f (question mark)
In the above sequence, the Unicode '8F 6F' is not picked up & converted properly by the WinAPI call and is just filled with the invalid Unicode '00 3F' - since '8F 6F' is present in codepage 936 only (ie., Simplified Chinese)
The .exe has just one translation table - as '\StringFileInfo\080404B0' - which refers to a language ID of '804' for Traditional Chinese only
How should one handle such cases - where the ProductName refers to Unicode from both 936 and 950 even though the translation table has one entry only ? Is there any other API call to use ?
Also, if I were to right-click on the exe and view 'details' tab, it shows the Productname correctly ! So it appears Microsoft uses a different API call or somehow
handle this correctly. I need to know how it so done.
Thanks in advance,
Venkat
It looks somewhat waierd to have contents compatible with codepage1 only in a block marked as codepage2. This is the source of your problem.
The best way to handle multi-codepages issues is obviously to turn your app to a Unicode-aware application. There will be no conversion to any codepages anymore, which will make everyone happy.
The LANGID (0804) is only an indication about the language of the contents in the block. If a version info has several blocks, you may program your app to lookup the block in the language of your user.
When you call VerQueryValue() in an ANSI application, this LANGID is not taken into account when converting the Unicode contents to ANSI: You're ANSI, so Windows assume you only understand the machine's default ANSI codepage.
Note about display in console
Beware of the console! It's an old creature that is not totally Unicode-aware. It is based on codepages. Therefore, you should expect display problems which can't be addressed. Even worse: It uses its own codepage (called OEM codepage) which may be different that the usual ANSI codepage (Although for East Asian languages, OEM codepage = ANSI codepage).
HTH.
I have a problem … a very peculiar one could you please guide.
Original message: Kevätsunnuntaisin lentää
The flow of data is HttpConnector -> WSDLConnector -> to the underlying system
The following is the encoding of the first 7 characters
4b 65 76 c3 a4 74 73 75 – In Http Connector – the request XML has UTF-8 encoding
4b 65 76 a3 74 73 75 – in WSDL Connector -
InputSource inputSource = new InputSource(myInputStream);
inputSource.setEncoding("UTF-8");
parser.parse(inputSource);
The original string gets converted to Kev£tsunnuntaisin lent££.Also, there is a loss of a byte.
Could you please guide me where I am going wrong? What must I do to avoid this character conversion?
Thanks for your help!!!
This is very simple: The data in myInputStream is not encoded as UTF-8, hence the decoding fails.
My guess is that you save the output of the HTML connector as a string and then use that as the input for the WSDL connector. In the string, the data is unicode, not UTF-8. Use String.getBytes('UTF-8') to get an array of bytes with the correct encoding.
As for all encoding issues: Always tell the computer with which encoding it should work instead of hoping that it will guess correctly. Bytes have no encoding and the computer is not telepathic :) And I hope it never will be ...