Why do some characters become ? and others become ☐ (␇) when encoding into a code page? - encoding

Question

Short version
What's the reasoning behind the mapper sometimes using ? and other times using ☐?
Unicode: €‚„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ
In 850: ?'".┼╬^%S<OZ''""☐--~Ts>ozY
^\_____________/^\_______/
| | | |
| best fit | best fit
| |
replacement replacement
CS Fiddle
Long Version
I was encoding some text to code page 850, and while a lot of characters that users use exist perfectly in the 850 code page, there are some that don't match exactly. Instead the mapper (e.g. .NET System.Text.Encoding, or Winapi WideStringToMultiByte) provides a best fit:
| Character | In code-page 850 |
|-----------|------------------|
| | |
| | |
| ‚ U+201A | ' |
| | |
| „ U+201E | " |
| … U+2026 | . |
| † U+2020 | ┼ |
| ‡ U+2021 | ╬ |
| ˆ U+02C6 | ^ |
| ‰ U+2030 | % |
| Š U+0160 | S |
| ‹ U+2039 | < |
| ΠU+0152 | O |
| | |
| Ž U+017D | Z |
| | |
| | |
| ‘ U+2018 | ' |
| ’ U+2019 | ' |
| “ U+201C | " |
| ” U+201D | " |
| | |
| – U+2013 | - |
| — U+2014 | - |
| ˜ U+02DC | ~ |
| ™ U+2122 | T |
| š U+0161 | s |
| › U+203A | > |
| œ U+0153 | o |
| | |
| ž U+017E | z |
| Ÿ U+0178 | Y |
These best fits are right, good, appropriate, wanted, and entirely reasonable:.
But some characters do not map:
| Character | In code-page 850 |
|-----------|------------------|
| € U+20AC | ? 0x37 | literally 0x37 (i.e. U+003F Question Mark)
| • U+2022 | ☐ 0x07 | literally 0x07 (i.e. U+0007 BELL)
What's the deal?
Why is it sometimes a question mark, and other times a ␇?
Note: This lack of mapping isn't terribly important to me. If the federal government doesn't support a reasonable encoding, then they'll take the garbage i give them. So i'm fine with it.
A problem comes later when i try to call MultiByteToWideChar to reverse the mapping, and the function fails due to invalid characters. And while i can try to figure out the issue with reverse encoding back into characters later; i'm curious what the encoding mapper is trying to tell me.
Bonus fun
The careful observer will understand why i chose the characters i did, in the order i did, and why there are gaps. I didn't want to mention it so as to confuse readers of the question.

Answer 1

The answer is both subtle, and obvious.
When performing a mapping, the encoder tries to perform a best-fit. And so while a lot of the characters don't exist in the target code-page; they can be approximated well enough.
Some characters don't have any equivalent, nor any best-fit mapping, and so are simply replaced with ?.
U+0037 QUESTION MARK
So the text:
The price of petrol is € 1.56 in Germany.
Will unfortunately become:
The price of petrol is ? 1.56 in Germany.
The question mark means that the character has no equivalent and was just lost.
The other character is more subtle
In ASCII, the first 32 characters are control characters, e.g.
13: Carriage Return (␍)
10: Line Feed (␊)
9: Horizontal Tab (␉)
11: Vertical Tab (␋)
7: Bell (␇)
27: Escape (␛)
30: Record Separator (␞)
These control code are generally unprintable. But code page 437 did something unique: they defined characters for those first 32 codes:
13: Eighth note (♪)
10: Inverse white circle (◙)
9: White circle (○)
11: Male Sign (♂)
7: Bullet (•)
27: Right Angle (∟)
30: Black up-pointing triangle (▲)
This has interesting implications if you had some text such as:
The price of petrol␍␊
• Germany: €1.56␍␊
• France: €1.49
When encoded in Code Page 850 becomes:
The price of petrol♪◙• Germany: ?1.56♪◙• France: €1.49
Notice 3 things:
The € symbol was lost; replaced with ?
The • symbol was retained
The CR LF symbols were lost; replaced with ♪ and ◙
Trying to decode the code page 437/850 back to real characters presents a problem:
If i want to retain my CRLF, i have to assume that the characters in the 1..32 range actually are ASCII control characters
The price of petrol␍␊
␇ Germany: €1.56␍␊
␇ France: €1.49
If i want to retain my characters (e.g. ¶, •, §) , i have to permanently lose my CRLF, and assume that the characters in 1..32 are actually characters:
The price of petrol♪◙• Germany: €1.56♪◙• France: €1.49
There's no good way out of this.
Ideally Code Page 437 would not have done this to the first 32 characters in the code page, and kept the control characters. And ideally anyone trying to convert the text to 437:
• The price of petrol is € 1.56 in Germany ♪sad song♪
would come back with
? The price of petrol is ? 1.56 in Germany ?sad song?
But that's not what the 437 code page is.
It's a horrible mess; where you have to pick your poison and die slowly.
Rest in Peace Michael Kaplan
This answer brought to you by "☼" (U+263c, a.k.a. WHITE SUN WITH RAYS)
A proud member of the glyph chars collection for more years than Seattle has seen sun
See Michael Kaplan's archived blog entry (🕗):
What the &%#$ does MB_USEGLYPHCHARS do?
I'm still angry at the Microsoft PM who shut down his blog out of spite.