BreakPermittedHere char in filename - unicode

Question

I recieved (mail attachment) a .pdf file with a \0082 (Break Permitted Here) character instead presumably of a é one.
Sciences Num<here>riques et Technologie.pdf
What could have happened ?

Answer 1

It's a simple mojibake case: sender and receiver applies different code pages.
Example for given characters .\Py\mojibakeWindows.py é \x82
Mojibake prove using 97 codecs
string é ['U+00e9'] Latin Small Letter E With Acute
versus ['U+0082'] ??? Cc
[b'\x82']
é ['cp437', 'cp720', 'cp775', 'cp850', 'cp852', 'cp857', 'cp858', 'cp860', 'cp861', 'cp863', 'cp865']
['cp1006', 'latin_1', 'iso8859_2', 'iso8859_3', 'iso8859_4', 'iso8859_5', 'iso8859_6', 'iso8859_7', 'iso8859_8', 'iso8859_9', 'iso8859_10', 'iso8859_13', 'iso8859_14', 'iso8859_15', 'iso8859_16', 'iso8859_11']
Here the mojibakeWindows.py script is as follows:
import sys
import codecs
import unicodedata
if len(sys.argv) == 3:
str1st = sys.argv[1].encode('raw_unicode_escape').decode('unicode_escape').encode('utf-16_BE','surrogatepass').decode('utf-16_BE');
str2nd = sys.argv[2].encode('raw_unicode_escape').decode('unicode_escape').encode('utf-16_BE','surrogatepass').decode('utf-16_BE');
else:
print( 'need two `string` parameters e.g. as follows:');
print( sys.argv[0], '"╧╤╪"', '"ÏÑØ"' )
sys.exit();
codec_list = ['ascii', 'big5', 'big5hkscs', 'cp037', 'cp424', 'cp437', 'cp500', 'cp720', 'cp737', 'cp775', 'cp850', 'cp852', 'cp855', 'cp856', 'cp857', 'cp858', 'cp860', 'cp861', 'cp862', 'cp863', 'cp864', 'cp865', 'cp866', 'cp869', 'cp874', 'cp875', 'cp932', 'cp949', 'cp950', 'cp1006', 'cp1026', 'cp1140', 'cp1250', 'cp1251', 'cp1252', 'cp1253', 'cp1254', 'cp1255', 'cp1256', 'cp1257', 'cp1258', 'euc_jp', 'euc_jis_2004', 'euc_jisx0213', 'euc_kr', 'gb2312', 'gbk', 'gb18030', 'hz', 'iso2022_jp', 'iso2022_jp_1', 'iso2022_jp_2', 'iso2022_jp_2004', 'iso2022_jp_3', 'iso2022_jp_ext', 'iso2022_kr', 'latin_1', 'iso8859_2', 'iso8859_3', 'iso8859_4', 'iso8859_5', 'iso8859_6', 'iso8859_7', 'iso8859_8', 'iso8859_9', 'iso8859_10', 'iso8859_13', 'iso8859_14', 'iso8859_15', 'iso8859_16', 'johab', 'koi8_r', 'koi8_u', 'mac_cyrillic', 'mac_greek', 'mac_iceland', 'mac_latin2', 'mac_roman', 'mac_turkish', 'ptcp154', 'shift_jis', 'shift_jis_2004', 'shift_jisx0213', 'utf_32', 'utf_32_be', 'utf_32_le', 'utf_16', 'utf_16_be', 'utf_16_le', 'utf_7', 'utf_8', 'utf_8_sig'] + ['cp273', 'cp1125', 'iso8859_11', 'koi8_t', 'kz1048']; # 'cp65001',
print( 'Mojibake prove using', len( codec_list ), 'codecs' );
str1stname = unicodedata.name(str1st, '??? {}'.format(unicodedata.category(str1st))) if len(str1st)==1 else ''
str2ndname = unicodedata.name(str2nd, '??? {}'.format(unicodedata.category(str2nd))) if len(str2nd)==1 else ''
print( 'string', str1st, ['U+{0:04x}'.format( ord(ch)) for ch in str1st], str1stname.title() );
print( 'versus', str2nd, ['U+{0:04x}'.format( ord(ch)) for ch in str2nd], str2ndname.title() );
str1list = [];
str2list = [];
strXlist = [];
for cod in codec_list:
for doc in codec_list:
if cod != doc:
# str1ste = codecs.encode( str1st,encoding=cod,errors='replace');
try:
str1ste = codecs.encode( str1st,encoding=cod,errors='strict');
except:
str1ste = b'?' * len(str1st);
# str2nde = codecs.encode( str2nd,encoding=doc,errors='replace');
try:
str2nde = codecs.encode( str2nd,encoding=doc,errors='strict');
except:
str2nde = b'?' * len(str2nd);
if ( str1ste == str2nde and b'?' not in str1ste ):
if cod not in str1list: str1list.append( cod );
if doc not in str2list: str2list.append( doc );
if str1ste not in strXlist: strXlist.append( str1ste );
print( strXlist );
print( str1st, str1list );
print( str2nd, str2list );
Another example applies my Alt KeyCode Finder script (see column Alt0 for ACP code and column Dec for OEMCP code): powershell -COMMAND .\PShell\MyCharMap.ps1 é,0x82
Ch Unicode Dec CP IME Alt Alt0 IME 0405/cs-CZ; CP65001; ACP 65001
é U+00E9 233 …233… Latin Small Letter E With Acute
130 CP437 en-US 0233 (ACP 1252) US & Western Eu
130 CP850 en-GB 0233 (ACP 1252) US & Western Eu
130 CP852 cs-CZ 0233 (ACP 1250) Central Europe
130 CP775 et-EE 0233 (ACP 1257) Baltic
130 CP857 tr-TR 0233 (ACP 1254) Turkish
130 CP720 ar-EG 0233 (ACP 1256) Arabic
vi-VN 0233 (ACP 1258) Vietnamese
U+0082 130 …130… Break Permitted Here
130 CP869 el-gr (ACP 1253) Greek-2
th-TH 0130 (ACP 874) Thai