This post is sequel to Conversion from ttf to type 2 CID font (type 42 base font)
It is futile to have a CID-Keyed font (containingCIDMap that enforces Identity Mapping i.e Glyph index = CID) without offering Unicode support inherently. Then, how to provide UTF-8 support for such a CID-keyed font externally by an application software?
Note: The application program that uses the CID-keyed font can be written in C, C++, Postscript or any other language.
The CID-keyed font NotoSansTamil-Regular.t42 has been converted from Google's Tamil ttf font.
You need this conversion because without this conversion, a postscript program can't access a truetype font!
Refer Post Conversion from ttf to type 2 CID font (type 42 base font) for conversion.
The CIDMap of t42 font enforces an identity mapping as follows:
Character code 0 maps to Glyph index 0
Character code 1 maps to Glyph index 1
Character code 2 maps to Glyph index 2
......
......
Character code NumGlyphs-1 maps to Glyph index NumGlyphs-1
It is clearly evident that there is no Unicode involved in this mapping inherently.
To understand concretely, edit the following postscript program tamil.ps that accesses t42 font through postscript's findfont operator.
%!PS-Adobe-3.0
/myNoTo {/NotoSansTamil-Regular findfont exch scalefont setfont} bind def
13 myNoTo
100 600 moveto
% தமிழ் தங்களை வரவேற்கிறது!
<0019001d002a005e00030019004e00120030002200030024001f002f0024005b0012002a0020007a00aa> show
100 550 moveto
% Tamil Welcomes You!
<0155017201aa019801a500030163018801a5017f01b101aa018801c20003016901b101cb00aa00b5> show
showpage
Issue the following Ghostscript command to execute the postscript program tamil.ps.
gswin64c.exe "D:\cidfonts\NotoSansTamil-Regular.t42" "D:\cidfonts\tamil.ps (on Windows Platform).
gs ~/cidfonts/NotoSansTamil-Regular.t42 ~/cidfonts/tamil.ps (on Linux Platform).
This will display two strings தமிழ் தங்களை வரவேற்கிறது! and Tamil Welcomes You! respectively in subsequent rows.
Note that the strings for show operator are in Hexadecimal format embedded within angular brackets. Operator show extracts 2 bytes at a time and maps this CID (16 bit value) to a Glyph.
For example, the first 4 Hex digits in the 1st string is 0019 whose decimal equivalent is 25. This maps to glyph த.
In order to use this font t42, each string (created from character set of a ttf) should be converted into hexadecimal string by hand which is practically impossible and therefore this font becomes futile.
Now consider the following C++ code that generates a postscript program called myNotoTamil.ps that accesses the same t42 font through postscript's findfont operator.
const short lcCharCodeBufSize = 200; // Character Code buffer size.
char bufCharCode[lcCharCodeBufSize]; // Character Code buffer
FILE *fps = fopen ("D:\\cidfonts\\myNotoTamil.ps", "w");
fprintf (fps, "%%!PS-Adobe-3.0\n");
fprintf (fps, "/myNoTo {/NotoSansTamil-Regular findfont exch scalefont setfont} bind def\n");
fprintf (fps, "13 myNoTo\n");
fprintf (fps, "100 600 moveto\n");
fprintf (fps, u8"%% தமிழ் தங்களை வரவேற்கிறது!\n");
fprintf (fps, "<%s> show\n", strps(ELang::eTamil, EMyFont::eNoToSansTamil_Regular, u8"தமிழ் தங்களை வரவேற்கிறது!", bufCharCode, lcCharCodeBufSize));
fprintf (fps, "%% Tamil Welcomes You!\n");
fprintf (fps, "<%s> show\n", strps(ELang::eTamil, EMyFont::eNoToSansTamil_Regular, u8"Tamil Welcomes You!", bufCharCode, lcCharCodeBufSize));
fprintf (fps, "showpage\n");
fclose (fps);
Although the contents of tamil.ps and myNotoTamil.ps are same and identical, the difference in the production of those ps files is like difference between heaven and earth!
Observe that unlike tamil.ps(handmade Hexadecimal strings), the myNotoTamil.ps is generated by a C++ program which uses UTF-8 encoded strings directly hiding the hex strings completely. The function strps produces hex strings from UTF-8 encoded strings which are the same and identical as the strings present in tamil.ps.
The futile t42 font has suddenly become fruitful due to strps function's mapping ability from UTF-8 to CIDs (every 2 bytes in Hex strings maps to a CID)!
The strps function consults a mapping table aNotoSansTamilMap (implemented as a single dimensional array constructed with the help of Unicode Blocks) in order to map Unicode Points (extracted from UTF-8 encoded string) to Character Identifiers (CIDs).
The buffer bufCharCode used in strps function (4th parameter) passes out hex strings corresponding to UTF-8 encoded strings to Postscript's show operator.
In order to benefit others, I released this UTF8Map program through GitHub on the following platforms.
Windows 10 Platform (Github Public Repository for UTF8Map Program on Windows 10)
Open up DOS command line and issue the following clone command to download source code:
git clone https://github.com/marmayogi/UTF8Map-Win
Or execute the following curl command to download source code release in zip form:
curl -o UTF8Map-Win-2.0.zip -L https://github.com/marmayogi/UTF8Map-Win/archive/refs/tags/v2.0.zip
Or execute the following wget command to download source code release in zip form:
wget -O UTF8Map-Win-2.0.zip https://github.com/marmayogi/UTF8Map-Win/archive/refs/tags/v2.0.zip
Linux Platform (Github Public Repository for UTF8Map Program on Linux)
Issue the following clone command to download source code:
git clone https://github.com/marmayogi/UTF8Map-Linux
Or execute the following curl command to download source code release in tar form:
curl -o UTF8Map-Linux-1.0.tar.gz -L https://github.com/marmayogi/UTF8Map-Linux/archive/refs/tags/v1.0.tar.gz
Or execute the following wget command to download source code release in tar form:
wget -O UTF8Map-Linux-1.0.tar.gz https://github.com/marmayogi/UTF8Map-Linux/archive/refs/tags/v1.0.tar.gz
Note:
This program uses t42 file to generates a ps file (a postscript program) which will display the following in a single page:
A welcome message in Tamil and English.
List of Vowels (12 + 1 Glyphs). All of them are associated with Unicode Points.
List of Consonants (18 + 6 = 24 Glyphs). No association of Unicode Points.
List of combined glyphs (Combination of Vowels + Consonants) in 24 lines. Each line displays 12 glyphs. Out of 288 Glyphs, 24 are associated with Unicode Points and rest do not.
List of Numbers in two lines. All 13 Glyphs for Tamil numbers are associated with Unicode Points.
A foot Note.
The two program files (main.cpp and mapunicode.h) are 100% portable. i.e. the contents of two files are same and identical across platforms.
The two mapping tables aNotoSansTamilMap and aLathaTamilMap are given in mapunicode.h file.
A README Document in Markdown format has been included with the release.
This software has been tested for t42 fonts converted from the following ttf files.
Google's Noto Tamil ttf
Microsoft`s Latha Tamil ttf
Related
I am looking for help in printing the club symbol from the Arial font in postscript.
It has a Unicode of 9827 (2663 hexadecimal).
The ampersand is Unicode 38 (26 hexadecimal)
This postscript code snippet
!PS
/ArialMT findfont
12 scalefont setfont
72 72 moveto
<26> show
showpage
produces the ampersand system when I run it through Adobe Distiller. It appears that postscript understands Unicodes with UTF-8 encoding by default.
I am unable to do the same with the club symbol.
My research indicates that I have to use Character Encoding and this where I am lost.
Could some kind soul show me (hopefully fairly short) how to show the club symbol using Character Encoding?
Alternatively if you could point me to a simple tutorial it would be greatly appreciated.
Reading the reference manual leaves my head spinning.
PostScript does not understand Unicode, not at all, or at least not as standard, though there are ways to deal with it.
Section 5.3 of the PostScript Language Reference Manual contains complete information on Character Encoding. You really need to read this in detail, what you are asking is a deceptively simple question, with no simple answer.
The way this works for PostScript fonts is that the characters in the document have a character code which lies between 0 and 255. When processing text, the interpreter takes the character code and looks it up in the Encoding attached to the font. If you didn't supply an Encoding to the font, then it will normally have a pre-defined StandardEncoding.
StandardEncoding has some congruence with UTF-8, for character codes 0x7F and below, but it's not exactly the same I don't think.
The Encoding maps the character code to a glyph name, For example 0x41 in StandardEncoding maps to /A (that's a name in PostScript). Note that is not UTF-8 or anything else, it's a mapping. It's entirely possible, and common practice for subset fonts, to map the first character used to character code 1, the second to character code 2 and so on.
So if we applied that scheme to 'Hello World' we would use an Encoding which maps
0x01->/H
0x02->/e
0x03->/l
0x04->/o
0x05->/space
0x06->/W
0x07->/r
0x08->/d
and then we might draw the text by :
<0102030304050604070308> show
Which, as you can see, bears no relation to UTF-8 at all.
Anyway, having retrieved the glyph name the interpreter then looks at the CharStrings dictionary in the font and locates the key associated with the character code. So for StandardEncoding we would map the 0x41 to /A and we'd then look in the CharStrings dictionary for the /A key. We then take the value associated with that key, which will be a PostScript glyph program and run it.
Your problem is that you are trying to use a TrueType font. PostScript does not support TrueType fonts in that way, it does support them when they are defined as Type42 fonts, because a Type42 font carries around some additional information which allows the PostScript interpreter to treat them, broadly speaking, the same way as PostScript fonts.
Many modern PostScript interpreters will load a TrueType font and create a Type42 font from it for you, but this involves guessing at the additional information, and there's no real way to tell in advance how any given interpreter will deal with this. I suspect that Adobe Distiller will behave similarly to Ghostscript and attempt to map the type42 to a StandardEncoding.
Essentially the Encoding maps the character code to a key in the CharStrings dictionary and the value associated with that key is the GID. The GID is used to index the GLYF table in the TrueType font, the TrueType rasteriser then reads that glyph program.
So in order to create a type42 font with an Encoding which will map a character code to a club symbol, you would need to know what the GID of the club symbol in the font actually is. This can be derived from one of the CMAP subtables in the TrueType font, which is how PostScript interpreters such as Ghostscript build the required Encoding when they load a TrueType font as a Type 42. You would then need to alter the CharStrings dictionary in the type42 font so that it maps to the correct GID. You would also need to alter the Encoding; choose a character code that you want to use, map the character code to the key in the CharStrings dictionary.
You would have to determine what kind of keys the Encoding and CharStrings dictionary is using. It might be names or it might be integers or anything else. You could figure that out of course by looking at the content of the Encoding array.
In all honesty unless you know a lot about TrueType fonts I think it would be hard for you to reverse-engineer the font to retrieve the correct GID and then re-encode the font that gets loaded by the interpreter. You would also need to examine the contents of the font dictionary returned by findfont to see what the existing mapping is. And crucially you may need to modify the CharStrings dictionary to map the key to the GID. It may be that Distiller returns a dictionary which is defined as no-access which will prevent you looking inside or (or at least, inside parts of it). You might be able to get away with looking at the Encoding in the font dictionary and modifying that, if the CharStrings dictionary already contains a key for every glyph in the font, which it may well do.
I could probably guide you through doing this with Ghostscript, but I have no idea how Adobe Distiller defines TrueType fonts loaded from disk.
You could use a CIDFont instead. These are defined in section 5.11.1 and it may be that if you were to use something like the pre-defined Identity-H or UCS2 CMaps you could create a CID-Keyed instance of ArialMT with TrueType outlines which would work for your Unicode code point.
But that would mean defining the font yourself, so you would need to include the whole TrueType font as part of your PostScript program. Again this isn't simple.
There is some good information here: Show Unicode characters in PostScript
I also have the ArialMT.ttf and made the ArialMT.ttf.t42 just to look inside. I found the /club glyph with GID 389 as described by KenS and tried this as described in the linked post with good results:
%!
100 300 moveto
/ArialMT.ttf 46 selectfont (ArialMT) show
100 200 moveto /club glyphshow
showpage
Note: I use ArialMT.ttf because the TT font wasn't installed in the ghostscript Fontmap just in the current directory so used gs -P for that reason. The normal /ArialMT findfont should work when the TT font is already installed in the search path. This is my first attempt with these glyphs and was just using trial and error.
There is a comprehensive Adobe list of glyphs that map many of the Unicode characters: https://github.com/adobe-type-tools/agl-aglfn/blob/master/glyphlist.txt.
If the desired Unicode character is in that list, say club;2663 or clubsuitblack;2663 or clubsuitwhite;2667, all one needs to say is
/club glyphshow and most modern fonts will know what to do. But #KenS says this "can cause problems".
Instead the preferred scheme that emerges from the recommended references is to:
create a composite font in the preamble, one for each of the fonts
you are using;
include the lower 256 characters as Font0;
add whatever Unicode characters you are planning to use, in chunks of
256 characters, as Font1, Font2 etc.;
remap the Unicode of the special characters onto a two-character
sequence, of the sub-font index within the composite font, followed
by the byte that is the index of the character within that sub-font.
The following is a complete example of both methods.
I use http://www.acumentraining.com/Acumen_Journal/AcumenJournal_May2002.zip, but with Font1 is a custom-remapping of a portion of the same font as Font0, re-using some of the well known ascii character(s).
This a complete file.eps:
%!PS-Adobe-3.0 EPSF-3.0
%%BoundingBox: 0 0 792 612
%%LanguageLevel: 2
%%EndComments
%%BeginProlog
userdict begin
%%EndProlog
%%BeginSetup
% The following encodes a few useful Unicode glyphs, if only a few are needed.
% Based on https://stackoverflow.com/questions/54840594/show-unicode-characters-in-postscript
% Usage: /Times-Roman /Times-Roman-Uni UniVec new-font-encoding
/new-font-encoding { <<>> begin
/newcodesandnames exch def
/newfontname exch def
/basefontname exch def
/basefontdict basefontname findfont def % Get the font dictionary on which to base the re-encoded version.
/newfont basefontdict maxlength dict def % Create a dictionary to hold the description for the re-encoded font.
basefontdict
{ exch dup /FID ne % Copy all the entries in the base font dictionary to the new dictionary except for the FID field.
{ dup /Encoding eq
{ exch dup length array copy % Make a copy of the Encoding field.
newfont 3 1 roll put }
{ exch newfont 3 1 roll put }
ifelse
}
{ pop pop } % Ignore the FID pair.
ifelse
} forall
newfont /FontName newfontname put % Install the new name.
newcodesandnames aload pop % Modify the encoding vector. First load the new encoding and name pairs onto the operand stack.
newcodesandnames length 2 idiv
{ newfont /Encoding get 3 1 roll put}
repeat % For each pair on the stack, put the new name into the designated position in the encoding vector.
newfontname newfont definefont pop % Now make the re-encoded font description into a POSTSCRIPT font.
% Ignore the modified dictionary returned on the operand stack by the definefont operator.
end} def
/Helvetica /Helvetica-Uni [
16#43 /club % ASCII 43 = C = /club
] new-font-encoding
/Helv
<<
/FontType 0
/FontMatrix [ 1 0 0 1 0 0 ]
/FDepVector [
/Helvetica findfont % this is Font0
/Helvetica-Uni findfont % this is Font1
]
/Encoding [ 0 1 ]
/FMapType 3
>> definefont pop
%%EndSetup
%%BeginScript
/Helv 20 selectfont
72 300 moveto
(The club character is \377\001C\377\000 a part of the string.) show
/Helvetica findfont 20 scalefont setfont
263 340 moveto
/club glyphshow
showpage
%%EOF
Which produces this
Obviously, this can be extended to more characters, but only 256 per sub-font. I am not aware of a "standard" convention for such re-encoding, although I would imagine a set of Greek letters alpha,beta,gamma... would map pretty obviously onto a,b,c... Perhaps somebody else is aware of such an implementation for all of the Unicode characters from the Adobe glyph list using multiple custom sub-fonts, and can provide a pointer..
I am using a printer which needs Cyrillic sets CP 1048 for printing Kazak and Russian language in text mode. How do we convert a text to CP 1048? CP 1048 is combined character set for Kazak and Russian languages. These languages come together in text files and this code page is available as a standard feature in the printer.
You convert text with some kind of text encoding converter. Since it wasn't specified, I'll use a Python script. Note this requires Python 3.5 or later. That version first defined the kz1048 codec.
Unicode string to KZ-1048 encoding:
>>> 'Россия/Ресей'.encode('kz1048')
b'\xd0\xee\xf1\xf1\xe8\xff/\xd0\xe5\xf1\xe5\xe9'
Note in Python b'\xd0\xee' denotes a byte string containing the byte hexadecimal values D0 and EE, which in KZ-1048 represent Р and о.
In many places we can read that, for example, "C# uses UTF-16 for its strings" (link). Technically, what does this mean?
My source file is just some text. Say I'm using Notepad++ to code a simple C# app; how the text is represented in bytes on disk, after I save the file, depends on N++, so that's probably not what people mean. Does that mean that:
The language specification requires/recommends that the compiler input be encoded as UTF-16?
The standard library functions are encoding-aware and treat the strings as UTF-16, for example String's operator [] (which returns the n-th character and not the n-th byte)?
Once the compiler produces an executable, the strings stored inside it are in UTF-16?
I've used C# as an example, but this question applies to any language of which one could say that it uses encoding Y for its strings.
"C# uses UTF-16 for its strings"
As far as I understand this concept, this is a simplification at best. A CLI runtime (such as the CLR) is required to store strings it loads from assemblies or that are generated at runtime in UTF-16 encoding in memory - or at least present them as such to the rest of the runtime and the application.
See CLI specification:
III.1.1.3 Character data type
A CLI char type occupies 2 bytes in memory and represents a Unicode code unit using UTF-16
encoding. For the purpose of stack operations char values are treated as unsigned 2-byte integers
(§III.1.1.1)
And C# specification:
4.2.4 The string type
Instances of the string class represent Unicode [being UTF-16 in .NET jargon] character strings.
I can't find that quickly which file encodings the C# compiler supports, but I'm quite sure you can have a source file stored in UTF-8 encoding, or even ASCII (or another non-unicode code page).
The standard library functions are encoding-aware and treat the strings as UTF-16
No, the BCL just treats strings as strings, being a wrapper around a char[] array. Only when transitioning outside the runtime, like in a P/Invoke call, the runtime "knows" which platform functions to invoke and how to marshal a string to those functions. See for example C++/CLI Converting from System::String^ to std::string
Once the compiler produces an [assembly], the strings are stored inside it in UTF-16?
Yes.
Let take a look at C/C++ char type. It is 8 bits long(1 byte). This means that it can store 255 different symbols. Now let's think what actually a font is. It is something like a map. Values from 0 to 255 (1 byte) are mapped to symbols. These type of fonts usually contains 2 type of characters (cyrillic and latin for example) and special symbols. There is no enough space (255 limit) to save greece or chinese letters.
Now let's see what is UTF-8. It is encoding, which stores some symbols using 8 bits and some using 16 bits. For example if you type in notepad word "word" and save file with UTF-8 encoding the resulting file will be exactly 4 bytes length, but if you type word "дума", which is again 4 symbols it will use 8 bytes on your storage. So some letters are stored as 1 byte, others as 2.
UTF-16 means that all symbols are stored in 2 bytes and logically UTF-32 = 4 bytes.
Let's see how this looks from programming sight. When you are typing symbols in notepad they are stored in RAM (in some format that notepad understands). When you save file on the disk notepad write a sequence of bytes on the disk. These sequence depends on chosen encoding. When you are reading (with C# or some other language) file you have to know its encoding. By knowing it you will know how to interpret the sequence written on the disk.
If a DICOM file does not define a Specific Character Set (0008,0005), what character set does it use by default? Is ASCII the default encoding for DICOM files?
TL;DR
A DICOM file contains German ä in one of the tags, but the file does not specify any character set. I assume that in this case the file is allowed to contain only ASCII symbols (the default character set) and report this file as invalid. Before I submit my change, I want to make sure that I understood DICOM correctly.
As specified in the Dicom Data Structures and Encoding
6.1.2.5.4 Levels of Implementation and Initial Designation
a) Attribute Specific Character Set (0008,0005) not present:
7-bit code
Implementation level: ISO 2022 Level 1 - Elementary 7-bit code (code-level identifier 1)
Initial designation: ISO-IR 6 (ASCII) as G0.
Code Extension shall not be used
Reference:
http://dicom.nema.org/medical/dicom/current/output/chtml/part05/chapter_6.html#sect_6.1.2.5.4
To add to answer by JonnyQ, DICOM standard also defines mechanisms when confronted with character sets that are unknown to implementations or unsupported (see PS 3.5 section 6.1.2.3). Implementations can print or display such characters by replacing all unknown characters with the four characters "\nnn", where "nnn" is the three digit octal representation of each byte.
An example given in the standard for an ASCII based machine as follows:
Character String: Günther
Encoded representation: 04/07 15/12 06/14 07/04 06/08 06/05 07/02
ASCII based machine: G\374nther
Implementations may also encounter Control Characters which they have no means to print or display.
Application may print or display such Control Characters by replacing the Control Character with the
four characters “\nnn”, where “nnn” is the three digit octal representation of each byte.
I'm using Ubuntu. My console (using Python or bash) does not print utf8 chars of 4 bytes, only 3 bytes or less.
Example (bash):
eran#eran-MS-7788:~/Dropbox/help$ echo -e "\xee\x8c\xb0"
# This prints the v sign
eran#eran-MS-7788:~/Dropbox/help$ echo -e "\xf0\x9f\x8f\x80"
# This should print the basketball sign but does not print so
I get the same from the Python console.
Any way to solve this?
Most so called Unicode fonts only implement characters present in Unicode BMP (Basic Multiligual Plane) - from U+0000 to U+FFFF, which all can be represented using just 3 bytes using UTF-8.
There are very few fonts that implement characters beyond BMP: that is, from U+010000 to U+1FFFFF (which are encoded using 4 bytes in UTF-8), and I would think that your Linux console or terminal is no exception. Because of this, you are not likely to see character you expect. Over time, it is possible that support for characters beyond BMP would become better, but it will probably take a long time. You can read more about this here.