I have a string named $title
Gardens and Anti-Gardens in Marie de France’s <i>Lais</i>
and I am getting this error
"\x{2019}" does not map to iso-8859-1
I try removing the italic tags but it still gives me the error i.e.
$title =~ s/<i>|<\/i>//g;
Thank you
Why do you think the HTML tags have anything to do with characters in the string?
If you google the \x{2019} the first hit is this.
Unicode Character 'RIGHT SINGLE QUOTATION MARK' (U+2019)
That's the ’. Typically Microsoft Word converts apostrophes (single quotes ') to those kinds of quotation marks. It looks like you are trying to print your string somewhere where its converted to the ISO-8859-1 encoding. You should be able to specifically convert that character to something that makes more sense, like the above mentioned single quote '.
$string =~ s/\x{2019}/'/g;
That should get rid of that one warning. But if you import something with unicode and then expect it to be output as latin-1 more characters will fail.
The encoding ISO-8859-1 does not contain the character U+2019.
Related
I have input that people will probably say "that's not really CSV", but I still have to parse it. (using Papa Parse)
comma is the delimiter. backslash is the escape. comma, double quote, backslash, r and n (to denote newlines) can all be escaped. There is no "quoting" of strings.
so... I see data like:
this is one\, field,1/2\" bolt,this is text with \\ and a new line \r\n embedded
and I want:
[0] this is one\, field
[1] 1/2\" bolt
[2] this is text with \\ and a new line \r\n embedded
but I'm getting
[0] this is one\
[1] field
[2] 1/2\" bolt
...
I can deal with the other \x things in post processing... I'd just like to get it to handle \, correctly.
I've tried the obvious values of quoteChar and escapeChar with no luck.
oh... and the Donate link is broken on https://www.papaparse.com/ if Matt Holt is listening.
const parsed = window.Papa.parse(csvText, {
escapeChar: '\\',
});
Seems like default escape character is ", but it can be overidden in the paramters.
Upd. though now that I look at it, it does not seem to work with your case. It only did fix an issue I had of 2.5\","Shell being considered single value because " was interpreted as escape character for ,.
I'm starting to get a feeling that the only way to escape coma is to enclose the field in the quotes.
Hope someone will post the right answer eventually...
I need to print label with special character like degree (°).
I'm using qz print applet on my website.
How i can say to applet that i'm going to print UtF-8 character?
Because it doesn't print correctly.
Thanks!
Well, you need to escape characters above ASCII by putting ^FH (Field Hexadecimal Indicator) before any ^FD field that might contain an UTF char and you also need to prefix the UTF Hex code with an underscore. Like happens in this other question: Unicode characters on ZPL printer
I am having trouble with non ascii characters being returned. I am not sure at which level the issue resides. It could be the actual PDF encoding, the decoding used by CAM::PDF (which is FlateDecode) or CAM::PDF itself. The following returns a string full of the commands used to create the PDF (Tm, Tj, etc).
use CAM::PDF;
my $filename = "sample.pdf";
my $cam_obj = CAM::PDF->new($filename) or die "$CAM::PDF::errstr\n";
my $tree = $cam_obj->getPageContentTree(1);
my $page_string = $tree->toString();
print $page_string;
You can download sample.pdf here
The text returned in the Tj often has one character which is non ASCII. In the PDF, the actual character is almost always a quote, single or double.
While reproducing this I found that the returned character is consistent within the PDF but varies amongst PDFs. I also noticed the PDF is using a specific font file. I'm now looking into font files to see if the same character can be mapped to varying binary values.
:edit:
Regarding Windows-1252. My PDF returns an "Õ" instead of apostrophes. The Õ character is hex 0xD5 in Windows-1252 and UTF-8. If the idea is that the character is encoded with Windows-1252, then it should be a hex 0x91 or 0x92 which it is not. Which is why the following does nothing to the character:
use Encode qw(decode encode);
my $page_string = 'Õ';
my $characters = decode 'Windows-1252', $page_string;
my $octets = encode 'UTF-8', $characters;
open STS, ">TEST.txt";
print STS $octets . "\n";
I'm the author of CAM-PDF. Your PDF is non-compliant. From the PDF 1.7 specification, section 3.2.3 "String Objects":
"Within a literal string, the backslash (\) is used as an escape
character for various purposes, such as to include newline characters,
nonprinting ASCII characters, unbalanced parentheses, or the backslash
character itself in the string. [...] The \ddd escape sequence provides
a way to represent characters outside the printable ASCII character set."
If you have large quantities of non-ASCII characters, you can represent them using hexadecimal string notation.
EDIT: Perhaps my interpretation of the spec is incorrect, given a_note's alternative answer. I'll have to revisit this... Certainly, the spec could be clearer in this area.
Sorry to intrude, and with all due respect, sir, but file IS compliant. Section 3.2.3 further states:
[The \ddd] notation provides a way to specify characters outside the
7-bit ASCII character set by using ASCII characters only. However,
any 8-bit value may appear in a string.
"receiving" - where? You get "Õ" instead of expected what? And doing exactly what? You know that windows command prompt uses dos code page, not windows-1252, right? (oops, new thread again... probably i should register here :-) )
I would like to replace characters with certain Unicode values in a variable with dash. I have two ideas which might work, but I do not know how to check for the value of character:
1/ processing variable as string, checking every characters value and placing these characters in a new variable (replacing those characters which are invalid)
2/ use these magic :-)
$variable = s/[$char_range]/-/g;
char_range should be similar to [0-9] or [A-Z], but it should be values for utf-8 characters. I need range from 0x00 to 0x7F to be exact.
The following expression should replace anything that is not ASCII with a hyphen, which is (I think) what you want to do:
s/[\N{U+0080}-\N{U+FFFF}]/-/g
There's no such thing as UTF-8 characters. There are only characters that you encode into UTF-8. Even then, you don't want to make ranges outside of the magical ones that Perl knows about. You're likely to get more than you expect.
To get the ordinal value for a character, use ord:
use utf8;
my $code_number = ord '😸'; # U+1F638
say sprintf "%#x", $code_number;
However, I don't think that's what you need. It sounds like you want to replace characters in the ASCII range with a -. You can specify ranges of code numbers:
s/[\000-\177]/-/g; # in octal
s/[\x00-\x7f]/-/g; # in hexadecimal
You can specify wide character ordinal values in braces:
s/[\x80-\x{10ffff}]/-/g; # wide characters, replace non-ASCII in this case
When the characters have a common property, you can use that:
s/\p{ASCII}/-/g;
However, if you are replacing things character for character, you might want a transliteration:
$string =~ tr/\000-\177/-/;
I have a HTML string in ISO-8859-1 encoding. I need to pass this string to HTML:Entities::decode_entities() for converting some of the HTML ASCII codes to respective chars. To so i am using a module HTML::Parser::Entities 3.65 but after decode_entities() operation my whole string changes to utf-8 string. This behavior seems fine as the documentation of the HTML::Parse. As i need this string back in ISO-8859-1 format for further processing so i have used Encode::encode("iso-8859-1",$str) to change the string back to ISO-8859-1 encoding.
My results are fine excepts for some chars, a question mark is coming instead. One example is single quote ' ASCII code (’)
Can anybody help me if there any limitation of Encode module? Any other pointer will also be helpful to solve the problem.
I am pasting the sample text having the char causing the issue:
my $str = "This is a test string to test the encoding of some chars like ’ “ ” etc these are failing to encode; some of them which encode correctly are é « etc.";
Thanks
There's a third argument to encode, which controls the checking it does. The default is to use a substitution character, but you can set it to FB_CROAK to get an error message.
The fundamental problem is that the characters represented by ’, “, and ” do not exist in ISO-8859-1. You'll have to decide what it is that you want to do with them.
Some possibilities:
Use cp1252, Microsoft's "extended" version of ISO-8859-1, instead of the real thing. It does include those characters.
Re-encode the entities outside the ISO-8859-1 range (plus &), before converting from utf-8 to ISO-8859-1:
my $toEncode = do { no warnings 'utf8'; "&\x{0100}-\x{10FFFF}" };
$string = HTML::Entities::encode_entities($string, $toEncode);
(The no warnings bit is needed because U+10FFFF hasn't actually been assigned yet.)
There are other possibilities. It really depends on what you're trying to accomplish.