Does anyone know about these 2 functions? Should the output of 'F'.atohex() be 0x16 or 0x46 (directly from the ASCII table)? I have googled this already, but some said the former one is correct while some said the other. Thanks.
Actually, the result is 0xF. These functions do not have the greatest names. What both do is convert an ASCII string in a particular radix to an integral value. atohex assumes the string is formatted in hexadecimal.
from LRM:
— str.atoi() returns the integer corresponding to the ASCII decimal representation in str.
— atohex interprets the string as hexadecimal.
— atooct interprets the string as octal.
— atobin interprets the string as binary.
NOTE—These ASCII conversion functions return a 32-bit integer value
So, the result of the following:
string a = "F";
a.atohex();
ia a 32-bit integer: 32'hF.
Related
From strings blog post :
Some people think Go strings are always UTF-8, but they are not: only
string literals are UTF-8. As we showed in the previous section,
string values can contain arbitrary bytes; as we showed in this one,
string literals always contain UTF-8 text as long as they have no
byte-level escapes.
To summarize, strings can contain arbitrary bytes, but when
constructed from string literals, those bytes are (almost always)
UTF-8.
Can you give me an example of a string literal that isn’t an utf-8 ?
What is the difference in go between "string literal" , "string value" , "string literal without byte-level escapes" .
Hope this helps:
As 32bitkid mentioned: The following character in Go source code is a string literal whose value is not UTF-8 encoded: "\xbd\xb2\x3d\xbc\x20\xe2\x8c\x98".
The idea of a "string literal" exists in Go source code only and has no representation in a compiled or even running program. A string literal in Go source code is written as "cat dog" and if your string literal needs to contain something your keyboard is missing (or your editor cannot display) you may use "byte level escapes" like this "cat\x07dog". Once your Go source code is compiled the notion of a string literal vanishes: There are only strings and they have some value. This value may be computed during the running time of your code or consist of values generated from "string literals" in your source.
"String literals" are to strings what "number literals" ar to ints: "abc" is a string literal and 20 is an int literal. Both may have different representations, e.g. "\x61bc" and 0x14. But once your code is compiled there is no difference whether your int value came from the literal 20 or 0x14. Same with strings. Only complication: Go source code is UTF-8 allways.
I use sprintf for conversion to hex - example >>
$hex = sprintf("0x%x",$d)
But I was wondering, if there is some alternative way how to do it without sprintf.
My goal is convert a number to 4-byte hex code (e.g. 013f571f)
Additionally (and optionally), how can I do such conversion, if number is in 4 * %0xxxxxxx format, using just 7 bits per byte?
sprintf() is probably the most appropriate way. According to http://perldoc.perl.org/functions/hex.html:
To present something as hex, look into printf, sprintf, and unpack.
I'm not really sure about your second question, it sounds like unpack() would be useful there.
My goal is convert a number to 4-byte hex code (e.g. 013f571f)
Hex is a textual representation of a number. sprintf '%X' returns hex (the eight characters 013f571f). sprintf is specifically designed to format numbers into text, so it's a very elegant solution for that.
...But it's not what you want. You're not looking for hex, you're looking for the 4-byte internal storage of an integer. That has nothing to do with hex.
pack 'N', 0x013f571f; # "\x01\x3f\x57\x1f" Big-endian byte order
pack 'V', 0x013f571f; # "\x1f\x57\x3f\x01" Little-endian byte order
sprintf() is my usual way of performing this conversion. You can do it with unpack, but it will probably be more effort on your side.
For only working with 4 byte values, the following will work though (maybe not as elegant as expected!):
print unpack("H8", pack("N1", $d));
Be aware that this will result in 0xFFFFFFFF for numbers bigger than that as well.
For working pack/unpack with arbitrary bit length, check out http://www.perlmonks.org/?node_id=383881
The perlpacktut will be a handy read as well.
For 4 * %0xxxxxxx format, my non-sprintf solution is:
print unpack("H8", pack("N1",
(((($d>>21)&0x7f)<<24) + ((($d>>14)&0x7f)<<16) + ((($d>>7)&0x7f)<<8) + ($d&0x7f))));
Any comments and improvements are very welcome.
Documentation all directs me to unicode support, yet I don't think my request has anything to do with Unicode. I want to work with raw bytes within the context of a single scalar; I need to be able to figure out its length (in bytes), take substrings of it (in bytes), write the bytes to disc, and over the network. Is there an easy way to do this, without treating the bytes as any sort of encoding in perl?
EDIT
More explicitly,
my $data = "Perl String, unsure of encoding and don't need to know";
my #data_chunked_into_1024_bytes_each = #???
Perl strings are, conceptually, strings of characters, which are positive 32-bit integers that (normally) represent Unicode code points. A byte string, in Perl, is just a string in which all the characters have values less than 256.
(That's the conceptual view. The internal representation is somewhat more complicated, as the perl interpreter tries to store byte strings — in the above sense — as actual byte strings, while using a generalized UTF-8 encoding for strings that contain character values of 256 or higher. But this is all supposed to be transparent to the user, and in fact mostly is, except for some ugly historical corner cases like the bitwise not (~) operator.)
As for how to turn a general string into a byte string, that really depends on what the string you have contains and what the byte string is supposed to contain:
If your string already is a string of bytes — e.g. if you read it from a file in binary mode — then you don't need to do anything. The string shouldn't contain any characters above 255 to being with, and if it does, that's an error and will probably be reported as such by the encryption code.
Similarly, if your string is supposed to encode text in the ASCII or ISO-8859-1 encodings (which encode the 7- and 8-bit subsets of Unicode respectively), then you don't need to do anything: any characters up to 255 are already correctly encoded, and any higher values are invalid for those encodings.
If your input string contains (Unicode) text that you want to encode in some other encoding, then you'll need to convert the string to that encoding. The usual way to do that is by using the Encode module, like this:
use Encode;
my $byte_string = encode( "name of encoding", $text_string );
Obviously, you can convert the byte string back to the corresponding character string with:
use Encode;
my $text_string = decode( "name of encoding", $byte_string );
For the special case of the UTF-8 encoding, it's also possible to use the built-in utf8::encode() function instead of Encode::encode():
utf8::encode( $string );
which does essentially the same thing as:
use Encode;
$string = encode( "utf8", $string );
Note that, unlike Encode::encode(), the utf8::encode() function modifies the input string directly. Also note that the "utf8" above refers to Perl's extended UTF-8 encoding, which allows values outside the official Unicode range; for strictly standards-compliant UTF-8 encoding, use "utf-8" with a hyphen (see Encode documentation for the gory details). And, yes, there's also a utf8::decode() function that does pretty much what you'd expect.
If I understood your question correctly, what you want is the pack/unpack functions: http://perldoc.perl.org/functions/pack.html
As long as your string doesn't contain characters above codepoint 255, it will mostly work as plain byte string, with length and substr operating on bytes. Additionally, most output functions like print expect octets/bytes by default and will actually complain if you try to stuff anything else to them.
You may need to explicitly encode/decode your output if it is known to be in some encoding, but more details can only be added if you ask another specific question for each problematic part of your program.
are there any hexadecimal encoding/decoding functions that mysql will recognize to avoid writing a whole algorithm out. base16_encode/decode does not work. Thanks
MySQL understands hexidecimal literals
MySQL supports hexadecimal values, written using X'val', x'val', or 0xval format, where val contains hexadecimal digits (0..9, A..F). Lettercase of the digits does not matter. For values written using X'val' or x'val' format, val must contain an even number of digits. For values written using 0xval syntax, values that contain an odd number of digits are treated as having an extra leading 0. For example, 0x0a and 0xaaa are interpreted as 0x0a and 0x0aaa.
http://dev.mysql.com/doc/refman/5.0/en/hexadecimal-literals.html
I have a problem using this query:
select decode(to_hex(ascii('ل')::int),'hex')
When I execute it, I get:
ERROR: invalid hexadecimal data: odd number of digits
decode(..., 'hex') doesn't mean convert this hexadecimal number to something. Hex encoding is a particular encoding format for bytes, and it requires two hexadecimal digits per octet. On the other hand, to_hex converts an integer to a hexadecimal representation, and that could have an even or odd number of digits.
So the answer is, you can't do that (without some manual fixups). And it's not clear why you would want to, either. It looks like you could just do 'ل'::bytea, but that might not be what you wanted either.
May be it's simpler to use something like this:
select encode('ل','escape');