I'm currently working on something that requires me to pass a Base64 string to a PowerShell script. But while decoding the string back to the original I'm getting some unexpected results as I need to use UTF-7 during decoding and I don't understand why. Would someone know why?
The Mozilla documentation would suggest that it's insufficient to use Base64 if you have Unicode characters in your string. Thus you need to use a workaround that consists of using encodeURIComponent and a replace. I don't really get why the replace is needed and shortened it to btoa(escape('✓ à la mode')) to encode the string. The result of that operation would be JXUyNzEzJTIwJUUwJTIwbGElMjBtb2Rl.
Using PowerShell to decode the string back to the original, I need to first undo the Base64 encoding. In order to do System.Convert can be used (which results in a byte array) and its output can be converted to a UTF-8 string using System.Text.Encoding. Together this would look like the following:
$bytes = [System.Convert]::FromBase64String($inputstring)
$utf8string = [System.Text.Encoding]::UTF8.GetString($bytes)
What's left to do is URL decode the whole thing. As it is a UTF-8 string I'd expect only to need to run the URL decode without any further parameters. But if you do that you end up with a accented a that looks like � in a file or ? on the console. To get the actual original string it's necessary to tell the URL decode to use UTF-7 as the character set. It's nice that this works but I don't really get why it's necessary since the string should be UTF-8 and UTF-8 certainly supports an accented a. See the last two lines of the entire script for what I mean. With those two lines you will end up with one line that has the garbled text and one which has the original text in the same file encoded as UTF-8
Entire PowerShell script:
Add-Type -AssemblyName System.Web
$inputstring = "JXUyNzEzJTIwJUUwJTIwbGElMjBtb2Rl"
$bytes = [System.Convert]::FromBase64String($inputstring)
$utf8string = [System.Text.Encoding]::UTF8.GetString($bytes)
[System.Web.HttpUtility]::UrlDecode($utf8string) | Out-File -Encoding utf8 C:\temp\output.txt
[System.Web.HttpUtility]::UrlDecode($utf8string, [System.Text.UnicodeEncoding]::UTF7) | Out-File -Append -Encoding utf8 C:\temp\output.txt
Clarification:
The problem isn't the conversion of the Base64 to UTF-8. The problem is some inconsistent behavior of the UrlDecode of C#. If you run escape('✓ à la mode') in your browser you will end up with the following string %u2713%20%E0%20la%20mode. So we have a Unicode representation of the check mark and a HTML entity for the á. If we use this directly in UrlDecode we end up with the same error. My current assumption would be that it's an issue with the encoding of the PowerShell window and pasting characters into it.
Turns out it actually isn't all that strange. It's just for what I want to do it's advantages to use a newer function. I'm still not sure why it works if you use the UTF-7 encoding. But anyways, as an explanation:
... The hexadecimal form for characters, whose code unit value is 0xFF or less, is a two-digit escape sequence: %xx. For characters with a greater code unit, the four-digit format %uxxxx is used.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/escape
As TesselatingHecksler pointed out What is the proper way to URL encode Unicode characters? would indicate that the %u format wasn't formerly standardized. A newer version to escape characters exists though, which is encodeURIComponent.
The encodeURIComponent() function encodes a Uniform Resource Identifier (URI) component by replacing each instance of certain characters by one, two, three, or four escape sequences representing the UTF-8 encoding of the character (will only be four escape sequences for characters composed of two "surrogate" characters).
The output of this function actually works with the C# implementation of UrlDecode without supplying an additional encoding of UTF-7.
The original linked Mozilla article about a Base64 encode for an UTF-8 strings modifies the whole process in a way to allows you to just call the Base64 decode function in order to get the whole string. This is realized by converting the URL encode version of the string to bytes.
I'm trying to fetch data from PostgreSQL with Erlang.
Here's my code that gets data from DB. However i have cyrrilic data in 'status' column. This cyrrilic data is not being fetched correctly.
I tried using UserInfo = io_lib:format("~tp ~n",[UserInfoQuery]), however this doesn't seem to work, because it crashes the app.
UserInfoQuery = odbc_queries:get_user_info(LServer,LUser),
UserInfo = io_lib:format("~p",[UserInfoQuery]),
?DEBUG("UserInfo: ~p",[UserInfo]),
StringForUserInfo = lists:flatten(UserInfo),
get_user_info(LServer, Id) ->
ejabberd_odbc:sql_query(
LServer,
[<<"select * from users "
"where email_hash='">>, Id, "';"]).
Here's the data that is fetched from DB
{selected,[<<"username">>,<<"password">>,<<"created_at">>,
<<"id">>,<<"email_hash">>,<<"status">>],
[{<<"admin">>,<<"admin">>,<<"2014-05-13 12:40:30.757433">>,
<<"1">>,<<"adminhash">>,
<<209,139,209,132,208,178,208,176,209,139,209,132,208,
178,208,176>>}]}
Question:
How can i extract data from column? For example only data from
'status' column?
How can i extract data in unicode from DB? Should i fetch the data from DB then use
io_lib:format("~tp~n") on it? Is there any better way to do it?
Additional question: is there any way to get string in human readable format, so that StringForUserInfo = 'ыфваыфва' from RowUnicode?
I tried this:
{selected, _, [Row]} = UserInfoQuery,
RowUnicode = io_lib:format("~tp~n", [Row]),
?DEBUG("RowUnicode: ~p",[RowUnicode]),
StringForUserInfo = lists:flatten(RowUnicode),
Error:
bad argument in call to erlang:iolist_size([123,60,60,34,97,100,109,105,110,34,
62,62,44,60,60,34,97,100,109,105,110,34,62,62,44,60,60,34,50,...])
Erlang ODBC driver perfectly fetched the status column from your database. Indeed, PostgreSQL encodes your data is UTF-8, and the value you get is UTF-8 encoded.
Status = <<209,139,209,132,208,178,208,176,209,139,209,132,208,178,208,176>>.
This is a binary representing the string ыфваыфва in UTF-8.
You can directly use UTF-8 encoded binaries in your code. If you want to use unicode character points instead of UTF-8 bytes, you can convert this to a list of integers (a string in Erlang parlance). Just use unicode:characters_to_list/1, which in your case will yield list [1099,1092,1074,1072,1099,1092,1074,1072]. This is a list representation of the same string. Unicode character 1099 (16#044B in hex) is ы (CYRILLIC SMALL LETTER YERU, cf Cyrillic excerpt unicode chart).
Erlang can handle unicode texts in the two representations above: lists of unicode characters as integers and binaries of UTF-8 encoded characters.
Let's examine a smaller example, string "ы". This string is composed of unicode character 044B CYRILLIC SMALL LETTER YERU, and it can be encoded as a binary as <<209,139>> or as a list as [16#044B] (= [1099]).
Historically, lists of integers as well as binaries were Latin-1 (ISO-8859-1) encoded. Unicode and ISO-8859-1 have the same values from 0 to 255, but UTF-8 transformation only matches ISO-8859-1 for characters in the 0-127 range. For this reason, Erlang's ~s format argument has a unicode translation modifier, ~ts. The following line will not work as expected:
io:format("~s", [<<209,139>>]).
It will output two characters, 00D1 (LATIN CAPITAL LETTER N WITH TILDE) and 008B (PARTIAL LINE FORWARD). This is because <<209,139>> is interpreted as a Latin-1 string and not as a UTF-8 encoded string.
The following line will fail:
io:format("~s", [[1099]]).
This is because [1099] is not a valid Latin-1 string.
Instead, you should write:
io:format("~ts", [<<209,139>>]),
io:format("~ts", [[1099]]).
Erlang's ~p format argument also has a unicode translation modifier, ~tp. However, ~tp will not do what you are looking for alone. Whether you use ~p or ~tp, by default, io_lib:format/2 will format the Status UTF-8 encoded binary above as:
<<209,139,209,132,208,178,208,176,209,139,209,132,208,178,208,176>>
Indeed, t modifier only means the argument shall accept unicode input. If you do use ~p, when formatting a string or a binary, Erlang will determine whether this could be represented as a Latin-1 string since input may be Latin-1 encoded. This heuristic allows Erlang to properly distinguish lists of integers and strings, most of the time. To see the heuristic at work, you can try something like:
io:format("~p\n~p\n", [[69,114,108,97,110,103], [1,2,3,4,5,6]]).
The heuristic detects that [69,114,108,97,110,103] actually is "Erlang", while [1,2,3,4,5,6] is just, well, a list of integers.
If you do use ~tp, Erlang will expect strings or binaries to be unicode-encoded, and then apply the default identification heuristic. And default heuristic happens to currently (R17) be latin-1 as well. Since your string cannot be represented with Latin-1, Erlang will display it as a list of integers. Fortunately, you can switch to Unicode heuristics by passing +pc unicode to Erlang on command line, and this will produce what you are looking for.
$ erl +pc unicode
So a solution to your problem is to pass +pc unicode and to use ~tp.
I don't understand why io:format("~tp") doesn't work, but you can extract the row and column you need and print that with io:format("~ts"):
> {selected, _, [Row]} = UserInfoQuery.
> io:format("~ts~n", [element(6, Row)]).
ыфваыфва
ok
Assuming I have a binary
Message = <<"string containing emoji">>.
How do I properly encode it in Unicode? I tried doing:
Encoded = <<Message/utf16>>.
I get this warning when compiling the file:
Warning: binary construction will fail with a 'badarg' exception
(invalid Unicode code point in a utf8/utf16/utf32 segment)
I tried this with /utf8 as well. Same warning.
Assuming that the binary you start with is encoded according to UTF-8, and you need to encode it as little-endian UTF-16, this should work:
unicode:characters_to_binary(<<"string containing emoji">>, utf8, {utf16, little})
See the documentation for the Unicode module for more information.
The reason why <<Message/utf16>> fails is that the utf8, utf16 and utf32 specifiers in bit syntax encode a single codepoint, not an entire string. So to encode the character U+1F64C, you could use:
2> <<16#1f64c/utf8>>.
<<240,159,153,140>>
3> <<16#1f64c/utf16>>.
<<"\330=\336L">>
4> <<16#1f64c/utf32>>.
<<0,1,246,76>>
You may need to add -*- coding: utf8 -*- as the first line of your module, and use /utf8.
My guess is that you are using Erlang/OTP < 17, meaning files are considered latin-1 unless specified otherwise.
I have a normal string in Powershell that is from a text file containing Base64 text; it is stored in $x. I am trying to decode it as such:
$z = [System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String($x));
This works if $x was a Base64 string created in Powershell (but it's not). And this does not work on the $x Base64 string that came from a file, $z simply ends up as something like 䐲券.
What am I missing? For example, $x could be YmxhaGJsYWg= which is Base64 for blahblah.
In a nutshell, YmxhaGJsYWg= is in a text file then put into a string in this Powershell code and I try to decode it but end up with 䐲券 etc.
Isn't encoding taking the text TO base64 and decoding taking base64 BACK to text? You seem be mixing them up here. When I decode using this online decoder I get:
BASE64: blahblah
UTF8: nVnV
not the other way around. I can't reproduce it completely in PS though. See sample below:
PS > [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String("blahblah"))
nV�nV�
PS > [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes("nVnV"))
blZuVg==
EDIT I believe you're using the wrong encoder for your text. The encoded base64 string is encoded from UTF8(or ASCII) string.
PS > [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String("YmxhaGJsYWg="))
blahblah
PS > [System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String("YmxhaGJsYWg="))
汢桡汢桡
PS > [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String("YmxhaGJsYWg="))
blahblah
There are no PowerShell-native commands for Base64 conversion - yet (as of PowerShell [Core] 7.1), but adding dedicated cmdlets has been suggested in GitHub issue #8620.
For now, direct use of .NET is needed.
Important:
Base64 encoding is an encoding of binary data using bytes whose values are constrained to a well-defined 64-character subrange of the ASCII character set representing printable characters, devised at a time when sending arbitrary bytes was problematic, especially with the high bit set (byte values > 0x7f).
Therefore, you must always specify explicitly what character encoding the Base64 bytes do / should represent.
Ergo:
on converting TO Base64, you must first obtain a byte representation of the string you're trying to encode using the character encoding the consumer of the Base64 string expects.
on converting FROM Base64, you must interpret the resultant array of bytes as a string using the same encoding that was used to create the Base64 representation.
Examples:
Note:
The following examples convert to and from UTF-8 encoded strings:
To convert to and from UTF-16LE ("Unicode") instead, substitute [Text.Encoding]::Unicode for [Text.Encoding]::UTF8
Convert TO Base64:
PS> [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes('Motörhead'))
TW90w7ZyaGVhZA==
Convert FROM Base64:
PS> [Text.Encoding]::Utf8.GetString([Convert]::FromBase64String('TW90w7ZyaGVhZA=='))
Motörhead
This page shows up when you google how to convert to base64, so for completeness:
$b = [System.Text.Encoding]::UTF8.GetBytes("blahblah")
[System.Convert]::ToBase64String($b)
Base64 encoding converts three 8-bit bytes (0-255) into four 6-bit bytes (0-63 aka base64). Each of the four bytes indexes an ASCII string which represents the final output as four 8-bit ASCII characters. The indexed string is typically 'A-Za-z0-9+/' with '=' used as padding. This is why encoded data is 4/3 longer.
Base64 decoding is the inverse process. And as one would expect, the decoded data is 3/4 as long.
While base64 encoding can encode plain text, its real benefit is encoding non-printable characters which may be interpreted by transmitting systems as control characters.
I suggest the original poster render $z as bytes with each bit having meaning to the application. Rendering non-printable characters as text typically invokes Unicode which produces glyphs based on your system's localization.
Base64decode("the answer to life the universe and everything") = 00101010
If anyone would like to do it with a pipe in Powershell (like a filter) (e.g. read file contents and decode it), it can be achieved with a one-liner like that:
Get-Content base64.txt | %{[Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($_))}
I had issues with spaces showing in between my output and there was no answer online at all to fix this issue. I literally spend many hours trying to find a solution and found one from playing around with the code to the point that I almost did not even know what I typed in at the time that I got it to work. Here is my fix for the issue: [System.Text.Encoding]::UTF8.GetString(([System.Convert]::FromBase64String($base64string)|?{$_}))
Still not a "built-in", but published to gallery, authored by MS:
https://github.com/powershell/textutility
TextUtility
ConvertFrom-Base64
Return a string decoded from base64.
ConvertTo-Base64
Return a base64 encoded representation of a string.
I've tried everything Google and StackOverflow have recommended (that I could find) including using Encode. My code works but it just uses UTF8 and I get the wide character warnings. I know how to work around those warnings but I'm not using UTF8 for anything else so I'd like to just convert it and not have to adapt the rest of my code to deal with it. Here's my code:
my $xml = XMLin($content);
# Populate the #titles array with each item title.
my #titles;
for my $item (#{$xml->{channel}->{item}}) {
my $title = Encode::decode_utf8($item->{title});
#my $title = $item->{title};
#utf8::downgrade($title, 1);
Encode::from_to($title, 'utf8', 'iso-8859-1');
push #titles, $title;
}
return #titles;
Commented out you can see some of the other things I've tried. I'm well aware that I don't know what I'm doing here. I just want to end up with a plain old ASCII string though. Any ideas would be greatly appreciated. Thanks.
The answer depends on how you want to use the title. There are 3 basic ways to go:
Bytes that represent a UTF-8 encoded string.
This is the format that should be used if you want to store the UTF-8 encoded string outside your application, be it on disk or sending it over the network or anything outside the scope of your program.
A string of Unicode characters.
The concept of characters is internal to Perl. When you perform Encode::decode_utf8, then a bunch of bytes is attempted to be converted to a string of characters, as seen by Perl. The Perl VM (and the programmer writing Perl code) cannot externalize that concept except through decoding UTF-8 bytes on input and encoding them to UTF-8 bytes on output. For example, your program receives two bytes as input that you know they represent UTF-8 encoded character(s), let's say 0xC3 0xB6. In that case decode_utf8 returns a representation that instead of two bytes, sees one character: ö.
You can then proceed to manipulate that string in Perl. To illustrate the difference further, consider the following code:
my $bytes = "\xC3\xB6";
say length($bytes); # prints "2"
my $string = decode_utf8($bytes);
say length($string); # prints "1"
The special case of ASCII, a subset of UTF-8.
ASCII is a very small subset of Unicode, where characters in that range are represented by a single byte. Converting Unicode into ASCII is an inherently lossy operation, as most of the Unicode characters are not ASCII characters. You're either forced to drop every character in your string which is not in ASCII or try to map from a Unicode character to their closest ASCII equivalents (which isn't possible in the vast majority of cases), when trying to coerce a Unicode string to ASCII.
Since you have wide character warnings, it means that you're trying to manipulate (possibly output) Unicode characters that cannot be represented as ASCII or ISO-8859-1.
If you do not need to manipulate the title from your XML document as a string, I'd suggest you leave it as UTF-8 bytes (I'd mention that you should be careful not to mix bytes and characters in strings). If you do need to manipulate it, then decode, manipulate, and on output encode it in UTF-8.
For further reading, please use perldoc to study perlunitut, perlunifaq, perlunicode, perluniintro, and Encode.
Although this is an old question, I just spent several hours (!) trying to do more or less the same thing! That is: read data from a UTF-8 XML file, and convert that data into the Windows-1252 codepage (I could also have used Latin1, ISO-8859-1 etc.) in order to be able to create filenames with accented letters.
After much experimentation, and even more searching, I finally managed to get the conversion working. The "trick" is to use Encode::encode instead of Encode::decode.
For example, given the code in the original question, the correct (or at least one :-) way to convert from UTF-8 would be:
my $title = Encode::encode("Windows-1252", $item->{title});
or
my $title = Encode::encode("ISO-8859-1", $item->{title});
or
my $title = Encode::encode("<your-favourite-codepage-here>", $item->{title});
I hope this helps others having similar problems!
You can use the following line to simply get rid of the warning. This assumes that you want to use UTF8, which shouldn't normally be a problem.
binmode(STDOUT, ":encoding(utf8)");