I am having difficulty trying to understand some data.
I have a Perl Script and all I know about that script is that it is sending some data packets over the network.
When I debug through the script the data that it sends looks like following: "KFD!P#"
I am very new to Perl and all I know is that it should be a Byte[]. Should I not see something like \dsdsds \dssds if it is a byte array?
Is this string represented in any expression that I am not able to understand?
Any ideas?
To print the contents of a perl string that is being used as a buffer, you need to convert it to a printable form first. Use unpack for that.
For example to convert it to a string of hex digits:
print unpack('H*', $buffer),"\n";
Read perlpacktut to learn more.
Run wireshark to see what it is putting over the wire.
Any string can be considered a byte array so you won't need any keywords like "byte" in the code.
Sharing the code somewhere or some portion of it would provide more context to work with and address your question.
Related
I have $stringF. Contained within $stringF is the following (the string is all one line, not word-wrapped as below):
http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=
AFQjCNHWQk0M4bZi9xYO4OY4ZiDqYVt2SA&clid=
c3a7d30bb8a4878e06b80cf16b898331&cid=52779892300270&ei=
H4IAW6CbK5WGhQH7s5SQAg&url=https://abcnews.
go.com/Lifestyle/wireStory/latest-royal-wedding-thousands-streets-windsor-55280649
I want to locate that string and make it look like this:
https://abcnews.go.com/Lifestyle/wireStory/latest-royal-
wedding-thousands-streets-windsor-55280649
Basically I need to use preg_replace to find the following string:
http://news.google.com/news/url?sa= ***SOME UNKNOWN CONTENT*** &url=http
and replace it with the following string:
http
I'm a little rusty with my php, and even rustier with regular expressions, so I'm struggling to figure this one out. My code looks like this:
$stringG = preg_replace('http://news.google.com/news/url?sa=*&url=http','http',$stringH);
except I know I can't use wildcards and I know I need to specially deal with the special characters (colon, forward slash, question mark, and sign, etc). Hoping someone can help me out here.
Also of note is that my $stringF contains multiple instances of such strings, so I need the preg_replace to be not greedy - otherwise it will replace a huge chunk of my string unnecessarily.
PHP has tools for that, no need to use a regex. parse_url to get the components of an url (scheme, host, path, anchor, query, ...) and parse_str to get the keys/values of the query part.
$url = 'http://news.google.com/news/url?sa=t&fd=R&ct2=us&usg=AFQjCNHWQk0M4bZi9xYO4OY4ZiDqYVt2SA&clid=c3a7d30bb8a4878e06b80cf16b898331&ci=52779892300270&ei=H4IAW6CbK5WGhQH7s5SQAg&url=https://abcnews.go.com/Lifestyle/wireStory/latest-royal-wedding-thousands-streets-windsor-55280649';
parse_str(parse_url($url, PHP_URL_QUERY), $arr);
echo $arr['url'];
I'm trying to upload a zip by post request. To do this I need to send string and byte data.
Ok the problem is the binary encoding still. The original zip encoding first line is PK........,Q?E?;, where as the uploaded copy is PK........,Q.E.; The original has ? whereas the copy converts them to ..
Convert it to a string:
[System.Text.Encoding]::ASCII.GetString([byte]$file)
More readable:
$byteArray = [byte]$file
$enc = [System.Text.Encoding]::ASCII
$stringContents = $enc.GetString($byteArray)
You can use other encodings if the content of the file is actually in Unicode or some other encoding.
To see the other encodings:
[System.Text.Encoding] | Get-Member -Type Properties -Static
Edit based on comment:
I misunderstood your question. You are concerned that the bytes are being converted to the ASCII representation of the byte's numerical value.
This is not actually what is happening. When you use Write-Host, it is then that the conversion is happening, for display purposes only.
The values in the byte array are the raw binary data.
Edit based on your edit:
You are embedding the byte array $bytes in another string. This is implicitly converting it to a display-able format, which is not what you want.
$bytes does contain the raw data. In this case, what you want is for each byte to be placed directly in the string as is. You should try using the first solution I posted, and then embed $stringContents instead of $bytes in $parameters. I think this will do what you are hoping for.
Don't confuse what's printed on the screen when you display a string with what it actually contains. Although a [String] object can be more complex, at its simplest a string is an array of bytes. In C for example, you don't declare strings, you declare an array of char (which is usually a byte).
I urge you to try the method whereby you convert it to a string and comment if it's not working.
How does one calculate the info_hash parameter? Aka the hash corresponding to the info dictionar??
From official specs:
info_hash
The 20 byte sha1 hash of the bencoded form of the info value from the metainfo file. Note that this is a substring of the metainfo file.
This value will almost certainly have to be escaped.
Does this mean simply get the substring from the meta-info file and do a sha-1 hash on the reprezentative bytes??
.... because this is how i tried 12 times but without succes meaning I have compared the resulting hash with the one i should end up with..and they differ ..that + tracker response is FAILURE, unknown torrent ...or something
So how do you calculate the info_hash?
The metafile is already bencoded so I don't understand why you encode it again?
I finally got this working in Java code, here is my code:
byte metaData[]; //the raw .torrent file
int infoIdx = ?; //index of 'd' right after the "4:info" string
info_hash = SHAsum(Arrays.copyOfRange(metaData, infoIdx, metaData.length-1));
This assumes the 'info' block is the last block in the torrent file (wrong?)
Don't sort or anything like that, just use a substring of the raw torrent file.
Works for me.
bdecode the metafile. Then it's simply sha1(bencode(metadata['info']))
(i.e. bencode only the info dict again, then hash that).
I decided to use Python 3 for making my website, but I encountered a problem with Unicode output.
It seems like plain print(html) #html is astr should be working, but it's not. I get UnicodeEncodeError: 'ascii' codec can't encode characters[...]: ordinal not in range(128). This must be because the webserver doesn't support unicode output.
The next thing I tried was print(html.encode('utf-8')), but I got something like repr output of the byte string: it is placed inside b'...' and all the escape characters are in raw form (e.g. \n and \xd0\x9c)
Please show me the correct way to output a Unicode (str) string as a raw UTF-8 encoded bytes string in Python 3.1
The problem here is that you stdout isn't attached to an actual terminal and will use the ASCII encoding by default. Therefore you need to write to sys.stdout.buffer, which is the "raw" binary output of sys.stdout. This can be done in various ways, the most common one seems to be:
import codecs, sys
writer = codecs.getwriter('utf8')(sys.stdout.buffer)
And the use writer. In a CGI script you may be able to replace sys.stdout with the writer so:
sys.stdout = codecs.getwriter('utf8')(sys.stdout.buffer)
Might actually work so you can print normally. Try that!
I've tried everything Google and StackOverflow have recommended (that I could find) including using Encode. My code works but it just uses UTF8 and I get the wide character warnings. I know how to work around those warnings but I'm not using UTF8 for anything else so I'd like to just convert it and not have to adapt the rest of my code to deal with it. Here's my code:
my $xml = XMLin($content);
# Populate the #titles array with each item title.
my #titles;
for my $item (#{$xml->{channel}->{item}}) {
my $title = Encode::decode_utf8($item->{title});
#my $title = $item->{title};
#utf8::downgrade($title, 1);
Encode::from_to($title, 'utf8', 'iso-8859-1');
push #titles, $title;
}
return #titles;
Commented out you can see some of the other things I've tried. I'm well aware that I don't know what I'm doing here. I just want to end up with a plain old ASCII string though. Any ideas would be greatly appreciated. Thanks.
The answer depends on how you want to use the title. There are 3 basic ways to go:
Bytes that represent a UTF-8 encoded string.
This is the format that should be used if you want to store the UTF-8 encoded string outside your application, be it on disk or sending it over the network or anything outside the scope of your program.
A string of Unicode characters.
The concept of characters is internal to Perl. When you perform Encode::decode_utf8, then a bunch of bytes is attempted to be converted to a string of characters, as seen by Perl. The Perl VM (and the programmer writing Perl code) cannot externalize that concept except through decoding UTF-8 bytes on input and encoding them to UTF-8 bytes on output. For example, your program receives two bytes as input that you know they represent UTF-8 encoded character(s), let's say 0xC3 0xB6. In that case decode_utf8 returns a representation that instead of two bytes, sees one character: รถ.
You can then proceed to manipulate that string in Perl. To illustrate the difference further, consider the following code:
my $bytes = "\xC3\xB6";
say length($bytes); # prints "2"
my $string = decode_utf8($bytes);
say length($string); # prints "1"
The special case of ASCII, a subset of UTF-8.
ASCII is a very small subset of Unicode, where characters in that range are represented by a single byte. Converting Unicode into ASCII is an inherently lossy operation, as most of the Unicode characters are not ASCII characters. You're either forced to drop every character in your string which is not in ASCII or try to map from a Unicode character to their closest ASCII equivalents (which isn't possible in the vast majority of cases), when trying to coerce a Unicode string to ASCII.
Since you have wide character warnings, it means that you're trying to manipulate (possibly output) Unicode characters that cannot be represented as ASCII or ISO-8859-1.
If you do not need to manipulate the title from your XML document as a string, I'd suggest you leave it as UTF-8 bytes (I'd mention that you should be careful not to mix bytes and characters in strings). If you do need to manipulate it, then decode, manipulate, and on output encode it in UTF-8.
For further reading, please use perldoc to study perlunitut, perlunifaq, perlunicode, perluniintro, and Encode.
Although this is an old question, I just spent several hours (!) trying to do more or less the same thing! That is: read data from a UTF-8 XML file, and convert that data into the Windows-1252 codepage (I could also have used Latin1, ISO-8859-1 etc.) in order to be able to create filenames with accented letters.
After much experimentation, and even more searching, I finally managed to get the conversion working. The "trick" is to use Encode::encode instead of Encode::decode.
For example, given the code in the original question, the correct (or at least one :-) way to convert from UTF-8 would be:
my $title = Encode::encode("Windows-1252", $item->{title});
or
my $title = Encode::encode("ISO-8859-1", $item->{title});
or
my $title = Encode::encode("<your-favourite-codepage-here>", $item->{title});
I hope this helps others having similar problems!
You can use the following line to simply get rid of the warning. This assumes that you want to use UTF8, which shouldn't normally be a problem.
binmode(STDOUT, ":encoding(utf8)");