Is there a way to convert MessagePack response on Charles/MITM to JSON?
I have a response from https://[...]/application/x-msgpack
Raw mode reads it:
xdd\xb8\xbd\xc9? \x03\xa2Xe\xccO\xc2O\xd6"\x06\x91\xcfB\x9c\xed\x0fl
Hex mode reads it:
0000000010 06 91 cf 42 9c ed 0f 6c 2e 14 ae f1 da 2d 34 e9 ...B...l.....-4.
Hex mode contains contains nonsense at the end of every line.
Other modes could not parse and fall back to Raw mode.
If I fail to give you any other info, let me know.
If you can decode it in Python (e.g. using Kaitai Struct), you can implement a mitmproxy contentview to display it in a human-readable way:
https://github.com/mitmproxy/mitmproxy/blob/master/examples/simple/custom_contentview.py
https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/contentviews
(if you go the kaitai struct route - we'd also be more than happy for a PR!)
Related
I am a new to linux/u-boot and just generally reading and writing to specific registers.
I have a KSZ9897 Ethernet switch, and my goal is to use u-boot i2c commands to set bits [15:13] to 001 to put the switch in the proper test mode.
The following screenshot is of the register that I am trying to read/write to:
KSZ9897 datasheet
The slave address of this part is 1011_111 = 0x5F
The address of the 16 bit register I am trying to read/write from is [0xN112-0xN113]
The PHY register is 0x09 (I am not sure how this fits in to the picture, but that may be the reason I am having issues)
using u-boot, I am sending the following command:
#i2c md 0x5F 0xN112.2 0x10
It reads back the following:
0000: 00 98 97 00 00 00 00 00 00 00 00 00 00 00 00 4c ...............L
My suspicion is that this is just reading the first 16 addr values starting at 0x0000. I know this because the 98 and 97 values match up with the default values that should be at address 0x0001 and 0x0002.
Can anyone tell me what I am doing wrong here and how I can access register 0x09? I think i am using the proper u-boot syntax, but clearly something is not right here. any help would be much appreciated.
The i2c command is wrong, because the third parameter must be a hexadecimal number and N is not hexa, the switch datasheet tells that:
Address field “N” specifies the portnumber. Valid values for “N” are 1 to 7 for some registers, 6 to 7 for MAC port specific registers, and 0 to 5 for PHYspecific registers.
You should be writing something like this: i2c md 0x5f 0x1112 1 , the last parameter is the number of bytes you read.
I am not sure I've got my nomenclature right, so please correct me :)
I've received a text file representing a Pāli dictionary: a list of words separated by newline \n (0x0a) characters. Supposedly, some of the special letters are encoded using UTF-8, but I doubt that.
Loading this text file into any of my editors (vim, Notepad, TextEdit, ..) shows quite scrambled text, for example
mhiti
A closer look at the actual bytes then reveal the following (using hexdump -C)
0a 0a 1e 6d 68 69 74 69 0a 0a ...mhiti..
which seems to me the Unicode code point U+1E6D ("ṭ" or LATIN SMALL LETTER T WITH DOT BELOW). That particular letter has UTF-8 encoding e1 b9 ad.
My question: is there a tool which helps me convert this particular file into actual UTF-8 encoding? I tried iconv but without success; I looked briefly into a Python script but would think there's an easier way to get this done. It seems that this is a useful link for this problem, but isn't there a tool that can get this done? Am I missing something?
EDIT: Just to make things a little bit more entertaining, there seem to be actual UTF-8 encoded characters scattered throughout as well. For example, the word "ākiñcaññāyatana" has the following sequence of bytes
01 01 6b 69 c3 b1 63 61 c3 b1 c3 b1 01 01 79 61 74 61 6e 61
ā k i ñ c a ñ ñ ā y a t a n a
where the "ā" is encoded by its Unicode code point U-0101, and the "ñ" is encoded by the UTF-8 sequence \xc3b1 which has Unicode code point U-00F1.
EDIT: Here's one that I can't quite figure out what it's supposed to be:
01 1e 37 01 01 76 61 6b 61
? ā v a k a
I can only guess, but that too doesn't make sense. The Unicode code point U+011e is a "Ğ" (UTF-8 \xc49e) but that's not a Pāli character AFAIK; then a "7" follows which doesn't make sense in a word. Then the Unicode code point U+1E37 is a "ḷ" (UTF-8 \xe1b8b7) which is a valid Pāli character. But that would leave the first byte \x01 by itself. If I had to guess I would think this is the name "Jīvaka" but that would not match the bytes. LATER: According to the author, this is "Āḷāvaka" — so assuming the heuristics of character encoding from above, again a \x00 is missing. Adding it back in
01 00 1e 37 01 01 76 61 6b 61
Ā ḷ ā v a k a
Are there "compressions" that remove \x00 bytes from UTF-16 encoded Unicode files?
I'm assuming in this context that "ṭhiti" makes sense as the contents of that file.
From your description, it looks like that file encodes characters < U+0080 as a single byte and characters > U+0100 as two-byte big-endian. That's not decodable, in general; two linefeeds (U+000A, U+000A) would have the same encoding as GURMUKHI LETTER UU (U+0A0A).
There's no invocation of iconv that'll decode it for you; you'll either need to take the heuristics you know, either based on character ranges or ordering in the file, to write a custom decoder (or ask for another copy in a standard encoding).
I think in the end this was my own fault, somehow. Browsing to this file showed a very mangled and broken version of the original UTF-16 encoded file; the "Save as" menu from the browser then saved that broken file which created the initial question for this thread.
It seems that a web browser tries to display that UTF-16 encoded file, removes non-printable characters like \x00 and converts some others to UTF-8, thus completely mangling the original file.
Using wget to fetch the file fixed the problem, and I could convert it nicely into UTF-8 and use it further.
I am attempting to read constant (or initilization) strings from an iPhone Mach-O binary file. I understand that the 3 relevant segment.sections are _TEXT._cstring _TEXT._ustring and _DATA._cfstring. Howver, even though I know the string information is stored in these three blocks of data, which I have extracted, I can not make any sense of it, and it all looks like garbage - I do not see any recognizable character strings. Can anyone shed some light on this and give me an idea of what steps need to be take to read the string data?
I have looked at some code (GetAddrOfConstantCFString() from http://llvm.org/svn/llvm-project/cfe/trunk/lib/CodeGen/CodeGenModule.cpp), but again, couldn't quite relate it to what I see in the binaries.
In my case the sizes of the sections in question are:
__TEXT.__cstring (99 K-bytes)
__TEXT.__ustring (<200 bytes)
__DATA.__cfstring (29 K-bytes)
To give you an idea, the first 32 bytes of the __cfstring section, which I though would contain the actual strings looks like this:
Dump _DATA._cfstring
00 00 00 00 c8 07 00 00 74 02 0d 00 15 00 00 00
00 00 00 00 c8 07 00 00 8c 02 0d 00 01 00 00 00
...
Thanks a lot for your help!
Well, I've found the answer.
1) the files are generally encrypted (this can be tested with otool -l prog_file|grep -i crypt ). Not all sections are encrypted but usually the first block including _TEXT._text (prog code) and _TEXT._cstring are. The _DATA._cfstring section was not encrypted in my case.
2) as expected __cfstring consists of 16-byte structures (NSConstantString), where the 3rd word is a pointer to memory where _TEXT._cstring is loaded. The 4th word is the length.
So in real life the trick is to decrypt the file first, and then all is visible and accessible. I still didn't get around doing it properly, but dumped a piece of memory in gdb, which then replaced the relevant section in the file.
In my iPhone App I have the capability to export data to an .txt file and send it via Mail – but have currently problems regarding the encoding.
To attach the file to mail I simply create a NSData instance from a NSString instance as followed:
NSData *dataToExport = [[NSData alloc] initWithData:[myExportString dataUsingEncoding:NSUTF8StringEncoding]];
[[NSFileManager defaultManager] createFileAtPath:pathToExportFile contents:dataToExport attributes:nil];
But this finally results in a file with content like that (wrong encoding):
√úberraschung
Instead using
[myExportString writeToFile:pathToExportFile atomically:NO encoding:NSUTF8StringEncoding error:&error];
... creates a file with the correct enconding:
Überraschung
Any ideas, how I'm able to get the right encoding with the first way?
Thank you,
Florian
NSString's writeToFile:atomically:encoding:error: method sets an extended attribute on the output file identifying the encoding. TextEdit checks for this attribute and uses that encoding when it's present.
Since you don't (and can't) tell NSFileManager's method what encoding to use (since it's for plain data, not necessarily text encoded as data), it won't set this attribute. Without it, TextEdit will guess, and it does so quite poorly.
I don't know whether MobileMail will preserve the extended attribute somehow. Try it both ways, and when you receive the messages, check their raw contents and see whether one of them specifies the file's encoding. At any rate, your hex dumps prove that the text is being encoded correctly either way.
More generally, when you want to write out text to a file, NSString's methods are the correct solution.
That's strange. Here ist the hexdump -C output.
"Wrong" encoding:
00000000 c3 9c 62 65 72 72 61 73 63 68 75 6e 67 |..berraschung|
0000000d
"Right" encoding:
00000000 c3 9c 62 65 72 72 61 73 63 68 75 6e 67 |..berraschung|
0000000d
I'm also trying different applications to view the documents:
TextEdit: wrong
Excel: wrong
SubEthaEdit: right
Numbers: right
Very strange,right?
I have a problem … a very peculiar one could you please guide.
Original message: Kevätsunnuntaisin lentää
The flow of data is HttpConnector -> WSDLConnector -> to the underlying system
The following is the encoding of the first 7 characters
4b 65 76 c3 a4 74 73 75 – In Http Connector – the request XML has UTF-8 encoding
4b 65 76 a3 74 73 75 – in WSDL Connector -
InputSource inputSource = new InputSource(myInputStream);
inputSource.setEncoding("UTF-8");
parser.parse(inputSource);
The original string gets converted to Kev£tsunnuntaisin lent££.Also, there is a loss of a byte.
Could you please guide me where I am going wrong? What must I do to avoid this character conversion?
Thanks for your help!!!
This is very simple: The data in myInputStream is not encoded as UTF-8, hence the decoding fails.
My guess is that you save the output of the HTML connector as a string and then use that as the input for the WSDL connector. In the string, the data is unicode, not UTF-8. Use String.getBytes('UTF-8') to get an array of bytes with the correct encoding.
As for all encoding issues: Always tell the computer with which encoding it should work instead of hoping that it will guess correctly. Bytes have no encoding and the computer is not telepathic :) And I hope it never will be ...