I am using mitmproxy and want to record every request and reponse to file,so I use "-w" option just as following:
mitmproxy -b 192.168.1.107 -p 9527 -w ~/Desktop/aaa.txt
but when I open the 'aaa.txt',it display unreadable content which is just as following:
[x§‡:ÖáHi4GÐL¿¤Ìé4Îæyùͧq¼<µYÂ&É‹¶Mñ+GÒ‡i8
avÅÆdT£ <_‰»ÚÀ—æÏÂÓSòo“çˆ$B6KƒßÛVÚ¼rq{”2w.®NÉRhÔ…x)¥qÕ¾0‡8éÙOøóŸüÍ—òÛ_þãnñ—‡"Ä‚NqiŠ¬#JÔî"œE§"CJ&0‡Í*NCBé r:G£O1yùè“æRQB4
I also try the script:https://github.com/mitmproxy/mitmproxy/blob/master/examples/flowwriter.py
it still doesn't work, so is there some encoding error?
mitmproxy -w writes a serialized (not primarily human-readable) dump file that can be read again using -r. If the content of a message are e.g. gzip-encoded, you'll see gzip-encoded data in the dumpfile. If you want human-readable output to a text file, I'd suggest running
mitmdump -r ~/Desktop/aaa.txt -n -dd
Explanation:
-r: Read an existing dump file
-n: Do not start a proxy server
-d: increase output details/verbosity (-ddd if you don't want contents to be cut off)
Related
I am trying to get tshark to show the payload of packages in clear text and I am failing in that. Which makes me really sad.
tshark -r pcap -T fiels -e data (or data.text) gives blank results.
tshark -r pcap -T fiels -e tcp.payload gives hex results.
So the question is how do I get tshark to show the tcp.payload als text (ascii).
I would like to know that is there any option in Wireshark to capture packets in the streaming network dynamically. Since I need to capture packets with out doing the export each time and packets capture automatically in a CSV file without exporting it periodically. Thanks.
You should be able to use tshark to achieve this. For example, suppose you want to capture the frame number and source and destination IP addresses of each packet (to keep the example simple), you could use:
tshark -i foo -T fields -E separator=, -E quote=d -E header=y -Y ip -e frame.number -e ip.src -i ip.dst > output.csv
You can specify as many fields as you want using the -e option
I have a bunch of pcap files that I got with tcpdump. I need to search through all of them for specific keywords and record which files contain these strings. Is there a way to automate the search for these keywords using a tcpdump command perhaps?
Probably the most generic solution using tshark would be to run something like:
tshark -r file.pcap -Y "frame contains foo"
... where foo is the string you're searching for. Refer to the wireshark-filter man page for more information on filtering using the contains and other operators, such as the matches operator which supports Perl compatible regular expressions.
Using that command, the output you'll see will be a 1-line summary of each packet matching the filter. You could tailor the output using a number of methods, but for example, suppose you only wanted to know the frame number of the matching packet, you could run:
tshark -r file.pcap -Y "frame contains foo" -T fields -e frame.number
Refer to the tshark man page for more information on the -T and -e options, as well as other options which may be of use to you.
There is more powerful version of tcpdump, tshark (it is the command line tool from wireshark package). You could use tshark -T fields|pdml|ps|psml|text to dump packets in format you like, and just grep it. tshark could read tcpdump dumps.
What does perl's -T function really do? From the man page on perlfunc:
-T File is an ASCII text file (heuristic guess).
-B File is a "binary" file (opposite of -T).
Is the -B option simply equivalent to ! -T, or is it simply an inversion of the heuristic, such that some of the time, a file may be true for both -B and -T. Does the heuristic have, say, a threshold for control characters? Does it ignore tabs, EOLs, EOFs and NULs?
From the same page:
The -T and -B switches work as follows.
The first block or so of the file is examined to see if it is valid UTF-8 that includes non-ASCII characters. If, so it's a -T file. Otherwise, that same portion of the file is examined for odd characters such as strange control codes or characters with the high bit set. If more than a third of the characters are strange, it's a -B file; otherwise it's a -T file. Also, any file containing a zero byte in the examined portion is considered a binary file. (If executed within the scope of a use locale which includes LC_CTYPE , odd characters are anything that isn't a printable nor space in the current locale.) If -T or -B is used on a filehandle, the current IO buffer is examined rather than the first block. Both -T and -B return true on an empty file, or a file at EOF when testing a filehandle. Because you have to read a file to do the -T test, on most occasions you want to use a -f against the file first, as in next unless -f $file && -T $file .
I have a directory containing a bunch of files, some text some binary, with no consistent naming. I want to search and replace a string in text files only. So I went with:
perl -i -pne 's#/some/text/to/replace#/replacement/text#' *
Remove the -i option and you will see that binary files get caught. How do I modify this one-liner to skip binary files?
ack -n --text --sort -f . | xargs perl -i -pne 's…'
Abusing ack goes much quicker than writing your own solution with -T.
Well, this is all based on what your definition of a text file is. Perl 5 has the -T filetest operator that will tell you if a filename or filehandle is a text file (using Perl 5's definition):
perl -i -pne 'BEGIN{#ARGV=grep-T,#ARGV}s#regex#replacement#' *
The BEGIN block will filter out any files that don't pass the -T test, so they won't even be read (except for their first block because that is what -T uses to determine if they are text).
From perldoc -f -X
The -T and -B switches work as follows. The first block or so of the file is examined for odd characters such as strange control codes or characters with the high bit set. If too many strange characters (>30%) are found, it's a -B file; otherwise it's a -T file. Also, any file containing a zero byte in the first block is considered a binary file. If -T or -B is used on a filehandle, the current IO buffer is examined rather than the first block. Both -T and -B return true on an empty file, or a file at EOF when testing a filehandle. Because you have to read a file to do the -T test, on most occasions you want to use a -f against the file first, as in next unless -f $file && -T $file .