I want to trace the http requests a program on my computer is making, actually what I really want it to find what it is setting as the useragent. How do I do this, a solution for Windows or OSX would be fine.
The tool to use is Wireshark
If you use firefox, there is the User Agent Switcher extension:
https://addons.mozilla.org/en-US/firefox/addon/user-agent-switcher/
Wireshark was mentioned, but I find Fiddler better fit for the task. It presents captured data more accessible than Wireshark.
Related
I have a script that I can configure to either use a socket library or use libcurl. I am using http access methods. When I use the socket library each request takes much longer as compared to when I switch to libcurl. Both of them work though.
I am curious to understand the difference. Is there a way I can look at the raw request and response for the network communication so I can investigate this? Something like what the browser shows for each page request would be ideal.
I am on a windows system. But if there is a software that can work on both windows and linux that would be great. Any help on how I can dig into this would be helpful.
Thanks.
I'm trying to write a sort-of web proxy but to make sure everything is working properly, I want to monitor an existing proxy server to see when the browser connects, disconnects, etc.
Are there any plugins for Firefox or Chrome (or Internet Explorer), to show me how many sockets it currently has open and what data was sent through them?
I know that I can use netstat to see a bit of information on socket state, but I'd really like to go a bit further and see what traffic was inside each socket specifically, etc.
Use a packet sniffer such as Wireshark, or a debugger such as Fiddler.
Use Microsoft Network Monitor. It's pretty easy to use. Once you capture packets, filter them just writing HTTP in the filter panel. In the left size, you will see a list of applications, choose Firefox, Chrome, IE or whatever you need, and it will display only packets sent and received by that application.
Fiddler is a great tool for testing a wide variety of http scenarios. However Fiddler also blocks all kinds of traffic that it shouldn't. This intrusive behavior can be annoying and time consuming.
Is there a way to specify what Fiddler does and does not listen to? So I want to say only monitor and report on traffic going to localhost or www.google.com, everything else would be ignored.
Is that possible ?
Its just rather tedious to have to close fiddler whenever it blocks something it should not.
Fiddler acts as a system proxy, so you can't have some traffic flow through it and some not, unless you can configure the client to not use the proxy for some connections.
You can use Fiddler's filter options to determine what is captured and shown in the sessions window.
However, what does it block ? I use Fiddler extensively, and have no issues with connections being blocked. Perhaps there is another solution than closing Fiddler.
I faced the same problem when setting up Fiddler just now (Win7). My problem was that Fiddler (Fiddler2) was not detecting and using my corporate Proxy settings. I had to go and set them manually in Fiddler Options > Gateway
That is why all my non-local, non-intranet traffic was appearing to blackhole. Hope that helps.
So I want to make my own Internet Filter. Don't worry, i'm not asking for a tutorial. I'm just wondering if Fiddler would make a good backbone for it. I'm a little worried because it seemed that there's a few things Fiddler can't always pick up - or that there are workarounds. So, my question:
Would Fiddler grab all web data? i.e, chats, emails, websites, etc.
Are there any known workarounds?
Any other reasons not to use it?
Thanks!
I think you mean FiddlerCore rather than Fiddler. Fiddler(Core) is a web proxy meaning it captures HTTP/HTTPS traffic; it won't capture traffic that uses other protocols (e.g. IRC, etc). To capture traffic from other protocols, you'll need a lower-level interception point (e.g. a Windows Firewall filter) which will capture everything, but it will not be able to decrypt HTTPS traffic, and parsing / modifying the traffic will prove MUCH harder.
I am trying to build an application that can inform a user about website specific information whenever they are visiting a website that is present in my database. This must be done in a browser independent way so the user will always see the information when visiting a website (no matter what browser or other tool he or she is using to visit the website).
My first (partially successful) approach was by looking at the data packets using the System.Net.Sockets.Socket class etc. Unfortunately I discoverd that this approach only works when the user has administrator rights. And of course, that is not what I want. My goal is that the user can install one relatively simple program that can be used right away.
After this I went looking for alternatives and found a lot about WinPcap and some of it's .NET wrappers (did I tell you I am programming c# .NET already?). But with WinPcap I found out that this must be installed on the user's pc and there is nog way to just reference some dll files and code away. I already looked at including WinPcap as a prerequisite in my installer but that is also to cumbersome.
Well, long story short. I want to know in my application what website my user is visiting at the moment it is happening. I think it must be done by looking at the data packets of the network but can't find a good solution for this. My application is build in C# .NET (4.0).
You could use Fiddler to monitor Internet traffic.
It is
a Web Debugging Proxy which logs all HTTP(S) traffic between your computer and the Internet. Fiddler allows you to inspect traffic, set breakpoints, and "fiddle" with incoming or outgoing data. Fiddler includes a powerful event-based scripting subsystem, and can be extended using any .NET language.
It's scriptable and can be readily used from .NET.
One simple idea: Instead of monitoring the traffic directly, what about installing a browser extension that sends you the current url of the page. Then you can check if that url is in your database and optionally show the user a message using the browser extension.
This is how extensions like Invisible Hand work... It scans the current page and sends relevant data back to the server for processing. If it finds anything, it uses the browser extension framework to communicate those results back to the user. (Using an alert, or a bar across the top of the window, etc.)
for a good start, wireshark will do what you want.
you can specify a filter to isolate and view http streams.
best part is wireshark is open source, and built opon another program api, winpcap which is open source.
I'm guessing this is what you want.
capture network data off the wire
view the tcp traffic of a computer, isolate and save(in part or in hole) http data.
store information about the http connections
number 1 there is easy, you can google for a winpcap tutorial, or just use some of their sample programs to capture the data.
I recomend you study up on the pcap file format, everything with winpcap uses this basic format and its structers.
now you have to learn how to take a tcp stream and turn it into a solid data stream without curoption, or disorginized parts. (sorry for the spelling)
again, a very good example can be found in the wireshark source code.
then with your data stream, you can simple read the http format, and html data, or what ever your dealing with.
Hope that helps
If the user is cooperating, you could have them set their browser(s) to use a proxy service you provide. This would intercept all web traffic, do whatever you want with it (look up in your database, notify the user, etc), and then pass it on to the original location. Run the proxy on the local system, or on a remote system if that fits your case better.
If the user is not cooperating, or you don't want to make them change their browser settings, you could use one of the packet sniffing solutions, such as fiddler.
A simple stright forward way is to change the comupter DNS to point to your application.
this will cause all DNS traffic to pass though your app which can be sniffed and then redirected to the real DNS server.
it will also save you the hussel of filtering out emule/torrent traffic as it normally work with pure IP address (which also might be a problem as it can be circumvented by using IP address to browse).
-How to change windows DNS Servers
-DNS resolver
Another simple way is to configure (programmaticly) the browsers proxy to pass through your server this will make your life easier but will be more obvious to users.
How to create a simple proxy in C#?