I am using WWW::Mechanize module in perl to create a script which goes to another website and downloads data. When I manually go to website and download data (using firefox browser) I can capture the packets in well-readable format using http-fox addon.
But, when I run perl script doing same thing and try to capture packets using Wireshark I can not understand how to understand the captured packets. All the information which was available in above diagram (firefox) is missing in wireshark.
Also, when I click "follow-stream". I get dotted text. Like this:
How to interpret this? May be the data is encrypted. In this case how to get the RSA key?
May be the data is encrypted
The packets in the TCP connection before packet 8 are to or from the "https" port (port 443), and packets 5 and 6 are a "TLSv1 Client Hello" and a "TLSv2 Server Hello", respectively, so this is HTTP-over-TLS traffic, which is likely to be encrypted.
Your browse is probably logging decrypted HTTP-over-SSL traffic in the http-fox module, but Wireshark is capturing the traffic going on the network, which is encrypted traffic.
See the Wireshark Wiki page on SSL for some information on how to have Wireshark decrypt that traffic in some cases.
Related
We are facing a technical issue with networking, which I'm not able to comprehend.
We are using Linphone Client to connect to x.x.x.125(Freeswitch box) via x.x.x.101 i.e .101 being a SIP proxy Flexisip.
The SIP flow look as below.
[ Linphone box ] <-> [ `.101` box ] <-> [ `.125` box ]
Now when we register i.e SIP REGISTER request (without VPN connected) to .125 via .101 it works i.e Registration is a success, saying this because I was sniffing around the traffic with TCP dump on both .101 and Linphone Client box (both had 200 OK Response for Register request).
Now here is the deal, when we make a call i.e send SIP INVITE request without VPN connected. We see no traffic on .101 box but same can be found on Linphone box hinting that request is directed to .101.(but at the same time I can constantly see OPTIONS request appearing from .101 to Linphone Machine and also a 200K response getting sent from Linphone machine for OPTIONS request.)
Now, The sooner we connect to VPN, we see the request appearing on the .101 box from Linphone box
Now, had this behaviour would have stayed constant, I would have suspected the Firewall rule but the it would work during SIP REGISTER and not using Regular INVITE invite is something I'm able to understand here.
And when we are connected to VPN it work.
NOTE: If I assume this is a packet loss as a part of UDP even the retransmission too does not go through and this happen in multiple run..
It's only the INVITE packet would not be sent always does not happen with REGISTER request.
Here how the sip flow looks like
Given the REGISTER requests are getting through that rules out an IP routing and firewall issue (assuming it's not doing deep inspection on SIP packets).
That leaves the two most likely culprits as the client software (in this case Linphone) using the wrong network interface,
Try running a test with a tool like sipp where you can explicitly set the local address to use and the type of SIP request to send.
# To test the user agent client scenario (which sends INVITE requests) use:
sipp -bind_local 10.1.10.1 -sn uac -m 1 x.x.x.125
Update:
Some observations from inspecting the packet captures:
In the no VPN case:
there is a NAPT between the softphone and the Proxy translating 10.1.10.1 to 49.36.13.47, e.g. 10.1.10.1:39248 mapped to 49.36.13.47:44150.
REGISTER responses do appear to be coming from the FreeSWITCH server at 63.211.239.125 according to the User Agent string of Flexisip... on the response.
in agreement with the original post there are no responses at all to INVITE requests in the capture.
there are no fragmented UDP packets captured.
In the VPN case:
there is no NAPT between the softphone and the Proxy. The softphone traffic is originating from 172.17.8.37.
the softphone is using an IP address of 192.168.29.134 in it's SDP offer which means the device likely has multiple network interfaces.
there is a fragmented UDP packet recorded when the softphone sends an INVITE request to the Proxy but it doesn't seem to be an issue as the Proxy happily forwards the request to the FreeSWITCH server.
Missing Data:
The no VPN capture does not contain the traffic between the Proxy and the FreeSWITCH server. This is the most crucial leg for the analysis since it could show whether the Proxy is forwarding the INVITE request or not.
Running tcpdump directly on the Proxy would be able to provide this missing information.
Updated Guess:
Based on the still incomplete information my best guess would now be that that Proxy has misconfigured (or perhaps deliberate) SIP settings and is silently dropping INVITE requests received on public interfaces.
When the softphone connects on the VPN the INVITE requests are forwarded because they are considered to originate from an internal network.
For REGISTER requests the Proxy could have a rule that says always forward them no matter which interface the Proxy receives them on since they are not as risky as INVITES.
If the path is different, the INVITE will go through different networks and they may not behave the same.
I can see 2 possible issues with the network used when VPN is not active:
A NAT has an ALG which, being broken, would drop the INVITE and let the other ones go through. This is unlikely, because it's happening for several User-Agent your tried.
The network is configured to drop packet larger than a specific size. This is very likely because the INVITE, with all User-Agent, is always the largest SIP message being sent.
I would advise you:
Try TCP: This should confirm it's a UDP only issue.
Try to remove all codecs and keep only PCMA, with UDP: if it works, it's likely a UDP/MTU/SIZE issue.
EDIT:
To make it clear, you certainly have an MTU issue.
Thus, I advise you to test your UDP network and the MTU size limit with and without the VPN. You don't have to use your android, but you need to use the same network.
On sip server, starts:
$> nc -u -l -p 2399
On LAN side, any PC with same network with netcat tool...
$> cat invite1000.example | nc -u sip.antisip.com 2399
$> cat invite1200.example | nc -u sip.antisip.com 2399
$> cat invite1500.example | nc -u sip.antisip.com 2399
$> cat invite2000.example | nc -u sip.antisip.com 2399
$> cat invite8000.example | nc -u sip.antisip.com 2399
Make several invitexxx.example files with any data, but with specific number of char in it.
In theory, nc/netcat on the server will stop receiving the packets when going over the MTU (or the packet will be incomplete).
Then, this will confirm it's an MTU issue.
This might be a bit weird to explain, but I'll try my best.
I have a Lua program that's intended to serve some data through the network. Specifically, the internet. The data the program is actually transmitting are only strings stored within UDP packets. Generalized, this is how the program operates:
The first client launches the program and specifies that they are the 'host' of the connection. The program opens a connection on UDP port 6000 and the main loop listens for any packets received on said port.
The second client launches the program and specifies that they are to connect to the 'host' on port 6000. The user enters the IP, and the client opens a UDP connection using a random port between 6050 and 7000
When the client successfully connects to the server, they send a 'connection' packet, simply containing a '202 OK' string. The 'host' receives this and registers the new client
Now that the connection has been initialized, the programs can send data between each other using the registered data.
Now, on a local network this program works fine. The purpose of the 'host' mode is to have multiple clients connect to the host and have the host relay packets from one clients to all the currently registered clients. Port selections are arbitrary and random port selection from the client was simply to allow debugging and testing from a single computer. This has been tested between two and more computers on a physical network, and worked successfully. However, when I attempt to run this over the internet it's a no go. I know that the ports are closed and that's why it's not working. But seeing as I'm going to be distributing this program (privately) I can't expect every person to open ports on their router (or know how to). Security is not currently a concern with the program, and should be disregarded in the current state. That being said, I recognise there's the potential for a lot to go wrong with the use of this program through the network and I accept that. Onto the main question, how can I have the host and client communicate over the internet without having to open ports?
I'll elaborate - for example, browsers. Although the technology is quite different to what I'm doing, it's easier to paint a picture - the browser requests data from a web server, and it gets sent back to the client. But wait, if the router is in it's default state (I hope) all the ports are closed? So how does the client receive this data if the port is closed?
I hope this makes some kind of sense and I don't sound like a complete fool.
I managed to find some suitable libraries and utilities to be able to communicate through the internet (NAT traversal is now a term I am familiar with), those libraries being that supplied by NMAP. These libraries include an implementation for STUN in LUA, among HEAPS of other useful networking-related libraries and scripts.
To actually answer my own question (very simply), the clients and servers are behind what's known as a NAT gateway. Due to the limitations of addresses of IPv4, NAT gateways were implemented to bypass this limitation of IPv4 (a total of about 4.2 billion addresses) by separating the clients' internal network from the external network - in this case the internet. The NAT is supplied with a single IP address, and the NAT then supplies all of its users within the internal network with an IP respective to the network they're on. As such, the devices cannot directly be accessed without forwarding connections from the NAT gateway (generally the router) to the client. However, when using UDP connections the NAT gateway opens a port for the purposes of this connection which gets closed after the connection dies. This port that is opened differs from what is specified by the client when they open the connection, which is where the STUN methods come in. STUN allows the host to find the port that the client is connecting from and send data back to this port so the user can receive it. Bear in mind this is an EXTREMELY simple explanation of how the technology works, and I'd suggest reading up on the Wiki and some of the Request for Comments for STUN.
Problem statement:
Suppose a parent server is hosted on a machine IP: 1.1.1.1 and that server some time communicates with three different servers say A (1.1.1.2), B (1.1.1.3), C (1.1.1.4). Those servers may be database servers or any other servers.
Now from your browser you can send a http request to 1.1.1.1/somePage.htm, as a result some TCP packet will go to the server 1.1.1.1, and 1.1.1.1 can send and receive some TCP packets from A,B,C as well.
Aim is to get the information of all TCP packets from the browser machine, without installing any agent software in any servers.
One solution is we can write a code at the 1.1.1.1 server machine that will filter all the TCP packets with respect to respective IPs. But I don’t want that solution.
Is there any way to solve this issue? Is it possible to introduce new protocol for this? But server codes can’t be modified.
Does "any agent software" includes something like Wireshark? Usually the way to look at all datagrams received is by using a sniffer like Wireshark or you can use tcpdump in Linux servers.
You can also use Netfilter to handle received packets in the server an take certain actions on them.
If all the above is included in what you don't want to do the only alternative I see is to add another server in the middle between the browser and the web server (or between the server and a load balancer if you have a load balancer) that acts only as a router or bridge. In that machine you can inspect and filter TCP segments with all the available tools.
Is there a way to convert wireshark .pcap files to a format that fiddler understands?
Every so often we get network traces generated from wireshark in .pcap files. Most of the time the traces contain mostly HTTP traffic which I find much easier to analyze using fiddler.
Fiddler 2 and Fiddler 4 now include import support for PCAP, PCAPNG, and NetMon files. Click File > Import > Packet Capture.
Note that only HTTP traffic will be imported (HTTPS traffic is encrypted, and FTP traffic wasn't a priority).
You can use ColaSoft Packet Player which enables to play captured .pcap files and open fiddler on capturing all processes and receive all data as it was captured.
Fiddler 4 supports pcap files.
Does someone know about a CPAN module on Win32 that captures network packets and edit them on the fly? As far as I know, the only Perl module on Win32 that deals with packets on the fly is Net::Pcap but it only support passive monitoring and not affet the TCP/IP stack.
Is there a such module could someone provide example /reference /documentation ?
As far as I know, libpcap allows you to read copies of incoming and outgoing packets, and some implementations allow you to inject a raw packet, but not rewrite a packet. You would basically have to drop the original packet (something libpcap cannot do) and then inject a new one in it's place.
Firewall apps that allow you to filter incoming and outgoing packets might be able to do something like this. However, since you're talking about Perl and Win32 your options are probably limited.
I think right answer is "implement proxy for this".
If it works in your scenario, try to implement proxy server. Listen on same port as your target service does and read all incoming traffic. If you need modification of packet, do it and pass all traffic to target service. Of course you have to implement both directions.
You can search for basic TCP deamon snippet in perl or maybe you can implement just module for existing proxy server for your service. Is it HTTP or what kind of traffic you need to handle?
I would suggest using Net::Pcap to capture traffic, then the Cygwin port of TCPReplay to modify and replay the traffic. Obviously a Linux setup would be more reliable since TCPreplay would work on it out of the box without requiring cygwin.