I am learning TCP/IP basics. I made a server-client chat application in which server opens a port 1024 and client can send message to it. I am a bit confused about the contents of TCP/IP packets exchanged by server and client. If client sends a message to server, it goes as packets via ethernet. In the ethernet frame from client, data field is encoded in TCP/IP format. In the TCP/IP frame, destination port will be 1024. But what will be source port's value ? Client is opening no port. Only server opens a port. Also I would like to know if there is any way to monitor these TCP/IP packets sent and received in PC.
Don't forget there's multiple layers involve here. TCP, IP and Ethernet are on different ones even if they are often used in conjunction. The separation is important to keep in mind. Ethernet (layer 2) is a protocol that connects individual computers together, but it doesn't care what IP addresses they have. IP connects computers over a much larger scale, it can be routed and sent over a variety of "layer two" network technologies where Ethernet is but one of those.
The great thing about IETF internet protocols is they're all thoroughly documented so you can find out how they work internally. In the case of TCP, which operates on top of IP, the port numbers are in the TCP layer. IP itself doesn't care about them, it's only concerned about source and destination addresses.
The key is right here in the diagram that describes the TCP header:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Destination Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Acknowledgment Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data | |U|A|P|R|S|F| |
| Offset| Reserved |R|C|S|S|Y|I| Window |
| | |G|K|H|T|N|N| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum | Urgent Pointer |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Both source and destination port must be populated. This is a key component of how your system's IP stack tracks which packets pertain to which connections.
Normally when you write code that connects to a server your connection originates from a (somewhat) random source port. When you create a server process that listens on a port, then that port can be automatically assigned or set specifically.
For services like HTTP you'd want that port pinned to 80 if you want other clients to connect to that service, so automatic assignment is of no help. Sometimes automatic assignment is preferable so there's no conflicts.
You can monitor all of this with tools like tcpdump or Wireshark among many others. They can dig into the various layers and show what's going on.
Port number is a logical entity/number used to identify a process running in server/client. Just like your server application has a port number ( which you decided ), client application will also have some port number associated with it, as assigned by the OS. Type in netstat -ab in cmd prompt, you can see the port number associated with your client application, in the list of processes and corresponding port numbers given by the command.
Related
I have a Windows 10 machine, and I would like to access a database which is set on another machine outside local network.
Is there any possibility of achieving that using postgresql?
Thank's a lot, and I'd appreciate your effort to help me overcome this situation.
It is possible, provided that:
The firewall of your local network allows outgoing connections to the PostgreSQL listen port (usually 5432).
The firewall of the other network allows incoming connections to the PostgreSQL listen port (usually 5432).
The firewall of the PostgreSQL server allows connection on its listen port (usually 5432).
The PostgreSQL server is configured to accept network connections.
You can use a network scanner such as Nmap to test things, thing to do is to get a laptop on the customer's network, and scan from there. If you can connect to the PostgreSQL from an address on the same subnet, then you know there is nothing else needed on the PostgreSQL server, and so your attention need to be on the customer's firewall. This is where things can get difficult, and you'll need to work with whoever controls that firewall / router.
Chances are that the customer's network is on an RFC 1918 subnet. If this is the case the firewall / router will need to be configured to port forward like this:
public internet
|
----public address--port nnn--
| |
| firewall |
| |
|-----rfc 1918 address--------|
|
|
|
----rfc 1918 address--port 5432--
| |
| PostgreSQL server |
| |
|--------------------------------|
Three computers connected to the same router will have the same public IP. If these computers send a request to my server, will they all have the same PORT as well or are there exceptions?
EDIT: When I get requests from the browser, the PORT is different for each connection it creates. Does the browser client just pick a random port that is available on the router?
Three computers connected to the same router will have the same public IP.
Correct, from the server's perspective, not from the client's perspective.
If these computers send a request to my server, will they all have the same PORT as well or are there exceptions?
No, they will not have the same Port on the router (they may on each client PC, though).
A TCP connection is uniquely identified by the tuple {protocol, local-ip, local-port, remote-ip, remote-port}. So, when multiple TCP connections have the same {remote-ip, remote-port} (IOW, when multiple clients are connected to the same server), then each {local-ip, local-port} must be unique. And vice versa, when multiple TCP connections have the same {local-ip, local-port} (IOW, when a client connects to multiple servers), then each {remote-ip, remote-port} must be unique.
When passing through a router, each TCP connection as seen on the client side will be {TCP, lan-ip, lan-port, server-ip, server-port}, while on the server side each connection will be seen as {TCP, listen-ip, listen-port, client-ip, client-port}, where {client-ip, client-port} will be the router's {public-ip, public-port}, so each {public-ip, public-port} must be unique.
So, multiple clients connecting to the same server through a router simply cannot use the same outgoing port on the router, otherwise the server would not be able to differentiate between the connections.
When I get requests from the browser, the PORT is different for each connection it creates.
Correct.
Does the browser client just pick a random port that is available on the router?
No, nor does the browser care that a router is present. The browser creates a local socket endpoint and binds it to an available {local-ip, local-port} and then uses it to connect to the server's {server-ip, server-port}. The packets go to the OS, the OS sends them to the router, the router opens its own available {public-ip, public-port} for each new connection and then forwards those packets to the server. When the server sends packets back, the router will receive those packets on its public NIC, forward them to the appropriate client OS, which will pass them to the appropriate socket endpoint.
-------------
| Client PC A |
-------------
{tcp, client-lan-ip, client-lan-port, server-ip, server-port}
/|\
|
\|/
{tcp, router-lan-ip, router-lan-port, client-lan-ip, client-lan-port}
--------
| Router |
--------
{tcp, router-public-ip, router-public-port, server-ip, server-port}
/|\
|
\|/
{tcp, listen-ip, listen-port, router-public-ip, router-public-port}
--------
| Server |
--------
I have a lot (ten of thousands) of connected mobile devices which are maintaining an opened connection to a server. If my understanding of the 64k connection limitation is correct, you cannot have more than 64k (because of the TCP/IP protocol) connections to a single port of a server per client IP (because of the range of ephemeral ports on the client side).
But most of the time, you are in a context where these devices are connected through a network provider which use NAT to translate addresses. (for example, a smartphone won't have a static IP address).
So in this context, my server will see the same ip address and nothing garantee that the source port won't be the same in 2 different clients.
My question is maybe dumb but there it is : how can my server identify the correct connection if we think of a connection as the 5-tuple (protocol, server port, server ip, client ip, client port) in this situation ? Is there a risk of losing a connection or conflicts between 2 different clients ?
my server will see the same ip address and nothing guarantees that the source port won't be the same in 2 different clients [...] Is there a risk of losing a connection or conflicts?
No, that's the job of the router performing the NAT: keeping the IP:port combinations at one side linked to the ones on the other side.
So:
Client | IP | Src | < NAT > | IP | Src | Dest | Dst
======================================================
1 | .1 | 42 | <-----> | .3 | 1 | Server | 80
2 | .2 | 84 | <-----> | .3 | 2 | Server | 80
Given two clients, with (source IP 10.0.0.1, source port 42) and (source IP 10.0.0.2, source port 84) wish to connect to your server at port 80, then NAT will translate their IP:port pair to a pair that is valid on the other (right) side of the NAT (e.g. 11.0.0.3), by giving them a unique source port (on that side of the NAT). It will keep this translation in memory in order to be able to send packets both ways.
You'll see that the tuples on the right side of the NAT (so what your server sees) are unique:
11.0.0.3:1 - Server:80
11.0.0.3:2 - Server:80
If the router determines that the possible tuples towards your server have exhausted (so after 11.0.0.3:65535 - Server:80), it may refuse to open new connections to it.
I have a specific question on implementing a load balancer or a TCP/IP server program that does TCP/IP.
Since port number is 16 bits, there are a max of only 65536 ports on a single Linux box at any given time.
And TCP/IP needs a port number to talk to the outside world.
1) when a client establishes a connection, an ephemeral port number is chosen.
2) when a server listening on a socket accepts a connection, a port number is assigned.
So in my understanding at any given time only maximum 65536 TCP/IP connections can exist on a given machine.
So how is it that some or most load balancers claim 200,000 or more concurrent connections?
Can someone please explain that?
Also regarding load balancers, once a load balancer has forwarded a request to one of the servers behind it, can the load balancer somehow pass some information to it, that will help the server to respond back to the originating client directly to avoid the latency of sending back the response via the load balancer?
Thanks everyone for your help.
Thambi
Since port number is 16 bits, there are a max of only 65536 ports on a single Linux box at any given time.
65535 actually, as you can't use port zero.
when a server listening on a socket accepts a connection, a port number is assigned.
No it isn't. The incoming connection uses the same port it connected to. No new port is assigned on accept().
So in my understanding at any given time only maximum 65536 TCP/IP connections can exist on a given machine.
No, see above. The actual limit is determined by kernel and process resources: open FDs, thread stack memory, kernel buffer space, ... Not by the 16-bit port number.
I TCP connection is uniquely identified by a (remote IP address, remote port, remote IP address, remote port) tuple.
For a typical server application, there is only one to three local IP addresses and one or two local ports. For example, a web server might listen on local addresses ::1, ::ffff:93.184.216.34, and 2606:2800:220:1:248:1893:25c8:1946 (possibly via wildcard addresses, but that's irrelevant), and local ports 80 and 443.
For the simple case of a single local address and port that's still 2128 + 16 (less a few for special purpose and broadcast addresses), which will be problematic if you wish to communicate with the entire Earth in units of less than 4 million atoms (which might be possible if you converted all matter on Earth into small viruses).
There has been a confusion among this question so I'll try to explain it with examples.
First a couple words about ports: everyone knows that they don't exist physically, they are just an extra identification information for a connection, and also a way to allow multiple servers listening on the same address (if there was no concept of port only one server could be listening on one address, or some other mechanism would have to be in place). Also port is unsigned short so it can have values between 0 and 65535 (64k).
Now, restriction about ports: they are on the server side when bind ing: a (server) socket (let's call it SS) can bind to an address and port: (unless SO_REUSEADDR is set before first binding,) only one socket can listen on a particular address and port at a time, so if someone is already listening on a port you can't listen too. There are some well known ports (e.g.: sshd - 22, httpd - 80, RDP - 3389, ...) that should be avoided when creating
SS, a general guidline is never to use a port number < 1k. For a complete list of "reserved" ports, visit www.iana.org.
As stated in the link I posted in the comment there's a 5 item tuple(2 pairs + 1 additional element) that identify a connection (LocalIP: LocalPort, RemoteIP: RemotePort, Protocol) (the last member is just for rigurousity, at this point we don't care about it). Now for a particular SS that listens on a IP:Port, one of the 2 pairs will be the same for all the clients (client sockets: CS) that connect to it depending where looking at the connection from:
server's endpoint: LocalIP: LocalPort
client's endpoint: RemoteIP: RemotePort
(just like looking in the mirror).
Now I'm going to exemplify on 2 machines (Centos(192.168.149.43) - server and Windows(192.168.137.10) - client). I created a dummy TCP server in Python (note that the code is not structured, no exception handling, only IPv4 capable, the purpose is not to have a Python class but to see some socket behavior):
import sys
import select
import socket
HOST = "192.168.149.43"
PORT = 4461
WAIT_TIME = 0.5
if __name__ == "__main__":
conns = list()
nconns = 0
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.bind((HOST, PORT))
srv.listen(0xFF)
print "Entering loop, press a key to exit..."
while sys.stdin not in select.select([sys.stdin], [], [], 0)[0]:
if select.select([srv], [], [], WAIT_TIME)[0]:
conn = srv.accept()
print "Accepted connection from: (%s, %d)" % conn[1]
conns.append(conn)
nconns += 1
print "Active connections:", nconns
for item in conns:
item[0].close()
srv.close()
print "Exiting."
Here's the netstat output on the server machine (before running the server app). I chose port 4461 for communication:
[cfati#xobved-itaf:~]> netstat -an | grep 4461
[cfati#xobved-itaf:~]>
So nothing related to this port. Now after starting the server (I had to trim some spaces so that the output fits here):
[cfati#xobved-itaf:~]> netstat -anp | grep 4461
tcp 0 0 192.168.149.43:4461 0.0.0.0:* LISTEN
As you can see there is a socket listening for connections on port 4461.
Now going on the client machine and starting the Python interpreter, running the following code in the console:
>>> import sys
>>> import socket
>>> HOST = "192.168.149.43"
>>> PORT = 4461
>>>
>>> def create(no=1):
... ret = []
... for i in xrange(no):
... s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
... s.connect((HOST, PORT))
... ret.append(s)
... return ret
...
>>> sockets=[]
>>> sockets.extend(create())
Just after typing the last line on the server machine we look at the server output:
Accepted connection from: (192.168.137.10, 64218)
Active connections: 1
And the corresponding netstat output:
[cfati#xobved-itaf:~]> netstat -an | grep 4461
tcp 0 0 192.168.149.43:4461 0.0.0.0:* LISTEN
tcp 0 0 192.168.149.43:4461 192.168.137.10:64218 ESTABLISHED
You see the ESTABLISHED connection (this is the accepted socket - AS): The connection was initiated from 192.168.137.10 on port 64218, to 192.168.149.43 on port 4461.
Here's the corresponding netstat output on the client machine (after creating the connection):
e:\Work\Dev>netstat -an | findstr 4461
TCP 192.168.137.10:64218 192.168.149.43:4461 ESTABLISHED
As you can see the Local and Remote (IP/Port) pairs (compared to the output on the server machine) are reversed (like I mentioned above about looking in the mirror). If I go again on the client machine in the interpreter and re-run the last line (create a new connection):
>>> sockets.extend(create())
the output of the server app will show another entry:
Accepted connection from: (192.168.137.10, 64268)
Active connections: 2
while the netstat output on the server machine:
[cfati#xobved-itaf:~]> netstat -an | grep 4461
tcp 0 0 192.168.149.43:4461 0.0.0.0:* LISTEN
tcp 0 0 192.168.149.43:4461 192.168.137.10:64268 ESTABLISHED
tcp 0 0 192.168.149.43:4461 192.168.137.10:64218 ESTABLISHED
I'm not posting what netstat will output on the client machine since it's obvious.
Now, let's look at the 2 pairs each corresponding to an active connection: 192.168.137.10:64268, 192.168.137.10:64218. The 2 ports are returned by the accept function (Ux or Win) called on SS.
The 2 ports (64268 and 64218) are used by connections, but that doesn't mean that they cannot be used anymore. Other socket servers can listen on them (I am talking here in the server machine context), or they can be present as used ports in connections from other addresses. Here's a hypothetical netstat output:
tcp 0 0 192.168.149.43:4461 192.168.137.45:64218 ESTABLISHED
So, port 64218 can be also present in a connection from 192.168.137.45 (note that I changed the last IP byte).
As a conclusion, you were somehow right: there can't be more than 65535 (excluding 0 as specified in the other solution) simultaneous connections from the same IP address. This is a huge number, I don't know if in the real world it can be met, but even if so, there are "tricks" to get around it (one example is have 2+ SSs listening on 2+ different ports, and configure the client that if connection to the server to one port fails to use another, so the max simultaneous connections number from the same address can be increased by a factor equal to the number of ports we have servers listening on).
Load balancers handle connections from multiple addresses, so their number can easily grow to hundreds of thousands.
I have a tun/tap device which is used to read incoming packets from one interface and send them as UDP packets via another interface. I could implement this and could read ICMP pakcets send to the tun/tap interface and also get them remotely using UDP. But the issue happens when I try to change the default gateway of the input interface to the tun/tap device so that I can read all the incoming data from the tun/tap. When this is done, I cant send the UDP packets as the routing isnt proper.
I also tried to you the "SO_BINDTODEVICE" option in socket comm but still didnt work. Please note that I havent used the write() method in the tun/tap. I just used the read() function, collected the data and send them via UDP socket communication.
Please let me know if my approach is wrong or any other work around to overcome this. Thanks.
/********More Details********/
Thanks Rob.
What I am trying to achieve is a simulation of IP based header commuication(ROHC) in a high latency channel.
For this I have 4 virtual machines. VM1 is a normal desktop machine. VM2 is a gateway which takes the packets using tun/tap(from VM1) and does the UDP based communication with VM4. VM3 is the channel where parameters like latency, error rate etc can be set. VM4 is connected to the WAN. The user in VM1 should be able to browse the WAN just like normal. Please find the diagram below.
IP Packets
|
| +------------------+ +--------------+ +----------------+
'---|eth1..... | | | | |
| | | | | | |
| tun/tap | | eth0|___|UDP Sock eth0|___
| | | | | | | | | |
| ..UDP Sock|_____|eth1 | | | | | |
| | | | | +tun/tap+ | '
+------------------+ +--------------+ +----------------+ WAN
VM2 VM3(Channel) VM4
Update:
Thanks Tommi. Your solution worked. I could get the UDP packets one way to the final NAT gateway. But I could not get the reverse way to work till now.
I tried enabling the masquerade using iptables and also setting up the host route to the tuntap at VM1 but it wasnot working.
I have a few queries regarding this.
1) In VM4 I receive the UDP data and write to the tun/tap. This will get routed to the WAN by the kernel. But for the incoming packet, do I again need to read using the tun/tap? In this case do I need to make the read and write in different threads? I am asking this because I need to transport them back also as UDP data. Let me know if I am missing something here.
Once again thanks a lot for your help.
Your udp packets will get routed to your tuntap interface, too. (well, depending on some settings they may just get discarded). You need to add a route rule for the udp peer you are sending them to, a host rule or a smaller network rule that wont interfere with your other communication.