I am aware of this and this stackoverflow questions which answers known methods to achieve "Reliable Multicast" but off-late I have come across some websites which mentions even routers should also be programmed to handle custom protocols which are designed over UDP, is that true?
Basically I want to use Multicast for my application and I want don't want to impose any restriction of changing router for configuring custom protocol to handle UDP in reliable way , for example I was thinking for implementing/using PGM protocol over UDP to handle multicast but someone said that router should also have support for PGM which restricts me in providing solution since customers should change infrastructure for my solution which is unwarranted.
Please let me know if there is any solution which I can implement to handle UDP packets in reliable way without any changes to network infrastructure.
Thanks in advance.
EDIT:
I don't mean to say that I don't want to enable multicast in router, I would definitely enable multicast routing in router. When I read about PGM implementation some one said even router should be PGM capable which I thought is different router than commercially available routers in stores. Is my understanding wrong?
If you can't or don't want to configure routers to forward multicast traffic or otherwise handle a third party protocol, you'll need to tunnel multicast traffic over a unicast link. UFTP is capable of multicast tunneling via the use of a UFTP proxy server.
From the man page:
The proxy can run in one of three modes: a server proxy, a client
proxy, or response proxy.
A server proxy is typically local to a server and acts as the upstream
end of a multicast tunnel. It listens on the public multicast address
(and private multicast address when specified) and forwards downstream
packets to a specific address downstream. Upstream packets are
forwarded back where the announcement originated from.
A client proxy is typically local to one or more clients and forms the
downstream end of a multicast tunnel. It receives unicast data from
one or more server proxies and forwards downstream traffic to the
multicast address specified in the packet header. Upstream traffic
from clients is gathered and forwarded back where the announcement
came from as an aggregated response.
Below is a diagram of a typical configuration where a server sends multicast messages to the local network, and one or more server proxies unicast the messages to a corresponding client proxy, which in turn multicasts the messages to its local network.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
x Network A x
x ---------- x
x | Server | x
x ---------- x
x | x
x | multicast x
x | x
x |----------------------------------------- x
x | | | x
x v v v x
x ---------------- ---------------- ---------- x
x | Server Proxy | | Server Proxy | | Client | x
x ---------------- ---------------- ---------- x
x | | x
x | unicast | unicast x
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
| |
| ------------
| |
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxx
x | Network B x x | Network C x
x v x x v x
x ---------------- x x ---------------- x
x | Client Proxy | x x | Client Proxy | x
x ---------------- x x ---------------- x
x | x x | x
x | multicast x x | multicast x
x | x x | x
x |------------- x x |------------ x
x | | x x | | x
x v v x x v v x
x ---------- ---------- x x ---------- ---------- x
x | Client | | Client | x x | Client | | Client | x
x ---------- ---------- x x ---------- ---------- x
x x x x
xxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxx
These proxies are also capable of working in a NATed environment:
If a client proxy is behind a firewall, the proxy can send a heartbeat
message to the upstream proxy to make a pinhole in the firewall that
the upstream server proxy can connect to. If the client proxy is also
NATed, the upstream server proxy may not know the IP/port of the
client proxy, so the server proxy can be configured to wait for a
heartbeat message, and use the IP/port the heartbeat came from as its
downstream address. If the server proxy is also behind a firewall or
NAT, a second server proxy on a machine with a publicly accessible IP
can be inserted between the first server proxy and the client proxy.
In this case, the first server proxy is set up to use the second as
its downstream address, and the second server proxy is set up to use
the first heartbeat it receives from a client proxy as its downstream
address.
I'm the author of this software, so if you need pointers regarding how to set this up, send me an email via the link at the bottom of the UFTP page and we'll see what we can do.
Update:
In the case of PGM, it can be configured to run at either the application layer (i.e. on top of UDP) or at the transport layer (i.e. directly on top of IP). If PGM is run at the transport layer, that's where you might need to worry about the router having special support for it. Conversely, UFTP runs strictly at the application layer.
If you use multicast you need to add multicast support requirement to the networking infrastructure. Unless multicast routing is enabled on networking devices (multicast routers) multicast will be available only within one LAN.
The only reliable way to go through Internet infrastructure is uni-cast.
Update: Multicast routing is performed on Network layer and multicast routers don't need to know anything about transport/application layers. In some situation knowledge about application layer is required on network layer but almost all that situation is related to NAT ALG (Application Level Gateway).
By the way one of possible solution to pass multicast through the Internet is tunneling protocols (GRE, IP over IP and so on).
Related
Basically I know how browsers are attaching different port to each TCP connection by choosing free ephemeral port and therefore connection is unique, however I don't know how it looks like on TCP level when two backend services connect to each other. Is that similar to how browsers work?
For example let's say I'm sending request from some http client to 'Service A' that is running on 'thread-per-connection' server and listening on port 'X'. Within choosen endpoint I am also sending http request to 'Service B' that listens on port 'Y' (similar service or database), how will it start unique TCP connection between these two services, do 'Service A' acts simlilarly to how browsers handle that?
The outside HTTP client application is acting as a client to Service A. So that app will use an ephemeral port when making that 1st connection.
Service A then acts as a client to Service B. So Service A will use an ephemeral port when making that 2nd connection.
---------- ------------- -------------
| client | ----> | service A | --------> | service B |
---------- ------------- -------------
^ ^ ^ ^
| | | |
x.x.x.x:e1 y.y.y.y:X y.y.y.y:e2 z.z.z.z:Y
What you describe is common to all TCP connection, including HTTP. The party creating the connection ("client") picks an ephemeral port (it is actually picked by the OS, not by the application) when connecting to a party accepting the connection ("server").
Note that the terms "client" and "server" might be confusing since they are used with several meanings. A "server" is often a hardware which provides services. It can be the service application itself which accepts connections. But it can also be the role in the communication, i.e. the client is the one initiating the connection and the server is the one accepting it. In your case a Service A which is a server application acts in the role of the client when initiating a TCP connection to Service B.
Three computers connected to the same router will have the same public IP. If these computers send a request to my server, will they all have the same PORT as well or are there exceptions?
EDIT: When I get requests from the browser, the PORT is different for each connection it creates. Does the browser client just pick a random port that is available on the router?
Three computers connected to the same router will have the same public IP.
Correct, from the server's perspective, not from the client's perspective.
If these computers send a request to my server, will they all have the same PORT as well or are there exceptions?
No, they will not have the same Port on the router (they may on each client PC, though).
A TCP connection is uniquely identified by the tuple {protocol, local-ip, local-port, remote-ip, remote-port}. So, when multiple TCP connections have the same {remote-ip, remote-port} (IOW, when multiple clients are connected to the same server), then each {local-ip, local-port} must be unique. And vice versa, when multiple TCP connections have the same {local-ip, local-port} (IOW, when a client connects to multiple servers), then each {remote-ip, remote-port} must be unique.
When passing through a router, each TCP connection as seen on the client side will be {TCP, lan-ip, lan-port, server-ip, server-port}, while on the server side each connection will be seen as {TCP, listen-ip, listen-port, client-ip, client-port}, where {client-ip, client-port} will be the router's {public-ip, public-port}, so each {public-ip, public-port} must be unique.
So, multiple clients connecting to the same server through a router simply cannot use the same outgoing port on the router, otherwise the server would not be able to differentiate between the connections.
When I get requests from the browser, the PORT is different for each connection it creates.
Correct.
Does the browser client just pick a random port that is available on the router?
No, nor does the browser care that a router is present. The browser creates a local socket endpoint and binds it to an available {local-ip, local-port} and then uses it to connect to the server's {server-ip, server-port}. The packets go to the OS, the OS sends them to the router, the router opens its own available {public-ip, public-port} for each new connection and then forwards those packets to the server. When the server sends packets back, the router will receive those packets on its public NIC, forward them to the appropriate client OS, which will pass them to the appropriate socket endpoint.
-------------
| Client PC A |
-------------
{tcp, client-lan-ip, client-lan-port, server-ip, server-port}
/|\
|
\|/
{tcp, router-lan-ip, router-lan-port, client-lan-ip, client-lan-port}
--------
| Router |
--------
{tcp, router-public-ip, router-public-port, server-ip, server-port}
/|\
|
\|/
{tcp, listen-ip, listen-port, router-public-ip, router-public-port}
--------
| Server |
--------
I am learning TCP/IP basics. I made a server-client chat application in which server opens a port 1024 and client can send message to it. I am a bit confused about the contents of TCP/IP packets exchanged by server and client. If client sends a message to server, it goes as packets via ethernet. In the ethernet frame from client, data field is encoded in TCP/IP format. In the TCP/IP frame, destination port will be 1024. But what will be source port's value ? Client is opening no port. Only server opens a port. Also I would like to know if there is any way to monitor these TCP/IP packets sent and received in PC.
Don't forget there's multiple layers involve here. TCP, IP and Ethernet are on different ones even if they are often used in conjunction. The separation is important to keep in mind. Ethernet (layer 2) is a protocol that connects individual computers together, but it doesn't care what IP addresses they have. IP connects computers over a much larger scale, it can be routed and sent over a variety of "layer two" network technologies where Ethernet is but one of those.
The great thing about IETF internet protocols is they're all thoroughly documented so you can find out how they work internally. In the case of TCP, which operates on top of IP, the port numbers are in the TCP layer. IP itself doesn't care about them, it's only concerned about source and destination addresses.
The key is right here in the diagram that describes the TCP header:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Port | Destination Port |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Acknowledgment Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data | |U|A|P|R|S|F| |
| Offset| Reserved |R|C|S|S|Y|I| Window |
| | |G|K|H|T|N|N| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Checksum | Urgent Pointer |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Options | Padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| data |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Both source and destination port must be populated. This is a key component of how your system's IP stack tracks which packets pertain to which connections.
Normally when you write code that connects to a server your connection originates from a (somewhat) random source port. When you create a server process that listens on a port, then that port can be automatically assigned or set specifically.
For services like HTTP you'd want that port pinned to 80 if you want other clients to connect to that service, so automatic assignment is of no help. Sometimes automatic assignment is preferable so there's no conflicts.
You can monitor all of this with tools like tcpdump or Wireshark among many others. They can dig into the various layers and show what's going on.
Port number is a logical entity/number used to identify a process running in server/client. Just like your server application has a port number ( which you decided ), client application will also have some port number associated with it, as assigned by the OS. Type in netstat -ab in cmd prompt, you can see the port number associated with your client application, in the list of processes and corresponding port numbers given by the command.
I have been reading about WebRTC and how it enables peer to peer communication. So, i did a experiment for NAT traversal.
The experiment objective was to test hole punching in NAT. I had two system both were running Ubuntu 16.04. I will refer to these systems as system A and system B.
Both systems connect to a server hosted on Amazon Web Services. System A used command nc -p 1234 -u IP_ADDRESS_OF_SERVER PORT_NUMBER_OF_SERVER and system B used command nc -p 1235 -u IP_ADDRESS_OF_SERVER PORT_NUMBER_OF_SERVER. System A and system B both were connected to the same WiFi router.
After connecting to the server, we got to know about public socket of both systems. After that, i tried to connect each other using the information of public socket we got from the server within few seconds.
System A disconnects itself from server and connects to system B by using the command nc -u -p 1234 PUBLIC_IP_OF_SYS_B PUBLIC_PORT_OF_SYS_B and system B also disconnects itself from server and connects to system A by using the command nc -u -p 1235 PUBLIC_IP_OF_SYS_A PUBLIC_PORT_OF_SYS_A.
I did this to punch a hole in our NAT. After it, system A cancel the above nc command and listen to it's port using nc -l -u -p 1234. But unfortunately, i wasn't able to receive any message typed in system B in system A.
Can anyone help me in getting this work?
So, i figure it out myself. What i was trying to do is called Hairpin Translation and hairpin translation is still much less common among existing NATs (as compared to hole punch translation).
The two systems happen to reside behind the same NAT, and are therefore located in the same private IP address realm. System A has established a UDP session with server S, to which the common NAT has assigned its own public port number(say x). System B has similarly established a session with S, to which the NAT has assigned public port number (say y).
Suppose that system A uses the hole punching technique to establish a UDP session with B, using server S as an introducer. System A sends S a message requesting a connection to B. S responds to A with B's public and private endpoints, and also forwards A's public and private endpoints to B. Both clients then attempt to send UDP datagrams to each other directly at each of these endpoints. The messages directed to the public endpoints will not reach their destination as our NAT doesn't support hairpin translation.
Anyone can read about hairpin over here.
I have a lot (ten of thousands) of connected mobile devices which are maintaining an opened connection to a server. If my understanding of the 64k connection limitation is correct, you cannot have more than 64k (because of the TCP/IP protocol) connections to a single port of a server per client IP (because of the range of ephemeral ports on the client side).
But most of the time, you are in a context where these devices are connected through a network provider which use NAT to translate addresses. (for example, a smartphone won't have a static IP address).
So in this context, my server will see the same ip address and nothing garantee that the source port won't be the same in 2 different clients.
My question is maybe dumb but there it is : how can my server identify the correct connection if we think of a connection as the 5-tuple (protocol, server port, server ip, client ip, client port) in this situation ? Is there a risk of losing a connection or conflicts between 2 different clients ?
my server will see the same ip address and nothing guarantees that the source port won't be the same in 2 different clients [...] Is there a risk of losing a connection or conflicts?
No, that's the job of the router performing the NAT: keeping the IP:port combinations at one side linked to the ones on the other side.
So:
Client | IP | Src | < NAT > | IP | Src | Dest | Dst
======================================================
1 | .1 | 42 | <-----> | .3 | 1 | Server | 80
2 | .2 | 84 | <-----> | .3 | 2 | Server | 80
Given two clients, with (source IP 10.0.0.1, source port 42) and (source IP 10.0.0.2, source port 84) wish to connect to your server at port 80, then NAT will translate their IP:port pair to a pair that is valid on the other (right) side of the NAT (e.g. 11.0.0.3), by giving them a unique source port (on that side of the NAT). It will keep this translation in memory in order to be able to send packets both ways.
You'll see that the tuples on the right side of the NAT (so what your server sees) are unique:
11.0.0.3:1 - Server:80
11.0.0.3:2 - Server:80
If the router determines that the possible tuples towards your server have exhausted (so after 11.0.0.3:65535 - Server:80), it may refuse to open new connections to it.