This might be a bit weird to explain, but I'll try my best.
I have a Lua program that's intended to serve some data through the network. Specifically, the internet. The data the program is actually transmitting are only strings stored within UDP packets. Generalized, this is how the program operates:
The first client launches the program and specifies that they are the 'host' of the connection. The program opens a connection on UDP port 6000 and the main loop listens for any packets received on said port.
The second client launches the program and specifies that they are to connect to the 'host' on port 6000. The user enters the IP, and the client opens a UDP connection using a random port between 6050 and 7000
When the client successfully connects to the server, they send a 'connection' packet, simply containing a '202 OK' string. The 'host' receives this and registers the new client
Now that the connection has been initialized, the programs can send data between each other using the registered data.
Now, on a local network this program works fine. The purpose of the 'host' mode is to have multiple clients connect to the host and have the host relay packets from one clients to all the currently registered clients. Port selections are arbitrary and random port selection from the client was simply to allow debugging and testing from a single computer. This has been tested between two and more computers on a physical network, and worked successfully. However, when I attempt to run this over the internet it's a no go. I know that the ports are closed and that's why it's not working. But seeing as I'm going to be distributing this program (privately) I can't expect every person to open ports on their router (or know how to). Security is not currently a concern with the program, and should be disregarded in the current state. That being said, I recognise there's the potential for a lot to go wrong with the use of this program through the network and I accept that. Onto the main question, how can I have the host and client communicate over the internet without having to open ports?
I'll elaborate - for example, browsers. Although the technology is quite different to what I'm doing, it's easier to paint a picture - the browser requests data from a web server, and it gets sent back to the client. But wait, if the router is in it's default state (I hope) all the ports are closed? So how does the client receive this data if the port is closed?
I hope this makes some kind of sense and I don't sound like a complete fool.
I managed to find some suitable libraries and utilities to be able to communicate through the internet (NAT traversal is now a term I am familiar with), those libraries being that supplied by NMAP. These libraries include an implementation for STUN in LUA, among HEAPS of other useful networking-related libraries and scripts.
To actually answer my own question (very simply), the clients and servers are behind what's known as a NAT gateway. Due to the limitations of addresses of IPv4, NAT gateways were implemented to bypass this limitation of IPv4 (a total of about 4.2 billion addresses) by separating the clients' internal network from the external network - in this case the internet. The NAT is supplied with a single IP address, and the NAT then supplies all of its users within the internal network with an IP respective to the network they're on. As such, the devices cannot directly be accessed without forwarding connections from the NAT gateway (generally the router) to the client. However, when using UDP connections the NAT gateway opens a port for the purposes of this connection which gets closed after the connection dies. This port that is opened differs from what is specified by the client when they open the connection, which is where the STUN methods come in. STUN allows the host to find the port that the client is connecting from and send data back to this port so the user can receive it. Bear in mind this is an EXTREMELY simple explanation of how the technology works, and I'd suggest reading up on the Wiki and some of the Request for Comments for STUN.
Related
For example, when you make an ssh connection, you are connected to port 22. What happens then? On a very high level brief overview, I know that if port 22 is open on the other end and if you can authenticate to it as a certain user, then you get a shell on that machine.
But I don't understand how ports tie into this model of services and connections to different services from remote machines? Why is there a need for so many specific ports running specific services? And what exactly happens when you try to connect to a port?
I hope this question isn't too confusing due to my naive understanding. Thanks.
Imagine your server as a house with 65536 doors. If you want to visit family "HTTP", you go to door 80. If you were to visit family "SMTP", you would visit door no. 25.
Technically, a port is just one of multiple possible endpoints for outgoing/incomming connections. Many of the port numbers are assigned to certain services by convention.
Opening/establishing a connection means (when the transport protocol is TCP, which are most of the “classical” services like HTTP, SMTP, etc.) that you are performing a TCP handshake. With UDP (used for things like streaming and VoIP), there's no handshake.
Unless you want to understand the deeper voodoo of IP networks, you could just say, that's about it. Nothing overly special.
TCP-IP ports on your machine are essentially a mechanism to get messages to the right endpoints.
Each of the possible 65536 ports (16 total bits) fall under certain categories as designated by the Internet Assigned Numbers Authority (IANA).
But I don't understand how ports tie into this model of services and
connections to different services from remote machines? Why is there a
need for so many specific ports running specific services?
...
And what exactly happens when you try to connect to a port?
Think of it this way: How many applications on your computer communicate with other machines? Web browser, e-mail client, SSH client, online games, etc. Not to mention all of the stuff running under the hood.
Now think: how many physical ports do you have on your machine? Most desktop machines have one. Occasionally two or three. If a single application had to take complete control over your network interface nothing else would be able to use it! So TCP ports are a way of turning 1 connection into 65536 connections.
For example, when you make an ssh connection, you are connected to
port 22. What happens then?
Think of it like sending a package. Your SSH client in front of you needs to send information to a process running on the other machine. So you supply the destination address in the form of "user#[ip or hostname]" (so that it knows which machine on the network to send it to), and "port 22" (so it gets to the right application running on the machine). Your application then packs up a TCP parcel and stamps a destination and a return address and sends it to the network.
The network finds the destination computer and delivers the package. So now it's at the right machine, but it still needs to get to the right application. What do you think would happen if your SSH packet got delivered to an e-mail client? That's what the port number is for. It effectively tells your computer's local TCP mailman where to make the final delivery. Then the application does whatever it needs to with the data (such as verify authentication) and sends a response packet using your machine's return address. The back and forth continues as long as the connection is active.
Hope that helps. :)
The port is meant to allow applications on TCP/IP to exchange data. Each machine on the internet has one single address which is its IP. The port allows different applications on one machine to send and receive data with multiple servers on the network/internet. Common application like ftp and http servers communicate on default ports like 21 and 80 unless network administrators change those default ports for security reasons
I connect to a web server supported by an embedded system with Internet Explorer 9. Windows 7 is on the client side.
The web page have many tabs and I browse across until the problem occurs. It takes about one minute to happen.
The embedded system freezes so it not possible to browse and it does not respond to ping. After a moment the embedded system will recover because it is designed to reboot. I joined a Wireshark trace in which you can see 92 connections (use the filter "tcp.stream eq 0" with values [0,91]) and you will see. I have the source code so I know that the embedded system does not support more than 37 simultaneous connections. Is the cause an exhaustion of the resources?
But I have a more basic question and I really more appreciate an answer to it. The web server is at 172.21.1.12 port 80 and the client is at
172.21.9.70 and variable port numbers (see the trace). Because the IP and port on the server side do not change, how many sockets are in use on the server side? The question is important because the more sockets are opened, the more probably there is an exhaustion of the resources.
If the answer is only 1 socket then I must conclude there is no lack of resources because it can support 37.
I also suggest you use the filter ip.addr == 172.21.1.12 in Wireshark.
I thought I could upload the wireshark file. I dont know how to share it with you. Help please?
Dropbox?
Under the caveat that you haven't specified your embedded system, most TCP stacks will create a new socket for each new connection, and the mapping from socket to connection is 1-1.
When a packet arrives to the network stack, it has to associate that packet to the right socket. Usually, this is accomplished by employing a map from the TCP 4-tuple to the socket, where the 4-tuple consists of [local-ip, local-port, remote-ip, remote-port].
A server makes its service available by listening on a fixed local port that is known to clients wanting to use the service. As you understand, this is usually port 80 for a web server, and the software interface for most TCP implementations dedicate a socket for the purpose of allowing the API to perform operations on the network parameters for this service. However, the socket is not fully connected (the last two parts of the 4-tuple are set to a special "not specified" value, usually all bits 0). When a new connection is accepted, a new socket is created where the 4-tuple consists of the local information of the listening socket and the remote information taken from the source address and port of the SYN packet that initiated the TCP connection.
The limit on the number of connections a server can support is based on how the operating system is configured (you say yours limits it to 37). Using the 4-tuple, a single service (that is a fixed local-ip and local-port) will have an absolute limit of (2ADDR_BITS - RESERVED_ADDRS) × (216 - RESERVED_PORTS). For IPv4, the number of bits is 32, while for IPv6, the number of bits is 128.
When creating a connection, the client will specify the destination address and port (which fills out the remote information for the 4-tuple), but usually leave the source information unspecified. The TCP stack will choose an appropriate source address based on routing, and select an available source port (which will become the local information to complete the 4-tuple). In theory, any source port that is not being used by the selected local interface to communicate to the same remote service can be used as the local port. Most stacks will dedicate a set of the higher numbered ports for this purpose (referred to as the ephemeral port range).
I'm building a system that relies on a central server to send the IP address and port of the first user (on mobile or desktop app) to a second user (on mobile or desktop app). The second user establishes a P2P encrypted connection with the first user, using the IP address and port sent by the central server, to send a large file directly (ideally, the actual file doesn't pass through the central server).
This system needs to work even if the users are behind different firewalls / NATs and on mobile or desktop devices, without requiring users to manually open ports.
I've been looking into NAT Traversal Protocol (Teredo IPv6), libjingle (Google's open source suite), STUN, direct socket connections, and direct VPNs between the users.
I'm confused if I'm approaching this correctly. Would all of these options solve this problem independently? Or am I approaching this wrong? Would direct IPv6 connections would straight out, even behind IPv4 routers?
P2P connection is not guaranteed to succeed always. It can fail for the following reasons:
1) Two peers are behind symmetric NAT. (Although Teredo works if one peer is behind symmetric nat.) 2) UDP is blocked
3) If the peer is behind proxy.
4) Double NAT scenarios.
There are three types of ipv6 address - link local, private address & global. Two peers can connect directly over the internet if they have global address. Global address prefix is (200:....). If your building P2P system, you should have fallback mechanism in which case the central server should relay the data between the peers. This way you can make your application reliable at the time make connection faster for most peers using p2p.
I am in the process of developing a peer to peer app,
I am a bit confused by the following scenario:
Lets say my application will use an outgoing port 1863 - which is also used for msn messenger(if this is not the port lets assume it is)
Now, client executes my app and connects to my server at port 1863.
I am a bit confused if this is going to produce any problems.
I know that 2 apps can use same port for outgoing communication. But what happens to the data coming back?
Also, does my client need to open port for my app to run correctly??
I know that 2 apps can use same port for outgoing communication. But
what happens to the data coming back?
That's exactly the problem the source port solves. The peer can always differentiate between 2 connections based on it. When it sends replies, what was the source port now becomes the destination port allowing the original receiver to correctly pass data to the rightful processes.
I want to develop an iPhone app with a simple IM feature. I am thinking about setting up an HTTP server on an iPhone. If the iPhone is using wifi and is behind a firewall, how can I make sure that other iPhone clients can connect to it?
It's not the firewall that will disturb the connection as much it is the NAT.
When you are connected through wireless router to connect the internet you are surfing via NAT. it means you dont really have an extenral IP but once you initiate connection the router will map your intenral IP to one of his externatl ports and for certain time window he will pass connections to you if he will get it to the right port.
That being said, there is no actual way of setting a server behind a NAT unless you can configure port forwarding in the router and internal static IP.
Hope i was clear enough, good luck
I do not really think that you need to get an HTPP server up and running on iPhone to make an application that can send and receive messages (IM). The idea of making one iPhone user to directly connect to one another does not seem right to me since the users will need to know IP addresses of one another to do that.
Interconnectivity between different users of the chat can be solved by making your application communicate via a dedicated TCP port. It is generally advisable to choose ports with a number higher than 1024 since those below are generally found on the list of so-called well-known ports and are used for Web (like port 80), FTP (port 21), SSH (22), DNS (53), etc., it will be the responsibility of the user to make sure the port used by your application is open on the firewall. In order to solve this problem you can actually use port 80 for communication if you find that the port you have selected is blocked. You can do this because you know that this port will not be blocked in most cases. Indeed Yahoo Messenger is reported to use this technique when the firewall blocks the port it uses for communication.
The port should be used by your application to connect to the Web-server that will actually store user credentials, perform authentication, message transmission, etc., and the server should reside on capable hardware to be able to support large number of simultaneous connections. I can suggest using either a VPS (like the one provided by Linode) or a cloud (like Amazon EC2, Google Application Engine, Rackspace).