I'm trying to build a web app that operates in real-time via a push model. I've used sockets in the past to achieve this, just not in a distributed service. My question is: Given that the host that receives the updated information (via Amazon
SQS) is not necessarily the host that the necessarily the host that
the web browser has a socket connection open with, How do we find the socket connection and then push the data.
Thanks
Related
We made a chat module in our project using socket.io. When the load is balanced and the manual deployed, if socket connections are switched to different servers, socket connections are disconnected and the messaging events are partially not processed. I solved the load balance problem with socket.io-redis library. It acts as a gateway and solves this problem thanks to redis.
Another problem is that when I deploy it manually, the pid of the servers changes and socketio connections are instantly disconnected on the client and then it is not connected even though it says connected.
Do you think that using tools such as Travis CI solves the problems in manual deploy process?
Another question is, if a system that goes to 3 servers with load balance then goes back to 2 servers, the socket connections will be closed again, what method may be required to solve this? I thought of separating the socket.io service from the monolithic structure and keeping it on a single server, and scaling the server vertically when the load increased.
We are using an Aws Elastic Beanstalk(EBS), it automatically performs load balance.
I'm working on a simple Swift app where one user can find other machines on a local network which are running an instance of my app and then send data to that machine using TCP sockets.
My question is how to find IP addresses of devices on same network which are running an instance of my app (cross-platform)?
I was thinking about listing all devices on local network and then
checking whether they have opened specific port (the port my app is
using)?
I also found that Apple provides service called Bonjour which could make my process discoverable. I'm not sure if this solution is good for cross-platform communication.
Apart of Mac-related stuff and high-level solutions (I believe keywords "network service discovery" will bring you to them), there are a couple of things that will work for a local network:
I Have a server that clients should report to. Some short hello-like UDP message and a timeout mechanism will be sufficient to keep a list of available clients in the network.
II Use IP or UDP multicast groups to notify others that a client has just connected to the network. Send a message to a multicast group and listen to this group to build a list of clients.
However, broadcasts and multicasts won't be transmitted through a router. So if your network is large enough only neighbouring clients will hear your notification. In order to overcome it,
III DHCP servers can be configured to provide custom data to clients via unassigned DHCP options. Large networks have usually such server. You probably can use it to send out a list of clients, but I'm not sure about this.
This might be a bit weird to explain, but I'll try my best.
I have a Lua program that's intended to serve some data through the network. Specifically, the internet. The data the program is actually transmitting are only strings stored within UDP packets. Generalized, this is how the program operates:
The first client launches the program and specifies that they are the 'host' of the connection. The program opens a connection on UDP port 6000 and the main loop listens for any packets received on said port.
The second client launches the program and specifies that they are to connect to the 'host' on port 6000. The user enters the IP, and the client opens a UDP connection using a random port between 6050 and 7000
When the client successfully connects to the server, they send a 'connection' packet, simply containing a '202 OK' string. The 'host' receives this and registers the new client
Now that the connection has been initialized, the programs can send data between each other using the registered data.
Now, on a local network this program works fine. The purpose of the 'host' mode is to have multiple clients connect to the host and have the host relay packets from one clients to all the currently registered clients. Port selections are arbitrary and random port selection from the client was simply to allow debugging and testing from a single computer. This has been tested between two and more computers on a physical network, and worked successfully. However, when I attempt to run this over the internet it's a no go. I know that the ports are closed and that's why it's not working. But seeing as I'm going to be distributing this program (privately) I can't expect every person to open ports on their router (or know how to). Security is not currently a concern with the program, and should be disregarded in the current state. That being said, I recognise there's the potential for a lot to go wrong with the use of this program through the network and I accept that. Onto the main question, how can I have the host and client communicate over the internet without having to open ports?
I'll elaborate - for example, browsers. Although the technology is quite different to what I'm doing, it's easier to paint a picture - the browser requests data from a web server, and it gets sent back to the client. But wait, if the router is in it's default state (I hope) all the ports are closed? So how does the client receive this data if the port is closed?
I hope this makes some kind of sense and I don't sound like a complete fool.
I managed to find some suitable libraries and utilities to be able to communicate through the internet (NAT traversal is now a term I am familiar with), those libraries being that supplied by NMAP. These libraries include an implementation for STUN in LUA, among HEAPS of other useful networking-related libraries and scripts.
To actually answer my own question (very simply), the clients and servers are behind what's known as a NAT gateway. Due to the limitations of addresses of IPv4, NAT gateways were implemented to bypass this limitation of IPv4 (a total of about 4.2 billion addresses) by separating the clients' internal network from the external network - in this case the internet. The NAT is supplied with a single IP address, and the NAT then supplies all of its users within the internal network with an IP respective to the network they're on. As such, the devices cannot directly be accessed without forwarding connections from the NAT gateway (generally the router) to the client. However, when using UDP connections the NAT gateway opens a port for the purposes of this connection which gets closed after the connection dies. This port that is opened differs from what is specified by the client when they open the connection, which is where the STUN methods come in. STUN allows the host to find the port that the client is connecting from and send data back to this port so the user can receive it. Bear in mind this is an EXTREMELY simple explanation of how the technology works, and I'd suggest reading up on the Wiki and some of the Request for Comments for STUN.
I connect to a web server supported by an embedded system with Internet Explorer 9. Windows 7 is on the client side.
The web page have many tabs and I browse across until the problem occurs. It takes about one minute to happen.
The embedded system freezes so it not possible to browse and it does not respond to ping. After a moment the embedded system will recover because it is designed to reboot. I joined a Wireshark trace in which you can see 92 connections (use the filter "tcp.stream eq 0" with values [0,91]) and you will see. I have the source code so I know that the embedded system does not support more than 37 simultaneous connections. Is the cause an exhaustion of the resources?
But I have a more basic question and I really more appreciate an answer to it. The web server is at 172.21.1.12 port 80 and the client is at
172.21.9.70 and variable port numbers (see the trace). Because the IP and port on the server side do not change, how many sockets are in use on the server side? The question is important because the more sockets are opened, the more probably there is an exhaustion of the resources.
If the answer is only 1 socket then I must conclude there is no lack of resources because it can support 37.
I also suggest you use the filter ip.addr == 172.21.1.12 in Wireshark.
I thought I could upload the wireshark file. I dont know how to share it with you. Help please?
Dropbox?
Under the caveat that you haven't specified your embedded system, most TCP stacks will create a new socket for each new connection, and the mapping from socket to connection is 1-1.
When a packet arrives to the network stack, it has to associate that packet to the right socket. Usually, this is accomplished by employing a map from the TCP 4-tuple to the socket, where the 4-tuple consists of [local-ip, local-port, remote-ip, remote-port].
A server makes its service available by listening on a fixed local port that is known to clients wanting to use the service. As you understand, this is usually port 80 for a web server, and the software interface for most TCP implementations dedicate a socket for the purpose of allowing the API to perform operations on the network parameters for this service. However, the socket is not fully connected (the last two parts of the 4-tuple are set to a special "not specified" value, usually all bits 0). When a new connection is accepted, a new socket is created where the 4-tuple consists of the local information of the listening socket and the remote information taken from the source address and port of the SYN packet that initiated the TCP connection.
The limit on the number of connections a server can support is based on how the operating system is configured (you say yours limits it to 37). Using the 4-tuple, a single service (that is a fixed local-ip and local-port) will have an absolute limit of (2ADDR_BITS - RESERVED_ADDRS) × (216 - RESERVED_PORTS). For IPv4, the number of bits is 32, while for IPv6, the number of bits is 128.
When creating a connection, the client will specify the destination address and port (which fills out the remote information for the 4-tuple), but usually leave the source information unspecified. The TCP stack will choose an appropriate source address based on routing, and select an available source port (which will become the local information to complete the 4-tuple). In theory, any source port that is not being used by the selected local interface to communicate to the same remote service can be used as the local port. Most stacks will dedicate a set of the higher numbered ports for this purpose (referred to as the ephemeral port range).
I am in the process of developing a peer to peer app,
I am a bit confused by the following scenario:
Lets say my application will use an outgoing port 1863 - which is also used for msn messenger(if this is not the port lets assume it is)
Now, client executes my app and connects to my server at port 1863.
I am a bit confused if this is going to produce any problems.
I know that 2 apps can use same port for outgoing communication. But what happens to the data coming back?
Also, does my client need to open port for my app to run correctly??
I know that 2 apps can use same port for outgoing communication. But
what happens to the data coming back?
That's exactly the problem the source port solves. The peer can always differentiate between 2 connections based on it. When it sends replies, what was the source port now becomes the destination port allowing the original receiver to correctly pass data to the rightful processes.