I looked online and can't seem to find too many similar stories, which is surprising.
I rented a server from a provider, and I run my server software on there. My iOS app connects to it. It's been up for over a month with no issues, and gets about dozens of connections a day.
Starting about 30 hours ago, someone began to connect to it every 2 seconds. Always same IP. I doubt he leaved his phone on for 30 hours? And my app has only <100 daily users, so I have no competitors who would gain from this.
I finally blocked him from my server using iptable. From the software engineering side, what is the common practice for preventing these kinds of things? Am I supposed to keep track from the server side and refuse to accept repeated connections? Do I use some sort of login/handshake, or what do I do?
To clarify, I do not use http or apache. I wrote a server based on BSD sockets using a custom protocol over TCP. I also have a crypto hash and would terminate a connection if the message doesn't hash correctly.
EDIT: I did a count in my connection log. 3 IPs connected totaling 10,000 times over the last 30 hours. 2 of them have since stopped. All from a tiny country in Asia which I won't name.
You need to implement some sort of antirobot in your application:
Drop incoming connection if there are already >= N established connections from the same IP address.
Drop incoming connection if total number of connections from this IP address for last 1, 5, and 30 minutes exceeds predefined thresholds (yes, you will need to keep track of number of connections made from each IP address).
More sophisticated antirobots used in high-load environments would also have means of detecting and blocking not only single IP addresses, but also whole ranges of IP addresses (networks). And also it would be better not to drop connections blindly but to redirect clients to captcha.
Anyway, there is no perfect solution to distinguish between robot and normal client: either you drop some good clients, or you allow some robots to sift through your antirobot.
Related
I have doubt while making my router as DHCP server that is:
suppose I made my router a DHCP server and I have connected one computer to the router which will listen to all the user requests now I have added more devices like mobile phones etc. But till now as we know we have only one device (server) which listens to user requests my question is, Is there any possibility that because of all other devices my server won't able to get user requests? I know question can be silly for you I want to know the answer.
Thanks.
DHCP protocol has an explicit mechanism for packet loss. Clients are supposed to do a retry on renew or fall back to start over with discover messages. If there is one more computer on the network is probably less important than the number of clients trying to renew their IP address at the same time. I have seen networks with 70,000 devices managed by a single DHCP server with no problem. Of course the DHCP server needs to be fast in all aspects: network latency, storage speed (for the lease data base) and processing speed. If the question is whether the DHCP protocol traffic can hinder other protocols? Then the answer is potentially yes, but probably only at times when many device go online (and need an IP address) at the same time. The overall traffic on the network generally is most likely not influenced by the very small amount of data single DHCP messages create.
I've got some programs that occasionally (anywhere from every few minutes to once an hour) need to send metrics to Graphite. Should I keep the socket to the graphite server open for the duration of my process or make a new connection every time I need to send some metrics? What are the considerations when doing one or the other?
Sounds like you need a TCP connection.
If you should keep the connection active or not depends on answers to points like:
- Would you like to monitor the "connected" clients at the server at any given time?
- Is there a limit at the Server side in relation to the previous point?
- The amount of such clients "connected" to the server?
- Is it a problem if the connection creation takes some time?
If you keep the connection open, just make sure to send keep-alive messages from time to time (application level proffered).
A large amount of clients connected to the server, even when not active, may consume resources of memory or objects (for example, if there is one thread per connection).
On the other hand, keeping the connection on, will allow the client to detect if there is a connection problem to the server much faster (if that even matters).
it all depends on when is needed.
I have a chat application (xmpp / muc) that is going to be served by apache (we might change to nginx later but right now it's not easily done). If a user is in 2 rooms, he'll have between 2 and 4 active connections to the server (long-polling connections), so if we have 200 users per room and we have 5 rooms, what should the ServerLimit, MaxClients be set to? For example, these are the default values:
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
Thanks,
Each user should only have a single connection to the XMPP server if your using something like BOSH to make the Web->XMPP connection. Each MUC room activity will be sent to the XMPP server with the user's JID as the target and that will cause the server to send that to the appropriate BOSH connection. Same for presence.
The information you need to load balance the Apache setup is the same as load balancing any long-polling connection - XMPP doesn't add anything exotic to the mix. An XMPP BOSH connection is a long-polling connection to your web server which is then sent to a persistent connection to the XMPP server - causing 2 sockets to be opened per user connect.
I'm guessing this a pre-fork Apache....
If you've got 1000 concurrent chat connections, then you need at least 1000 webservers (presumably you'll be serving up static content too, and you say that each presence results in 2 connections - so you can double the 1000) - a ServerLimit/MaxClients of 256 is not going to cut it, you probably need around 2200 to support this (but without hard metrics its hard to give an exact figure).
This is a ridiculously large amount. To support this I'd be looking for 3 boxes, each with about 2Gb free memory before the webserver is started.
MaxRequestsPerChild is not really relevant other than the fact you want some turnover of processes, particularly if you are using long polling.
This is one of the reasons why COMET is not a good idea. Using AJAX polling would be much more efficient. Assuming this is not something you can change, you might want to have a look at using a threaded Apache webserver which is marginally more memory efficient,
C.
Recently I've added some load-balancing capabilities to a piece of software that I wrote. It is a networked application that does some data crunching based on input coming from a SQL database. Since the crunching can be pretty intensive I've added the capability to have multiple instances of this application running on different servers to split the load but as it is now the load balancing is a manual act. A user must specify which instances take which portion of the input domain.
I would like to take that to the next level and program the instances to automatically negotiate the diving up of the input data and to recognize if one of them "disappears" (has crashed or has been powered down) so that the remaining instances can take on the failed instance's workload.
In order to implement this I'm considering using a simple heartbeat protocol between the instances to determine who's online and who isn't and while this is not terribly complicated I'd like to know if there are any established heartbeat network protocols (based on UDP, TCP or both).
Obviously this happens a lot in the networking world with clustering, fail-over and high-availability technologies so I guess in the end I'd like to know if maybe there are any established protocols or algorithms that I should be aware of or implement.
EDIT
It seems, based on the answers, that either there are no well established heart-beat protocols or that nobody knows about them (which would imply that they aren't so well established after all) in which case I'm just going to roll my own.
While none of the answers offered what I was looking for specifically I'm going to vote for Matt Davis's answer since it was the closest and he pointed out a good idea to use multicast.
Thank you all for your time~
Distribued Interactive Simulation (DIS), which is defined under IEEE Standard 1278, uses a default heartbeat of 5 seconds via UDP broadcast. A DIS heartbeat is essentially an Entity State PDU, which fully defines the state, including the position, of the given entity. Due to its application within the simulation community, DIS also uses a concept referred to as dead-reckoning to provide higher frequency heartbeats when the actual position, for example, is outside a given threshold of its predicted position.
In your case, a DIS Entity State PDU would be overkill. I only mention it to make note of the fact that heartbeats can vary in frequency depending on the circumstances. I don't know that you'd need something like this for the application you described, but you never know.
For heartbeats, use UDP, not TCP. A heartbeat is, by nature, a connectionless contrivance, so it goes that UDP (connectionless) is more relevant here than TCP (connection-oriented).
The thing to keep in mind about UDP broadcasts is that a broadcast message is confined to the broadcast domain. In short, if you have computers that are separated by a layer 3 device, e.g., a router, then broadcasts are not going to work because the router will not transmit broadcast messages from one broadcast domain to another. In this case, I would recommend using multicast since it will span the broadcast domains, providing the time-to-live (TTL) value is set high enough. It's also a more automated approach than directed unicast, which would require the sender to know the IP address of the receiver in order to send the message.
Broadcast a heartbeat every t using UDP; if you haven't heard from a machine in more than k*t, then it's assumed down. Be careful that the aggregate bandwidth used isn't a drain on resources. You can use IP broadcast addresses, or keep a list of specific IPs you're doing work for.
Make sure the heartbeat includes a "reboot count" as well as "machine ID" so that you know previous server state isn't around.
I'd recommend using MapReduce if it fits. It would save a lot of work.
I'm not sure this will answer the question but you might be interested by the way Weblogic Server clustering work under the hood. From the book Mastering BEA WebLogic Server:
[...] WebLogic Server clustering provides a loose coupling of the servers in the cluster. Each server in the cluster is independent and does not rely on any other server for any fundamental operations. Even if contact with every other server is lost, each server will continue to run and be able to process the requests it receives. Each server in the cluster maintains its own list of other servers in the cluster through periodic heartbeat messages. Every 10 seconds, each server sends a heartbeat message to the other servers in the cluster to let them know it is still alive. Heartbeat messages are sent using IP multicast technology built into the JVM, making this mechanism efficient and scalable as the number of servers in the cluster gets large. Each server receives these heartbeat messages from other servers and uses them to maintain its current cluster membership list. If a server misses receiving three heartbeat messages in a row from any other server, it takes that server out of its membership list until it receives another heartbeat message from that server. This heartbeat technology allows servers to be dynamically added and dropped from the cluster with no impact on the existing servers’ configurations.
Cisco content switches are a hardware solution for this problem. They implement a virtual IP address as a front end to multiple real servers, whose real IP addresses are known to the switch. The switch periodically sends HTTP HEAD requests to the web servers, to verify they are still running (which the switch software calls a "keepalive", although this doesn't keep the server itself alive). The Cisco switch accepts traffic on the virtual IP and forwards it to the actual web servers, using configurable load balancing such as round-robin, or user-defined load balancing.
These switches retail in the $3-10K range, although my business partner picked one up on eBay for about $300 a year ago. If you can afford one, they do represent a proven hardware solution to the question of how to have a service spread transparently across multiple servers. Redhat includes a built-in port configuration so that you could implement your own Cisco switch using a cheap RedHat box. Google for "virtual ip address" and "cisco content router" for more information.
In addition to trying hardware load-balancers, you can also try a free-open-source load-balancing software application such as HAProxy, available for Linux and the BSDs.
Is it possible to multiplex sa ocket connection?
I need to establish multiple connections to yahoo messenger and i am looking for a way to do this efficiently without having to hold a socket open for each client connection.
so far i have to use one socket for each client and this does not scale well above 50,000 connections.
oh, my solution is for a TELCO, so i need to at least hit 250,000 to 500,000 connections
i'm planing to bind multiple IP addresses to a single NIC to beat the 65k port restriction per IP address.
Please i would any help, insight i can get.
**most of my other questions on this site have gone un-answered :) **
Thanks
This is an interesting question about scaling in a serious situation.
You are essentially asking, "How do I establish N connections to an internet service, where N is >= 250,000".
The only way to do this effectively and efficiently is to cluster. You cannot do this on a single host, so you will need to be able to fragment and partition your client base into a number of different servers, so that each is only handling a subset.
The idea would be for a single server to hold open as few connections as possible (spreading out the connectivity evenly) while holding enough connections to make whatever service you're hosting viable by keeping inter-server communication to a minimum level. This will mean that any two connections that are related (such as two accounts that talk to each other a lot) will have to be on the same host.
You will need servers and network infrastructure that can handle this. You will need a subnet of ip addresses, each server will have to have stateless communication with the internet (i.e. your router will not be doing any NAT in order to not have to track 250,000+ connections).
You will have to talk to AOL. There is no way that AOL will be able to handle this level of connectivity without considering cutting your connection off. Any service of this scale would have to be negotiated with AOL so both you and they would be able to handle the connectivity.
There are i/o multiplexing technologies that you should investigate. Kqueue and epoll come to mind.
In order to write this massively concurrent and teleco grade solution, I would recommend investigating erlang. Erlang is designed for situations such as these (multi-server, massively-multi-client, massively-multithreaded telecommunications grade software). It is currently used for running Ericsson telephone exchanges.
While you can listen on a socket for multiple incoming connection requests, when the connection is established, it connects a unique port on the server to a unique port on the client. In order to multiplex a connection, you need to control both ends of the pipe and have a protocol that allows you to switch contexts from one virtual connection to another or use a stateless protocol that doesn't care about the client's identity. In the former case you'd need to implement it in the application layer so that you could reuse existing connections. In the latter case you could get by using a proxy that keeps track of which server response goes to which client. Since you're connecting to Yahoo Messenger, I don't think you'll be able to do this since it requires an authenticated connection and it assumes that each connection corresponds to a single user.
You can only multiplex multiple connections over a single socket if the other end supports such an operation.
In other words it's a function protocol - sockets don't have any native support for it.
I doubt yahoo messenger protocol has any support for it.
An alternative (to multiple IPs on a single NIC) is to design your own multiplexing protocol and have satellite servers that convert from the multiplex protocol to the yahoo protocol.
I'll trow in another approach you could consider (depending on how desperate you are).
Note that operating system TCP/IP implementations need to be general purpose, but you are only interested in a very specific use-case. So it might make sense to implement a cut-down version of TCP/IP (which only handles your use-case, but does that very well) in your application code.
For example, if you are using Linux, you could route a couple of IP addresses to a tun interface and have your application handle the IP packets for that tun interface. That way you can implement TCP/IP (optimised for your use-case) entirely in your application and avoid any operating system restriction on the number of open connections.
Of course, it's quite a bit of work doing the TCP/IP yourself, but it really depends on how desperate you are - i.e. how much hardware can you afford to throw at the problem.
500,000 arbitrary yahoo messenger connections - is your telco doing this on behalf of Yahoo? It seems like whatever solution has been in place for many years now should be scalable with the help of Moore's Law - and as far as I know all the IM clients have been pretty effective for a long time, and there's no pressing increase in demand that I can think of.
Why isn't this a reasonable problem to address with hardware plus traditional solutions?