What's the server setup used for data communication between a mobile app (e.g. Android) and a server? Would an Apache http server with PHP be sufficient? Assume you need to transfer 1 GB of binary or text data once per day, or spread it out over small chunks and transfer at different times of the day. What about Tomcat + Java servlet?
All suggestions are welcome.
From the server side PHP/Apache should be fine, but is this all going to the same client?
Related
I am building a distributed system that consists of potentially millions of clients which all need to keep an open (preferrably HTTP) connection to wait for a command from the server (which is running somewhere else). The load of messages / commmands will not be very high, maybe one message / sec / 1000 clients which means it would be 1000 msg/sec # 1 million clients. => it's basically about the concurrent connections.
The requirements are simple too. One way messaging (server->client), only 1 client per "channel".
I am pretty open in terms of technology (xmpp / websockets / comet / ...). I am using Google App Engine as server, but their "channels" won't work for me unfortunately (too low quotas and no Java client). XMPP was an option but is quite expensive. So far I was using URL Fetch & pubnub, but they just started charging for connections (big time).
So:
Does anyone know of a service out there that can do that for me in an affordable way? Most I have found restrict or heavily charge for connections.
Any experience with implementing such a server yourself? I have actually done that already and it works pretty well (based on Tomcat & NIO) but I haven't had the time yet to actually set up a large load test environment (partially because this is still a fallback solution, I'd prefer a battle hardened msg server). Any experience to how many users you get per GB? Any hard limits?
My architecture also allows to fragment the msg servers, but I'd like to maximize the concurrent connections because the msg processing CPU overhead is minimal.
I have meanwhile implemented my own message server using netty.io. Netty makes use of Java NIO and scales extremely well. For idle connections I get a memory footprint of 500 bytes per connection. I am doing only very simple message forwarding (no caching, storage or other fancy stuff) but with that am easily getting 1000 - 1500 msg / sec (each half a KB) on the small amazon instance (1ECU / 1.6GB).
Otherwise if you are looking for a (paid) service then I can recommend spire.io (they do not charge for connections but have a higher price per message) or pubnub (they do charge for connections but are cheaper per message).
You have to look more in architecture of making such environment.
First of all, if you will write sockets management by yourself, then don't use Thread per Client Socket. Use Asynchronous methods for receiving and sending data.
WebSockets might be too heavy if your messages are small. Because it implements framing, which has to be applied to each message for each socket individually (caching can be used for different versions of WebSockets protocols), that makes them slower to process both directions: for receive and for send, especially because of data masking.
It is possible to create millions of sockets, but only most advanced technologies are capable to do so. Erlang is able to handle millions connections, and is pretty scalable.
If you would like to have millions of connections using other higher level technologies, then you need to think about clustering of what you are trying to accomplish.
For example using gateway server that will keep track of all processing servers. And have data of them (IP, ports, load (if it will be one internal network, firewalling and port forwarding might be handy here).
Client software connects to that gateway server, gateway server checks the least loaded server and sends ip and port to client. Client creates connection directly to working server using provided address.
That way you will have gateway which as well can handle authorization, and wont hold connections for long, so one of them might be enough. And many workers that are doing publishing of data and keeping connections.
This is very related to your needs, and might not be suitable for your solutions.
How do they do this? I would like to have web pages with data fields that change in real time as a person views the web page. Here is an example.
How do they do this? JQuery? PHP?
I need to connect my field data to mySQL database.
There are two approaches:
Polling
Client requests data on a regular basis. Uses network and server resources even when there is no data. Data is not quite 'live'. Extremely easy to implement, but not scalable.
Push
Server sends data to the client, so client can simply wait for it to arrive instead of checking regularly.
This can be achieved with a socket connection (since you are talking about web pages, this doesn't really apply unless you are using Flash, since support for sockets in the browser in the browser is currently immature) - or by using the technique known as 'comet'.
Neither socket connections nor comet are particularly scalable if the server end is implemented naively.
- To do live data on a large scale (without buying a boat load of hardware) you will need server software that does not use a thread for each client.
I did it with JavaScript timer set execution in milliseconds, each time timer executed function that queried Server with Ajax and returned value(possibly JSON format), then you you update your field with the value. I did it each 5 sec and it works perfectly. In ASP.NET I think it called Ajax Timer Control.
There are two things needed to do this:
Code that runs on the browser to fetch the latest data. This could be Javascript or something running in a plugin such as Silverlight or Flash. This will need to periodically request updated content from the server.
Which leads to a need for...
Code that runs on the server to retrieve and return the latest data (from the database). This could be created with any server sided scripting language.
I am tasked with creating a text messaging system with low bandwidth server to server connections. The other developers already use protobuf to send data for other parts of the system between these same server locations, and it would be helpful to continue that trend for the text messaging portion. Server to client connections are not bandwidth constrained. It would be great to be able to use an unmodified chat client and openfire xmpp server.
What is better to program in this situation, a component for openfire or a transport for Kraken?
Have you tried enabling XEP-138 compression on the server-to-server link? Even if OpenFire doesn't support XEP-138, it will be easy to add, and should provide better results that almost any naive translation to protobufs.
I want to take a new streaming server for my website which generally holds videos and audio files. But how do we maintain backup of the streaming server if storage size is increasing day by day.
Generally Database server, like Sql Server, backup can be easily taken and restored very easily as it does not occupy much space for medium range application.
On the other hand how can we take backup of streaming server. If the server fails, the there should be an alternative server / solution that should decrease downtime of the server.
How the back-end architecture of YouTube built to handle this.
The backend architecture of YouTube probably uses Google's BigTable which stores objects redundantly over several different servers. If you are using a single server solution your only real options are backing up to an attached disk, backing up to another server or using an offsite storage system like Amazon S3 (which you could then use with their CDN to do basic HTTP streaming of content in the case of a failure).
I've been talking with someone about the possibility of a iPhone development contract gig. All I really know at this point is that there is a company that wants to make an iPhone app that will hit their internal database. I'm not sure what the database type is( Oracle, MySQL, etc...).
I've wanted to know if the database type was Oracle or MySQL if there is a big learning curve for connecting to one of these across the internet?
If it's a real pain I may do more research before accepting the conract.
I would advise against directly accessing the database from the iPhone application.
Usually, you would create a web service which accesses the database, and then you consume that web service from the iPhone application.
Create a web service. This allows you to make the iphone app more of a thin client. Let the application push commands to the web service for processing and interaction with the database returning only the data needed by the app.
This option is better for the app, the database, and the customer's security.
You can easily perform the connection over the internet, the same way you would locally, but you are opening the database up to attacks if it will accept communication from any remote IP address. Typically you will just connect via a socket open to the server's remote IP address over the open port, MySQL's default port is 3306.
I would recommend against this sort of system in general unless there is some critical reason they want their internal database exposed to the world's hacker community.
What I am doing is creating a web service using Sinatra to access the online database.
Those answers from 2009 are mostly obsolete now.
http://ODBCrouter.com/ipad (new) has XCode client-side ODBC libraries, header files and multi-threaded Objective C objects that let your apps send SQL to server-side ODBC drivers and get back binary results! This reduces the need to stop and separately maintain SOAP/REST servers that can get pretty frightening anyway after a while maintaining it.
The XML schemes were okay for transferring static configurations to mobile devices "every once in a while", but XML was meant for infrequent inter-company type transfers in a "server environment" (with power cords, wired networks and air conditioning) and is definitely not efficient for frequent database queries coming in from n-copies of a mobile app. There are third-party JSON libraries that help things, but even with JSON, everything has to be encoded (and decoded) from the binary representation in the database to text representation on the server (only fine if it's going to be shown to the user in a web browser anyway, but not fine if the mobile app is going to translate it right back into binary so that it can perform calculations "behind the scenes" to what is going on with the user). Aside from the higher network overhead and battery power the mobile CPU will draw with XML and JSON, it will also make you buy more RAM and CPU power on the back-end server faster than just using an ODBC connection to the database.