I'm trying to coordinate a triggered event across many smartphones within as small a time-frame as possible (ideally have them start within half a second or less of each other).
The problem:
From my experience, the local time value on the devices cant be relied on, and additionally latency can contribute to issues with syncing a value for the current time (imagine trying to get the updated time from some remote source and eventually approaching a decently close idea of the current time locally based on that remote source, ideally within a fraction of a second of said source).
Are there any established techniques, mechanisms, or more accurate sources of some time value reference point that would allow for a planned event to be triggered on multiple devices within a fraction of a second of one another? The more I search, the more I realize this is not a trivial issue, however I thought it would be worth it to query the great minds of stackoverflow.
Thanks in advance for any and all help.
I've developed a technology that achieves synchronization of smartphones down to 10 milliseconds. Each devices takes the UTC time from many clocks and makes a non-trivial convolution.
I have applied this to a massive event (http://massivesymphony.org) and I'm now providing the technology for several corporate events.
In case you are interested in more details, my contact is
José I. Latorre
Dept. of Physics, U. Barcelona
j.i.latorre#gmail.com
Related
For various reasons I need to get from the iPhone the current date and time that can't be meddled with by the user. Yes, I've seen how one can check a server (e.g., here), but that's not invulnerable to tampering if you take a moment to reflect.
There are two knee-jerk reactions I'm expecting to hear:
Use the GPS time.
It can't be done.
In answer to another question, I've described my researches into this matter. To summarize them:
The GPS time shifts with the user-defined settings.
The iPhone definitely has an internal tamper-proof time and date, as shown when date-time reverts after Set Automatically in Settings > General > Time & Date is turned back to on even in a fallout shelter.
What I want to know is how to access this tamper-proof time.
Edit
Just to be clear, the server-based solution is not suitable. For one, it could be faked. For another, the app needs to work without a network connection.
Assuming you always have Internet available, you could implement a class or object that connects to a remote Network Time Protocol server.
Here's an open source GitHub project that should get you started, and the related StackOverflow question I found it at.
This is a bit of an open ended question, but I'm looking for an open ended answer. I'm looking for a resource that can help explain how to benchmark different systems, but more importantly how to analyze the data and make intelligent choices based on the results.
In my specific case, I have a 4 server setup that includes mongo that serves as the backend for an iOS game. All servers are running Ubuntu 11.10. I've read numerous articles that make suggestions like "if CPU utilization is high, make this change." As a new-comer to backend architecture, I have no concept of what "high CPU utilization" is.
I am using Mongo's monitoring service (MMS), and I am gathering some information about it, but I don't know how to make choices or identify bottlenecks. Other servers serve requests from the game client to mongo and back, but I'm not quite sure how I should be benchmarking or logging important information from them. I'm also using Amazon's EC2 to host all of my instances, which also provides some information.
So, some questions:
What statistics are important to log on a backend setup? (CPU, RAM, etc)
What is a good way to monitor those statistics?
How do I analyze the statistics? (RAM usage is high/read requests are low, etc)
What tips should I know before trying to create a stress-test or benchmarking script for my architecture?
Again, if there is a resource that answers many of these questions, I don't need an explanation here, I was just unable to find one on my own.
If more details regarding my setup are helpful, I can provide those as well.
Thanks!
I like to think of performance testing as a mini-project that is undertaken because there is a real-world need. Start with the problem to be solved: is the concern that users will have a poor gaming experience if the response time is too slow? Or is the concern that too much money will be spent on unnecessary server hardware?
In short, what is driving the need for the performance testing? This exercise is sometimes called "establishing the problem to be solved." It is about the goal to be achieved-- because if there is not goal, why go through all the work of testing the performance? Establishing the problem to be solved will eventually drive what to measure and how to measure it.
After the problem is established, a next set is to write down what questions have to be answered to know when the goal is met. For example, if the goal is to ensure the response times are low enough to provide a good gaming experience, some questions that come to mind are:
What is the maximum response time before the gaming experience becomes unacceptably bad?
What is the maximum response time that is indistinguishable from zero? That is, if 200 ms response time feels the same to a user as a 1 ms response time, then the lower bound for response time is 200 ms.
What client hardware must be considered? For example, if the game only runs on iOS 5 devices, then testing an original iPhone is not necessary because the original iPhone cannot run iOS 5.
These are just a few question I came up with as examples. A full, thoughtful list might look a lot different.
After writing down the questions, the next step is decide what metrics will provide answers to the questions. You have probably comes across a lot metrics already: response time, transaction per second, RAM usage, CPU utilization, and so on.
After choosing some appropriate metrics, write some test scenarios. These are the plain English descriptions of the tests. For example, a test scenario might involve simulating a certain number of games simultaneously with specific devices or specific versions of iOS for a particular combination of game settings on a particular level of the game.
Once the scenarios are written, consider writing the test scripts for whatever tool is simulating the server work loads. Then run the scripts to establish a baseline for the selected metrics.
After a baseline is established, change parameters and chart the results. For example, if one of the selected metrics is CPU utilization versus the number of of TCP packets entering the server second, make a graph to find out how utilization changes as packets/second goes from 0 to 10,000.
In general, observe what happens to performance as the independent variables of the experiment are adjusted. Use this hard data to answer the questions created earlier in the process.
I did a Google search on "software performance testing methodology" and found a couple of good links:
Check out this white paper Performance Testing Methodology by Johann du Plessis
Have a look at the Methodology section of this Wikipedia article.
We’re creating a massively-multiplayer social game. We expect up to 1 million concurrent users. The game is not real-time, instead it’s turn-based. We need reliable messaging between our clients and the server, preferably over HTTP protocol.
Besides the multiplayer functionality, we’ll also need a content delivery service.
Could you please recommend a server-side technology for us, so we’ll start searching for the right people to hire?
Is it correct assumption that no single server will hold that amount of load so it must scale horizontally?
Will Windows Azure do the job?
Thanks in advance.
Hmmm... gaming, concurrency, server?
G-WAN (200 KB, full-ANSI C scripts included).
This is the best candidate -by far. And it lets you grow horizontally with load-balancing as time goes (you will not have 1 million users the day you ship the game).
I know they are workng on applets (client-side) so you might benefit asking them the question.
[quote]a million concurrent users IS NOT a real number by any means[/quote]
There are games that have this concurrency, and more. Most of the popular Facebook games do, while they have their 15 days in the sun. That being said, having to solve that problem is a nice problem to have :-)
It's probably possible to write such a system on Azure, but you'd probably be piloting in uncharted waters, and you'd also have to pay Microsoft for the hosting. Compare to Amazon ECC for pricing, for example, and perhaps another approach would be better.
Other technologies to consider, depending on what it is you're really trying to do:
- J2EE
- Erlang/OTP
- Python/Twisted
Also, the networking and multiplayer game FAQ on gamedev.net: http://www.gamedev.net/community/forums/showfaq.asp?forum_id=15
Is it [a] correct assumption that no single server will hold that amount of load so it must scale horizontally?
Yes. It depends on how much work the server has to do per person, but I'd say 1 million concurrent users would require more than one server.
Will Windows Azure do the job?
Windows Azure will provide the computers and the storage for a fee. You have to provide the software and make sure the software can scale horizontally.
Is it correct assumption that no
single server will hold that amount of
load so it must scale horizontally?
No, that is nao avalid assumption. There are servers that are HUGH - 1000+ processors (not on a cluster). Also, a million concurrent users IS NOT a real number by any means - that would be way too much a slice of the concurrent facebook users. And it totally depends no what you do in your game. TUrn based could be chess, and I would not have a problem hostin 1.000.000 concurrent chess boards on a high end server with let's say 256gb memory.
Realistically, though, you possibly will scale horizontaly. First, it makes no sense to ahve a million people in one game / world (even eve online scales horizontally by solar system), second it is likely cheaper than buying a super big computer.
Will Windows Azure do the job?
Hahaha. Seriously. Scaling horizontally - yes.
Look at the price, calcualte up an nistance for a month, compare to dedicated server and laugh on the way to the shop. Nice for very varsying load, bad for base load.
Comapre mid range server (8-12cores, 64gb RAM) to an azure instance and iti s clear ONE azure instance is not going to compare.
My users use the site pretty equally 24/7. Is there a meme for build timing?
International audience, single cluster of servers on eastern time, but gets hit well into the morning, by international clients.
1 db, several web servers, so if no db, simple, whenever.
But when the site has to come down, when would you, as a programmer be least mad to see SO be down for say 15 minutes.
If there's truly no good time from the users' perspective, then I'd suggest doing it when your team has the most time to recover from any build-related disaster.
Here's what I have done and its worked well for me:
Get a site traffic analysis tool
which will graph hourly user load
Select low-point in graph for doing
updates
If you're small, then yeah, find when your lowest usage period is, and do it then (for us personally, usually around 1AM-3AM PST is the lowest dip...but it never drops to 0 of course). Once you start growing to having a larger userbase, if you want people to take you seriously you'll need to design your application such that you can upgrade without downtime. This is not simple, and it often involves having multiple servers.
I've spent ages trying to get our application to this point, the best I've come up with so far is for a couple hours run both the old version and new version at the same time. Users logged in at the time of the switchover stay on the old version, until they log out. Next time they come in they go to the new version. Any users coming on after the switchover get sent straight to the new version. It's still not foolproof, but it's pretty good.
What kind of an application is it? Most sites that I use tend to update around 2AM or 3AM.
Use a second site, and hotswap as needed.
The issue with hot-swapping, is database would still be shared, and breaking changes would bring stand in down as well.
I guess you have to ask your clients.
In any case, there's the wee hours of the morning. If you're talking about a locally available website, I do not think users will mind if they get an "under maintenance" notice at 2 am in their time zone.
Depends on your location: 4AM East Coast/1AM West Coast is typlically the lightest time.
Pick a few times that you'd like to do it and offer them as choices to the decider-types. Whatever you do, put up a "down for routine maintenance" page while you deploy.
Check the time of least usage
Clone/copy/update latest production code to another directory
If there exists any database migrations to be done, perform any that are required, and non conflicting with the old code base
At time of least usage, move symlink to point to latest code
First use an analysis tool to try and determine your typically "light" traffic times. Depending on the site and your location in the world in comparison to most of your users, it could be 4am, it could be 1pm, who knows. Then, once you have a good timeframe nailed down, make sure to have your deployment process as automated as possible, so that it happens quickly to minimize the downtime of your site.
Does anyone know how sites that have a real-time feed of a lot of data work? I am referring to something like a stock site, where they can tell you in real time (well, 20 minute delay mostly, but still real-time - 20 minutes as I understand it).
They have thousands of data pieces delivered to them every second, I would imagine: MSFT 25.00 +.23 VOL 12000 ???? for each stock that had a change during some interval.
So, is there just a constant feed of small pushes going on? Or do you think a site will pull from the place that has the real data and say "give me all changes since 12:23:45 CST to now" type query?
I ask this because at work we might have a situation where we need to have at our application's fingertips real time information like this, and it won't make sense to hit our third party provider over and over and over again every second...
Generally there is a server/client protocol defined between the 2 parties. In the company I work for the connection is maintained at all times.
Here is info on real time data feeds to go with your stock example
NYSE,NASDAQ
It is common for data providers to also have FTP sites with (delayed) batched data. One that comes to mind is the NWS EMWIN
Sites like Twitter feed data to certain approved sites in real-time via XMPP (Wiki link).
In the broadest terms, a push model is going to be the best way of achieving "real time" transfer, particularly if you're talking about a large amount of data.
However you do always have a problem when using a purely push model of how to recover from missed data.
Depending on the nature of your data that may not be a problem (thinking of video delivery as an analogue, where the amount of data is huge but there is sufficient redundancy for it to recover from missing data). And if you have any control over the data you may be able to build some redundancy in. For example, on every change event you can provide absolute values rather than changes, or previous value and new value.
I've done this making an attempt to retrieve the stock quote from the source, and falling back to a timestamped on-disk cache of the quote when the main source fails or times out.