What are ways to handle too many database connections? - mongodb

I have a website that caters to daily fantasy sports players and on days with big slates (i.e. a lot of games), the site gets hammered before the lineups get locked. "Hammered" in my case means 50-100 concurrent users, which I'm sure doesn't sound like a lot, but it seems to kill my database host (Mongolab). During "normal" operations, the max number of open connections I see is less than 500, but during these peak times, there can be upwards of 10,000 open connections!
I think what happens is that users open the site, see that it is not responding quickly enough, and then they proceed to open a bunch more sessions thinking one of them will work. This causes the downward spiral.
Last week I added multi-key indexes which helped quite a bit, and I thought had "solved" the issues. But today the site got hammered and even the new indexes haven't proved to be enough. What else can I do?

Related

General question about components updating in live time

Im curious how this is accomplished. Let's say a platform has a voting system on a webpage. A piece of content can be voted on positive or negative. As a user scrolls through the platform, they can see a live change in the amount of votes each piece of content has.
I figured this would be done one of two ways but I am not sure which is the most accepted one.
A timer to call the server ever x amount of seconds and update the components on a loop.
A socket to connect to the backend every time a user opens said page.
I wont know which is best, but I feel like 2 probably wins considering its a little less repetitive.

Google Adsense revenue slowing

I don't know where to ask this, so I'm going to just ask it here and handle the backlash when ever it comes.
First let me state that I'm familiar with the recent issues relating to Pepsi and YouTube how ever what I am currently unsure of is, does this effect website publishers also, or has my traffic just all around stopped converting?
I'm currently getting around 300,000 impressions per day and my revenue has dropped to as low as 0.19cpc when before it was sitting almost around a dollar for every registered click.
Question basically comes down to, are you guys getting effected by this also? What can I do in the mean time if this is in correlation with the advertisers opting out, because it's getting really hard to manage servers with no revenue.
Yes it seems to have had a drastic impact on my earnings as well, although I'm not doing anywhere close to the impressions you're doing.
My cpc and rpm have hit an all time low and its disastrous. I'm seeing the same cpc as you btw, we're not alone other publishers are complaining about the same issue.
I'm considering dropping a few adsense units till i see some better figures.

How long does an app in development get banned from Facebooks if it exceeds limits?

I have an app I'm developing against Facebook that timed out a few hours ago during my first production use. Of course I tried to get it do too much and the http call timed out. So, I rewrote what I was doing to use threaded connections, which sped up the interaction significantly! However, I was so engrossed in getting my interaction to speed up (it equated to about 25-50 calls, not exactly sure, I was expecting 25 but some of my results show it was 50 times), I didn't even stop to think about how fast I was hitting facebook.
So, I started getting the "Uncaught OAuthException: It looks like you were misusing this feature by going too fast. You窶况e been blocked from using it." which is what I now get even if I try to run my program with only 1 hit. I've added a sleep into my system to limit the hits at 1/second, but I'm concerned that my app (that was not making public posts so no one could have been bothered by them) is now forever banned from facebook, as it says I'm banned from the feature with a reference to learn about blocks in the Help Center; except I can't find any reference in the Help Center to my specific situation.
Does anyone know how long my app is out of commission?
And what are the specific (reference please, because I've search the hell out of fb and can't find one) limits regarding speed at which you can access facebook?
It depends on what has blocked you. In this case it was a spam bot that stopped me from posting comments into a group. Apparently there is a non-specific number of times you can post comments in a group in a short amount of time. The amount varies, but hovers around 150ish give or take 50 (at the time of my tests).
The ban appeared to be consistently set to about 19 hours at that time (May 2014). I've confirmed by continued testing in test groups and subsequent bans. However, Facebook developers are unable to give a solid set of numbers as they say it's controlled by a spam algorithm which changes based on server usage. So, 150 comments within about 3 minutes = ban for about 19 hours.

Memcached - Pros and Cons

We have a website at swalif.com which is like a news website based on forums. We are currently using a mysql database and things are getting slow. We decided to go with the Sphinx Search Server to speedup things and its been going quiet well.
Recently we heard of something called 'memcached' and just going through it we think we should look into it before moving to a search server completely.
My question is what are the pros and cons of using 'memcached' as it is a fairly new topic to us.
Thanking you
I just got my site set up with memcached a couple months back and it's amazing. The pros are rather obvious. It can be used to cache information that is, perhaps difficult to gather. The best example is an expensive mysql query. Check your slow query log, that would be a good starting point for things to parts to target. I had this one main page that took 2.5 seconds to echo from the server (horrible, I know). I had thought about changing the way it was written and it would have been very complicated. I put in memcached on the "difficult" parts of that page and now it's down to 0.001 seconds to parse. It's just amazing.
There is one main con that I've run into. If you update your content, you have to delete all associated keys related to that new content so that your front end will refetch the data and cache the new data. If not, you get stale content. I have tens of thousands of entries in my memcached and it's difficult to delete all the appropriate ones. If you don't, you'll get old content. One solution is to just set your key expiration duration to something short (24 hours). If you do that, you know that your site will reflect the newest content, at worst, 24 hours after a change. So if you can live with that, this problem is rather moot.
Bottom line, it's one of the best tools I've ever seen. It took me less than a day to install it on the lions share of my rather big site and the impact was tremendous.

What time should I build to production?

My users use the site pretty equally 24/7. Is there a meme for build timing?
International audience, single cluster of servers on eastern time, but gets hit well into the morning, by international clients.
1 db, several web servers, so if no db, simple, whenever.
But when the site has to come down, when would you, as a programmer be least mad to see SO be down for say 15 minutes.
If there's truly no good time from the users' perspective, then I'd suggest doing it when your team has the most time to recover from any build-related disaster.
Here's what I have done and its worked well for me:
Get a site traffic analysis tool
which will graph hourly user load
Select low-point in graph for doing
updates
If you're small, then yeah, find when your lowest usage period is, and do it then (for us personally, usually around 1AM-3AM PST is the lowest dip...but it never drops to 0 of course). Once you start growing to having a larger userbase, if you want people to take you seriously you'll need to design your application such that you can upgrade without downtime. This is not simple, and it often involves having multiple servers.
I've spent ages trying to get our application to this point, the best I've come up with so far is for a couple hours run both the old version and new version at the same time. Users logged in at the time of the switchover stay on the old version, until they log out. Next time they come in they go to the new version. Any users coming on after the switchover get sent straight to the new version. It's still not foolproof, but it's pretty good.
What kind of an application is it? Most sites that I use tend to update around 2AM or 3AM.
Use a second site, and hotswap as needed.
The issue with hot-swapping, is database would still be shared, and breaking changes would bring stand in down as well.
I guess you have to ask your clients.
In any case, there's the wee hours of the morning. If you're talking about a locally available website, I do not think users will mind if they get an "under maintenance" notice at 2 am in their time zone.
Depends on your location: 4AM East Coast/1AM West Coast is typlically the lightest time.
Pick a few times that you'd like to do it and offer them as choices to the decider-types. Whatever you do, put up a "down for routine maintenance" page while you deploy.
Check the time of least usage
Clone/copy/update latest production code to another directory
If there exists any database migrations to be done, perform any that are required, and non conflicting with the old code base
At time of least usage, move symlink to point to latest code
First use an analysis tool to try and determine your typically "light" traffic times. Depending on the site and your location in the world in comparison to most of your users, it could be 4am, it could be 1pm, who knows. Then, once you have a good timeframe nailed down, make sure to have your deployment process as automated as possible, so that it happens quickly to minimize the downtime of your site.