I am trying to perform performance testing for one of my application using MongoDB. I am using 2.5.5 development version. Sharding works fine when I try to read and write data using mongos.
To perform performance testing I need to start 600-700 mongoconnection threads to the mongos. Each thread queries around 2000 documents which is distributed on two shards. This test runs fine for few minutes but after sometime it stops working with the error "Connection refused by one of the shard". Looking closely at it I found that server runs out of ports when these many threads request data.
Could anyone please have a look and let me know if it is a MongoDB bug in the dev version or is it something which I am doing wrong while connecting to database.
your help will be much appreciated.
Thanks,
Vibhu
Related
I am using Meteor JS in production and it first looked like we suffered from a memory leak in our application, but now it seems like MongoDB is the culprit or perhaps Meteor?
We're heavy users of Meteor methods and don't manage connection to Mongo ourselves, since Meteor does that for us. However, in production it seems like mongo processes just hangs around forever and consumes more and more memory until the application becomes unresponsive.
As you clearly can see, there are a lot of mongo processes here, some that has been living for hours. Yesterday, after a reboot of the server the memory consumption was about 900MB, now it's closing onto 4GB.
Since rebooting every once in a while in unacceptable, what can we do to fix this? I don't really know where to start since Meteor manages connections to Mongo and I am a bit uncertain what's causing this issue.
Any tips or direction is helpful, even if it's just a tip on how to debug our application.
MeteorJS version: 1.10
Mongo version: 4.2.1
I'm running MongoDB 2.4.5 and recently I've started digging into Replica Set to get some kind of redundancy.
I started same mongo instance with --replSet parameter and also added an Arbiter to running Replica Set. What happened was writing to mongo slowed down significantly (from 15ms to 30-60ms, sometimes even around 300ms). As soon as I restarted it in not-replicaset-mode performance went back to normal.
I also set up the newest 3.0 version of MongoDB with no data and run same tester as before and the result was quite similar - writes were at least 50% slower while running the ReplicaSet mode.
I could not find many examples of such behaviour online so I guess something is wrong with my mongo configuration or OS configuration.
Any ideas? Thanks for help.
It sounds like you're using "replica acknowledged" write concern, which means that the operation will not return until the data has been written to both the primary and replica. The write concern can be set when doing any write operation (from 2.6 onwards - it looks from the 2.4 documentation that calling getLastError causes a write concern of replica acknowledged in 2.4, are you doing that in your test code?).
Read this section (v3)) or this section (v2.4) of the MongoDB documentation to understand the implications of different write concerns and try explicitly setting it to Acknowledged.
Okay so the problem was C# library. I used a native C# driver (works fine even with 2.4.5 MongoDB) and there seems to be no difference in performance. Thanks for help.
I'm testing the performance of MongoDB on a single system using YCSB. I'd like to get a sense of the performance using SSDs compared to spinning disks.
I have CentOS, MongoDB, and YCSB installed. I have stumbled around a bit with basic examples, but have yet to see a step by step of starting from this setup to loading to running to reviewing. I keep seeing bits and pieces, but not enough to get me up and running.
If anyone could please provide a command line for these steps, it would be most appreciated!
Thanks
Here's a guide on how to run Yahoo! Cloud System Benchmark (YCSB) using Mongodb.
https://github.com/samanca/YCSB/tree/master/mongodb
https://github.com/brianfrankcooper/YCSB/wiki
Working example using Python and Java to test Mongodb:
https://github.com/richcar58/MongoDBTools/blob/master/RunYcsb/runycsb/fabfile.py
Our mongodb cluster in production, is a sharded cluster with 3 replica sets with 3 server each one and, of course, another 3 config servers.
We also have 14 webservers that connect directly to mongoDb throw the mongos process that are running in each of this webservers (clients).
The entire cluster receive 5000 inserts per minute.
Sometimes, we start getting exceptions from our java applications when it wants to perform operations to the mongoDb.
This is the stackTrace:
caused by com.mongodb.MongoException: writeback
com.mongodb.CommandResult.getException(CommandResult.java:100)
com.mongodb.CommandResult.throwOnError(CommandResult.java:134)
com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:142)
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:183)
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)
com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:270)
com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:226)
com.mongodb.DBCollection.insert(DBCollection.java:147)
com.mongodb.DBCollection.insert(DBCollection.java:90)
com.mongodb.DBCollection$insert$0.call(Unknown Source)
If I check the mongos process throw the rest _status command that it provides, it returns a 200OK. We could fix the problem restarting the tomcat that we are using and restarting the mongos process but I would like to find a final solution to this problem. It's not a happy solution to have to restart everything in the middle of the night.
When this error happens, maybe 2 or 3 another webservers got the same error at the same time, so I imagine that there is a problem in the entire mongoDb cluster, no a problem in a single isolated webserver.
Does anyone know why mongo returns a writeback error? and how to fix it?
I'm using mongoDb 2.2.0.
Thanks in advance.
Fer
I believe you are seeing the Writeback error "leak" into the getLastError output and then continue to be reported even when the operation in question had not errored. This was an issue in the earlier versions of MongoDB 2.2, and has since been fixed, see:
https://jira.mongodb.org/browse/SERVER-7958
https://jira.mongodb.org/browse/SERVER-7369
https://jira.mongodb.org/browse/SERVER-4532
As of writing this answer, I would recommend 2.2.4, but basically whatever the latest 2.2 branch is, to resolve your problem.
I have used NORM driver in production. New year holidays - it is pretty cool, so my project get high loading and i want to set up a replication set, but have a problem - Norm does not support replication set :( . as far as i understand sharding too?
Help me :) Who did use mongodb csharp or official 10gen driver with replset? Is there any problem on production? If i choose another driver I'll have to rewrite the repository, but I do not want it to be in vain. Is there some issues?
Sharding should not depend on driver-specific support. When you shard, you connect to a router application mongos and this router behaves exactly like mongod.
So you should be able to shard. But you will probably need to change the "connection string". The suggested setup is to have one mongos per application server (instead of your current single mongod).