I have recently started having my mongodb instance crash on an ubuntu machine at random times, it usually stays up for a day or so. The mongo log has no trace of the crash, just the last operation and when I restarted the server. I need some guidance in finding out the problem and the log doesn't have any information. Is there another log I should be looking at?
The setup is fairly straightforward, single instance (no sharding), of mongodb 2.2 running on an ubuntu box, with pretty much default install.
The only change I have done recently which seems to coincide with this in timing is I have replaced some simple map reduce execution with the aggregate framework.
Thanks.
MongoDB unofficially have Mtools try download and run with your logs in this tool. it can give you why it went down, how many time it restarted and many more details.
you can get this in github
Through Logs you can view the actual cause of crash
/var/log/mongodb/mongod.log ####Common path of logs
or use this command
db.adminCommand( { getLog : "global" } )
Related
I am using Meteor JS in production and it first looked like we suffered from a memory leak in our application, but now it seems like MongoDB is the culprit or perhaps Meteor?
We're heavy users of Meteor methods and don't manage connection to Mongo ourselves, since Meteor does that for us. However, in production it seems like mongo processes just hangs around forever and consumes more and more memory until the application becomes unresponsive.
As you clearly can see, there are a lot of mongo processes here, some that has been living for hours. Yesterday, after a reboot of the server the memory consumption was about 900MB, now it's closing onto 4GB.
Since rebooting every once in a while in unacceptable, what can we do to fix this? I don't really know where to start since Meteor manages connections to Mongo and I am a bit uncertain what's causing this issue.
Any tips or direction is helpful, even if it's just a tip on how to debug our application.
MeteorJS version: 1.10
Mongo version: 4.2.1
I'm situation were I am trying to get a broken member of a replica set back online. I have followed the normal procedure of deleting the data in /data/db and trying to sync. Also tried to move all the data from one secondary to the other and restart mongo. Each time I get the standard error above complaining about compatibility issues. Now the servers are the same version 3.6. So why was this an issue?... Must be that the featureCompatibilityVersion=3.6 while on the other server are set to featureCompatibilityVersion=3.4 which they are. However to see this the server needs to startup but since it kept restarting I could not check that. So I removed it from the replica set started it up and seen it was set to featureCompatibilityVersion=3.6. So I changed this to 3.4 and started it up again while part of replica set but still had the same issue. How do I get round this?
The other servers have featureCompatibilityVersion=3.4 so if I was syncing correctly would the other server not pull in admin collection which would have featureCompatibilityVersion=3.4. So why does this keep complaining?
If not how do you change this on a server which does not startup? Is there something in mongo config I can set?
Thanks for the help in advance.
here's probably not a helpful answer, just a sanity check: remove replSet parameter and spin up the mongod totally solo.... does that work ok?
I only suggest it cause I can't see what your issue is and maybe this would find something...wouldn't take long, may not help... the versioning I don't quite get...
I Came by a problem where i have an Ops Manager that suppose to run a MongoDB cluster as an automated cluster.
Suddenly the servers started going down, unexpectedly - while there are no errors in any of the log files indicating on when is the problem.
The Ops Manager gets stuck on the blue label
We are deploying your changes. This might take a few minutes
And it just never goes away.
Because this environment is based on the automation feature, the mms is managing the user on the servers and runs all of the processes from "mongod" which i can't access even as a Root (administrator).
As far as the Ops Manager goes it shows that a shard in a replica set is down although it's live, and thinks that a mongos that is dead is alive.
Has someone got into this situation before and may be able to help ?
Thanks,
Eliran.
Problem found: there was an ntp mismatch between the servers in the cluster somehow, so what happened was that the servers were not synced and everytime the ops manager did something it got responses with wrong times and could not use it's time limits.
After re-configuring all the ntp's back to the same one - everything got back to how it should have been :)
I'm running MongoDB 2.4.5 and recently I've started digging into Replica Set to get some kind of redundancy.
I started same mongo instance with --replSet parameter and also added an Arbiter to running Replica Set. What happened was writing to mongo slowed down significantly (from 15ms to 30-60ms, sometimes even around 300ms). As soon as I restarted it in not-replicaset-mode performance went back to normal.
I also set up the newest 3.0 version of MongoDB with no data and run same tester as before and the result was quite similar - writes were at least 50% slower while running the ReplicaSet mode.
I could not find many examples of such behaviour online so I guess something is wrong with my mongo configuration or OS configuration.
Any ideas? Thanks for help.
It sounds like you're using "replica acknowledged" write concern, which means that the operation will not return until the data has been written to both the primary and replica. The write concern can be set when doing any write operation (from 2.6 onwards - it looks from the 2.4 documentation that calling getLastError causes a write concern of replica acknowledged in 2.4, are you doing that in your test code?).
Read this section (v3)) or this section (v2.4) of the MongoDB documentation to understand the implications of different write concerns and try explicitly setting it to Acknowledged.
Okay so the problem was C# library. I used a native C# driver (works fine even with 2.4.5 MongoDB) and there seems to be no difference in performance. Thanks for help.
Our mongodb cluster in production, is a sharded cluster with 3 replica sets with 3 server each one and, of course, another 3 config servers.
We also have 14 webservers that connect directly to mongoDb throw the mongos process that are running in each of this webservers (clients).
The entire cluster receive 5000 inserts per minute.
Sometimes, we start getting exceptions from our java applications when it wants to perform operations to the mongoDb.
This is the stackTrace:
caused by com.mongodb.MongoException: writeback
com.mongodb.CommandResult.getException(CommandResult.java:100)
com.mongodb.CommandResult.throwOnError(CommandResult.java:134)
com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:142)
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:183)
com.mongodb.DBTCPConnector.say(DBTCPConnector.java:155)
com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:270)
com.mongodb.DBApiLayer$MyCollection.insert(DBApiLayer.java:226)
com.mongodb.DBCollection.insert(DBCollection.java:147)
com.mongodb.DBCollection.insert(DBCollection.java:90)
com.mongodb.DBCollection$insert$0.call(Unknown Source)
If I check the mongos process throw the rest _status command that it provides, it returns a 200OK. We could fix the problem restarting the tomcat that we are using and restarting the mongos process but I would like to find a final solution to this problem. It's not a happy solution to have to restart everything in the middle of the night.
When this error happens, maybe 2 or 3 another webservers got the same error at the same time, so I imagine that there is a problem in the entire mongoDb cluster, no a problem in a single isolated webserver.
Does anyone know why mongo returns a writeback error? and how to fix it?
I'm using mongoDb 2.2.0.
Thanks in advance.
Fer
I believe you are seeing the Writeback error "leak" into the getLastError output and then continue to be reported even when the operation in question had not errored. This was an issue in the earlier versions of MongoDB 2.2, and has since been fixed, see:
https://jira.mongodb.org/browse/SERVER-7958
https://jira.mongodb.org/browse/SERVER-7369
https://jira.mongodb.org/browse/SERVER-4532
As of writing this answer, I would recommend 2.2.4, but basically whatever the latest 2.2 branch is, to resolve your problem.