I tried doing a db.repairDatabase() command from a mongo shell on a healthy but large MongoDB database. It was running for about 10 hours and it still did not complete. For better or worse, I hit Ctrl-C to cancel it.
It appears that the cluster has been left in some locked state. Commands like "show dbs" all fail with "Operation timed out":
mongos> show dbs
2016-06-10T09:38:10.179-0400 E QUERY [thread1] Error: listDatabases failed:{ "code" : 50, "ok" : 0, "errmsg" : "Operation timed out" } :
_getErrorWithCode#src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs#src/mongo/shell/mongo.js:62:1
shellHelper.show#src/mongo/shell/utils.js:760:19
shellHelper#src/mongo/shell/utils.js:650:15
#(shellhelp2):1:1
It has been like this for about 10 more hours now after I killed the db.repairDatabase().
What is the correct way to recover from this?
My cluster info: I am running MongoDB 3.2.5 everywhere. I have 3 config servers, 11 shards, each shard is a replica set consisting of 2 nodes plus an arbiter. And I have about 40 nodes running mongos instances. The 3 config servers are still 3.0-style (not yet upgraded to replica-set).
Well for what it's worth I was able to bring the cluster back as follows:
Restarted all mongos services.
Restarted all mongod arbiters.
Restarted mongod for all 3 config servers.
Restarted mongod for 1 node from each of my 11 shards' replica sets.
Restarted mongod for the other 1 node from each of my 11 shards' replica sets.
Steps 1 thru 4 didn't fix anything.
But after I ran step 5 I was able to once again use all the databases. Things seem to be back to normal now.
Related
I am trying to accomplish sharding within 2 machines with config server, router, 1 shard in machine A and another shard in machine B. I am finding it hard to do this as I am a beginner and also can't find much documentation/ tutorials online. I have started a two mongod instances one as config server and another as shard, but clueless on how to proceed.
Below is the sharding configuration in two of my mongod (config and shard ) conf files:
Config server:
sharding:
clusterRole: configsvr
Shard:
sharding:
clusterRole : shardsvr
As per the documentation , the next step is to execute the command rs.initiate(), but I don't require replication. I still tried to execute just in case and received below error:
{
"ok" : 0,
"errmsg" : "This node was not started with the replSet option",
"code" : 76,
"codeName" : "NoReplicationEnabled"
}
Is it mandatory to have replication while sharding? How to do sharding without replication within 2 machines?
That's not possible, see sharding Options:
Note
Setting sharding.clusterRole requires the mongod instance to be
running with replication. To deploy the instance as a replica set
member, use the replSetName setting and specify the name of the
replica set.
But you can have a replica set with just one member, that's no problem.
The replica set will have only the primary, should work.
We'v set up sharded cluster with replica set for each shard in our production ENV. Last night we encountered a issue while connecting to the cluster and issuing building index command through mongo shell , by connecting to the mongos instance , not directly to specific shard.
The issue is : once starting building the index, connections created from mongos to this shard increases rapidly and 'too many connections' errors show up in the primary shard's log file very soon.
The below is link for primary shard's log summary:
At the very beginning for index
Then very soon, the connections numbers reached to 10000:
Connection limit exceeded
From the three mongos' log, all the connections are initiated from mongos. We have googled and find related issue link as : https://jira.mongodb.org/browse/SERVER-28822
But there is no trigger conditions. And the same time, I tried to reproduce the question in test ENV ,but not occurred again. So, please help.
here is configurations for mongos:
mongos' configuration
and here is for shard:
primary shard's configuration
Found the answer.
It was because the index creation issued by mongorestore command was foreground, not background. I mistook the way which mongorestore took and not checked the meta file for the table schema.
I have a mongodb replicaset running in a docker container (mongo:3.0.11) in a aws vpc (for this specific case just one node, primary).
This server is shutdown every night and started again in the next morning.
After a few months running seamlessly, I'm having a few errors in the past few weeks. Happens that once or twice a week the mongo startup fails.
rs.status() returns stateStr: REMOVED
and as error message: errmsg : "Our replica set config is invalid or we are not a member of it"
Looking at the mongo logs I have:
2016-06-07T12:01:48.724+0000 W NETWORK [ReplicationExecutor] getaddrinfo("database.my_vpc_dns.net") failed: Name or service not known
When this error happens, a simple restart on the docker container will fix, but I'm struggling to understand what is causing this error to happen occasionally.
Probably the replica loses the configuration when doing the restart. It is possible that the replica loses the reading of the DNS reason why it does not manage to raise when the server is started.
What you can do is to point to the machine directly through the domain.my-machine in the Execute db.isMaster() in primary to not restart.
I am getting started with MongoDB and cannot find the answer to the question.
For test purposes I want to create a 3 Datanode Cluster, but so far I am not sure how many machines i will need to start a cluster with 3 Datanodes. I want to have 2 routingservers in the cluster.
My current understanding is that I will need 4 machines.
Machine (Configserver and Routingserver): runs mongod --configsrv and mongos
Machine (Shard and Routingserver): runs mongod and mongos
Machine (Shard): runs only the mongod
Machine (Shard): runs only the mongod
So in my opinion a mongod --configsrv cannot be a shard at the same time?
In MongoDB the config server will store any data other than metadata for a sharded cluster. If you manually connect to the config server and try to write data, you get this error:
WriteResult({
"nInserted" : 0,
"writeError" : {
"code" : 14037,
"errmsg" : "can't create user databases on a --configsvr instance"
}
})
Regarding the number of servers, each shard should run on its own machine. As you only have two shards, you can get away with 2 machines, however, 4 would be desirable so you can have a primary and a secondary replica set for both shards. The config server and routing servers can be run on any of the four machines, so you only need 4 machines.
I am supposed to setup mongodb on 4 servers (a school project to benchmark mongodb, vs others). So I am thinking of using only sharding without replication. So I attempted to setup the 1st 3 server to run mongod --configsvr, and the 4th just a normal mongod instance. Then for all servers, I run mongos. Then I am at the part where I run sh.addShard("...") and I get
{
"ok" : 0,
"errmsg" : "the specified mongod is a --configsvr and should thus not be a shard server"
}
Seems like I cant have a config server running as a shard too? How then should I set things up?
No, the config server is not a shard. The config server is a special mongod instance that stores the sharded cluster metadata, essentially how the data is distributed across the sharded cluster.
You really only need 1 mongos instance for a simple setup like this. When you start mongos, you will provide it with the hostname(s) of your config server(s).
The Sharded Cluster Deployment Tutorial explains all of the steps that you need to follow.