I did a mistake when configuring replica sets in mongodb. I think that what I did wrong is that I did a rs.initialize() on both nodes, which made them confused in some way. I'm not sure.
Now all I want to do is start over, but I couldn't find a way to de-initialize a node. So I followed the advice to delete the local* db files, thus resetting the configurations. I did that, and now nothing works.
> rs.initiate()
{
"info2" : "no configuration explicitly specified -- making one",
"me" : "0.0.0.0:27017",
"errmsg" : "couldn't initiate : can't find self in the replset config",
"ok" : 0
}
> rs.conf()
null
I tried to remove and reinstall the package (I'm doing this on Ubuntu servers) which just meant that my mongodb.conf disappeared and my init script stopped working. This is of course easy enough to solve.
So how do I start over?
Note: I did look at this answer, but since rs.conf() doesn't work this doesn't work either.
You'll also get this error if your machine's hostname doesn't map back to 127.0.0.1. Update your /etc/hosts and/or your /etc/hostname, and rs.initiate() with no config should work.
If you force a reconfig with a config that you have generated, does it resolve the issue?
You could do this similar to the follow from the {{mongo}} shell:
> cfg = {
... "_id" : "rs0",
... "version" : 1,
... "members" : [
... {
... "_id" : 0,
... "host" : "0.0.0.0:27017"
... }
... ]
... }
>rs.reconfig(cfg, {force:true})
You may need to tune the cfg variable to have your hostname and portname, as the can't find self in new replset config error will be returned to the shell if the repl set can't find the node it is running from in the config.
If you just comment out bind_ip in /etc/mongod.conf this will achieve the correct result so that you can reissue a rs.initiate() command to set-up or reconfig the replica.
Related
How to configure removed members of a replica set to form new replica set?
I have a replica set with 4 mongod instances
Output of rs.config()
{
"_id" : "rs0",
"members" : [
{
"_id" : 0,
"host" : "localhost:27031"
},
{
"_id" : 1,
"host" : "localhost:27032"
},
{
"_id" : 2,
"host" : "localhost:27033"
},
{
"_id" : 3,
"host" : "localhost:27034"
}
],
"settings" : {
"replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2")
}
}
I removed 2 instances from the replica set
rs.remove("localhost:27033")
rs.remove("localhost:27034")
Now my requirement is to form a new replica set with these 2 removed members. What is the best way for that?
My current solution
connect to removed member
mongo --port 27033
and execute
conf = {
"_id" : "rs0",
"members" : [
{
"_id" : 2,
"host" : "localhost:27033"
},
{
"_id" : 3,
"host" : "localhost:27034"
}
],
"settings" : {
"replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2")
}
}
and then
rs.reconfig(conf, {force:true})
Outcome
This solution worked fine practically.
The removed members formed a replicaset, one of them became primary and other became secondary. Data was replicated among them.
And this replica set seems to be isolated from the the initial replica set from which they were removed.
Concerns
1) I had to use forced reconfiguration. Not sure about the consequences.
"errmsg" : "replSetReconfig should only be run on PRIMARY, but my state is REMOVED; use the \"force\" argument to override",
2) Is the new replica set actually new one? In the rs.config()
replicaSetId is same as old one.
"replicaSetId" : ObjectId("5cf22332f5b9d21b01b9b6b2")
I had to use same value for _id of members as in config of old replica set
"errmsg" : "New and old configurations both have members with host of localhost:27034 but in the new configuration the _id field is 1 and in the old configuration it is 3 for replica set rs0",
Is this solution good?
Is there any better solution?
Note: I need to retain data from old replica set (data which was present at the time of removal) in the new replica set.
As you have suspected, the procedure did not create a new replica set. Rather, it's a continuation of the old replica set, albeit superficially they look different.
There is actually a procedure in the MongoDB documentation to do what you want: Restore a Replica Set from MongoDB Backups. The difference being, you're not restoring from a backup. Rather, you're using one of the removed secondaries to seed a new replica set.
Hence you need to modify the first step in the procedure mentioned in the link above. The rest of the procedure would still be the same:
Restart the removed secondary as a standalone (without the --replSet parameter) and connect to it using the mongo shell.
Drop the local database in the standalone node:
use local
db.dropDatabase()
Restart the ex-secondary, this time with the --replSet parameter (with a new replica set name)
Connect to it using the mongo shell.
rs.initiate() the new set.
After this, the new set should have a different replicaSetId compared to the old set. In my quick test of the procedure above, this is the result I see:
Old set:
> rs.conf()
...
"replicaSetId": ObjectId("5cf45d72a1c6c4de948ff5d8")
...
New set
> rs.conf()
...
"replicaSetId": ObjectId("5cf45d000dda9e1025d6c65e")
...
As with any major deployment changes like this, please ensure that you have a backup, and thoroughly test the procedures before doing it on a production system.
I have got this issue while working on replica sets. Server is successfully turning on but after executing rs.initiate() and rs.status I am getting errors.
"info2" : "no configuration explicitly specified -- making one",
"errmsg" : "exception: bad --replSet config string format is: <setname>[host1>,<seedhost2>,...]",
"code" : 13093,
"ok" : 0
I ran into this problem as well. What happened was I configured the replica set in /etc/mongo.conf, went into the mongo client and executed rs.initiate(). What I forgot to do was restart mongo! A simple sudo service mongod restart fixed it.
Consider, I have 4 replicate sets and the config is as follows:
{
"_id": "rs_0",
"version": 5,
"members" : [
{"_id": 1, "host": "127.0.0.1:27001"},
{"_id": 2, "host": "127.0.0.1:27002"},
{"_id": 3, "host": "127.0.0.1:27003"},
{"_id": 4, "host": "127.0.0.1:27004"}
]
}
I am able to connect to all sets using
mongo --port <port>
There are documents for getting information on Convert a Standalone to a Replica Set, but can anyone tell me how to convert back to standalone from replica set?
Remove all secondary hosts from replica set (rs.remove('host:port')), restart the mongo deamon without replSet parameter (editing /etc/mongo.conf) and the secondary hosts starts in standalone mode again.
The Primary host is tricky one, because you can't remove it from the replica set with rs.remove.
Once you have only the primary node in the replica set, you should exit mongo shell and stop mongo. Then you edit the /etc/mongo.conf and remove the replSet parameter and start mongo again.
Once you start mongo you are already in standalone mode, but the mongo shell will prompt a message like:
2015-07-31T12:02:51.112+0100 [initandlisten] ** WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset
to remove the warning you can do 2 procedures:
1) Droping the local db and restarting mongo:
use local
db.dropDatabase();
/etc/init.d/mongod restart
2)Or if you don't want to be so radical, you can do:
use local
db.system.replset.find()
and it will prompt a message like:
{ "_id" : "replicaSetName", "version" : 1, "members" : [ { "_id" : 0, "host" : "hostprimary:mongoport" } ] }
then you will erase it using:
db.system.replset.remove({ "_id" : "replicaSetName", "version" : 1, "members" : [ { "_id" : 0, "host" : "hostprimary:mongoport" } ] })
and it will probably prompt:
WriteResult({ "nRemoved" : 1 })
Now, you can restart the mongo and the warning should be gone, and you will have your mongo in standalone mode without warnings
Just remove a host from replica set (rs.remove('host:port')), relaunch it without replSet parameter and it's standalone again.
On an Ubuntu Machine
Stop your mongo server
open /etc/mongod.conf
Comment the replication and replSetName line
#replication:
#replSetName: rs0
Start your mongo server and go to mongo shell
drop local database
use local
db.dropDatabase()
Restart mongo
The MongoDB Documentation suggests the following to perform maintenance on a replica set member, which brings the the replica set member into standalone mode for further operations. With little modification it can be made standalone:
If node in concern is the only node in a shard, drain the chunks to other shards as per MongoDB documentation here, or else the sharded database will break, i.e.
Make sure balancer is enabled by connecting to mongos and run sh.startBalancer(timeout, interval)
For the shard in concern, go to admin database and db.adminCommand( { removeShard: "mongodb0" } )
Check draining status by repeating above removeShard command, wait for draining to complete
If node in concern is primary, do rs.stepDown(300)
Stop the node by running db.shutdownServer()
Change the yaml config by:
commenting out replication.replSetName (--replSetName in command line)
commenting out sharding.clusterRole for shard or config server (--shardsvc and --configsvr in command line)
commenting out net.port, then change it to a different port (--port in command line)
Start the mongod instance
If change is permanent, go to other mongod instance and run rs.remove("host:port")
After this, the node in concern should be up and running in standalone mode.
Follow below steps :
Go to mongo shell on Secondary servers
Stop the secondary servers by using below command :
use admin
db.shutdownServer()
Go to Linux shell- on secondary servers and type below command :
sudo service mongod stop
Starting the MongoDB replication -
Go to Linux shell - on secondary servers and type below command :
sudo service mongod start
Starting the MongoDB replication -
Go to primary and type below commands to start the replication :
a] rs.initiate()
b] rs.add("Secondar -1:port no")
c] rs.add("Secondary-2:port no")
d] rs.add({ "_id" : 3, "host" : "Hidden_member:port no", "priority" : 0,
"hidden" : true })
e] rs.status()
I am using the MongoDB 2.4.3, and following the wizard:
http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
But when adding the other members into replica-set, get the following error:
root#vm3:~# mongo
MongoDB shell version: 2.4.3
connecting to: test
rs1:PRIMARY> rs.add("vm1")
{
"errmsg" : "exception: set name does not match the set name host vm1:27017 expects",
"code" : 13145,
"ok" : 0
}
rs1:PRIMARY> rs.add("vm4")
{
"errmsg" : "exception: set name does not match the set name host vm4:27017 expects",
"code" : 13145,
"ok" : 0
}
vm1, vm3 and vm4 know each other because I configured their /etc/hosts files correctly.
Any idea? I don't understand what does this error message mean!
After restarting all vms, it works now.
root#vm3:~# mongo
MongoDB shell version: 2.4.3
connecting to: test
rs1:PRIMARY> rs.add("vm4")
{ "ok" : 1 }
rs1:PRIMARY> rs.add("vm1")
{ "ok" : 1 }
In my case, just restart virtual machines, every thing is fine.
If you are re-installing a MongoDB instance, the replSet may be living in your data file on the drive. I had the same problem setting up a new replica set. The problem was from changing the replica set name after bringing up instances with an older replSet name. I deleted the data files, ran my install scripts again and it worked just fine.
I need to grab (within the C# driver for MongoDB) a list of all the config servers connected to my instance of Mongo-s. Or, failing that, I would settle for a way to grab ALL the servers and a way to go through them one by one telling which are configsvr and which are something else. I was thinking of the getShardMap command, but I still have no idea how to look at a server (programmatically) and decide if it's a configsvr or not.
Thanks.
mongos> db.runCommand("getShardMap")
{
"map" : {
"node2:27021" : "node2:27021",
"node3:27021" : "node3:27021",
"node4:27021" : "node4:27021",
"config" : "node2:27019,node3:27019,node4:27019",
"shard0000" : "node2:27021",
"shard0001" : "node3:27021",
"shard0002" : "node4:27021"
},
"ok" : 1
}
getShardMap command gives the config string that is passed to mongos server. You can parse the string to get the list of config servers.
The only way I can think of to get this info is to run the getCmdLineOpts command on a mongos and look at the --configdb argument it was passed. I'm not sure how you run admin commands in the C# driver, but I'd imagine it's something like:
db.RunCommand("getCmdLineOpts");