I have a mongo 2 node cluster running, with this replica set config.
config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.100:15000'}]
}
I have to move these both nodes on to new servers. I have copied everything from old to new servers, but I'm running into issues while reconfiguring the replica config due to ip change on the 2nd node.
I have tried this.
config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.200:15000'}]
}
rs.reconfig(config)
{
"startupStatus" : 1,
"errmsg" : "loading local.system.replset config (LOADINGCONFIG)",
"ok" : 0
}
It shows above message, but change is not happening.
I also tried changing replica set name but pointing to the same data dirs.
I am getting the following error:
rs.initiate()
{
"errmsg" : "local.oplog.rs is not empty on the initiating member. cannot initiate.",
"ok" : 0
}
What are the right steps to change the IP but keeping the data on the 2nd node, or do i need to recreate/resync the 2nd node?
Well , I had the same problem.
I had to delete all replication and oplog.
use local
db.dropDatabase()
restart your mongo with new set name
config = {_id: "repl1", members:[
{_id: 0, host: 'localhost:15000'},
{_id: 1, host: '192.168.2.100:15000'}]
}
rs.initiate(config)
I hope this works for you too
You can use force option when reconfiguring replica set:
rs.reconfig(config, {force: true})
Note that, as Adam already suggested in the comments, you should have at least 3 nodes: 2 full nodes and 1 arbiter (minimum supported configuration) or 3 full nodes (minimum recommended configuration) so that primary node can be elected.
I realise this is an old post, but I discovered I was getting this exact same error when trying to change the port used by secondaries in my replica set.
In my case, I needed to stop the secondary whose config I was changing, and bring it up on its new address and port BEFORE applying the changed config on the Primary.
This is in the mongo documentation, but the order in which I had to bring things up and down was something I'd mis-read on the first pass, so for clarity I've repeated that here:
Shut down the secondary member of the replica set you are moving.
Bring that secondary back up at its new address
Make the configuration change as detailed in the original post above
You can use rs.reconfig option. First retrieve the current configuration with rs.conf(). Modify the configuration document as needed, and then pass the modified document to rs.reconfig()
More info in docs.
Related
I want to create a MongoDB replica set, and according to the documentation I need to run sth like this in my first mongo instance in order to config the replica set and this works fine. However, I was wondering if there is a way to automate this process and don't ssh to the server and run this piece of code every time. I tried putting it in a config file but it didn't work
rs.initiate( {
_id : "rs0",
members: [
{ _id: 0, host: "mongodb0.example.net:27017" },
{ _id: 1, host: "mongodb1.example.net:27017" },
{ _id: 2, host: "mongodb2.example.net:27017" }
]
})
Are you talking about situation where you create replica set(s) with docker, terraform, ... kind of tools?
You can always include local init -script what have that rs.initiate(...) command.
Replica set needs ALWAYS initial initiate -command. Without it, it doesn't exist.
Although the question is a bit old, I think it is important to show a possible solution to future readers!
This repo https://github.com/deRemo/dockerized-mongodb presents a python script to automate the configuration of a replica set in a "dockerized" scenario. Have a look at configure_rs.py
Here I extrapolated the important bits:
#!/usr/bin/python3
import os,sys
if len(sys.argv) < 2:
print("USAGE: ./script ip1 ip2 ... (hostname are also valid, if recognised)")
exit()
print("Configuring replica set...")
members = []
for i in range(1, len(sys.argv)):
prio = 1 if i == 1 else 0 #prioritize first member for the primary election
members.append("{_id: "+str(i-1)+", host: \'"+str(sys.argv[i])+"\', priority: "+str(prio)+"}")
#stringify members and prepare rs config
members = ', '.join(d for d in members)
cfg = "{_id: 'rs0', members: ["+members+"]}"
os.system("./mongo --host "+sys.argv[1]+" --eval \"JSON.stringify(db.adminCommand({'replSetInitiate' : "+cfg+"}))\"")
The script takes as input the hostnames of the machines where your mongod instances are deployed (make sure that they are reachable over your network)
I have encountered a problem after adding a shard to mongodb cluster.
I did the following operations:
1. deploy a mongodb cluster with primary shard named 'shard0002'(10.204.8.155:27010) for all databases.
2. for some reason I removed it and add a new shard of different host (10.204.8.100:27010, was automaticlly named shard0002 too) after migrating finished.
3. then add another shard (the one removed in step 1), named 'shard0003'
4. executing query on a sharded collection.
5. the following errors appeared:
mongos> db.count.find()
error: {
"$err" : "setShardVersion failed host: 10.204.8.155:27010 { errmsg: \"exception: gotShardName different than what i had before before [shard0002] got [shard0003] \", code: 13298, ok: 0.0 }",
"code" : 10429
}
I tried to rename the shardname, but it's not allowed.
mongos> use config
mongos> db.shards.update({_id: "shard0003"}, {$set: {_id: "shard_11"}})
Mod on _id not allowed
I have also tried to remove it, draining stared but processing seems hung up.
what can I do ?
------------------------
lastupate (24/02/2014 00:29)
I found the answer on google. since mongod has it's own cache for mongod configuration, just restart the sharded mongod process, the problem will be fixed.
Consider, I have 4 replicate sets and the config is as follows:
{
"_id": "rs_0",
"version": 5,
"members" : [
{"_id": 1, "host": "127.0.0.1:27001"},
{"_id": 2, "host": "127.0.0.1:27002"},
{"_id": 3, "host": "127.0.0.1:27003"},
{"_id": 4, "host": "127.0.0.1:27004"}
]
}
I am able to connect to all sets using
mongo --port <port>
There are documents for getting information on Convert a Standalone to a Replica Set, but can anyone tell me how to convert back to standalone from replica set?
Remove all secondary hosts from replica set (rs.remove('host:port')), restart the mongo deamon without replSet parameter (editing /etc/mongo.conf) and the secondary hosts starts in standalone mode again.
The Primary host is tricky one, because you can't remove it from the replica set with rs.remove.
Once you have only the primary node in the replica set, you should exit mongo shell and stop mongo. Then you edit the /etc/mongo.conf and remove the replSet parameter and start mongo again.
Once you start mongo you are already in standalone mode, but the mongo shell will prompt a message like:
2015-07-31T12:02:51.112+0100 [initandlisten] ** WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset
to remove the warning you can do 2 procedures:
1) Droping the local db and restarting mongo:
use local
db.dropDatabase();
/etc/init.d/mongod restart
2)Or if you don't want to be so radical, you can do:
use local
db.system.replset.find()
and it will prompt a message like:
{ "_id" : "replicaSetName", "version" : 1, "members" : [ { "_id" : 0, "host" : "hostprimary:mongoport" } ] }
then you will erase it using:
db.system.replset.remove({ "_id" : "replicaSetName", "version" : 1, "members" : [ { "_id" : 0, "host" : "hostprimary:mongoport" } ] })
and it will probably prompt:
WriteResult({ "nRemoved" : 1 })
Now, you can restart the mongo and the warning should be gone, and you will have your mongo in standalone mode without warnings
Just remove a host from replica set (rs.remove('host:port')), relaunch it without replSet parameter and it's standalone again.
On an Ubuntu Machine
Stop your mongo server
open /etc/mongod.conf
Comment the replication and replSetName line
#replication:
#replSetName: rs0
Start your mongo server and go to mongo shell
drop local database
use local
db.dropDatabase()
Restart mongo
The MongoDB Documentation suggests the following to perform maintenance on a replica set member, which brings the the replica set member into standalone mode for further operations. With little modification it can be made standalone:
If node in concern is the only node in a shard, drain the chunks to other shards as per MongoDB documentation here, or else the sharded database will break, i.e.
Make sure balancer is enabled by connecting to mongos and run sh.startBalancer(timeout, interval)
For the shard in concern, go to admin database and db.adminCommand( { removeShard: "mongodb0" } )
Check draining status by repeating above removeShard command, wait for draining to complete
If node in concern is primary, do rs.stepDown(300)
Stop the node by running db.shutdownServer()
Change the yaml config by:
commenting out replication.replSetName (--replSetName in command line)
commenting out sharding.clusterRole for shard or config server (--shardsvc and --configsvr in command line)
commenting out net.port, then change it to a different port (--port in command line)
Start the mongod instance
If change is permanent, go to other mongod instance and run rs.remove("host:port")
After this, the node in concern should be up and running in standalone mode.
Follow below steps :
Go to mongo shell on Secondary servers
Stop the secondary servers by using below command :
use admin
db.shutdownServer()
Go to Linux shell- on secondary servers and type below command :
sudo service mongod stop
Starting the MongoDB replication -
Go to Linux shell - on secondary servers and type below command :
sudo service mongod start
Starting the MongoDB replication -
Go to primary and type below commands to start the replication :
a] rs.initiate()
b] rs.add("Secondar -1:port no")
c] rs.add("Secondary-2:port no")
d] rs.add({ "_id" : 3, "host" : "Hidden_member:port no", "priority" : 0,
"hidden" : true })
e] rs.status()
I have a 200gb data base on a sharded four node cluster and and I would like to drop the databse and delet all the files associated to it from the node. I am connecting to my mongos and call dropDatabase on it. The system comes back with ok but if call show dbs, it will show the database again and shows that it is still occupying the 200gb. What I am doing wrong?
I think you are running into this issue:
https://jira.mongodb.org/browse/SERVER-4804
In most cases it seems like the database is in fact removed but the mongos still reports it as being there. You can verify it is gone by either trying to use the DB and getting an error or by logging into the shards directly and checking.
The bug refers to issues with dropping databases while a migration is happening. You can workaround the cause of the issue by doing something like this (sub in your own dbname):
mongos> use config
switched to db config
// 1. stop the balancer
mongos> db.settings.update({_id: "balancer"}, {$set: {stopped: true}}, true)
// 2. wait for in-progress migrations to finish, this may take a few seconds
mongos> while (db.locks.findOne({_id: "balancer", state: {$ne: 0}}) != null) { sleep(1000); }
// 3. now you can safely drop the database
mongos> use <dbname>
switched to db <dbname>
mongos> db.dropDatabase()
{ "dropped" : "<dbname>", "ok" : 1 }
You may want to run the flushRouterConfig on the mongoses to refresh the config info:
mongos> use config
switched to db config
mongos> var mongoses = db.mongos.find()
mongos> while (mongoses.hasNext()) { new Mongo(mongoses.next()._id).getDB("admin").runCommand({flushRouterConfig: 1}) }
{ "flushed" : true, "ok" : 1 }
Of course, the real fix will only come along when the fix is committed - looks like it is targeted for 2.1
If you are in a broken state, you can try this, but it is tricky:
To "reset" the sharding metadata to recover from this issue, please try to do the following
First, stop the balancer (as above) and wait for migrations to finish (also as above)
Next, ensure there is no activity from the app servers on the database in question
Now, ensure that there are no collection entries in config.collections for namespaces beginning with "TestCollection." If so, remove those entries through a mongos:
mongos> use config
mongos> db.collections.find({_id: /^TestCollection\./})
// if any records found, delete them
mongos> db.collections.remove({_id: /^TestCollection\./})
Next up, ensure there is no database entry in config.databases for "TestCollection", and if so, remove it through mongos:
mongos> use config
switched to db config
mongos> db.databases.find({_id: "TestCollection"})
// if any records found, delete them
mongos> db.databases.remove({_id: "TestCollection"})
Now, ensure there are no entries in config.chunks for any namespaces in the database (like the default test namespace). If there are any, remove through mongos:
mongos> use config
switched to db config
mongos> db.chunks.find({ns: /^test\./})
// if any records found, delete them
mongos> db.chunks.remove({ns: /^test\./})
Then, flushRouterConfig on all mongoses:
mongos> use config
switched to db config
mongos> var mongoses = db.mongos.find()
mongos> while (mongoses.hasNext()) { new Mongo(mongoses.next()._id).getDB("admin").runCommand({flushRouterConfig: 1}) }
{ "flushed" : true, "ok" : 1 }
...
Finally, manually connect to each shard primary, and drop the database on the shards (not all shards may have the database, but it's best to be thorough and issue the dropDatabase() call on all
Regarding in-progress migrations, you can use this snippet:
// 2. wait for in-progress migrations to finish, this may take a few seconds
mongos> while (db.locks.findOne({_id: "balancer", state: {$ne: 0}}) != null) { sleep(1000); }
When done, don't forget to reenable the balancer:
mongos> use config
switched to db config
mongos> db.settings.update({_id: "balancer"}, {$set: {stopped: false}}, true)
I had this exact same problem, and discovered that issuing the db.dropDatabase as a regular user failed silently but doing the same as sudo worked, in case that helps anyone.
Recently we made live the mongodb sharding concept and its working fine in production server. But we have configured the public IP address instead of internal IP. So we have to change the internal ip in mongodb db sharding.
Please clarify whether its possible or not. If possible means, please share your input.
public ip example:
conf = {_id : "data1",members : [{_id : 0, host : "10.17.18.01:10001", votes : 2},{_id : 1, host : "10.17.19.02:10002", votes : 1},{_id:2, host: "10.17.19.03:10003", votes : 3, arbiterOnly: true}]}
internal ip example
conf = {_id : "data1",members : [{_id : 0, host : "20.17.18.01:10001", votes : 2},{_id : 1, host : "20.17.19.02:10002", votes : 1},{_id:2, host: "20.17.19.03:10003", votes : 3, arbiterOnly: true}]}
whether it will work. Pls suggest.
Regards,
Kumaran
You said you're trying to update the IPs in the sharding system, but the config documents you provided as an example look like a replica set configuration. If it's actually your replica set configuration you want to update, you should just be able to remove the entry for the old IP address from the replica set configuration, then add the node back in with the new IP. See http://www.mongodb.org/display/DOCS/Replica+Set+Configuration and http://www.mongodb.org/display/DOCS/Reconfiguring+when+Members+are+Up for more details.
If it's actually the sharding configuration you want to update, it will be a bit more complicated.
Throwing in an answer, even though this is a dated question for anyone else who might view this.
I would recommend using host names / host entries on your servers to handle local and external ips. However, to update the hosts in your case, you would have to change the replica set config.
Log into the Primary in the replica set then do the following:
> cfg = rs.conf()
> cfg.members[0].host = "[new host1]:[port]"
> cfg.members[1].host = "[new host2]:[port]"
> cfg.members[2].host = "[new host3]:[port]"
cfg.members is obviously a zero-index array, you can reuse that for how every many replicas you have.
> rs.reconfig( cfg )
From there, you would want to re-add your shards with the newly specified hosts.
from inside mongos.
simply use the following command to update the IPs of the shard servers:
db.shards.update({_id: <<"shard name">>} , {$set: {"host" : "newIP:27018"}})
Example:
db.shards.update({_id: "shard000"} , {$set: {"host" : "172.31.1.1:27018"}})
172.31.1.1 is the private ip address of your shard server in the private network.
avoid using a dynamic ip address.
If you want to do any modification in the Shard configuration then you should
use config
db.shards.update( { _id : } , { $set : { ... } } )
Please make sure that you restart your config server and mongos after making this change.