mongoDB servers are not available when the master is down - mongodb

Once I turn off the master node, the other nodes are down, too.
I can NOT get data from other nodes (if the master is down/off )
The following errors happends when the master replica is down
ConnectionPool::PoolShuttingDownError (ConnectionPool::PoolShuttingDownError):
Moped::Errors::ConnectionFailure (Could not connect to a primary node for replica set #<Moped::Cluster:70117691586500 #seeds=[<Moped::Node resolved_address="0.0.0.0:27017">]>
I have 3 mongoDB replicas.
It seems the primary replica is 172.19.16.109:27017
I don't understand why I shutdown the machine 172.19.16.109,
And remaining 2 replicas are unavailable to load data from its database.
Is it not making sense for the replicas ? Once if the master is shutdown, the others are also unavailable, too.
I expect that even if any replica is down, the others should work correctly.
My configurations are as below.
vvtk_dqa:PRIMARY> rs.conf()
{
"_id" : "vvtk_dqa",
"version" : 4,
"members" : [
{
"_id" : 1,
"host" : "172.19.16.109:27017"
},
{
"_id" : 2,
"host" : "172.19.16.104:27017"
},
{
"_id" : 3,
"host" : "192.168.14.7:27017"
}
]
}
mongodb.conf
# Where to store the data.
dbpath=/var/lib/mongodb
#where to log
logpath=/var/log/mongodb/mongodb.log
logappend=true
bind_ip = 0.0.0.0
#port = 27017
# Enable journaling, http://www.mongodb.org/display/DOCS/Journaling
journal=true
replSet=vvtk_dqa
isMaster
vvtk_dqa:PRIMARY> rs.isMaster()
{
"setName" : "vvtk_dqa",
"setVersion" : 4,
"ismaster" : true,
"secondary" : false,
"hosts" : [
"172.19.16.109:27017",
"192.168.14.7:27017",
"172.19.16.104:27017"
],
"primary" : "172.19.16.109:27017",
"me" : "172.19.16.109:27017",
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 1000,
"localTime" : ISODate("2015-02-11T02:05:12.021Z"),
"maxWireVersion" : 2,
"minWireVersion" : 0,
"ok" : 1
}
After I shutdown the 172.19.16.109, and type rs.status() on 192.168.14.7
and still can not get any data from 192.168.14.7 or 172.19.16.104
vvtk_dxa:PRIMARY> rs.status()
{
"set" : "vvtk_dqa",
"date" : ISODate("2015-02-12T01:51:37Z"),
"myState" : 1,
"members" : [
{
"_id" : 1,
"name" : "172.19.16.109:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(1423649845, 1),
"optimeDate" : ISODate("2015-02-11T10:17:25Z"),
"lastHeartbeat" : ISODate("2015-02-12T01:51:28Z"),
"lastHeartbeatRecv" : ISODate("2015-02-11T10:31:04Z"),
"pingMs" : 0,
"syncingTo" : "192.168.14.7:27017"
},
{
"_id" : 2,
"name" : "172.19.16.104:27017",
"health" : 1,
"state" : 4,
"stateStr" : "FATAL",
"uptime" : 64832,
"optime" : Timestamp(1423545748, 1),
"optimeDate" : ISODate("2015-02-10T05:22:28Z"),
"lastHeartbeat" : ISODate("2015-02-12T01:51:37Z"),
"lastHeartbeatRecv" : ISODate("2015-02-12T01:51:35Z"),
"pingMs" : 0
},
{
"_id" : 3,
"name" : "192.168.14.7:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 64835,
"optime" : Timestamp(1423704914, 1),
"optimeDate" : ISODate("2015-02-12T01:35:14Z"),
"self" : true
}
],
"ok" : 1
}

Related

pymongo unable to connect to primary

I'm trying to find_one by connecting to my replica set's primary node.
MongoClient(hostname, replicaSet="rs0", read_preference=ReadPreference.PRIMARY)
But it results in an error:
ServerSelectionTimeoutError: No replica set members match selector
"Primary()"
I'm able to successfully read using SECONDARY_PREFERRED. I also tried connecting using MongoReplicaSetClient with no success. I'm guessing this due to bad configuration, but what should I be looking for?
rs.status:
rs0:PRIMARY> rs.conf()
{
"_id" : "rs0",
"version" : 111313,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 1,
"host" : "ANDROMEDA:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "mongo02.db.com:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0.5,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 3,
"host" : "mongo03.db.com:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0.5,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}
rs0:SECONDARY> rs.status()
{
"set" : "rs0",
"date" : ISODate("2016-08-04T08:58:02.293Z"),
"myState" : 2,
"term" : NumberLong(90),
"syncingTo" : "mongo03.db.com:27017",
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 1,
"name" : "ANDROMEDA:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 2503,
"optime" : {
"ts" : Timestamp(1470299746, 1),
"t" : NumberLong(90)
},
"optimeDate" : ISODate("2016-08-04T08:35:46Z"),
"lastHeartbeat" : ISODate("2016-08-04T08:58:01.109Z"),
"lastHeartbeatRecv" : ISODate("2016-08-04T08:58:01.803Z"),
"pingMs" : NumberLong(28),
"electionTime" : Timestamp(1469600522, 1),
"electionDate" : ISODate("2016-07-27T06:22:02Z"),
"configVersion" : 111313
},
{
"_id" : 2,
"name" : "mongo02.db.com:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 7604104,
"optime" : {
"ts" : Timestamp(1470299746, 1),
"t" : NumberLong(90)
},
"optimeDate" : ISODate("2016-08-04T08:35:46Z"),
"syncingTo" : "mongo03.db.com:27017",
"configVersion" : 111313,
"self" : true
},
{
"_id" : 3,
"name" : "mongo03.db.com:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2503,
"optime" : {
"ts" : Timestamp(1470299746, 1),
"t" : NumberLong(90)
},
"optimeDate" : ISODate("2016-08-04T08:35:46Z"),
"lastHeartbeat" : ISODate("2016-08-04T08:58:01.948Z"),
"lastHeartbeatRecv" : ISODate("2016-08-04T08:58:01.802Z"),
"pingMs" : NumberLong(28),
"syncingTo" : "ANDROMEDA:27017",
"configVersion" : 111313
}
],
"ok" : 1
}
In cases with an error message similar to ServerSelectionTimeoutError: No replica set members match selector "Primary()" and where replica set status function output, rs.status(), shows the Primary member using a non-FQDN (ANDROMEDA:27017 as in this case) then it is highly likely the application is unable to resolve the Primary host on the network.
You can easily check this using the following commands from the command line of the host running your application:
$ dig ANDROMEDA
$ ping ANDROMEDA
$ mongo --host ANDROMEDA:27017
If you don't have the Mongo Shell installed on the host running your application you can use Telnet instead.
$ telnet ANDROMEDA 27017
These outputs will allow you to check connectivity between your application host and your mongod host to determine if this is causing the problem.
I use this connection string:
MongoClient('mongodb://mongo01.db.com:27017,mongo02.db.com:27017,mongo03.db.com:27017/mydb',replicaSet="rs0", read_preference=ReadPreference.PRIMARY)

MongoDb primary replica becomes secondary if secondary fails

I have 2 mongo replicas. One is primary and the second one is secondary. If I stop the secondary replica, the primary one becomes secondary and I lost the write permissions.
Here are my configs:
replica:PRIMARY> rs.status()
{
"set" : "replica",
"date" : ISODate("2016-02-26T11:27:50.140Z"),
"myState" : 1,
"members" : [
{
"_id" : 1,
"name" : "192.168.5.44:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 80,
"optime" : Timestamp(1456486069, 1),
"optimeDate" : ISODate("2016-02-26T11:27:49Z"),
"electionTime" : Timestamp(1456485992, 1),
"electionDate" : ISODate("2016-02-26T11:26:32Z"),
"configVersion" : 82935,
"self" : true
},
{
"_id" : 2,
"name" : "192.168.5.34:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 79,
"optime" : Timestamp(1456486067, 2),
"optimeDate" : ISODate("2016-02-26T11:27:47Z"),
"lastHeartbeat" : ISODate("2016-02-26T11:27:48.874Z"),
"lastHeartbeatRecv" : ISODate("2016-02-26T11:27:48.930Z"),
"pingMs" : 1,
"syncingTo" : "192.168.5.44:27017",
"configVersion" : 82935
}
],
"ok" : 1
}
replica:SECONDARY> rs.status()
{
"set" : "replica",
"date" : ISODate("2016-02-26T11:21:38.574Z"),
"myState" : 2,
"syncingTo" : "192.168.5.44:27017",
"members" : [
{
"_id" : 1,
"name" : "192.168.5.44:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 1306,
"optime" : Timestamp(1456485695, 3),
"optimeDate" : ISODate("2016-02-26T11:21:35Z"),
"lastHeartbeat" : ISODate("2016-02-26T11:21:36.602Z"),
"lastHeartbeatRecv" : ISODate("2016-02-26T11:21:37.412Z"),
"pingMs" : 0,
"electionTime" : Timestamp(1456484393, 1),
"electionDate" : ISODate("2016-02-26T10:59:53Z"),
"configVersion" : 82935
},
{
"_id" : 2,
"name" : "192.168.5.34:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1306,
"optime" : Timestamp(1456485697, 1),
"optimeDate" : ISODate("2016-02-26T11:21:37Z"),
"syncingTo" : "192.168.5.44:27017",
"configVersion" : 82935,
"self" : true
}
],
"ok" : 1
}
replica:SECONDARY> cfg = rs.conf()
{
"_id" : "replica",
"version" : 82935,
"members" : [
{
"_id" : 1,
"host" : "192.168.5.44:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 2,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 2,
"host" : "192.168.5.34:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatTimeoutSecs" : 10,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}
Even if I shutdown the primary replica, the secondary won't become primary.
If you need any other details notify me.
Thank you for help!
MongoDB need the majority of members to accomplish an election. In a 2 members replication set, both member must be available to reach the majority. If one is down, another cannot be elected to primary.

MongoDB 3.0.1 Replication Setup

I am using MongoDB 3.0.1 version. i am try to setup for mongodb replicaiton in
our machine. i have use three mongodb in septate machine for replication.
machine1 - master
machine2 - slave
machine3 - slave
I refer this url http://docs.mongodb.org/manual/tutorial/deploy-replica-set/
I have configure for every machine in mongodb.conf file
replSet = rs1
fork = true
and i added two members in master machine using this command
rs.add(192.168.1.2)
rs.add(192.168.1.3)
but i insert a document in master machine but not replicate other two slave machine. i did check slave machine throw the following error
> show dbs
2015-05-18T12:43:22.020+0530 E QUERY Error: listDatabases failed:{ "note" : "from execCommand", "ok" : 0, "errmsg" : "not master" }
at Error (<anonymous>)
at Mongo.getDBs (src/mongo/shell/mongo.js:47:15)
at shellHelper.show (src/mongo/shell/utils.js:630:33)
at shellHelper (src/mongo/shell/utils.js:524:36)
at (shellhelp2):1:1 at src/mongo/shell/mongo.js:47
>
> rs.conf()
2015-05-18T12:43:38.692+0530 E QUERY Error: Could not retrieve replica set config: {
"info" : "run rs.initiate(...) if not yet done for the set",
"ok" : 0,
"errmsg" : "no replset config has been received",
"code" : 94
}
at Function.rs.conf (src/mongo/shell/utils.js:1011:11)
at (shell):1:4 at src/mongo/shell/utils.js:1011
>
Please help me to solve the problem. thanks & Advance.
EDIT:
rs1:PRIMARY> rs.conf()
{
"_id" : "rs1",
"version" : 4,
"members" : [
{
"_id" : 0,
"host" : "analyzer-xubuntu:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 1,
"host" : "192.168.1.31:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0.75,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 2,
"host" : "192.168.1.33:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0.5,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatTimeoutSecs" : 10,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}
rs1:PRIMARY> rs.status()
{
"set" : "rs1",
"date" : ISODate("2015-05-18T09:07:31.767Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "analyzer-xubuntu:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 9236,
"optime" : Timestamp(1431939509, 2),
"optimeDate" : ISODate("2015-05-18T08:58:29Z"),
"electionTime" : Timestamp(1431931054, 2),
"electionDate" : ISODate("2015-05-18T06:37:34Z"),
"configVersion" : 4,
"self" : true
},
{
"_id" : 1,
"name" : "192.168.1.31:27017",
"health" : 1,
"state" : 0,
"stateStr" : "STARTUP",
"uptime" : 8953,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2015-05-18T09:07:29.831Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 1,
"configVersion" : -2
},
{
"_id" : 2,
"name" : "192.168.1.33:27017",
"health" : 1,
"state" : 0,
"stateStr" : "STARTUP",
"uptime" : 8946,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2015-05-18T09:07:30.533Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 1,
"configVersion" : -2
}
],
"ok" : 1
}
Let us go step by step.
Your MongoDB config is ok, the rest of work may be done in the shell. I will use machine names instead of IPs.
First connect to machine1, and run the following there:
> conf = {
_id: "rs1",
members:
[
{_id : 0, host : "machine1:27017"},
{_id : 1, host : "machine2:27017"},
{_id : 2, host : "machine3:27017"}
]
}
> rs.initiate(conf)
Then just run rs.slaveOk() on the secondaries. The secondaries will start replicating, and you will be able to query them and see your data, inserted from the primary.

MongoDB replicaset configuration

Could you please tell me if this will cause any issues with failover? For example, what would happen if host mongo2.local is down? (assuming the original host and the arbiter go down and only 2 members are left). Will the rest of the members be able to elect a new primary ever?
I know that there shouldn't be an arbiter here as it makes things worse but I wanted to know if a failover will occur in case of this setup and mongo2.local go down.
mongo:ARBITER> rs.status()
{
"set" : "mongo",
"date" : ISODate("2015-02-12T09:00:08Z"),
"myState" : 7,
"members" : [
{
"_id" : 0,
"name" : "mongo1.local:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2572473,
"optime" : Timestamp(1423731603, 4),
"optimeDate" : ISODate("2015-02-12T09:00:03Z"),
"lastHeartbeat" : ISODate("2015-02-12T09:00:07Z"),
"lastHeartbeatRecv" : ISODate("2015-02-12T09:00:07Z"),
"pingMs" : 0,
"syncingTo" : "mongo2.local:27017"
},
{
"_id" : 1,
"name" : "mongo2.local:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 12148099,
"optime" : Timestamp(1423731603, 4),
"optimeDate" : ISODate("2015-02-12T09:00:03Z"),
"lastHeartbeat" : ISODate("2015-02-12T09:00:08Z"),
"lastHeartbeatRecv" : ISODate("2015-02-12T09:00:08Z"),
"pingMs" : 0,
"electionTime" : Timestamp(1423711411, 1),
"electionDate" : ISODate("2015-02-12T03:23:31Z")
},
{
"_id" : 2,
"name" : "mongo3.local:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 5474488,
"optime" : Timestamp(1423731603, 4),
"optimeDate" : ISODate("2015-02-12T09:00:03Z"),
"lastHeartbeat" : ISODate("2015-02-12T09:00:07Z"),
"lastHeartbeatRecv" : ISODate("2015-02-12T09:00:08Z"),
"pingMs" : 139,
"syncingTo" : "mongo2.local:27017"
},
{
"_id" : 3,
"name" : "mongo2.local:27020",
"health" : 1,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 12148101,
"self" : true
}
],
"ok" : 1
}
and:
mongo:ARBITER> rs.config()
{
"_id" : "mongo",
"version" : 5,
"members" : [
{
"_id" : 0,
"host" : "mongo1.local:27017",
"priority" : 0.5
},
{
"_id" : 1,
"host" : "mongo2.local:27017"
},
{
"_id" : 2,
"host" : "mongo3.local:27017",
"priority" : 0.5
},
{
"_id" : 3,
"host" : "mongo2.local:27020",
"arbiterOnly" : true
}
]
}
If you have less than a majority of the votes in a replica set available, the replica set will not be able to elect or maintain a primary and the replica set will be unhealthy and will be read-only. Ergo, if you only have 2 of your 4 embers up, you will not have a primary. No automatic failover will occur as there aren't enough votes for an election.
Don't have an even number of nodes in a replica set. It increases the chances that there will be problems, just because there are more servers, without increasing the failure tolerance of the set. With 3 or 4 replica set members, 2 down servers will render the set unhealthy.

Couldn't connect to new shard ReplicaSetMonitor no master found for set

I'm trying to deploy sharded cluster in Mongodb
I followed up the tutorial here
http://docs.mongodb.org/manual/tutorial/convert-replica-set-to-replicated-shard-cluster/
first I deployed a Replica set with test data on a separate machine with IP 192.168.1.212
and this is the status after I finished deploying it
firstset:PRIMARY> rs.status();
{
"set" : "firstset",
"date" : ISODate("2014-03-24T10:54:06Z"),
"myState" : 1,
"members" : [
{
"_id" : 1,
"name" : "localhost:10001",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 117,
"optime" : Timestamp(1395650164, 10507),
"optimeDate" : ISODate("2014-03-24T08:36:04Z"),
"self" : true
},
{
"_id" : 2,
"name" : "localhost:10002",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 65,
"optime" : Timestamp(1395650164, 10507),
"optimeDate" : ISODate("2014-03-24T08:36:04Z"),
"lastHeartbeat" : ISODate("2014-03-24T10:54:05Z"),
"lastHeartbeatRecv" : ISODate("2014-03-24T10:54:05Z"),
"pingMs" : 0,
"lastHeartbeatMessage" : "syncing to: localhost:10001",
"syncingTo" : "localhost:10001"
},
{
"_id" : 3,
"name" : "localhost:10003",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 51,
"optime" : Timestamp(1395650164, 10507),
"optimeDate" : ISODate("2014-03-24T08:36:04Z"),
"lastHeartbeat" : ISODate("2014-03-24T10:54:05Z"),
"lastHeartbeatRecv" : ISODate("2014-03-24T10:54:04Z"),
"pingMs" : 0,
"lastHeartbeatMessage" : "syncing to: localhost:10001",
"syncingTo" : "localhost:10001"
}
],
"ok" : 1
}
Then I deployed three config server on separate machine then run mongos instance on another machine
Then I wanted to add Replica shard using the following command
sh.addShard("firstset/192.168.1.212:10001,192.168.1.212:10002,192.168.1.212:10003")
But I get the following error
mongos> sh.addShard('firstset/192.168.1.212:10001,192.168.1.212:10002,192.168.1.212:10003');
{
"ok" : 0,
"errmsg" : "couldn't connect to new shard ReplicaSetMonitor no master found for set: firstset"
}
I found the solution to this problem with Sammaye's help
the problem was that when replica set is initiated you should take care of the IPs you use because when the router will try to connect to the replica set, it reads its configuration file
So if you use rs.initiate() with setting your configuration, the configuration will be like that
{
"_id" : "firstset",
"version" : 1,
"members" : [
{
"_id" : 1,
"host" : "localhost:10001"
},
{
"_id" : 2,
"host" : "localhost:10002"
},
{
"_id" : 3,
"host" : "localhost:10003"
}
]
}
So the router will try to search at localhost to find the primary replica set but it won't find it because it is on other machine
If you use different machines for testing so initialize Replica set manually as following
rsconf ={
"_id" : "firstset",
"version" : 1,
"members" : [
{
"_id" : 1,
"host" : "machine_ip:machine_port"
},
{
"_id" : 2,
"host" : "machine_ip:machine_port"
},
{
"_id" : 3,
"host" : "machine_ip:machine_port"
}
]
}
rs.initiate(rsconf);
Also if you use either “localhost” or “127.0.0.1” as the host identifier, then you must use “localhost” or “127.0.0.1” for all host settings for any MongoDB instances in the cluster. This applies to both the host argument to addShard and the value to the mongos --configdb run time option. If you mix localhost addresses with remote host address, MongoDB will produce errors.