MongoDb primary replica becomes secondary if secondary fails - mongodb

I have 2 mongo replicas. One is primary and the second one is secondary. If I stop the secondary replica, the primary one becomes secondary and I lost the write permissions.
Here are my configs:
replica:PRIMARY> rs.status()
{
"set" : "replica",
"date" : ISODate("2016-02-26T11:27:50.140Z"),
"myState" : 1,
"members" : [
{
"_id" : 1,
"name" : "192.168.5.44:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 80,
"optime" : Timestamp(1456486069, 1),
"optimeDate" : ISODate("2016-02-26T11:27:49Z"),
"electionTime" : Timestamp(1456485992, 1),
"electionDate" : ISODate("2016-02-26T11:26:32Z"),
"configVersion" : 82935,
"self" : true
},
{
"_id" : 2,
"name" : "192.168.5.34:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 79,
"optime" : Timestamp(1456486067, 2),
"optimeDate" : ISODate("2016-02-26T11:27:47Z"),
"lastHeartbeat" : ISODate("2016-02-26T11:27:48.874Z"),
"lastHeartbeatRecv" : ISODate("2016-02-26T11:27:48.930Z"),
"pingMs" : 1,
"syncingTo" : "192.168.5.44:27017",
"configVersion" : 82935
}
],
"ok" : 1
}
replica:SECONDARY> rs.status()
{
"set" : "replica",
"date" : ISODate("2016-02-26T11:21:38.574Z"),
"myState" : 2,
"syncingTo" : "192.168.5.44:27017",
"members" : [
{
"_id" : 1,
"name" : "192.168.5.44:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 1306,
"optime" : Timestamp(1456485695, 3),
"optimeDate" : ISODate("2016-02-26T11:21:35Z"),
"lastHeartbeat" : ISODate("2016-02-26T11:21:36.602Z"),
"lastHeartbeatRecv" : ISODate("2016-02-26T11:21:37.412Z"),
"pingMs" : 0,
"electionTime" : Timestamp(1456484393, 1),
"electionDate" : ISODate("2016-02-26T10:59:53Z"),
"configVersion" : 82935
},
{
"_id" : 2,
"name" : "192.168.5.34:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 1306,
"optime" : Timestamp(1456485697, 1),
"optimeDate" : ISODate("2016-02-26T11:21:37Z"),
"syncingTo" : "192.168.5.44:27017",
"configVersion" : 82935,
"self" : true
}
],
"ok" : 1
}
replica:SECONDARY> cfg = rs.conf()
{
"_id" : "replica",
"version" : 82935,
"members" : [
{
"_id" : 1,
"host" : "192.168.5.44:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 2,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
},
{
"_id" : 2,
"host" : "192.168.5.34:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : 0,
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatTimeoutSecs" : 10,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}
Even if I shutdown the primary replica, the secondary won't become primary.
If you need any other details notify me.
Thank you for help!

MongoDB need the majority of members to accomplish an election. In a 2 members replication set, both member must be available to reach the majority. If one is down, another cannot be elected to primary.

Related

Multiple Primary node in Mongodb replicaset

Sometimes our Mongodb replicaset has multiple primary nodes!
in this case mongo-b node is our real primary and mongo-c node is not responding and just have primary role in fake.
Number of nodes in replicaset: 4 (3 normal + 1 delay node without voting).
network partition and access checked and there is no partion.
we have had this problem several times since updating the cluster from version 4.2.5 to version 4.4.5.
each time we fix the problem by cleaning datadir on fake node and restarting it.
replica conf:
{ [25/599]
"_id" : 0,
"host" : "mongo-a:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 3,
"host" : "mongo-h:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : true,
"priority" : 0,
"tags" : {
},
"slaveDelay" : NumberLong(900),
"votes" : 0
},
{
"_id" : 5,
"host" : "mongo-c:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 6,
"host" : "mongo-b:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
rs.status() result (partial):
{ [1881/1930]
"_id" : 5,
"name" : "mongo-c:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 3435,
"optime" : {
"ts" : Timestamp(1637525581, 2),
"t" : NumberLong(87)
},
"optimeDurable" : {
"ts" : Timestamp(1637525443, 1),
"t" : NumberLong(87)
},
"optimeDate" : ISODate("2021-11-21T20:13:01Z"),
"optimeDurableDate" : ISODate("2021-11-21T20:10:43Z"),
"lastHeartbeat" : ISODate("2021-11-24T07:08:32.108Z"),
"lastHeartbeatRecv" : ISODate("2021-11-24T07:08:34.867Z"),
"pingMs" : NumberLong(2355),
"lastHeartbeatMessage" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1637524307, 1),
"electionDate" : ISODate("2021-11-21T19:51:47Z"),
"configVersion" : 23,
"configTerm" : 87
},
{
"_id" : 6,
"name" : "mongo-b:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 7995003,
"optime" : {
"ts" : Timestamp(1637737714, 24),
"t" : NumberLong(88)
},
"optimeDate" : ISODate("2021-11-24T07:08:34Z"),
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1637525581, 1),
"electionDate" : ISODate("2021-11-21T20:13:01Z"),
"configVersion" : 23,
"configTerm" : 88,
"self" : true,
"lastHeartbeatMessage" : ""
}

pymongo unable to connect to primary

I'm trying to find_one by connecting to my replica set's primary node.
MongoClient(hostname, replicaSet="rs0", read_preference=ReadPreference.PRIMARY)
But it results in an error:
ServerSelectionTimeoutError: No replica set members match selector
"Primary()"
I'm able to successfully read using SECONDARY_PREFERRED. I also tried connecting using MongoReplicaSetClient with no success. I'm guessing this due to bad configuration, but what should I be looking for?
rs.status:
rs0:PRIMARY> rs.conf()
{
"_id" : "rs0",
"version" : 111313,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 1,
"host" : "ANDROMEDA:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "mongo02.db.com:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0.5,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 3,
"host" : "mongo03.db.com:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0.5,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
}
}
}
rs0:SECONDARY> rs.status()
{
"set" : "rs0",
"date" : ISODate("2016-08-04T08:58:02.293Z"),
"myState" : 2,
"term" : NumberLong(90),
"syncingTo" : "mongo03.db.com:27017",
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 1,
"name" : "ANDROMEDA:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 2503,
"optime" : {
"ts" : Timestamp(1470299746, 1),
"t" : NumberLong(90)
},
"optimeDate" : ISODate("2016-08-04T08:35:46Z"),
"lastHeartbeat" : ISODate("2016-08-04T08:58:01.109Z"),
"lastHeartbeatRecv" : ISODate("2016-08-04T08:58:01.803Z"),
"pingMs" : NumberLong(28),
"electionTime" : Timestamp(1469600522, 1),
"electionDate" : ISODate("2016-07-27T06:22:02Z"),
"configVersion" : 111313
},
{
"_id" : 2,
"name" : "mongo02.db.com:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 7604104,
"optime" : {
"ts" : Timestamp(1470299746, 1),
"t" : NumberLong(90)
},
"optimeDate" : ISODate("2016-08-04T08:35:46Z"),
"syncingTo" : "mongo03.db.com:27017",
"configVersion" : 111313,
"self" : true
},
{
"_id" : 3,
"name" : "mongo03.db.com:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2503,
"optime" : {
"ts" : Timestamp(1470299746, 1),
"t" : NumberLong(90)
},
"optimeDate" : ISODate("2016-08-04T08:35:46Z"),
"lastHeartbeat" : ISODate("2016-08-04T08:58:01.948Z"),
"lastHeartbeatRecv" : ISODate("2016-08-04T08:58:01.802Z"),
"pingMs" : NumberLong(28),
"syncingTo" : "ANDROMEDA:27017",
"configVersion" : 111313
}
],
"ok" : 1
}
In cases with an error message similar to ServerSelectionTimeoutError: No replica set members match selector "Primary()" and where replica set status function output, rs.status(), shows the Primary member using a non-FQDN (ANDROMEDA:27017 as in this case) then it is highly likely the application is unable to resolve the Primary host on the network.
You can easily check this using the following commands from the command line of the host running your application:
$ dig ANDROMEDA
$ ping ANDROMEDA
$ mongo --host ANDROMEDA:27017
If you don't have the Mongo Shell installed on the host running your application you can use Telnet instead.
$ telnet ANDROMEDA 27017
These outputs will allow you to check connectivity between your application host and your mongod host to determine if this is causing the problem.
I use this connection string:
MongoClient('mongodb://mongo01.db.com:27017,mongo02.db.com:27017,mongo03.db.com:27017/mydb',replicaSet="rs0", read_preference=ReadPreference.PRIMARY)

mongo replica set - data not replicating

I setup a test replcia set using mongo v3.2 on a single node and the rs.status() command in the mongo shell seems to indicate that the replica set is established (output below).
However, when I insert data into the primary, I can see that it gets inserted into the primary but I cannot seem to run the find operation on any of the secondary processes (message given below)
message on secondary
Error: error: { "ok" : 0, "errmsg" : "not master and slaveOk=false", "code" : 13435 }
rs.status()
{
"set" : "set0",
"date" : ISODate("2016-07-21T19:53:41.882Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "127.0.0.1:27049",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 2388,
"optime" : {
"ts" : Timestamp(1469130790, 2),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-07-21T19:53:10Z"),
"electionTime" : Timestamp(1469128700, 1),
"electionDate" : ISODate("2016-07-21T19:18:20Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "127.0.0.1:27050",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2132,
"optime" : {
"ts" : Timestamp(1469130790, 2),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-07-21T19:53:10Z"),
"lastHeartbeat" : ISODate("2016-07-21T19:53:40.964Z"),
"lastHeartbeatRecv" : ISODate("2016-07-21T19:53:40.823Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "127.0.0.1:27049",
"configVersion" : 1
},
{
"_id" : 2,
"name" : "127.0.0.1:27051",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2132,
"optime" : {
"ts" : Timestamp(1469130790, 2),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2016-07-21T19:53:10Z"),
"lastHeartbeat" : ISODate("2016-07-21T19:53:40.963Z"),
"lastHeartbeatRecv" : ISODate("2016-07-21T19:53:40.823Z"),
"pingMs" : NumberLong(0),
"syncingTo" : "127.0.0.1:27049",
"configVersion" : 1
}
],
"ok" : 1
}
output from rs.conf() on primary
{
"_id" : "set0",
"version" : 1,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "127.0.0.1:27049",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 1,
"host" : "127.0.0.1:27050",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 2,
"host" : "127.0.0.1:27051",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("57911ff0bd131f1aeb2ef878")
}
}
After I ran the following command on the secondary process, I was able to read the data from the secondary:
rs.slaveOk()
or
db.getMongo().setSlaveOk()
After running the following command from the Mongo shell, I was able to get the required documents from Mongo using the find command.

I want to know how to connect my mongo client to my replication set

I realized that the data oplog.rs is different from my client oplog.rs. Is the connection supposed to automatic once I initiated the replication set ? Or is there a step I missed in the process.
I was following this tutorial http://www.acemyskills.com/replica-sets-in-mongodb/
config = {
_id:"acemyskillsrepsets",
members: [
{_id: 0, host: "localhost:27017"},
{_id: 1, host: "localhost:27018"},
{_id: 2, host: "localhost:27019"}
]
};acemyskillsrepsets:PRIMARY> rs.status()
{
"set" : "acemyskillsrepsets",
"date" : ISODate("2016-07-24T01:35:16.869Z"),
"myState" : 1,
"term" : NumberLong(19),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 63,
"optime" : {
"ts" : Timestamp(1469324080, 1),
"t" : NumberLong(19)
},
"optimeDate" : ISODate("2016-07-24T01:34:40Z"),
"electionTime" : Timestamp(1469324079, 1),
"electionDate" : ISODate("2016-07-24T01:34:39Z"),
"configVersion" : 2,
"self" : true
},
{
"_id" : 1,
"name" : "localhost:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 25,
"optime" : {
"ts" : Timestamp(1469324080, 1),
"t" : NumberLong(19)
},
"optimeDate" : ISODate("2016-07-24T01:34:40Z"),
"lastHeartbeat" : ISODate("2016-07-24T01:35:15.260Z"),
"lastHeartbeatRecv" : ISODate("2016-07-24T01:35:16.014Z"
),
"pingMs" : NumberLong(0),
"syncingTo" : "localhost:27019",
"configVersion" : 2
},
{
"_id" : 2,
"name" : "localhost:27019",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 42,
"optime" : {
"ts" : Timestamp(1469324080, 1),
"t" : NumberLong(19)
},
"optimeDate" : ISODate("2016-07-24T01:34:40Z"),
"lastHeartbeat" : ISODate("2016-07-24T01:35:16.020Z"),
"lastHeartbeatRecv" : ISODate("2016-07-24T01:35:15.622Z"
),
"pingMs" : NumberLong(0),
"syncingTo" : "localhost:27017",
"configVersion" : 2
}
],
"ok" : 1
}

How come that primary member behind secondary in one mongodb replica set

On our production environment we have strange behavior of a mongo replica, our primary is always behind secondaries.
rs.status():
{
"set" : "repl01",
"date" : ISODate("2014-02-20T11:11:28.000Z"),
"myState" : 2,
"syncingTo" : "prodsrv04:27018",
"members" : [
{
"_id" : 0,
"name" : "prodsrv02:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 6271,
"optime" : Timestamp(1392894670, 97),
"optimeDate" : ISODate("2014-02-20T11:11:10.000Z"),
"self" : true
},
{
"_id" : 1,
"name" : "prodsrv03:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 6270,
"optime" : Timestamp(1392894670, 68),
"optimeDate" : ISODate("2014-02-20T11:11:10.000Z"),
"lastHeartbeat" : ISODate("2014-02-20T11:11:28.000Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00.000Z"),
"pingMs" : 2
},
{
"_id" : 2,
"name" : "prodsrv04:27018",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 6270,
"optime" : Timestamp(1392894669, 113),
"optimeDate" : ISODate("2014-02-20T11:11:09.000Z"),
"lastHeartbeat" : ISODate("2014-02-20T11:11:27.000Z"),
"lastHeartbeatRecv" : ISODate("2014-02-20T11:11:28.000Z"),
"pingMs" : 6
}
],
"ok" : 1
}
Master optime: Timestamp(1392894669, 113);
Slave optime : Timestamp(1392894670, 68);
How come?