mongodb replica set changes take two attempts to work - mongodb

Question

I'm using using Mongodb 2.6.1
I have a 4 node solution across two data centres (2 dbs, 2 arbiters but one arbiter is always out of the replicate set)
{
"_id": "prelRS",
"members": [
{
"_id": 1,
"host": "serverInDataCenter1:27011",
"priority": 6
},
{
"_id": 3,
"host": "serverInDataCenter2:27013",
"priority": 0
},
{
"_id": 5,
"host": "serverInDataCenter1:27015",
"arbiterOnly": true
}
]
}
when we have a DR situation and need to use the DataCenter2 only, we will
and when I try to take the primary out of the relica set and make the secondary be the primary it takes two attempts to force the configuration to apply, for what seems like transient state issues. below was applied to the 27013 node, and all done in the space of a few seconds.
prelRS:SECONDARY> cfg={
... "_id": "prelRS",
... "members": [
... {
... "_id": 3,
... "host": "serverInDataCenter2:27013",
... "priority": 4
... },
... {
... "_id": 6,
... "host": "serverInDataCenter2:27016",
... "arbiterOnly": true
... }
... ]
... }
{
"_id" : "prelRS",
"members" : [
{
"_id" : 3,
"host" : "serverInDataCenter2:27013",
"priority" : 4
},
{
"_id" : 6,
"host" : "serverInDataCenter2:27016",
"arbiterOnly" : true
}
]
}
prelRS:SECONDARY> rs.reconfig(cfg, {force : true})
{
"errmsg" : "exception: need most members up to reconfigure, not ok : serverInDataCenter2:27016",
"code" : 13144,
"ok" : 0
}
prelRS:SECONDARY> rs.reconfig(cfg, {force : true})
{ "ok" : 1 }
prelRS:SECONDARY>
this also seems to be the case when I am adding the 27011 node back in as well (as a lower priority replica) from the 27013 node
cfg={
"_id": "prelRS",
"members": [
{
"_id": 1,
"host": "serverInDataCenter1:27011",
"priority": 2
},
{
"_id": 3,
"host": "serverInDataCenter2:27013",
"priority": 4
},
{
"_id": 5,
"host": "serverInDataCenter1:27015",
"arbiterOnly": true
}
]
}
prelRS:PRIMARY> rs.reconfig(cfg)
{
"errmsg" : "exception: need most members up to reconfigure, not ok : serverInDataCenter1:27015",
"code" : 13144,
"ok" : 0
}
prelRS:PRIMARY> rs.reconfig(cfg)
2014-08-08T20:53:03.192+0100 DBClientCursor::init call() failed
2014-08-08T20:53:03.193+0100 trying reconnect to 127.0.0.1:27013 (127.0.0.1) failed
2014-08-08T20:53:03.193+0100 reconnect 127.0.0.1:27013 (127.0.0.1) ok
reconnected to server after rs command (which is normal)
but it doesn't seem to happen when i make this node primary again with the first config i mentioned (applied on the 27011 node)
but this action isn't adding arbiters to the set, so maybe that's a clue as to what is going on?
I realize now that I probably need to leave 27011 in the replica set as priority 0 during DR, even though it is not available, but if all of dataCenter1 was not available, i would still have to add 27016 to the set and take 27105 out, and would face the error above when invoking DR
any suggestions why this takes two attempts to work?
thanks

Answer 1

The problem is that your arbiters aren't "real" replica set members, so for some operations you're not going to be able to reach a quorum on the set.