How mongo db replicaset works in Azure - mongodb

Question

I have deployed the Mongo DB replica set using this template mongodb-replica-set-centos.
Mongo DB VM 1 (primary):
ps aux | grep mongo
root 10161 0.7 0.5 797140 40900 ? SLl 05:18 0:05 mongod --dbpath /var/lib/mongo/ --replSet repset --logpath /var/log/mongodb/mongod.log --fork --config /etc/mongod.conf
sshuser 10347 0.0 0.0 112640 960 pts/0 S+ 05:29 0:00 grep --color=auto mongo
Mongo DB database:-
mongo -u mongoadmin -p mongoadmin admin
MongoDB shell version: 3.2.19
connecting to: admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2018-03-23T05:18:21.137+0000 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-03-23T05:18:21.137+0000 I CONTROL [initandlisten]
repset:PRIMARY> rs.status()
{
"set" : "repset",
"date" : ISODate("2018-03-23T07:38:45.694Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "52.170.83.3:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 8426,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"electionTime" : Timestamp(1521782318, 1),
"electionDate" : ISODate("2018-03-23T05:18:38Z"),
"configVersion" : 2,
"self" : true
},
{
"_id" : 1,
"name" : "10.0.1.5:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 8407,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"lastHeartbeat" : ISODate("2018-03-23T07:38:45.538Z"),
"lastHeartbeatRecv" : ISODate("2018-03-23T07:38:42.546Z"),
"pingMs" : NumberLong(1),
"configVersion" : 2
}
],
"ok" : 1
}
Mongo DB VM 2 (secondary):
ps aux | grep mongo
root 10115 0.4 0.5 447908 37892 ? SLl 05:11 0:17 mongod --dbpath /var/lib/mongo/ --config /etc/mongod.conf --replSet repset --logpath /var/log/mongodb/mongod.log --fork
sshuser 10269 0.0 0.0 112640 960 pts/0 S+ 06:21 0:00 grep --color=auto mongo
Mongo DB database:-
mongo -u mongoadmin -p mongoadmin admin
MongoDB shell version: 3.2.19
connecting to: admin
2018-03-23T07:38:54.311+0000 E QUERY [thread1] Error: Authentication failed. :
DB.prototype._authOrThrow#src/mongo/shell/db.js:1441:20
#(auth):6:1
#(auth):1:2
exception: login failed
Mongo DB VM 3(secondary):-
ps aux | grep mongo
root 10122 0.6 0.5 795472 40420 ? SLl 05:12 0:26 mongod --dbpath /var/lib/mongo/ --config /etc/mongod.conf --replSet repset --logpath /var/log/mongodb/mongod.log --fork
sshuser 10381 0.0 0.0 112640 960 pts/0 S+ 06:21 0:00 grep --color=auto mongo
Mongo DB database:-
mongo -u mongoadmin -p mongoadmin admin
MongoDB shell version: 3.2.19
connecting to: admin
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
http://docs.mongodb.org/
Questions? Try the support group
http://groups.google.com/group/mongodb-user
Server has startup warnings:
2018-03-23T05:12:19.613+0000 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-03-23T05:12:19.613+0000 I CONTROL [initandlisten]
repset:SECONDARY> rs.status()
{
"set" : "repset",
"date" : ISODate("2018-03-23T07:39:04.009Z"),
"myState" : 2,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "52.170.83.3:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 8425,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"lastHeartbeat" : ISODate("2018-03-23T07:39:02.571Z"),
"lastHeartbeatRecv" : ISODate("2018-03-23T07:39:03.573Z"),
"pingMs" : NumberLong(1),
"electionTime" : Timestamp(1521782318, 1),
"electionDate" : ISODate("2018-03-23T05:18:38Z"),
"configVersion" : 2
},
{
"_id" : 1,
"name" : "10.0.1.5:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 8806,
"optime" : {
"ts" : Timestamp(1521782318, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2018-03-23T05:18:38Z"),
"infoMessage" : "could not find member to sync from",
"configVersion" : 2,
"self" : true
}
],
"ok" : 1
}
Questions:-
Why I am unable to login into Mongo DB VM 2?
After I shutdown Mongo DB VM 3, will Mongo DB VM 2 acts as a secondary node?
If I shutdown Mongo DB VM 1, Will any one of secondary node act as a primary node?

Answer 1

All three questions are answered by the same fact: DB VM2 is not part of the replica set. It's clear from the rs.status() information that only two nodes are registered as part of the replica set, VM1 and VM3.
The implications are that:
On DB VM2, it is not part of the replica set so it does not have the authentication credentials you are trying to log in with
No, DB VM2 will not act as a secondary node; because it is not part of the replica set
In the current setup, with only 2 nodes in the replica set, if you shut down either node (VM1 or VM3) then the other node will not elect itself primary, because it cannot command a majority in an election.
Take a look at the docs on Replica Set Elections to understand what the majority is and why it matters; and take a look at DB VM2 to understand why it is not part of your replica set. Did you ever actually add it?