MonogoDB Replica Set Status Not changing from Startup to Secondary - mongodb

Question

I have setup a MongoDB replica set with 3 nodes(vm's running CentOS). One node became Primary other 2 stuck in Startup. When these 2 nodes will change their states from startup to secondary.
aryabhata:PRIMARY> rs.status()
{
"set" : "aryabhata",
"date" : ISODate("2016-04-30T08:10:45.173Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "localhost.localdomain:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 69091,
"optime" : Timestamp(1461935462, 1),
"optimeDate" : ISODate("2016-04-29T13:11:02Z"),
"electionTime" : Timestamp(1461934754, 1),
"electionDate" : ISODate("2016-04-29T12:59:14Z"),
"configVersion" : 459192,
"self" : true
},
{
"_id" : 1,
"name" : "repset1.com:27017",
"health" : 1,
"state" : 0,
"stateStr" : "STARTUP",
"uptime" : 92,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2016-04-30T08:10:44.485Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0,
"configVersion" : -2
},
{
"_id" : 2,
"name" : "repset2.com:27017",
"health" : 1,
"state" : 0,
"stateStr" : "STARTUP",
"uptime" : 68382,
"lastHeartbeat" : ISODate("2016-04-30T08:10:43.974Z"),
"lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
"pingMs" : 0,
"configVersion" : -2
}
],
"ok" : 1
}

Answer 1

My problem fix with set ip address for Primary instead hostname
cfg = rs.conf()
cfg.members[0].host = "public-or-private-primary-ip:27017"
rs.reconfig(cfg)
after that secondary state change to STARTUP2

Answer 2

From primary check whether you are able to connect to secondary
mongo --host repset1.com --port 27017
When the above one fails may be firewall or BindIP issue.
Check bind_ip (should be 0.0.0.0, change in mongodb.conf is it's 127.0.0.1):
netstat -plunt | grep :27017 | grep LISTEN
Look at the log-files of secondaries, why they are stuck. Did they receive the configuration details?
Try to reconfigure, see mongo replicaset reconfigure

Answer 3

For me the problem was that primary had authorization enabled. In this case the secondaries always stayed in STARTUP.
To use authorization you need to set keyFile in configuration file of all nodes (primary and secondary).
Create mongodb key file on linux:
openssl rand -base64 741 > mongodb.key
chmod 600 mongodb.key
chown mongod:mongod mongodb.key
mongod.conf file:
replication:
replSetName: rs0
security:
authorization: enabled
keyFile: /home/mongodb.key
Source MongoDB replica set with simple password authentication

Answer 4

It requires that both the primary resolves the host name to IP of the secondary and also the secondary resolves the hostname of the primary to an IP.
In my case I forgot to add in hosts file for the secondary to resolve the hostname of the primary. Once I updated the hosts file in the secondary, the state of the secondary transitioned to STARTUP2 and then to SECONDARY.