mongodb replica set unreachable - mongodb

Question

I am trying to configure a standalone mongodb replica set with 3 instances. I seem to have gotten into a funky state. Two of my instances went down, and I was left with all secondary nodes. I tried to follow this: http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/
I got this error though:
rs0:SECONDARY> rs.reconfig(cfg, {force : true})
{
"errmsg" : "exception: need most members up to reconfigure, not ok : obfuscated_hostname:27019",
"code" : 13144,
"ok" : 0
}
When I look at the logs I see this:
Fri Aug 2 20:45:11.895 [initandlisten] options: { config: "/etc/mongodb1.conf",
dbpath: "/var/lib/mongodb1", logappend: "true", logpath: "/var/log/mongodb/mongodb1.log",
port: 27018, replSet: "rs0" }
Fri Aug 2 20:45:11.897 [initandlisten] journal dir=/var/lib/mongodb1/journal
Fri Aug 2 20:45:11.897 [initandlisten] recover begin
Fri Aug 2 20:45:11.897 [initandlisten] recover lsn: 0
Fri Aug 2 20:45:11.897 [initandlisten] recover /var/lib/mongodb1/journal/j._0
Fri Aug 2 20:45:11.899 [initandlisten] recover cleaning up
Fri Aug 2 20:45:11.899 [initandlisten] removeJournalFiles
Fri Aug 2 20:45:11.899 [initandlisten] recover done
Fri Aug 2 20:45:11.923 [initandlisten] waiting for connections on port 27018
Fri Aug 2 20:45:11.925 [websvr] admin web console waiting for connections on port 28018
Fri Aug 2 20:45:11.927 [rsStart] replSet I am hostname_obfuscated:27018
Fri Aug 2 20:45:11.927 [rsStart] replSet STARTUP2
Fri Aug 2 20:45:11.929 [rsHealthPoll] replset info hostname_obf:27017 thinks that we are down
Fri Aug 2 20:45:11.929 [rsHealthPoll] replSet member hostname_obf:27017 is up
Fri Aug 2 20:45:11.929 [rsHealthPoll] replSet member hostname_obf:27017 is now in state SECONDARY
Fri Aug 2 20:45:12.587 [initandlisten] connection accepted from ip_obf:52446 #1 (1 connection now open)
Fri Aug 2 20:45:12.587 [initandlisten] connection accepted from ip_obf:52447 #2 (2 connections now open)
Fri Aug 2 20:45:12.588 [conn1] end connection ip_obf:52446 (1 connection now open)
Fri Aug 2 20:45:12.928 [rsSync] replSet SECONDARY
I'm unable to connect to the mongo instances, even though the logs say that it is up and running. Any ideas on what to do here?

Answer 1

You did not mention which version of mongodb you are using, but I assume it is post-2.0.
I think the problem with your forced reconfiguration is that after this reconfiguration, you still need to have the minimum number of nodes for a functioning replica set, i.e. 3. But since you originally had 3 members and lost 2, there is no way you could turn that single surviving node into a functioning replica set.
Your only option for recovery would be to bring up the surviving node as a stand-alone server, backup the database, and then create a new 3-node replica set with that data.

Answer 2

Yes you can turn up a single secondary replica to primary if the secondary server is running fine.Do follow the below simple steps:
Step 1: Connect to member and check the current configuration
rs.conf()
Step 2: Save the current configuration to another variable.
x = rs.conf()
Step 3: Select the id,host and port of the member that is to be made as primary.
x.members = [{"_id":1,"host" : "localhost.localdomain:27017"}]
Step 4: Reconfigure the new replica set by force.
rs.reconfig(x, {force:true})
Now the desired member will be promoted as the primary.