I have a replica of three mongod instances, one of them is the primary instance, and the other two are secondary. I connected a mongo instance to the primary, and then I turned off the mongod primary instance, simulating a failure.
Typically, as soon as I initiated a command, mongo generated a DBClientCursor::init call() failed error.
My question is, in case of such a failure, where the primary cannot go back to service, and there will definitely be an election for the new primary, is there a way to tell mongo to automatically search and connect to the new primary of the replica set?
Possibly your problem is that you're only connecting to one node, the one which is primary at that time. What you need to to do is to specify, when connecting, what the replicaset is and what the address is of each available node - so your client can reconnect to a different node if its primary falls over.
The documentation on the Mongo URI connection string format explains this:
hostX : Optional. You can specify as many hosts as necessary. You would specify multiple hosts, for example, for connections to replica sets.
It gives this example:
EXAMPLE
To describe a connection to a replica set named test, with the following mongod hosts:
db1.example.net on port 27017 and
db2.example.net on port 2500.
You would use a connection string that resembles the following:
mongodb://db1.example.net,db2.example.net:2500/?replicaSet=test
You may need to check your driver's documentation also, in case it requires a particular way of specifying the replica set and the list of nodes.
This was not the problem. The replica set was well defined and configured. The problem was really weird, I'm guessing it is a bug in the terminal. All I did was I closed the terminal, opened a new one, and it was working like a charm
Related
I'm fairly new to mongodb, just a couple of months. I just converted my mongodb database to support a secondary replica set so I can watch collections. I only added one secondary which I'm guessing now may not be the best after reading you should create an odd number, but it is a localhost on one machine. I went through the instructions, got replication working fine for for half a day running my programs. But for some reason recently it has switched the database for port 27017 from primary to secondary. Primary was previously on localhost:27017 and secondary was on localhost:27027. Now my normal program can't connect to localhost:27017 without an error, which I believe it is because it is a secondary replica set now when it was primary before, assuming you can only connect to a primary. Here is the error msg.
Exception in thread "main" com.mongodb.MongoNotPrimaryException: Command failed with error 10107 (NotWritablePrimary): 'not master' on server localhost:27017.
I'm perplexed why mongodb switched the replica set primary in the first place. I doubt an error occurred, but certainly possible, but I haven't had a single "localhost" error in months of development.
For now, ideally how would I switch 27017 back to be the primary. How do I do that so my existing programs can function again?
Eventually when in production, what is the best methodology to handle this, assuming a lookup to a DNS entry to an ip address and suddenly the primary gets changed because of a fail over?
Given question 3 is a bit more involved, is there something I can do in my development environment to better simulate a production environment.
I use StackOverflow extensively but this is my first post so thanks for anyone who can provide advice.
Without knowing more about the replica configuration and circumstances of the switch over I'm not sure anyone could confidently answer question 1 but it may not be important compared to question 3.
When you want to manually switch the primary you can manipulate the priority settings:
https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/
Or run manual commands to freeze or step down the current primary:
https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/#force-a-member-to-be-primary-using-database-commands
The safest option is to ensure your application is aware of all replicas in the replica set. Then when you have these situations where something unexpected has happened the application will fail over to a writable db without any issues.
https://mongodb.github.io/mongo-java-driver/3.4/driver/tutorials/connect-to-mongodb/#connect-to-a-replica-set
I can only suggest setting up some VMs or containers as replica set members to better represent a production environment.
https://hub.docker.com/_/mongo
I was able to solve the problem by using the connect string which comprised both replica sets, which I was unaware I needed to do. Such as for java:
mongoClient = MongoClients.create("mongodb://localhost:27017,localhost:27027");
This also worked for Mongo Compass so I was able to connect to the secondary database. I didn't know you needed to provide paths to all replica sets when trying to connect, but in retrospect makes goods sense if something is down.
If you need a replica set for testing, you can create a single-node RS. Follow the instructions for creating a RS but only add one node.
I have some questions about the mongo replica
mongo replica
If I make 1 primary and 2 secondary MongoDB for replication. So I have 3 endpoints to 3 different DB and my apps can only write on primary DB. what if suddenly my primary shutdown and secondary DB take over the primary. Then how to automatically change the endpoint in my apps? should I use mongos (mongo routes)? but it needs sharding if I remember correctly.
Thank you.
All nodes in a replica set work together to have identical data. Secondary nodes may lag behind the primary, but you don't get "3 different DB". There is only one database of which copies exist on each node.
All MongoDB drivers know to monitor replica set members and discover which is the primary one automatically. You need to configure some drivers to do so by providing the replica set name, others do it automatically by default when they connect to a replica set node. Look up "connecting to replica set" in your driver documentation.
In a proper connection string you will provide all three RS members, e.g.
mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?replicaSet=myRepl
The client will detect the PRIMARY and will use it. I guess, most drivers will re-connect automatically if the PRIMARY node changes.
Most drivers will detect the PRIMARY automatically if you provide the ReplicaSet name, i.e.
mongodb://mongodb0.example.com:27017/?replicaSet=myRepl
would connect to the PRIMARY even if it is not mongodb0.example.com. However, if mongodb0.example.com does not run, then you don't connect at all. So, it is beneficial to provide all ReplicaSet members in the connection string.
See Connection String URI Format
mongos is needed only to connect to a Sharded Cluster.
I am using MongoDB with Loopback in my application with a loopback connector to the MongoDB. My application was working fine but now it throws an error
not master and slaveOk=false.
try running rs.slaveOk() in a mongoDB shell
You are attempting to connect to secondary replica whilst previously your app (connection) was set to connect likely to the primary, hence the error. If you use rs.secondaryOk() (slaveOk is deprecated now) you will possibly solve the connection problem but it might not be what you want.
To make sure you are doing the right thing, think if you want to connect to the secondary replica instead of primary. Usually, it's not what you want.
If you have permissions to amend the replica configuration
I suggest to connect using MongoDB Compass and execute rs.status() first to see the existing state and configuration for the cluster. Then, verify which replica is primary.
If necessary, adjust priorities in the replicaset configuration to assign primary status to the right replica. The highest priority number sets the replica as primary. This article shows how to do it right.
If you aren't able to change the replica configuration
Try a few things:
make sure your hostname points to the primary replica
if it is a local environment issue - make sure you added your local replica hostnames to the /etc/hosts pointing to 127.0.0.1
experiment with directConnection=true
experiment with multiple replica hosts and ?replicaSet=<name> - read this article (switch tabs to replica)
The best bet is that your database configuration has changed and your connection string no longer reflects it correctly. Usually, slight adjustments in the connection string are needed or just checking to what instance you want to connect.
I created a replica set with three members.
now I read the following link: Replica Set Elections
which explains only in an abstract way what is going on.
So they say that when the primary becomes unavailable the elections occur.
So I was trying to make the primary unavailable by ctrl+c in the terminal which holds the mongod instance, the other two secondaries were sending each other heartbeats to connect but I got in each terminal:
Failed to connect to 127.0.1.1:27017, reason: errno:111 Connection refused
So maybe that's not the way to make the primary unavailable?
Can you help me please, I cant find examples..
Thanks.
Simplest option provided by Mongo is,
rs.stepdown()
Use this function on primary replica to make it unavailable for specified period of time. This causes other members to reelect new primary.
http://docs.mongodb.org/manual/reference/method/rs.stepDown/
I had the same problem, it turns out that Ubuntu had the FQDN set to 127.0.1.1 in /etc/hosts.
Either make mongod listen on 127.0.1.1 or edit /etc/hosts to remove the 127.0.1.1 entry or change it to 127.0.0.1.
This resolved my issue.
If a replica set is set up in Mongo with only two nodes, an arbiter needs to be added to ensure that there is always a majority in a vote for a new master. The arbiter never becomes the master itself, it is purely there to provide the casting vote in an otherwise neck-and-neck election.
When connecting a client (in my case Java) to a MongoDB cluster, we are supposed to specify all the nodes of the cluster in the connection configuration:
List addrs = new ArrayList();
addrs.add( new ServerAddress( "localhost" , 27017 ) );
addrs.add( new ServerAddress( "localhost" , 27018 ) );
Mongo mongo = new Mongo(addrs);
Should the arbiter be included in the connection configuration? I would guess not as they:
don't have a copy of the data and will never become the primary node (or even a readable secondary)
(Taken from here)
...but I just wanted to double check!
No, you shouldn't need to include arbiters in the connection.
After all, as you suspected, there would be little point in your code trying to connect to one of those as there is no data there. They just do their stuff behind the scenses to help auto-failover.
You don't even have to specify all the servers in your connection configuration (not even the master) - as long as one of the servers you do mention returns a response, it can find the master from there. Though IMHO, the more you name the better, just incase a number of them go down.
The list is just a seed list. The actual members of the replica set are determined after connecting to one. You could just specify the one at 27017 (but it would be bad if that one were down).