and thanks in advance for your time.
For a given sharded setup, mongos is launched while specifying the config server(s) to talk to. Say we start with the following mongos option:
--configdb=cf1,cf2,cf3
Everything is all fine and dandy. If you were to relaunch mongos (or launch a different mongos) with:
-- configdb=cf3,cf2,cf1
It results in the following error:
Tue Jul 9 23:32:41 uncaught exception: error: { "$err" : "could not initialize sharding on connection rs1/db1.me.net:27017,db2.me.net:27017,db3.me.net:27017, :: caused by :: mongos specified a different config database string : stored :cfg1:27017,cfg2:27017,cfg3:27017 vs given :cfg3:27017,cfg2:27017,cfg1:27017","code" : 15907}
My question is, what is the reasoning mongo sensitive to the order of the config server string? I would imagine at some point it parses the different servers hostnames/port, so why not just compare the set? I know you can see from the source code that its just a string comparison, but my question is the underlying reason for this.
Some context to this problem: I am using chef for my mongo deployments. We recently went through the exercise of migrating the config server with the same hostname. However, this still ended up being a disruptive process because the order the chef picked up the config servers had changed, thus changing order mongos starts its process with. I understand that this issue is directly because of chefs functionality, but I am curious as to why Mongo is not this flexible.
Thanks again for your time.
When mongos process changes metadata for sharded cluster, it has to change it in all three config servers "simultaneously" (i.e. all three must agree in order to have a valid metadata change).
If the system were to go down in the middle of such a metadata change, if the config database order was not fixed, there would be a lot more possible permutations of incorrect states to unwind. Requiring a fixed sequence of config dbs allows (a) simpler checking of whether all members of the cluster are viewing the same metadata (b) significant reduction of possible states when a system crashes or otherwise stops unexpectedly.
In addition it reduces chances for "race condition" sorts of bugs if different mongos' could initiate the same operations on different config servers. Even as simple a change as mongos process taking a "virtual" distributed lock to see if a chance is necessary - how could you handle the case of different mongos' checking config servers in different order to check on (and take out) the lock?
As a summary, the three config servers are not a replica set, but one of them still has to be the one that always accepts the changes "first" - think of the order of configdbs to mongos as designation of such "first" status.
Related
I'm fairly new to mongodb, just a couple of months. I just converted my mongodb database to support a secondary replica set so I can watch collections. I only added one secondary which I'm guessing now may not be the best after reading you should create an odd number, but it is a localhost on one machine. I went through the instructions, got replication working fine for for half a day running my programs. But for some reason recently it has switched the database for port 27017 from primary to secondary. Primary was previously on localhost:27017 and secondary was on localhost:27027. Now my normal program can't connect to localhost:27017 without an error, which I believe it is because it is a secondary replica set now when it was primary before, assuming you can only connect to a primary. Here is the error msg.
Exception in thread "main" com.mongodb.MongoNotPrimaryException: Command failed with error 10107 (NotWritablePrimary): 'not master' on server localhost:27017.
I'm perplexed why mongodb switched the replica set primary in the first place. I doubt an error occurred, but certainly possible, but I haven't had a single "localhost" error in months of development.
For now, ideally how would I switch 27017 back to be the primary. How do I do that so my existing programs can function again?
Eventually when in production, what is the best methodology to handle this, assuming a lookup to a DNS entry to an ip address and suddenly the primary gets changed because of a fail over?
Given question 3 is a bit more involved, is there something I can do in my development environment to better simulate a production environment.
I use StackOverflow extensively but this is my first post so thanks for anyone who can provide advice.
Without knowing more about the replica configuration and circumstances of the switch over I'm not sure anyone could confidently answer question 1 but it may not be important compared to question 3.
When you want to manually switch the primary you can manipulate the priority settings:
https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/
Or run manual commands to freeze or step down the current primary:
https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/#force-a-member-to-be-primary-using-database-commands
The safest option is to ensure your application is aware of all replicas in the replica set. Then when you have these situations where something unexpected has happened the application will fail over to a writable db without any issues.
https://mongodb.github.io/mongo-java-driver/3.4/driver/tutorials/connect-to-mongodb/#connect-to-a-replica-set
I can only suggest setting up some VMs or containers as replica set members to better represent a production environment.
https://hub.docker.com/_/mongo
I was able to solve the problem by using the connect string which comprised both replica sets, which I was unaware I needed to do. Such as for java:
mongoClient = MongoClients.create("mongodb://localhost:27017,localhost:27027");
This also worked for Mongo Compass so I was able to connect to the secondary database. I didn't know you needed to provide paths to all replica sets when trying to connect, but in retrospect makes goods sense if something is down.
If you need a replica set for testing, you can create a single-node RS. Follow the instructions for creating a RS but only add one node.
We are currently setting up our mongodb environment for production. At the moment we only have one dedicated mongodb database server. We will expand this in the near future with a 2nd server and I already indicated to the management that for the ideal situation we should get a 3rd server as well.
Since I already know we're going to use sharding and replication in the near future I want to be prepared for it.
The idea I have now is to start now with the Development Configuration (as mongo's documentation names it).
Whenever our second server comes available I would like to expand this setup to a configuration with 2 configuration servers en 2 shards (replica sets).
And of course when our third server comes available have the fully functional sharded cluster configuration.
While reading mongo's documentation I was getting triggered by the note that de Development setup should not be used in production.
MongoDb Development Configuration
Keeping in mind that we will add more servers soon, would it be a bad idea to already configure the Development Configuration already so we can easily add the 2nd server to the cluster when it comes available?
After setting up the 'development sharded setup' I've found my anwser. Of course i'm happy to share in case anybody runs into the same questions as I do when starting with this.
In my case, it was ok to start with the development setup untill my new servers arrived. It was a temporary situation and when my new servers arived I was able to easily expand my replicasets. There are a number of reasons why this isn't adviced for production:
To state the obvious, there is no replication yet. Since I was running shards on one machine there is a single point of failure. If the machine, or one node goes down, the cluster won't work anymore.
Now this part is interesting. After I added a second server, I did have primary and secondary nodes. Primary nodes were used for writing and secondary for reading. I've eliminated the issue that there was no replication AND my data had a higher availability. However, I noticed with the 2-member replica sets, if one member of the replicaset went down (even is this was a secondary), the primary stepped down to a secondary node as well. This had to do with the voting mechanism that MongoDb uses. See Markus' more detailed answer on this.. Since there are no more primaries in the replicaset, my cluster won't function anymore. Now, if i were to use an arbiter I could eliminate this problem as well.
When you have a 3-member replicataset, automatic failover kicks in. Whenever a node goes down, another primary is assigned automatically and the cluster will continue performing as before.
During my tests I also got to a point where one of my MongoD.exe instances stopped working due to a "Out of memory exception". I was running a cluster with 3 replicasets, meaning every machine had at least 4 mongod.exe processes running (3 for the replicaset shards and one for the configuration server replicaset). Besides having a query which wasn't optimized yet I also noticed that the WiredTiger storage engine by default can use up to 50% of ram minus one gigabyte. Perhaps it wasn't the best choise to have multiple replicaset-shards on one machine but I was able to eliminate the problem by capping the wiredtiger memory usage.
I hope this answer helps anybody who's starting to set up replication and sharding for MongoDb.
I've created a small MongoDB v2.6.4 cluster with one mongos router, three config servers, and handful of mongod data servers. It's working fine, I can do CRUD operations just fine from the mongos client, which directs to the appropriate shards.
I'm now wanting to experiment with the rest service.
When I tried this with a single MongoDB data server using mongod, it worked exactly as prescribed.
Even now, I can connect to each shard which is running with --rest and get a web response. But, the results are a little unexpected. I see the overview, clients, dbtop, write lock, log, etc. And if I click on listDatabase, I actually see my database listed. However, any REST attempts I try return with a JSON object showing offest 0, no rows, total of 0 roes, no query, and 0 milliseconds.
At the moment I'm of the opinion that this doesn't matter, as I've directly connected to a shard and would anticipate only seeing just the contents of that shard -- although further experimentation has been unable to produce even that result.
Again, I'll stress that in the current sharded configuration, mongos, NodeJS, and PHP all can see, query, and manipulate the data within my MongoDB collection just fine.
However, what I'm trying to do (if it's even possible) is connect to the mongos shard service via REST, since it does offer the --httpinterface option, figuring it will expose access to the shards and do the right thing.
It delivers the standard status page, but clicking on any links or hand rolling any REST results in a message about --rest not being enabled and that I should enable it.
REST is not enabled, use --rest to turn on.
check that port 28017 is secured for the network too.
Problem is, mongos doesn't allow that as an option! (mongod, however does)
It is entirely possible this is correct behavior, but I'm not sure how REST services would then work in a sharded configuration.
I'm not using a configuration file, specifying everything by the command line. That said, I tried passing a config file with RESTInterfaceEnabled: true in the yaml, and it didn't help.
mongos is returning a web interface, with the List all commands, db version, and log. It's missing the other diagnostic information (that's usually mongod specific), which I'd perhaps expect. But the links it offers doesn't work -- all requesting me to turn on an option that isn't there.
I've manually checked each shard data server (running mongod with --shardsvr), and directly connecting to them does provide REST access.
Everything else about the sharded cluster is working perfectly. I just can't to REST, like I can in a single-node unsharded solution and access my database or collections.
What might I have missed? Is this even possible? Any ideas?
AFAIK the REST Interface is inly to do monitoring/management not to access the documents. MongoDB does not provide official REST interface to do CRUD operations. For this you need either to create your own or find a library in the ecosystem to do this, look at http://docs.mongodb.org/ecosystem/tools/http-interfaces/
Do you have specific requirements/needs in mind?
PS : I do not use the --rest parameter in any of my deployment/development.
We are using MongoDB in production environment and now, due to some issues of current servers, I'm going to change the server and start a new MongoDB instance.
We have a replica set and a single mongod instance (two different MongoDB networks for different purposes). Now, first I should migrate the single mongod instance and then the whole replica set to the new server.
What I want to know is, how can I migrate both instances with no down-time? I don't want to shutdown the server or stop write operations.
Thanks in advance.
So first of all you should never run mongodb as a single instance for production. At a minimum you should have 1 primary, 1 secondary and 1 arbiter.
Second, even with a replica set you will always have a bit of write downtime when you switch primaries, as writes are not possible during the election process. From the docs:
IMPORTANT Elections are essential for independent operation of a
replica set; however, elections take time to complete. While an
election is in process, the replica set has no primary and cannot
accept writes. MongoDB avoids elections unless necessary.
Elections are going to occur when for example you bring down the primary to move it to a new server or virtual instance, or upgrade the database version (like going from 2.4 to 2.6).
You can keep downtime to a minimum with an existing replica set by setting the appropriate options to allow queries to run against secondaries. Again from the docs:
Maintaining availability during a failover. Use primaryPreferred if
you want an application to read from the primary under normal
circumstances, but to allow stale reads from secondaries in an
emergency. This provides a “read-only mode” for your application
during a failover.
This takes care of reads at least. Writes are best dealt with by having your application retry failed writes, or queue them up.
Regarding your standalone the documented procedures for converting to a replica set are well tested and can be completed very quickly with minimal downtime:
http://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/
You cannot have no downtime (a new mongod will run on new IP so you need to at least connect to it). But you can minimize downtime by making geographically distributed replica set.
Please Read
http://docs.mongodb.org/manual/tutorial/deploy-geographically-distributed-replica-set/
Use the given process but please note:
Do not set priority 0 of instances at New Location so that they become primary when old ones at Old Location step down.
You still need to restart mongod in replica set mode at Old Location.
You need 3 instances including an arbiter at New Location, if you want it to be
replica set.
When complete data is in sync with instances at New Location, step down instances at Old Location (one by one). Now everything will go to New Location but the problem is that it is directed through a distant mongod.
So stop mongod at Old Location and start a new one at new Location. Connect your applications to New Location Mongod.
Note: I have not done the same so far. I had planned it once but then I got the problem and it was not of hosting provider. Practically you may get some issues.
Replica Set is the feature provided by the Mongodb database to achieve high availability and automatic failover.
It is kinda traditional master-slave configuration but have capability of automatic failover.
It is basically group/cluster of the mongod instances which communicates, replicates to each other to provide high availability and to do automatic failover
Basically, in replica sets there are minimum 2 and maximum of 12 mongod instances can exist
In replica set following types of server exist. out of all, one server is always primary.
http://blog.ajduke.in/2013/05/31/setup-mongodb-replica-set-in-4-steps/
John answer is right, btw in your case you have no way to avoid downtime, you can just try to make it shorter as possible.
You can prepare the new replica set and save its configuration.
Same for the single mongod instance, prepare a js file with specific configuration (ie: stuff going on the admin database).
disable client connections on production servers.
copy the datafiles from the old servers to the new ones (http://docs.mongodb.org/manual/core/backups/#backup-with-file-copies)
apply your previous saved replica set config and configuration.
done
you can use diffent ways as add an hidden secondary member on the replica set if you have a lot of data, so you can wait it's is up-to-date before stopping the production server. Basically for the replica set you have many ways to handle a migration, with the single instance instead you don't have such features.
I am using 64-bit MongoDB, and i am undergoing test on multiple-shards. If i keep multiple shards in a single machine. Its working fine but if i keep shards in different machine, its failed in sharding to second shard. I have restricted the first-shard size to 10MB, once its reaches the limited size in first shard it should start sharding to second-shard but not happening so.Instead failed to store in second-shard updating to first shard. The following are my shard details. In my environment initially i have two shards. The first shard is on my first-machine running along with my application. The Second-shard is on my second machine.
Configuration as follows:-
*)On both of my shards, shard-server,configserver,mongos and i have connected mongo through mongos as follows ./mongo hostname:27017/admin and i have added both the shards in first & second shard and enabled sharding for database and collection level by using shard-key.
Please, let me know if i gone wrong anywhere in the configuration.
Advance Thanks,
Your post could use some editing, this is very difficult to read.
It looks like you have 2 machines. On each machine you have:
mongod process serving as one shard
mongod process serving as a config
mongos process
a copy of your application connecting to localhost:27017/admin
Please, let me know if i gone wrong anywhere in the configuration.
There are several possible problems here. Please check the following:
You can only have 1 or 3 config processes. It looks like you have 2, this will not work.
When you connect to localhost:27017/admin are you connecting to mongos or mongod? Either one could be running on those ports. Can you specify the ports for each process to help clarify? You must connect to mongos or the sharding will not happen.
Please look at the logs, they generally have output indicating what the server is doing. If there is no indication of "splits" or "chunks" happening, then your database may be configured incorrectly.
Your best bet is to start from top and test each piece one at a time.