We have some servers that serve a specific amount of data (5TB).
The problem is that we have lack in space o the servers and we want to add another server.
The question is, what'll happend when I'll add the new shard while all the other servers already running?
Will MongoDB be possible to transfer the data to the new server?
You were commented on right away and the response is correct. What will simply happen is that the balancer will periodically attempt to re-allocate chunks across the available shards, including the new one.
The process is automatic after the shard is added so yes is is possible to transfer new data to the server.
Perhaps the main consideration (as pointed to in the document link) is that the balancer is applied on one instance of mongos you have running and of how the balancer behaves. A lot will depend on your configutation, but for more information read the following document in addition:
http://docs.mongodb.org/manual/core/sharding-balancing/#sharding-balancing-internals
Related
What is the easiest way to move Google Cloud SQL instance(Postgres 9.6) from one google project to another with minimum or zero downtime? Instance size is about 20 GB
There is a service called "Migration job" which looks very relevant https://cloud.google.com/database-migration/docs/postgres/create-migration-job . But I cannot understand whether it can be used to move instance from one google project to another.
Simple restoring from backup is not really my case because I want to achieve minimum possible downtime, so I'm looking for something like 2 running instances with the synced real-time data.
PS. I also have configured VM with pgbouncer
Yes, Database Migration Service could be used to move one Cloud SQL instance from one GCP project to another. This is a cheaper way than the next approach, and although it requires more setup, it should be faster too. A connection profile can be created for the existing Cloud SQL instance, and a Cloud SQL target must be created in the destination project, but once everything is set up, most of the migration will be automatic. This is a well documented procedure, of which you can find information in our documentation.
Developers sometimes want to migrate their (normal) relational database with “zero” downtime. While downtime can be reduced, migration cannot be done without any impact on applications (that is, zero downtime). Replication causes replication lag.
The instant the decision is made to “migrate” all applications from one replica to another, applications (and therefore developers) have to wait (that is, downtime) for at least as long as the “replication lag” before using the new database. In practice, the downtime is a few orders of magnitude higher (minutes to hours) because:
Database queries can take multiple seconds to complete, and in flight queries must be completed or aborted at the time of migration.
The database has to be “warmed up” if it has substantial buffer memory - common in large databases.
If database shards have duplicate tables, some writes may need to be paused while the shards are being migrated.
Applications must be stopped at source and restarted in GCP, and connection to the GCP database instance must be established.
Network routes to the applications must be rerouted. Based on how DNS entries are set up, this can take some time.
All of these can be reduced with some planning and “cost” (some operations not permitted for some time before/after migration).
Decreasing the load on the source DB until the migration completes might help, and downtime might be less disruptive.
Other considerations:
Increase the machine types to increase network throughput.
Increase SSD size for higher IOPS/MBps.
More about.
The most intuitive way would be to export the data from the Cloud SQL instance to a GCS bucket, and import it to a new instance in the new project. This would imply some downtime, and you would have to manually create the instance in the target project with the same configuration as the original; it does require some manual steps, but it would be a simple and verifiable way to copy the data across an instance in a different project.
I'm reading "Seven Databases in Seven Weeks". Could you please explain me the text below:
One downside of a distributed system can be the lack of a single
coherent filesystem. Say you operate a website where users can upload
images of themselves. If you run several web servers on several
different nodes, you must manually replicate the uploaded image to
each web server’s disk or create some alternative central system.
Mongo handles this scenario by its own distributed filesystem called
GridFS.
Why do you need replicate manually uploaded images? Does they mean some of the servers will have linux and some of them Windows?
Do all replicated data storages tend to implement own filesystem?
On the need for data distribution and its intricacies
Let us dissect the example in a bit more detail. Say you have a web application where people can upload images. You fire up your server, save the images to the local machine in /home/server/app/uploads, the users use the application. So far, so good.
Now, your application becomes the next big thing, you have tens of thousands of concurrent users and your single server simply can not handle that load any more. Luckily, aside from the fact that you store the images in the local file system, you implemented the application in a way that you could easily put up another instance and distribute the load between them. But now here comes the problem: the second instance of your application would not have access to the images stored on the first instance – bad thing.
There are various ways to overcome that. Let us take NFS as an example. Now your second instance can access the images, and even store new ones, but that puts all the images on one machine, which sooner or later will run out of disk space.
Scaling storage capacity can easily become a very expensive part of an application. And this is where GridFS comes to help. It uses the rather easy means of MongoDB to distribute data across many machines, a process which is called sharding. Basically, it works like this: Instead of accessing the local filesystem, you access GridFS (and the contained files within) via the MongoDB database driver.
As for the OS: Usually, I would avoid mixing different OSes within a deployment, if at all possible. Nowadays, there is little to no reason for the average project to do so. I assume you are referring to the "different nodes" part of that text. This only refers to the fact that you have multiple machines involved. But they perfectly can run the same OS.
Sharding vs. replication
Note The following is vastly simplified, because going into details would well exceed the scope of one or more books.
The excerpt you quoted mixes two concepts a bit and is not clear enough on how GridFS works.
Lets first make the two involved concepts a bit more clear.
Replication is roughly comparable to a RAID1: The data is stored on two or more machines, and each machine holds all data.
Sharding (also known as "data partitioning") is roughly comparable to a RAID0: Each machine only holds a subset of the data, albeit you can access the whole data set (files in this case) transparently and the distributed storage system takes care of finding the data you requested (and decides where to store the data when you save a file)
Now, MongoDB allows you to have a mixed form, roughly comparable to RAID10: The data is distributed ("partitioned" or "sharded") between two or more shards, but each shard may (and almost always should) consist of a replica set, which is an uneven number of MongoDB instances which all hold the same data. This mixed form is called a "sharded cluster with a replication factor of X", where X denotes the non-hidden members per replica set.
The advantage of a sharded cluster is that there is no single point of failure any more:
Depending on your replication factor, one or more replica set members can fail, and the cluster is still working
There are servers which hold the metadata (which part of the data is stored on which shard, for example). Those are called config servers. As of MongoDB version 3.0.x (iirc), they form a replica set themselves – not much of a problem if a node fails.
You access a sharded cluster via a the mongos sharded cluster query router of which you usually have one per instance of your application (and most often on the same server as your application instance). But: most drivers can be given multiple mongos instances to connect to. So if one of those mongos instances fails, the driver will happily use the next one you configured.
Another advantage is that in case you need to add additional storage or have more IOPS than your current system can handle, you can add another shard: MongoDB will take care of distributing the existing data between the old shards and the new shard automagically. The details on how this is done are covered in the introduction to Sharding in the MongoDB docs.
The third advantage – and the one that has the most impact, imho – is that you can distribute (and replicate) data on relatively cheap commodity hardware, whereas most other technologies offering the benefits of GridFS on a sharded cluster require you to have specialized and expensive hardware.
A disadvantage is of course that this setup only is feasible if you have a lot of data, since many machines are necessary to set up a sharded cluster:
At least 3 config servers
At least a single shard, which should consist of a replica set. The minimal setup would be two data bearing nodes plus an arbiter
But: in order to use GridFS in general, you do not even need a replica set ;).
To stay within our above example: Both instances of your application could well access the same MongoDB instance holding a GridFS.
Do all replicated data storages tend to implement own filesystem?
Replicated? Not necessarily. There is DRBD for example, which could be described as "RAID1 over ethernet".
Assuming we have the same mixup of concepts here as we had above: Distributed file systems by their very definition implement a file system.
In this case,IMHO, author was stating that each web server has own disk storage, not shared with others - having that - upload path could be /home/server/app/uploads and as it is part of server filesystem is not shared at all as a kind of security with service provider. To populate those we need to have a script/job which will sync data to other places behind the scenes.
This scenario could be a case to use GridFS with mongo.
How gridFS works:
GridFS divides the file into parts, or chunks 1, and stores each
chunk as a separate document. By default, GridFS uses a chunk size of
255 kB; that is, GridFS divides a file into chunks of 255 kB with the
exception of the last chunk. The last chunk is only as large as
necessary. Similarly, files that are no larger than the chunk size
only have a final chunk, using only as much space as needed plus some
additional metadata.
In reply to comment:
BSON is binary format, and mongo has special replication mechanism for replicating collection data (gridFS is a special set of 2 collections). It uses OpLog to send diffs toother servers in replica set. More here
Any comments welcome!
For my application I need to move old data periodically from a mongodb server to another one (ie, two distinct servers). I also want to be able to query those data as if they were the same database.
In short terms, I want to be able to see two mongodb instances (on two different servers) as one and be able to control when and where the data is stored.
I read about the concept of sharding and chunks and rapidly saw the moveChunk function which can easily do what I want.
The problem is that it seems to be impossible to configure such architecture in mongodb. Am I missing something here?
Archiving Deleted Documents
For the problem of keeping the deleted documents, you have no option to achieve this with build-in features/mechanisms like sharding or replication. The only way to do it is to handle that case manually, e.g. holding a separate collection for deleted documents, and simply move documents to that collections instead of deleting them.
For your global problem of moving data you have the following two options:
Sharding
Using sharding you will split your data into pieces which will be stored on two (in your case) different servers. In this scenario you can use the moveChunk method as you have mentioned. But this method is very tricky, as for that you will need to disable the built-in automatic balancer to have a full manual control over your chunks. Anyway, this is not recommended by the MongoDB:
Only use the moveChunk in special circumstances such as preparing your sharded cluster for an initial ingestion of data, or a large bulk import operation. In most cases allow the balancer to create and balance chunks in sharded clusters
Besides this will only allow to split data, and finally, to get to your goal, you will end up with one full and one empty shard.
Replication
The replication approach is much more safe and easy to achieve. You can simply configure a replica set and add your second server to that set.
If the data is too big, you can configure your second server as hidden. So that no reads will be performed towards that server, so no inconsistent data will be received. After the data replication is finished, you will have the copy of your data on both servers.
As for using both servers as a single server, if you need to balance the requests between these tow, you can configure your readPreference to secondary, which will assure that all the reads are being sent to the secondary server, and writes by default are done on the primary.
In this case your code will be unaware of what server you are querying. You will just run your client methods, and the rest will be done behind the drivers.
Conclusion
So my advice would be to use the replication approach as more clean, pain-less and safe solution.
I would like to know if it is at all possible to have mongodb fail overs only using a single address. I know replica sets are typically used for this while relying on the driver to make the switch over, but I was hoping there may be a solution out there that would allow one address or hostname to automatically change over when the mongodb instance was recognized as being down.
Any such luck? I know there are solutions for MySQL, but I haven't had much luck with finding something for MongoDB.
Thanks!
Yes it is possible, the driver holds a cache map of your replica set which it will query for a new primary when the set suffers an election. This map is refreshed once every so often however, if your application restarts (process is quit or something, or each request of PHP fork mode) then the driver has no choice but to refresh its map. At this point you will suffer connectivity problems.
Of course the best thing to do is to add a seedlist.
Using a single IP defies the redundancy that is in-built into MongoDB.
Now I have two shards:shard3(16g),shard4(15g) and three machines:
the deploy like this:
10.10.10.5:(mongoS,configureserver,shard3 primary,shard4 primary)
10.10.10.6:(mongoS,configureserver,shard3 secondary,shard4 secondary)
10.10.10.7:(mongoS,configureserver,shard3 arbitor,shard4 arbitor)
now I want to make a performance testing(about adding new shards),I know I can't use the production environment to test since that will impact the production performance,So I want to copy all of the data to my testing three machines:20.20.20.5,20.20.20.6 and 20.20.20.7,I read the manual book but can't find a better way,So dear guys could you please give me an advice.
by the way,ask two little questions:First:like my production environment,how do I change the arbitor node to secondary,i.e I want change 10.10.10.7 to secondary shards,because I wanna 10.7 share the read pressure with 10.6.
Second:how do I indicate the Mongos read primary node but secondary node,you know the mongoS writes on primary while read on secondary,but I wanna both read and write on primary node for getting the newest data immediately.
Thanks inadvance
Jack
You should look at the following documentation: http://www.mongodb.org/display/DOCS/Import+Export+Tools. You can likely use mongoexport and mongoimport for what you want to do. Or you can also use mongodump & mongorestore. This will allow you to backup and restore your data onto the testing environment.
First question: You Can't "convert" an arbiter into a secondary. The only real way to do this would be to add a new node to the replica-set, and then take down the arbiter and later remove it from the replica-set. You can add a new mongod to the replica-set using rs.add() on an existing replica set node. You do it in this order to avoid downtime. If you don't mind downtime, then order doesn't particularly matter. Documentation on adding a node to a replica set can be found here: http://www.mongodb.org/display/DOCS/Replica+Sets+-+Basics
Also, if you are doing readScaling and using SlaveOK() it's important you keep in mind that if there is replication lag from the primary to either of the secondaries, there is the potential for you to read stale data. If this is acceptable for your application than its fine, but it's important for you to realize that if you simultaneously query two nodes in a replica-set, you may read in two different values for the same query based on replication lag.
Second Question: If you want to always only read/write from the primary than you should not run with SlaveOK. SlaveOK off is the default, but if you already turned it on just call it again passing in false.
As long as you have a primary up and a majority of your replica set, the replica set will be writable. So naturally, as long as the primary is not overloaded and you have not taken down 2 out of 4 of your nodes, you will be able to write to the primary. It just will only replicate to the majority of the nodes that are up.
In general, we try to discourage even numbered replica-sets so I would take the arbiter offline and stick to your 3 replica sets. This is because you really don't win anything from having the arbiter. If 2 nodes fail either way you do not have a majority and the replica set will become read only. If one node goes down you will still be up for writes. The arbiter doesn't help anything in an even number set.
When you tried to perform these queries that went to only one secondary, was the new secondary all caught up datawise? If it was in "Recovery" state, then it's possible that the reads would not have gone to it yet since it had not replicated all of the data. Aside from that, there are ways to specify read preferences.
Documentation on all of the different read preferences and how to use them can be found here: http://docs.mongodb.org/manual/applications/replication/#read-preference