Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 7 years ago.
Improve this question
I have some questions about Backup with Filesystem Snapshots.
I just wonder that only taking the snapshot is equal to backup? It's because snapshot is the pointer to the disk block in the mongo manner ( copy on write )
If the original disc is broken, it can't be restored.
So the question is that if we want to need to backup, is only the filesystem snapshot enough or we need to implement additionally ?
A filesystem snapshot itself is surely not enough to backup your data. When using a filesystem snapshot, this does not help you with corrupted disks, etc. So you need to move the data in the snapshot to somewhere else.
When you create a snapshot, the changes applied to the original data files will fill up the remaining space, in case of LVM snapshot even just the space explicitly allocated. So you need to over provision your disks by at least the amount the data you write within the time you need to process the backup and store it on a remote location.
Note In the following description, most security considerations are set aside for the sake of brevity. You must apply the security measures appropriate for your data, for example using stunnel for the encryption of the data transfer when using the first approach.
One approach
Here is what I tend to do to keep the need for over provisioning your disk size, CPU and RAM as low as possible while still being able to create snapshots and use them properly. Note that this is sort of a poor mans solution (when being really tight on budget) and you really should carefully think about wether this solution suits you.
I usually have a cheapo VM as a backup system, with some big, cheap, slow(ish) storage attached. On that VM, I open a listening netcat, piping it's output through some sort of compressor. I use my own snap for that, which is a crude implementation utilizing the snappy compression algorithm. The reason is that snappy is optimized for speed rather than size, and that is what I am interested in here to reduce the time needed for getting the snapshot from the MongoDB server.
nc -l <internalIp> <somePort> | snap > /mnt/datastore/backup-$(date +%Y-%m-%dT%T%Z).tar.sz
Make sure you only listen to an internal IP or one you have limited access to via a firewall.
Next, I mount my snapshot (how to do that varies) on the MongoDB server I took the backup at. Then I jump to the directory I mounted said snapshot into (the one containing the data folder) and run tar over it, piping the output to another netcat, which sends the tarred data to the netcat listening on the backup server
tar -cv <directory> | nc <internalIpOfBackupServer> <listeningPort>
Another approach
This is basically the same, though it utilizes ssh instead of nc:
tar -cv <directory> | ssh user#backupserver "snap > /mnt/datastore/backup-$(date +%Y-%m-%dT%T%Z).tar.sz"
We are trading speed for security here. This approach will take considerably longer and needs more resources on the machine you take the backup from.
Some thoughts
You really want the snapshot be moved and archived on another server as fast as possible so that you can destroy the snapshot and return to normal operations. This way, you can reduce the necessary over provisioning of your resources to a bare minimum. Since RAM is a very precious resource on any MongoDB deployment, you want the compression to be done on a different machine.
If space is an issue on your backup machine, you can either use gzip or bzip2 instead of snap right away (reducing the speed with which the backup is done), or you can do the following on the backup machine after the backup is finished:
snap -c -u yourBackup.tar.sz | bzip2 -v9 > yourBackup.tar.bz2
rm yourBackup.tar.sz
Related
I’m trying to set up daily backups (using Persistent Disk snapshots) for a PostgreSQL instance I’m running on Google Compute Engine and whose data directory lives on a Persistent Disk.
Now, according to the Persistent Disk Backups blog post, I should:
stop my application (PostgreSQL)
fsfreeze my file system to prevent further modifications and flush pending blocks to disk
take a Persistent Disk snapshot
unfreeze my filesystem
start my application (PostgreSQL)
This obviously brings with it some downtime (each of the steps took from seconds to minutes in my tests) that I’d like to avoid or at least minimize.
The steps of the blog post are labeled as necessary to ensure the snapshot is consistent (I’m assuming on the filesystem level), but I’m not interested in a clean filesystem, I’m interested in being able to restore all the data that’s in my PostgreSQL instance from such a snapshot.
PostgreSQL uses fsync when committing, so all data which PostgreSQL acknowledges as committed has made its way to the disk already (fsync goes to the disk).
For the purpose of this discussion, I think it makes sense to compare a Persistent Disk snapshot without stopping PostgreSQL and without using fsfreeze with a filesystem on a disk that has just experienced an unexpected power outage.
After reading https://wiki.postgresql.org/wiki/Corruption and http://www.postgresql.org/docs/current/static/wal-reliability.html, my understanding is that all committed data should survive an unexpected power outage.
My questions are:
Is my comparison with an unexpected power outage accurate or am I missing anything?
Can I take snapshots without stopping PostgreSQL and without using fsfreeze or am I missing some side-effect?
If the answer to the above is that I shouldn’t just take a snapshot, would it be idiomatic to create another Persistent Disk, periodically use pg_dumpall(1) to dump the entire database and then snapshot that other Persistent Disk?
1) Yes, though it should be even safer to take a snapshot. The fsfreeze stuff is really to be 100% safe (anecdotally: I never use fsfreeze on my PDs and have not run into issues)
2) Yes, but there is no 100% guarantee that it will always work (paranoid solution: take a snapshot, spin up a temp VM with that snapshot, check the disk is ok, and delete the VM. This can be automated)
3) No, I would not recommend this over snapshots. It will take a lot more time, might degrade your DB performance, and what happens if something happens in the middle of a dump? Also, PDs are very expensive for incremental backups. Snapshots are diffed, so you don't have to pay for the whole disk every copy (just the first one), only the changes.
Possible recommendation:
Do #3, but then create a snapshot of the new PD and then delete the PD.
https://cloud.google.com/compute/docs/disks/persistent-disks#creating_snapshots has recently been updated and now includes this new paragraph:
If you skip this step, only data which was successfully flushed to disk by the application will be included in the snapshot. The application experiences this scenario as if it was a sudden power outage.
So the answers to my original questions are:
Yes
Yes
N/A, since the answer to ② is Yes.
I am working on an application in which there is a pretty dramatic difference in usage patterns between "hot" data and other data. We have selected MongoDB as our data repository, and in most ways it seems to be a fantastic match for the kind of application we're building.
Here's the problem. There will be a central document repository, which must be searched and accessed fairly often: it's size is about 2 GB now, and will grow to 4GB in the next couple years. To increase performance, we will be placing that DB on a server-class mirrored SSD array, and given the total size of the data, don't imagine that memory will become a problem.
The system will also be keeping record versions, audit trail, customer interactions, notification records, and the like. that will be referenced only rarely, and which could grow quite large in size. We would like to place this on more traditional spinning disks, as it would be accessed rarely (we're guessing that a typical record might be accessed four or five times per year, and will be needed only to satisfy research and customer service inquiries), and could grow quite large, as well.
I haven't found any reference material that indicates whether MongoDB would allow us to place different databases on different disks (were're running mongod under Windows, but that doesn't have to be the case when we go into production.
Sorry about all the detail here, but these are primary factors we have to think about as we plan for deployment. Given Mongo's proclivity to grab all available memory, and that it'll be running on a machine that maxes out at 24GB memory, we're trying to work out the best production configuration for our database(s).
So here are what our options seem to be:
Single instance of Mongo with multiple databases This seems to have the advantage of simplicity, but I still haven't found any definitive answer on how to split databases to different physical drives on the machine.
Two instances of Mongo, one for the "hot" data, and the other for the archival stuff. I'm not sure how well Mongo will handle two instances of mongod contending for resources, but we were thinking that, since the 32-bit version of the server is limited to 2GB of memory, we could use that for the archival stuff without having it overwhelm the resources of the machine. For the "hot" data, we could then easily configure a 64-bit instance of the database engine to use an SSD array, and given the relatively small size of our data, the whole DB and indexes could be directly memory mapped without page faults.
Two instances of Mongo in two separate virtual machines Would could use VMWare, or something similar, to create two Linux machines which could host Mongo separately. While it might up the administrative burden a bit, this seems to me to provide the most fine-grained control of system resource usage, while still leaving the Windows Server host enough memory to run IIS and it's own processes.
But all this is speculation, as none of us have ever done significant MongoDB deployments before, so we don't have a great experience base to draw upon.
My actual question is whether there are options to have two databases in the same mongod server instance utilize entirely separate drives. But any insight into the advantages and drawbacks of our three identified deployment options would be welcome as well.
That's actually a pretty easy thing to do when using Linux:
Activate the directoryPerDB config option
Create the databases you need.
Shut down the instance.
Copy over the data from the individual database directories to the different block devices (disks, RAID arrays, Logical volumes, iSCSI targets and alike).
Mount the respective block devices to their according positions beyond the dbpath directory (don't forget to add the according lines to /etc/fstab!)
Restart mongod.
Edit: As a side note, I would like to add that you should not use Windows as OS for a production MongoDB. The available filesystems NTFS and ReFS perform horribly when compared to ext4 or XFS (the latter being the suggested filesystem for production, see the MongoDB production notes for details ). For this reason alone, I would suggest Linux. Another reason is the RAM used by rather unnecessary subsystems of Windows, like the GUI.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 9 years ago.
Improve this question
I've been playing around with Amazon's Web Services for over a year now, however I don't quite understand how it works. When I for example select an AMI of my choice from the EC2 console and I continue through the wizard, I reach the "Storage Configuration Tab". There are several options here.
There is the root volume tab and then there is the EBS volume tab. How do both of these differ? What is the maximum size I can allocate for each? How can I configure the EBS Volumes to work with my Instance? Say for example I decide to create 8 EBS volumes each with 25 GB of storage....now for something like a Postgresql database which naturally lives on the root device, how I configure it so the database is stored across all 8 EBS volumes? In a sense, the 8 EBS volumes becoming one 200 GB drive and the postgres database data stored across that whole drive.
Any form of clarification will be appreciated.
You should read the benefits of EBS vs instance store. I also wrote a bit about the PostgreSQL angle of this on my work blog recently. See also what root device to use for a new EC2 instance and the other questions listed in the Related sidebar.
Instance store will eventually EAT YOUR DATA unless you carefully set up replication and regular backups. If an instance fails or is terminated you cannot get your data back if it's on an instance store. You need good backups anyway, it's just more important with instance store and you need to be more careful about having near-real-time replication set up.
On the other hand, EBS is more likely to be affected by outages and faults that render it unavailable for a time; your data may still exist, but if you can't get to it for a couple of hours you can't fail over until the fault is fixed. So you really need good backups and replication anyway.
Quick answer, I'll leave the detailed explanations to the post:
The root volume is either EBS or instance store, depending on AMI type.
In the volumes tab you can add additional volumes. You can choose whether these are EBS or instance store volumes at volume creation time, irrespective of the AMI type. Different instance sizes have different limits on number and size of instance store volumes, but all have the same limits on EBS volumes.
The maximum size of an instance store volume is defined by the instance type. See the documentation for your instance. The maximum size of an EBS volume is in the first paragraph of the EBS documentation:
Amazon EBS volumes are created in a particular Availability Zone and can be from 1 GB to 1 TB in size.
The PostgreSQL database doesn't "naturally live on the root volume" really. It lives where you put it. If you're using a package-manager installed version it'll usually be put in /var/lib/pgsql or /var/lib/postgres, but you can either change the startup script options to move it elsewhere, replace that with a symlink to the desired location, or mount a new volume at that point. There are ample discusions of how to move PostgreSQL on Stack Overflow, dba.stackexchange.com and serverfault so I won't repeat all that here.
To combine multiple EBS volumes use Linux's software RAID (md). EBS is just like any other disk as far as Linux is concerned, so see the usual documentation for setting up Linux software RAID.
Personally I've been quite unimpressed with the performance of EC2, at least with PostgreSQL. You can get a very fast database running, but only at a pretty crushing price. It's very convenient if you want to fire up some big databases for a short term job, but it isn't economic as a long lived hosting option, you're better off looking at VPS providers that offer better I/O performance. Search ServerFault, dba.stackexchange.com, etc.
Finally, a reminder: Instance store on high I/O instances seems to be faster than the other options ... but if you have to shut down or reboot your instance or the instance fails you will lose all data on your instance store volumes, so you must have good backups and real-time replication if you're going to use the instance store.
The shorter answer is:
For quick and dirty, you can just have instance store on all your EC2 instances, and do backups to S3. The advantage of EBS over instance store is that when you kill that server, the EBS will stay and can be reused, and an instance store won't.
200Gb is small space, you can just get one storage device (instance store) for it, and then backup to S3 or replicate the whole 200Gb thing. Chances are, you won't be using RAID or haddrive replication to improve your database's reliability/availability.
tl;dr
Use instance store unless you need the volumes to be transferable between servers.
I am developing a web system to handle a very large set of small images, about 100 millions images of 50kb ~ 200kb, working on ReiserFS.
For now, it is very difficult to backup and sync those large number of small files.
My question is that if it a good idea to store these small images to a key/value store or other nosql database such as GridFS (Mongodb), Tokyo Tyrant, Voldemort to gain more performance and bring better backup support?
First off, have a look at this: Storing a millon images in the filesystem. While it isn't about backups, it is a worthwile discussion of the topic at hand.
And yes, large numbers of small files are pesky; They take up inodes, require space for filenames &c. (And it takes time to do backup of all this meta-data). Basically it sounds like you got the serving of the files figured out; if you run it on nginx, with a varnish in front or such, you can hardly make it any faster. Adding a database under that will only make things more complicated; also when it comes to backing up. Alas, I would suggest working harder on a in-place FS backup strategy.
First off, have you tried rsync with the -az-switches (archive and compression, respectively)? They tend to be highly effective, as it doesn't transfer the same files again and again.
Alternately, my suggestion would be to tar + gz into a number of files. In pseudo-code (and assuming you got them in different sub-folders):
foreach prefix (`ls -1`):
tar -c $prefix | gzip -c -9 | ssh -z destination.example.tld "cat > backup_`date --iso`_$prefix.tar.gz"
end
This will create a number of .tar.gz-files that are easily transferred without too much overhead.
Another alternative is to store the images in SVN and actually have the image folder on the web server be an svn sandbox of the images. That simplifies backup, but will have zero net effect on performance.
Of course, make sure you configure your web server to not serve the .svn files.
If all your images, or at least the ones most accessed, fit into memory, then mongodb GridFS might outperform the raw file system. You have to experiment to find out.
Of course, depending on your file-system, breaking up the images into folders or not would affect images. In the past I noticed that ReiserFS is better for storing large numbers of files in a single directory. However, I don't know if thats still the best file system for the job.
I have a perl script which monitors several windows network share drive usages. It currently monitors the free space of several network drives using the cygwin df command. I'd like to add the individual drive usages as well. When I use the
du -s | grep total
command, it takes for ever. I need to look at the shared drive usages because there are several network drives that are shared from a single drive on the server. Thus, filling one network drive fills them all (yes I know, not the best solution, not my choice).
So, I'd like to know if there is a quicker way to get the folder usage that doesn't take for ever.
du -s works by recursively querying the size of every directory and file. If your filesystem implementation doesn't store this total value somewhere, this is the only way to determine disk usage. Therefore, you should investigate which filesystem and drivers you are using, and see if there is a way to directly query for this data. Otherwise, you're probably SOL and will have to suck up the time it takes to run du.
1) The problem possibly lies in the fact that they are network drives - local du is acceptably fast in most cases. Are you doing du on the exact server where the disk is housed? If not, try to approach the problem from a different angle - run an agent on every server hosting the drives which calculates the local du summaries and then report the totals to a central process (either IPC or heck, by writing a report into a file on that same share filesystem).
2) If one of the drives is taking a significantly larger share of space (on average) than the rest of them, you can optimize by doing du on all but the "biggest" one and then calculate the biggest one by subtracting the sum of others from df results
3) Also, to be perfectly honest, it sounds like a suboptimal solution from design standpoint - while you indicated that it's not your choice, I'd strongly recommend that you post a question on how you can improve the design within the parameters you were given (to ServerFault website, not SO)