I am working on AWS ec2 instance. I have configured MongoDb on it.
1. I have 1TB storage space for mongo data to be store.
2. Other 50 GB for application to run.
Since cost of taking snapshot of everything is huge, Can I take snapshot of only folder where my mongodb data is stored.
e.g my folder for mongodb storage is /home/ubuntu/mongodb
So I want snap shot or LVM of only mongodb folder instead of taking it for 1 TB instance on AWS.
you can take dumps of your database.
mongodump --dbpath /data/db/ --out /data/backup/
or
mongodump --host mongodb.example.net --port 27017
and then store it on s3. You can also run cron job for this to take backup of your data at desired time frequency.
Related
I have a MongoDB Atlas cloud database,
and I want to "sync" to a local server instance with mongod server running.
I have written an automated backup script that backs up a website, which then also does a mongodump to create an archive file from the (local) MongoDB, which then all gets dumped to an AWS bucket.
It's been working great, but I just realized that it's getting the local disk's mongo data, and not the "live" data on the Mongo Atlas cloud.
Is there a way mongodump can dump the MongoDB Atlas stuff to the local disk?
I hope there is an easier way than to "find" all on individual Atlas collections in my database, and "update" to the local disk.
I was successfully able to get a dump from Atlas to my local server using mongodump.
mongodump --forceTableScan --url="mongodb+srv://<username>:<password>#yourmongoserver.something.mongodb.net/<database name>"
Note this failed until I included the --forceTableScan, which then after was seemingly successful.
I'm a SQL Server DBA trying to use MongoDB for some particular cases. What I have at the moment is 3 node replica set with 2 data bearing nodes and 1 arbiter. The thing that i'm struggling at the moment is finding a clear answer on ho to create Backups that will allow me a point in time restores, similar to what you have in MS SQL with FULL and LOG backups. How can i do that?
Mongodb provides different methods to Backup and restore
1. Back Up with Atlas(cloud based AWS services)
2. Back Up with MongoDB Cloud Manager or Ops Manager(Enterprise edition only. Supports backing up and restoring MongoDB replica sets and sharded clusters from a graphical user interface.)
3. Back Up with file system snapshot on OS( on Linux, the Logical Volume Manager (LVM) can create snapshots. Similarly, Amazon’s EBS storage system for EC2 supports snapshots)
To get a correct snapshot of a running mongod process, you must have journaling enabled. Without journaling enabled, there is no guarantee that the snapshot will be consistent or valid.
To create a snapshot with LVM, issue a command as root in the following format:
lvcreate --size 100M --snapshot --name mdb-snap01 /dev/vg0/mongodb
This command creates an LVM snapshot (with the --snapshot option) named mdb-snap01 of the mongodb volume in the vg0 volume group.
This example creates a snapshot named mdb-snap01 located at /dev/vg0/mdb-snap01. The location and paths to your systems volume groups and devices may vary slightly depending on your operating system’s LVM configuration.
To restore a snapshot , issue the following sequence of commands:
umount /dev/vg0/mdb-snap01
lvcreate --size 1G --name mdb-new vg0
dd if=/dev/vg0/mdb-snap01 of=/dev/vg0/mdb-new
mount /dev/vg0/mdb-new /srv/mongodb
for more details https://docs.mongodb.com/manual/tutorial/backup-with-filesystem-snapshots/#back-up-and-restore-using-lvm-on-linux
4.Back Up with mongodump (Terminal based Mongodb tools)
mongodump and mongorestore are simple and efficient tools for backing up and restoring small MongoDB deployments
mongodump and mongorestore operate against a running mongod process
If you dont specify any database it captures all the databases and copies into seperate folder along with indexes(json format) for every database
By default, mongodump does not backup the local database(which contains Replicaset configuration & oplog.rs collection).
For replica sets, mongodump provides the --oplog option to include in its output oplog entries that occur during the mongodump operation. This allows the corresponding mongorestore operation to replay the captured oplog. To restore a backup created with --oplog, use mongorestore with the --oplogReplay option.
Mongorestore captures only database files. indexes must be rebuild after restoring the data.
https://docs.mongodb.com/manual/tutorial/backup-and-restore-tools/#
commands:
mongodump --out /data/backup/ (It backups all the databases and indexes)
mongodump --collection myCollection --db test (specified database & collection)
mongorestore --port
we operate for our customer a server with a single mongo instance, gradle, postgres and nginx running on it. The problem is we had massiv performance problmes until mongodump is running. The mongo queue is growing and no data be queried. The next problem is the costumer want not invest in a replica-set or a software update (mongod 3.x).
Has somebody any idea how i clould improve the performance.
command to create the dump:
mongodump -u ${MONGO_USER} -p ${MONGO_PASSWORD} -o ${MONGO_DUMP_DIR} -d ${MONGO_DATABASE} --authenticationDatabase ${MONGO_DATABASE} > /backup/logs/mongobackup.log
tar cjf ${ZIPPED_FILENAME} ${MONGO_DUMP_DIR}
System:
6 Cores
36 GB RAM
1TB SATA HDD
+ 2TB (backup NAS)
MongoDB 2.6.7
Thanks
Best regards,
Markus
As you have heavy load, adding a replica set is a good solution, as backup could be taken on secondary node, but be aware that replica need at least three servers (you can have an master/slave/arbiter - where the last need a little amount of resources)
MongoDump makes general query lock which will have an impact if there is a lot of writes in dumped database.
Hint: try to make backup when there is light load on system.
Try with volume snapshots. Check with your cloud provider what are the options available to take snapshots. It is super fast and cheaper if you compare actual pricing used in taking a backup(RAM and CPU used and if HDD then transactions const(even if it is little)).
full disclosure: I am a complete n00b to mongodb and am just getting my feet wet with using mongo on AWS (but have 2 decades working in IT so not a total n00b :P)
I setup an EBS volume and installed mongo on a EC2 instance.
My problem is that I provisioned too small an EBS volume initially.
When I realized this I:
created a new larger EBS volume
mounted it on the server
stopped mongo ( $ sudo service mongod stop)
copied all my /data/db files into the new volume
updated conf files and fstab (dbpath, logpath, pidfilepath and mount point for new volume respectively)
restarted mongod
When I execute: $ sudo service mongod start
- everything runs fine.
- I can futz about in the admin and local databases.
However, when I run the mongos command: > show databases
- I only see the admin and local.
- the database I copied into the new volume (named encompass) is not listed.
I still have a working local copy of the database so my data is not lost, just not sure how best to move mongo data around other than:
A) start all over importing the data to the db on the AWS server (not what I would like since it is already loaded in my local db)
B) copy the local db to the new EBS volume again (also not preferred but better that importing all the data from scratch again!).
NOTE: originally I secure copied the data into the EBS volume with this command:
$ scp -r -i / / ec2-user#:/
then when I copied between volumes I used a vanilla cp command.
Did I miss something here?
The best I could find on SO and the web was this process (How to scale MongoDB?), but perhaps I missed a switch in a command or a nuance to the process that rendered my database files inert/useless?
Any idea how I can get mongo to see my other database files and collections?
Or did I make a irreversible error somewhere along the way?
Thanks for any help!!
Are you sure you conf file is being loaded? You can, for a test, load mongod.exe and specify the path directly to your db for a test, i.e.:
mongod --dbpath c:\mongo\data\db (unix syntax may vary a bit, this is windows)
run this from the command line and see what, if anything, mongo complains about.
A database has a very finicky algorithm that is easy to damage. Before copying from one database to another you should probably seed the database, a few dummy entries will tell you the database is working.
I use the below to create a new database in mongo
sudo mkdir -p /data/db2/
When I start mongo e.g.
mongod --port 27019 --dbpath /data/db2 --replSet rtb/test:27017 --rest
Mongo creates a 3 gig file. I am in dev and to reduce the size to e.g. 100 Megs. How to I do that?
MongoDB will pre-allocate data files. when it starts running. By default a new database will contain the following files (where test is the name of the DB)
test.ns (16MB)
test.0 (16MB)
test.1 (32MB)
As more files are needed, the pattern continues: 64MB, 128MB, 256MB, 1024MB, 2048MB. After 2GB, each new file will be 2GB.
In your case, you are running a replica set. Replica Sets require a database called local. This database contains the oplog which is used replication. The oplog is a capped collection.
The capped collection is a special collection which must be pre-allocated. In your case it is likely defaulting to about 3GB.
You can control the size of the local database using the --oplogSize parameter.
If you want to keep your local copy small you can also use the --smallfiles option which will slow down the speed of pre-allocation.