Moving MongoDB's data folder? - mongodb

I have 2 computers in different places (so it's impossible to use the same wifi network).
One contains about 50GBs of data (MongoDB files) that I want to move to the second one which has much more computation power for analysis. But how can I make MongoDB on the second machine recognize that folder?

When you start mongodprocess you provide an argument to it --dbpath /directory which is how it knows where the data folder is.
All you need to do is:
stop the mongod process on the old computer. wait till it exits.
copy the entire /data/db directory to the new computer
start mongod process on the new computer giving it --dbpath /newdirectory argument.
The mongod on the new machine will use the folder you indicate with --dbpath. There is no need to "recognize" as there is nothing machine specific in that folder, it's just data.

I did this myself recently, and I wanted to provide some extra considerations to be aware of, in case readers (like me) run into issues.
The following information is specific to *nix systems, but it may be applicable with very heavy modification to Windows.
If the source data is in a mongo server that you can still run (preferred)
Look into and make use of mongodump and mongorestore. That is probably safer, and it's the official way to migrate your database.
If you never made a dump and can't anymore
Yes, the data directory can be directly copied; however, you also need to make sure that the mongodb user has complete access to the directory after you copy it.
My steps are as follows. On the machine you want to transfer an old database to:
Edit /etc/mongod.conf and change the dbPath field to the desired location.
Use the following script as a reference, or tailor it and run it on your system, at your own risk.
I do not guarantee this works on every system --> please verify it manually.
I also cannot guarantee it works perfectly in every case.
WARNING: will delete everything in the target data directory you specify.
I can say, however, that it worked on my system, and that it passes shellcheck.
The important part is simply copying over the old database directory, and giving mongodb access to it through chown.
#!/bin/bash
TARGET_DATA_DIRECTORY=/path/to/target/data/directory # modify this
SOURCE_DATA_DIRECTORY=/path/to/old/data/directory # modify this too
echo shutting down mongod...
sudo systemctl stop mongod
if test "$TARGET_DATA_DIRECTORY"; then
echo removing existing data directory...
sudo rm -rf "$TARGET_DATA_DIRECTORY"
fi
echo copying backed up data directory...
sudo cp -r "$SOURCE_DATA_DIRECTORY" "$TARGET_DATA_DIRECTORY"
sudo chown -R mongodb "$TARGET_DATA_DIRECTORY"
echo starting mongod back up...
sudo systemctl start mongod
sudo systemctl status mongod # for verification

quite easy for windows, just move the data folder to the target location
run cmd
"C:\your\mongodb\bin-path\mongod.exe" --dbpath="c:\what\ever\path\data\db"

In case of Windows in case you need just to configure new path for data, all you need to create new folder, for example D:\dev\mongoDb-data, open C:\Program Files\MongoDB\Server\6.0\bin\mongod.cfg and change there path :
Then, restart your PC. Check folder - it should contains new files/folders with data.

Maybe what you didn't do was export or dump the database.
Databases aren't portable therefore must be exported or created as a dumpfile.
Here is another question where the answer is further explained

Related

Where is the MongoDB Data?

This is more of a general question about how MongoDB works.
But I've been using MongoDB for a while and everything seems to be working for me. The part that currently confuses me, though, is that when I visit the directory where the MongoDB data is saved (I'm using the default data/db) the directory is empty. The data is being persisted, I'm just confused - why does the directory appears empty on my computer?
I'm on Windows, if that's worth anything.
You can execute :
db.serverCmdLineOpts().parsed.storage.dbPath
from inside the mongo process and find what the dbPath startup parameter show to find where your data is saved
or alternatively check from the config file in windows:
<install directory>/bin/mongod.cfg
dbPath can be confusing.
When you run mongod without --dbpath option or unset it in config file then it defaults to \data\db
However, when you use the default config files which comes along the installation then the dpPath is different (I don't remember by heart which). So you should really check path with db.serverCmdLineOpts() as suggested by R2D2
I know, for Linux the default dbPath is /data/db but the pre-installed config file /etc/mongod.conf has set /var/lib/mongo

MongoDB data not stored in /data/db, although it is saving data... but where?

I'm completely new to MongoDB - I've installed all the programs and set it up so that I run mongod.exe and then mongo.exe. I created a new database called 'test' and inserted some data. However, I can't see any files being created in the /data/db directory. The data is definitely being stored somewhere, and when I closed down all the cmd boxes and started the processes again, the data I initially pushed onto the DB is still there.
I tried running the command 'db.adminCommand("getCmdLineOpts")' which apparently should show my DB Path, but it doesn't:
Can anyone offer some insight on this? Thanks :)
For using mongod.exe without parameters default is C:\data\db. Check start up parameters if you run server as service.
You can run this command to retrieve the dbpath if you in a Linux environment:
grep dbpath /etc/mongod.conf
db.adminCommand("getCmdLineOpts") is not running the db path as it was not specified a a command line parameter when the mongod process was started. It seems the db path is configured in the mongod.conf configuration file and the above command should return it.

Mongo error on I control hotfix

I have tried to start mongod.exe from my 2008 R2 server and im getting this error:
I CONTROL Hotfix kb2731284 or a later update is not installed, will zero-out files.
I didnt find any update or what is I CONTROL, someone have encounter with the problem?
Thank you.
Well, I just faced the same issue. I installed the fix, but saw nothing but same error. So that's how I got over it: just create a folder structure in your C catalog like this one: C:\data\db. So it worked for me perfectly. I use Windows 7 x64.
I had to run the command in following format & it worked for me:
C:\mongodb\bin\mongod.exe --dbpath d:\test\mongodb\data
Ref: Original documentation link
In my case the problem was that I had a space in my data path: E:\Program Files\MongoDB\Data\
I ran dir /x within E:\ to get the short path for Program Files and used the short path instead: mongod.exe --dbpath E:\PROGRA~1\MongoDB\Data\
It worked. Shocking that spaces in paths are still an issue!
The above error comes when mongodb not found the directory like "C:\data\db" which is by default when you install mongodb on your machine.
So just go to c drive create a directory like "data\db" and now run the command "mongodb.exe".
If you want to relocate the database location to any other drive say D drive in that case create a directory like "D:\mongodb\data" and run the command like "mongodb.exe --dbpath d:\mongodb\data".
I got the similar error
I CONTROL Hotfix kb2731284 or a later update is installed, will zero-out files.
Notice the message says later update is installed not later update is not installed in my case. But It was not starting the server.
Creating a different folder structure for the db Drive\<mongo-db-folder>\<yourdb>and providing it with --dbpath flag while running the mongod worked for me. The short name for Program Files PROGRA~1 didn't work for me. Maybe this has something to do with the rights managment.
This known issue is in Windows 7. In Windows 8 this problem haven't appeared yet.
Firstly I would recommend to change your mongo installation path to path with folders names without gaps. For example, default mongo installation path is C:\program files\bla-bla... to avoid a wasting of time for looking for solutions of this issue, change your path to C:\mongo.. and make sure that you haven't names with gaps in your path.
After that install this hot-fix.
Another thing I suggest you to do: Create folder data\db in disc C.
You could also change your mongo work path each time you use mongo shell, but it's overwork by my modest opinion.
I've done these steps and my mongo has started to work.
Getting started on mongodb was not that difficult. Well I have been facing similar error on mongodb to start.
I have getting error
Services
Windows could not start the MongoDB on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 100.
OK
very simple way to overcome.
Solution is the Mongodb service has not been started
to do so. Open CMD in admin mode and type NET START "Mongodb" > This will start the service.
Inspite of this if services is not getting started or mongo.exe is not starting. It is coz your mongodb has locked the file. Mongodb is shared file system and it gets locked. You need to for mongo.lock file in your created dir. Delete the file and start the service. You should be able to start now.
Cheers
By just creating a folder structure like below in my "C:\" drive , i resolved the issue :=>
c:\data\db
Sometime it works without creating the above mentioned folder structure, i.e. MONGODB itself creates the same (as it worked for me in Windows 8 laptop) but sometimes it needs to be explicitly done (as i need to for my windows 7 installed desktop)
Also you can browse through the complete installation and connection process for MONGODB.
Similiar to above, but I had a typo in my mongodb.cfg file which meant that Mongo couldn't find the data directory on startup. Fixed the typo and Mongo starts. Given the previous response I suspect that the failure to start is more of an issue over finding the data directory than the Hotfix message
If you had got the below error message::
2015-05-29T05:05:54.207+0530 I CONTROL Hotfix KB2731284 or later update is not
installed, will zero-out data files
2015-05-29T05:05:54.219+0530 I STORAGE [initandlisten] exception in initAndLis
en: 29 Data directory D:\data\db\ not found., terminating
2015-05-29T05:05:54.219+0530 I CONTROL [initandlisten] dbexit: rc: 100
then just create a directory "D:\data\db\""
then restart the mongod
it should work fine
you need to have mongod.exe running in one cmd window. then use another cmd window, write mongo and hit enter
You need to create the C:\Data\Db directory if you don't specify an existing directory when you run mongod.exe.
This directory is the default one.
http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/
You can do fix [initandlisten] exception in initAndListen: 29 Data directory /data/db not found., terminating by following given two steps.
You can create the D:\data\db\ directory and restart the MongoDB server.
You can create the data folder as per your choice and run the server with the path of your data folder and restart the MongoDB server
i.e. D:\MongoDB\Server\3.0\bin>mongod.exe --dbpath D:\MongoDB\Server\data​
I just deleted mongod.lock in my db directory, and now it works fine.
1 : Download and install the fix from here :
http://hotfixv4.microsoft.com/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix405791/7600/free/451413_intl_x64_zip.exe
them reboot windows
2 : this you should make a folder called "data" and within it another one called "db"
c:/data/db ;
I would start with the second step , it's most likely to solve The problem .
As for what "I CONTROL" means, the "I" is for "Information" (we have W for warning and E for error) and CONTROL is the component writing that log/message (we have also NETWORK, WRITE, QUERY and COMMAND components).
So you could get also I COMMAND ..., I QUERY ..., I WRITE ... :)
Well, that is a common problem when you shut down your machine and mongodb server is running, it is pretty clear that MongoDb has created a lock in your db folder.
Just go to c:\data\db and remove MONGOD.LOCK
Run Mongo and it will work again.

MongoDB does not see database or collections after migrating from localhost to EBS volume

full disclosure: I am a complete n00b to mongodb and am just getting my feet wet with using mongo on AWS (but have 2 decades working in IT so not a total n00b :P)
I setup an EBS volume and installed mongo on a EC2 instance.
My problem is that I provisioned too small an EBS volume initially.
When I realized this I:
created a new larger EBS volume
mounted it on the server
stopped mongo ( $ sudo service mongod stop)
copied all my /data/db files into the new volume
updated conf files and fstab (dbpath, logpath, pidfilepath and mount point for new volume respectively)
restarted mongod
When I execute: $ sudo service mongod start
- everything runs fine.
- I can futz about in the admin and local databases.
However, when I run the mongos command: > show databases
- I only see the admin and local.
- the database I copied into the new volume (named encompass) is not listed.
I still have a working local copy of the database so my data is not lost, just not sure how best to move mongo data around other than:
A) start all over importing the data to the db on the AWS server (not what I would like since it is already loaded in my local db)
B) copy the local db to the new EBS volume again (also not preferred but better that importing all the data from scratch again!).
NOTE: originally I secure copied the data into the EBS volume with this command:
$ scp -r -i / / ec2-user#:/
then when I copied between volumes I used a vanilla cp command.
Did I miss something here?
The best I could find on SO and the web was this process (How to scale MongoDB?), but perhaps I missed a switch in a command or a nuance to the process that rendered my database files inert/useless?
Any idea how I can get mongo to see my other database files and collections?
Or did I make a irreversible error somewhere along the way?
Thanks for any help!!
Are you sure you conf file is being loaded? You can, for a test, load mongod.exe and specify the path directly to your db for a test, i.e.:
mongod --dbpath c:\mongo\data\db (unix syntax may vary a bit, this is windows)
run this from the command line and see what, if anything, mongo complains about.
A database has a very finicky algorithm that is easy to damage. Before copying from one database to another you should probably seed the database, a few dummy entries will tell you the database is working.

Where are the db files? at /var/lib/mongodb I cant find any increase in size. I ran very big loop to create lakhs of objects

I am using UBUNTU, from /etc/mongod.conf, I found that /var/lib/mongdb is the path for data.
I have found some files like collectionname.0, .1, .ns in that directory. but when I run a very big loop(lakhs), I am able to get them back using mongo shell, but that mongodb directory size is not increasing, so there must be someother place where this data is being stored
What is that place?
There is no another place. As indicated by #itsbruce, in Ubuntu it's /var/lib/mongodb.
On a non-packaged installation (on Linux), i.e. without a /etc/mongodb.conf file, the default is /data/db/ (unless otherwise specified).
You can modify the switch "--dbpath" on the command line to designate a different directory for the mongod instance to store its data. Typical locations include: /srv/mongodb, /var/lib/mongodb or /opt/mongodb.
(Windows systems use the \data\db directory.) If you installed using a package management system.
I recommend using the db.collection.stats command as outlined here to monitor the size of your collection as you insert data. The link also explains what each field (in the output) means.
That is the correct data location for MongoDB on Ubuntu. MongoDB pre-allocates filespace. Are you sure you have generated more data than would fit into the initial pre-allocated files? Try blowing away any existing data files and restarting Mongo with the --noprealloc flag. Then add data.