I have 40,000 lines that I wish to insert into a mongo database but I'm not sure the best way to do this.
Would I use the command line 'mongo' and paste it (I've been messing with this idea and it doesn't seem like it is ideal)
Are there any optimal or best practices here? Should I just do it in mongoid instead of mongo?
Ugh. I posted an answer here and then realized there is a command called mongoimport that can directly write the DB files.
Lets assume you have your data in JSON format, one object per line, like so:
{"a":"a1"}
{"a":"a2"}
{"a":"a3"}
You would then do:
mongoimport --dbpath PATH_TO_YOUR_DB_DIR -d DB_NAME -c COLLECTION_NAME --file JSON_FILE_NAME
The use of dbpath needs to lock the data directory, so it cannot be used if a mongod is currently accessing the same path. mongoimport --help gives you pretty useful info.
Related
I am looking to make a new db on my localhost from a mongodbdump file. the file looks like this:
C:\Users\ME\FILEPATH\database\somedata-mongodbdump.tar.gz
How can I use mongo shell commands to "restore" this data to a new db on my local host? I know how to set up a new DB and insert records no problem, but I am not sure how to unload this data into one.
I am not finding anything in my searches or the documentation as to how to straightforwardly do this.
I'd like to get this file into a DB that I can host locally and use as the back end for testing an app.
I was misguided here and the previous answers is incorrect. While it may be pretty beginner stuff for some people, I will answer my own question in case anyone else runs into this:
A .tar.gz file is effectively a double zipped file. For Mongo, we need a .bson file. Unzip the file with 7zip or through CMD.
Once you have the BSON in a directory, in CMD (not mongo shell) type:
mongorestore --db 'yourdbname' --collection 'yourcollectionname' C\filepath\filepath\filename.bson
use mongo restore
mongorestore -u 'user' -p 'password -d 'database' -c 'collection' 'path to file'
if you are using a different database for auth then add
--authenticationDatabase='db for auth'
So I have a MongoDB in an Azure Cosmos DB service. It contains a collection of 1500 documents and I want to download this whole collection in a JSON format. I've tried several methods without success, namely
test_collection.find({})
Which gave me a cursor timeout. Using
{ timeout : false }
Did not help. Then I tried to use mongoexport:
mongoexport -h host_name --port 1234 -u user_name -p password
-d admin -c collection_name -o data.json --ssl
which gives me 0 exported records. The firewall IP access control is off and I can connect to the database through Mongo shell just fine. Trying to export other collections doesn't work either. Also, it has to be by ssl otherwise I get a "database not found" right away.
I've thought about using skip and limit but it doesn't seem to be a good idea with large (and expanding) collections? Could someone please give me some advise as to how I best achieve or overcome these obstacles to download my collection? It doesn't matter how, I just simply need to download the collection. Thank you.
You possibly have a few incorrect parameters, and a missing parameter:
Are you sure your database name is admin?
You need to specify --sslAllowInvalidCertificates
For host/port: this should look something like:
/h yourcosmosaccount.documents.azure.com:10255
If you take a look at the "quick start" tab in your Cosmos DB settings, you'll see the command line string for mongo (well, mongo.exe in the example). Just grab those parameters and use them for mongoexport.
I just ran this against a sample Cosmos DB (MongoDB API) database of mine, with no issue:
Here's the generic command-line equivalent:
mongoexport /h <host:port> /u <username> /p <password> /ssl /sslAllowInvalidCertificates /d <database> /c <collection> /o <outputfile>.json
Using mongodb and am trying to get a specific value from a collection in the db. I am able to get the complete export using
mongoexport --db database --collection name
But the output is a large file and I am trying to get a specific set of key/pair in it.
ex: "Name": "Value"
there are several names and I just need to print all the names in the collection.
What would be the command syntax from a UNIX shell ?
I looked at this but that is from with in the mongo shell.
thanks
To request all fields from collection yourCollection in MyDatabase :
mongo --quiet 127.0.0.1/MyDatabase --eval 'printjson(db.yourCollection.find().toArray());'
To request only fields name field from collection yourCollection in MyDatabase :
mongo --quiet 127.0.0.1/MyDatabase --eval 'printjson(db.yourCollection.find({},{"_id":0,"name":1}).toArray());'
You could also have a script to execute and save you the time of manually writing in those commands. Execute something like
mongo localhost:27017/test myfile.js
and in the javascript file input
db.name.findOne()
db.name().find({},{"_id":0,"Name":1}).toArray());
please see https://docs.mongodb.com/manual/tutorial/write-scripts-for-the-mongo-shell/ and [How to execute mongo commands through shell scripts?
If your goal is to export documents matching a specific condition from the database, you can pass a query to mongoexport using the -q parameter. For example:
mongoexport -d db -c coll -q '{"Name":"Value"}'
This will export all documents containing the field "Name" having the value "Value".
You can also pass the --quiet parameter to mongoexport if you prefer to have the output without any informative content, such as number of exported documents.
Please see https://docs.mongodb.com/manual/reference/program/mongoexport/ for more information regarding the mongoexport tool.
I am following the example given on MongoDB's website here, but I am running into trouble when trying to import sample data.
When running the command
mongoimport --db test --collection restaurants --drop --file primer-dataset.json
I get the error:
Failed: open primer-dataset.json: The system cannot find the file specified
The problem is, I am not sure what directory MongoDB expects this file to be in. I tried placing it in data/db, but that did not work. Note that I am only using default settings.
I know this is a somewhat trivial question and I feel stupid for asking it but I can not find documentation on this anywhere. Where is MongoDB expecting import files?
MongoDB expects the file to be in the directory from where you are running the command mongoimport.
If you place your file under data/db then set mongodb path as global environment variable and execute the command from data/db directory.
Additionally if you have security enabled for your mongodb then you need to execute command as below
mongoimport --username admin --password password --db test --collection restaurants --drop --file primer-dataset.json
here admin is the user authorized to perform db operations for test database and restaurants is the collection name.
For Windows!
Save file using notepad++ in .json format at MongoDB/bin where mongoimport command is present.
Simple notepad has trouble doing it.
It happened to me as well. The issue was that though the file was visible as restaurants.json actually the file was restaurants.json.json (since saved in JSON format). The issue was resolved after properly changing the name.
i have trouble like you, check you path to file, mongoimport.exe and your file may be stay in another folders.
use mongoimport -d test1 -c restaraunts companies.json for import to mongodb.
Check the filename extension, and make sure it's a ".json" file;
After this I successfully run the
mongoimport --db test --collection restaurants --drop --file [path\to\Json file]
command;
In my case, I removed the --drop parameter and it worked perfectly. I guess, it is throwing this error:
Failed: open paht/file-name.json: The system cannot find the file specified.
because the collection it wants to drop is not available, because I have not created any before.
you must copy your json file into C:\Windows\System32 and write this command on cmd:
mongoimport --db test --collection mongotest --type json --file yournamefile.json
I was given a data dump of bson files. In the mongo db, the database and the collections exists. These are updates to each of the collections in the database. So, in the given directory, there are about 30 bson files for each collection.
From the command line, I am using ubuntu, how do I append and load? Mongo is on my localhost with no username or password.
Thanks
Took me a while to get around this excuse for an error. In the end I went to the directory outside of my dump folder, and did the following...
For a full DB restore:
mongorestore --drop dump/mydb
Note that the mongodump operation will create individual folders for each database within the dump folder it creates, so you need to specify the full relative path, as above.
For a single collection:
mongorestore --drop -d mydb -c mycollection dump/mydb/mycollection.bson
The usual syntax is:
mongorestore -d dbname -c collectionname dir/file.bson
are you looking for mongorestore? http://www.mongodb.org/display/DOCS/Import+Export+Tools#ImportExportTools-mongorestore
import Bson
mongorestore -d dbname -c collectionname dir/file.bson
import Json
mongoimport --collection NAME --file NAME.
http://docs.mongodb.org/v2.2/reference/mongoimport/
Since Mongo restore does not update the current records this would not be a good choice.
Mongorestore only appends new records as stated:
mongorestore just does inserts with the data to restore; if existing
data (like with the same _id) is there it will not be replaced.
You may wish to build a BSON parser in your language of choice and make a more complex tool than mongorestore, since mongorestore is only designed to "restore" (as the name kinda suggests) a database/collection you will need to write something a little more complicated to do what you want and that depends heavily on your server-side language.
Edit
This is actually better done with mongoexport and mongoimport:
http://www.mongodb.org/display/DOCS/Import+Export+Tools#ImportExportTools-mongoimport
With mongoexport you could export a JSON file and give the command line for it do upserts. So I would personally go back to the person who gave this file and tell them that you actually want a mongo export file instead.
1) Go to the directory where the "dump" folder is located in CMD.
2) Run the mongorestore command.