I would like to know about an ETL tool to upload data into a MongoDB, I've tried with Pentaho and Talend, but I would prefer something a bit lighter, because my process is so simple.
Thanks!
Assume you have a JSON file 'test.json' (and also assume it is valid JSON) you can use command line program mongoimport.
Example:
mongoimport --host localhost:27017 --db test --collection test --username johndoe --password my secret --authenticationDatabase admin --file test.json
Related
I need to create a mongodump of a specific collection from a specific database in a replica set. I am issuing the following command from the windows command shell:
mongodump --uri="mongodb://dd-domain-dev-shard-0-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-1-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-2-abcde.azure.mongodb.net:27017/?replicaSet=dd-domain-dev-shard-0" --db=TargetDatabase_dev --collection=abcCollection
When doing so, the following error occurs:
error parsing command line options: illegal argument combination: cannot specify --db and --uri
How can I create an acceptable command that includes both the database and collection? I must specify both because the replica set contains several databases, and I only want that single collection in the output file.
MongoDb version: 4.2.11
MongoDump version: r4.0.1
Include the database name in the URI.
mongodump --uri="mongodb://dd-domain-dev-shard-0-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-1-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-2-abcde.azure.mongodb.net:27017/TargetDatabase_dev?replicaSet=dd-domain-dev-shard-0" --collection=abcCollection
If that doesn't work:
mongodump --uri="mongodb://dd-domain-dev-shard-0-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-1-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-2-abcde.azure.mongodb.net:27017/TargetDatabase_dev?replicaSet=dd-domain-dev-shard-0&authSource=admin" --collection=abcCollection
The previous answer is correct but you will need to include most options in the --uri string.
If you want to keep those as dashed options (--option) you can do it
by using --host instead.
mongodump --host="dd-domain-dev-shard-0/dd-domain-dev-shard-0-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-1-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-2-abcde.azure.mongodb.net:27017" --db=TargetDatabase_dev
--collection=abcCollection
You will probably need to authenticate though, like so
mongodump --host="dd-domain-dev-shard-0/dd-domain-dev-shard-0-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-1-abcde.azure.mongodb.net:27017,dd-domain-dev-shard-2-abcde.azure.mongodb.net:27017/" --db=TargetDatabase_dev --collection=abcCollection --authenticationDatabase admin -u <username> -p <password>
Here is the relevant MongoDB docs
i migrate local db into atlas using following process
export db collection one by one using mongoexport --db bla
--collection usersettings --jsonArray --out ~/Desktop/users.json command
import these collection on atlas using mongoimport --host
cluster0-shard-00-00-c7jiq.mongodb.net:27017 --db Eltar --type json
--file ~/Desktop/userotp.json --authenticationDatabase admin --ssl --username name --password pass command
Now, when i connect to local mongo shell and run the query db.users.find() it shows all the record but when i run the same query db.users.find() after connecting atlas shell it shows only one record.
Records are showing on atlas but unable to get them using query
dont know what i am doing wrong here, any help will be appreciated thanks.
Done by migrating mongo db to mlab instead of mongo db atlas, still don't know the issue. But everything works fine on mlab
I am trying to restore data to the Azure Cosmos DB Emulator from MongoDB in order to test my Application. I originally used the Data Import Tool but realised after reading the Documentation this is for use with the SQL API for Cosmos DB while I want to be using the MongoDB API.
I successfully exported my Data from Mongo to json file with the command line:
D:\MongoDb\bin>mongoexport.exe --db Vehicles --collection Cars --out C:\Temp\Cars.json
2018-09-18T10:02:21.210-0400 connected to: localhost
2018-09-18T10:02:21.212-0400 exported 100 records
I am then trying to import this into Azure Cosmos DB Emulator with below command
D:\MongoDb\bin> mongoimport.exe --host localhost:10255 -u admin -p C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== --ssl --sslAllowInvalidCertificates --db Vehicles --collection Cars --type json --file "C:\Temp\Cars.json"
However I am getting the following Error:
2018-09-18T11:00:38.829-0400 Failed: error connecting to db server: Database Account admin does not exist
ActivityId: ada5953a-0000-0000-0000-000000000000, Microsoft.Azure.Documents.Common/1.22.0.0
2018-09-18T11:00:38.830-0400 imported 0 documents
I had created a DB in the Azure Cosmos DB Emulator called Vehicles with a Cars collection so not sure what I am doing wrong - or can the Azure Cosmos Emulator not be used for what I am attempting?
I tried changing the admin to Vehicles in the command line script but get the same error.
From the Azure Cosmos DB Emulator this is my Mongo Connection string which is where I was pulling the details from for the import
mongodb://localhost:C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==#localhost:10255/admin?ssl=true
Ideally I want to get the Emulator running as a way to test things locally
The DatabaseAccount name in the CosmosDB emulator is localhost not admin so your mongoimport.exe string should look like this:
D:\MongoDb\bin> mongoimport.exe --host localhost:10255 -u localhost -p C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== --ssl --sslAllowInvalidCertificates --db Vehicles --collection Cars --type json --file "C:\Temp\Cars.json"
I have a collection named dashboard in one db and i want to copy that collection to another db using robomongo. How can i do this? I tried creating a new collection in 2nd db and tried copying but it failed. so somebody please help me
Another db - another connection. But Robomongo works only with one connection in one period of time. That is why it is impossible.
I suggest you to use mongoimport/mongoexoprt tools for your task. They comes with mongo, are located in same folder as mongod.exe and allows to move collections via databases, by exporting to and importing from a json file.
Code sample:
mongoexport --db testFrom --port portFrom --username userFrom --password passwordFrom --collection yourCollection --out test.json
mongoimport --db testTo --port portTo --username userTo --password passwordTo --collection yourCollection --file test.json
I was trying to backup my mongo database on the product sever.and then restore then back to the staging server.
and here comes some problem, there are a lot of collections in db, I want to igonre some collections that I don't want to restore on staging server.
I can approach this by dumpping the staging db, dumpping the producting db, and then restore the prodct to staging useing --drop option. and restore the specified collections in staging db. uh..it's really bad.
1. dump producting db
mongodump --host product-server-host --username abcd --password bcda -d db -o pruduct-dump-dir
2. dump staging db
mongodump --host staging-server-host --username abcd --password bcda -d db -o staging -dump-dir
3. restore all collection, then restore the collection back
restore pruduct-dump-dir to staging server
mongorestore --host staging-server-host --username abcd --password bcda --drop pruduct-dump-dir
mongorestore --host staging-server-host --username abcd --password bcda --drop --collection coll pruducting-dump-dir
Is there any option like ignore-collection when I'm dumpping?
any suggestion will be appreciated :3
Now available from version 3.0.0
--excludeCollection <collection_name>
--excludeCollectionsWithPrefix <collection_prefix>
Repeat to exclude more than 1
Checkout the documentation
mongodump --db test --excludeCollection=users --excludeCollection=salaries
You can add --collection COLLECTION_NAME to dump the collection you need. By default, if you do not specify a collection to dump from a database, MongoDump will dump all collections in that database.
As of Mongo 3.4, you can now specify an --nsExclude <namespace pattern> option when restoring from a Mongo database dump, which will exclude the specified namespaces from the restore operation. This is particularly useful when the mongodump operation has already occurred.
Official documentation here: https://docs.mongodb.com/manual/reference/program/mongorestore/#cmdoption-nsexclude
You can exclude multiple collections with wildcards:
mongorestore --db test --nsExclude 'test.*_tmp'
Or alternatively, specifying multiple --nsExclude options work as well:
mongorestore --db test --nsExclude 'test.collection1' --nsExclude 'test.collection2'
I had to do the same while backing up a mongo db. If you use python (or any other language), you can use similar approach as well. After doing the mongodump, you simple have to remove the unwanted collection's bson & the metadata.json files.
import os
EXCLUDE_COLLECTIONS = ['collection_1', 'collection_2']
db_dump_path = "/Path/to/mongodump"
db_name = "name_of_db"
for collection_name in EXCLUDE_COLLECTIONS:
bson_file_path = os.path.join(db_dump_path, db_name, '{}.bson'.format(collection_name)
meta_file_path = os.path.join(db_dump_path, db_name, '{}.metadata.json'.format(collection_name)
if os.path.exists(bson_file_path) and os.path.exists(meta_file_path):
os.remove(bson_file_path)
os.remove(meta_file_path)