I would like to know about an ETL tool to upload data into a MongoDB, I've tried with Pentaho and Talend, but I would prefer something a bit lighter, because my process is so simple.
Thanks!
Assume you have a JSON file 'test.json' (and also assume it is valid JSON) you can use command line program mongoimport.
Example:
mongoimport --host localhost:27017 --db test --collection test --username johndoe --password my secret --authenticationDatabase admin --file test.json
i migrate local db into atlas using following process
export db collection one by one using mongoexport --db bla
--collection usersettings --jsonArray --out ~/Desktop/users.json command
import these collection on atlas using mongoimport --host
cluster0-shard-00-00-c7jiq.mongodb.net:27017 --db Eltar --type json
--file ~/Desktop/userotp.json --authenticationDatabase admin --ssl --username name --password pass command
Now, when i connect to local mongo shell and run the query db.users.find() it shows all the record but when i run the same query db.users.find() after connecting atlas shell it shows only one record.
Records are showing on atlas but unable to get them using query
dont know what i am doing wrong here, any help will be appreciated thanks.
Done by migrating mongo db to mlab instead of mongo db atlas, still don't know the issue. But everything works fine on mlab
I dump the mongodb database using mongodump --db dbName. Now when on different system I tried to restore it using mongorestore it is showing me error:
Failed: mindcentral.user: error creating collection mindcentral.user: error running create command: BSON field 'OperationSessionInfo.create' is a duplicate field
My dump folder is on desktop, so I am using command
mongorestore --db DBName Desktop/dump/DBName
You just Follow:
Mongorestore:
Syntax:
mongorestore -d databasename databasepath.
Example:
mongorestore -d user ./Desktop/dump/user
where the user is a database name.
I run this
mongorestore --db dbName
and I got
the --db and --collection args should only be used when restoring from
a BSON file. Other uses are deprecated and will not exist in the
future; use
--nsInclude instead
so I used
mongorestore --nsInclude 'dbName.*'
and I got
2018-01-06T21:28:02.106+0200 using default 'dump' directory
2018-01-06T21:28:02.142+0200 preparing collections to restore from
2018-01-06T21:28:02.147+0200 done
but I don't see any db of dbName that created.
I found the answer here Mongorestore of a db causing me trouble
mongorestore --db you_db_name --drop dump/you_db_name
Hope this helps.
I was trying to backup my mongo database on the product sever.and then restore then back to the staging server.
and here comes some problem, there are a lot of collections in db, I want to igonre some collections that I don't want to restore on staging server.
I can approach this by dumpping the staging db, dumpping the producting db, and then restore the prodct to staging useing --drop option. and restore the specified collections in staging db. uh..it's really bad.
1. dump producting db
mongodump --host product-server-host --username abcd --password bcda -d db -o pruduct-dump-dir
2. dump staging db
mongodump --host staging-server-host --username abcd --password bcda -d db -o staging -dump-dir
3. restore all collection, then restore the collection back
restore pruduct-dump-dir to staging server
mongorestore --host staging-server-host --username abcd --password bcda --drop pruduct-dump-dir
mongorestore --host staging-server-host --username abcd --password bcda --drop --collection coll pruducting-dump-dir
Is there any option like ignore-collection when I'm dumpping?
any suggestion will be appreciated :3
Now available from version 3.0.0
--excludeCollection <collection_name>
--excludeCollectionsWithPrefix <collection_prefix>
Repeat to exclude more than 1
Checkout the documentation
mongodump --db test --excludeCollection=users --excludeCollection=salaries
You can add --collection COLLECTION_NAME to dump the collection you need. By default, if you do not specify a collection to dump from a database, MongoDump will dump all collections in that database.
As of Mongo 3.4, you can now specify an --nsExclude <namespace pattern> option when restoring from a Mongo database dump, which will exclude the specified namespaces from the restore operation. This is particularly useful when the mongodump operation has already occurred.
Official documentation here: https://docs.mongodb.com/manual/reference/program/mongorestore/#cmdoption-nsexclude
You can exclude multiple collections with wildcards:
mongorestore --db test --nsExclude 'test.*_tmp'
Or alternatively, specifying multiple --nsExclude options work as well:
mongorestore --db test --nsExclude 'test.collection1' --nsExclude 'test.collection2'
I had to do the same while backing up a mongo db. If you use python (or any other language), you can use similar approach as well. After doing the mongodump, you simple have to remove the unwanted collection's bson & the metadata.json files.
import os
EXCLUDE_COLLECTIONS = ['collection_1', 'collection_2']
db_dump_path = "/Path/to/mongodump"
db_name = "name_of_db"
for collection_name in EXCLUDE_COLLECTIONS:
bson_file_path = os.path.join(db_dump_path, db_name, '{}.bson'.format(collection_name)
meta_file_path = os.path.join(db_dump_path, db_name, '{}.metadata.json'.format(collection_name)
if os.path.exists(bson_file_path) and os.path.exists(meta_file_path):
os.remove(bson_file_path)
os.remove(meta_file_path)