Using mongodb and am trying to get a specific value from a collection in the db. I am able to get the complete export using
mongoexport --db database --collection name
But the output is a large file and I am trying to get a specific set of key/pair in it.
ex: "Name": "Value"
there are several names and I just need to print all the names in the collection.
What would be the command syntax from a UNIX shell ?
I looked at this but that is from with in the mongo shell.
thanks
To request all fields from collection yourCollection in MyDatabase :
mongo --quiet 127.0.0.1/MyDatabase --eval 'printjson(db.yourCollection.find().toArray());'
To request only fields name field from collection yourCollection in MyDatabase :
mongo --quiet 127.0.0.1/MyDatabase --eval 'printjson(db.yourCollection.find({},{"_id":0,"name":1}).toArray());'
You could also have a script to execute and save you the time of manually writing in those commands. Execute something like
mongo localhost:27017/test myfile.js
and in the javascript file input
db.name.findOne()
db.name().find({},{"_id":0,"Name":1}).toArray());
please see https://docs.mongodb.com/manual/tutorial/write-scripts-for-the-mongo-shell/ and [How to execute mongo commands through shell scripts?
If your goal is to export documents matching a specific condition from the database, you can pass a query to mongoexport using the -q parameter. For example:
mongoexport -d db -c coll -q '{"Name":"Value"}'
This will export all documents containing the field "Name" having the value "Value".
You can also pass the --quiet parameter to mongoexport if you prefer to have the output without any informative content, such as number of exported documents.
Please see https://docs.mongodb.com/manual/reference/program/mongoexport/ for more information regarding the mongoexport tool.
Related
I'm trying to create a mongo dump for 1 single document inside a mongo collection.
When I do this on windows command line:
mongodump /host:x.x.x.x /port:27017 /username:my_user /password:my_pass -d my_db -o C:\fsdump -c "fs.files" -q '{_id: ObjectId("28ad7bkjia3e927d690385ec")}'
I get this error:
positional arguments not allowed: [ObjectId(28ad7bkjia3e927d690385ec)}']
when I change the id in mongo from ObjectId("28ad7bkjia3e927d690385ec") to "28ad7bkjia3e927d690385ec", and when I dump like this:
mongodump /host:x.x.x.x /port:27017 /username:my_user /password:my_pass -d my_db -o C:\fsdump -c "fs.files" -q '{_id: "28ad7bkjia3e927d690385ec"}'
then it works as expected
so my question is how can I use mongodump and do filtering on specific ObjectId's ?
or is there another way to create an export for a subset of documents in a collection (instead of the entire collection)?
ObjectId("28ad7bkjia3e927d690385ec") is a javascript function call to the ObjectId constructor. This is not valid JSON. "28ad7bkjia3e927d690385ec" is also not a valid ObjectId.
Mongodump uses an extended form of JSON, which had a tag to indicate the field type, so you would specify an ObjectId like this:
-q '{"_id":{"$oid":"5f76b7cc0311bd14f80a3dec"}}'
I'm trying to insert the TWDS1E1.json file into mongodb through the command prompt:
db.collections.insert( TWDS1E1.json )
But getting the error:
TWDS1E1.json is not defined.
Mongo is not my thing, what am I doing wrong here?
In command prompt whose directory path is the path where mongoimport.exe is available type the commands
For normal JSON
mongoimport -d test -c docs --file example2.json
For array type JSON
mongoimport --jsonArray -d test -c docs --file example2.json
Please see docs for more information
You cannot use the collection.insert() command to insert a file.
insert() is used to insert actual objects, e.g.
db.myCollection.insert({"name":"buzz"});
To bulk load a JSON file, use mongoimport
I have a file named services.json containing a data base that I exported from a windows mongodb, and I want to import that file into robomongo (connected to mongodb installed by npm) on Ubuntu.
I'm a beginner and I don't know how to proceed, which terminal use (robomongo or Ubuntu)?
to import data for a collection in Robomongo:
Right click on collection.
Select 'insert Document'.
Paste your json data
Click validate.
Click save.
Ok, I found the answer. In shell Mac OS X or Unix type:
$ mongoimport -d your Database Name -c your Collection Name --file /path/to/my/fileThatIwantToImport.json
For anyone wishing to use mongoimport with a remote db (#andi-giga), here's what I did to make it work:
mongoimport -h xxx.mlab.com --port 2700 -d db_name -c collection_name -u user_name -p password --type json --file /Path/to/file.json
Arguments should be self-explanatory.
-h hostname
More information at this link
I don't have enough points to comment on Varun's answer, but if you use export jsonArray and then import using Robo3T (Robomongo), make sure to remove the commas in between the objects, as well as remove the square brackets.
It's not really a JSON format that ROBO 3T accepts, but rather bunch of JSON objects separated by newlines.
(if you use export Standard, then it's already formatted for document insert)
if this is not a bson, and only json, you can use mongoimport --jsonArray . refference Insert json file into mongodb
RoboMongo is just the UI for your mongod which is the primary daemon process for the MongoDB system.
The only option to import from RoboMongo is
Right Click on Collection -> Insert Document
Apart from this you can import using the mongoimport command from terminal.
Open terminal and type mongo
Now in mongo interactive shell
Use the following command to import the json file as collection
mongoimport -d database_name -c collection_name --file < path to the
json file
Tested :
mongoimport --jsonArray -d <DataBase Name> -c <Collection Name> --file /path/to/my/fileThatIwantToImport.json
It works very well!
Insert Document will insert all the JSON file data under a single document.
Apparently the tool does not support JSON import.
There are two ways to import the database into MongoDB. one is with robomongo/Robo 3T and one is with a shell command. I always choose the second method due to fewer and easy steps.
FIRST METHOD
Install MongoDB on your machine. Also, check it was installed properly or not by using mongod command on your terminal. So, for importing a new DB on your MongoDB write this command on your terminal
mongostore -host <HostIp | 127.0.0.1> -port <mongoPort | 27017> -db <DBname> <Directory-path>
So, for example you’re running MongoDB on local machine with default port i.e 27017 and your DB files are store at /usr/library/userDatabase then write this command and check DB is imported in your MongoDB
mongostore -host 127.0.0.1 -port 27017 -db userDatabase /usr/library/userDatabase
For more details check this article.
Import MongoDB using shell and robomongo/Robo 3T
How do I export the Collection from Mongo DB to xsl file.
I am using MongoVue export, but it is just displaying the Document and Number of keys in it, but not all the Key and Value pairs in the document..
Please help how to export a mongodb collection to xsl.
I'm not sure of using MongoVue (I haven't used that before) but if you are ok with an extra manual step you could always use mongoexport to output to a CSV file. You can then open and save as XSL via excel.
Here's the syntax for using mongoexport:
mongoexport -d <database> -c <collection> --csv --fields <field1,field2,...> -o <filename>
This will use the localhost mongo instance by defaykt so if you need to connect to another server see the docs:
http://docs.mongodb.org/manual/reference/program/mongoexport/
Can you use mongodump to dump the latest "x" documents from a collection? For example, in the mongo shell you can execute:
db.stats.find().sort({$natural:-1}).limit(10);
Is this same capability available to mongodump?
I guess the workaround would be to dump the above documents into a new temporary collection and mongodump the entire temp collection, but would be great to just be able to do this via mongodump.
Thanks in advance,
Michael
mongodump does not fully expose the cursor interfaces.
But you can work around it, using the --query parameter.
First get the total number of documents of the collection
db.collection.count()
Let's say there are 10000 documents and you want the last 1000.
To do so get the id of first document you want to dump.
db.collection.find().sort({_id:1}).skip(10000 - 1000).limit(1)
In this example the id was "50ad7bce1a3e927d690385ec".
Now you can feed mongodump with this information, to dump all documents a with higher or equal id.
$ mongodump -d 'your_database' -c 'your_collection' -q '{_id: {$gte: ObjectId("50ad7bce1a3e927d690385ec")}}'
UPDATE
The new parameters --limit and --skip were added to mongoexport will be probably available in the next version of the tool: https://github.com/mongodb/mongo/pull/307
Building off of Mic92's answer, to get the most recent 1000 items from a collection:
Find the _id of the 1000th most recent item:
db.collection.find('', {'_id':1}).sort({_id:-1}).skip(1000).limit(1)
It will be something like 50ad7bce1a3e927d690385ec.
Then pass this _id in a query to mongodump:
$ mongodump -d 'your_database' -c 'your_collection' -q '{"_id": {"$gt": {"$oid": "50ad7bce1a3e927d690385ec"}}}'
mongodump supports the --query operator. If you can specify your query as a json query, you should be able to do just that.
If not, then your trick of running a query to dump the records into a temporary collection and then dumping that will work just fine. In this case, you could automate the dump using a shell script that calls a mongo with a javascript command to do what you want and then calling mongodump.
I was playing with a similar requirement (using mongodump) where I wanted to do sequential backup and restore. I would take dump from last stored timestamp.
I couldn't get through --query '{ TIMESTAMP : { $gte : $stime, $lt : $etime } }'
Some points to note:
1) use single quote instead of double
2) do not escape $ or anything
3) replacing $stime/$etime with real numbers will make the query work
4) problem I had was with getting $stime/$etime resolved before mongodump executes itself
under -x it showed as
+ eval mongodump --query '{TIMESTAMP:{\$gte:$utc_stime,\$lt:$utc_etime}}'
++ mongodump --query '{TIMESTAMP:$gte:1366700243}' '{TIMESTAMP:$lt:1366700253}'
Hell, the problem was evident. query gets converted into two conditionals.
The solution is tricky and I got it after repeated trials....
escape { and } ie use { ..} . This fixes the problem.
try this:
NUM=10000
doc=selected_doc
taskid=$(mongo 127.0.0.1/selected_db -u username -p password --eval "db.${doc}.find({}, {_id: 1}).sort({_id: -1}).skip($NUM).limit(1)" | grep -E -o '"[0-9a-f]+"')
mongodump --collection $doc --db selected_db --host 127.0.0.1 -u username -p password -q "{_id: {\$gte: $taskid}}" --out ${doc}.dump
_id-based approaches may not work if you use a custom _id for your collection (such as returned by a 3rd party API). In that case, you should depend on a createdAt or equivalent field:
COL="collectionName"
HOW_MANY=10000
DATE_CUTOFF=$(mongo <host, user, pass...> dbname --quiet \
--eval "db.$COL.find({}, { createdAt: 1 }).sort({ createdAt: -1 }).skip($HOW_MANY).limit(1)"\
| grep -E -o '(ISODate\(.*?\))')
echo "Copying $HOW_MANY items after $DATE_CUTOFF..."
mongodump <host, user, pass...> -d dbname -c ${COL}\
-q "{ createdAt: { \$gte: $DATE_CUTOFF} }" --gzip
Strategy is simple but there are some challenges to doing that. I am assuming we are using _id field to do that. And we know _id field is incremental thus it is a good indicator to find recent documents.
Find X'th record accordingly in the collection
Extract the _id field of document
Use the _id field in mongodump --query
Find X'th record accordingly in the collection
We can achive that by using --eval with mongo tool.
Sort the documents newest to oldest
limit X record reverse sort it
take the first document (limit:1)
stringify Id
mongo --host=$mongodb_uri --quiet --eval db.myCollection.aggregate([{\$sort:{_id:-1}},{\$limit:$MAX_DOCUMENT},{\$sort:{_id:1}},{\$limit:1},{\$project:{_id:{\$toString:\"\$_id\"}}}])
result={ "_id" : "62440d84c18a957093f6c8a3" }
Extract the _id field of document
We need the exact value of _id so we do some regex
$(echo $result | sed -e 's/{ "_id" : "\(.*\)" }/\1/')
lastId=62440d84c18a957093f6c8a3
Use the _id field in mongodump --query
mongodump does not accept ObjectId so we should use $oid to indicate ObjectId fields.
query="{\"_id\":{\"\$gte\":{\"\$oid\":\"$lastId\"}}}"
Here is the complete bash script
dump()
{
local lastIdQuery="db.$collection.aggregate([{\$sort:{_id:-1}},{\$limit:$MAX_DOCUMENT},{\$sort:{_id:1}},{\$limit:1},{\$project:{_id:{\$toString:\"\$_id\"}}}])"
echo "lastIdQuery $lastIdQuery"
local lastIdResult=$(mongo --host=$mongodb_uri --quiet --eval "$lastIdQuery")
echo "lastIdResult $lastIdResult"
local lastId=$(echo $lastIdResult | sed -e 's/{ "_id" : "\(.*\)" }/\1/')
echo $lastId
query="{\"_id\":{\"\$gte\":{\"\$oid\":\"$lastId\"}}}"
echo "query $query"
mongodump --uri=$mongodb_uri --collection $collection --query="$query" --out=$outFolder
}
mongodb_uri='mongodb://localhost:27017/myDb'
outFolder=./backup
MAX_DOCUMENT=100
collection="users"
dump