I am trying to export a collection to csv which has the following fields:
_id
number
name
price
pollingTime
I can see the polling time data when I open the collection in RoboMongo or try to access the collection through mongoshell, but when I export that into a CSV, the pollingTime field comes out blank.
Here's my mongoexport command:
mongoexport --db=itemDB --collection=itemprice1 --type=csv --fieldFile=fields.txt --out items.csv
I need to send this data to some non-tech business folks; any idea if I need to make any changes in the fields.txt. Fields.txt is like this:
_id
number
name
totalPrice
pollingTimme
Apologies - I discovered immediately after while reviewing my question itself that I was making a mistake in the spelling of pollingTime field. I still want to keep the question and answer on StackOverflow so that others searching such a problem will try to look at their spellings :-)
Related
As I was importing a new json file to my mongodb collection I've accidentally use just one '-' instead of 2. Eg.:
mongoimport --host=127.0.0.1 --db=dataBaseName -collection=people --file=importFile.json
I believe that due to the lack of the second '-', now I'm stuck with the following results when I type show collections:
people
ollection=people
I can't access, drop or interact with the second one. Apart from droping the database and starting over, is there a way around this issue?
You can rename the collection like:
> use YourDatabase
// Might wanna drop people collection first
> db.getCollection("ollection=people").renameCollection("people")
Hope This helps!
I am using MongoDB at mLab. I have multiple collections - 1 main and other supporting. Therefore, the main collection consists of IDs pointing to supporting collections. I would like to export the actual data from the main collection to a CSV file. So I need to populate the data first and then export the result.
I see I can export collections individually but then the data are not populated. I suppose I should use bash script to do this but I do not know how.
Could you point me the right direction or suggest a way to do this?
Thank you!
Using the mongo shell will be the better idea in your case, as per the official documents below is the steps to write the bash script to read the data from mongo collection in bash shell scripts:
Simple example to get the data count from a collection with updated date time with greater than 10 days.
DATE2=$(date -d '10 days ago' "+%Y-%m-%dT%H:%M:%S.%3NZ");
counter = $(mongo --quiet dbName --eval 'db.dbCollection.find({"updatedAt":{"$gt":new ISODate("'$DATE'")}}).count()')
echo counter;
Or you can get the list of data and iterate over it to populate it as per your requirements.
For more on mongo shell query click here
i have a few fields in my collection at the mongoDB.
i have tried exported out everything.
which looking like this
{"_id":{"$oid":"5a5ef05dbe83813f55141a51"},"comments_data":{"id":"211","comments":{"paging":{"cursors":{"after":"WzZANVFV4TlRVME5qUXpPUT09","before":"WTI5dEF4TlRVNE1USTVNemczTXpZAMk56YzZANVFV4TlRBMU9ERTFNQT09"}},"data":[{"created_time":"2018-01-04T09:29:09+0000","message":"Super","from":{"name":"M Mun","id":"1112"},"id":"1111"},{"created_time":"2018-01-07T22:25:08+0000","message":"Happy bday..Godbless you...","from":{"name":"L1","id":"111"},"id":"1111"},{"created_time":"2018-01-10T00:22:00+0000","message":"Nelson ","from":{"name":"Boon C","id":"1111"},"id":"10111"},{"created_time":"2018-01-10T01:07:19+0000","message":"Thank to SingTel I like to","from":{"name":"Sarkar WI","id":"411653482605703"},"id":"10155812413346677_10155825869201677"}]}},"post_id":"28011986676_10155812413346677","post_message":"\"Usher in the New Year with deals and rewards that will surely perk you up, exclusively for Singtel customers. Find out more at singtel.com/rewards\"",
but now i want to export just a single field which is the 'message' from the 'comments_data' from the collection.
i tried using this mongoexport --db sDB --collection sTest --fields data.comments_data --out test88.json
but when i check my exported file, it just contains something like this
{"_id":{"$oid":"5a5ef05dbe83813f55141a51"}}
which is something not i have expected.
i just want something like "message":"Happy bday..Godbless you..."
but when i query out at the mongoshell with db.sTest.find({}, {comments_data:1, _id:0})i can roughly get what i want.
If this ...
db.sTest.find({}, {'comments_data.message':1, _id:0})
... selects the data you are interested in then the equivalent mongoexport command is:
mongoexport --db sDB --collection sTest --fields 'comments_data.message' --type csv --out test88.csv
Note: this uses --type csv because, according to the docs, use of the JSON output format causes MongoDB to export all fields in the selected sub document ...
For csv output formats, mongoexport includes only the specified field(s), and the specified field(s) can be a field within a sub-document.
For JSON output formats, mongoexport includes only the specified field(s) and the _id field, and if the specified field(s) is a field within a sub-document, the mongoexport includes the sub-document with all its fields, not just the specified field within the document.
If you must have JSON format and limit your output to a single field then I think you'll need to write the reduced documents to a separate collection and export that collection, as per this answer.
How to dump all collections by date? If my records hasn't timestamp field?
Fields: _id, name, email, carnumber... etc.
And how to look/find() in archived/dumped database?
I need to create search mechanism, for searching in archive
You can pass a query to mongodump that will make it dump only a portion of your data. If you can't make a query that finds a required portion of data, then you're out of luck.
Result of mongodump is a collection of bson files. They are not directly queryable. But you can load them into another database and query that. Or you can use mongoexport utility that creates JSON documents. JSON is a little bit easier to work with.
Although what Sergio says is broadly true, let me expand a bit:
First, You mention using _id - if that is an ObjectID (the default), then it contains a timestamp - the first 4 bytes are a unix style timestamp:
http://www.mongodb.org/display/DOCS/Object+IDs#ObjectIDs-BSONObjectIDSpecification
Next, the problem with using mongoexport is that JSON does not preserve all BSON types (http://bsonspec.org/#/specification) - BSON has more types than JSON does and so storing as JSON can be problematic unless you have rules to re-import
If you keep the data in BSON format there is the bsondump to inspect things as-is in the files:
http://www.mongodb.org/display/DOCS/Import+Export+Tools#ImportExportTools-bsondump
Or, if you had an "archive" MongoDB instance, you could just use mongodump/mongorestore, which works directly with the BSON files and does not have the JSON issues seen with mongoexport etc.:
http://www.mongodb.org/display/DOCS/Import+Export+Tools#ImportExportTools-mongodumpandmongorestore
What I want:
I have a master collection of products, I then want to filter them and put them in a separate collection.
db.masterproducts.find({category:"scuba gear"}).copyTo(db.newcollection)
Of course, I realise the 'copyTo' does not exist.
I thought I could do it with MapReduce as results are created in a new collection using the new 'out' parameter in v1.8; however this new collection is not a subset of my original collection. Or can it be if I use MapReduce correctly?
To get around it I am currently doing this:
Step 1:
/usr/local/mongodb/bin/mongodump --db database --collection masterproducts -q '{category:"scuba gear"}'
Step 2:
/usr/local/mongodb/bin/mongorestore -d database -c newcollection --drop packages.bson
My 2 step method just seems rather inefficient!
Any help greatly appreciated.
Thanks
Bob
You can iterate through your query result and save each item like this:
db.oldCollection.find(query).forEach(function(x){db.newCollection.save(x);})
You can create small server side javascript (like this one, just add filtering you want) and execute it using eval
You can use dump/restore in the way you described above
Copy collection command shoud be in mongodb soon (will be done in votes order)! See jira feature.
You should be able to create a subset with mapreduce (using 'out'). The problem is mapreduce has a special output format so your documents are going to be transformed (there is a JIRA ticket to add support for another format, but I can not find it at the moment). It is also going to be very inefficent :/
Copying a cursor to a collection makes a lot of sense, I suggest creating a ticket for this.
there is also toArray() method which can be used:
//create new collection
db.creatCollection("resultCollection")
// now query for type="foo" and insert the results into new collection
db.resultCollection.insert( (db.orginialCollection.find({type:'foo'}).toArray())