I wanted to use MongoVue to create a quick export of all the ids for portion of a given collection.
I have a document that has an _id field which is a composite key.
For example.
{
"_id" : {
"GroupID" : 3,
"ThingyID" : 320486
},
"HowManyOwned" : 42,
"IsAwesome" : true
}
I want to create an export of all the ThingyIDs for Group 3.
Of course if I make my query something like this.
db.GroupThingy.find({ "_id.GroupID" : 3 }, { "_id.ThingyID" : 1 })
I'll get back all the composite keys. I wanted to use MongoVue to just quickly create this export. If I use that query I get back an export of.
Document[2 Keys]
Document[2 Keys]
Document[2 Keys]
What I was hoping to get was
either
3,12345
3,3838
3,3777
3,1111
Or even better would be just
12345
3838
3777
1111
I could write a program for this but there has to be a quick way to accomplish this that I just don't know about.
Aggregation Framework doesn't help me get a csv export... but something like this will only support 20k documents
db.GroupThingy.group(
{
key: {
"_id.ThingyID": 1,
"_id.GroupID":1
},
cond: { "_id.GroupID": 3 },
reduce: function(curr, result){
result.ThingyID2 = curr._id.ThingyID
},
initial: { "ThingyID2": 0 }
});
Well I finally found out how to get the export using mongoexport thanks to this question:
how to export collection to csv in mongodb
However I kept getting the error:
ERROR: too many positional options... and this post helped me find the answer:
What does "too many positional options" mean when doing a mongoexport?
So my final solution to run the mongoexport was:
mongoexport
--host myHostName
--db theDB
--collection GroupThingy
--fields "_id.ThingyID"
--csv
--query "{'_id.GroupID':3}"
(options on there own lines for readability)
Related
Not able to backup mongo documents with command below.
Error:
Failed: error parsing query as Extended JSON: invalid JSON input. Position: 15. Character: O
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt": ObjectId("5e25b7a5f4c9b92aaa8a4131")}}'
ObjectId method returns a new ObjectId value which is not the ID value you passed in. The new ObjectId is invalid since is not part of JSON object.
Based on this documentation, you can passed your ID value which is 5e25b7a5f4c9b92aaa8a4131 to the nested query $oid (object id).
Query
{
"_id": {
"$lt": {
"$oid": "5e25b7a5f4c9b92aaa8a4131"
}
}
}
Final Solution
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt":{"$oid": "5e25b7a5f4c9b92aaa8a4131"}}}'
You will have to use MongoDB Extended JSON for this parse as json.
Your query becomes {"_id" : {"$lt" : {"$oid" : "5e25b7a5f4c9b92aaa8a4131"}}}
I'm trying to export phone numbers from a collection. Below is the sample document
{ "_id" : ObjectId("5ad5cf864717256ff02b4923"),"userName":"9619324746", "firstName" : "D H", "contactPhone" : 9619324746}
The export command that I used is below
mongoexport --db dbname --collection accounts --type=json --out accounts.json --fields contactPhone,userName
And the contents of JSON looks like below
{"_id":{"$oid":"5ad5cf864717256ff02b4923"},"userName":"9619324746","contactPhone":9.619324746e+09}
Can somebody help me to get the contactPhone value not converted? Thank you.
-Srini
If mongoexport exported 123 as 123.0, then 123 was a Double type in the document. You should try inserting the value as a 32- or 64-bit integer
db.collection.insert({
"tweetId" : NumberLong(1234567)
})
mongoexport exports JSON, using strict mode JSON representation, which inserts some type information into the JSON so MongoDB JSON parsers (like mongoimport) can reproduce the correct BSON data types while the exported JSON still conforms to the JSON standard
{ "tweetId" : { "$numberLong" : "1234567" } }
To preserve all the type information, use mongodump/mongorestore instead. To export all field values as strings, you'll need to write your own script with a driver that fetches each doc and stringifies all the values.
I can take mongodb data backup but I am not sure about mongodb schama backup.
Is there any way to take dump of MONGODB schema only not the data ?
You need to use mongorestore... which is used for things like importing json, or csv, etc.
You can read more about mongorestore in the docs below; I'd take a look and read up on them as they are very helpful.
http://www.mongodb.org/display/DOCS/Import+Export+Tools#ImportExportTools-mongorestore
You can also check out http://learnmongo.com for tips and help!
or you can visit the links
How to use the dumped data by mongodump? hope this may be helpful for you.
MongoDB is an NoSQL Database.
There is no fixed schema for any collection, so there are no functions available in mongo shell to find the collection schema.
Fixed Schema is applicable for RDBMS databases. In NoSQL DB, such as mongodb it is not required, but you can enforce same schema using your implementation logic, if required.
A document in a same collection, can be of different schema's. Please see example below
db.mycollection.insert([
{ "_id":1, "name":"A"},
{ "_id":2, "name":"CD", "age":29},
{ "_id":3, "name":"AB", "age":28},
{ "_id":4, "name":"ABC", "age":27, "emailId":"abc#xyz.com"},
{ "_id":5, "name":"ABCD", "age":29, "emailId":"abcd#xyz.com"}]);
db.mycollection.find();
{ "_id" : 1, "name" : "A" }
{ "_id" : 2, "name" : "CD", "age" : 29 }
{ "_id" : 3, "name" : "AB", "age" : 28 }
{ "_id" : 4, "name" : "ABC", "age" : 27, "emailId" : "abc#xyz.com" }
{ "_id" : 5, "name" : "ABCD", "age" : 29, "emailId" : "abcd#xyz.com" }
An approach to find the schema
In Mongo Shell
var k = db.mycollection.findOne();
for ( i in k){print (i)};
_id
name
this approach will work for you if all the documents in your collection follows the same schema.
Here's how I did it:
mongodump --uri="mongodb://localhost/mydb" -o ./mydb-dump
find ./mydb-dump -name *.bson -exec truncate -s 0 {} \;
Explanation: I'm dumping the whole database, then truncating all the .bson files (which hold collection data) to zero bytes.
Limitation: Obviously, this is only practical if the source database is small, otherwise you're generating a huge data dump only to throw away most of it.
To restore this-
mongorestore --uri="mongodb://some-other-server/mydb" ./mydb-dump
If there's a better way to do this, I'd love to know what it is!
MongoDB Compass GUI has a way to export the schema to JSON.
At the time of this post, there doesn't seem to be a way to do this by bulk, so this will have to be done for each collection one by one.
From the docs:
You can export your schema after analyzing it. This is useful for
sharing your schema and comparing schemas across collections.
If you have not already done so, analyze your schema:
Select your desired collection and click the Schema tab. Click
Analyze Schema.
Once your schema has been analyzed, export your schema:
In the top menu bar, click Collection. From the dropdown, click Share
Schema as JSON.
Your schema is copied to your clipboard as a JSON object.
See full docs here ~ https://www.mongodb.com/docs/compass/master/schema/export/
I want to output the nested json array into csv.
sample.json
{
"DocId":"ABC",
"User":[
{
"Id":1234,
"Username":"sam1234",
"Name":"Sam",
"ShippingAddress":{
"Address1":"123 Main St.",
"Address2":null,
"City":"Durham",
"State":"NC"
}
},
{
"Id":5678,
"Username":"sam5678",
"Name":"Sam",
"ShippingAddress":{
"Address1":"5678 Main St.",
"Address2":null,
"City":"Durham",
"State":"NC"
}
}
]
}
enter code here
Above is the sample file, DocID must not be printed, and output in csv must be only for array contents
Id Username Name ShippingAddress
1234 sam1234 Sam 123 Main St.Durham NC
5678 sam5678 Sam 5678 Main St.Durham NC
How to print with headers, and with out headers in csv
One way to do is to do it in two steps
Perform aggregation on this collection and change the structure of the collection docs and output them in another collection
Use mongoexport to export the collection created in step 1 as CSV [This step can be used directly ^-^].
For step 1, Lets say I have db -> test and collection -> stack, so aggregation query is:
db.stack.aggregate([
{ $unwind:"$User"},
{ $project : { Id : "$User.Id" , Username:"$User.Username", Name:"$User.Name", ShippingAddress:"$User.ShippingAddress", _id:0} },
{ $out: "result" }
])
For step 2, use mongoexport terminal utility:
mongoexport --db test --collection result --csv --fields "Id,Username,Name,ShippingAddress" --out file.csv
Very new to Mongodb and MapReduce, but loving what I can do with it and getting on mostly ok. I am however having trouble exporting the results of a MapReduce to CSV.
Here's what i'm doing:
var mapFunction1 = function() {
emit({isrc: this.isrc, country: this.country}, this.amount_payable);
};
var reduceFunction1 = function(keyIsrc, valuesAmountPayable) {
return Array.sum(valuesAmountPayable);
};
db.sales.mapReduce(
mapFunction1,
reduceFunction1,
{ out: "sales_with_total_by_country_and_isrc" }
)
db.sales_with_total_by_country_and_isrc.find()
When I run the find above I can see the results I want, ISRCs, Countrys and Values are all present.
I'm running my export as follows and as I say, this runs ok, all columns are present but I only have values in the value column, not ISRC or Country.
mongoexport --csv -d test -c sales_with_total_by_country_and_isrc -q '{value: {$ne: 0}}' -f "isrc","country","value" -o sales_with_total_by_country_and_isrc.csv
What am I doing wrong? As far as I can tell i'm passing the fields I want to the export with -f in the correct way.
Thanks in advance.
Ok, I sussed this out myself.
As, the resulting collection looked like:
{ "_id" : { "isrc" : "", "country" : "AE" }, "value" : 0.391081 }
I just needed to call -f "_id.isrc","_id.country","value" in my mongoexport call.
Hope this can help some others.