to print nested json array values into csv using MongoDB - mongodb

I want to output the nested json array into csv.
sample.json
{
"DocId":"ABC",
"User":[
{
"Id":1234,
"Username":"sam1234",
"Name":"Sam",
"ShippingAddress":{
"Address1":"123 Main St.",
"Address2":null,
"City":"Durham",
"State":"NC"
}
},
{
"Id":5678,
"Username":"sam5678",
"Name":"Sam",
"ShippingAddress":{
"Address1":"5678 Main St.",
"Address2":null,
"City":"Durham",
"State":"NC"
}
}
]
}
enter code here
Above is the sample file, DocID must not be printed, and output in csv must be only for array contents
Id Username Name ShippingAddress
1234 sam1234 Sam 123 Main St.Durham NC
5678 sam5678 Sam 5678 Main St.Durham NC
How to print with headers, and with out headers in csv

One way to do is to do it in two steps
Perform aggregation on this collection and change the structure of the collection docs and output them in another collection
Use mongoexport to export the collection created in step 1 as CSV [This step can be used directly ^-^].
For step 1, Lets say I have db -> test and collection -> stack, so aggregation query is:
db.stack.aggregate([
{ $unwind:"$User"},
{ $project : { Id : "$User.Id" , Username:"$User.Username", Name:"$User.Name", ShippingAddress:"$User.ShippingAddress", _id:0} },
{ $out: "result" }
])
For step 2, use mongoexport terminal utility:
mongoexport --db test --collection result --csv --fields "Id,Username,Name,ShippingAddress" --out file.csv

Related

Can't get lists of sorted documents based on _id

Not able to backup mongo documents with command below.
Error:
Failed: error parsing query as Extended JSON: invalid JSON input. Position: 15. Character: O
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt": ObjectId("5e25b7a5f4c9b92aaa8a4131")}}'
ObjectId method returns a new ObjectId value which is not the ID value you passed in. The new ObjectId is invalid since is not part of JSON object.
Based on this documentation, you can passed your ID value which is 5e25b7a5f4c9b92aaa8a4131 to the nested query $oid (object id).
Query
{
"_id": {
"$lt": {
"$oid": "5e25b7a5f4c9b92aaa8a4131"
}
}
}
Final Solution
mongodump --db test-data --collection foo --out=dump --query '{"_id":{"$lt":{"$oid": "5e25b7a5f4c9b92aaa8a4131"}}}'
You will have to use MongoDB Extended JSON for this parse as json.
Your query becomes {"_id" : {"$lt" : {"$oid" : "5e25b7a5f4c9b92aaa8a4131"}}}

How do I export this MongoDB data into a CSV file (a "join" of two collections)?

I have a database with two collections.
Collection cars has documents that look like this:
{ license_number: "123456", name: "tesla" }
{ license_number: "654321", name: "ford" }
{ license_number: "987654", name: "volvo" }
Collection greatCars has documents that look like this:
{ license_number: "123456" }
I want to export a "join" of these two collections into a CSV file that looks like this:
license_number, name, isGreat
123456, tesla, TRUE
654321, ford, FALSE
987654, volvo, FALSE
(Please don't take offense. It's just an example and I know nothing about cars.)
You can create a short javascript program that will transfer the data you want to export into a new temporary collection, that can be exported.
create a file export.js:
//initialise the export results collection
db.export.results.drop()`;
//create a cursor containing the contents of the list1 collection
cursor = db.list1.find();
while (cursor.hasNext()) {
doc = cursor.next();
//Check if the document exists in the list2 collection
list2 = db.list2.find({"<id_fieldname>": doc.<id_fieldname>});
if (list2.hasNext()) {
//if it does exist, add the document from list1 to the new export collection
db.export.results.insert(doc);
}
}
print(db.export.results.count() + " matching documents found");
Run this from the cmd line:
# mongo "localhost:27017/<dbname>" export.js
It will create a collection called export.results containing the document from the list1 collection with documents in the list2 (in your case cars and greatCars)collection with a matching id field. You can then export or dump this collection:
# mongoexport --db <dbname> -c export.results -type csv -o <file_name>

Save a result from a mongodb query

I write this in the shell of mongo (line by line). When I write "documentos", the results are displayed in the shell of mongo.
var ZZZ = db.mycollection.distinct( 'name', { event: 'a' } );
var documentos = db.mycollection.find({ nombre: { $in: ZZZ }});
documentos
How can I to create a collection from that query? or.. "save" or "export" the result?
Thx!
Execute the below script in the shell
var ZZZ = db.mycollection.distinct("name", {"event": "a"} );
db.mycollection.find({"nombre":{$in:ZZZ}}).forEach(
function(doc){db.newcollection.insert(doc);}
)
It will iterate each document and insert the documents into a new collection.
Querying the companies collection and inserting the result in new collection.
newCollection doesn't exists.
Here is how mongo shell looks like
x = db.companies.findOne();
db.newCollection.insert(x);
WriteResult({ "nInserted" : 1 })
You can export using
mongoexport --db companies --collection companies --csv --fieldFile D:\fields.txt --out D:\company.csv
where fields.txt contains contains a line-separated list of fields to export.
Modified
> cities = db.zips.distinct("city", {pop:{$gt: 81000}} ); n = db.zips.findOne({c
ity :{$in: cities}});db.newCollection.insert(n);
WriteResult({ "nInserted" : 1 })

Mongodb MapReduce - Mongoexport missing column values

Very new to Mongodb and MapReduce, but loving what I can do with it and getting on mostly ok. I am however having trouble exporting the results of a MapReduce to CSV.
Here's what i'm doing:
var mapFunction1 = function() {
emit({isrc: this.isrc, country: this.country}, this.amount_payable);
};
var reduceFunction1 = function(keyIsrc, valuesAmountPayable) {
return Array.sum(valuesAmountPayable);
};
db.sales.mapReduce(
mapFunction1,
reduceFunction1,
{ out: "sales_with_total_by_country_and_isrc" }
)
db.sales_with_total_by_country_and_isrc.find()
When I run the find above I can see the results I want, ISRCs, Countrys and Values are all present.
I'm running my export as follows and as I say, this runs ok, all columns are present but I only have values in the value column, not ISRC or Country.
mongoexport --csv -d test -c sales_with_total_by_country_and_isrc -q '{value: {$ne: 0}}' -f "isrc","country","value" -o sales_with_total_by_country_and_isrc.csv
What am I doing wrong? As far as I can tell i'm passing the fields I want to the export with -f in the correct way.
Thanks in advance.
Ok, I sussed this out myself.
As, the resulting collection looked like:
{ "_id" : { "isrc" : "", "country" : "AE" }, "value" : 0.391081 }
I just needed to call -f "_id.isrc","_id.country","value" in my mongoexport call.
Hope this can help some others.

MongoDb/MongoVue Export part of a composite key

I wanted to use MongoVue to create a quick export of all the ids for portion of a given collection.
I have a document that has an _id field which is a composite key.
For example.
{
"_id" : {
"GroupID" : 3,
"ThingyID" : 320486
},
"HowManyOwned" : 42,
"IsAwesome" : true
}
I want to create an export of all the ThingyIDs for Group 3.
Of course if I make my query something like this.
db.GroupThingy.find({ "_id.GroupID" : 3 }, { "_id.ThingyID" : 1 })
I'll get back all the composite keys. I wanted to use MongoVue to just quickly create this export. If I use that query I get back an export of.
Document[2 Keys]
Document[2 Keys]
Document[2 Keys]
What I was hoping to get was
either
3,12345
3,3838
3,3777
3,1111
Or even better would be just
12345
3838
3777
1111
I could write a program for this but there has to be a quick way to accomplish this that I just don't know about.
Aggregation Framework doesn't help me get a csv export... but something like this will only support 20k documents
db.GroupThingy.group(
{
key: {
"_id.ThingyID": 1,
"_id.GroupID":1
},
cond: { "_id.GroupID": 3 },
reduce: function(curr, result){
result.ThingyID2 = curr._id.ThingyID
},
initial: { "ThingyID2": 0 }
});
Well I finally found out how to get the export using mongoexport thanks to this question:
how to export collection to csv in mongodb
However I kept getting the error:
ERROR: too many positional options... and this post helped me find the answer:
What does "too many positional options" mean when doing a mongoexport?
So my final solution to run the mongoexport was:
mongoexport
--host myHostName
--db theDB
--collection GroupThingy
--fields "_id.ThingyID"
--csv
--query "{'_id.GroupID':3}"
(options on there own lines for readability)