Related
For example, I have 6 items in collection
{ _id: 1, list: ["A", "B"] }
{ _id: 2, list: ["C", "A"] }
{ _id: 3, list: ["E", "F"] }
{ _id: 4, list: ["E", "D"] }
{ _id: 5, list: ["U", "I"] }
{ _id: 6, list: ["D", "K"] }
I would do a query to merge all the items which its list have at least 1 element matches. So the result will be:
{ _id: 7, list: ["A", "B", "C"] }
{ _id: 8, list: ["E", "F", "D", "K"] }
I'm new to MongoDB so anyone help me for this query ? Thanks alot.
I found this solution which almost solves your problem.
db.lists.aggregate([
{$unwind:"$list"},
{$group:{_id:"$list", merged:{$addToSet:"$_id"}, size:{$sum:1}}},
{$match:{size: {$gt: 1}}},
{$project:{_id: 1, merged:1, size: 1, merged1: "$merged"}},
{$unwind:"$merged"},
{$unwind:"$merged1"},
{$group:{_id:"$merged", letter:{$first:"$_id"}, size:{$sum: 1}, set: {$addToSet:"$merged1"}}},
{$sort:{size:1}},
{$group:{_id: "$letter", mergedIds:{$last:"$set"}, size:{$sum:1}}},
{$match: {size:{$gt:1}}}
])
I have tested this in my mongo shell which gives the following output:
{ "_id" : "E", "matchedIds" : [ 6, 3, 4 ], "size" : 2 }
{ "_id" : "A", "matchedIds" : [ 1, 2 ], "size" : 2 }
The matchedIds represents the docs id-s which have common value in the list array.
I think in the above aggregation can be done some optimization, but initially I found this, will try to find other ways. In addition you can use $lookup aggregation at the end of aggregation pipline to match the id-s with the set values. I couldn't test this because my mongo version doesn't support $lookup. But you can manually get that values inside some for loop if you use Node.js or something else.
Edited
This algorithm will only work if the amount of intersected lists for each list is no more than 3.
For example this will work:
{ "_id" : 1, "list" : [ "A", "B" ] }
{ "_id" : 2, "list" : [ "C", "A" ] }
{ "_id" : 3, "list" : [ "E", "F" ] }
{ "_id" : 4, "list" : [ "E", "D" ] }
{ "_id" : 5, "list" : [ "U", "I" ] }
{ "_id" : 6, "list" : [ "D", "K" ] }
{ "_id" : 7, "list" : [ "A", "L" ] }
but this will not:
{ "_id" : 1, "list" : [ "A", "B" ] }
{ "_id" : 2, "list" : [ "C", "A" ] }
{ "_id" : 3, "list" : [ "E", "F" ] }
{ "_id" : 4, "list" : [ "E", "D" ] }
{ "_id" : 5, "list" : [ "U", "I" ] }
{ "_id" : 6, "list" : [ "D", "K" ] }
{ "_id" : 7, "list" : [ "L", "K" ] }
Here the lists with ids of 7, 6, 4, 3 has intersection, so the number of intersected lists is 4, in this case the provided algorithm will not work. It will work only if the amount of intersection is less than 4 for each list
Final notice
It seems you can't achieve to your desired result by doing merge computation in the mongo database layer. If you are building an application then it will be better to do computation also in the application layer.
Im working on balancing some text over multiple lines and I am adding manual newLines to split the text evenly over the lines. To figure it out i need to see where the "\n" are added for testing my code, but the only way ive found is to print it all out as character:
["T", "i", "r", "s", "d", "a", "g", " ", "f", "i", "k", "k", " ", "1", "5", " ", "b", "a", "r", "n", " ", "i", " ", "G", "r", "a", "n", "s", "t", "u", "b", "b", "e", "n", " ", "b", "a", "r", "n", "e", "h", "a", "g", "e", " ", "h", "a", "g", "e", "t", " ", "s", "e", "g", " ", "b", "a", "m", " ", "b", "a", "m", " ", "s", "h", "a", "r", "k", " ", "w", "e", "e", "k", " ", "f", "o", "r", " ", "l", "i", "f", "e", " ", "a", "n", "d", " ", "f", "o", "r", " ", "e", "v", "e", "r"]
Its a pain to find new \n symbols in here and I am now wondering if i could see it as a string in any way. Any suggestions?
you can split the string by newline and join it with any visible character you like (in the example with ~~~)
myString.components(separatedBy: CharacterSet.newlines).joined(separator: "~~~")
Because '\' is an escaped charecter so you can print it like this
print("something \\n something")
I have the following data in mongoDB
{ "_id" : "ee477d7a-7a0c-4420-b476-c402012c74a9",
"_class" : "com.TrackingData",
"modified" : ISODate("2014-07-09T20:23:33.117Z"),
"eventtype" : "Test",
"eventdata" : "{\"QueryDate\":\"01-APR-2014\",
\"SearchQuery\":{\
"keyword\":\"Java\",\
"location\":\"Santa Clara, CA\",
\"Facet\":\"skill~java~perl|workAuth~USC\",
\"SearchAgentId\":\"4299\"
},
\"Viewed\":[
{\"ViewedID\":\"8992ade400a\",
\"Dockey\":\"3323aba3233\",
\"PID\":\"32399a\",
\"actionsTaken\":\"email|direct message|report seeker\",
\"viewDate\":\"01-APR-2014\",
\"MessageSent\":\"true\",
\"Message\":[
{\"MessageID\":\"123aca323\",
\"Delivered\":\"True\",
\"Opened\":\"True\",
\"ClickThroughRate\":\"NotBad\",
\"MessageDate\":\"02-APR-2014\",
\"Response\":[{\"ResponseId\":\"a323a9da\",\"ResponseDate\":\"23-APR-2014\"}]}]}]}",
"eventsource" : "API-Dev Test - JMachine",
"sourceip" : "myIp",
"entityid" : "Test",
"groupid" : "ice",
"datecreated" : ISODate("2014-07-09T20:23:33.112Z") }
evendata is stored as a String I have tried to convert this to an object through the mongo shell:
db.mydb.find( { 'eventdata' : { $type : 2 } } ).forEach( function (x) { x.eventdata = new Object(x.eventdata); db.mydb.save(x);});
However is simply seemed to split these into a map:
"eventdata" : { "0" : "{", "1" : "\"", "2" : "V", "3" : "i", "4" : "e", "5" : "w", "6" : "e", "7" : "d", "8" : "\"", "9" : ":", "10" : "[", "11" : "{", "12" : "\"", "13" : "V", "14" : "i", "15" : "e", "16" : "w", "17" : "e", "18" : "d", "19" : "I", "20" : "D", "21" : "\"", "22" : ":", "23" : "\"", "24" : "8", "25" : "9", "26" : "9", "27" : "2", "28" : "a", "29" : "d", "30" : "e", "31" : "4", "32" : "0", "33" : "0", "34" : "a", "35" : "\"", "36" : ",", "37" : "\"", "38" : "D", "39" : "o", "40" : "c", "41" : "k", "42" : "e", "43" : "y", "44" : "\"", "45" : ":", "46" : "\"", "47" : "1", "48" : "7", "49" : "2", "50" : "9", "51" : "f", "52" : "7", "53" : "a", "54" : "f", "55" : "c", "56" : "d", "57" : "1", "58" : "f", "59" : "7", "60" : "6", "61" : "3", "62" : "9", "63" : "3", "64" : "a", "65" : "b", "66" : "6", "67" : "6", "68" : "d", "69" : "5", "70" : "c", "71" : "6", "72" : "4", "73" : "8", "74" : "a", "75" : "f", "76" : "3", "77" : "7", "78" : "b", "79" : "\"", "80" : ",", "81" : "\"", "82" : "P", "83" : "I", "84" : "D", "85" : "\"", "86" : ":", "87" : "\"", "88" : "\"", "89" : ",", "90" : "\"", "91" : "a", "92" : "c", "93" : "t", "94" : "i", "95" : "o", "96" : "n", "97" : "s", "98" : "T", "99" : "a", "100" : "k", "101" : "e", "102" : "n", "103" : "\"", "104" : ":", "105" : "\"", "106" : "\"", "107" : ",", "108" : "\"", "109" : "v", "110" : "i", "111" : "e", "112" : "w", "113" : "D", "114" : "a", "115" : "t", "116" : "e", "117" : "\"", "118" : ":", "119" : "\"", "120" : "0", "121" : "9", "122" : "-", "123" : "J", "124" : "U", "125" : "L", "126" : "-", "127" : "2", "128" : "0", "129" : "1", "130" : "4", "131" : " ", "132" : "2", "133" : "0", "134" : ":", "135" : "3", "136" : "1", "137" : ":", "138" : "2", "139" : "3", "140" : "\"", "141" : "}", "142" : "]", "143" : "}" },
Which still does not recongised my nested queries such as:
db.mydb.find({'eventdata.SearchQuery.keyword' :'keywordValue' }).skip(0).limit(20)
So given my original data structure. What need to be done in terms of transformation to allow me to drill down into this eventdata attribute.
In your example with forEach use JSON.parse(x.eventdata) instead of new Object(x.eventdata).
Of course you cannot query it directly with .find() - it's just string from the MongoDB point of view. You should have store it as BSON if you wanted to query it... All you can use is documented here: http://docs.mongodb.org/manual/reference/operator/query/ After the transformation from JSON to native BSON (the forEach example) it will be possible to query it.
How do I get a clean insert without extraneous characters being added when trying to insert an array of object. If I manually do an insert from mongodb shell I get the expected results, otherwise it doesn't seem to work.
What I'm trying to achieve is the results from mongodb shell:
db.test.insert([{name:"john"},{name:"jane"}]);
which yields:
db.test.find()
{ "_id" : ObjectId("53bb0768dc2469c1f440a3c2"), "name" : "john" }
{ "_id" : ObjectId("53bb0768dc2469c1f440a3c3"), "name" : "jane" }
But I don't get that, so I used the code snippet below to test several ways to insert the array of objects hoping to find the right combination:
test = new Meteor.Collection("test");
a = new Array();
a.push({name:"john"});
a.push({name:"jane"});
console.log(a);
test.insert(a);
console.log(a.toString());
test.insert(a.toString());
console.log(JSON.stringify(a));
test.insert(JSON.stringify(a));
test.insert([{name:"john"},{name:"jane"}]);
test.insert([{"name":"john"},{"name":"jane"}]);
What I get in the console:
[ { name: 'john' }, { name: 'jane' } ]
[object Object],[object Object]
[{"name":"john"},{"name":"jane"}]
What I get in the database:
db.test.find()
{ "0" : { "name" : "john" }, "1" : { "name" : "jane" }, "_id" : "SYkv79XLNQsWgkYmw" }
{ "0" : "[", "1" : "o", "2" : "b", "3" : "j", "4" : "e", "5" : "c", "6" : "t", "7" : " ", "8" : "O", "9" : "b", "10" : "j", "11" : "e", "12" : "c", "13" : "t", "14" : "]", "15" : ",", "16" : "[", "17" : "o", "18" : "b", "19" : "j", "20" : "e", "21" : "c", "22" : "t", "23" : " ", "24" : "O", "25" : "b", "26" : "j", "27" : "e", "28" : "c", "29" : "t", "30" : "]", "_id" : "SiQ3ZpGfeBqj4mXB2" }
{ "0" : "[", "1" : "{", "2" : "\"", "3" : "n", "4" : "a", "5" : "m", "6" : "e", "7" : "\"", "8" : ":", "9" : "\"", "10" : "j", "11" : "o", "12" : "h", "13" : "n", "14" : "\"", "15" : "}", "16" : ",", "17" : "{", "18" : "\"", "19" : "n", "20" : "a", "21" : "m", "22" : "e", "23" : "\"", "24" : ":", "25" : "\"", "26" : "j", "27" : "a", "28" : "n", "29" : "e", "30" : "\"", "31" : "}", "32" : "]", "_id" : "kKRiR8NjNJefBYRya" }
{ "0" : { "name" : "john" }, "1" : { "name" : "jane" }, "_id" : "RBrvkrw5xZaEGdczF" }
{ "0" : { "name" : "john" }, "1" : { "name" : "jane" }, "_id" : "2cfWJqHY4aJ6yF68s" }
I expected a simple 'test.insert(a)' to give me what I want, but it includes the array indexes. How do I build an array of objects to insert into mongodb from meteor without the array indexes? Stringify seemed to build a clean looking serialization of the array, but apparently I just don't know how to do this? The purpose of this is so I can build a complex array of objects in memory and do a bulk insert.
Meteor only lets you store root level documents as objects, if you give it an array it will try to convert it to an object. This is why you're getting this weird result. You would have to adjust your document to store arrays as part of the root document
test = new Meteor.Collection("test");
a = new Array();
a.push({name:"john"});
a.push({name:"jane"});
var doc = {
names: a
}
test.insert(a);
It won't be possible to store a document as [].
Template.first.rendered=function(){
var a=[];
a.push({name:"rahul"});
a.push({name:"vidu"});
Meteor.call("array", a , function(error,result){
});
};
I have a collection of entities, which represents a tree. Each entity has a property containing an array of attributes.
For example:
{
"_id" : 1,
"parent_id" : null,
"attributes" : [ "A", "B", "C" ]
}
I would like to use MapReduce to generate another collection which is similar to the original collection, but for each item in the collection it not only contains the attributes directly associated with the entity, but also those of its ancestors, all the way up to the root of the hiearchy.
So given the following entities:
{
"_id" : 1,
"parent_id" : null,
"attributes" : [ "A", "B", "C" ]
}
{
"_id" : 2,
"parent_id" : 1,
"attributes" : [ "D", "E", "F" ]
}
{
"_id" : 3,
"parent_id" : 2,
"attributes" : [ "G", "H", "I" ]
}
The result of the MapReduce job would be the following:
{
"_id" : 1,
"attributes" : [ "A", "B", "C" ]
}
{
"_id" : 2,
"attributes" : [ "A", "B", "C", "D", "E", "F" ]
}
{
"_id" : 3,
"attributes" : [ "A", "B", "C", "D", "E", "F", "G", "H", "I" ]
}
I've managed produce MapReduce jobs which do simple things like count the attributes for each entity but can't get my head round how I might deal with a hierarchy. I am open to alternative ways of storing the data but don't want to store the whole hierarchy in a single document.
Is this kind of thin possible with MapReduce in MongoDB or am I just thinking about the problem in the wrong way?
Ok, so I don't think this will be very performant/scalable, because you have to recursively find the parent ids from the child nodes. However, it does provide the output you want.
var mapFunc = function(doc, id) {
// if this is being invoked by mapReduce, it won't pass any parameters
if(doc == null) {
doc = this;
id = this._id;
} else if (doc.parent_id != null) {
// if this is a recursive call, find the parent
doc = db.test.findOne({_id:doc.parent_id});
}
// emit the id, which is always the id of the child node (starting point), and the attributes
emit(id, {attributes: doc.attributes});
// if parent_id is not null, call mapFunc with the hidden parameters
if(doc.parent_id != null) {
// recursive mapFunc call
mapFunc(doc, id);
}
}
// since we're going to call this from within mapReduce recursively, we have to save it in the system JS
db.system.js.save({ "_id" : "mapFunc", "value" : mapFunc});
var reduceFunc = function(key, values) {
var result = {attributes:[]};
values.forEach(function(value) {
// concat the result to the new values (I don't think order is guaranteed here)
result.attributes = value.attributes.concat(result.attributes);
});
return result;
}
// this just moves the attributes up a level
var finalize = function(key, value) {return value.attributes};
// quick test...
db.test.mapReduce(mapFunc, reduceFunc, {out: {inline: 1}, finalize: finalize});
Provides:
"results" : [
{
"_id" : 1,
"value" : [
"A",
"B",
"C"
]
},
{
"_id" : 2,
"value" : [
"A",
"B",
"C",
"D",
"E",
"F"
]
},
{
"_id" : 3,
"value" : [
"A",
"B",
"C",
"D",
"E",
"F",
"G",
"H",
"I"
]
}
],
"timeMillis" : 2,
"counts" : {
"input" : 3,
"emit" : 6,
"reduce" : 2,
"output" : 3
},
"ok" : 1,
}