Meteor: Speeding up MongoDB Join's for Big Data? - mongodb

Question

I have two collections: Data and Users. In the Data collection, there is an array of user IDs consisting of approximately 300 to 800 users.
I need to join together the countries of all of the users for each row in the Data collection, which hangs my web browser due to there being too much data being queried at once.
I query for about 16 rows of the Data collection at once, and also there are 18833 users so far in the Users collection.
So far I have tried to make both a Meteor method and a transform() JOIN for the Meteor collection which is what's hanging my app.
Mongo Collection:
UserInfo = new Mongo.Collection("userInfo")
GlyphInfo = new Mongo.Collection("GlyphAllinOne", {
transform: function(doc) {
doc.peopleInfo = doc.peopleInfo.forEach(function(person) {
person.code3 = UserInfo.findOne({userId: person.name}).code3;
return person;
})
return doc;
}
});
'code3' designates user's country.
Publication:
Meteor.publish("glyphInfo", function (courseId) {
this.unblock();
var query = {};
if (courseId) query.courseId = courseId;
return [GlyphInfo.find(query), UserInfo.find({})];
})
Tested Server Method:
Meteor.methods({
'glyph.countryDistribution': function(courseId) {
var query = {};
if (courseId) query.courseId = courseId;
var glyphs = _.map(_.pluck(GlyphInfo.find(query).fetch(), 'peopleInfo'), function(glyph) {
_.map(glyph, function(user) {
var data = Users.findOne({userId: user.name});
if (data) {
user.country = data ? data.code3 : null;
console.log(user.country)
return user;
}
});
return glyph;
});
return glyphs;
}
});
Collection Data:
There is an option of preprocessing my collection so that countries would already be included, however I'm not allowed to modify these collections. I presume that having this JOIN be done on startup of the server and thereafter exposed through an array as a Meteor method may stall startup time of the server for far too long; though I'm not sure.
Does anyone have any ideas on how to speed up this query?
EDIT: Tried out MongoDB aggregation commands as well and it appears to be extremely slow on Meteor's minimongo. Took 4 minutes to query in comparison to 1 second on a native MongoDB client.
var codes = GlyphInfo.aggregate([
{$unwind: "$peopleInfo"},
{$lookup: {
from: "users",
localField: "peopleInfo.name",
foreignField: "userId",
as: "details"
}
},
{$unwind: "$details"},
{$project: {"peopleInfo.Count": 1, "details.code3": 1}}
])

Answer 1

I would approach this problem slightly differently, using reywood:publish-composite
On the server I would publish the glyphinfo using publish-composite and then include the related user and their country field in the publication
I would join up the country on the client wherever I needed to display the country name along with the glyphinfo object
Publication:
Meteor.publishComposite('glyphInfo', function(courseId) {
this.unblock();
return {
find: function() {
var query = {};
if (courseId) query.courseId = courseId;
return GlyphInfo.find(query);
},
children: [
{
find: function(glyph) {
var nameArray = [];
glyph.person.forEach(function(person){
nameArray.push(person.name);
};
return UserInfo.find({ userId: {$in: nameArray }});
}
]
}
});

Answer 2

Solved the problem by creating a huge MongoDB aggregation call, with the biggest factor in solving latency being indexing unique columns in your database.
After carefully implementing indices into my database with over 4.6 million entries, it took 0.3 seconds on Robomongo and 1.4 seconds w/ sending data to the client on Meteor.
Here is the aggregation code for those who'd like to see it:
Meteor.methods({
'course.countryDistribution': function (courseId, videoId) {
var query = {};
if (courseId) query.courseId = courseId;
var data = GlyphInfo.aggregate([
{$unwind: "$peopleInfo"},
{$lookup: {
from: "users",
localField: "peopleInfo.name",
foreignField: "userId",
as: "details"
}
},
{$unwind: "$details"},
{$project: {"peopleInfo.Count": 1, "details.code3": 1}},
{$group: {_id: "$details.code3", count: {$sum: "$peopleInfo.Count"}}}
])
return data;
}
});
If anyone else is tackling similar issues, feel free to contact me. Thanks everyone for your support!