Recursive Publishing - Angular-meteor - mongodb

Question

I am wanting to load a topic, 25 of its comments and up to 5 sub comments for each comment, repeated recursively over each comment/sub-comment until all related comments are found.
I'm currently using an angular directive to recursively subscribe and add to the local collection whenever the comment has children. It works quite well, but there is some lag (to be expected) between loading the initial 25 comments, and loading their children, then their children and so on.
This issue isn't a problem when just loading a page at a time. It becomes an issue when using infinite scrolling and increasing that initial 25 comment limit. It will cause the page to jump up and down a bit as the sub comments disappear and reappear once loaded again.
I was wondering how I could recursively look up all comments prior to sending to the local client so I don't need to make more than one round trip for each topic.
I have a demo loaded up at ck-gaming.com
If you scroll to the bottom it will load more and you'll see it jump all over as the sub comments are reloaded into the page.
The two options I can think of would be to use a resolve to wait for all collections prior to loading the page or using recursive publish to get them all first.
Thoughts? Ideas?
Ok, my first attempt that I would like some thoughts on if possible.
For the publishing I decided to go with publish-composite to make publishing from the same collection easier.
for the publication I wrote:
Meteor.publishComposite('oneDiscussion', function (slug, options) {
var query = {};
query.find = function () {
return Discussions.find({ slug: slug }, { limit: 1 });
};
var mainChildQuery = Comments.find({ slug: slug }, { limit: 1 });
query.children = [];
query.children[0] = {};
query.children[0].find = function (discussion) {
return mainChildQuery;
};
query.children[0].children = [];
query.children[0].children[0] = {};
query.children[0].children[0].find = function (comment) {
return Meteor.users.find({ _id: comment.author.id }, { limit: 1, fields: { profile: 1, roles: 1, createdAt: 1, username: 1 } });
};
query.children[0].children[1] = {};
query.children[0].children[1].find = function (parent) {
Counts.publish(this, 'numberOfComments', Comments.find(
{ parent_id: parent._id }
), { noReady: true });
console.log(options)
return Comments.find({ parent_id: parent._id }, options);
};
// var parentQuery = Comments.find({ slug: slug });
var parent = mainChildQuery.fetch();
var children = Comments.find({ parent_id: parent[0]._id }, { limit: 25 }).fetch();
var childrenIds = _.pluck(children, '_id');
var getChildren = function (children_ids, thisParent) {
var i = 0;
thisParent.children = [];
var recursive = function getEm(children, parent) {
_.each(children, function (id) {
// parent.children[i] = new Children(id);
var query = Comments.find({ parent_id: id }, { limit: 5, sort: { date: -1 } });
parent.children[i] = {
find: function () {
return Comments.find({ parent_id: id }, { limit: 5, sort: { date: -1 } });
}
};
var children1 = query.fetch();
var newChildrenIds = _.pluck(children1, '_id');
i++;
if (newChildrenIds.length > 0) {
getEm(newChildrenIds, parent);
}
});
}
recursive(children_ids, thisParent);
};
getChildren(childrenIds, query.children[0].children[1]);
return query;
});
Seems to be working ok so far, though running it on my desktop it's not as performant as I would think it should be. I'll deploy it and see if there's a difference online. I'll update when I get home and can update the live site. If anyone can find something wrong with what I've written it would be much appreciated.

Answer 1

I've came up with what I think is the best solution. I improved on the function above and so far I'm really enjoying the results.
Here is the publish function:
Meteor.publishComposite('comments', function (item_id, options) {
/**
* TODO: Add query to find a user for each comment.
*/
/**
* Start building our query.
* Add the latest 25 (depending on options) child comments of the viewed item
* to the query.
*/
var query = {
find: function () {
return Comments.find({ parent_id: item_id }, options);
}
};
// Query the database for the first 25? comments, we'll need their _id's
var mainChildren = Comments.find({ parent_id: item_id }, options).fetch();
// pluck the id's from the initial comments
var mainChildrenIds = _.pluck(mainChildren, '_id');
/**
* Builds the remaining query based on the id's plucked from the children
* above.
* #param childrens_id The id's we just plucked from the above query
* #param thisParent This is the parent query
*/
var getChildren = function (children_ids, parentQuery) {
// initiate i to 0
var i = 0;
// add a child array to the current parent query.
parentQuery.children = [];
var recursive = function getem(children, parent) {
_.each(children, function (id) {
var query = Comments.find({ parent_id: id }, { limit: 5, sort: { date: 1 } });
parent.children[i] = {
find: function () {
return query;
}
};
var children1 = query.fetch();
var newChildrenIds = _.pluck(children1, '_id');
i++;
if (newChildrenIds.length > 0) {
getem(newChildrenIds, parent);
}
});
};
// recursively build the query if there are children found.
recursive(children_ids, parentQuery);
};
// initiate the query build function
getChildren(mainChildrenIds, query);
return query;
});
I created an example app you can get on GitHub here
And you can view it running on meteorpad here
what it does
All the function does is build the publishComposite query, recursively looping over the children id's, as long as there are children id's. When there are no more children it stops.
to use it
You want a collection of comments (or whatever you're nesting) that have a parent_id field. This field will be filled with either the parent post Id, parent Item id (if say making a store with reviews/comments). The parent post id would of course be the comment or post you are commenting on. See the example for more information.