Joining two aggregation results in mongodb using $out of aggregate - mongodb

Question

I have a reporting app, and I generate mongodb commands, and it involves running three aggregate calls. The aggregate calls have [match,group,project] in their pipes.
RESULT OF AGGREGATE 1-3
{_id: <XXX>, ...}
The grouping "_id" for these calls are same, but, because their $match are different they cannot be in the same aggregate call. I need to join all of these aggregation results. I know that one way to solve this is using conditions during the $group stage, but the problem is the conditions are really complicated to mix with the already complex $group pipe.
To give some context why that solution is very difficult if not impossible; the data is quite huge, each doc has 700 attributes, and the docs are coming it at around 1k per day. Generating such complicated condition into EACH field in the $group stage will make a mess.
I have seen answers that are running map-reduce to combine these aggregation results, but I am looking for other solutions. As I've researched, aggregate has an $out pipe. Is there any way that I can manipulate that $out pipe to join these aggregation results? (The reason for thinking of $out is that I have to save ALL the results anyway as a report)

Answer 1

If you indeed want to go ahead with merging the aggregation results, then you can create an output collection using bulk upserts. For performance you can create a compound index on this output collection, which has your grouping attributes.
dataArray.map(function(data) {
data.forEach(function(err, row){
var setOnInsert = {grouping_attrs: row.grouping_values, v1: row.v1}
var set = {v2: row.v2}
var query = {grouping_attrs: row.grouping_values}
bulk.find(query).upsert().update({$setOnInsert: setOnInsertStmt, $set: set});
})
})
Here your dataArray is created using find on the $out collections.