Aggregation with condtion in embedded documents in mongo db - mongodb

Question

I'm stuck with aggregation in mongodb. The premise is I have to get data for particular ads within a time range.
So suppose I query for ads within a range of 22nd April to 24th April, here is what I should get, summation of spend from source2, and revenue, session, bounces etc from source1.
[{ "_id" : ObjectId("560bbd5dfabc614611000e95"),
"spend": 470,
"revenue": 440,
"sessions": 3
},....
]
Here is the query, I was attempting which gives me correct data but takes really long - 24seconds for only 22k entires.
db.getCollection('tests').aggregate([{
$match: {
ad_account_id: 40
}
}, {
"$unwind": "$source1"
}, {
"$unwind": "$source2"
}, {
"$group": {
"_id": "$internal_id",
"transactionrevenue": {
"$sum": {
"$cond": [{
"$and": [{
"$gte": [
"$source1.created_at", ISODate("2015-04-22T00:00:00.000Z")
]
}, {
"$lte": [
"$source1.created_at", ISODate("2015-04-25T00:00:00.000Z")
]
}]
}, "$source1.transactionrevenue", 0]
}
},
"sessions": {
"$sum": {
"$cond": [{
"$and": [{
"$gte": [
"$source1.created_at", ISODate("2015-04-22T00:00:00.000Z")
]
}, {
"$lte": [
"$source1.created_at", ISODate("2015-04-25T00:00:00.000Z")
]
}]
}, "$source1.sessions", 0]
}
},
"spend": {
"$sum": {
"$cond": [{
"$and": [{
"$gte": [
"$source2.created_at", ISODate("2015-04-22T00:00:00.000Z")
]
}, {
"$lte": [
"$source2.created_at", ISODate("2015-04-25T00:00:00.000Z")
]
}]
}, "$source2.spend", 0]
}
}
},
}]);
Problems are how to unwind multiple times, how to get summation of multiple things in source1 and not having to do aggregation again and again? It takes 24seconds, for 22 entries....Please suggest on what I should index (I have none), and also if document size of average 4mb suggests there is something wrong with the schema?
Would map reduce be better even though aggregation is usually considered faster in mongodb?
If you think the document design is wrong, I'm all ears, as we're just working on the migration. Much better to correct things now, rather than later.
Here is a sample document
{
"_id" : ObjectId("560bbd5dfabc614611000e95"),
"internal_id": 1,
"created_at" : ISODate("2015-04-21T00:50:02.593Z"),
"updated_at" : ISODate("2015-09-15T12:20:39.154Z"),
"name" : "LookalikeUSApr21_06h19m",
"ad_account_id" : 40,
"targeting" : {
"age_max" : 44,
"age_min" : 35,
"genders" : [
1
],
"page_types" : [
"desktopfeed"
]
},
"auto_optimization" : false,
"source1" : [
{
"id" : 119560952,
"created_at" : ISODate("2015-04-23T12:35:09.467Z"),
"updated_at" : ISODate("2015-05-19T05:20:58.374Z"),
"transactionrevenue" : 320,
"sessions" : 1,
"bounces" : 1
},
{
"id" : 119560955,
"created_at" : ISODate("2015-05-01T12:35:09.467Z"),
"updated_at" : ISODate("2015-05-19T05:20:58.374Z"),
"transactionrevenue" : 230,
"sessions" : 10,
"bounces" : 1
},
{
"id" : 119560954,
"created_at" : ISODate("2015-04-23T10:35:09.467Z"),
"updated_at" : ISODate("2015-05-19T05:20:58.374Z"),
"transactionrevenue" : 120,
"sessions" : 2,
"bounces" : 1
},
{
"id" : 119560953,
"created_at" : ISODate("2015-04-25T12:35:09.467Z"),
"updated_at" : ISODate("2015-05-19T05:20:58.374Z"),
"transactionrevenue" : 100,
"sessions" : 3,
"bounces" : 2
}
],
"source2" : [
{
"id" : 219560952,
"created_at" : ISODate("2015-04-22T12:35:09.467Z"),
"updated_at" : ISODate("2015-05-19T05:20:58.374Z"),
"spend" : 300
},
{
"id" : 219560955,
"created_at" : ISODate("2015-04-23T12:35:09.467Z"),
"updated_at" : ISODate("2015-05-19T05:20:58.374Z"),
"spend" : 170
},
{
"id" : 219560954,
"created_at" : ISODate("2015-04-25T10:35:09.467Z"),
"updated_at" : ISODate("2015-05-19T05:20:58.374Z"),
"spend" : 450
}
]
}

Answer 1

The very first thing you should be doing is adding an index to both the source1 and source2 arrays for their "created_at" field. You will likely reduce a lot of possible results and improve speed greatly by simply querying for these possible matches being present in the documents you select.
The next main improvements are to combine the arrays and filter as one, and notably before you process $unwind. This is going to save a lot of cycles and document expansion in the arrays.
Moreover, it's going to give you the correct totals. When you $unwind two arrays, then one array's details get repeated by the number of items in the second array. This gives you incorrect results for the array content that you "unwound" first. You can always do each separately, but it's far better to merge them into one:
db.getCollection('tests').aggregate([
{ "$match": {
"ad_account_id": 40,
"$or": [
{
"source1": {
"$elemMatch": {
"created_at": {
"$gte": new Date("2015-04-22"),
"$lte": new Date("2015-04-25")
}
}
}
},
{
"source2": {
"$elemMatch": {
"created_at": {
"$gte": new Date("2015-04-22"),
"$lte": new Date("2015-04-25")
}
}
}
}
]
}},
{ "$project": {
"_id": 0,
"internal_id": 1,
"source": {
"$setDifference": [
{ "$map": {
"input": { "$setUnion": [ "$source1", "$source2" ] },
"as": "source",
"in": {
"$cond": [
{ "$and": [
{ "$gte": [ "$$source.created_at", new Date("2015-04-22") ] },
{ "$lte": [ "$$source.created_at", new Date("2015-04-25") ] }
]},
"$$source",
false
]
}
}},
[false]
]
}
}},
{ "$unwind": "$source"},
{ "$group": {
"_id": "$internal_id",
"transactionrevenue": { "$sum": { "$ifNull": [ "$source.transactionrevenue", 0 ] } },
"sessions": { "$sum": { "$ifNull": [ "$source.sessions", 0 ] } },
"spend": { "$sum": { "$ifNull": [ "$source.spend", 0 ] } }
}}
])
Which is going to give the result on your sample:
{ "_id" : 1, "transactionrevenue" : 440, "sessions" : 3, "spend" : 470 }
So probably the great big architecture hint in what is being done here it that it would be very wise to to combine the arrays into a single array in your general application usage. You can always add another field for "type" if you must to discern between the two different types of items, but just about all processing should benefit from a singular array.
The main lesson for the query aside from that, is that you always $match first to filter out as much content as possible. Whilst the initial $match stage cannot of course remove items from arrays that do not meet the conditions, what it can importantly do is "match the documents". because you do not want to process documents that don't have that information at all. That always adds time.
The second part other than the combined array is that basically you want to filter out any content before unwinding the array where possible for much the same reasons, as you don't want to be processing items you don't need to.
Short lesson, filter first to reduce what you are processing. Conditional sums are fine, but really only should be used for selection of content and not raw filtering. It's basically about getting rid of the undesired data first rather than just ignoring it. Process less and do it faster.