MongoDB optimizing big aggregation queries - mongodb

Question

I have a collection of documents in MongoDB, representing some entity. For every entity there are some statistics data gathered on a daily basis. The statistics are put as a separate documents into different collections.
Entity collection schema:
{
_id: ObjectId,
filterField1: String, //indexed
filterField2: String, //indexed
}
Example schema of statistics collection:
{
_id: ObjectId,
entityId: ObjectId, //indexed
statisticsValue: Int32,
date: Date //indexed
}
There is a dashboard that needs to display some aggregated statistics based on the gathered data over some time period e.x. average value, sum, count etc. The dashboard enables filtering in/out entities and applying different date ranges which makes precalculating those aggregated statistics impossible.
As for now, I've been using aggregation pipeline to:
apply the filters on the entities collection (using match stage)
make necessary lookups stages to acquire statistics for aggregation
make grouping and aggregation (avg, sum, count, etc.)
Here is the pipeline:
db.getCollection('entities').aggregate([
{ $match: { $expr: { $and: [
// ENTITIES FILTERS based on filterField1 and filterField2 fields
] } } },
{ $lookup: {
from: 'statistics',
let: { entityId: '$_id' },
pipeline: [{ $match: { $expr: { $and: [
{ $eq: ["$entityId", "$$entityId"] },
{ $gte: [ "$date", new ISODate("2022-06-01T00:00:00Z") ] },
{ $lte: [ "$date", new ISODate("2022-06-01T23:59:59Z") ] },
] } } },
as: 'stats_start_date_range',
} },
{ $lookup: {
from: 'statistics',
let: { key: '$_key' },
pipeline: [{ $match: { $expr: { $and: [
{ $eq: ["$entityId", "$$entityId"] },
{ $gte: [ "$date", new ISODate("2022-06-31T00:00:00Z") ] },
{ $lte: [ "$date", new ISODate("2022-06-31T23:59:59Z") ] },
] } } },
as: 'stats_end_date_range',
} },
{ $addFields:
{
start_stats: { $first: "$stats_start_date_range" },
end_stats: { $first: "$stats_end_date_range" }
}
},
{
$group: {
_id: null,
avg_start: { $avg: "$start_stats.statisticsValue" },
avg_end: { $avg: "$end_stats.statisticsValue" }
}
}
])
In case of this query, the expected result is the average value of the statisticsValue field for the start and end date for every entity matching the filters.
I applied the index on the field used to left join collections in lookup stage. as well as on the date field used for getting statistics for a specific date.
The problem is that the query takes about 1 second for the max number of documents after the match stage (about 1000 documents). And I need to perform 4 such queries. The statistic collection contains 800k documents and the number is growing every day.
I was wondering, if I can do anything to make the query execution faster, I considered:
time series collection
reorganizing collections structure (don't know how)
merging those 4 separate queries into 1, using facet stage
But I'm not sure if MongoDB is suitable data source for such operations and maybe I should consider another data source if I want to perform such queries.

Answer 1

Hard to guess, what you would like to get. An approach could be this one:
const entities = db.getCollection('entities').aggregate([
{ $match: { filterField1: "a" } }
]).toArray().map(x => x._id)
db.getCollection('statistics').aggregate([
{
$match: {
entityId: { $in: entities },
date: {
$gte: ISODate("2022-06-01T00:00:00Z"),
$lte: ISODate("2022-06-31T23:59:59Z")
}
}
},
{
$facet: {
stats_start_date_range: [
{
$match: {
date: {
$gte: ISODate("2022-06-01T00:00:00Z"),
$lte: ISODate("2022-06-01T23:59:59Z")
}
}
}
],
stats_end_date_range: [
{
$match: {
date: {
$gte: ISODate("2022-06-31T00:00:00Z"),
$lte: ISODate("2022-06-31T23:59:59Z")
}
}
}
]
}
},
{
$addFields: {
start_stats: { $first: "$stats_start_date_range" },
end_stats: { $first: "$stats_end_date_range" }
}
},
{
$group: {
_id: null,
avg_start: { $avg: "$start_stats.statisticsValue" },
avg_end: { $avg: "$end_stats.statisticsValue" }
}
}
]);