Find missing documents in MongoDB using aggregation pipeline - mongodb

Question

I am trying to find the missing documents in MongoDB. Scenario is like this: I have a collection where the documents have a hour-stamp field. I want to find which hours are missing given the time range.
Since I am writing this question on Metabase, I am limited to use only one aggregation pipeline, meaning I can't use $out to make temperate collection and do $lookup for join.
I can only fill in the code of db.collection.aggregate(my code)
Any idea how can I achieve this? Thanks a lot!

Answer 1

Was able to achieve this so just sharing my solution.
Idea:
Generate an array for the hours needed to be checked. Notice I use
hour-diff from current hour, so I can dynamically check if (-9
hours) is missing. Reason for doing this is that I cannot find a way
to programmatically generate this array using absolute hour-stamp
(2022-07-11 10:00:00).
Calculate the hour-diff from current of the data's hour-stamp.
Use $setDifference to find the missing hours.
Calculate the absolute hour-stamp from the hour-diff value to get the missing hours.
Works for my need, and hope this will help someone.
Code snippet (I use this for finding missing hours between -6 to -30 hours for each data_source) :
db.getCollection(<collection_name>).aggregate(
[
{ $project: {
_id: 1,
data_source: 1,
available_date_time: { $toDate: "$available_date_time"},
current_hour: { $dateFromString: { dateString: { $dateToString: { format: "%Y-%m-%dT%H", date: ISODate() } }, format: "%Y-%m-%dT%H" } },
}},
{ $project: {
_id:1,
data_source:1,
available_date_time: 1,
current_hour: 1,
current_hour_minus_6hr: { $subtract: [ "$current_hour", { $multiply: [6, 60, 60, 1000] }] },
current_hour_minus_30hr: { $subtract: [ "$current_hour", { $multiply: [30, 60, 60, 1000] }] },
}},
{ $project: {
_id:1,
data_source:1,
available_date_time: 1,
current_hour: 1,
past_6hr_comp: { $subtract: [ { $toDate: "$available_date_time"}, "$current_hour_minus_6hr" ] },
past_30hr_comp: { $subtract: [ { $toDate: "$available_date_time"}, "$current_hour_minus_30hr" ] },
}},
{ $match: {
$and: [
{ past_30hr_comp: { $gte: 0 } },
{ past_6hr_comp: { $lte: 0} }
]
}},
{ $project: {
_id: 1,
data_source:1,
available_date_time: 1,
current_hour: 1,
hour_diff_from_current: { $divide: [ {$subtract: [ "$current_hour", "$available_date_time" ] }, 3600000 ] }
}},
{ $group: {
_id: { data_source: "$data_source" },
count: { $sum: 1 },
available_hour_diff_set: { $addToSet: "$hour_diff_from_current" },
current_hour: { $first: "$current_hour" }
}},
{ $project: {
_id: 0,
data_source: "$_id.data_source",
available_hour_count: "$count",
available_hour_diff_set: "$available_hour_diff_set",
required_hour_diff_set: [30, 29, 28, 27, 26, 25, 24, 23, 22, 21, 20, 19, 18, 17, 16, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6],
current_hour: "$current_hour"
}},
{ $project: {
data_source: 1,
available_hour_count: 1,
unavailable_hour_diff_set: { $setDifference: [ "$required_hour_diff_set", "$available_hour_diff_set" ] },
current_hour:1
}},
{ $unwind: "$unavailable_hour_diff_set" },
{ $project: {
data_source: 1,
available_hour_count: 1,
current_hour: 1,
unavailable_hour_diff: "$unavailable_hour_diff_set",
}},
{ $project: {
data_source: 1,
available_hour_count: 1,
current_hour: 1,
unavailable_hour_diff: "$unavailable_hour_diff",
missing_hour: { $subtract: [ "$current_hour", { $multiply: [ "$unavailable_hour_diff", 60, 60, 1000 ] } ] }
}}
]