MongoDB: get latest full document for each id by date/time - mongodb

Question

I need to get the latest documents that are in an array of ids based on data/time. I have the following query that does this, but it only returns the _id and acquiredTime fields. How can I get it to return the full document with all the fields?
db.trip.aggregate([
{ $match: { tripId: { $in: ["trip01", "trip02" ]}} },
{ $sort: { acquiredTime: -1} },
{ $group: { _id: "$tripId" , acquiredTime: { $first: "$acquiredTime" }}}
])
The collection looks something like:
[{
"tripId": "trip01",
"acquiredTime": 1000,
"name": "abc",
"value": "abc"
},{
"tripId": "trip02",
"acquiredTime": 1000,
"name": "xyz",
"value": "xyz"
},{
"tripId": "trip01",
"acquiredTime": 2000,
"name": "def",
"value": "abc"
},{
"tripId": "trip02",
"acquiredTime": 2000,
"name": "ghi",
"value": "xyz"
}]
At the moment I get:
[{
"tripId": "trip01",
"acquiredTime": 2000
},{
"tripId": "trip02",
"acquiredTime": 2000
}]
I need to get:
[{
"tripId": "trip01",
"acquiredTime": 2000,
"name": "def",
"value": "abc"
},{
"tripId": "trip02",
"acquiredTime": 2000,
"name": "ghi",
"value": "xyz"
}]

Answer 1

Your approach is the right approach, but the thing is that $group and $project just don't work that way and require you to name all of the fields you want in the result.
If you don't mind the structure looking a bit different, then you can always use $$ROOT in MongoDB versions 2.6 and greater:
db.trip.aggregate([
{ "$match": { "tripId": { "$in": ["trip01", "trip02" ]}} },
{ "$sort": { "acquiredTime": -1} },
{ "$group": { "_id": "$tripId" , "doc": { "$first": "$$ROOT" }}}
])
So the whole document is there, but just all contained as a sub-document to "doc" in the results.
For anything else or prettier you are going to have to specify every field that you want. It's just a data structure so you could always generate it from code anyway.
db.trip.aggregate([
{ "$match": { "tripId": { "$in": ["trip01", "trip02" ]}} },
{ "$sort": { "acquiredTime": -1} },
{ "$group": {
"_id": "$tripId" ,
"acquiredTime": { "$first": "$acquiredTime" },
"name": { "$first": "$name" },
"value": { "$first": "$value" }
}}
])

Answer 2

To my understading, the above solution suffers from performance and RAM problems when there is a large number of unique documents to be returned, as the output of $match is sorted in memory, no matter what indices you may have.
Reference: https://docs.mongodb.com/manual/tutorial/sort-results-with-indexes/
To maximise performance and minimise RAM usage:
Create a unique index [(tripId, 1), (acquiredTime, -1)]
Have the sort to operate exactly along the index
This of course will cost you an index, which will slow down inserts - there's no free meal :)
Additionally, the cosmetic problem of having the original document moved to a sub-document can be easily solved with $replaceRoot, without needing to explicltly list the document keys.
db.trip.aggregate([
{ "$match": { "tripId": { "$in": ["trip01", "trip02" ]}} },
{ "$sort": SON([("tripId", 1), ("acquiredTime", -1)],
{ "$group": { "_id": "$tripId" , "doc": { "$first": "$$ROOT" }}},
{ "$replaceRoot": { "newRoot": "$doc"}}
])
Finally, it's worth noting that if acquiredTime is just your server time, you can get rid of it, as the _id already embeds the creation timestamp. So the unique index would go on [(tripId, 1), (_id, -1)], and the query becomes:
db.trip.aggregate([
{ "$match": { "tripId": { "$in": ["trip01", "trip02" ]}} },
{ "$sort": SON([("tripId", 1), ("_id", -1)],
{ "$group": { "_id": "$tripId" , "doc": { "$first": "$$ROOT" }}},
{ "$replaceRoot": { "newRoot": "$doc"}}
])
This is also better as date objects in MongoDB have a resolution of 1 millisecond, which - depending on the frequency of your inserts - may result in extremely hard to reproduce race conditions, whereas auto-generated _id are guaranteed to be strictly incremental.