I am working with 1.000 documets in a collection in MongoDb. Each document can be made of many topics, and a topic can be made of many keywords.
The mongo structure for each document is the following:
_id:ObjectId(6d5fc0922982bb550e08502d),
id_doc:"1234-678-436-42"
topic:Array
keywords:Array
The key topic is an array of objects o this type
type:"topic"
label:"work"
While, the keywords key is an array of objects, very similar to "topic":
type:"keyword"
value:"programmer"
label:"work"
Label represents in both cases the topic of the doc!
What I want is to list all the documents (id_doc) where a topic appears in the "topic" array, but never in the "keyword" array.
Query
takes the intersection of topic.label with keywords.label
if empty then no common members, so document passes the filter
*not sure if you want this, if not if you can give 1-2 documents in json text, and the expected output
Playmongo
aggregate(
[{"$match":
{"$expr":
{"$eq":
[{"$setIntersection": ["$topic.label", "$keywords.label"]}, []]}}}])
One option is using $filter:
count the number of items in topic which their label is not present in as a value of an item on keywords. Save this count as condMatch
$match only document with condMatch greater than 0
db.collection.aggregate([
{$set: {condMatch: {
$size: {
$filter: {
input: "$topic",
cond: {$not: {$in: ["$$this.label", "$keywords.value"]}}
}
}
}
}
},
{$match: {condMatch: {$gt: 0}}},
{$unset: "condMatch"}
])
See how it works on the playground example
I'm trying to look up all records that match a certain condition, in this case _id being certain values, and then return only the top 2 results, sorted by the name field.
This is what I have
db.getCollection('col1').aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$sort: {fk: 1, name: -1}},
{$group: {_id: "$fk", items: {$push: "$$ROOT"} }},
{$project: {items: {$slice: ["$items", 2]} }}
])
and it works, BUT, it's not guaranteed. According to this Mongo thread $group does not guarantee document order.
This would also mean that all of the suggested solutions here and elsewhere, which recommend using $unwind, followed by $sort, and then $group, would also not work, for the same reason.
What is the best way to accomplish this with Mongo (any version)? I've seen suggestions that this could be accomplished in the $project phase, but I'm not quite sure how.
You are correct in saying that the result of $group is never sorted.
$group does not order its output documents.
Hence doing a;
{$sort: {fk: 1}}
then grouping with
{$group: {_id: "$fk", ... }},
will be a wasted effort.
But there is a silver lining with sorting before $group stage with name: -1. Since you are using $push (not an $addToSet), inserted objects will retain the order they've had in the newly created items array in the $group result. You can see this behaviour here (copy of your pipeline)
The items array will always have;
"items": [
{
..
"name": "Michael"
},
{
..
"name": "George"
}
]
in same order, therefore your nested array sort is a non-issue! Though I am unable to find an exact quote in documentation to confirm this behaviour, you can check;
this,
or this where it is confirmed.
Also, accumulator operator list for $group, where $addToSet has "Order of the array elements is undefined." in its description, whereas the similar operator $push does not, which might be an indirect evidence? :)
Just a simple modification of your pipeline where you move the fk: 1 sort from pre-$group stage to post-$group stage;
db.getCollection('col1').aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$sort: {name: -1}},
{$group: {_id: "$fk", items: {$push: "$$ROOT"} }},
{$sort: {_id: 1}},
{$project: {items: {$slice: ["$items", 2]} }}
])
should be sufficient to have the main result array order fixed as well. Check it on mongoplayground
$group doesn't guarantee the document order but it would keep the grouped documents in the sorted order for each bucket. So in your case even though the documents after $group stage are not sorted by fk but each group (items) would be sorted by name descending. If you would like to keep the documents sorted by fk you could just add the {$sort:{fk:1}} after $group stage
You could also sort by order of values passed in your match query should you need by adding a extra field for each document. Something like
db.getCollection('col1').aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$addField:{ifk:{$indexOfArray:[[1, 2],"$fk"]}}},
{$sort: {ifk: 1, name: -1}},
{$group: {_id: "$ifk", items: {$push: "$$ROOT"}}},
{$sort: {_id : 1}},
{$project: {items: {$slice: ["$items", 2]}}}
])
Update to allow array sort without group operator : I've found the jira which is going to allow sort array.
You could try below $project stage to sort the array.There maybe various way to do it. This should sort names descending.Working but a slower solution.
{"$project":{"items":{"$reduce":{
"input":"$items",
"initialValue":[],
"in":{"$let":{
"vars":{"othis":"$$this","ovalue":"$$value"},
"in":{"$let":{
"vars":{
//return index as 0 when comparing the first value with initial value (empty) or else return the index of value from the accumlator array which is closest and less than the current value.
"index":{"$cond":{
"if":{"$eq":["$$ovalue",[]]},
"then":0,
"else":{"$reduce":{
"input":"$$ovalue",
"initialValue":0,
"in":{"$cond":{
"if":{"$lt":["$$othis.name","$$this.name"]},
"then":{"$add":["$$value",1]},
"else":"$$value"}}}}
}}
},
//insert the current value at the found index
"in":{"$concatArrays":[
{"$slice":["$$ovalue","$$index"]},
["$$othis"],
{"$slice":["$$ovalue",{"$subtract":["$$index",{"$size":"$$ovalue"}]}]}]}
}}}}
}}}}
Simple example with demonstration how each iteration works
db.b.insert({"items":[2,5,4,7,6,3]});
othis ovalue index concat arrays (parts with counts) return value
2 [] 0 [],0 [2] [],0 [2]
5 [2] 0 [],0 [5] [2],-1 [5,2]
4 [5,2] 1 [5],1 [4] [2],-1 [5,4,2]
7 [5,4,2] 0 [],0 [7] [5,4,2],-3 [7,5,4,2]
6 [7,5,4,2] 1 [7],1 [6] [5,4,2],-3 [7,6,5,4,2]
3 [7,6,5,4,2] 4 [7,6,5,4],4 [3] [2],-1 [7,6,5,4,3,2]
Reference - Sorting Array with JavaScript reduce function
There is a bit of a red herring in the question as $group does guarantee that it will be processing incoming documents in order (and that's why you have to sort of them before $group to get an ordered arrays) but there is an issue with the way you propose doing it, since pushing all the documents into a single grouping is (a) inefficient and (b) could potentially exceed maximum document size.
Since you only want top two, for each of the unique fk values, the most efficient way to accomplish it is via a "subquery" using $lookup like this:
db.coll.aggregate([
{$match: {fk: {$in: [1, 2]}}},
{$group:{_id:"$fk"}},
{$sort: {_id: 1}},
{$lookup:{
from:"coll",
as:"items",
let:{fk:"$_id"},
pipeline:[
{$match:{$expr:{$eq:["$fk","$$fk"]}}},
{$sort:{name:-1}},
{$limit:2},
{$project:{_id:0, fk:1, name:1}}
]
}}
])
Assuming you have an index on {fk:1, name:-1} as you must to get efficient sort in your proposed code, the first two stages here will use that index via DISTINCT_SCAN plan which is very efficient, and for each of them, $lookup will use that same index to filter by single value of fk and return results already sorted and limited to first two. This will be the most efficient way to do this at least until https://jira.mongodb.org/browse/SERVER-9377 is implemented by the server.
I somehow created duplicates of every single entry in my database. Currently, there are 176039 documents and counting, half are duplicates. Each document is structured like so
_id : 5b41d9ccf10fcf0014fe8917
originName : "Hartsfield Jackson Atlanta International Airport"
destinationName : "Antigua"
totalDuration : 337
Inside the MongoDB Compass Community App for Mac under the Aggregations tab, I was able to find duplicates using this pipeline
[
{$group: {
_id: {originName: "$originName", destinationName: "$destinationName"},
count: {$sum: 1}}},
{$match: {count: {"$gt": 1}}}
]
I'm not sure how to move forward and delete the duplicates at this point. I'm assuming it has something to do with $out.
Edit: Something I didn't notice until now is that the values for totalDuration on each double are actually different.
Add
{$project:{_id:0, "originName":"$_id.originName", "destinationName":"$_id.destinationName"}},
{ $out : collectionname }
This will replace the documents in your current collection with documents from aggregation pipeline. If you need totalDuration in the collection then add that field in both group and project stage before running the pipeline
is there a way to sort the documents returned from limit?
example:
//returns 10 documents:
db.users.find()
.sort({last_online_timestamp:-1})
.limit(10)
//returns 10 another documents
db.users.find()
.sort({last_online_timestamp:-1})
.limit(10)
.sort({messages_count:1});
what I want is to get 10 last logged in users and then sort them by messages count.
You can use an aggregate such as
db.users.aggregate([
{"$sort": {"last_online_timestamp":1}},
{"$limit": 10},
{"$sort": {"messages_count": 1}}
])
This will go through stages where the the documents in the collection will be:
Sorted by the last_online_timestamp field in ascending order
Limited to 10 documents
Those 10 documents will be sorted by the messages_count field in ascending order
You may want to change {"$sort": {"last_online_timestamp":1} to {"$sort": {"last_online_timestamp":-1} depending on the actual values of last_online_timestamp
For more information on Mongo aggregation, see https://docs.mongodb.com/manual/aggregation/.
I have a mongo collection which looks something like this:
{
title: String,
category: String
}
I want to write a query that selects various categories, similar to this:
Collection.find({category: {$in: ['Books', 'Cars', 'People']});
But I want to only select a limited number of each category, for example 5 of each book, car, people. How do I write such a query? Can I do it one query or must I use multiple ones?
You can do it using mongodb aggregation. Take a look at this pipeline:
Filter all documents by categories(using $match).
Group data by categories and create array for items with the same category(using $group and $push).
Get a subset of each array with a limited maximum length(using $project and $slice).
Try the following query:
db.collection.aggregate([
{$match: {category: {$in: ['Books', 'Cars', 'People']}}},
{$group: {_id: "$category", titles: {$push: "$title"}}},
{$project: {titles: {$slice: ["$titles", 5]}}}
])