How to remove duplicates based on property after an aggregate query? - mongodb

Question

I'm trying to figure out how to remove duplicates based on the url, as the aggregate query can match the same document twice if say "APPL" and "TSLA" are in stocks and included in the same document.
const res = await ReadSchema.find({ 'tickers.symbol': { $in: stocks } })
.select(['-content', '-htm'])
.sort({ createdAt: -1 })
.limit(100)
.filter((v, i, a) => a.findIndex((t) => t.url === v.url) === i);
I'm doing this in js after the results, but I'd rather do it before the .limit(100) so that I remove duplicate urls before I grab the first 100.
Here is an example document:
{
"_id" : ObjectId("6001eb414110e6001cb5b226"),
"summary" : {
"topics" : [
"buds",
"pro",
"galaxy",
"anc",
"live",
"sound",
"samsung",
"noise",
"ear",
"app"
],
"sentiment" : 0,
"words" : 0,
"difficulty" : 0,
"minutes" : 0
},
"views" : 1,
"title" : "Samsung takes on Apple's AirPods Pro with new Galaxy Buds Pro",
"byline" : "Alex Perry",
"dir" : null,
"content" : "<div id=\"readability-page-1\" class=\"page\"><section>\n\n<p>Sometimes making things less fun is for the best. Case in point: The Samsung Galaxy Buds Pro. </p>\n<p>Just a handful of months after Samsung graced us with the delightfully bean-shaped (and genuinely good) Galaxy Buds Live, we've already got a more, uh, \"professional\" follow-up. At $199.99, the Galaxy Buds Pro are only marginally more expensive than the Buds Live, but offer a totally different look and a few extra features, largely revolving around active noise cancellation.</p>\n<p>The end result is something that should please Samsung fans (and Android users in general). But is there any reason to spend that extra $30 instead of getting the <em>already totally fine</em> Galaxy Buds Live? </p>\n<div>\n\n<div>\n<div>\n<div>\n<p>\nMuch more comfortable than Buds Live • Improved ANC • Sound quality is still great • Strong app-based customization\n</p></div>\n<div>\n<p>\nVoice detection feature is a dud • ANC struggles with outdoor bustle\n</p></div>\n\n</div>\n\n\n\n</div>\n</div>\n\n<h2>The Good: Improved ANC and comfort, SmartThings Find, sound quality</h2>\n<p>The most noticeable difference between the Buds Live and the Buds Pro from the jump is their appearance. As we've discussed, the Buds Live were shaped like something you'd find in a burrito and, on a practical level, let a decent amount of outside noise in due to the lack of a seal of any kind. Their ANC features did a little bit to offset this, but it wasn't good enough. It was pretty sweet that they attracted next to no earwax because they didn't actually go inside the ear, though.</p>\n<p>The Buds Pro aren't nearly as eye-catching but they make up for it in the ways that actually matter. Like Apple's AirPods Pro equivalent, the Buds Pro have rubber ear cups that you insert into the ear canal with a rounded touch panel that sits outside featuring basic playback controls. Tap once to pause, tap twice to skip ahead, and so on. Samsung is shipping the Buds Pro with three different ear cup sizes to accommodate different ear shapes, as well.</p>\n<div>\n<div>\n\n<figure><div>\n <p><img alt=\"Galaxy Buds Live\" data-fragment=\"m!0fdc\" data-image=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591715%252F873bf2ac-34d3-4a4a-b2cd-f3befe3f05fe.png%252Ffull-fit-in__1200x2000.png?signature=NZalyPHmMKjSvE_-Md19lx1Ovfk=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" data-micro=\"1\" loading=\"lazy\" naturalwidth=\"950\" sizes=\"(max-width: 300px) 280w,(max-width: 400px) 356w,(max-width: 700px) 638w,950w\" src=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591715%252F873bf2ac-34d3-4a4a-b2cd-f3befe3f05fe.png%252Ffull-fit-in__950x534.png?signature=0P5F2WHmrx7IopVdxva_M-CrBCs=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" src_type=\"png\" srcset=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591715%252F873bf2ac-34d3-4a4a-b2cd-f3befe3f05fe.png%252Ffull-fit-in__280x157.png?signature=KLY0Uk8B1s5bL03gF9ehpT1HFM4=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 280w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591715%252F873bf2ac-34d3-4a4a-b2cd-f3befe3f05fe.png%252Ffull-fit-in__356x205.png?signature=Z-q_8wMfWcP-cbOWGMcJOcIcps0=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 356w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591715%252F873bf2ac-34d3-4a4a-b2cd-f3befe3f05fe.png%252Ffull-fit-in__638x368.png?signature=dUn8anfa1mPCeLNJAwiejq4X83E=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 638w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591715%252F873bf2ac-34d3-4a4a-b2cd-f3befe3f05fe.png%252Ffull-fit-in__950x534.png?signature=0P5F2WHmrx7IopVdxva_M-CrBCs=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 950w\">\n</p></div>\n\n\n<figcaption><p>Galaxy Buds Live\n</p></figcaption><p>Image: alex perry / mashable\n</p></figure>\n</div>\n<div>\n\n<figure><div>\n <p><img alt=\"Galaxy Buds Pro\" data-fragment=\"m!a70e\" data-image=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591722%252Fb72cbc58-58fc-484c-8a54-fae6c2306ebc.jpg%252Ffull-fit-in__1200x2000.jpg?signature=dFshI8fkvoBN6_Rg8ZYPqfRwxb4=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" data-micro=\"1\" loading=\"lazy\" naturalwidth=\"950\" sizes=\"(max-width: 300px) 280w,(max-width: 400px) 356w,(max-width: 700px) 638w,950w\" src=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591722%252Fb72cbc58-58fc-484c-8a54-fae6c2306ebc.jpg%252Ffull-fit-in__950x534.jpg?signature=SpjFrkhxftv8Z4eGTzVzosVqvhI=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" src_type=\"jpg\" srcset=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591722%252Fb72cbc58-58fc-484c-8a54-fae6c2306ebc.jpg%252Ffull-fit-in__280x157.jpg?signature=HXfNokeRqH-JA92c1-GP_w-fQlI=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 280w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591722%252Fb72cbc58-58fc-484c-8a54-fae6c2306ebc.jpg%252Ffull-fit-in__356x205.jpg?signature=d2iT0s-p_vsJpWa0pK1kHpbm6Qg=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 356w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591722%252Fb72cbc58-58fc-484c-8a54-fae6c2306ebc.jpg%252Ffull-fit-in__638x368.jpg?signature=5eWyZZdHOKWwE4y2-2ogtFJHUg4=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 638w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591722%252Fb72cbc58-58fc-484c-8a54-fae6c2306ebc.jpg%252Ffull-fit-in__950x534.jpg?signature=SpjFrkhxftv8Z4eGTzVzosVqvhI=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 950w\">\n</p></div>\n\n\n<figcaption><p>Galaxy Buds Pro\n</p></figcaption><p>Image: alex perry / mashable\n</p></figure>\n</div>\n</div>\n<p>From a purely physical standpoint, the Buds Pro are fundamentally a step up from the Buds Live because of how much better they feel in the ear. The \"beans\" fit snugly and were never a threat to fall out, but the insides of my ears didn't appreciate long sessions with them, and signaled that with minor aching. By virtue of keeping hard plastic away from the sensitive skin inside my ears, the Buds Pro were <em>significantly</em> more comfortable for lengthy periods of time. </p>\n<p>The ear tips not only make the Buds Pro more comfortable, but they also do a much better job of blocking unwanted outside noise than the Buds Live's open form factor did. While the ANC here still isn't perfect (more on that in a bit), it's a real improvement. When it's enabled and you're not in close proximity to chaos, you'll be able to listen to your music or podcasts in peace.</p>\n<p>Just like the Buds Live, Samsung's newest earbuds can connect to the Galaxy Wearable Android app for all of your settings and customization needs. That app is Android-only, so iPhone owners should probably buy some AirPods instead. Within the Wearable app you can set ANC to high or low (I didn't notice a huge difference between the two), turn it off entirely, or turn on an ambient sound mode that uses the on-board microphones to amplify sounds around you in situations where you might need to hear what people are saying. </p>\n<p>Beyond that, you can adjust what the long-press command on the earbuds' outer panels does (taps are set in stone), choose from a handful of different equalizer options like \"bass boost\" and \"treble boost,\" or even shift the audio balance to favor the left or right side if your hearing is better in one ear than the other. This amount of customization was a strong aspect of the Buds Live and it's still strong here, as well. It isn't super different, but it's appreciated nonetheless.</p>\n<p>Another side of the Buds Pro that picks up where Buds Live left off is sound quality. I was really impressed with how deep and rich the sound was in the \"beans\" despite their unusual shape. That hasn't changed with the shift to a more traditional build. The audio output here isn't dramatically better than what I experienced in the \"beans,\" but the 11mm woofer and 6.5mm tweeter inside do an excellent job of making songs sound like they should. </p>\n<figure><div>\n <p><img alt=\"The Galaxy Wearable app\" data-fragment=\"m!beae\" data-image=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591421%252Fc7dad555-1148-4700-8268-77002a63f642.jpg%252Foriginal.jpg?signature=mVprQdLxhltKq-tGPj21dYhKOzk=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" data-micro=\"1\" loading=\"lazy\" naturalwidth=\"950\" sizes=\"(max-width: 300px) 280w,(max-width: 400px) 356w,(max-width: 700px) 638w,950w\" src=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591421%252Fc7dad555-1148-4700-8268-77002a63f642.jpg%252Ffull-fit-in__950x534.jpg?signature=cZGICWc0BfWlzHT4dBKEa7BDPos=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" src_type=\"jpg\" srcset=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591421%252Fc7dad555-1148-4700-8268-77002a63f642.jpg%252Ffull-fit-in__280x157.jpg?signature=I9LxxqU9xNEBTe6ZDZ3x8Pf6MFo=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 280w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591421%252Fc7dad555-1148-4700-8268-77002a63f642.jpg%252Ffull-fit-in__356x205.jpg?signature=Vxxu-64ZaXie39nPyghsfKgryA0=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 356w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591421%252Fc7dad555-1148-4700-8268-77002a63f642.jpg%252Ffull-fit-in__638x368.jpg?signature=6fP9FBkiCqJdSAbT6qOBW6EOT2s=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 638w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591421%252Fc7dad555-1148-4700-8268-77002a63f642.jpg%252Ffull-fit-in__950x534.jpg?signature=cZGICWc0BfWlzHT4dBKEa7BDPos=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 950w\">\n</p></div>\n\n\n<figcaption><p>The Galaxy Wearable app\n</p></figcaption><p>Image: alex perry / mashable\n</p></figure><p>Instruments and vocals at the front of the mix are appropriately prominent, while aspects of a song that fill out the sound without calling much attention to themselves are fully audible if you focus on them. Some wireless earbuds can flatten songs out due to small, chintzy speakers, but the Buds Pro avoid that particular pitfall. I would still say the Bose QC Earbuds are <em>slightly</em> richer in sound, but they're also $80 more expensive. </p>\n<p>Battery life is also acceptable, if not remarkable. People who like to use ANC will notice battery drain, as the Buds Pro are rated for about five hours of ANC listening time with an additional 13 hours in the charging case, which is in line with my experience. Turn ANC off and you should get closer to eight hours on a single charge, so if you can live without that key feature, you'll have a better time.</p>\n<p>Oh, and one last thing: Galaxy Buds Pro work with Samsung's recently launched device tracking feature within the SmartThings app. SmartThings Find will show you a GPS location for your Buds Pro even if they're off. If they get lost or stolen, this is huge. Simply open the SmartThings app while the Buds Pro are paired to the phone to pair them to the app. It's quick, easy, and eminently useful.</p>\n<h2>The Bad: Sensitive voice detection, ANC is better indoors than outdoors</h2>\n<figure><div>\n <p><img alt=\"The case is almost identical to the Buds Live case.\" data-fragment=\"m!6d74\" data-image=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591414%252F2bf3f78c-75f9-4720-af0b-0e8494231004.jpg%252Foriginal.jpg?signature=c1GLO0Mcnadapz4KHB3k_jiZgB0=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" data-micro=\"1\" loading=\"lazy\" naturalwidth=\"950\" sizes=\"(max-width: 300px) 280w,(max-width: 400px) 356w,(max-width: 700px) 638w,950w\" src=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591414%252F2bf3f78c-75f9-4720-af0b-0e8494231004.jpg%252Ffull-fit-in__950x534.jpg?signature=qMk3EQVFbzAZSCBsJrTc1Hpg5V0=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" src_type=\"jpg\" srcset=\"https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591414%252F2bf3f78c-75f9-4720-af0b-0e8494231004.jpg%252Ffull-fit-in__280x157.jpg?signature=sKmiyBUFQdAvwwN_hQDYwFSCOGo=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 280w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591414%252F2bf3f78c-75f9-4720-af0b-0e8494231004.jpg%252Ffull-fit-in__356x205.jpg?signature=MrGzmcAVT-ha4f0pyB_cjLjtaV8=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 356w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591414%252F2bf3f78c-75f9-4720-af0b-0e8494231004.jpg%252Ffull-fit-in__638x368.jpg?signature=K03SKbqjuUqgjS0hTi-xENqm3Js=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 638w,https://mondrian.mashable.com/uploads%252Fcard%252Fimage%252F1591414%252F2bf3f78c-75f9-4720-af0b-0e8494231004.jpg%252Ffull-fit-in__950x534.jpg?signature=qMk3EQVFbzAZSCBsJrTc1Hpg5V0=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com 950w\">\n</p></div>\n\n\n<figcaption><p>The case is almost identical to the Buds Live case.\n</p></figcaption><p>Image: alex perry / mashable\n</p></figure><p>Unfortunately, one of the more distinct ANC-centric features of the Buds Pro was a dud for me. Samsung is touting the \"voice detect\" feature (enabled in the Galaxy Wearable app) as a way to get through your day without constantly turning ANC on or off, but I ended up keeping the feature disabled entirely. </p>\n<p>The way it works is that the Buds Pro can detect when you start speaking, at which point they will turn whatever you're listening to down and turn on the ambient noise mode so you can theoretically carry a conversation for a moment. Once you stop speaking for 10 seconds, everything goes back to the way it was. My issue is that it reacted a little too sensitively to things like sighing and clearing my throat, which would then ruin a song I was listening to for 10 seconds. </p>\n<p>I wish there were a way to shorten the time before ambient sound turns off, but for now, 10 seconds is the only option you have. I can see people getting use out of this, but personally, I found it more useful to long press one of the earbuds to manually turn ambient sound mode on or off.</p>\n<p>The only other minor criticism I have is less of a specific problem with the Buds Pro and more of a general issue with noise-cancelling wireless earbuds. Samsung's ANC here is better than it was in the previous model, but it can still let too much noise in when you're walking down a busy street. In other words, you can drown out the noise of an office or an apartment, but cars whizzing by you will still make noise. Use over-ear noise-cancelling headphones if you want to <em>really</em> escape from the world.</p>\n<h2>The Verdict</h2>\n<p>If I owned a Samsung smartphone, the Galaxy Buds Pro would be a no-brainer for me. They work just as seamlessly with something like a Galaxy Note 20 (which I used for testing) as AirPods do with an iPhone, provide terrific sound quality and noise cancellation, and can sit in my ears for hours at a time without bothering me. The improved ANC and additional comfort are enough to make me wholeheartedly recommend these over the Galaxy Buds Live for Android owners.</p>\n<p>That said, I wish the ANC did a better job of muting louder outdoor noises, but maybe that's a few years away. It's also unfortunate that the voice detection feature can be more of a hindrance than a help, at least in my experience. Despite those minor issues, as well as middling battery life when ANC is turned on, the Galaxy Buds Pro are a worthy successor to the Galaxy Buds Live and something Android owners should consider.</p>\n<p>With all due respect to the \"beans,\" this is a case where boring is better.</p>\n<h2>WATCH: Unboxing the BTS Edition Samsung Galaxy Buds+</h2>\n<div data-source=\"blueprint\" data-template=\"embedded\" data-video=\"O4oKA2YZnd\">\n<div>\n <p><img alt=\"Uploads%252fvideo uploaders%252fdistribution thumb%252fimage%252f95136%252f96587d2f 7dea 48ef 87c8 0ed74bd00239.png%252f930x520.png?signature=dkg2rtrnetqd20fsap ou2byynm=&source=https%3a%2f%2fblueprint api production.s3.amazonaws\" src=\"https://mondrian.mashable.com/uploads%252Fvideo_uploaders%252Fdistribution_thumb%252Fimage%252F95136%252F96587d2f-7dea-48ef-87c8-0ed74bd00239.png%252F930x520.png?signature=dkg2rtRNEtQd20FSap-Ou2bYYnM=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" data-fragment=\"m!3c0b\" data-image=\"https://mondrian.mashable.com/uploads%252Fvideo_uploaders%252Fdistribution_thumb%252Fimage%252F95136%252F96587d2f-7dea-48ef-87c8-0ed74bd00239.png%252Foriginal.png?signature=b1YV-6FyrAnr5O_ELQa8Vr7FPPE=&source=https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com\" data-micro=\"1\">\n</p></div>\n\n</div>\n\n\n\n</section></div>",
"length" : 8417,
"excerpt" : "Samsung's newest earbuds are less fun to look at, but better to use.",
"siteName" : "Mashable",
"tickers" : [
{
"_id" : ObjectId("6001eb414110e6001cb5b227"),
"name" : "Apple Inc.",
"symbol" : "AAPL",
"exchange" : "nasdaq"
}
],
"cryptos" : [
{
"_id" : ObjectId("6001eb414110e6001cb5b228"),
"name" : "Strong",
"symbol" : "STRONG"
},
{
"_id" : ObjectId("6001eb414110e6001cb5b229"),
"name" : "Tap",
"symbol" : "XTP"
}
],
"meta" : {
"title" : "Samsung takes on Apple's AirPods Pro with new Galaxy Buds Pro",
"thumb" : "https://mondrian.mashable.com/2021%252F01%252F15%252Fdd%252Fac21db42899b4219ab9b4e35fc74c043.002a7.jpg%252F1200x630.jpg?signature=I4q1zJF08njPyxmiZgxnHrSsTsM="
},
"url" : "https://mashable.com/review/samsung-galaxy-buds-pro-review/",
"host" : "mashable.com",
"createdAt" : ISODate("2021-01-15T19:21:37.931Z"),
"updatedAt" : ISODate("2021-01-15T19:21:37.931Z"),
"__v" : 0
}

Answer 1

You can use $group stage after $match stage,
$group by url and get first root document using $$ROOT, this will return document in root field
{
$group: {
_id: "$url",
root: { $first: "$$ROOT" }
}
}
this is optional, if you want to replace this root document to root, try $replaceRoot,
{ $replaceRoot: { newRoot: "$root" } }
Your final query would be,
const res = await ReadSchema.aggregate([
{ $match: { "tickers.symbol": { $in: stocks } } },
{
$group: {
_id: "$url",
root: { $first: "$$ROOT" }
}
},
{ $sort: { "root.createdAt": -1 } },
{ $limit: 100 },
{
$project: {
content: 0,
htm: 0
}
},
{ $replaceRoot: { newRoot: "$root" } }
])
Playground

Answer 2

await ReadSchema.aggregate([
{ $match: { 'tickers.symbol': { $in: stocks } } },
{
$project: {
content: 0,
htm: 0,
},
},
{
$group: {
_id: '$url',
root: { $first: '$$ROOT' },
},
},
{ $sort: { 'root.createdAt': -1 } },
{ $limit: 50 },
{ $replaceRoot: { newRoot: '$root' } },
]).allowDiskUse(true);
```