How to convert to ObjectId and match dates on MongoDB lookup? - mongodb

I am grabbing an id from a nested value in a schema, then using that to lookup the id from another table and also matching on a couple dates from that table. I've tried a regular match/lookup/unwind/match and also a lookup/let/pipeline technique. In both cases, it ignores matching on the date for some reason. What am I missing?
Here is one method for reference. I'm not sure where to put the sort either since it doesn't seem to pull $meeting out to sort on.
EXAMPLE RECORDS
PRODUCT
{
"_id" : ObjectId("5f36c0df6d5553e6af208cac"),
"items" : [
{
"paramType" : "Meeting",
"paramValue" : "5f36c0df6d5553e6af208cab"
}
],
"ownerId" : ObjectId("12345678901234567")
}
MEETING
{
"_id" : ObjectId("5f36c0df6d5553e6af208cab"),
"startDate" : ISODate("2020-08-18T10:00:00.000+0000"),
"endDate" : ISODate("2020-08-18T11:00:00.000+0000")
}
AGGREGATE
db.getCollection("products").aggregate(
[
{
$match: {
"ownerId": ObjectId("12345678901234567")
}
},
{
$unwind: "$items"
},
{
$lookup: {
from: "meetings",
let: { "meetingId": '$items.paramValue' },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ["$_id", "$$meetingId"] },
{
$eq: ["meeting.startDate", {
"$gte": ["$meeting.startDate", ISODate("2020-08-01T00:00:00.000Z")]
}]
},
{
$eq: ["meeting.endDate", {
"$lte": ["$meeting.endDate", ISODate("2020-08-31T23:59:59.999Z")]
}]
}
],
},
},
},
],
as: "meeting"
}
},
{
$unwind: "$meeting"
},
{
$project: {
"_id": 1,
"items": 1,
"meeting": "$meeting"
}
},
{
$sort: {
'meeting.startDate': 1
}
},
]
);
It might be because item.paramValue is not converted to an ObjectId before the lookup. But can't figure out how to convert it inside an aggregate. I tried this, but no go
{
$addFields: {
"convertedMeetingId": { $toObjectId: "$items.paramValue" }
}}
let: { "meetingId": "$convertedMeetingId" }

There are quick fixes in $lookup other looks good,
let you can convert meetingId string to ObjectId here using $toObjectId
$gte and $lte, you used $meeting.startDate and $meeting.endDate it should be $startDate and $endDate because you are already inside meeting lookup.
i am not sure why you have used $eq and with $gte and $lte, if i am not wrong i have corrected and removed $eq it will work directly.
{
$lookup: {
from: "meetings",
let: { meetingId: { $toObjectId: "$items.paramValue" } },
pipeline: [
{
$match: {
$expr: {
$and: [
{ $eq: ["$_id", "$$meetingId"] },
{ $gte: ["$startDate", ISODate("2020-08-01T00:00:00.000Z")] },
{ $lte: ["$endDate", ISODate("2020-08-31T23:59:59.999Z")] }
]
}
}
}
],
as: "meeting"
}
},
Playground

Related

MongoDB aggregation matching ObjectId against string

I have the following document which is also available in the mongo playground at:
https://mongoplayground.net/p/zhcoi1BF0Ny
db={
MyCollectionOne: [
{
"firstId": "10",
"secondId": "123456789012345678901234"
},
{
"firstId": "11",
"secondId": "999999999999999999999999"
}
],
MyCollectionTwo: [
{
"_id": ObjectId("123456789012345678901234"),
"otherFieldOne": "Some Data",
"otherFieldTwo": [
{
someNumber: 7
}
]
},
{
"_id": ObjectId("999999999999999999999999"),
"otherFieldOne": "Some Other Data",
"otherFieldTwo": [
{
someNumber: 9
},
{
someNumber: 39
}
]
}
]
}
Given a firstId, I need to determine the correct MyCollectionTwo entry to return. So for example, if I was given a firstId of 11, I would use that to lookup its corresponding secondId (which is "999999999999999999999999"). I would need to convert the secondId value to an ObjectId and then look through the MyCollectionTwo _id fields until I find the matching one.
I gave it a try and am very close but cannot figure out how to correctly do the string->objectId conversion.
db.MyCollectionTwo.aggregate([
{
$lookup: {
from: "MyCollectionOne",
localField: "_id",
foreignField: "secondId",
as: "Temp"
}
},
{
$unwind: "$Temp"
},
{
$match: {
"Temp.firstId": "11"
}
},
{
$project: {
_id: 1,
otherFieldOne: 1,
otherFieldTwo: 1
}
}
]).find()
I am aware there is a let/pipeline which can use a $toObjectId but I can't get that working in the above context.
Any help would be appreciated. Thanks!
Your $lookup with pipeline should be as below:
$lookup: {
from: "MyCollectionOne",
let: {
id: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
{
$toObjectId: "$secondId"
},
"$$id"
]
}
}
}
],
as: "Temp"
}
Sample Mongo Playground
You can try this
db.MyCollectionTwo.aggregate([
{
$lookup: {
//searching collection name
from: "MyCollectionOne",
//setting variable [searchId] where your string converted to ObjectId
let: {
"searchId": {
$toObjectId: "$secondId"
}
},
//search query with our [searchId] value
"pipeline": [
//searching [searchId] value equals your field [_id]
{
"$match": {
"$expr": [
{
"_id": "$$searchId"
}
]
}
},
],
as: "Temp"
}
},
{
$unwind: "$Temp"
},
{
$match: {
"Temp.firstId": "11"
}
},
{
$project: {
_id: 1,
otherFieldOne: 1,
otherFieldTwo: 1
}
}
]).find()
https://mongoplayground.net/p/es0j0AiPDCj

Mongo DB Join on Primary/Foreign Key

I have two collections, viz: clib and mp.
The schema for clib is : {name: String, type: Number} and that for mp is: {clibId: String}.
Sample Document for clib:
{_id: ObjectId("6178008397be0747443a2a92"), name: "c1", type: 1}
{_id: ObjectId("6178008397be0747443a2a91"), name: "c2", type: 0}
Sample Document for mp:
{clibId: "6178008397be0747443a2a92"}
{clibId:"6178008397be0747443a2a91"}
While Querying mp, I want those clibId's that have type = 0 in clib collection.
Any ideas how this can be achieved?
One approach that I can think of was to use $lookUp, but that doesnt seem to be working. Also, I m not sure if this is anti-pattern for mongodb, another approach is to copy the type from clib to mp while saving mp document.
If I've understood correctly you can use a pipeline like this:
This query get the values from clib where its _id is the same as clibId and also has type = 0. Also I've added a $match stage to not output values where there is not any coincidence.
db.mp.aggregate([
{
"$lookup": {
"from": "clib",
"let": {
"id": "$clibId"
},
"pipeline": [
{
"$match": {
"$expr": {
"$and": [
{
"$eq": [
{
"$toObjectId": "$$id"
},
"$_id"
]
},
{
"$eq": [
"$type",
0
]
}
]
}
}
}
],
"as": "result"
}
},
{
"$match": {
"result": {
"$ne": []
}
}
}
])
Example here
db.mp.aggregate([
{
$lookup: {
from: "clib",
let: {
clibId: "$clibId"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{
$eq: [ "$_id", "$$clibId" ],
}
]
}
}
},
{
$project: { type: 1, _id: 0 }
}
],
as: "clib"
}
},
{
"$unwind": "$clib"
},
{
"$match": {
"clib.type": 0
}
}
])
Test Here

MongoDb return records only if $lookup cond is occur

I have a class model which has field ref.
I'm trying to fetch only records that match the condition in lookup.
so what i did:
{
$lookup: {
from: 'fields',
localField: "field",
foreignField: "_id",
as: 'FieldCollege',
},
},
{
$addFields: {
"FieldCollege": {
$arrayElemAt: [
{
$filter: {
input: "$FieldCollege",
as: "field",
cond: {
$eq: ["$$field.level", req.query.level]
}
}
}, 0
]
}
}
},
The above code works fine and returning the FieldCollege if the cond is matched.
but the thing is, i wanted to return the class records only if the FieldCollege is not empty.
I'm totally new to mongodb. so i tried something like this:
{
$match: {
'FieldCollege': { $exists: true, $ne: [] }
}
},
Obv this didn't work.
does mongodb support something like this or am i complicating things?
EDIT:
the result from the above code:
"Classes": [
{
"_id": "613245664c6ea614e001fcef",
"name": "test",
"language": "en",
"year_cost": "3232323",
"FieldCollege":[] // with $unwind
}
],
expected Result:
"Classes": [
// FieldCollege is empty
],
I think the good option is to use lookup with pipeline, and see the final version of your query,
$lookup with fields collection and match your both conditions
$limit to result one document
$match FieldCollege is not empty []
$addElemAt to get first element from result FieldCollege
[
{
$lookup: {
from: "fields",
let: { field: "$field" },
pipeline: [
{
$match: {
$and: [
{ $expr: { $eq: ["$$field", "$_id"] } },
{ level: req.query.level }
]
}
},
{ $limit: 1 }
],
as: "FieldCollege"
}
},
{ $match: { FieldCollege: { $ne: [] } } },
{
$addFields: {
FieldCollege: { $arrayElemAt: ["$FieldCollege", 0] }
}
}
]

Join multiple collections after parallel aggregation in Mongodb

I have a collection called "Reel" which has embedded Objects.
{
"_id":"reel_1",
"category":[
{
"_id" : "cat_1",
"videos": [ {"_id":"vid_1"},{"_id":"vid_2"} ] //_id is reference of Video collection
},
{
"_id" : "cat_2",
"videos": [ {"_id":"vid_3"},{"_id":"vid_4"} ]
}
]
}
Video is another collection whose _id is referred inside reel-> category -> videos -> _id
{
"_id":"vid_1",
"title":"title 1",
"groups":[{"_id":"group_1"},{"_id":"group_2"}]
},
{
"_id":"vid_2",
"title":"title 2",
"groups":[{"_id":"group_1"},{"_id":"group_4"}]
},
{
"_id":"vid_3",
"title":"title 3",
"groups":[{"_id":"group_1"},{"_id":"group_2"}]
},
{
"_id":"vid_4",
"title":"title 4",
"groups":[{"_id":"group_3"},{"_id":"group_4"}]
}
The Document collection which holds _id of Reel and _id of Category
{
"_id":"doc_1",
"title":"document title",
"assessments":[
{
"reel":"reel_1", // reel reference _id
"category":"cat_1", // category reference _id
"groups":[{"_id":"group_1"},{"_id":"group_2"}
]
}
]
}
I need to join and find all related embedded Objects which has group_1.
I have done joining between Reel collection and Video collection and working fine,
{ $unwind: { path: '$category', preserveNullAndEmptyArrays: true }},
{ $unwind: { path: '$category.videos', preserveNullAndEmptyArrays: true }},
{
$lookup: {
from: 'video',
localField: 'category.videos._id',
foreignField: '_id',
as: 'joinVideo'
}
},
{ $unwind: { path: "$joinVideo", preserveNullAndEmptyArrays: true }},
{ $unwind: { path: "$joinVideo.groups", preserveNullAndEmptyArrays: true }},
{ $match: { "joinVideo.groups._id": "group_1" }},
{ $addFields: { "category.videos": "$joinVideo" }},
{
$group: {
_id: {
_id: "$_id",
category: "$category._id"
},
videos: {
$addToSet: "$category.videos"
}
}
}, {
$group: {
_id: "$_id._id",
category: {
$addToSet: {
"_id": "$_id.category",
"videos": "$videos"
}
}
}
}
The document collection should be embedded inside the category object based on reel _id and and category _id filtered by group_1. My expected result is
{
"_id":"reel_1",
"category":[
{
"_id" : "cat_1",
"videos": [
{
"_id":"vid_1",
"title":"title 1",
"groups":[ {"_id":"group_1"},{"_id":"group_2"}]
},
{
"_id":"vid_2",
"title":"title 2",
"groups":[{"_id":"group_1"},{"_id":"group_4"}]
}
],
"documents":[
{ // this document comes by reel="reel_1", category="cat_1", filtered by "group_1"
"_id":"doc_1",
"title":"document title",
}
]
},
{
"_id" : "cat_2",
"videos": [
{
"_id":"vid_3",
"title":"title 3",
"groups":[{"_id":"group_1"},{"_id":"group_2"}]
}
]
}
]
}
I tried in many ways. Since I'm new to Mongodb, I couldn't sort this out.
Since MongoDB v3.6, $lookup allows perform uncorrelated sub-queries. This allows us perform non-standard queries to join two or more collections.
Note: Explanation why we need to use $expr inside $lookup pipeline
Explanation
We apply $unwind to flatten $category
We perform $lookup with 2 conditions:
video.groups._id == 'group_1' and video._id in reel.category.videos._id
Since $reel.category.videos._id returns an array, we need to use $in operator
Again we perform $lookup with 2 conditions. It creates documents field for every document
To remove fields dynamically, we need to use Aggregation expressions called $$REMOVE which allows us exclude conditionally a field from document
We perform $group stage to transform into desired result
db.reel.aggregate([
{
$unwind: {
path: "$category",
preserveNullAndEmptyArrays: true
}
},
{
$lookup: {
from: "video",
let: {
videos: "$category.videos._id"
},
pipeline: [
{
$match: {
"groups._id": "group_1",
$expr: {
$in: [
"$_id",
"$$videos"
]
}
}
}
],
as: "category.videos"
}
},
{
$lookup: {
from: "document",
let: {
reel_id: "$_id",
category_id: "$category._id"
},
pipeline: [
{
$match: {
$expr: {
$and: [
{
$in: [
"$$reel_id",
"$assessments.reel"
]
},
{
$in: [
"$$category_id",
"$assessments.category"
]
}
]
}
}
},
{
$project: {
_id: 1,
title: 1
}
}
],
as: "category.documents"
}
},
{
$addFields: {
"category.documents": {
$cond: [
{
$eq: [
{
$size: "$category.documents"
},
0
]
},
"$$REMOVE",
"$category.documents"
]
}
}
},
{
$group: {
_id: "$_id",
category: {
$push: "$category"
}
}
}
])
MongoPlayground

mongodb aggregate apply a function to a field

As part of an aggregate I need to run this transformation:
let inheritances = await db.collection('inheritance').aggregate([
{ $match: { status: 1 }}, // inheritance active
{ $project: { "_id":1, "name": 1, "time_trigger": 1, "signers": 1, "tree": 1, "creatorId": 1, "redeem": 1, "p2sh": 1 } },
{ $lookup:
{
from: "user",
let: { creatorId: { $concat: [ "secretkey", { $toString: "$creatorId" } ] }, time_trigger: "$time_trigger"},
pipeline: [
{ $match:
{ $expr:
{ $and:
[
{ $eq: [ "$_id", sha256( { $toString: "$$creatorId" } ) ] },
{ $gt: [ new Date(), { $add: [ { $multiply: [ "$$time_trigger", 24*60*60*1000 ] }, "$last_access" ] } ] },
]
}
}
},
],
as: "user"
},
},
{ $unwind: "$user" }
]).toArray()
creatorId comes from a lookup, and in order to compare it to _id I first need to do a sha256.
How can I do it?
Thanks.
External functions will not work with the aggregation framework. Everything is parsed to BSON by default. It is all basically processed from BSON operators to native C++ code implementation, This is by design for performance.
Basically in short, you can't do this. I recommend just storing the hashed value on every document as a new field, otherwise you'll have to do it in code just before the pipeline.