-
Hi , i have an aggregation query with lookup, i need to project specific fields from this lookup and slice them. This is what I've done so far.
{
$lookup: {
from: 'users',
localField: 'users',
foreignField: '_id',
as: 'users',
}},
I've added the unwind statement
{
$unwind: {
path: '$users',
preserveNullAndEmptyArrays: true
}},
I've added the group statement
{
$group: {
_id: {
_id: '$_id',
createdAt: '$createdAt',
updatedAt: '$updatedAt'
},
users: {
$addToSet: '$users',
}
}
},
And to project specific fields in array of users i did:
{
$project: {
_id: '$_id._id',
createdAt: '$_id.createdAt',
updatedAt: '$_id.updatedAt',
// users: {
// $slice: [
// "$users",
// skip,
// limit
// ]
// },
users: {
$map: {
input: '$users',
as: 'user',
in: {
email: '$$user.email',
name: '$$user.name',
username: '$$user.username',
updatedAt: '$$user.updatedAt'
}
}
}
}},
My question is , How can i use $slice in this scope ?
I don't know how 'legit' is this but , I've added fields to $addToSet statement in $group , so now i can use $slice with mapped fields.
{
$group: {
_id: {
_id: '$_id',
createdAt: '$createdAt',
updatedAt: '$updatedAt',
},
users: {
$addToSet: {
_id: '$users._id',
email: '$users.email',
name: '$users.name',
username: '$users.username',
updatedAt: '$users.updatedAt'
}
}
}
}
Now i can easily do $slice in $project statement.
{
$project: {
_id: '$_id._id',
users: {
$slice: [
"$users",
skip,
limit
]
}
},
}
If someone has a better solution , i would like to know.
Related
I use Atlas Search to return a list of documents (using Mongoose):
const searchResults = await Resource.aggregate()
.search({
text: {
query: searchQuery,
path: ["title", "tags", "link", "creatorName"],
},
}
)
.match({ approved: true })
.addFields({
score: { $meta: "searchScore" }
})
.exec();
These resources can be up and downvoted by users (like questions on Stackoverflow). I want to boost the search score depending on these votes.
I can use the boost operator for that.
Problem: The votes are not a property of the Resource document. Instead, they are stored in a separate collection:
const resourceVoteSchema = mongoose.Schema({
_id: { type: String },
userId: { type: mongoose.Types.ObjectId, required: true },
resourceId: { type: mongoose.Types.ObjectId, required: true },
upDown: { type: String, required: true },
After I get my search results above, I fetch the votes separately and add them to each search result:
for (const resource of searchResults) {
const resourceVotes = await ResourceVote.find({ resourceId: resource._id }).exec();
resource.votes = resourceVotes
}
I then subtract the downvotes from the upvotes on the client and show the final number in the UI.
How can I incorporate this vote points value into the score of the search results? Do I have to reorder them on the client?
Edit:
Here is my updated code. The only part that's missing is letting the resource votes boost the search score, while at the same time keeping all resource-votes documents in the votes field so that I can access them later. I'm using Mongoose syntax but an answer with normal MongoDB syntax will work for me:
const searchResults = await Resource.aggregate()
.search({
compound: {
should: [
{
wildcard: {
query: queryStringSegmented,
path: ["title", "link", "creatorName"],
allowAnalyzedField: true,
}
},
{
wildcard: {
query: queryStringSegmented,
path: ["topics"],
allowAnalyzedField: true,
score: { boost: { value: 2 } },
}
}
,
{
wildcard: {
query: queryStringSegmented,
path: ["description"],
allowAnalyzedField: true,
score: { boost: { value: .2 } },
}
}
]
}
}
)
.lookup({
from: "resourcevotes",
localField: "_id",
foreignField: "resourceId",
as: "votes",
})
.addFields({
searchScore: { $meta: "searchScore" },
})
.facet({
approved: [
{ $match: matchFilter },
{ $skip: (page - 1) * pageSize },
{ $limit: pageSize },
],
resultCount: [
{ $match: matchFilter },
{ $group: { _id: null, count: { $sum: 1 } } }
],
uniqueLanguages: [{ $group: { _id: null, all: { $addToSet: "$language" } } }],
})
.exec();
It could be done with one query only, looking similar to:
Resource.aggregate([
{
$search: {
text: {
query: "searchQuery",
path: ["title", "tags", "link", "creatorName"]
}
}
},
{$match: {approved: true}},
{$addFields: {score: {$meta: "searchScore"}}},
{
$lookup: {
from: "ResourceVote",
localField: "_id",
foreignField: "resourceId",
as: "votes"
}
}
])
Using the $lookup step to get the votes from the ResourceVote collection
If you want to use the votes to boost the score, you can replace the above $lookup step with something like:
{
$lookup: {
from: "resourceVote",
let: {resourceId: "$_id"},
pipeline: [
{
$match: {$expr: {$eq: ["$resourceId", "$$resourceId"]}}
},
{
$group: {
_id: 0,
sum: {$sum: {$cond: [{$eq: ["$upDown", "up"]}, 1, -1]}}
}
}
],
as: "votes"
}
},
{$addFields: { votes: {$arrayElemAt: ["$votes", 0]}}},
{
$project: {
"wScore": {
$ifNull: [
{$multiply: ["$score", "$votes.sum"]},
"$score"
]
},
createdAt: 1,
score: 1
}
}
As you can see on this playground example
EDIT: If you want to keep the votes on the results, you can do something like:
db.searchResults.aggregate([
{
$lookup: {
from: "ResourceVote",
localField: "_id",
foreignField: "resourceId",
as: "votes"
}
},
{
"$addFields": {
"votesCount": {
$reduce: {
input: "$votes",
initialValue: 0,
in: {$add: ["$$value", {$cond: [{$eq: ["$$this.upDown", "up"]}, 1, -1]}]}
}
}
}
},
{
$addFields: {
"wScore": {
$add: [{$multiply: ["$votesCount", 0.1]}, "$score"]
}
}
}
])
As can be seen here
So I have 2 models user & form.
User Schema
firstName: {
type: String,
required: true,
},
lastName: {
type: String,
required: true,
},
email: {
type: String,
required: true,
}
Form Schema
approvalLog: [
{
attachments: {
type: [String],
},
by: {
type: ObjectId,
},
comment: {
type: String,
},
date: {
type: Date,
},
},
],
userId: {
type: ObjectId,
required: true,
},
... other form parameters
When returning a form, I'm trying to aggregate the user info of every user in the approvalLog into their respective objects as below.
{
...other form info
approvalLog: [
{
attachments: [],
_id: '619cc4953de8413b548f61a6',
by: '619cba9cd64af530448b6347',
comment: 'visit store for disburement',
date: '2021-11-23T10:38:13.565Z',
user: {
_id: '619cba9cd64af530448b6347',
firstName: 'admin',
lastName: 'user',
email: 'admin#mail.com',
},
},
{
attachments: [],
_id: '619cc4ec3ea3e940a42b2d01',
by: '619cbd7b3de8413b548f61a0',
comment: '',
date: '2021-11-23T10:39:40.168Z',
user: {
_id: '619cbd7b3de8413b548f61a0',
firstName: 'sam',
lastName: 'ben',
email: 'sb#mail.com',
},
},
{
attachments: [],
_id: '61a9deab8f472c52d8bac095',
by: '61a87fd93dac9b209096ed94',
comment: '',
date: '2021-12-03T09:08:59.479Z',
user: {
_id: '61a87fd93dac9b209096ed94',
firstName: 'john',
lastName: 'doe',
email: 'jd#mail.com',
},
},
],
}
My current code is
Form.aggregate([
{
$lookup: {
from: 'users',
localField: 'approvalLog.by',
foreignField: '_id',
as: 'approvedBy',
},
},
{ $addFields: { 'approvalLog.user': { $arrayElemAt: ['$approvedBy', 0] } } },
])
but it only returns the same user for all objects. How do I attach the matching user for each index?
I've also tried
Form.aggregate([
{
$lookup: {
from: 'users',
localField: 'approvalLog.by',
foreignField: '_id',
as: 'approvedBy',
},
},
{
$addFields: {
approvalLog: {
$map: {
input: { $zip: { inputs: ['$approvalLog', '$approvedBy'] } },
in: { $mergeObjects: '$$this' },
},
},
},
},
])
This adds the right user to their respective objects, but I can only add the to the root object and not a new one.
You can try the approach,
$map to iterate loop of approvalLog
$filter to iterate loop of approvedBy array and search for user id by
$arrayElemAt to get first element from above filtered result
$mergeObjects to merge current object properties of approvalLog and filtered user
$$REMOVE don't need approvedBy now
await Form.aggregate([
{
$lookup: {
from: "users",
localField: "approvalLog.by",
foreignField: "_id",
as: "approvedBy"
}
},
{
$addFields: {
approvalLog: {
$map: {
input: "$approvalLog",
as: "a",
in: {
$mergeObjects: [
"$$a",
{
user: {
$arrayElemAt: [
{
$filter: {
input: "$approvedBy",
cond: { $eq: ["$$a.by", "$$this._id"] }
}
},
0
]
}
}
]
}
}
},
approvedBy: "$$REMOVE"
}
}
])
Playground
The second approach using $unwind,
$unwind deconstruct the approvalLog array
$lookup with user collection
$addFields and $arrayElemAt to get first element from lookup result
$group by _id and reconstruct the approvalLog array and get first value of other required properties
await Form.aggregate([
{ $unwind: "$approvalLog" },
{
$lookup: {
from: "users",
localField: "approvalLog.by",
foreignField: "_id",
as: "approvalLog.user"
}
},
{
$addFields: {
"approvalLog.user": {
$arrayElemAt: ["$approvalLog.user", 0]
}
}
},
{
$group: {
_id: "$_id",
approvalLog: { $push: "$approvalLog" },
userId: { $first: "$userId" },
// add your other properties like userId
}
}
])
Playground
Given following collections.
I have to get the title field and combine to identifier
CREDENTIAL:
{
_id: ..
title: ..
}
USER_CREDENTIAL:
{
_id: ..
credential_id: .. (from credential collection)
created_at: ..
identifier: {
first_name: ..
middle_name: ..
last_name: ..
}
}
The response should be:
{
user_credential_id:
member: {
first_name:
middle_name:
last_name:
title:
created_at:
}
}
$lookup - Join user and user_credential_id with a pipeline to match the condition ($match) and decorate the document ($project).
$unwind - Deconstruct member array to multiple documents.
$project - Decorate the output document.
db.user.aggregate([
{
"$lookup": {
"from": "user_credential",
let: {
"user_credential_id": "$_id",
"title": "$title"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$credential_id",
"$$user_credential_id"
]
}
}
},
{
$project: {
first_name: "$identifier.first_name",
middle_name: "$identifier.middle_name",
last_name: "$identifier.last_name",
title: "$$title",
created_at: "$created_at",
}
}
],
"as": "member"
}
},
{
$unwind: "$member"
},
{
$project: {
_id: 0,
user_credential_id: "$_id",
member: 1
}
}
])
Sample Mongo Playground
You can try this query
db.user.aggregate([
{
$lookup: {
from: "credentials",
localField: "credential_id",
foreignField: "_id",
as: "user_credential"
}
},
{
$unwind: "$user_credential"
},
{
$project: {
_id: 0,
user_credential_id: "$credential_id",
member: {
first_name: "$identifier.first_name",
middle_name: "$identifier.middle_name",
last_name: "$identifier.last_name",
title: "$user_credential.title",
created_at: "$created_at",
}
}
}
])
You can check it out here
I have two collections:
Users: [{
_id: 'xxx',
name: 'xxx',
},
...
]
Posts: [{
_id: 'xxx',
userId: 'xxx',
},
...
]
So a user can have multiple posts. I want to get users with the number of posts that each user has. If the user doesn't have any post, it should load 0.
So the result will be:
[{
_id: 'xxx',
name: 'xxx',
numberOfPosts: 'xxx'
},
...
]
Below query is what I have wrote:
$lookup: {
from: 'Posts',
let: {
userId: '$_id',
},
pipeline: [{
$match: {
$expr: {
$eq: ["$userId", "$$userId"]
}
}
}, {
$count: 'posts'
}, {
$project: {
posts: {
$cond: [{
$ifNull: ['$posts', true]
},
'$posts',
0
]},
}
}],
as: 'posts'
}
It doesn't give me 0 if a user doesn't have any post. What was wrong in the query?
You can try simple way,
use $lookup without pipeline
$addFields to count the number of posts using $size
db.Users.aggregate([
{
$lookup: {
from: "Posts",
localField: "_id",
foreignField: "userId",
as: "numberOfPosts"
}
},
{
$addFields: {
numberOfPosts: {
$size: "$numberOfPosts"
}
}
}
])
Playground
The code example is in Mongo Playground
https://mongoplayground.net/p/W4Qt4oX0ZRP
Assume the following documents
[
{
_id: "5df1e6f75de2b22f8e6c30e8",
user: {
name: "Tom",
sex: 1,
age: 23
},
dream: [
{
label: "engineer",
industry: "5e06b16fb0670d7538222909",
type: "5e06b16fb0670d7538222951",
},
{
label: "programmer",
industry: "5e06b16fb0670d7538222909",
type: "5e06b16fb0670d7538222951",
}
],
works: [
{
name: "any engineer",
company: "5dd7fd51b0ae1837a08d00c8",
skill: [
"5dc3998e2cf66bad16efd61b",
"5dc3998e2cf66bad16efd61e"
],
},
{
name: "any programmer",
company: "5dd7fd9db0ae1837a08d00e2",
skill: [
"5dd509e05de2b22f8e67e1b7",
"5dd509e05de2b22f8e67e1bb"
],
}
]
}
]
I tried to use aggregate $lookup $unwind
db.coll.aggregate([
{
$unwind: {
path: "$dream",
}
},
{
$lookup: {
from: "industry",
localField: "dream.industry",
foreignField: "_id",
as: "dream.industry"
},
},
{
$unwind: {
path: "$dream.industry",
}
},
{
$lookup: {
from: "type",
localField: "dream.type",
foreignField: "_id",
as: "dream.type"
},
},
{
$unwind: {
path: "$dream.type",
}
},
{
$unwind: {
path: "$works",
}
},
{
$lookup: {
from: "company",
localField: "works.company",
foreignField: "_id",
as: "works.company"
},
},
{
$unwind: {
path: "$works.company",
}
},
{
$lookup: {
from: "skill",
localField: "works.skill",
foreignField: "_id",
as: "works.skill"
},
},
])
Executing the above code did not get the desired result!
This is what i expect
{
_id: "5df1e6f75de2b22f8e6c30e8",
user: {
name: 'Tom',
sex: 1,
age: 23
},
dream: [
{
label: 'engineer',
industry: {
_id: "5e06b16fb0670d7538222909", // Industry doc _id
name: 'IT',
createdAt: "2019-12-28T01:35:44.070Z",
updatedAt: "2019-12-28T01:35:44.070Z"
},
type: {
_id: "5e06b16fb0670d7538222951", // Type doc _id
name: 'job',
createdAt: "2019-12-28T01:35:44.070Z",
updatedAt: "2019-12-28T01:35:44.070Z"
},
},
{
label: 'programmer',
industry: {
_id: "5e06b16fb0670d7538222909", // Industry doc _id
name: 'IT',
createdAt: "2019-12-28T01:35:44.070Z",
updatedAt: "2019-12-28T01:35:44.070Z"
},
type: {
_id: "5e06b16fb0670d7538222951", // Type doc _id
name: 'job',
createdAt: "2019-12-28T01:35:44.070Z",
updatedAt: "2019-12-28T01:35:44.070Z"
}
}
],
works: [
{
name: 'any engineer',
company: {
_id: "5dd7fd51b0ae1837a08d00c8", // Company doc _id
name: 'alibaba',
area: 'CN',
},
skill: [
{
_id: "5dc3998e2cf66bad16efd61b", // Skill doc _id
name: 'Java'
},
{
_id: "5dc3998e2cf66bad16efd61e", // Skill doc _id
name: 'Php'
},
]
},
{
name: 'any programmer',
company: {
_id: "5dd7fd9db0ae1837a08d00e2", // Company doc _id
name: 'microsoft',
area: 'EN',
},
skill: [
{
_id: "5dd509e05de2b22f8e67e1b7", // Skill doc _id
name: 'Golang'
},
{
_id: "5dd509e05de2b22f8e67e1bb", // Skill doc _id
name: 'Node.js'
}
]
},
]
}
The expected result is dream is an array, works is an array, and dream.industry changed from ObjectId to document, dream.type changed from ObjectId to document, works.company changed from ObjectId to document
When I use populate, I can do it easily
Model.find()
.populate('dream.industry')
.populate('dream.type')
.populate('works.company')
.populate('works.skill')
.lean()
I refer to the following questions
mongoose aggregate lookup array (Almost the same as my question, But not resolved)
$lookup on ObjectId's in an array
hope to get everyone's help, thank you!
To make it easier i would not change the current pipeline but just add a $group stage to end of it in order to re-structure the data.
{
$group: {
_id: "$_id",
user: {$first: "$user"},
dream: {$addToSet: "$dream"},
works: {$addToSet: "$works"}
}
}
With that said if you are using Mongo version 3.6+ i do recommend you use the "newer" version of $lookup to re-write your pipeline to be a bit more efficient by avoiding all these $unwind's.