I have a pipeline that works great for what I need... but I think there is some redundant data that can be removed from the pipeline.
Expected output
This is what I want the output to look like
{
"_id": "5ecee2189fdd1b0004056936",
"name": "Mike",
"history": [
{
"_id": "5ecb263c166b8500047c1411",
"what": "Log IN"
},
{
"_id": "5ecb263c166b8500047c1422",
"what": "Log OUT"
}
]
}
Current output
This is what the output currently looks like
{
"docs": [
{
"_id": "5ecee2189fdd1b0004056936",
"name": "Mike",
"history": {
"_id": "5ecb263c166b8500047c1411",
"what": "Log IN"
},
"historyIndex": 0
},
{
"_id": "5ecee2189fdd1b0004056936",
"name": "Mike",
"history": {
"_id": "5ecb263c166b8500047c1422",
"what": "Log OUT"
},
"historyIndex": 1
}
]
}
User doc
In real life there will be more users than this... of course...
{
"_id": "5ecee2189fdd1b0004056936",
"name": "Mike",
}
History docs
again, to make it simple, I am keeping data short
[
{
"_id": "5ecb263c166b8500047c1411",
"userId": "5ecee2189fdd1b0004056936",
"what": "Log IN"
},
{
"_id": "5ecb263c166b8500047c1422",
"userId": "5ecee2189fdd1b0004056999",
"what": "Log IN"
},
{
"_id": "5ecb263c166b8500047c1433",
"userId": "5ecee2189fdd1b0004056936",
"what": "Log OUT"
},
{
"_id": "5ecb263c166b8500047c1444",
"userId": "5ecee2189fdd1b0004056999",
"what": "Log OUT"
}
]
mongoose-aggregate-paginate-v2 middleware
I am also using mongoose-aggregate-paginate-v2, but I don't think that is my issue, but it definitely comes into play when the results are returned. it needs to have the docs flattened so it can count and paginate them:
"totalDocs": 941,
"limit": 500,
"page": 1,
"totalPages": 2,
"pagingCounter": 1,
"hasPrevPage": false,
"hasNextPage": true,
"prevPage": null,
"nextPage": 2
Pipeline
Here is my pipeline
var agg_match = {
$match:
{
_id: mongoose.Types.ObjectId(userId)
}
};
var agg_lookup = {
$lookup: {
from: 'it_userhistories',
localField: '_id',
foreignField: 'userId',
as: 'history'
}
}
var agg_unwind = {
$unwind: {
path: "$history",
preserveNullAndEmptyArrays: true,
includeArrayIndex: 'historyIndex',
}
}
var agg = [
agg_match,
agg_lookup,
agg_unwind,
agg_project,
];
var pageAndLimit = {
page:page,
limit:limit
}
User.aggregatePaginate(myAggregate, pageAndLimit)
You can use $map operator to do this. Following query will be helpful (I have not included the match stage in the pipeline, you can easily include it):
db.user.aggregate([
{
$lookup: {
from: "history",
localField: "_id",
foreignField: "userId",
as: "history"
}
},
{
$project: {
name: 1,
history: {
$map: {
input: "$history",
as: "h",
in: {
_id: "$$h._id",
what: "$$h.what"
}
}
}
}
}
])
MongoPLayGroundLink
Related
I have the following data model:
{
"_id": {
"$oid": "63b6da81661f0ecd23cd9830"
},
"Plan": [
{
"_id": {
"$oid": "63b6311e0871625f7ceb85ad"
},
"Name": "Straight ankle lock",
"Date": {
"$date": {
"$numberLong": "1672725600000"
}
},
"Notes": "Christian taught ankle locks",
"TeamId": {
"$oid": "63a291ebb60592854e23b8fb"
}
}
],
"User": [
{
"_id": {
"$oid": "6240fd2ee1335b45680bee9d"
},
"FirstName": "Test",
"LastName": "User",
"TeamId": {
"$oid": "639fd03bb31c7995a9d4b28c"
}
}
]
}
And I'd like to show a new object via aggregation that looks like:
{
"_id": {
"$oid": "63b6da81661f0ecd23cd9830"
},
"PlanName": "Straight ankle lock",
"UserName": "Test User"
}
I've been trying to figure this out for a few days, but at this point not sure if it is even possible. Any ideas?
Thanks.
Newer model based on Ray's input using project:
{
"_id": {
"$oid": "63b6da81661f0ecd23cd9830"
},
"InsertDate": {
"$date": {
"$numberLong": "1672927873507"
}
},
"Plan": {
"Name": "Straight ankle lock"
},
"User": {
"FirstName": "Adam",
"LastName": "Gusky"
},
"Team": {
"TeamName": "GB2 No Gi"
}
}
The query I'm using to get the above data:
[
{
$lookup: {
from: "Plans",
localField: "PlanId",
foreignField: "_id",
as: "Plan",
},
},
{
$lookup: {
from: "Teams",
localField: "TeamId",
foreignField: "_id",
as: "Team",
},
},
{
$lookup: {
from: "Users",
localField: "UserId",
foreignField: "_id",
as: "User",
},
},
{
$project: {
Plan: {
$first: "$Plan",
},
User: {
$first: "$User",
},
Team: {
$first: "$Team",
},
InsertDate: 1,
},
},
{
$project: {
"Plan.Name": 1,
"User.FirstName": 1,
"User.LastName": 1,
"Team.TeamName": 1,
InsertDate: 1,
},
},
]
You can simply set the value you want in the $project stage.
db.collection.aggregate([
{
$project: {
_id: 1,
PlanName: {
$first: "$Plan.Name"
},
UserName: {
"$concat": [
{
"$first": "$User.FirstName"
},
" ",
{
"$first": "$User.LastName"
}
]
}
}
}
])
Mongo Playground
I have Orders collection and iam getting the data from it as shown below:
[
{
"_id": "628216b7b30bb8aa80c8fd1a",
"promotionsDetails": {
"companyTotalPrice": 27,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 27,
"companyId": "621dd85eb45ca2ae292d9a36"
},
{
"_id": "621de063bb5f9f0bf510897d",
"price": 19,
"companyId": "621dd85eb45ca2ae292d9a32"
}
]
}
},
{
"_id": "628214fcb30bb8aa80c8fd18",
"promotionsDetails": {
"companyTotalPrice": 46,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 46,
"companyId": "621dd85eb45ca2ae292d9a32",
}
]
},
}
]
what I am trying to do is to get the company details from the companies collection using the companyId objectId in each object in the array, like below:
[
{
"_id": "628216b7b30bb8aa80c8fd1a",
"promotionsDetails": {
"companyTotalPrice": 27,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 27,
"companyId": "621dd85eb45ca2ae292d9a36",
"companyData": { "title": "..." }
},
{
"_id": "621de063bb5f9f0bf510897d",
"price": 19,
"companyId": "621dd85eb45ca2ae292d9a32",
"companyData": { "title": "..." }
}
]
}
},
{
"_id": "628214fcb30bb8aa80c8fd18",
"promotionsDetails": {
"companyTotalPrice": 46,
"promotionsData": [
{
"_id": "621de063bb5f9f0bf510897f",
"price": 46,
"companyId": "621dd85eb45ca2ae292d9a32",
"companyData": { "title": "..." }
}
]
}
}
]
i have tried to use lookup and pipeline, but I'm not getting the desired result, thanks!
Actuality $lookup supports arrays, so there is no need to $unwind and change the structure. This will return your expected results:
db.Orders.aggregate([
{
$lookup: {
from: "Company",
localField: "promotionsDetails.promotionsData.companyId",
foreignField: "_id",
as: "companyfullData"
}
},
{
$set: {
"promotionsDetails.promotionsData": {
$map: {
input: "$promotionsDetails.promotionsData",
in: {
$mergeObjects: [
"$$this",
{
companyData: {
$arrayElemAt: [
"$companyfullData",
{$indexOfArray: ["$companyfullData.id", "$$this.id"]}
]
}
}
]
}
}
}
}
},
{$unset: "companyfullData"}
])
Playground
Here, to make it more clear take a look at Mongo playground
db.Orders.aggregate([
{
$unwind: "$promotionsDetails.promotionsData"
},
{
"$lookup": {
"from": "Company",
"localField": "promotionsDetails.promotionsData.companyId",
"foreignField": "_id",
"as": "promotionsDetails.promotionsData.companyData"
}
},
])
Let's say i have 2 collections
// Post collection:
[
{
"_id": "somepost1",
"author": "firstuser",
"title": "First post"
},
{
"_id": "somepost2",
"author": "firstuser",
"title": "Second post"
},
{
"_id": "somepost3",
"author": "firstuser",
"title": "Third post"
}
]
// User collection:
[
{
"_id": "firstuser",
"nickname": "John",
"posts": {
"voted": []
}
},
{
"_id": "seconduser",
"nickname": "Bob",
"posts": {
"voted": [
{
"_id": "somepost1",
"vote": "1"
},
{
"_id": "somepost3",
"vote": "-1"
}
]
}
}
]
And i need to get this result:
[
{
"_id": "somepost1",
"author": {
"_id": "firstuser",
"nickname": "John"
},
"title": "First post",
"myvote": "1"
},
{
"_id": "somepost2",
"author": {
"_id": "firstuser",
"nickname": "John"
},
"title": "Second post",
"voted": "0"
},
{
"_id": "somepost3",
"author": {
"_id": "firstuser",
"nickname": "John"
},
"title": "Third post",
"myvote": "-1"
}
]
How can i make a request with aggregation, which will display this output with dynamic _id of elements?
I have problem with using current _id of post in second $match and setting "myvote" to 0 if there are no element in "posts.voted" associated with current post.
Here what i've tried: https://mongoplayground.net/p/v70ZUioVSpQ
db.post.aggregate([
{
$match: {
author: "firstuser"
}
},
{
$lookup: {
from: "user",
localField: "author",
foreignField: "_id",
as: "author"
}
},
{
$addFields: {
author: {
$arrayElemAt: [
"$author",
0
]
}
}
},
{
$lookup: {
from: "user",
localField: "_id",
foreignField: "posts.voted._id",
as: "Results"
}
},
{
$unwind: "$Results"
},
{
$unwind: "$Results.posts.voted"
},
{
$match: {
"Results.posts.voted._id": "ID OF CURRENT POST"
}
},
{
$project: {
_id: 1,
author: {
_id: 1,
nickname: 1
},
title: 1,
myvote: "$Results.posts.voted.vote"
}
}
])
From the $match docs:
The query syntax is identical to the read operation query syntax
The query syntax does not allow usage of document values. which is what you're trying to do.
What we can do is use $expr within the $match stage, this allows us to use aggregation oprerators, thus also giving access to the document values. like so:
{
$match: {
$expr: {
$eq: ['$Results.posts.voted._id', '$_id'],
}
},
},
I have a problem with how to lookup nested array, for example i have 4 collections.
User Collection
"user": [
{
"_id": "1234",
"name": "Tony",
"language": [
{
"_id": "111",
"language_id": "919",
"level": "Expert"
},
{
"_id": "111",
"language_id": "920",
"level": "Basic"
}
]
}
]
Language Collection
"language": [
{
"_id": "919",
"name": "English"
},
{
"_id": "920",
"name": "Chinese"
}
]
Job
"job": [
{
"_id": "10",
"title": "Programmer",
"location": "New York"
}
],
CvSubmit Collection
"cvsubmit": [
{
"_id": "11",
"id_user": "1234",
"id_job": "11"
}
]
And my query aggregation is:
db.cvsubmit.aggregate([
{
$lookup: {
from: "user",
localField: "id_user",
foreignField: "_id",
as: "id_user"
}
},
{
$lookup: {
from: "language",
localField: "id_user.language.language_id",
foreignField: "_id",
as: "id_user.language.language_id"
}
},
])
But the result is:
[
{
"_id": "11",
"id_job": "11",
"id_user": {
"language": {
"language_id": [
{
"_id": "919",
"name": "English"
},
{
"_id": "920",
"name": "Chinese"
}
]
}
}
}
]
I want the result like this, also showing all user data detail like name:
[
{
"_id": "11",
"id_job": "11",
"id_user": {
"_id": "1234",
"name": "Tony"
"language": [
{
"_id": "919",
"name": "English",
"Level": "Expert"
},
{
"_id": "920",
"name": "Chinese",
"level": "Basic"
}
]
}
}
]
Mongo Playground link https://mongoplayground.net/p/i0yCucjruey
Thanks before.
$lookup with user collection
$unwind deconstruct id_user array
$lookup with language collection and return in languages field
$map to iterate look of id_user.language array
$reduce to iterate loop of languages array returned from collection, check condition if language_id match then return name
db.cvsubmit.aggregate([
{
$lookup: {
from: "user",
localField: "id_user",
foreignField: "_id",
as: "id_user"
}
},
{ $unwind: "$id_user" },
{
$lookup: {
from: "language",
localField: "id_user.language.language_id",
foreignField: "_id",
as: "languages"
}
},
{
$addFields: {
languages: "$$REMOVE",
"id_user.language": {
$map: {
input: "$id_user.language",
as: "l",
in: {
_id: "$$l._id",
level: "$$l.level",
name: {
$reduce: {
input: "$languages",
initialValue: "",
in: {
$cond: [
{ $eq: ["$$this._id", "$$l.language_id"] },
"$$this.name",
"$$value"
]
}
}
}
}
}
}
}
}
])
Playground
You database structure is not accurate as per NoSQL, there should be max 2 collections, loot of join using $lookup and $unwind will cause performance issues.
I'm trying to build a voting system where you can have X num of options to vote for on an entry.
The query I'm building now is when retrieving an entry, I would like to get the numbers of votes per option on an entry.
I have a very clear understanding of how I would do this in SQL but grasping to understand the concepts of aggregation, lookup, and group in MongoDB
The model looks like this:
Entries
{
"_id": "5fc2765938401a2308e18ac5",
"options": [
{
"name": "First Option"
"_id": "5fc2765938401a2308e18ac6",
},
{
"name": "Second Option"
"_id": "5fc2765938401a2308e18are",
},
{
"name": "Third Option"
"_id": "5fc2765938401a2308e18aef",
},
],
},
{
"_id": "5fc2766438401a2308e18ac8",
"options": [
{
"name": "Some other option"
"_id": "5fc2766438401a2308e18ac9",
},
{
"_id": "5fc2766438401a2308e18aca",
"name": "This is also an option"
}
],
}
Votes
{
"_id": "5fc2765938401a2308e18ac5",
"entryId": "5fc2765938401a2308e18ac6"
},
{
"_id": "5fc2765938401a2308e18aer",
"entryId": "5fc2765938401a2308e18are"
},
{
"_id": "5fc2765938401a2308e18ek",
"entryId": "5fc2765938401a2308e18ac6"
}
...
And I want the results of Entry to look like this.
{
"_id": "5fc2765938401a2308e18ac5",
"options": [
{
"name": "First Option"
"_id": "5fc2765938401a2308e18ac6",
"votes": 1,
},
{
"name": "Second Option"
"_id": "5fc2765938401a2308e18are",
"votes": 0,
},
{
"name": "Third Option"
"_id": "5fc2765938401a2308e18aef",
"votes": 5,
},
],
},
{
"_id": "5fc2766438401a2308e18ac8",
"options": [
{
"name": "Some other option"
"_id": "5fc2766438401a2308e18ac9",
"votes": 3,
},
{
"_id": "5fc2766438401a2308e18aca",
"name": "This is also an option"
"votes": 10,
}
],
}
$lookup to join votes collection, pass local field optoins._id and foreign field entryId
$project get options votes, $map to iterate loop of options array, $filter to get matching entryId records and $size to get count of element in return array, merge votes field and current object using $mergeObjects
db.entries.aggregate([
{
$lookup: {
from: "votes",
localField: "options._id",
foreignField: "entryId",
as: "votes"
}
},
{
$project: {
options: {
$map: {
input: "$options",
as: "a",
in: {
$mergeObjects: [
"$$a",
{
votes: {
$size: {
$filter: {
input: "$votes",
cond: { $eq: ["$$this.entryId", "$$a._id"] }
}
}
}
}
]
}
}
}
}
}
])
Playground