MongoDB retrun array of Items from other arrays, with reduced datapoints - mongodb

I have a user object that looks (roughly) like the example below - a user containg Arrays of Objects for "autos" "boats" and "planes" - each object COULD have an image URL, but many do not, I want an array of ALL the image URLs form a user (regardless of which group they are in) and I want SOME but NOT ALL of the object that is associated with that image.
DATA MODEL
{
user: STRING,
autos: [
{
make: STRING,
model: STRING,
year: NUMBER,
price: NUMBER,
image: STRING
}
],
boats: [
{
make: STRING,
model: STRING,
year: NUMBER,
price: NUMBER,
image: STRING
}
],
planes: [
{
make: STRING,
model: STRING,
year: NUMBER,
price: NUMBER,
image: STRING
}
]
}
This is a reduced "object" in each of these arrays, the real data contains many points that are specific to each of these TYPES, and I cannot change the model at this point.
So I am able to pull back an array of Image URLs from the arrays, and concat them into a single array, But then I just have URLS... what I want is the "make" and "model" and "image URL" for each
Here is the Query I've got
query = [
{
$match: matchCriteria
},
{
$project: {
_id: 0,
image: {
$filter: {
input: {
$concatArrays: [
'$autos.image',
'$boats.image',
'$planes.image'
]
},
cond: {
$and: [ // skip records without image data
{ $eq: [{ $type: "$$this" }, "string"] },
{ $ne: ["$$this", ""] }
]
}
}
}
}
}
];
this gives me data looking like this
"data": [
"https://www.yourPics.com/see?image=image4",
"https://www.yourPics.com/see?image=image5",
"https://www.yourPics.com/see?image=image6",
"https://www.yourPics.com/see?image=image1",
"https://www.yourPics.com/see?image=image2",
"https://www.yourPics.com/see?image=image3",
"https://www.yourPics.com/see?image=image7",
"https://www.yourPics.com/see?image=image8",
"https://www.yourPics.com/see?image=image9",
"https://www.yourPics.com/see?image=image10",
"https://www.yourPics.com/see?image=image11"
],
But I WANT something that looks like this
"data": [
{
make:"some make4",
model:"some model4",
image: "https://www.yourPics.com/see?image=image4"
},
{
make:"some make5",
model:"some model5",
image: "https://www.yourPics.com/see?image=image5"
},
{
make:"some make6",
model:"some model6",
image: "https://www.yourPics.com/see?image=image6"
},
...
],
Any help is greatly appreciated.

I'm not sure why you're concating just the image attribute of those objects. Doing that, you will certainly end up with just an array of images. Why not concat the entire arrays? As in:
$concatArrays: ["$autos", "$boats"]
So I have put this Playground together for you. Using this project:
{
$project: {
items: {
$filter: {
input: {
$concatArrays: [
"$autos",
"$boats"
]
},
as: "items",
cond: {
$and: [
{
$ne: [
"$$items.image",
undefined
]
},
{
$ne: [
"$$items.image",
""
]
}
]
}
}
}
}
}
You will get this result:
[
{
"_id": ObjectId("5a934e000102030405000000"),
"items": [
{
"image": "https://pont.com",
"make": "Pontiac",
"model": "Sunfire",
"year": "2004"
},
{
"image": "https://aluma.com",
"make": "Aluma",
"model": "X4",
"price": "100,000",
"year": "2021"
}
]
}
]
From here, excluding the keys you don't want (e.g year, price) is a trivial matter. Just project only what you want:
{
$project: {
_id: 0,
"items.make": 1,
"items.model": 1,
"items.image": 1
}
}
Playground: https://mongoplayground.net/p/qFIwLhuXR84

Related

How to map on array fields with a dynamic variable in MongoDB, while projection (aggregation)

I want to serve data from multiple collections, let's say product1 and product2.
Schemas of both can be referred to as -:
{ amount: Number } // other fields might be there but not useful in this case.
Now after multiple stages of aggregation pipeline, I'm able to get the data in the following format-:
items: [
{
amount: 10,
type: "product1",
date: "2022-10-05"
},
{
amount: 15,
type: "product2",
date: "2022-10-07"
},
{
amount: 100,
type: "product1",
date: "2022-10-10"
}
]
However, I want one more field added to each element of items - The sum of all the previous amounts.
Desired Result -:
items: [
{
amount: 10,
type: "product1",
date: "2022-10-05",
totalAmount: 10
},
{
amount: 15,
type: "product2",
date: "2022-10-07",
totalAmount: 25
},
{
amount: 100,
type: "product1",
date: "2022-10-10",
totalAmount: 125
}
]
I tried adding another $project stage, which goes as follows -:
{
items: {
$map: {
input: "$items",
in: {
$mergeObjects: [
"$$this",
{ totalAmount: {$add : ["$$this.amount", 0] } },
]
}
}
}
}
This just appends another field, totalAmount as the sum of 0 and the amount of that item itself.
I couldn't find a way to make the second argument (currently 0) in {$add : ["$$this.amount", 0] } as a variable (initial value 0).
What's the way to perform such action in MongoDb aggregation pipeline ?
PS-: I could easily perform this action by a later mapping in the code itself, but I need to add limit (for pagination) to it in the later stage.
You can use $reduce instead of $map for this:
db.collection.aggregate([
{$project: {
items: {
$reduce: {
input: "$items",
initialValue: [],
in: {
$concatArrays: [
"$$value",
[{$mergeObjects: [
"$$this",
{totalAmount: {$add: ["$$this.amount", {$sum: "$$value.amount"}]}}
]}]
]
}
}
}
}}
])
See how it works on the playground example

Convert String field to an Encoded String in MongoDB

My requirement is to write a Mongo aggregation which returns a List of "virtual" Documents by grouping some existing "actual" Documents from the collection.
I intend to use this result as-is on my UI project, I'm looking for ways I can add a unique and decodable ID to it during the aggregation itself.
Example:
[
{... pipeline stages},
{
$group: {
_id: {
bookCode: '$bookCode',
bookName: '$bookName'
}
books: {
$push: '$bookId'
}
}
},
{
$project: {
//virtual unique Id by combining bookCode and bookName
virtualId: {
$concat: [
{
$ifNull: [ '$_id.bookCode', '~' ]
},
'-',
{
$ifNull: [ '$_id.bookName', '~' ]
}
]
},
books: '$books'
}
}
]
Sample Output:
[
{
virtualId: 'BC01-BOOKNAME01'
books: ['BID01', 'BID02']
},
{
virtualId: 'BC02-BOOKNAME01'
books: ['BID03', 'BID04']
},
{
virtualId: '~-BOOKNAME01'
books: ['BID05', 'BID06']
},
{
virtualId: 'BC02-~'
books: ['BID07', 'BID08']
},
{
virtualId: '~-~'
books: ['BID09', 'BID10']
},
]
This method of concatenating grouping fields to generate virtualId works, but is there a way to make it more terse?
Perhaps some way I could convert this to an unreadable by human but decodable format.
TLDR: I'm looking for ways to create an ID for each result document in the aggregation query itself, that would give back it's contributing fields if I decode it later.
MongoDB Version: 4.0.0
use this aggregation we use funtion and generate code with js function
db.collection.aggregate([
{
"$project": {
books: 1,
virtualId: {
"$function": {
"body": "function(a){var t = '';for(i=0;i<a.length;i++){t=a.charCodeAt(i)+t;};return t;}",
"args": [
"$virtualId"
],
"lang": "js"
}
}
}
}
])
https://mongoplayground.net/p/Lm_VjIG54BG

Combine $facet result with subdocument and then conditionally excluding document

Suppose after the $facet stage I have a result including two arrays: roomInfo and hotelInfo.
Which looks like this:
{
"roomInfo": [
{
_id: 'ab1',
booked: 3
},
{
_id: 'ab2',
booked: 1
}
],
"hotelInfo": [
{
name: 'Radison Blue',
roomDetails: [
{
_id: 'ab1',
roomCount: 5
},
{
_id: 'xy1',
roomCount: 5
}
]
},
{
name: 'Intercontinental',
roomDetails: [
{
_id: 'ab2',
roomCount: 5
}
]
}
]
};
Expected Result
I want an output like this:
[
{
name: 'Radison Blue',
roomDetails: [
{
_id: 'ab1',
roomCount: 5,
booked: 3
},
{
_id: 'xy1',
roomCount: 5,
booked: 0
}
]
},
{
name: 'Intercontinental',
roomDetails: [
{
_id: 'ab2',
roomCount: 5,
booked: 1
}
]
}
];
Basically, adding the booked property from roomInfo into the hotelInfo's roomDetails field after matching their ids.
Additionally, after getting the above output result I want to exclude those hotels on which all the rooms(not for a single room) have the value of fields roomCount and booked equal. I want to do this inside the aggregation pipeline stage as I will have to deal with $skip and $limit later on.
How to achieve these use cases?
Thanks!
Basically the approach will be iterating over the hotels and matching each room accordingly, here is a quick working code sample:
db.collection.aggregate([
{
$unwind: "$hotelInfo"
},
{
$project: {
name: "$hotelInfo.name",
"roomDetails": {
$filter: {
input: {
$map: {
input: "$hotelInfo.roomDetails",
as: "info",
in: {
"$mergeObjects": [
"$$info",
{
"$arrayElemAt": [
{
$filter: {
input: "$roomInfo",
as: "room",
cond: {
$eq: [
"$$room._id",
"$$info._id"
]
}
}
},
0
]
}
]
}
}
},
as: "proccessedInfo",
cond: {
$ne: [
"$$proccessedInfo.roomCount",
"$$proccessedInfo.booked"
]
}
}
}
}
}
])
Mongo Playground
With that said you mention you'd like to paginate calls in the future. the current approach does not seem scaleable, because these are "real data points" aka hotels it's fine if your scale is somewhat capped ( no more than several thousands hotels ). but if it's not I recommend you ask another question with the entire pipeline you have so we can adjust it to work better.

How to avoid adding duplicate objects to an array in MongoDB

this is my schema:
new Schema({
code: { type: String },
toy_array: [
{
date:{
type:Date(),
default: new Date()
}
toy:{ type:String }
]
}
this is my db:
{
"code": "Toystore A",
"toy_array": [
{
_id:"xxxxx", // automatic
"toy": "buzz"
},
{
_id:"xxxxx", // automatic
"toy": "pope"
}
]
},
{
"code": "Toystore B",
"toy_array": [
{
_id:"xxxxx", // automatic
"toy": "jessie"
}
]
}
I am trying to update an object. In this case I want to update the document with code: 'ToystoreA' and add an array of subdocuments to the array named toy_array if the toys does not exists in the array.
for example if I try to do this:
db.mydb.findOneAndUpdate({
code: 'ToystoreA,
/*toy_array: {
$not: {
$elemMatch: {
toy: [{"toy":'woddy'},{"toy":"buzz"}],
},
},
},*/
},
{
$addToSet: {
toy_array: {
$each: [{"toy":'woddy'},{"toy":"buzz"}],
},
},
},
{
new: false,
}
})
they are added and is what I want to avoid.
how can I do it?
[
{
"code": "Toystore A",
"toy_array": [
{
"toy": "buzz"
},
{
"toy": "pope"
}
]
},
{
"code": "Toystore B",
"toy_array": [
{
"toy": "jessie"
}
]
}
]
In this example [{"toy":'woddy'},{"toy":"buzz"}] it should only be added 'woddy' because 'buzz' is already in the array.
Note:when I insert a new toy an insertion date is also inserted, in addition to an _id (it is normal for me).
As you're using $addToSet on an object it's failing for your use case for a reason :
Let's say if your document look like this :
{
_id: 123, // automatically generated
"toy": "buzz"
},
{
_id: 456, // automatically generated
"toy": "pope"
}
and input is :
[{_id: 789, "toy":'woddy'},{_id: 098, "toy":"buzz"}]
Here while comparing two objects {_id: 098, "toy":"buzz"} & {_id: 123, "toy":"buzz"} - $addToSet consider these are different and you can't use $addToSet on a field (toy) in an object. So try below query on MongoDB version >= 4.2.
Query :
db.collection.updateOne({"_id" : "Toystore A"},[{
$addFields: {
toy_array: {
$reduce: {
input: inputArrayOfObjects,
initialValue: "$toy_array", // taking existing `toy_array` as initial value
in: {
$cond: [
{ $in: [ "$$this.toy", "$toy_array.toy" ] }, // check if each new toy exists in existing arrays of toys
"$$value", // If yes, just return accumulator array
{ $concatArrays: [ [ "$$this" ], "$$value" ] } // If No, push new toy object into accumulator
]
}
}
}
}
}])
Test : aggregation pipeline test url : mongoplayground
Ref : $reduce
Note :
You don't need to mention { new: false } as .findOneAndUpdate() return old doc by default, if you need new one then you've to do { new: true }. Also if anyone can get rid of _id's from schema of array objects then you can just use $addToSet as OP was doing earlier (Assume if _id is only unique field), check this stop-mongoose-from-creating-id-property-for-sub-document-array-items.

Issues with lookup and match multipe collections

Having issues with aggregate and lookup in multiple stages. The issue is that I cannot match by userId In the last lookup. If I omit the { $eq: ['$userId', '$$userId'] } it works and match by the other criteria. But not by the userid.
I've tried added pools as a let and use it as { $eq: ['$userId', '$$pools.userId'] } in the last stage but that doesn't work either. I get an empty coupon array.
I get this with the below query. I think I need to use $unwind in some way? But haven't got that to work yet. Any pointers?
There is three collections total to be joined. First the userModel, it should contain pools and then the pools should contain a users coupons.
{
"userId": "5df344a1372f345308dac12a", // Match this usedId with below userId coming from the coupon
"pools": [
{
"_id": "5e1ebbc6cffd4b042fc081ab",
"eventId": "id999",
"eventStartTime": "some date",
"trackName": "tracky",
"type": "foo bar",
"coupon": []
}
]
},
I need the coupon array to be filled with the correct data (below) which has a matching userId in it.
"coupon": [
{
"eventId": "id999",
"userId": "5df344a1372f345308dac12a", // This userId need to match the above one
"checked": true,
"pool": "a pool",
}
poolProject:
const poolProject = {
eventId: 1,
eventStartTime: 1,
trackName: 1,
type: 1,
};
Userproject:
const userProjection = {
_id: {
$toString: '$_id',
},
paper: 1,
correctBetsLastWeek: 1,
correctBetsTotal: 1,
totalScore: 1,
role: 1,
};
The aggregate query
const result = await userModel.aggregate([
{ $project: userProjection },
{
$match: {
$or: [{ role: 'User' },
{ role: 'SuperUser' }],
},
},
{ $addFields: { userId: { $toString: '$_id' } } },
{
$lookup: {
from: 'pools',
as: 'pools',
let: { eventId: '$eventId' },
pipeline: [
{ $project: poolProject },
{
$match: {
$expr: {
$in: ['$eventId', eventIds],
},
},
},
{
$lookup: {
from: 'coupons',
as: 'coupon',
let: { innerUserId: '$$userId' },
pipeline: [
{
$match: {
$expr: {
$eq: ['$userId', '$$innerUserId'],
},
},
},
],
},
},
],
},
},
]);
Thanks for any input!
Edit:
If i move the second lookup (coupon) so they are in the same "level" it works but i would like to have it inside of the pool. If I add as: 'pools.coupon', in the last lookup it overwrites the lookedup pool data.
When you access fields with the $$ prefix it means they are defined as "special" system variables by Mongo.
We don't know exactly how Mongo the magic happens but you're naming two variables with the same name, which causes a conflict as it seems.
So either remove userId: '$userId' from the first lookup as you're not even using it.
Or rename or second userId: '$userId' a different name like innerUserId: '$userId' to avoid conflicts when you access it.
Just dont forget to change { $eq: ['$userId', '$$userId'] } to { $eq: ['$userId', '$$innerUserId'] } after.
EDIT:
Now that its clear theres no field userId in pools collection just change the variable in the second lookup collection from:
let: { innerUserId: '$userId' } //userId does not exist in pools.
To:
let: { innerUserId: '$$userId' }