$addField with subQuery in MongoDB aggregation - mongodb

I have 2 collections: Chargers and Reservation. I would like for the reservations in the next 7 days to be appended as a field when querying for chargers (i.e. time range from now until the same day next week).
The model for chargers looks like this:
{
"name":"charger 2",
"address":"test, test, USA",
"current_type": 1,
"charge_level" : 2,
"plug_type" : 1,
}
and the reservation model looks like this:
{
"charger_id": ObjectId("test ID"),
"start_time": "31-01-2021",
"end_time": "1-02-2021"
}
I found the aggregation $addField that adds a field within an aggregation pipeline, but I was wondering if I could use the $addfield on a "subquery". Essentiallly, for each charger that match, get all the reservations within a given time range where the chargerID is the one from the match, and add the array of reservations as a field. The resulting model would look like this:
{
"name":"charger 2",
"address":"test, test, USA",
"current_type": 1,
"charge_level" : 2,
"plug_type" : 1,
"reservations" : [
...
]
}
The current way of getting the data is with 2 queries at the application result, but that becomes quite taxing with the network latencies.
Query 1 - chargers:
{"_id":<charger_id>}
once I get the result, I query again.
Query 2 - Reservations:
{
"charger_id": <charger_id>,
"start_time" : {
"$lte" : <date in a week>
}
}
This is not too bad with a get by ID, because I already have the ID before querying for the charger in the first place, but with a getAll, or any other query that doesn't query by ID, it can get really taxing.

To bring in data from another collection you need to use $lookup. $addFields can be used to add fields based on data you already have in the result set.

Related

Next.js/MongoDB - Query Optimization

I am building a website using Next.js and MongoDB. On one of my website page, I have implemented filters to help search for products. To retrieve and update the filters (update item count each time a filter is changing), I have an api endpoint which query my MongoDB Collection. This specific collection contains ~200.000 items. Each item have several fields such as brand, model, place etc...
I have 9 fields which I use to filter and thus must fetch through my api each time there's a change. Therefore I have 9 queries running through my api, on for each field/filter and the query on MongoDB looks like :
var models = await db_collection
.aggregate([
{
$match: {
$and: [filter],
},
},
{
$group: { _id: '$model', count: { $sum: 1 } },
},
{ $sort: { _id: 1 } },
])
.toArray();
The problem is that, as 9 queries are running, the update of the page (mainly due to the queries) takes ~4secs which is too long. I would like to reach <1sec. I would like to now if there is a good practice I am missing such as doing one query instead of one for each filter or maybe a database optimization on my database.
Thank you,
I have tried using a $project argument before $groupon aggregate pipeline for the query to reduce the number of field returned, using distinct and then sorting instead of aggregate but none of these solutions seem to improve efficiency.
EDIT :
As suggested by R2D2, I am posting the structure of a document on MongoDB in my collection :
{
_id : ObjectId('example_id')
source : string
date : date
brand : string
family : string
model : string
size : string
color : string
condition : string
contact : string
SKU : string
}
Depending on the pages, I query unique values of each field of interest (source, date, brand, family, model, size, color, condition, contact) and their count depending on filters (e.g. Number for each unique values of model for selected brands, I also query documents based on specific values of these fields.
As mentioned, you indexes are important and if you are querying by those field I recomand to create compound indexes, see here for indexes optimisation : https://learnmongodbthehardway.com/schema/indexes/
As far as the aggregation pipeline goes, nothing is out of the ordinary, but this specific aggregation just return the number of items per model matching the criteria, not the matching document. If it is all the data you need you might find it usefull to create a new collection when you perform pre-caculation for common search daily (how many items have the color black, ...) this way, when the page loads, you don't have to look in you 200k+ items, but just in your pre-calculated statistical collection. Schedule a cron task or use a lambda function to invoke a route on your api that will calculate all your stats once a day and upsert them in a new collection.
Also I believe the "and" is useless useless since you can use the implicit $and. You can look for an object like :
{
color : {$in : ['BLACK', 'BLUE']},
size : 3
}
rather than :
[{color : 'BLACK'}, {color : 'BLUE'}, {size : 3}]
Reserve the explicit $and for when you really need it.

How to get the last unique transactions from a collection in MongoDB

I have a collection that contains the following fields: agentId, postBalance, preBalance, etc. I want to fetch the last unique record for an agent that contains the field stated earlier based on a date filter.
db.transaction.find(
{
"createdAt" : {
"$gte": ISODate("2022-09-01T00:00:00Z"),
"$lt": ISODate("2022-09-02T00:00:00Z")
}
},
{
“agentId”: 1,
“walletBalance”: 1
}
)
The query above returns duplicate values and not the latest one. How best do I optimise this query. I am using Mongo Compass so I don't mind any query that comes in that format. I have read up on $last, $natural but they don't seem to solve my issue.
Have you tried to add sort by "createdAt" and limit of 1, or just using findOne method with same sort?

exclude row by maping id and fetch records by mapping name in mongodb

Guys help me out mongodb query that I want write that.
skip the row by id and find by name matching row. I put my query below suggest me the right way
db.college.find({"_id" : {$ne: ObjectId("5b965696b8bba46518cbefda")}},{ $and:[{ "Name" : "KK" }]})
here am getting all rows of records. but I want to get only matching recor
Both conditions should be in one object like this:
db.college.find({"_id" : { $ne: ObjectId("5b965696b8bba46518cbefda")}, "Name": "KK" })
Second parameter of find represents projection

Execution time of a query - MongoDB

I have two collections: coach and team.
Coach collection contains information about coaches like name, surname, age and an array coached_Team that contains the _id of the team that a coach coached.
The team collection contains data about teams like _id, common name, official name, country, championship....
If I want to find, for example, the official name of all teams coached by Allegri, I have to do two queries, the first on coach collection:
>var x = db.coach.find({surname:"Allegri"},{_id:0, "coached_Team.team_id":1})
>var AllegriTeams
>while(x.hasNext()) AllegriTeams=x.next()
{
"coached_Team" : [
{
"team_id" : "Juv.26
},
{
"team_id" : "Mil.74
},
{
"team_id" : "Cag.00
}
]
}
>AllegriTeams=AllegriTeams.coached_Team
[
{
"team_id" : "Juv.26"
},
{
"team_id" : "Mil.74"
},
{
"team_id" : "Cag.00"
}
]
And then I have to execute three queries on team collection:
> db.team.find({ _id:AllegriTeams[0].team_id}, {official_name:1,_id:0})
{official_name : "Juventus Football Club S.p.A."}
> db.team.find({ _id:AllegriTeams[1].team_id}, {official_name:1,_id:0})
{official_name : "Associazione Calcio Milan S.p.A"}
> db.team.find({ _id:AllegriTeams[2].team_id}, {official_name:1,_id:0})
{official_name:"Cagliari Calcio S.p.A"}
Now consider I have about 100k documents on collection team and collection coach. The first query, on coach collection, needs about 71 ms plus the time of while cycle. The three queries on team collection, using cursor.explain("executionStats") needs 0 ms. I don't understand why this query takes 0.
I need executionTimeMillis of these three queries to have the execution time of the query "find official names of all teams coached by Allegri". I want to add the execution time of the query on coach collection(71ms) with the execution time of these three. If the time of these three queries is 0 what can I say about the execution time of the query mainly?
I think the more important observation here is that 71ms is a long time for a simple fetch of one item. Looks like your "surname" field needs an index. The other "three" queries are simple lookups of a primary key, which is why they are relatively fast.
db.coach.createIndex({ "surname": 1 })
If that surname is actually "unique" then add that too:
db.coach.createIndex({ "surname": 1 },{ "unique": true })
You can also simplify your "three" queries as as one by simply mapping the array, and applying the $in operator:
var teamIds = [];
db.coach.find(
{ "surname": "Allegri" },
{ "_id":0, "coached_Team.team_id":1 }
).forEach(function(coach) {
teamIds = coach.coached_Team.map(function(team) {
return team.team_id }).concat(teamIds);
});
});
db.team.find(
{ "_id": { "$in": teamIds" }},
{ "official_name": 1, "_id": 0 }
).forEach(function(team) {
printjson(team);
});
And then certainly the overall execution time is way down, as well as removing the overhead of multiple operations down to just the two queries requried.
Also remembering here that despite what is in the execution plan stats, the more queries to make to and from the server then the longer the overal real time execution will be for making each request and retriving the data. So it is best to keep things as minimal as possible.
Therefore even more logical would be that where to "need" this information regularly, storing the "coach name" on the "team itself" ( and indexing that data ) leads to the fastest possible response and only a single query operation.
It's easy to get caught up in observing execution stats. But really, think of what is "best" and "fastest" as a pattern for the sort of queries you want to do.

mongodb: inc high level document and embedded document

I want to increment two values, one in high level document and one in embedded document:
{
studentId: "x1"
numberOfAttending: 2
courses: [
{
courseId:"y1"
numberOfAttending: 1
},
{
courseId:"y2"
numberOfAttending: 1
}
]
}
How could i inc the number of attending for student and for the course (upsert). and could i do it with a single update query ?
That's going to be tough since courses is an array. You'll need to know the index of the course you want to update, then do something like:
{ '$inc' : {numberOfAttending : 1, 'courses.1.numberOfAttending' : 1}}
Have you thought about switching it to a single embedded doc with courseId as a key for each course? If so, you can run a command like this to increment both. This doesn't depend on position so it's going to be less fragile:
{ '$inc' : { numberOfAttending : 1, 'courses.y2.numberOfAttending' : 1}}