I have an issue with a MongoDB query that I am having challenges with. In my class, I have a structure of [instance].events.destination.estimatedDateTime, and the value of estimatedDateTime is an ISO Date (eg. 2014-09-23 21:48:00.000Z). We are using org.springframework.data.mongodb.core.query.Criteria to append criteria to our query like this:
criteria = criteria.and("events.destination.estimatedDateTime").gte(today)
When this is generated, it looks like this when evaluated at runtime:
"events.destination.estimatedDateTime" : { "$gte" : { "$date" : "2014-10-28T05:00:00.000Z"} }
Unfortunately that is returning no records. I have taken that same condition and attempted to pull records in RoboMongo, but again there are no results. However, I have found that when I do the following in RoboMongo, I do indeed get results
"events.destination.estimatedDateTime" : { $gte : ISODate("2014-10-28T05:00:00.000Z") }
Is there a way using the Criteria class to generate ISODate(date) rather than "$date":date?
Related
I am building a website using Next.js and MongoDB. On one of my website page, I have implemented filters to help search for products. To retrieve and update the filters (update item count each time a filter is changing), I have an api endpoint which query my MongoDB Collection. This specific collection contains ~200.000 items. Each item have several fields such as brand, model, place etc...
I have 9 fields which I use to filter and thus must fetch through my api each time there's a change. Therefore I have 9 queries running through my api, on for each field/filter and the query on MongoDB looks like :
var models = await db_collection
.aggregate([
{
$match: {
$and: [filter],
},
},
{
$group: { _id: '$model', count: { $sum: 1 } },
},
{ $sort: { _id: 1 } },
])
.toArray();
The problem is that, as 9 queries are running, the update of the page (mainly due to the queries) takes ~4secs which is too long. I would like to reach <1sec. I would like to now if there is a good practice I am missing such as doing one query instead of one for each filter or maybe a database optimization on my database.
Thank you,
I have tried using a $project argument before $groupon aggregate pipeline for the query to reduce the number of field returned, using distinct and then sorting instead of aggregate but none of these solutions seem to improve efficiency.
EDIT :
As suggested by R2D2, I am posting the structure of a document on MongoDB in my collection :
{
_id : ObjectId('example_id')
source : string
date : date
brand : string
family : string
model : string
size : string
color : string
condition : string
contact : string
SKU : string
}
Depending on the pages, I query unique values of each field of interest (source, date, brand, family, model, size, color, condition, contact) and their count depending on filters (e.g. Number for each unique values of model for selected brands, I also query documents based on specific values of these fields.
As mentioned, you indexes are important and if you are querying by those field I recomand to create compound indexes, see here for indexes optimisation : https://learnmongodbthehardway.com/schema/indexes/
As far as the aggregation pipeline goes, nothing is out of the ordinary, but this specific aggregation just return the number of items per model matching the criteria, not the matching document. If it is all the data you need you might find it usefull to create a new collection when you perform pre-caculation for common search daily (how many items have the color black, ...) this way, when the page loads, you don't have to look in you 200k+ items, but just in your pre-calculated statistical collection. Schedule a cron task or use a lambda function to invoke a route on your api that will calculate all your stats once a day and upsert them in a new collection.
Also I believe the "and" is useless useless since you can use the implicit $and. You can look for an object like :
{
color : {$in : ['BLACK', 'BLUE']},
size : 3
}
rather than :
[{color : 'BLACK'}, {color : 'BLUE'}, {size : 3}]
Reserve the explicit $and for when you really need it.
I have a collection that contains the following fields: agentId, postBalance, preBalance, etc. I want to fetch the last unique record for an agent that contains the field stated earlier based on a date filter.
db.transaction.find(
{
"createdAt" : {
"$gte": ISODate("2022-09-01T00:00:00Z"),
"$lt": ISODate("2022-09-02T00:00:00Z")
}
},
{
“agentId”: 1,
“walletBalance”: 1
}
)
The query above returns duplicate values and not the latest one. How best do I optimise this query. I am using Mongo Compass so I don't mind any query that comes in that format. I have read up on $last, $natural but they don't seem to solve my issue.
Have you tried to add sort by "createdAt" and limit of 1, or just using findOne method with same sort?
I'm trying to order my collection using the following query:
db.getCollection('trip').find().sort({'itinerary.0.timestamp': 1})
The result is not being correctly sorted, however I exported the full collection to a local mongoDB database and the same query works like a charm. In order to perform that sort in cosmos DB I had to create the index 'itinerary.0.timestamp'.
data example:
{
"_id" : ObjectId("6087104ca68f171ce7715448"),
"tripId" : NumberLong(38533184),
"itinerary" : [
{
"transId" : NumberLong(39800097),
"timestamp" : NumberLong(1619372446291)
},
{
"transId" : NumberLong(39800576),
"timestamp" : NumberLong(1619372446321)
},
],
"results" : [],
"tripTimeSent" : ISODate("2021-04-29T14:44:53.253Z")
}
What am I missing?
Thanks!!
The solution was to create a new field, itiTimestamp, outside the array containing the value 'itinerary.0.timestamp'. Then just order by itiTimestamp
It's true that you need to create an index for the sort field. Here's the doc related:
To apply a sort to a query, you must create an index on the fields
used in the sort operation.
==========================================
I've tested in my side, after creating wildcard index on itinerary, sort query could be executed but has no luck. I also refer to this answer(new BasicDBObject("labels.0.value", 1)) and this one(db.testCollection.find().sort({"someArray.0": 1})), they all don't work for the date format Op provided.
But when I added a properity "score":[20,55,80] in each item in the collection, I found it can be sorted by the first item when sort by score directly.
I assume that this feature hasn't supported.
Mongodb contains a collection that has following objects,
Objects with lastTrip :
{
"usage" : 0,
"lastMaintenanceDate" : NumberLong(1572600811104),
"lastTrip" : {
"direction" : "DOWN",
"startLevel" : 5
}
}
Objects without lastTrip:
{
"usage" : 0,
"lastMaintenanceDate" : NumberLong(1572600811104),
}
And following query is used to retrieve data with sorting,
Query one:
db.getCollection('some_collection').find({}).sort({"lastTrip.startLevel":1})
This only returns objects with the field lastTrip, even though collection contains objects without that field.
This issue is causing me getting wrong count on following query,
Query two:
db.getCollection('some_collection').find({}).sort({"lastTrip.startLevel":1}).count()
Expected result:
Query one > 20 documents
Query two > 20
Actual result:
Query one > 15 documents
Query two > 20
What is the proper way of querying(for query one) to get objects without the field lastTrip as well?
2 days old to Mongo, so bear with me.
I have a collection from which, I only want to retrieve specific values contingent to another key existing in the MongoDB environment.
Here is what I am doing:
db.results.find({'someKeyThatShouldExist':{$exists:true}}, {"parentKey.childKey.theKeyWoseValueIwant":1}
This yields data in the following format for me:
{ "_id" : ObjectId("532a2c2b6803fa486b8b456a"), "parentKey" : { "childKey" : { "theKeyWhoseValueIWant" : 102982577 }}}.....
Now, all I really want is the value 102982577, not everything else.
How can I do this ?
You can suppress the _id by adding _id:0 to the projection criteria.
db.results.find(
{"someKeyThatShouldExist":{$exists:true}},
{_id:0, "parentKey.childKey.theKeyWoseValueIwant":1}
)
To get just the value, you could do something like:
db.results.find(
{"someKeyThatShouldExist":{$exists:true}},
{_id:0, "parentKey.childKey.theKeyWoseValueIwant":1}
)[0].parentKey.childKey.theKeyWoseValueIwant