I want to create a partial index in MongoDB which excludes documents that are older than 30 days in the past from the current date.
Something like this (pseudo code):
partialFilterExpression: { date: { $gte: { $currentDate - 30 days }} }
The $currentDate should be the actual current date, in other words the date is a dynamic value, not a static value.
It this possible?
As far as I know, dynamic values in the filter expression are not currently supported with partial indexes & the filter expression only supports a subset of regular mongo query operators (and that subset does not include the $date operator).
One way to approximate your desired behavior is by creating a shouldIndex boolean field on your documents, have {shouldIndex: true} be your filter expression, and have a script update that field once a day for documents that are older than current_date - 30.
Yes that's possible,
you can use normal createIndex operation with partialFilterExpression which contains you condition.
I think code in you quenstion will only work
You can use
let today = new Date(); //your date condition
db.collection_name.createIndex({ field1 : type1, field2: type2},
partialFilterExpression: { date: { $gte: { today }} })
Read more about parital indexes here,
however it comes with some restrictions too.
Related
I cant spot what the issue here is. This returns nothing
db.mycoll.aggregate([
{$project:{CreatedAt:"$CreatedAt",Now:"$$NOW",DateFloor:{$add:["$$NOW",-24*60*60000]}}},
{$match:{CreatedAt:{$gte:"$DateFloor"}}}
])
But this returns results - substituting DateFloor with actual value
db.mycoll.aggregate([
{$project:{CreatedAt:"$CreatedAt",Now:"$$NOW",DateFloor:{$add:["$$NOW",-24*60*60000]}}},
{$match:{CreatedAt:{$gte: ISODate("2020-04-28T23:17:56.547Z")}}}
])
Issue with your query is when you're doing :
{$match:{CreatedAt:{$gte:"$DateFloor"}}}
You're actually checking for documents where CreatedAt field's value to be greater than input string value "$DateFloor", So $match is not considering "$DateFloor" as another field in the same document rather it's considering it as a string value. So to compare two fields from same document you need to use $gte with $expr (which will let you use aggregation expressions within the query language).
{
$match: {
{
$expr: {
$gte: ["$CreatedAt", "$DateFloor"];
}
}
}
}
So you might get confused when I say aggregation expressions & why $gte needs to be wrapped inside $expr - In your actual query $gte refers to comparison operator in MongoDB but in this above query $gte refers to aggregation pipeline operator where both technically does the same but which is what needed to compare two fields from same document.
I need to query Mongo using the FIND function, I can't use the aggregate function.
My documents are like this:
{
"name": "Tom",
"priDate":2010-04-11T00:00:00.000Z
}
The query I would like to make is:
Find all documents where ("priDate" + 1 year) is lte today.
Is it possible to do this without using an aggregation query? I can't use the field value in find ..
The query that I would need, I think, would be like this one I made:
db.system.profile.find({
"priDate" :
{
$gte: new Date(ITSELF + 1 year??) ,
$lt : new Date()
}
})
Can you help me?
many thanks, i'm going crazy :)
see if this works:
db.collection.find(
{
$expr: {
$lte: ['$priDate', { $subtract: ['$$NOW', 31536000000] }]
}
}
)
https://mongoplayground.net/p/QJ3BbHTQlgh
Adding "1 year" can be be difficult because of leap years or daylight-saving-times.
I suggest moment.js, then solution would be
db.system.profile.find(
{
priDate: {
$gte: moment().add(1, "year").toDate(),
$lt: moment().toDate()
}
}
)
However priDate >= "today + 1 year" AND priDate < "today" is not possible. Change the condition according to your needs.
MongoDB stores dates as milliseconds since epoch, so you can advance a date one year by adding the number of milliseconds in a year using $add inside $expr, then test with $lte:
db.system.profile.find({$expr:{$lte:[{$add:["$priDate",31536000000]},new ISODate()]}})
Note that this will be off by a day around leap year unless you adjust the number of milliseconds for that.
You expressed the constraint that you cannot use aggregate, but with $expr in Mongo 3.6 onwards, you can use any and all aggregation operators in a find query as well.
https://docs.mongodb.com/manual/reference/operator/query/expr/#definition
I have this query which isn't working and I've tried looking for material to better understand the syntax behind it, but I'm not having much luck. If someone could point out some resources for me to look into, I would appreciate it. Specifically, I'm having trouble with ISODate(:#{#start}) and what roles the :# or {#parameter} play in this query.
Also, if you happen to know how to fix this up. Please share!
Edit The dates in the db should be engulfed by the date parameters supplied. I originally had 'foo': :#{#parameter} which worked for looking up values, but here I've had to wrap the string with ISODate() in order to convert it for comparison.
#Query("
{ 'user.id': :#{#id} },
{ 'date.events.start': { $gte: ISODate(:#{#start}) } },
{ 'date.events.end': { $lte: ISODate(:#{#currentDate}) } }
")
List<Case> getUsersInPeriod(#Param("id") String id,
#Param("start") String start, #Param("currentDate") String currentDate);
Edit Playing around in Mongo Compass query box with the following:
{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-02-13')} }, {'date.events.end': {$lte: ISODate('2020-02-31')}}
and it does not work as expected. So I am not sure what the issue is.
Sample Mongo document values:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
Sample Comparison values:
start: "2020-01-16" (String)
end: "2020-01-31" (String)
Which didn't work, and so I wrapped the comparison values with ISODate() and that still didn't work, which now has me looking at document string values.
Edit: #3
I've converted some values in the document to Date and changed the query in Mongo Compass to:
{$and: [{'user.id': '5df2b19006f31c190cc13288' }, { 'date.events.start': {$gte: ISODate('2020-01-21')} }, {'date.events.end': {$lte: ISODate('2020-01-31')}}]}
which only picks up the document values formed as Date instead of String... so, I think I narrowed down the problem to two things (My original issue still persists with the syntax).
How do I deal with Mongo documents with string dates? Is it possible to parse the document string dates as ISODate while doing the query?
#prasad_ answer to formatting: can parse String or Date, but both must be of the same type. Ideally Date is used.
Edit #4: What I know so far...
User.id checks out. There's no issue there. I know the $and usage is correct.
Sample Document entries:
start: "2020-01-20T08:30:00.000-06:00" (String)
end: "2020-01-20T15:30:00.000-06:00" (String)
The above values should be engulfed by the following parameters:
start: 2020-01-16T19:57:54.949-06:00
end: 2020-01-31T23:59:59.999-06:00
I've converted both sets of strings to String or to Date and neither has returned results from the query made in my application; however ...
MongoDB Compass Community Query filter:
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Does filter strings correctly, and...
{'$and':[{'user.id':'5df2b19006f31c190cc13288'},{'date.events.start':{'$gte':'2020-01-22T08:30:00.000-06:00'}},{'date.events.end':{'$lte':'2020-01-31T15:30:00.000-06:00'}}]}
Works when the document fields are of type Date and
start: 2020-01-21T14:30:00.000+00:00 (Date)
end: 2020-01-21T21:30:00.000+00:00 (Date)
Since I can get results in Mongo Compass Community, there must be something wrong with my application's query here:
#Query("{'$and': [{ 'user.id': :#{#id} }, { 'date.events.start': { '$gte': :#{#startPeriod} } }, { 'date.events.end': { '$lte': :#{#currentDate} } } ]}")
List<Case> getUsersInPeriod(#Param("id") String id, #Param("startPeriod") String startPeriod, #Param("currentDate") String currentDate);
The document entry is structured as:
date: (Object)
events: (Array)
[0]: (Object)
start: (String)
end: (String)
[1]: (Object)
(...)
(...)
Solution
I was able to find something that put me on the right path:
Internally, MongoDB can store dates as either Strings or as 64-bit integers. If you intend to do any operations using the MongoDB query or aggregate functions, or if you want to index your data by date, you'll likely want to store your dates as integers. If you're using the built-in "Date" data type, or a date wrapped in the ISODate() function, you're also storing your date as an integer.
https://www.compose.com/articles/understanding-dates-in-compose-mongodb/
So I changed everything to Date and now it's working as expected. Not sure what I had done wrong the first time I checked Date types, but oh well.
I still don't understand the syntax I originally asked, so if someone wants to help by providing something to read, please and thank you.
I want to store/update statistics for each year. I have the following model
{
entityid: ObjectId,
stats: [
{ year: 2018, value: 25 }
]
}
(This model is a bit simplified, in reality the year has also an array with months -> days -> hours. But the problem stays the same for the simplified model)
For updating I can simply use $inc like
db.statistics.updateOne(
{entityid, 'stats.year': 2018},
{$inc: { 'stats.$.year': 1}}
)
But now a problem arises when a new year begins because there will be no { year: 2019, value: 0 } inside the stats array. Upsert can not really be used because of the positional operator $.
The current solution is to check the result of the update query above if we actually modified a document. If no changes were applied we execute a push to insert the array element for the new year and execute the update again.
The solution feels like a hack and produces some problem with race conditions where multiple objects are pushed for the same year, although this can be fixed easily.
Can the update/push operation be performed in one go? Or is there a better database model to store this information?
You can either follow your hack or make database like this and use upsert on the year key while using $inc on value
{
entityid: ObjectId,
year: 2018,
value: 25
}
and use $group on entityid while fetching data if you want to group data.
Let's say I have four documents in my collection:
{u'a': {u'time': 3}}
{u'a': {u'time': 5}}
{u'b': {u'time': 4}}
{u'b': {u'time': 2}}
Is it possible to sort them by the field 'time' which is common in both 'a' and 'b' documents?
Thank you
No, you should put your data into a common format so you can sort it on a common field. It can still be nested if you want but it would need to have the same path.
You can use use aggregation and the following code has been tested.
db.test.aggregate({
$project: {
time: {
"$cond": [{
"$gt": ["$a.time", null]
}, "$a.time", "$b.time"]
}
}
}, {
$sort: {
time: -1
}
});
Or if you also want the original fields returned back: gist
Alternatively you can sort once you get the result back, using a customized compare function ( not tested,for illustration purpose only)
db.eval(function() {
return db.mycollection.find().toArray().sort( function(doc1, doc2) {
var time1 = doc1.a? doc1.a.time:doc1.b.time,
time2 = doc2.a?doc2.a.time:doc2.b.time;
return time1 -time2;
})
});
You can, using the aggregation framework.
The trick here is to $project a common field to all the documents so that the $sort stage can use the value in that field to sort the documents.
The $ifNull operator can be used to check if a.time exists, it
does, then the record will be sorted by that value else, by b.time.
code:
db.t.aggregate([
{$project:{"a":1,"b":1,
"sortBy":{$ifNull:["$a.time","$b.time"]}}},
{$sort:{"sortBy":-1}},
{$project:{"a":1,"b":1}}
])
consequences of this approach:
The aggregation pipeline won't be covered by any of the index you
create.
The performance will be very poor for very large data sets.
What you could ideally do is to ask the source system that is sending you the data to standardize its format, something like:
{"a":1,"time":5}
{"b":1,"time":4}
That way your query can make use of the index if you create one on the time field.
db.t.ensureIndex({"time":-1});
code:
db.t.find({}).sort({"time":-1});