If I have documents in the following schema saved in my mongoDB:
{
createdDate: Date,
lastUpdate: Date
}
is it possible to query for documents where the period of time between creation and the last update is e.g. greater than one day?
Best option is to use the $redact aggregation pipeline stage:
db.collection.aggregate([
{ "$redact": {
"$cond": {
"if": {
"$gt": [
{ "$subtract": [ "$lastUpdate", "$createdDate" ] },
1000 * 60 * 60 * 24
]
},
"then": "$$KEEP",
"else": "$$PRUNE"
}
}}
])
So you are looking at the milliseconds value from the difference being greater than the milliseconds value for one day. The $subtract does the math for the difference, and when two dates are subtracted the difference in miliseconds is returned.
The $redact operator takes a logical expression as "if", and where that condition is true it takes the action in "then" which is to $$KEEP the document. Where it is false then the document is removed from results with $$PRUNE.
Note that since this is a logical condition and not a set value or a range of values, then an "index" is not used.
Since the operations in the aggregation pipeline are natively coded, this is the fastest execution of such a statement that you can get though.
The alternate is JavaScript evaluation with $where. This takes a JavaScript function expression that needs to similarly return a true or false value. In the shell you can shorthand like this:
db.collection.find(function() {
return ( this.lastUpdate.valueOf() - this.createdDate.valueOf() )
> ( 1000 * 60 * 60 * 24 );
})
Same thing, except that JavaScript evalution requires interpretation and will run much slower than the .aggregate() equivalent. By the same token, this type of expression cannot use an index to optimize performance.
For the best results, store the difference in the document. Then you can simply query directly on that property, and of course you can index it as well.
You can use $expr ( 3.6 mongo version operator ) to use aggregation functions in regular query.
Compare query operators vs aggregation comparison operators.
db.col.find({$expr:{$gt:[{"$subtract":["$lastUpdate","$createdDate"]},1000*60*60*24]}})
Starting in Mongo 5, it's a perfect use case for the new $dateDiff aggregation operator:
// { created: ISODate("2021-12-05T13:20"), lastUpdate: ISODate("2021-12-06T05:00") }
// { created: ISODate("2021-12-04T09:20"), lastUpdate: ISODate("2021-12-05T18:00") }
db.collection.aggregate([
{ $match: {
$expr: {
$gt: [
{ $dateDiff: { startDate: "$created", endDate: "$lastUpdate", unit: "hour" } },
24
]
}
}}
])
// { created: ISODate("2021-12-04T09:20"), lastUpdate: ISODate("2021-12-05T18:00") }
This computes the number of hours of difference between the created and lastUpdate dates and checks if it's more than 24 hours.
Related
First occasion experimenting with dates and times in MongoDB.
Currently, I have correctly inserted UTC dates into my documents:
{
"Value": 10
"DateTime": {
"$date": "2013-08-23T08:00:00.000Z"
},
}
I want to query the data to return all of the 'value' and datetime fields for all documents in which the hour is 08. I have gotten as far as:
db.readings.aggregate([{$project:{hour:{$hour:"$DateTime"}}},{$match:{hour:{"$in":[08]}}}])
Which returns the _id and "hour": 8 for all matching entries but I'm confused at how to proceed. It seems an unnecessarily complicated way to search and so I wonder if I am barking up the wrong tree here? Admittedly, I am somewhat out of my depth so some guidance would be appreciated.
You can try $expr expression operator to use aggregation operators ($hour) in query,
$expr to use aggregation operators
$hour will return an hour from the date
$eq to match hour and input hour 8
db.collection.find({
$expr: {
$eq: [
{ $hour: "$DateTime" },
8
]
}
})
Playground
I have a collection T, with 2 fields: Grade1 and Grade2, and I want to select those with condition Grade1 > Grade2, how can I get a query like in MySQL?
Select * from T Where Grade1 > Grade2
You can use a $where. Just be aware it will be fairly slow (has to execute Javascript code on every record) so combine with indexed queries if you can.
db.T.find( { $where: function() { return this.Grade1 > this.Grade2 } } );
or more compact:
db.T.find( { $where : "this.Grade1 > this.Grade2" } );
UPD for mongodb v.3.6+
you can use $expr as described in recent answer
You can use $expr ( 3.6 mongo version operator ) to use aggregation functions in regular query.
Compare query operators vs aggregation comparison operators.
Regular Query:
db.T.find({$expr:{$gt:["$Grade1", "$Grade2"]}})
Aggregation Query:
db.T.aggregate({$match:{$expr:{$gt:["$Grade1", "$Grade2"]}}})
If your query consists only of the $where operator, you can pass in just the JavaScript expression:
db.T.find("this.Grade1 > this.Grade2");
For greater performance, run an aggregate operation that has a $redact pipeline to filter the documents which satisfy the given condition.
The $redact pipeline incorporates the functionality of $project and $match to implement field level redaction where it will return all documents matching the condition using $$KEEP and removes from the pipeline results those that don't match using the $$PRUNE variable.
Running the following aggregate operation filter the documents more efficiently than using $where for large collections as this uses a single pipeline and native MongoDB operators, rather than JavaScript evaluations with $where, which can slow down the query:
db.T.aggregate([
{
"$redact": {
"$cond": [
{ "$gt": [ "$Grade1", "$Grade2" ] },
"$$KEEP",
"$$PRUNE"
]
}
}
])
which is a more simplified version of incorporating the two pipelines $project and $match:
db.T.aggregate([
{
"$project": {
"isGrade1Greater": { "$cmp": [ "$Grade1", "$Grade2" ] },
"Grade1": 1,
"Grade2": 1,
"OtherFields": 1,
...
}
},
{ "$match": { "isGrade1Greater": 1 } }
])
With MongoDB 3.4 and newer:
db.T.aggregate([
{
"$addFields": {
"isGrade1Greater": { "$cmp": [ "$Grade1", "$Grade2" ] }
}
},
{ "$match": { "isGrade1Greater": 1 } }
])
In case performance is more important than readability and as long as your condition consists of simple arithmetic operations, you can use aggregation pipeline. First, use $project to calculate the left hand side of the condition (take all fields to left hand side). Then use $match to compare with a constant and filter. This way you avoid javascript execution. Below is my test in python:
import pymongo
from random import randrange
docs = [{'Grade1': randrange(10), 'Grade2': randrange(10)} for __ in range(100000)]
coll = pymongo.MongoClient().test_db.grades
coll.insert_many(docs)
Using aggregate:
%timeit -n1 -r1 list(coll.aggregate([
{
'$project': {
'diff': {'$subtract': ['$Grade1', '$Grade2']},
'Grade1': 1,
'Grade2': 1
}
},
{
'$match': {'diff': {'$gt': 0}}
}
]))
1 loop, best of 1: 192 ms per loop
Using find and $where:
%timeit -n1 -r1 list(coll.find({'$where': 'this.Grade1 > this.Grade2'}))
1 loop, best of 1: 4.54 s per loop
I have a collection T, with 2 fields: Grade1 and Grade2, and I want to select those with condition Grade1 > Grade2, how can I get a query like in MySQL?
Select * from T Where Grade1 > Grade2
You can use a $where. Just be aware it will be fairly slow (has to execute Javascript code on every record) so combine with indexed queries if you can.
db.T.find( { $where: function() { return this.Grade1 > this.Grade2 } } );
or more compact:
db.T.find( { $where : "this.Grade1 > this.Grade2" } );
UPD for mongodb v.3.6+
you can use $expr as described in recent answer
You can use $expr ( 3.6 mongo version operator ) to use aggregation functions in regular query.
Compare query operators vs aggregation comparison operators.
Regular Query:
db.T.find({$expr:{$gt:["$Grade1", "$Grade2"]}})
Aggregation Query:
db.T.aggregate({$match:{$expr:{$gt:["$Grade1", "$Grade2"]}}})
If your query consists only of the $where operator, you can pass in just the JavaScript expression:
db.T.find("this.Grade1 > this.Grade2");
For greater performance, run an aggregate operation that has a $redact pipeline to filter the documents which satisfy the given condition.
The $redact pipeline incorporates the functionality of $project and $match to implement field level redaction where it will return all documents matching the condition using $$KEEP and removes from the pipeline results those that don't match using the $$PRUNE variable.
Running the following aggregate operation filter the documents more efficiently than using $where for large collections as this uses a single pipeline and native MongoDB operators, rather than JavaScript evaluations with $where, which can slow down the query:
db.T.aggregate([
{
"$redact": {
"$cond": [
{ "$gt": [ "$Grade1", "$Grade2" ] },
"$$KEEP",
"$$PRUNE"
]
}
}
])
which is a more simplified version of incorporating the two pipelines $project and $match:
db.T.aggregate([
{
"$project": {
"isGrade1Greater": { "$cmp": [ "$Grade1", "$Grade2" ] },
"Grade1": 1,
"Grade2": 1,
"OtherFields": 1,
...
}
},
{ "$match": { "isGrade1Greater": 1 } }
])
With MongoDB 3.4 and newer:
db.T.aggregate([
{
"$addFields": {
"isGrade1Greater": { "$cmp": [ "$Grade1", "$Grade2" ] }
}
},
{ "$match": { "isGrade1Greater": 1 } }
])
In case performance is more important than readability and as long as your condition consists of simple arithmetic operations, you can use aggregation pipeline. First, use $project to calculate the left hand side of the condition (take all fields to left hand side). Then use $match to compare with a constant and filter. This way you avoid javascript execution. Below is my test in python:
import pymongo
from random import randrange
docs = [{'Grade1': randrange(10), 'Grade2': randrange(10)} for __ in range(100000)]
coll = pymongo.MongoClient().test_db.grades
coll.insert_many(docs)
Using aggregate:
%timeit -n1 -r1 list(coll.aggregate([
{
'$project': {
'diff': {'$subtract': ['$Grade1', '$Grade2']},
'Grade1': 1,
'Grade2': 1
}
},
{
'$match': {'diff': {'$gt': 0}}
}
]))
1 loop, best of 1: 192 ms per loop
Using find and $where:
%timeit -n1 -r1 list(coll.find({'$where': 'this.Grade1 > this.Grade2'}))
1 loop, best of 1: 4.54 s per loop
Given the following schema
var schema = new Schema({
activeDuration: {type: Number, required: true},
}, {
timestamps: true
});
activeDuration is a time given in days. How can I create a query which is equal to SELECT FROM schema WHERE (UNIX_TIMESTAMP(createdAt) + (activeDuration * 24 * 60 * 60)) < CURRENT_TIMESTAMP;
So I should receive all objects, which are not "active" anymore (the time where the object was inserted to the database + the given duration is smaller than the current time).
You would require an aggregation pipeline that uses $redact operator as it allows you to process otherwise complex logical conditions in a pipeline with the $cond operator and uses the special operations $$KEEP to "keep" the document where the logical condition is true or $$PRUNE to "remove" the document where the condition was false.
This operation is similar to having a $project pipeline that selects the fields in the collection and creates a new field that holds the result from the logical condition query and then a subsequent $match, except that $redact does all the computations within a single pipeline stage which can be more efficient.
As for the logical condition, there are Date Aggregate Operators as well as Arithmetic ones to aid with the computations.
Consider the following example pipeline:
Model.aggregate([
{
"$redact": {
"$cond": [
{
"$lt": [
{
"$add": [
"$createdAt",
{ "$multiply": ["$activeDuration", 24 * 60 * 60]}
]
},
new Date()
]
},
"$$KEEP",
"$$PRUNE"
]
}
}
], (err, data) => {
if (err) throw err;
console.log(data);
})
And if going with the find() method, then $where can be applied as
Model.find({ "$where": "(this.createdAt.valueOf() + (this.activeDuration * 24 * 60 * 60)) < +new Date()" })
.exec((err, data) => {
if (err) throw err;
console.log(data);
})
Bear in mind the performance costs when using $where as the operator calls the JavaScript engine to evaluate JavaScript code on every document and checks the condition for each. It is advisable to combine with indexed queries if you can so that the query may be faster.
As stated in my comment, I added a endsAt field, which contains a Date(), when the game ends. The query is as easy as:
var query = Game.find({"endsAt": {"$lt": new Date()}});
I have time series data stored in a mongodb database, where one of the fields is an ISODate object. I'm trying to retrieve all items for which the ISODate object has a zero value for minutes and seconds. That is, all the objects that have a timestamp at a round hour.
Is there any way to do that, or do I need to create separate fields for hour, min, second, and query for them directly by doing, e.g., find({"minute":0, "second":0})?
Thanks!
You could do this as #Devesh says or if it fits better you could use the aggregation framework:
db.col.aggregate([
{$project: {_id:1, date: {mins: {$minute: '$dateField'}, secs: {$second: '$dateField'}}}},
{$match: {mins: 0, secs: 0}}
]);
Like so.
Use the $expr operator along with the date aggregate operators $minute and $second in your find query as:
db.collection.find({
'$expr': {
'$and': [
{ '$eq': [ { '$minute': '$dateField' }, 0 ] },
{ '$eq': [ { '$second': '$dateField' }, 0 ] },
]
}
})
Can you have one more column added in the collection only containing the datetime without minutes and seconds . It will make your query faster and easy to use. It will be datetime column with no minutes and seconds parts