MongoDB - Dates between using $match - mongodb

So I try to use MongoDB $match to get data between 2 dates, but it turns out that the data is not returning a.k.a empty here. What it looks like:
db.collection.aggregate([
{
$match: {
date: {
$gte: new Date("2022-10-23"),
$lt: new Date("2022-10-25"),
},
}
},
{
$group: {
_id: "$title",
title: {
$first: "$title"
},
answer: {
$push: {
username: "$username",
date: "$date",
formId: "$formId",
answer: "$answer"
}
}
}
},
])
Here is the data that I try to run on the Mongo playground:
https://mongoplayground.net/p/jKx_5kZnJNz
I think there is no error with my code anymore... but why it gives an empty return.

Migrate the comment to the answer post for the complete explanation.
Issue 1
The document contains the date field as a string type while you are trying to compare with Date which leads to incorrect output.
Ensure that you are comparing both values in the exact type.
Either that migrate the date value to Date type or
converting the date field to Date type in the query via $toDate.
{
$match: {
$expr: {
$and: [
{
$gte: [
{
$toDate: "$date"
},
new Date("2022-10-23")
]
},
{
$lt: [
{
$toDate: "$date"
},
new Date("2022-10-25")
]
}
]
}
}
}
Issue 2
Since you are using $lt ($lt: new Date("2022-10-25")), it won't include the documents with date: new Date("2022-10-25").
For inclusive end date, you shall use $lte.
Demo # Mongo Playground

Related

Mongo query to search between given date range while date is stored as string in db

I have db schema that has string date format("date":"2020-09-01 16:07:45").
I need to search between given date range, I know this is possible if we're using ISO date format but I'm not sure if we can query with date format being string.
I have tried the following query, it doesn't seem to show accurate results.
db.daily_report_latest.find({"date":{$gte: "2021-01-01 00:00:00", $lte:"2021-03-01 00:00:00"}},{"date":1})
Is there any alternative to this? Appreciate your help.
You're right, you can't query a date field with a string, but you can just cast it to date type like so:
Mongo Shell:
db.daily_report_latest.find({
"date": {$gte: ISODate("2021-01-01T00:00:00Z"), $lte: ISODate("2021-03-01T00:00:00Z")}
}, {"date": 1});
For nodejs:
db.daily_report_latest.find({
"date": {$gte: new Date("2021-01-01 00:00:00"), $lte: new Date("2021-03-01 00:00:00")}
}, {"date": 1});
For any other language just check what the mongo driver date type is and do the same.
Note that the mongo shell isn't able to parse the string input in the format you provided, you should read here about the supported formats and transform your string pre-query like I did.
Another thing to consider for the nodejs usecase is timezones, the string will be parsed as the machine current timezone so again you need to adjust to that.
You can use $dateFromString feature of aggregation. (Documentation)
pipeline = []
pipeline.append({"$project": {document: "$$ROOT", "new_date" : { "$dateFromString": { "dateString": '$date', "timezone": 'America/New_York' }}}})
pipeline.append({"$match":{"new_date": {"$gte": ISODate("2021-01-01 00:00:00"), "$lte":ISODate("2021-03-01 00:00:00")}}})
data = db.daily_report_latest.aggregate(pipeline=pipeline)
So in the both the solutions, first typecast the date field in DB to date and then compare it with your input date range.
SOLUTION #1: For MongoDB Version >= 4.0 using $toDate.
db.daily_report_latest.find(
{
$expr: {
$and: [
{ $gte: [{ $toDate: "$date" }, new Date("2021-01-01 00:00:00")] },
{ $lte: [{ $toDate: "$date" }, new Date("2021-03-01 00:00:00")] }
]
}
},
{ "date": 1 }
)
SOLUTION #2: For MongoDb version >= 3.6 using $dateFromString.
db.daily_report_latest.find(
{
$expr: {
$and: [
{ $gte: [{ $dateFromString: { dateString: "$date" }}, new Date("2021-01-01 00:00:00")] },
{ $lte: [{ $dateFromString: { dateString: "$date" }}, new Date("2021-03-01 00:00:00")] }
]
}
},
{ "date": 1 }
)

Date range not working in aggregation pipeline, but works in find()

I am trying to filter data by a date range. Example return the data that was created no more than 14 days ago.
I can do this in find with the following:
{
$match: {
eventTime: { $gte: startTime.toDate(), $lte: endTime.toDate() }
}
}
eventTime is an ISO date as well as startTime and endTime
I am using an aggregation and a lookup and trying to implement the same thing:
{
$lookup:
{
from: "data",
let: { dataId: "$dataId", patientId: "$patientId" },
pipeline: [
{
$match:
{
$expr:
{
$and:
[
{ $eq: ["$patientId", patientId] },
{ $eq: ["$dataId", "$$dataId"] },
{ $gte: ["$eventTime", startTime.toDate()] },
{ $lte: ["$eventTime", endTime.toDate()] },
]
}
}
}
],
as: "data"
}
}
But no data results are returned. If I remove the dates I get all the correct data based on dataId and patient. so the join is working.. but somehow the date range is not.
Again both the eventTime and startTime and endTime are all ISO dates.
example :
let endTime = Moment(new Date());
let startTime = Moment().subtract(days, "days");
"eventTime": "2019-08-07T03:37:40.738Z"
startTime "2019-07-30T00:02:11.611Z"
endTime "2019-08-13T00:02:11.610Z"
End time is 'today'
so in the example here the data time is between the two dates and should be returned.
I looked there : https://docs.mongodb.com/manual/reference/operator/aggregation/gte/
and it should work.. but not the case
I tried:
{eventTime: { '$gte': new Date(startTime), $lte: new Date(endTime)}}
and I get:
MongoError: An object representing an expression must have exactly one field: { $gte: new Date(1564495211043), $lte: new Date(1565704811042) }
also tried:
{ eventTime: {'$gte': new Date(startTime)}}
and get:
MongoError: Expression $gte takes exactly 2 arguments. 1 were passed in.
also tried:
{ $eventTime: {'$gte': new Date(startTime)}}, {$eventTime: {'$lte': new Date(endTime)}}
and get: MongoError: Unrecognized expression '$eventTime'
Any insight would certainly be appreciated
I was able to get it working via toDate:
{
$match:
{
$expr:
{
$and:
[
{ $eq: ["$patientId", patientId] },
{ $eq: ["$dataId", "iraeOverallAlert"] },
{ "$gte": [ {$toDate: "$eventTime"}, startTime.toDate()] },
{ "$lte": [ {$toDate: "$eventTime"}, endTime.toDate()] },
]
}
}
},
Note: This was not needed in the find, but somehow was needed using aggregation. Makes no sense but yah for trial and error.

Find avg difference in dates stored as strings

I have a Mongo database and I have stored dates as strings. Per document I have a field called "creationdate" and a field called "completiondate". The dates format is "YYYY-MM-dd" (ex "2011-12-18"). Even I can execute simple aggregation like greaterThan, greaterThanEqual, I cannot find the difference in dates, which I have to find to calculate the average days difference between completion and creation date.
The above query I have to write it on spring-boot with MongoTemplate if it is possible.
I am trying something like this but it doesn't work.
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("creationdate").gte(date1).lte(date2).andOperator(Criteria.where("completiondate").ne(""))),
Aggregation.project("servicerequesttype").and(DateOperators.DateFromString.fromStringOf("completiondate").withFormat("%Y-%m-%d")).minus(DateOperators.DateFromString.fromStringOf("creationdate").withFormat("%Y-%m-%d")).as("diff"),
Aggregation.group("servicerequesttype").avg("diff").as("average")
);
date1, date2 are given strings like "2011-01-01"
Is this what you are looking for?
db.collection.aggregate([
{
$project: {
creationdate: {
$dateFromString: {
dateString: "$creationdate",
format: "%Y-%m-%d"
}
},
completiondate: {
$dateFromString: {
dateString: "$completiondate",
format: "%Y-%m-%d"
}
}
}
},
{
$project: {
difference: {
$subtract: [
"$completiondate",
"$creationdate"
]
}
}
},
{
$group: {
_id: null,
average: {
$avg: "$difference"
}
}
},
{
$project: {
_id: 0,
dayAverage: {
$divide: [
"$average",
86400000
]
}
}
}
])
I have created interactive demo here: https://mongoplayground.net/p/wGRw12m3UbB
Hope it helps :)
Spring-Boot
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("creationdate").gte(date1).lte(date2).andOperator(Criteria.where("completiondate").ne(""))),
Aggregation.project("servicerequesttype").and(DateOperators.DateFromString.fromStringOf("creationdate").withFormat("%Y-%m-%d")).as("creationdate").and(DateOperators.DateFromString.fromStringOf("completiondate").withFormat("%Y-%m-%d")).as("completiondate"),
Aggregation.project("servicerequesttype").and("completiondate").minus("creationdate").as("difference"),
Aggregation.group("servicerequesttype").first("servicerequesttype").as("servicerequesttype").avg("difference").as("temp"),
Aggregation.project("servicerequesttype").and("temp").divide(86400000).as("average")
);

MongoDB exception Resolving [duplicate]

I am getting an issue for running the following aggregate query:
db.snippets.aggregate([ { '$project': { month: { '$month': '$created_at' }} } ])
The error message for the same is:
assert: command failed: {
"errmsg" : "exception: can't convert from BSON type EOO to Date",
"code" : 16006,
"ok" : 0 } : aggregate failed
How do I get around this issue? I found a related question: MongoDB: can't convert from BSON type EOO to Date.
But it doesn't tell me how to get things done.
You likely have one or more docs with a created_at value that's not a BSON Date and you'll need to fix that by converting those values to Date or removing them.
You can find those docs with a $not query that uses the $type operator like:
db.snippets.find({created_at: {$not: {$type: 9}}})
If the created_at values are date strings, you can find the docs that need updating and then update them in the shell using code like:
db.snippets.find({created_at: {$not: {$type: 9}}}).forEach(function(doc) {
// Convert created_at to a Date
doc.created_at = new Date(doc.created_at);
db.snippets.save(doc);
})
try this one, its help for me above problem.
db.snippets.aggregate([{
'$project': {
month: { $substr: ["$created_at", 5, 2] }
}
}]);
above code get month wise
data is entered into the database in ISO format which can then be easily worked with.
In some situations, some documents are supposed to have empty Date fields. In those cases, you could try this (using your example):
db.snippets.aggregate([ { '$project': { month:
{ $cond: [{ $ifNull: ['$created_at', 0] }, { $month: '$created_at' }, -1] }} } ])
In this example, we would get -1 in the cases whenever no field '$created_at' is found. For all the other cases, we would get the Date month.
I had a related issue, but in my case the Date fields were the members of an array, so the error was "can't convert BSON type Object to Date".
I needed to get the day of week from the dates in the possibleTripDateTimes array.
Sample document:
{
"possibleTripDateTimes" : [
{
"tripDateTime" : ISODate("2015-08-01T06:00:00.000-0700")
}
]
}
The fix was simply to use dot notation to address the array member fields.
db.trips.aggregate([
{
$project: {
departTime: {
$map: {
input: "$possibleTripDateTimes.tripDateTime",
as: "dateTime",
in: { $dayOfWeek: "$$dateTime" }
}
}
}
}
]
);
I hope this helps someone who also gets zero search results on the "BSON type Object" search
I had the same problem, I figured that the date field is missing for some of the documents causing the conversion to fail. I just added a match clause to filter these out. But ofcourse i am investigating on my app side why they are not being populated.
db.snippets.aggregate([
{
'$match': {
'created_at': {
"$exists": true
}
}
},
{
'$project': {
month: {
'$month': '$created_at'
}
}
}
])
I had a similar problem, and solved it checking if the date existed.
db.users.aggregate([
{$project:{day: { $cond: ["$bd", { $dayOfMonth: "$bd" }, -1] },
month: { $cond: ["$bd", { $month: "$bd" }, -1] },
year: { $cond: ["$bd", { $year: "$bd" }, -1] }
}},
{$match:{"month":1, "day":15}}
])
My date field is bd and with that match I'm getting all users that have their birthday on January 15th.
This error can also appear if you have incorrectly named your properties in your aggregation relative to what they are in your database.
For example my code was
$group: {
_id: {$week: "$projects.timeStamp"},
total: { $sum: "$projects.hours" }
}
But I hadn't camelcased timestamp in my database so simply modifying to projects.timestamp fixed it.
First, you need to check whether the data type is in ISODate. IF not you can change the data type as the following example.
db.collectionName.find().forEach(function(each_object_from_collection){each_object_from_collection.your_date_field=new ISODate(each_object_from_collection.your_date_field);db.collectionName.save(each_object_from_collection);})
Now you can find it in two ways
db.collectionName.find({ $expr: {$eq: [{ $year: "$your_date_field" }, 2017]}});
Or by aggregation
db.collectionName.aggregate([{$project: {field1_you_need_in_result: 1,field12_you_need_in_result: 1,your_year_variable: {$year: '$your_date_field'}, your_month_variable: {$month: '$your_date_field'}}},{$match: {your_year_variable:2017, your_month_variable: 3}}])
First, you can identify the particular field which is causing an issue as follows:
db.collectionName.find( { 'dateField' : { $type : 2 } } )
The above line checks and finds all the documents with field name 'dateField' having type as String(referred as $type - 2).
Once it is identified and verified, we can modify those records as follows:
db.collectionName.find( { 'dateField' : { $type : 2 } } ).forEach( function (x) {
x.dateField = new ISODate(x.dateField);
db.collectionName.save(x);
});
In my case I had to use "$toDate" and it worked:
db.snippets.aggregate([ { '$project': { month: { '$month': {$toDate: '$created_at'} }} } ])

MongoDB $gt not working

Below query gives me no results, though there are fields matching the geo location and time stamp.
I have tried changing the $gt to $ne and it returns the sum of all $values.
I have even tried checking timeStamp greater than 0. still no result. Here timestamp is a double datatype.
db.csv2.aggregate([{
$match: {
$and: [{
loc: {
$geoWithin: {
$box: [
[-128.232422, 26.382028],
[-109.6875, 43.624147]
]
}
}
}, {
timeStamp: {
$gt: 1432201420790
}
}]
}
}, {
$group: {
_id: null,
total: {
$sum: "$value"
}
}
}]);
The error given to you is explicit. Read the doc to see how to implement the $gt aggregation function : documentation
Basically, the condition is an array
{ $gt: ["$timeStamp", 1432201420790] }
[EDIT]
Your issue probably comes from the syntax with the dollar sign before the timestamp field which is missing in your code. Try this
{ $gt: ["$timeStamp", 1432201420790] }
instead of
{ timeStamp: { $gt: 1432201420790 }}