Find avg difference in dates stored as strings - mongodb

I have a Mongo database and I have stored dates as strings. Per document I have a field called "creationdate" and a field called "completiondate". The dates format is "YYYY-MM-dd" (ex "2011-12-18"). Even I can execute simple aggregation like greaterThan, greaterThanEqual, I cannot find the difference in dates, which I have to find to calculate the average days difference between completion and creation date.
The above query I have to write it on spring-boot with MongoTemplate if it is possible.
I am trying something like this but it doesn't work.
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("creationdate").gte(date1).lte(date2).andOperator(Criteria.where("completiondate").ne(""))),
Aggregation.project("servicerequesttype").and(DateOperators.DateFromString.fromStringOf("completiondate").withFormat("%Y-%m-%d")).minus(DateOperators.DateFromString.fromStringOf("creationdate").withFormat("%Y-%m-%d")).as("diff"),
Aggregation.group("servicerequesttype").avg("diff").as("average")
);
date1, date2 are given strings like "2011-01-01"

Is this what you are looking for?
db.collection.aggregate([
{
$project: {
creationdate: {
$dateFromString: {
dateString: "$creationdate",
format: "%Y-%m-%d"
}
},
completiondate: {
$dateFromString: {
dateString: "$completiondate",
format: "%Y-%m-%d"
}
}
}
},
{
$project: {
difference: {
$subtract: [
"$completiondate",
"$creationdate"
]
}
}
},
{
$group: {
_id: null,
average: {
$avg: "$difference"
}
}
},
{
$project: {
_id: 0,
dayAverage: {
$divide: [
"$average",
86400000
]
}
}
}
])
I have created interactive demo here: https://mongoplayground.net/p/wGRw12m3UbB
Hope it helps :)

Spring-Boot
Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("creationdate").gte(date1).lte(date2).andOperator(Criteria.where("completiondate").ne(""))),
Aggregation.project("servicerequesttype").and(DateOperators.DateFromString.fromStringOf("creationdate").withFormat("%Y-%m-%d")).as("creationdate").and(DateOperators.DateFromString.fromStringOf("completiondate").withFormat("%Y-%m-%d")).as("completiondate"),
Aggregation.project("servicerequesttype").and("completiondate").minus("creationdate").as("difference"),
Aggregation.group("servicerequesttype").first("servicerequesttype").as("servicerequesttype").avg("difference").as("temp"),
Aggregation.project("servicerequesttype").and("temp").divide(86400000).as("average")
);

Related

MongoDB - Dates between using $match

So I try to use MongoDB $match to get data between 2 dates, but it turns out that the data is not returning a.k.a empty here. What it looks like:
db.collection.aggregate([
{
$match: {
date: {
$gte: new Date("2022-10-23"),
$lt: new Date("2022-10-25"),
},
}
},
{
$group: {
_id: "$title",
title: {
$first: "$title"
},
answer: {
$push: {
username: "$username",
date: "$date",
formId: "$formId",
answer: "$answer"
}
}
}
},
])
Here is the data that I try to run on the Mongo playground:
https://mongoplayground.net/p/jKx_5kZnJNz
I think there is no error with my code anymore... but why it gives an empty return.
Migrate the comment to the answer post for the complete explanation.
Issue 1
The document contains the date field as a string type while you are trying to compare with Date which leads to incorrect output.
Ensure that you are comparing both values in the exact type.
Either that migrate the date value to Date type or
converting the date field to Date type in the query via $toDate.
{
$match: {
$expr: {
$and: [
{
$gte: [
{
$toDate: "$date"
},
new Date("2022-10-23")
]
},
{
$lt: [
{
$toDate: "$date"
},
new Date("2022-10-25")
]
}
]
}
}
}
Issue 2
Since you are using $lt ($lt: new Date("2022-10-25")), it won't include the documents with date: new Date("2022-10-25").
For inclusive end date, you shall use $lte.
Demo # Mongo Playground

MongoDB: Is there another way to match by date other than ISODate in an aggregation?

My question is relatively simple. I have a collection of documents that look like this
{
_id:"61f52b609f97a100092842f2"
timestampISO:2022-01-29T11:56:14.000+00:00
}
The timestampISO is a date type.
Is there a way to match by date without using ISODate("...") as seen in this answer ? I am trying not to use ISODate('') as I would actually like to save the pipeline as a string in Mongo but that is not important right now.
I would like to have a match like this
$match: {
timestampISO: {
$gte: {
$dateFromString: {
dateString: '2022-01-01T11:56:14.000+00:00',
}
},
$lte: {
$dateFromString: {
dateString: '2022-01-29T11:56:14.000+00:00',
}
},
}
}
which does not work. It seems as though ISODate() seems to be the only way? Is this correct?
Thank you for your time.
ISODate() is just an alias in the Mongo shell for new Date().
{ $dateFromString: { dateString: '2022-01-01T11:56:14.000+00:00' } } is more or less the same, however you have to use
{
$match: {
$expr: {
$gte: [
"$timestampISO",
{ $dateFromString: { dateString: '2022-01-01T11:56:14+00:00' } }
]
}
}
}
or maybe a bit simpler:
{ $match:
{$expr: {$gte: ["$timestampISO", { $toDate: '2022-01-01T11:56:14.000+00:00' } ] }}
}

MongoDB $type check on a field inside $cond

I'm new to MongoDB and my requirement is to convert a string date to date. But that particular field is sometimes in date format sometimes in string format.
Effectively, If the date is in string format I want to convert it to date else leave as it is.
Sample data:
paymentDate:2021-11-19T05:36:32.596+00:00
paymentDate:'2021-11-19T05:36:32.596+00:00'
My attempt is
{
convertedDate: {
$cond: {
if:
{'$eq': [{$type:"$paymentDate"},9]},
then:"$newField",
else:{
$dateFromString: {
dateString: '$paymentDate'
}
}
}
}
}
You're almost to the answer. Specify the compare value in $type as "date".
db.collection.find({},
{
convertedDate: {
$cond: {
if: {
"$eq": [
{
$type: "$paymentDate"
},
"date"
]
},
then: "$paymentDate",
else: {
$dateFromString: {
dateString: "$paymentDate"
}
}
}
}
})
Sample Mongo Playground
References
Available type | $type
If you are using MongoDB 4.2+, you may simply use $toDate to convert your field in an update with aggregation pipeline operation.
db.collection.update({},
[
{
"$set": {
"paymentDate": {
"$toDate": "$paymentDate"
}
}
}
])
Here is the Mongo playground for your reference.

MongoDB date math with aggregation variable

I'm trying to build an aggregation of things that haven't reported in by some interval (heartbeat) - I need to calculate a value based on a stored heartbeat:
db.things.aggregate([
{$project: {"lastmsg":1, "props.settings":1}},
{$unwind: "$props.settings"},
{$project: {
_id:0,
"lastmsg": "$lastmsg",
"heartbeat": {$multiply: [{$toInt: "$props.settings.heartbeat"},2000]},
"now": new Date(), "subtracted": new Date(new Date().getTime()- "$heartbeat")
}
}
])
Result returned is like this:
{ "lastmsg" : ISODate("2020-04-23T12:41:37.667Z"), "heartbeat" : 240000, "now" : ISODate("2020-05-14T16:26:11.824Z"), "subtracted" : ISODate("1970-01-01T00:00:00Z") }
{ "lastmsg" : ISODate("2020-05-14T16:24:24.228Z"), "heartbeat" : 240000, "now" : ISODate("2020-05-14T16:26:11.824Z"), "subtracted" : ISODate("1970-01-01T00:00:00Z") }
The "subtracted" projection is not doing the date math as expected. I can plug in a specific number and it works but this defeats the purpose...
As a last step I will match to see what of these things hasn't checked in within the interval of heartbeat:
{ $match: { "lastmsg":{$gte: "$subtracted")}
Any help would be greatly appreciated...
I don't know how your data is like (you should post your data to help), but I think this can solve the problem.
You can use the $$NOW variable, that returns the current date in ISODate format.
Test data:
[
{
"lastmsg": ISODate("2020-04-23T12:41:37.667Z"),
"heartbeat": 240000
},
{
"lastmsg": ISODate("2020-05-14T16:24:24.228Z"),
"heartbeat": 240000
}
]
Query:
db.collection.aggregate([
{
$addFields: {
"now": "$$NOW",
"subtracted": {
$subtract: [
"$$NOW",
"$heartbeat"
]
}
}
},
{
$match: {
"lastmg": {
$gte: "$subtracted"
}
}
}
])

Date range not working in aggregation pipeline, but works in find()

I am trying to filter data by a date range. Example return the data that was created no more than 14 days ago.
I can do this in find with the following:
{
$match: {
eventTime: { $gte: startTime.toDate(), $lte: endTime.toDate() }
}
}
eventTime is an ISO date as well as startTime and endTime
I am using an aggregation and a lookup and trying to implement the same thing:
{
$lookup:
{
from: "data",
let: { dataId: "$dataId", patientId: "$patientId" },
pipeline: [
{
$match:
{
$expr:
{
$and:
[
{ $eq: ["$patientId", patientId] },
{ $eq: ["$dataId", "$$dataId"] },
{ $gte: ["$eventTime", startTime.toDate()] },
{ $lte: ["$eventTime", endTime.toDate()] },
]
}
}
}
],
as: "data"
}
}
But no data results are returned. If I remove the dates I get all the correct data based on dataId and patient. so the join is working.. but somehow the date range is not.
Again both the eventTime and startTime and endTime are all ISO dates.
example :
let endTime = Moment(new Date());
let startTime = Moment().subtract(days, "days");
"eventTime": "2019-08-07T03:37:40.738Z"
startTime "2019-07-30T00:02:11.611Z"
endTime "2019-08-13T00:02:11.610Z"
End time is 'today'
so in the example here the data time is between the two dates and should be returned.
I looked there : https://docs.mongodb.com/manual/reference/operator/aggregation/gte/
and it should work.. but not the case
I tried:
{eventTime: { '$gte': new Date(startTime), $lte: new Date(endTime)}}
and I get:
MongoError: An object representing an expression must have exactly one field: { $gte: new Date(1564495211043), $lte: new Date(1565704811042) }
also tried:
{ eventTime: {'$gte': new Date(startTime)}}
and get:
MongoError: Expression $gte takes exactly 2 arguments. 1 were passed in.
also tried:
{ $eventTime: {'$gte': new Date(startTime)}}, {$eventTime: {'$lte': new Date(endTime)}}
and get: MongoError: Unrecognized expression '$eventTime'
Any insight would certainly be appreciated
I was able to get it working via toDate:
{
$match:
{
$expr:
{
$and:
[
{ $eq: ["$patientId", patientId] },
{ $eq: ["$dataId", "iraeOverallAlert"] },
{ "$gte": [ {$toDate: "$eventTime"}, startTime.toDate()] },
{ "$lte": [ {$toDate: "$eventTime"}, endTime.toDate()] },
]
}
}
},
Note: This was not needed in the find, but somehow was needed using aggregation. Makes no sense but yah for trial and error.