How to query with Dates in Aggregate functions in mongodb - mongodb

I'm using Metabase v0.41 and writing query to Cosmos DB as a mongoDb.
Metabase only allows execution of db.collection.aggregate, and I first want a match stage that does a basic string match and date range.
In SQL it'd be something like this:
SELECT *
FROM TABLE
WHERE TABLE.COLUMN = 'WORD' AND TABLE.CREATEDAT > DATE_ADD(NOW(), INTERVAL -1 DAY)
and this is what I have for the equivalent mongo query.
So here are several attempts that I tried in the match query
1)
{
$and: [
{
column: {$eq: "word" }
},
{
createdAt:{$lt : //new ISODate()
{
$dateAdd:{
startDate: new ISODate(),
unit: "day",
amount: -1
}
}
}
}
]
}
-> doesn't work even on mongo compass,this returns no results even if there are matching records in the database. but no syntax error either.
Subtract the difference in millisec
{
$and: [
{
column: {$eq: "word" }
},
{
createdAt:{$lt :new ISODate() -1231312321}
}
]
}
-> this does work on mongo compass, but not on Metabase.
the parser on the Metabase doesn't parse the number properly.
"JSON reader was expecting a name but found '-1231312321'."
I tried the subtraction using $subtract but still doesn't work.
Have any ideas?

Try
{ $expr:
{ $lt: [
"$createdAt",
{
$dateSubtract:{
startDate: "$$NOW",
unit: "day",
amount: 1
}
}]
}
}

Related

MongoDB - Dates between using $match

So I try to use MongoDB $match to get data between 2 dates, but it turns out that the data is not returning a.k.a empty here. What it looks like:
db.collection.aggregate([
{
$match: {
date: {
$gte: new Date("2022-10-23"),
$lt: new Date("2022-10-25"),
},
}
},
{
$group: {
_id: "$title",
title: {
$first: "$title"
},
answer: {
$push: {
username: "$username",
date: "$date",
formId: "$formId",
answer: "$answer"
}
}
}
},
])
Here is the data that I try to run on the Mongo playground:
https://mongoplayground.net/p/jKx_5kZnJNz
I think there is no error with my code anymore... but why it gives an empty return.
Migrate the comment to the answer post for the complete explanation.
Issue 1
The document contains the date field as a string type while you are trying to compare with Date which leads to incorrect output.
Ensure that you are comparing both values in the exact type.
Either that migrate the date value to Date type or
converting the date field to Date type in the query via $toDate.
{
$match: {
$expr: {
$and: [
{
$gte: [
{
$toDate: "$date"
},
new Date("2022-10-23")
]
},
{
$lt: [
{
$toDate: "$date"
},
new Date("2022-10-25")
]
}
]
}
}
}
Issue 2
Since you are using $lt ($lt: new Date("2022-10-25")), it won't include the documents with date: new Date("2022-10-25").
For inclusive end date, you shall use $lte.
Demo # Mongo Playground

Mongo database query operation question on dates as string

Let’s say some external component inserts data into my mongo database with Field called “startDate”. They store the value as string in mongo db in format yyyymmdd-HHmmss.
Can I still run the date time queries to fetch documents between two dates ?
Will below work as expected for above scenario?
db.test.find({ startDate:{$gt: “20201121”, $lte: “20210121”}})
Yes, it will but not for the same end date. It'd be better to add a day to cater for trailing zeros.
//data
[
{
startDate: "20210101-010101"
},
{
startDate: "20210121-000000"
},
{
startDate: "20210121-010000"
}
]
// query 1 - not successful maybe
[
{
startDate: "20210101-010101"
},
{
startDate: "20210121-000000"
},
{
startDate: "20210121-010000"
}
]
//query 2, successful
db.collection.find({
startDate: {
$gt: "20201121",
$lt: "20210122"
}
})

How do I do calculation in sort in aggreation mongoose

I am trying to do this calculation but I am having some trouble. Anyone have any ideas how I can do calculation inside my aggregation? I wanna do something like this:
const test = await Test.aggregate([
{
$sort: {
$divide: [
'value',
Math.pow(1.1, new Date() - 'date'),
],
},
},
]);
For example here, I wanna do 1.1^number of days has passed. The Test schema has a "value" of type Float and a date of type date.
You have two issues here:
You're trying to use javascript functions within the pipeline, while this is possible by using $function it is not recommended, especially if you can execute the same logic using Mongodb operators.
$sort stage has this following structure:
{ $sort: { <field1>: <sort order>, <field2>: <sort order> ... } }
As you can tell it's not being followed in your example as you're trying to use an expression.
So how can we solve these?
Well you can use $pow instead of Math.pow, $$NOW instead of new Date() and $subtract instead of the - javascript operator.
You will also need to add a "sortField" to sort by to match the $sort stage structure, all of this would look like this:
db.collection.aggregate([
{
"$addFields": {
"sortField": {
$divide: [
"$value",
{
$pow: [
1.1,
{
$subtract: [
"$$NOW",
"$date"
]
}
]
}
]
}
}
},
{
$sort: {
sortField: 1
}
}
])
Mongo Playground
Mind you subtracting dates will give you result in miliseconds, you will have to divide it by the required number ( 60 * 1000 * 60 * 24 for a day ) to get the right mesaurement.

Date range not working in aggregation pipeline, but works in find()

I am trying to filter data by a date range. Example return the data that was created no more than 14 days ago.
I can do this in find with the following:
{
$match: {
eventTime: { $gte: startTime.toDate(), $lte: endTime.toDate() }
}
}
eventTime is an ISO date as well as startTime and endTime
I am using an aggregation and a lookup and trying to implement the same thing:
{
$lookup:
{
from: "data",
let: { dataId: "$dataId", patientId: "$patientId" },
pipeline: [
{
$match:
{
$expr:
{
$and:
[
{ $eq: ["$patientId", patientId] },
{ $eq: ["$dataId", "$$dataId"] },
{ $gte: ["$eventTime", startTime.toDate()] },
{ $lte: ["$eventTime", endTime.toDate()] },
]
}
}
}
],
as: "data"
}
}
But no data results are returned. If I remove the dates I get all the correct data based on dataId and patient. so the join is working.. but somehow the date range is not.
Again both the eventTime and startTime and endTime are all ISO dates.
example :
let endTime = Moment(new Date());
let startTime = Moment().subtract(days, "days");
"eventTime": "2019-08-07T03:37:40.738Z"
startTime "2019-07-30T00:02:11.611Z"
endTime "2019-08-13T00:02:11.610Z"
End time is 'today'
so in the example here the data time is between the two dates and should be returned.
I looked there : https://docs.mongodb.com/manual/reference/operator/aggregation/gte/
and it should work.. but not the case
I tried:
{eventTime: { '$gte': new Date(startTime), $lte: new Date(endTime)}}
and I get:
MongoError: An object representing an expression must have exactly one field: { $gte: new Date(1564495211043), $lte: new Date(1565704811042) }
also tried:
{ eventTime: {'$gte': new Date(startTime)}}
and get:
MongoError: Expression $gte takes exactly 2 arguments. 1 were passed in.
also tried:
{ $eventTime: {'$gte': new Date(startTime)}}, {$eventTime: {'$lte': new Date(endTime)}}
and get: MongoError: Unrecognized expression '$eventTime'
Any insight would certainly be appreciated
I was able to get it working via toDate:
{
$match:
{
$expr:
{
$and:
[
{ $eq: ["$patientId", patientId] },
{ $eq: ["$dataId", "iraeOverallAlert"] },
{ "$gte": [ {$toDate: "$eventTime"}, startTime.toDate()] },
{ "$lte": [ {$toDate: "$eventTime"}, endTime.toDate()] },
]
}
}
},
Note: This was not needed in the find, but somehow was needed using aggregation. Makes no sense but yah for trial and error.

MongoDB exception Resolving [duplicate]

I am getting an issue for running the following aggregate query:
db.snippets.aggregate([ { '$project': { month: { '$month': '$created_at' }} } ])
The error message for the same is:
assert: command failed: {
"errmsg" : "exception: can't convert from BSON type EOO to Date",
"code" : 16006,
"ok" : 0 } : aggregate failed
How do I get around this issue? I found a related question: MongoDB: can't convert from BSON type EOO to Date.
But it doesn't tell me how to get things done.
You likely have one or more docs with a created_at value that's not a BSON Date and you'll need to fix that by converting those values to Date or removing them.
You can find those docs with a $not query that uses the $type operator like:
db.snippets.find({created_at: {$not: {$type: 9}}})
If the created_at values are date strings, you can find the docs that need updating and then update them in the shell using code like:
db.snippets.find({created_at: {$not: {$type: 9}}}).forEach(function(doc) {
// Convert created_at to a Date
doc.created_at = new Date(doc.created_at);
db.snippets.save(doc);
})
try this one, its help for me above problem.
db.snippets.aggregate([{
'$project': {
month: { $substr: ["$created_at", 5, 2] }
}
}]);
above code get month wise
data is entered into the database in ISO format which can then be easily worked with.
In some situations, some documents are supposed to have empty Date fields. In those cases, you could try this (using your example):
db.snippets.aggregate([ { '$project': { month:
{ $cond: [{ $ifNull: ['$created_at', 0] }, { $month: '$created_at' }, -1] }} } ])
In this example, we would get -1 in the cases whenever no field '$created_at' is found. For all the other cases, we would get the Date month.
I had a related issue, but in my case the Date fields were the members of an array, so the error was "can't convert BSON type Object to Date".
I needed to get the day of week from the dates in the possibleTripDateTimes array.
Sample document:
{
"possibleTripDateTimes" : [
{
"tripDateTime" : ISODate("2015-08-01T06:00:00.000-0700")
}
]
}
The fix was simply to use dot notation to address the array member fields.
db.trips.aggregate([
{
$project: {
departTime: {
$map: {
input: "$possibleTripDateTimes.tripDateTime",
as: "dateTime",
in: { $dayOfWeek: "$$dateTime" }
}
}
}
}
]
);
I hope this helps someone who also gets zero search results on the "BSON type Object" search
I had the same problem, I figured that the date field is missing for some of the documents causing the conversion to fail. I just added a match clause to filter these out. But ofcourse i am investigating on my app side why they are not being populated.
db.snippets.aggregate([
{
'$match': {
'created_at': {
"$exists": true
}
}
},
{
'$project': {
month: {
'$month': '$created_at'
}
}
}
])
I had a similar problem, and solved it checking if the date existed.
db.users.aggregate([
{$project:{day: { $cond: ["$bd", { $dayOfMonth: "$bd" }, -1] },
month: { $cond: ["$bd", { $month: "$bd" }, -1] },
year: { $cond: ["$bd", { $year: "$bd" }, -1] }
}},
{$match:{"month":1, "day":15}}
])
My date field is bd and with that match I'm getting all users that have their birthday on January 15th.
This error can also appear if you have incorrectly named your properties in your aggregation relative to what they are in your database.
For example my code was
$group: {
_id: {$week: "$projects.timeStamp"},
total: { $sum: "$projects.hours" }
}
But I hadn't camelcased timestamp in my database so simply modifying to projects.timestamp fixed it.
First, you need to check whether the data type is in ISODate. IF not you can change the data type as the following example.
db.collectionName.find().forEach(function(each_object_from_collection){each_object_from_collection.your_date_field=new ISODate(each_object_from_collection.your_date_field);db.collectionName.save(each_object_from_collection);})
Now you can find it in two ways
db.collectionName.find({ $expr: {$eq: [{ $year: "$your_date_field" }, 2017]}});
Or by aggregation
db.collectionName.aggregate([{$project: {field1_you_need_in_result: 1,field12_you_need_in_result: 1,your_year_variable: {$year: '$your_date_field'}, your_month_variable: {$month: '$your_date_field'}}},{$match: {your_year_variable:2017, your_month_variable: 3}}])
First, you can identify the particular field which is causing an issue as follows:
db.collectionName.find( { 'dateField' : { $type : 2 } } )
The above line checks and finds all the documents with field name 'dateField' having type as String(referred as $type - 2).
Once it is identified and verified, we can modify those records as follows:
db.collectionName.find( { 'dateField' : { $type : 2 } } ).forEach( function (x) {
x.dateField = new ISODate(x.dateField);
db.collectionName.save(x);
});
In my case I had to use "$toDate" and it worked:
db.snippets.aggregate([ { '$project': { month: { '$month': {$toDate: '$created_at'} }} } ])