Timezone query on MongoDB - mongodb

I'm building an application wherein users can be located in various timezones, and the queries I run are sensitive to their timezones.
The problem I'm having is that MongoDB seems to be ignoring the timezone on query time!
This is an example of a date field "2019-09-29T23:52:13.495000+13:00", here's the full json:
And this is an example of a query:
{
"at": {
"$gte": "2019-09-29T00:00:00+00:00",
"$lte": "2019-09-30T00:00:00+00:00"
}
}
All of my dates are saved either with +12 or +13 because those are the timezones where my customers are at the moment, essentially what that means for the above query is that I should be seeing some results from the 2019-10-01:00:00:00+13:00 due to the first of October still being the 30th of September in UTC, and I'm not.
I'm new to this and not too far down the rabbit hole so I'm open to refactoring/change of thinking if it will make things easier.
For context, in case it makes a difference I'm using PyMongo, and my MongoDB version is 4.2
EDIT!
I've tried converting the "at" field into date, but the timezone seem to have been suppressed or is not visible
With that I also had to change the way I query
{ "at": {
"$gte": ISODate("2019-09-29T00:00:00+00:00"),
"$lte": ISODate("2019-09-30T00:00:00+00:00")
}
}
Didn't help

MongoDB uses the UTC timezone when it stores date. Using your example date:
> ISODate("2019-09-29T23:52:13.495000+13:00")
ISODate("2019-09-29T10:52:13.495Z")
So the +13:00 timezone is converted to Z (UTC) timezone by ISODate(), and this is the value stored in the database.
> db.test.insert({_id:0, at: ISODate("2019-09-29T23:52:13.495000+13:00")})
> db.test.find()
{ "_id" : 0, "at" : ISODate("2019-09-29T10:52:13.495Z") }
Note that "at" is stored in UTC time instead of +13 as inserted.
Querying using ISODate() as per your example works as expected:
> db.test.find({ "at": {
... "$gte": ISODate("2019-09-29T00:00:00+00:00"),
... "$lte": ISODate("2019-09-30T00:00:00+00:00")
... }
... })
{ "_id" : 0, "at" : ISODate("2019-09-29T10:52:13.495Z") }
The timezone information is not visible due to the visual tool you're using. It is visible in the mongo shell as Z, as in the above example.
The advantage of storing UTC is that you can convert them to any other timezones without worrying about daylight saving time. There are date specific aggregation operators that deals with timezones and convert between them. See Date Expression Operators and https://stackoverflow.com/a/48535842/5619724 for examples.

Related

MongoDB: Migrate some YYYY-MM-DD dates to ISO 8601

I have a field with dates stored in different formats (due to a code change at some point):
Some documents in the same collection use ISO 8601 ("2022-06-27T00:00:00.000Z"), while some others use the YYYY-MM-DD syntax ("2022-06-27").
This is a problem, because this query fails to fetch the YYYY-MM-DD documents (while it does fetch the ISO ones):
filter:
{"where":{"and":[{"notificationOfNeedDate":{"gte":"2019-04-08T21:00:00.000Z"}}]},"order":["aCode DESC"],"limit":24,"skip":0}
I believe the solution is to migrate all YYYY-MM-DD to ISO dates
Is there a command to run in mongodb in order to "UPDATE" all YYYY-MM-DD to ISO dates (for that collection and field), respecting the locale timezone offset and the DST settings for that day?
Thanks
Another example why date values should never be stored as string, it's a design flaw. Store always proper Date objects, so you go in the right direction.
Your question is not really clear, string "YYYY-MM-DD" is also a valid format according to ISO 8601 (a date, just without time information).
You can use $dateFromString:
db.collection.updateMany(
{ timestamp: { $type: "string" } },
[ {
$set: {
notificationOfNeedDate: {
$dateFromString: {
dateString: '$notificationOfNeedDate',
timezone: 'Europe/Athens'
}
}
}
} ]
)
You don't need to specify format, because ISO-8601 format is the default.
Then in your query you need to filter also on Date values:
db.collection.find({ notificationOfNeedDate: { $gte: ISODate("2019-04-08T21:00:00.000Z") } })
Of course, ISODate("2019-04-09T00:00:00+03:00") works also

Azure Data Factory (ETL) from MongoDB to Azure Synapse Anlytics - Date Conversion

I am building ETL pipeline using Azure DataFactory from MongoDB to Azure Synapse Anlytics.
This is my sample document
{
"_id" : ObjectId("62b6f74ce3a7b8ee664c675d"),
"event_name" : "click_search",
"screen_name" : "home",
"product_sku" : "",
"screen_id" : "5005",
"action_id" : "1023",
"created_on" : ISODate("2022-06-25T16:53:48.000+05:00")
}
I used simple Copy Data activity and then preview data for source (MongoDB) is showed me like this
{
"_id": {
"$oid": "62b6f74ce3a7b8ee664c675d"
},
"event_name": "click_search",
"screen_name": "home",
"product_sku": "",
"screen_id": "5005",
"action_id": "1023",
"created_on": {
"$date": 1656158028000
}
}
My First question is why date is coming in different format in datafactory?
Secondly
while mapping it show me like this
due to this when I ran pipeline it gives this error
Message=Column 'created_on' contains an invalid value '1656158028000'. Cannot convert
'1656158028000'to type 'DateTime'.,Source=Microsoft.DataTransfer.Common,
''Type=System.FormatException,Message=String was not recognized as a valid
DateTime.,Source=mscorlib,'
I tried to find this issue in documentation Copy data from or to MongoDB using Azure Data Factory or Synapse Analytics
but didn't get any luck, please let me know what's happening under the hood? Why MongoDB send date like this and how can we resolve it. Thank you.
MongoDB stores date as BSON date
BSON Date is a 64-bit integer that represents the number of milliseconds since the Unix epoch (Jan 1, 1970). This results in a representable date range of about 290 million years into the past and future.
For your case, you can use toUTC to convert UNIX timestamp to proper date in ADF. One thing to note is that MongoDB stores unix timestamp in milliseconds, so you may need to divide by 1000 before converting to UTC date.

MongoDB aggregate lookup let variables problem

I've create a mongoplayground to make it easier to understand.
https://mongoplayground.net/p/8zhkcMyLkmy
Basically, I'm stuck between line 108 ~ 111, because in MongoDB Compass the same aggregate pipeline doesn't work.
Here's the images:
If I just change '$$date' to its date:
Even if I insert the variable inside date function:
Does anyone have any thoughts in why it doesn't work in compass and works in mongoplayground?
For best practice, when comparing date string, it is required to convert as date first (via $toDate).
$lte: [ { $toDate: "$date" }, { $toDate: "$$date" }]
Comparison/Sort Order (MongoDB Manual) - String section
By default, MongoDB uses the simple binary comparison to compare strings.
Hence, comparing dates in string format will lead to an inaccurate result.

Reversing date conversion using specific format in MongoDB

I convert the date to a string so that I can use it for various grouping related work. However in certain groupings the reverse conversion fails.
db.test.aggregate({
$addFields: {
date: "$date",
grouped_format: {
"$dateToString": {
date: '$date',
format: "%Y-%m",
timezone: 'Europe/London',
}
}
}
}, {
$addFields: {
converted: {
"$dateFromString": {
dateString: '$grouped_format',
format: "%Y-%m", // if i comment this out, it works,
timezone: 'Europe/London',
}
}
}
})
The error I get is
{
"message" : "an incomplete date/time string has been found, with elements missing: '2016-01'",
"ok" : 0,
"code" : 241,
"codeName" : "ConversionFailure",
"name" : "MongoError"
}
If I comment the format under $dateFromString, I get the desired result.
{
"_id" : ObjectId("60ca299f10949130abbe5c8e"),
"date" : ISODate("2014-04-21T00:00:00.000Z"),
"grouped_format" : "2014-04",
"converted" : ISODate("2014-03-31T23:00:00.000Z") // London is one hour ahead due to DST so this is correct. -> start of the month
},
{
"_id" : ObjectId("60ca2aaf10949130abbe5c8f"),
"date" : ISODate("2021-01-02T00:00:00.000Z"),
"grouped_format" : "2021-01",
"converted" : ISODate("2021-01-01T00:00:00.000Z") // London is even with UTC so this is correct. -> start of the month
}
Unfortunately, I want to be able to specify any custom format in $dateToString and have it be converted back.
As an example, if the custom format is "%G-%V"which is the ISO week year and ISO week number, specifying the format in $dateToString and $dateFromString works correctly.
I've tried the following formats and here are the results. As you can see, the No. 4. in the table does not specify the minute but it still works fine. However any variation of the date from No.6. to 8. fails.
If I can't reliably reverse the value then I will need to create a special function that covers all edge cases but to do that I need to understand why this is happening in the first place.
No
Format
Works/Fails
1
'%Y-%m-%dT%H:%M:%S.%L'
Works
2
'%Y-%m-%dT%H:%M'
Works
3
'%Y-%m-%dT%H'
Works
4
'%Y-%m-%d'
Works
5
'%G-%V'
Works
6
'%Y-%m'
Fails
7
'%Y-%m-01'
Fails
8
'%Y-%m-00'
Fails
According to the documentation, the default formatter is "%Y-%m-%dT%H:%M:%S.%LZ" but specifying it explicitly also does not work and errors out. No string or null/undefined value seems to work while converting "%Y-%m" unless you omit the format string entirely in $dateFromString. This makes the behaviour very strange.

mongodb date query don't work with ISODate?

I can find an item with this search:
db.item.find({"startdate":{$gte: 1485521569000 }})
The date seems to correspond to
> new Date(1485521569000)
ISODate("2017-01-27T12:52:49Z")
But if I search
db.item.find({"startdate":{"$gte" : ISODate("2017-01-01T00:00:00Z")}})
I don't get any results. What am I doing wrong?
PS
the only way I found is
db.item.find({"startdate":{"$gte" : (new Date("2017-01-01")).getTime()}})
is that right or there is a better way?
Below one will work.
db.getCollection('test').find({
"key": {
"$gte": (ISODate("2017-01-01")).getTime()
}
})
Reason:
You have your data is in int64 or long
In the new Date query, you are converting date to time which return int. So get performs integer comparison.
In ISODate you are passing a date, it doesn't convert date to integer i.e milliseconds. So if you convert, both will work.
new Date() returns the current date as a Date object. The mongo shell wraps the Date object with the ISODate helper
var d = ISODate("2017-01-01")
print(d); //Sun Jan 01 2017 05:30:00 GMT+0530 (IST)
Hence, the comparison fails.
If i understand, you have timestamp in startdate field,
This is the option if helps you, Can do it with aggregate() function, using $toDate,
$toDate convert timestamp to ISO date
db.collection.aggregate([
{
$match: {
$expr: {
$gte: [
{
$toDate: "$startdate"
},
ISODate("2017-01-27T12:52:49.000Z")
]
}
}
}
])
Playground: https://mongoplayground.net/p/H7SFHsgOcCu