I will begin using MongoDB in a project and made the following observation:
A ‘Date’, e.g.: ISODate("2000-07-24T00:00:00.000Z") is stored in as UTC 0 (or Zulu).
Let’s assume that an end user submits a plain date, such as “2000-07-24” and his or her local timezone is -5.
Because of the ‘Z’ character in the time (indicates UTC 0), storing the following would be incorrect because the user’s 5 hour difference:
ISODate("2000-07-24T00:00:00.000Z")
I am assuming that it would be more proper to consider the end user’s local time zone into account and store the date as follows:
ISODate(“2000-07-23T19:00:00.000Z”)
By doing this, I can see that any queries would need a timezone adjustment for 'Date'.
By storing all ‘Date’ fields as UTC 0, we can guarantee that exact points in time from people posting from different continents is properly handled.
Does this seem like a good way to handle timezones in MongoDB?
Related
Let's say that I need to create 'chat-like' system. Of course I need to deal with dates somehow.
I read about it a lot and I have some knowledge but I really don't know how to use it.
I would like to store message date in UTC (postgres 12)
Each user should be able to select his time zone and this time zone should be saved into database (standard approach)
When message is retrieved from database I need to convert message date into valid local date based on user selected timezone.
This is really all I need to do and here problem starts:
In postgres date is stored with offset f.e 2020-05-01 00:00:00+02, but I want to store timezone in another table, not here
How can I store user timezone? I should use names like "EST5EDT" or use time offsets as integer?
Where can I find list of all timezones to present user? (Each global website f.e. facebook has list of timezones with offsets, where can I find list of all valid timezones?)
How can I select date with user appropriate timezone? f.e.
SELECT convert_to_user_date("Date", "timezonename??")
FROM "Messages"
Is this correct way to achieve my goal?
These days you don't have to resort to UTC. You can store full timestamps with time zone. This way e.g. you won't lose DST status at the moment the timestamp was recorded in the database.
https://www.postgresql.org/docs/current/datatype-datetime.html
You can easily select the timestamp stored with any time zone shifted to target user's time zone (assuming it's stored somewhere in user preferences). The syntax is SELECT ... AT TIME ZONE
https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-ZONECONVERT
You are very close to a workable setup.
First, use timestamp or timestamp without timezone column data type to store those UTC date/time stamps.
Second, store your users' preferred time zones in varchar(63) columns in the form Asia/Kolkata or America/Halifax.
Third, use postgresql's built in view pg_timezone_names to get a list of valid time zones. You can use it to populate a pulldown list of choices in your user-settings screen.
If you have time for some real excellence in your user-settings screen, you can suggest time zone settings you guess from the users' ip adresses and allow them to change them if your guess was wrong. Read this. How to get Time Zone through IP Address in PHP
Then, when your application starts using postgresql on behalf of any user, look up that user's chosen time zone in your users table, and use it in this SQL command. SET TIME ZONE 'America/Halifax'; (whatever the user's choice is).
Then when you retrieve your time stamps, they will be rendered in the user's local time, and when you store the they'll be in UTC.
The 'toobz are full of advice about this. Here's something that might be useful. How to get Time Zone through IP Address in PHP
Use the data type timestamp with time zone. Don't be worried by the name — that data type really represents an absolute point of time and does not store time zone information.
The only thing you have to do is to set the timezone parameter correctly for the time zone of the client connection, then the value will be represented correctly in that time zone. PostgreSQL does all the work for you.
If you don't like the string representation (e.g., you are disturbed by the time zone offset displayed), use to_char to format the output the way you like:
CREATE TABLE dates (x timestamp with time zone NOT NULL);
SET timezone = 'Europe/Vienna';
INSERT INTO dates VALUES ('2020-06-01 03:00:00');
SET timezone = 'Asia/Kolkata';
SELECT to_char(x, 'YYYY-MM-DD HH24:MI:SS') FROM dates;
to_char
---------------------
2020-06-01 06:30:00
(1 row)
I save my date as a local date, but when I read it back, it treats it as if it was a UTC date so it slips by several hours.
The dates are passed in as strings in the form '2020-03-05 09:05:23' as query parameters but when they are retrieved they might look like '2020-03-04 10:05:23' because I am 13 hours ahead of Greenwich.
For MariaDB (or MySQL):
Use DATETIME as a picture of the clock on the wall.
Use TIMESTAMP to adjust to the system's timezone.
Set the system's timezone according to where it lives in the world.
Is there a standard for encoding a date as a timestamp? My thoughts:
This should be 12:00pm UTC in local time, eg 9:00am at T-3, therefore anyone consuming the timestamp, regardless of their -12/+12 offset, will recognize the same date, regardless of whether they parse at the UTC timezone
It could be 12:00pm at UTC
It could be the start of the day (12:00am) at UTC
It could be start of the day (12:00am UTC) in local time eg 9:00pm at T-3
Is there an official spec or standard to adhere to?
It would be easy to point to this document and say 'this is the standard' as opposed to being unaware and having to change our logic down the line.
There isn't a standard for this, because a date and a timestamp are logically two very different concepts.
A date covers the entire range of time on that day, not a specific point in time.
It may be a different date for a person in another time zone at any given point in time, but dates themselves do not have any association with time zones. Visualize a date as just a square on a calendar, not a point on a timeline.
Many APIs will use midnight (00:00) as the default time when a date-only value is assigned to a date+time value. However:
Whether it is UTC based or local-time based is very dependent on that particular API. There is no standard for this, nor is one answer necessarily better than the other.
Assigning a local-time midnight can be problematic for time zones with transitions near midnight. For example, in Santiago, Chile on 2019-09-08, the day started at 01:00 due to the start of DST. There was no 00:00 on that day.
Also, you tagged your question with momentjs. Since a Moment object is basically a timestamp (not a date), then Moment.js will generally assign the start of the day if provided a date-only value. The time zone involved is key to deciding which moment that actually is, which illustrates my prior points.
For example:
// Parsing as UTC
moment.utc('2019-09-08').format() //=> "2019-09-08T00:00:00Z"
// Parsing as Local Time (my local time zone is US Pacific Time)
moment('2019-09-08').format() //=> "2019-09-08T00:00:00-07:00"
// Parsing in a specific time zone (on a day without midnight)
moment.tz('2019-09-08', 'America/Santiago').format() //=> "2019-09-08T01:00:00-03:00"
Also keep in mind that sometimes APIs can be misnamed. The JavaScript Date object is not a date-only value, but actually a timestamp, just like a moment.
I have a table that I am using to store iso dates with timezones. I realize that dates should "always" be stored as utc but I have an exception to that rule. The timestamps aren't in any way related to the server they are running on. I want to be able to store an iso date like this:
2016-03-06T01:15:52-06:00
And regardless of the time zone of the server or anything else I want the timestamp returned as:
2016-03-06T01:15:52-06:00
Currently if I insert an iso date it automatically converts it to whatever the server timezone is. My above date gets converted to:
2016-03-06 07:15:52+00 (server is utc)
The only thing I can think of is storing the timezone offset in a separate column, storing my date as utc and then converting using the offset column, horribly messy. Surely there is a way to store my date in one column and get it out the way it was originally created?
Your proposed solution is correct. Or more precisely, it is one of several correct implementations. Any of the following would work:
Store the UTC timestamp in one field, store the offset in another.
Store the local timestamp in one field, store the offset in another.
Store the local date in one field, and store a time with time zone in another. (though time with time zone is generally discouraged...)
Store the UTC timestamps in one field and the local timestamp in another.
The easiest by far is the first one, which you already proposed.
I'd avoid against storing timestamps in text fields, as they tend not to be very efficiently searchable.
Also note - if you're coming from a SQL Server background, you might recall its datetimeoffset type, which stores the local datetime and offset in the field, and uses the UTC equivalent during indexing. It's common to think that Postgres and MySQL's timestamp with time zone would have the same behavior, but they don't. They simply use the session time zone to convert to/from UTC. SQL Server has no concept of a session time zone, and thus the discrepancy.
Be sure to read this part of the Postgres docs.
I have various timestamps stored in Mongo collections, some as floats and some as ints.
They are all stored in BST and the server will be switched to UTC soon. How do I convert them within Mongo to be UTC timestamps?
In MySQL I can do this:
UPDATE `table` SET `field` = CONVERT_TZ(`field`, 'Europe/London', 'UTC');
Is there a Mongo equivalent?
You'll have to use your language of choice and update them one at a time. It should be as simple as a for loop that loads the data and rewrites it.
Just double-check how your language of choice handles timestamps across timezones. Making such a data change can have all kinds of unexpected effects on production code.
Timestamps are generally in UTC and not in a specific timezone. All date/time libraries I've worked with return timestamps that are the number of seconds (or milliseconds) since Jan 1st 1970 UTC. Check the documentation for the library you used to create the timestamp to be sure.
This means that you should be ok, unless you have used a date/time library that does not follow this convention, or somehow calculated the timestamps yourself and accounted for the timezone.
For example in JavaScript if you store the value returned from new Date().getTime() and later pass that value to new Date(...) on a different system you will end up with the same absolute date/time, regardless of the timezones of the two systems. The same goes for Ruby, do Time.new.to_i on one machine, and run Time.at(...) on another and you will get the same absolute date/time. When I say "absolute date/time" I mean that it's UTC time will be the same, most likely the system will display it in the local time zone, but that is what you want.
Some points to consider about date in Mongo:
All dates are stored in UTC in MongoDB
MongoDB stores dates internally as a 64 bit integer representing
milliseconds since 1970-01-01T00:00:00Z
If the date value you provide is not already in UTC it
will be converted to UTC before being stored in MongoDB by the driver
The recommendation is not to use DateTime.Parse.
You will get into all sorts timezone issues regarding how DateTimes are formatted.
Instead just use one of the DateTime constructors with UTC flavor.
Example:
var dateTime = new DateTime(2011, 5, 5, 0, 0, 0, DateTimeKind.Utc);
Hope you find it useful.