I have a node.js application that stores many dates in a database. They are stored in the ISO format, such as '2016-11-02T16:30:12-04:00'.
Some fields which are dates are just dates, other are date/times. An example of a date/time would be "last modified" for a record, where a person's birthday is just a date.
The question is about best practices for storage and query patterns on these things. Because a date always has a time, you must choose how to store for example a birthday. Following the 5 laws of API dates and times this is of course done in UTC.
There are edge cases though where proper API behavior seems unclear. Suppose someone submits a birthdate to the API of '2016-11-02T16:30:12-04:00'. This is bad news, because a search like /users?birthdate=2016-11-02 will fail, as that date will get converted to '2016-11-02T00:00:00Z' and fail to match in the DB. What then should correct behavior be?
When someone POSTs a user, convert date fields into dates at midnight UTC, and then have the convention that querying birthdates should assume the same?
Convert date queries for certain fields into implicit ranges, i.e. searching for 2016-11-02 is really looking for 2016-11-02T00:00:00Z <= x <= 2016-11-02:23:59:59Z?
Match only on the exact moment, and rely on the client to know that a birthday of '2016-11-02T16:30:12-04:00' really means 4:30PM EST, and does not mean just on November 2nd?
What's the established pattern / best practice here for distinguishing between dates and datetimes?
I have been studying REST best practices and standards a lot for a while and I can't recall reading anything about that, but for the usage of ISO standard. From your description it seems to be something that really depends on the application and its use-cases.
I would go for your option #2: if a GET request comes with a date but no time, consider it a query for the whole day, and do the "conversion" in your GET response server code. Maybe you'd want to support both a "date" and a separate "time" query string parameters if the precise time might matter occasionally. This can also help you to keep clients "unaware" of the database storage format you choose, and may even allow you to support localized date formats.
The problem here is the usage of UTC, which implies that there's a time associated with it. There's not, a birthdate is considered (in iCalendar) a 'floating date' and does not have a specific time associated with it.
If your birthdate is November 3rd, and you move to Australia, your birthdate does not actually change to November 2nd, because your birthdate does not have a time, does not have a timezone and is the same where ever you are in the world.
The solution is simple. If you allow users to submit a date/time for birthday searches, then you should just 'cut off' the time and timezone. Assume that you're only going to be using the date portion and just search your database based on that.
Ideally you don't allow users to submit a time at all though. I think this just creates confusion. Just force api clients to submit a date only.
Those '5 laws' are an extreme over-simplification and don't apply to many situations.
Related
I am creating a form (using Typeform, but doesn't really matter) in which I need to understand the time zone of my customers. So far I have left this as an open questions, but I would like to provide an extensive list, to ensure consistency on my database.
I could provide a list of time zones abbreviation (such as this one), but then I'd have the problem of daylight saving times. That is, let's say a customer is in the UK, they will see both "Greenwich Mean Time" and "British Summer Time" on the list, and would be answering differently depending on the time of the year.
How can a produce a meaningful non-redundant plaintext list of timezones?
Discussing data time-formats, someone mentioned to me how he stores datetime (in a human-readable format) using floats as yyyymmdd.hhmmss, so 2019-09-18, 11:29:30am would become 20190918.112930
I'm trying to find out if this guy has invented his own format or if it is used (and described) elsewhere too - and if so, how is it even called...?
It’s probably homespun
I have seen a lot of date and time formats, and I have not seen this one before. My go is that his guy or his organization invented it themselves.
Edit: Thank you for confirming in the comment. Since comments are not always permanent on Stack Overflow, I quote here, you said:
Finally got confirmation from the source: it's homespun indeed.
As an aside I don’t like it. A float is stored in a binary format internally, and only after formatting it into decimal does it become human readable. Using a float for a “human readable” date and time was not what formatting of floating-point numbers was meant for, it’s a hack.
Use ISO 8601
For a human-readable format I recommend ISO 8601. Here 2019-09-18, 11:29:30am becomes 2019-09-18T11:29:30. Or even better and still within ISO 8601, convert to UTC and append a Z to denote UTC. So if your original time was in Europe/Berlin time zone, it would become 2019-09-18T09:29:30Z. As you can see, ISO 8601 is even more human readable than you friend’s format, and it is sortable as strings (as long as the years don’t go beyond 9999).
While he may have come up with it himself, it is also a formatting option in zipinfo.
The manual doesn't explicitly name it, but describes it as a sortable decimal format and decimal format.
Not sure if we are talking about SQL date format. If so, this date format is present in SQL Statements.
Not sure about the name, it's called in different ways: non-standard, ISO, Other format and so on.
Is present also in PHP.
According to Wikipedia, this would be similar to the ISO 8601, which permits, all of the following for date and time combined:
2019-09-18T09:18:26+00:00
2019-09-18T09:18:26Z
20190918T091826Z
except that the T to separate the time from the date is replaced by . and the time-zone information is dropped.
That specific format has limited popularity either in the yyyymmdd.HHMMSS or the C's strftime()-compatible %Y%m%d.%H%M%S form.
EDIT
As far as using float for date and time the way you suggests, it depends on the precision and machine representation.
If the system is following IEEE 754 basic standard (which is what most modern C compiler stick to), you would need at least float64.
However, it is not common to do so.
This might be in part because it may be difficult to correctly predict the accuracy of the time information, and it is not as bit-efficient as the Unix time.
Given that the only positive feature it has is that it can rely on standard %f from sprintf(), I would only see it advantageous when strftime() is not available or a performance bottleneck.
What is the best way to transfer Dates and Times across. I am using GWT on the client/browser side and .NET C Sharp on the server and I am using JSON as data-interchange format. I am currently storing all the dates and times on the server as .NET DateTime. Now I have noticed, that if I use the GWT DatePicker or DateBox to pick a date and send it to the server as miliseconds (by doing date.getTime()) where the server takes this param as DateTime, I can see an hour offset due to the BST. I have situations where I have to have the date and time in separate boxes on the UI and the time setting along with the correct date is crucial because of scheduling.
The best way to interchange Date and Time values would be to serialize them into culture-independent, UTC based strings like: 2010-09-18T18:37:11. The problem is, Date and Time related operations tend to be implemented incorrectly...
As for your problem, I assume that it pops up during deserialization of JSON time, i.e. .Net treats this time as local (DateTimeKind.Local or DateTimeKind.Unspecified), thus converting it. Not sure how to deal with it, the brute force would be probably sending serialized string like above and deserializing manually like this:
DateTime date = DateTime.Parse(dateString, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal);
I'd recommend using standard such as ISO 8601 to transfer date time info in a string form. At my company, date time information encoded in JSON object is almost always in this format, e.g. "2015-10-12T18:41:11+01:00". This string can be parsed and understood correctly in all clients with different programming languages (Obj-C, Java, C/C++).
I've been searching for a definitive answer to this, and the XML schema data types document seems to suggest that timezones are accepted, yet I found at least one implementation which does not properly convert time zones ( NUSOAP ).
To make sure that the problem is not at my end, I'd like to know if a format such as 2009-11-05T11:53:22+02:00 is indeed valid and should be parsed with timezone information, i.e. as 2009-11-05T13:53:22.
Given the following sentences from the w3c schema documentation:
"Local" or untimezoned times are
presumed to be the time in the
timezone of some unspecified locality
as prescribed by the appropriate legal
authority;
and
When a timezone is added to a UTC
dateTime, the result is the date and
time "in that timezone".
it does not sound like there is a definitive answer to this. I would assume that it is the usual ambiguity: Both versions are principally valid, and the question of what version to use depends on the configuration/behavior/expectations of the system one is interfacing with.
And even if there where a definitive answer, I would definitely not rely on it, but rather expect that every other web service and library had its own way of dealing with this :/
You converted the timezone incorrectly.
2009-11-05T11:53:22+02:00
is equivalent to
2009-11-05T09:53:22Z
Is that what NUSOAP did?
Should I store it in a single timestamp/datetime field or separate date and time fields? I prefer to store it as a single timestamp field but I need to know when a user didn't enter a time portion.
How to best store this in the database? I'm using postgresql.
There are definitely reasons why this is a bad idea. There are also reasons why your choices are limited. It's a bad idea because it's a single data item and for the more practical reason that you can't store a timezone if you have two fields.
As mentioned above, nulls are the obvious benefit of using two fields.
You might also want to consider using a single datetime field and storing a flag to indicate whether or not the user entered a time. This could be a boolean flag. However, you will still need to think about how you are going to use this data - entering only a date into a datetime field will lead to a time component being set to midnight. This will have implications in sorting and selection. Additionally, if you are storing timezones, you will have to be very careful when you use the data.
In order to fulfill your requirement of knowing whether or not a time was entered you will need to have two fields. You do not need the second field to be a time though.
The obvious answer is to use two separate fields; then you can use NULL values.
If you choose to use one field you will need to choose a magic time part that signifies "didn't enter a real time", which has the danger of coinciding with a real time (however unlikely).
Also, if you intend to use the date and time part separately often, then it might also be convenient to use separate fields; otherwise you will often need to use selection functions for extracting the relevant part of a field.