I'm currently writing a driver for MongoDB, so I have to dig a little deeper and so I find this:
BSON spec for DateTimeUTC:
"\x09" e_name int64
BSON spec for int64:
"\x12" e_name int64
BSON spec for timeStamp (although I know its almost always used internally, its just to show BSON makes use of unsigned integers):
"\x11" e_name uint64
It seems a bit controversial to me. Why are int64 and utc millis even separated? Does mongoDB use different ways to compare different BSON dateTimeUTCs?
And why is dateTimeUTC NOT a uint64 but a signed integer? millis are always > 0. Is there a reason behind this? Am I missing something?
DateTimeUTC is used to represent a point in time. It predates BSON, and has historically been using signed integer. This is to enable the use of DateTimeUTC to point to a date before the epoch. Otherwise, it won't be possible to represent dates before 1970-01-01 using DateTimeUTC.
In contrast, timestamp is for mostly internal use, and is expected to be used for mostly current dates that have little need to represent a time before the epoch (e.g. the timestamp of an operation).
There's a related question in UNIX StackExchange regarding this: Why does Unix store timestamps in a signed integer?
Related
we need sending some objects from database of various types within long-polling by rest. Data are sent and each record contains timestamp. When client receive new data from server he should create another poll request with record's timestamp as parameter which helps to specify following data records.
I consider about epoch unix time and store this value in each record in database to filtering and also this value will be sent with each poll requests.
What do you think about this solution? Is that usage fine or should I worry about something? Thanks.
EDIT:
I forget notice these data will be added by clients in different time-zones. This is also another reason why I consider use unix time.
Any format of storing the timestamp is fine, as long as users will be able to unambiguously interpret it. There is no reason for timestamp format in API to be the same as in database. Idea of API is to decouple model from database.
Personally I would choose one format from ISO 8601 Basic and Extended Notations. Example: 2008-09-15T15:53:00. In virtually all programing languages there are methods to handle this format (cast to unix timestamp or to internal date/time classes). For java you would use java.time.LocalDateTime#parse
Unix timestamp has some issues (they may be or not may be issues for you)
unable to represent dates before January 1st, 1970
unable to represent dates after January 19, 2038
not human-readable
does not contain timezone (timestamp itself does not have concept of timezone, but it may be useful to send client timezone along with timestamp. server may always normalise the value to UTC)
The data I'm receiving has timestamps down to nanoseconds (which we actually care about). Is there a way for Postgres timestamps to go to nanoseconds?
As others have pointed out, Postgres doesn't provide such type out of the box. However, it's relatively simple to create an extension that supports nanosecond resolution due to the open-source nature of Postgres. I faced similar issues a while ago and created this timestamp9 extension for Postgres.
It internally stores the timestamp as a bigint and defines it as the number of nanoseconds since the UNIX epoch. It provides some convenience functions around it that make it easy to view and manipulate the timestamps. If you can live with the limited time range that these timestamps can have, between the year 1970 and the year 2262, then this is a good solution.
Disclaimer: I'm the author of the extension
Nope, but you could trim timestamps to milliseconds, and store nanosecond part to a separate column.
You can create index on both, and view or function to return your wanted nanosecond timestamp, and you can even create index on your function.
On the one hand, the documented resolution is 1 microsecond.
On the other hand, PostgreSQL is open source. Maybe you can hack together something to support nanoseconds.
I have stored date in unix timestamp format in mongodb >> 1449060622
now I want to add milliseconds as well so if records inserted in same seconds can be sorted properly.
can someone suggest me that using js new Date() is more better or simply (new Date).getTime() ?
Whenever you store times in MongoDB you should really consider using the native Date type instead. Not only does it provide you with millisecond precision, it also unlocks a lot of features which are unavailable for simple integer, like date aggregation operators for example.
If you really don't want to use native dates for some obscure reason (I couldn't think of a good one) or don't want to convert your whole database (really, you should) and need a higher precision, you might consider to add new values as floating point values. This ensures interoperability with the old data because integers and floating point values usually can be converted and compared between each other easily.
I am just starting to learn about MongoDB and hoping to slowly migrate from MySQL.
In MySQL, there are two different data types - DATE ('0000-00-00') and DATETIME ('0000-00-00 00:00:00'). In my MySQL, I use the DATE type, but I am not sure how to transfer them into MongoDB. In MongoDB, there is a Date object, which is comparable to DATETIME. It seems it would be most appropriate to use Date objects, but that would be wasting space, since hours, min, sec are not utilized. On the other hand, storing dates as strings seems wrong.
Is there a golden standard on storing dates ('0000-00-00') in MongoDB?
I'm actually in the process of converting a MongoDB database where dates are stored as proper Date() types to instead store them as strings in the form yyyy-mm-dd. Why, considering that every other answerer says that this is a horrible idea? Simply put, because of the neverending pain I've been suffering trying to work with dates in JavaScript, which has no (real) concept of timezones. I had been storing UTC dates in MongoDB, i.e. a Date() object with my desired date and the time set as midnight UTC, but it's unexpectedly complicated and error-prone to get a user-submitted date correctly converted to that from whatever timezone they happen to be in. I've been struggling to get my JavaScript "whatever local timezone to UTC" code to work (and yes, I'm aware of Sugar.js and Moment.js) and I've decided that simple strings like the good old MySQL standard yyyy-mm-dd is the way to go, and I'll parse into Date() objects as needed at runtime on the client side.
Incidentally, I'm also trying to sync this MongoDB database with a FileMaker database, which also has no concept of timezones. For me the simplicity of simply not storing time data, especially when it's meaningless like UTC midnight, helps ensure less-buggy code even if I have to parse to and from the string dates now and then.
BSON (the storage data format used by mongo natively) has a dedicated date type UTC datetime which is a 64 bit (so, 8 byte) signed integer denoting milliseconds since Unix time epoch. There are very few valid reasons why you would use any other type for storing dates and timestamps.
If you're desperate to save a few bytes per date (again, with mongo's padding and minimum block size and everything this is only worth the trouble in very rare cases) you can store dates as a 3 byte binary blob by storing it as an unsigned integer in YYYYMMDD format, or a 2 byte binary blob denoting "days since January 1st of year X" where X must be chosen appropriately since that only supports a date range spanning 179 years.
EDIT: As the discussion below demonstrates this is only a viable approach in very rare circumstances. Basically; use mongo's native date type ;)
If you really care about saving 4 bytes per field (in case you have many DATE fields per document) you can store dates as int32 fields in form 20110720 (note MySQL DATE occupies 3 bytes, so the storage will be greater in any case). Otherwise I'd better stick to standard datetime type.
Taking into consideration performance and readability, how should dates be encoded in JSON for use in iPhone applications? The ideal encoding would work with NSDate easily, but also be future proofed for use in a web based back end (i.e. Not using an API exclusive to the iPhone, something crossplatform). Here are some encodings I have been considering:
# UNIX Epoc based integer encoding
{"Date":123456789}
# ISO 8601 String format
{"Date":"2011-03-25T20:00Z"}
Will the parsing of the ISO 8601 string impact performance too much when thousands of dates are processed? Is the performance hit too low to matter when compared to Epoc? Is there an encoding missing that accomplishes the mentioned requirements?
I've always used Unix epoch based encoding when I have control of the server stack. The pros outweigh the cons for my use cases
epoch pros
faster (may be an issue if dealing with large volume of dates.)
needs less code
supported by just about every date/time library/language
epoch cons
not human readable
bogus input (as long as it's a number) will likely still give you a valid date
ISO pros
very readable
bogus input (malformed) will likely not give you a valid date
ISO cons
takes more bytes
need to write/find your own Cocoa implementation (let me know if you can't find any, I have a category lying around somewhere)
ISO parsing is not hard, but you must check yourself on your own implementation. Cocoa doesn't have the fastest string parser. I have to believe that it will be noticeably slower (likely more than 2x).