What does a negative integer Date value mean in MongoDB? - mongodb

I've discovered an issue with some of data being stored in MongoDB. We have a field that stores a Date, and normally this includes values like ISODate("1992-08-30T00:00:00.000Z") or ISODate("1963-08-15T00:00:00.000Z"). That's nice and straight-forward; I can easily look at those dates and see August 30, 1992 or August 15, 1963.
However, I've noticed a couple of entries where the date looks something like this instead:
Date(-61712668800000)
I'm honestly not sure how the data got persisted that way in the first place, as it should have been stored the former way. And I'll have to address the software bug with my code that is intermittently causing it to be stored that way.
However, the bigger problem is what to do with data entries that look like that. I'm not even sure what date that was supposed to be. My first assumption is that it's just milliseconds, like a UNIX timestamp or something, but that's not right. Even if I flip the negative sign and remove some of the trailing zeros, that still ends up being a date way in the future (e.g. July 23, 2165), and that's not correct. It should be a date in the past.
And the other big problem is that I'm not sure how to even search for this in the database. I can't utilize a $type query because the type is still 9 (i.e. it still thinks it's a "Date").
Has anyone else encountered these weird date entries before? How can I find them? And how can I recover the actual date from them?

The problem seems to be that your code is storing dates prior to the epoch, which are furthermore so far into the past that they cannot be represented using an ISODate wrapper:
As per the documentation
(emphasis added)
Date
BSON Date is a 64-bit integer that represents the number of
milliseconds since the Unix epoch (Jan 1, 1970). This results in a
representable date range of about 290 million years into the past and
future.
The official BSON specification refers to the BSON Date type as the
UTC datetime.
Changed in version 2.0: BSON Date type is signed. [2] Negative values
represent dates before 1970.
Although not explicitly stated in the Mongo documentation, it appears that they are following a strict interpretation of the ISO 8601 standard and not one of the variants which are allowed "by trading partner agreement" based on what I found at wikipedia
Years
YYYY ±YYYYY ISO 8601 prescribes, as a minimum, a
four-digit year [YYYY] to avoid the year 2000 problem. It therefore
represents years from 0000 to 9999, year 0000 being equal to 1 BC and
all others AD. However, years prior to 1583 are not automatically
allowed by the standard. Instead "values in the range [0000] through
[1582] shall only be used by mutual agreement of the partners in
information interchange."[9]
To represent years before 0000 or after 9999, the standard also
permits the expansion of the year representation but only by prior
agreement between the sender and the receiver.[10] An expanded year
representation [±YYYYY] must have an agreed-upon number of extra year
digits beyond the four-digit minimum, and it must be prefixed with a +
or − sign[11] instead of the more common AD or BC (or the less widely
used BCE/CE) notation; by convention 1 BC is labelled +0000, 2 BC is
labeled -0001, and so on.[12]
If you read through the rest of the article you will also see that the reason the number of digits must be pre-defined is so that the date can be stored unambiguously without using separator characters such as "-" between the components.

Related

What is the perfect way of storing date in firestore?

I'm using flutter. In which way a DateTime should be stored in firestore so that I can make a filter based on the DateTime. Should it be converted to String or something else? What is the perfect way here?
Firestore has a native Timestamp type that can be used to store timestamps, which:
A Timestamp represents a point in time independent of any time zone or calendar, represented as seconds and fractions of seconds at nanosecond resolution in UTC Epoch time
This type is perfectly suited for filtering on date/time ranges, for example when you want to get all documents with a field between January 7, 2021 at 7:41:00AM and January 20, 2021 at 2:00:42PM.
In cases where I'm purely interested in a date and not in a time, I often store that as a string in "20210107" format (so yyyyMMdd). I find this format slightly easier when I want to get documents for a specific date, as I can use an equality filter. But the trade-off is that I love the precision of time. As you can probably guess from my phrasing this is a personal preference, not necessarily a best way for all cases or everyone.
It must be a string, you can split or format it as the day / month / year you want after the string value

Operating with datetimes in SQLite

I'm interested in knowing the different possibilities to operate with datetimes in SQLite and understand its pros and cons. I did not find anywhere a detailed explanation of all the alternatives.
So far I have learned that
SQLite doesn't actually have a native storage class for timestamps /
dates. It stores these values as NUMERIC or TEXT typed values
depending on input format. Date manipulation is done using the builtin
date/time functions, which know how to convert inputs from the other
formats.
(quoted from here)
When any operation between datetimes is needed, I have seen two different approaches:
julianday function
SELECT julianday(OneDatetime) - julianday(AnotherDatetime) FROM MyTable;
Number of days is returned, but this can be fractional.
Therefore, you can also get some other measures of time with some extra operations. For instance, to get minutes:
SELECT CAST ((
julianday(OneDatetime) - julianday(AnotherDatetime)
) * 24 * 60 AS INTEGER)
Apparently julianday could cause some problems:
Bear in mind that julianday returns the (fractional) number of 'days'
- i.e. 24hour periods, since noon UTC on the origin date. That's usually not what you need, unless you happen to live 12 hours west of
Greenwich. E.g. if you live in London, this morning is on the same
julianday as yesterday afternoon.
More information in this post.
strftime function
SELECT strftime("%s", OneDatetime)-strftime("%s", AnotherDatetime) FROM MyTable;
Number of seconds is returned. Similarly, you can also get some other measures of time with some extra operations. For instance, to get minutes:
SELECT (strftime("%s", OneDatetime)-strftime("%s", AnotherDatetime))/60 FROM MyTable;
More information here.
My conclusion so far is: julianday seems easier to use, but can cause some problems. strftime seems more verbose, but also safer. Both of them provide only as results a single unit (either days or hours or minutes or seconds), but not a combination of many.
Question
1) Is there any other possibility to operate with datetimes?
2) What would be the best way to get directly the difference of two datetimes in time format (or date or datetime), where datetime would be formatted as 'YYYY-mm-dd HH:MM:SS', and the result would be something in the same format?
I would have imagined that something like the following would work, but it does not:
SELECT DATETIME('2016-11-04 08:05:00') - DATETIME('2016-11-04 07:00:00') FROM MyTable;
> 01:05:00
Julian day numbers are perfectly safe when computing differences.
The only problem would be if you tried to convert them into a date by truncating any fractional digits; this would result in noon, not midnight. (The same could happen if you tried to store them in integer variables.) But that is not what you're doing here.
SQLite has no built-in function to compute date/time differences; you have to convert date/time values into some number first. Whether you use (Julian) days or seconds does not really matter from a technical point of view; use whatever is easier in your program.
If you started with a different format, you might want to convert the resulting difference back into that format, e.g.:
time(difference_value, 'unixepoch') -- from seconds to hh:mm:ss
time(0.5 + difference_value) -- from Julian days to hh:mm:ss

How to handle dates in neo4j

I'm an historian of medieval history and I'm trying to code networks between kings, dukes, popes etc. over a period of time of about 50 years (from 1220 to 1270) in medieval Germany. As I'm not a specialist for graph-databases I'm looking for a possibility to handle dates and date-ranges.
Are there any possibilities to handle over a date-range to an edge so that the edges, which represents a relationship, disappears after e.g. 3 years?
Are there any possibility to ask for relationships who have their date-tag in a date-range?
The common way to deal with dates in Neo4j is storing them either as a string representation or as millis since epoch (aka msec passed since Jan 01 1970).
The first approach makes the graph more easily readable the latter allows you to do math e.g. calculate deltas.
In your case I'd store two properties called validFrom and validTo on the relationships. You queries need to make sure you're looking for the correct time interval.
E.g. to find the king(s) in charge of France from Jan 01 1220 to Dec 31st 1221 you do:
MATCH (c:Country{name:'France'})-[r:HAS_KING]->(king)
WHERE r.validFrom >= -23667123600000 and r.validTo <=-23604051600000
RETURN king, r.validFrom, r.validTo
addendum
Since Neo4j 3.0 there's the APOC library which provides couple of functions for converting timestamps to/from human readable date strings.
You can also store the dates in their number representation in the following format: YYYYMMDD
In your case 12200101 would be Jan 1st 1220 and 12701231 would be Dec 31st 1270.
It's a useful and readable format and you can perform range searches like:
MATCH (h:HistoricEvent)
WHERE h.date >= 12200101 AND h.date < 12701231
RETURN h
It would also let you order by dates, if you need to.
As of Neo4J 3.4, the system handles duration and dates, see the official documentation. See more examples here.
An example related to the original question: Retrieve the historical events that happened in the last 30 days from now :
WITH duration({days: 30}) AS duration
MATCH (h:HistoricEvent)
WHERE date() - duration < date(h.date)
RETURN h
Another option for dates that keeps the number of nodes/properties you create fairly low is a linked list years (earliest year of interest - latest year), one of months (1-12), and one of dates in a month (1-31). Then every "event" in your graph can be connected to a year, month, and day. This way you don't have to create a new node for every new combination of a year month and day. You just have a single set of months, one of days, and one year. I scale the numbers to make manipulating them easier like so
Years are yyyy*10000
Months are mm*100
Date are dd
so if you run a query such as
match (event)-[:happened]->(t:time)
with event,sum(t.num) as date
return event.name,date
order by date
You will get a list of all events in chronological order with dates like Janurary 17th, 1904 appearing as 19040117 (yyyymmdd format)
Further, since these are linked lists where, for example,
...-(t0:time {num:19040000})-[:precedes]->(t1:time {num:19050000})-...
ordering is built into the nodes too.
This is, so far, how I have liked to do my event dating

Convert FM (FileMaker) timestamp to DateTime

I have some FileMaker timestamp which I don't know how to handle. (I discovered it by trial...)
Does someone know an algorithm to convert FM (File Maker) timestamp into DateTime?
I have read about the format on this page. Which includes a "FM dec Timestamp" button which makes the desired conversion, but gives no reference on how it does so!
Also, my timestamps differs in format from the one required in the site, mine has a size of 18 digits, whearas the site only allows 11.
Inserting 634890864000000000 and removing the trailing zeroes (to leave 11 digits), I got this date:
Wednesday, 2012-11-21 10:20:00
If you have FileMaker this should be as simple as:
Importing the number as text,
Making a new calculation field, resultingTimestamp, which takes the left 11 characters and converts to a TimeStamp:
GetAsTimestamp( Left( myImportedTimestamp ; 11 ) )
Doing conversion to Unix format, either programmatically or through display on the resultingTimestamp field on a Layout.
If you don't have FileMaker:
Take the left 11 digits of the FileMaker timestamp.
Subtract 62135596800 from the FileMaker timestamp to get the Unix (epoch) timestamp.
(Verified by taking the same date in each and subtracting the FileMaker date from the Unix date.)
Convert epoch time to human readable, for example according to one of the formulas found in the "Convert from epoch to human readable date" section of epochconverter.com.
To get your date:
create a calculation field with the following calculation:
TimeStamp/864000000000+1
set the return type to Date.
Also, I think the extra zeroes are fractions of a second, regardless the given formula deals with these.

UniVerse native date format

I am in the process of optimizing some UniVerse data access code we have which uses UniObjects. After some experimentation, it seems that using a UniSession.OConv call to parse certain things such as decimal numbers (most we have a MR4 or MR2 or MR2$) and dates (almost all are D2/) is extremely slow (I think it might make a call back to the server to parse it).
I have already built a parser for the MR*[$] codes, but I was wondering about the dates as they are stored so I can build one for D2/. Usually they seem to be stored as a 5 digit number. I thought it could be number of days since the Unix Epoch since our UniVerse server runs on HP-UX, but after finding '15766' as a last modified date and multiplying it by 86400 (seconds per day), I got March 02, 2013 which doesn't make sense as a last modified date since as far as I know that is still in the future.
Does anyone know what the time base of these date numbers are?
It is stored as a number of days. Just do a conversion on 0 and you will get the start date.
Edit:
As noted by Los, the Epoch used in UniVerse (and UniData) is 31st Dec 1967.
In Universe and any other Pick database, Dates and Times are stored as separate values.
The internal date is the number of days before of after 31/12/1967, which is day zero.
The internal time is the number of seconds after midnight. It can be stored as a decimal but is not normally.
In TCL there is a CDT command (stands for Convert Date) that converts dates from human readable to numeric and and vice versa:
CDT 9/28/2017
* Result: 18169
CDT 18169
* Result: 09/28/2017