While this isn't a programming question per se, it IS related.
So I'm trying to figure out how to parse the SMS DB that gets backed up from the iPhone. I'm looking at the "messages" table, specifically the "date" field. I noticed that the more recent messages are using a different numbering system to indicate the date/time. I've narrowed it down to the switch to iMessage, as I have a message sent at 1318470904, with a reply sent at 340164736. I know for a fact that these messages were sent less than an hour apart, yet they're indicating > 30 years' difference.
Anybody know how to accurately calculate the date using this newer system? Is it using a different epoch or is there some crazy math I need to do?
Edit: Recent messages are affected as well. Texts (green bubbles) are stored with the date set normally, and anything through iMessage (blue bubbles) is stored with the different date representation.
Since the backup is exported to SQLite database format, here's how to convert the number to a real date in SQLite:
select
datetime(date + strftime('%s', '2001-01-01 00:00:00'),
'unixepoch', 'localtime') as date,
*
from message
I don't know about getting the correct date given two versions present, but when I did this today, I noticed the date column was not the standard unix time but a longer number with seemingly nine zeros at the end, like 444548608000000000. This is what I did to get the correct date:
select
datetime(substr(date, 1, 9) + 978307200, 'unixepoch', 'localtime') as f_date,
text
from message
It is in seconds since 1/1/2001 instead of the others which are Unix based off of 1/1/1970. So to convert it to say an Excel time your formula would be =Cell/(60*60*24) + "1/1/2001".
Apple uses Mac Absolute Time (MacTime). This is counted from 01-01-2001. The other timestamp you see is UnixTime. This starts from 01-01-1970.
You have to add 31 years to MacTime to get UnixTime. This is a PHP-snippit:
$macTime = $d['ZMESSAGEDATE']; // MacTime column (from Whatsapp)
$unixTime = $macTime + 978307200;
echo date("Y-m-d H:i:s", $unixTime);
The time difference is calculated using this website:
https://www.timeanddate.com/date/durationresult.html?d1=1&m1=1&y1=1970&d2=1&m2=1&y2=2001&h1=0&i1=0&s1=0&h2=0&i2=0&s2=0
There may be another answer.
=Cell/(60*60*24) + "1/1/1970"
works with my current version of the iPhone/iOS => 4.3.3
Fomurla with time of messages:
=Cell/(60*60*24) + "1/1/2001 7:00"
Since date in mac is calculated from 2001 and not 1970, we have to add some extra to this mac date.
978307200000 is equivalent to milliseconds until 2001-01-01
Also multiplying by 1000 is required to convert to milli-seconds.
macDate * 1000 + 978307200000
Bohemian♦ is right, but there's a little typo in his answer:
use %S (capitals) instead of %s, since the time is represented in seconds since 2001 and not 1970!
Doc from https://www.sqlite.org/lang_datefunc.html
%s seconds since 1970-01-01
%S seconds: 00-59
select
datetime(date + strftime('%S', '2001-01-01 00:00:00'),
'unixepoch', 'localtime') as date,
*
from message
Related
I have been working on a readmission issue using a clinical graph dataset. Let's say a patient is being readmitted within 30 days. So, this means I need add 30 days to the first date (visit date) for the second visit date.
Here is the Cypher query:
MATCH(p:Person)-[r:PATIENT_HAS]->(e:Encounter)
WITH p,e
MATCH (p)-[r:PATIENT_HAS]-(e2:Encounter) WHERE e2.ADMIT_DATE < (e.ADMIT_DATE + 30)
This query won't work because the date property is in YYYYMMDD format. For example, if it is 20151225, it will give 20151255. But I need to get it as 20160124 after adding 30 days. Is there any other way to use a different format than YYYYMMDD. I know that there is string format as YYYY-MM-DD, but what is the way to use this format for adding days?
How can accomplish this?
I would appreciate your help.
The best way to deal with dates in Neo4j in my opinion is to have them saved as UNIX epoch time in miliseconds or seconds. We have a great plugin in Neo4j called apoc procedures that allows you to use awesome procedures. In your specific case I would utilize apoc.date.*procedures and parse your date format to epoch time in seconds.
MATCH (e:Encounter)
WITH e,apoc.date.parse(e.ADMIT_DATE,"s","YYYYMMDD") as unix
set e.unix = unix
So now your query would look like :
MATCH(p:Person)-[r:PATIENT_HAS]->(e:Encounter)
MATCH (p)-[r:PATIENT_HAS]-(e2:Encounter)
WHERE e2.unix < e.unix + (30 * 24 * 60 * 60)
Ofcourse you can simplify this query and make it shorter:
MATCH (e2:Encounter)-[r:PATIENT_HAS]-(p:Person)-[r:PATIENT_HAS]->(e:Encounter)
WHERE e2.unix < e.unix + (30 * 24 * 60 * 60)
You can construct a Duration, and add it directly to a date.
Here is an example in Cypher:
RETURN Date() + Duration({days: 30})
Reference: https://neo4j.com/docs/cypher-manual/current/functions/temporal/duration/
I'm currently using Neo4j 4.0.3 and the following works:
RETURN REPLACE(toString(date('20151225') + duration('P30D')),'-','')
// "20160124"
We found out a temporary solution: We are setting a specific date in the past (1/1/2009 in our case), and are getting the difference between the first visit (e.ADMIT_DATE-1/1/2009) and the second visit (e2.ADMIT_DATE-1/1/2009) as days. Then we are able to get the difference between the first and second visits (e2.DAYS-e.DAYS) to figure out 30 days for the readmission criteria.
I had checked out the apoc procedures plugin and it seemed a bit complicated. But I can definitely retry it. Many thanks for your great responses folks.
Is there a standard way to represent a date as a single integer? (I only need to store dates, not full timestamps.)
If it matters, the reason I want to do this is because I'm storing the dates in a SQLite database on Android and would like to store them as numbers so they can be compared/sorted efficiently to return results from queries.
One good option might be YYYYMMDD, for example encoding today (Jan 13, 2014) as the integer 20140113.
Advantages:
It works for comparisons, as long as you only care about <, ==, and >;
It's reasonably human-readable;
It's compatible with the ISO 8601 standard.
Disadvantages:
It's not as easy to compute differences between dates;
SQLite won't recognize it as a date.
On the last point: The SQLite3 documentation says that SQLite3 has no specific storage types for dates and/or times. Instead, it recommends using one of:
TEXT as ISO8601 strings ("YYYY-MM-DD HH:MM:SS.SSS").
REAL as Julian day numbers, the number of days since noon in Greenwich on November 24, 4714 B.C. according to the proleptic Gregorian calendar.
INTEGER as Unix Time, the number of seconds since 1970-01-01 00:00:00 UTC.
all of which apparently can be processed using SQLite's built-in date and time functions.
The latter argues in favor of the solution in bdf's answer. Picking an arbitrary time within the specified day is admittedly problematic, but I suggest picking noon UTC is unlikely to cause too many problems as long as you're careful to use it consistently. (Noon UTC can be on a different day if your time zone offset is 12 hours or more, but that's not an issue for most of the world.)
just set the time for every day to an arbitrary time of your choosing, such as 2 am.
Storing them as timestamps anyway might still be a good idea, since you'd have a lot more date formatting options.
An important detail to keep in mind are time zones. If e.g. your time falls in the gap between your time zone offset and GMT you might get unexpected results. So at first I propose we discuss date as the one visible to the user which is usually the one of the local time zone.
So if we assume local time, and we want to make use of the Date objects, there are 2 possible solutions, which I will present as JavaScript unit test style. First one is the one presented by Keith Thompson previously:
let date = new Date('1987-12-31T01:02:03')
let simpleDateInteger = (
date.getFullYear() * 10000 +
(date.getMonth() + 1) * 100 +
date.getDate()
)
expect(simpleDateInteger).toBe(19871231)
let fromSimpleDateInteger = new Date(
simpleDateInteger / 10000, // year
simpleDateInteger / 100 % 100 - 1, // month
simpleDateInteger % 100 // day
)
expect(fromSimpleDateInteger.toDateString()).toEqual(date.toDateString())
If you need more compact integers and each integer +1 representing the next day, i.e. a continuous representation you can go with this one:
let date = new Date('1987-12-31T00:01:02')
const DAY_IN_MILLISECONDS = 86400 * 1000
let timeZoneInMilliSeconds = date.getTimezoneOffset() * 60 * 1000
let continuousDateInteger = Math.floor(
(date.getTime() - timeZoneInMilliSeconds) / DAY_IN_MILLISECONDS
)
expect(continuousDateInteger).toBe(6573)
let fromContinuousDateInteger = new Date(
continuousDateInteger * DAY_IN_MILLISECONDS + timeZoneInMilliSeconds
)
expect(fromContinuousDateInteger.toDateString()).toEqual(date.toDateString())
I have some FileMaker timestamp which I don't know how to handle. (I discovered it by trial...)
Does someone know an algorithm to convert FM (File Maker) timestamp into DateTime?
I have read about the format on this page. Which includes a "FM dec Timestamp" button which makes the desired conversion, but gives no reference on how it does so!
Also, my timestamps differs in format from the one required in the site, mine has a size of 18 digits, whearas the site only allows 11.
Inserting 634890864000000000 and removing the trailing zeroes (to leave 11 digits), I got this date:
Wednesday, 2012-11-21 10:20:00
If you have FileMaker this should be as simple as:
Importing the number as text,
Making a new calculation field, resultingTimestamp, which takes the left 11 characters and converts to a TimeStamp:
GetAsTimestamp( Left( myImportedTimestamp ; 11 ) )
Doing conversion to Unix format, either programmatically or through display on the resultingTimestamp field on a Layout.
If you don't have FileMaker:
Take the left 11 digits of the FileMaker timestamp.
Subtract 62135596800 from the FileMaker timestamp to get the Unix (epoch) timestamp.
(Verified by taking the same date in each and subtracting the FileMaker date from the Unix date.)
Convert epoch time to human readable, for example according to one of the formulas found in the "Convert from epoch to human readable date" section of epochconverter.com.
To get your date:
create a calculation field with the following calculation:
TimeStamp/864000000000+1
set the return type to Date.
Also, I think the extra zeroes are fractions of a second, regardless the given formula deals with these.
I am in the process of optimizing some UniVerse data access code we have which uses UniObjects. After some experimentation, it seems that using a UniSession.OConv call to parse certain things such as decimal numbers (most we have a MR4 or MR2 or MR2$) and dates (almost all are D2/) is extremely slow (I think it might make a call back to the server to parse it).
I have already built a parser for the MR*[$] codes, but I was wondering about the dates as they are stored so I can build one for D2/. Usually they seem to be stored as a 5 digit number. I thought it could be number of days since the Unix Epoch since our UniVerse server runs on HP-UX, but after finding '15766' as a last modified date and multiplying it by 86400 (seconds per day), I got March 02, 2013 which doesn't make sense as a last modified date since as far as I know that is still in the future.
Does anyone know what the time base of these date numbers are?
It is stored as a number of days. Just do a conversion on 0 and you will get the start date.
Edit:
As noted by Los, the Epoch used in UniVerse (and UniData) is 31st Dec 1967.
In Universe and any other Pick database, Dates and Times are stored as separate values.
The internal date is the number of days before of after 31/12/1967, which is day zero.
The internal time is the number of seconds after midnight. It can be stored as a decimal but is not normally.
In TCL there is a CDT command (stands for Convert Date) that converts dates from human readable to numeric and and vice versa:
CDT 9/28/2017
* Result: 18169
CDT 18169
* Result: 09/28/2017
I have the following from a Microsoft SQL Server database for date/time value:
0x00009CEF00A25634
I found this post:
Help me translate long value, expressed in hex, back in to a date/time
Which seemed to be on the right track but by using the code I didn't get the right dates, are my hex dates in a different format? How would I convert them to a normal date, I am using PHP/PostgreSQL.
select CAST (0x00009CEF00A25634 as datetime) gives 2009-12-30 09:51:03.000
This is two integers. One for the date part 0x00009CEF (decimal 40175) and one for the time part 00A25634 (decimal 10638900). The date part is a signed integer giving number of days since 1 Jan 1900. The time part is an integer representing number of ticks.
There are 300 ticks in a second.
It can be seen that the following also returns the same result
SELECT DATEADD(MILLISECOND,10638900*10/3.0, DATEADD(DAY,40175, '19000101'))
You will need to figure out how to apply this to postgres.
Edit: an answer here apparently does this. I haven't tested it myself.
This works for me while migrating from SQL to MySQL :
SELECT (CAST('1900-01-01 00:00:00' + INTERVAL CAST(CONV(substr(HEX( 0x0000A249004576D0 ),1,8), 16, 10) AS SIGNED) DAY + INTERVAL CAST(CONV(substr(HEX( 0x0000A249004576D0 ),9,8), 16, 10) AS SIGNED)* 10000/3 MICROSECOND AS DATETIME)) AS newdate