How to use DateTime or FileTime date types in Matlab - matlab

I would like to parse/convert DateTime (or FileTime) data type which is int64 and measured in "ticks" which are 100-nanoseconds units, to a datenum or a better date & time format in Matlab.
I must use a counter with a high resolution 64bit as the difference between time stamps can be a couple of microseconds and the "spacing" isn't even.
Using latest Matlab version, the "ticks" variable is read as a string..
Any recommendations?
Thank you for your help

Datenums probably won't represent this effectively. They are doubles, an approximate type, with 1.0 = 1 day; the resolution limit for recent dates is about 10 microseconds. Also, the datenum related functions aren't written for high precision processing, and probably won't work well here.
>> datenum_precision = eps(now)
datenum_precision =
1.1642e-010
>> millisecond = 1.0/(24*60*60*1000)
millisecond =
1.1574e-008
>>
You're probably better off creating your own class that wraps the "ticks" as uint64 and provides methods for converting to human-readable presentation (like datestr()), doing addition and subtraction, parsing the strings you're getting to the uint64 values, and so on. Something with an interface like this.
classdef filetime
properties
tick; % An array of uint64 ticks
end
methods (Static = true)
out = parse(str); % Parse strings to #filetime
end
methods
out = filetimestr(obj); % human-readable strings
out = minus(a,b);
out = plus(a,b);
out = sort(obj);
out = datenum(obj); % maybe a lossy conversion to datenum, for convenience
end
end
You'll need R2010b to do arithmetic on 64-bit ints; older versions don't support it.
The tricky part would be if you wanted to do plots using the filetimetick objects; the Handle Graphics plots don't support user-defined objects.

You can convert your numbers to MATLAB's serial date with a simple equation:
step = 1e-7; % 100 nanosec
ticks = int64(250000);
timenum = double(ticks)*step/24/60/60;
First, for very large integer values converting to double might loose precision. If this is the case you probably better to stay with your numbers without converting to serial date.
Second, DATESTR/DATETICK functions support precision up to milliseconds. Try:
datestr(timenum,'HH:MM:SS.FFF')
So, think about it and please answer Andrew's comment - what do you need this conversion for?

D is a uint64 value of 100 ns since 01-01-1601, the result must be a datenum. This means days passed since 00-Jan-0000.
step = 100E-9; % 100 nanoseconds
step = step/(24*60*60); %fraction of a day
D = datenum('01-01-1601')+double(D)*step;
%D is now a datenum
Modern version using datetime (matlab >2014b)
%uint64 value of 100 ns since 01-01-1601
step = 100E-9; % 100 nanoseconds
step = step/(24*60*60); %fraction of a day
D = datetime('01-01-1601')+days(double(D)*step);
As mentioned above, both methods lose some of the accuracy that we had. In the case of datetimes this can probably be circumvented. It happens at double(D)*step because D is big for current dates. Many nanoseconds have passed since 1601. By converting D to a more recent offset one could get a more accurate conversion.

Related

Are certain MATLAB functions only precise to a certain decimal? How precise is MATLAB really?

I am converting a program from MATLAB 2012 to 2016. I've been getting some strange errors, which I believe some of are due to a lack of precision in MATLAB functions.
For instance, I have a timeseries oldTs as such:
Time Data
-----------------------------
1.00000000000000001 1.277032377439511
1.00000000000000002 1.277032378456123
1.00000000000000003 1.277032380112478
I have another timeseries newTs with similar data, but many more rows. oldTs may have half a million rows, whereas newTs could have a million. I want to interpolate the data from the old timeseries with the new timeseries, for example:
interpolatedTs = interp(oldTs.time, oldTs.data, newTs.time)
This is giving me an error: x values must be distinct
The thing is, my x values are distinct. I think that MATLAB may be truncating some of the data, and therefore believing that some of the data is not unique. I found that other MATLAB functions do this:
test = [1.00000000000000001, 1.00000000000000002, 1.0000000000000000003]
unique(test)
ans =
1
test2 = [10000000000000000001, 10000000000000000002, 10000000000000000003]
unique(test2)
ans =
1.000000000000000e+19
MATLAB thinks that this vector only has one unique value in it instead of three! This is a huge issue for me, as I need to maintain the highest level of accuracy and precision with my data, and I cannot sacrifice any of that precision. Speed/Storage is not a factor.
Do certain MATLAB functions, by default, truncate data at a certain nth decimal? Has this changed from MATLAB 2012 to MATLAB 2016? Is there a way to force MATLAB to use a certain precision for a program? Why does MATLAB do this to begin with?
Any light shed on this topic is much appreciated. Thanks.
No, this has not changed since 2012, nor since the very first version of MATLAB. MATLAB uses, and has always used, double precision floating point values by default (8 bytes). The first value larger than 1 that can be represented is 1 + eps(1), with eps(1) = 2.2204e-16. Basically you have less than 16 decimal digits to play with. Your value 1.00000000000000001 is identical to 1 in double precision floating point representation.
Note that this is not something specific to MATLAB, it is a standard that your hardware conforms to. MATLAB simply uses your hardware's capabilities.
Use the variable precision arithmetic from the Symbolic Math Toolbox to work with higher precision numbers:
data = [vpa(1) + 0.00000000000000001
vpa(1) + 0.00000000000000002
vpa(1) + 0.00000000000000003]
data =
1.00000000000000001
1.00000000000000002
1.00000000000000003
Note that vpa(1.00000000000000001) will not work, as the number is first interpreted as a double-precision float value, and only after converted to VPA, but the damage has already been done at that point.
Note also that arithmetic with VPA is a lot slower, and some operations might not be possible at all.

Is there an easy way to read integer time data from a netcdf file using the time-attribute?

I am trying to read time-coordinate data from a netCDF file using matlab. I have a netCDF file (which I created) that has a time variable in the format of a double corresponding to the number of hours from a specific time (see below).
Variable attributes:
double time(Time) ;
time:standard_name = "Time" ;
time:units = "hours since 2002-01-01 0:0:0" ;
time:calendar = "proleptic_gregorian" ;
When I read the time variable using ncread) into matlab, it just prints out an integer e.g.,1. However, if I use "ncdump" to explore the file, I see the time variable in it's coordinate time e.g., 2002-01-01 01.
Specifically: "ncdump -t -v time ncfile.nc"
I'm relatively new to matlab, and I was wondering if anyone knew if there was a similar, or an equally simple, way to read this time variable as its coordinate time into matlab, either as a string, or numerical date. Specifically, I would like to avoid having to parse the attribute string and code up a bunch of pointers and conditions to convert the integer data to an actual date.
Alternatively, should I just create a new time variable in these files that is just an array of dates as strings?
Any information is very much appreciated!
Thanks!
NetCDF stores time as an offset from an epoch. From your variable attribute, your epoch is 2002-01-01 0:0:0, and the time is hours since then. Matlab has a similar methodology called date numbers, although it is based off of days since an epoch (which they call pivot years). There are two functions that you should look into: datenum and datestr. The first converts a string into a date number and the other converts a date number into a date string.
You can convert your time variable into a compatible Matlab date number by dividing by 24 and then use the datestr function to format it however you like. Here is a simple example:
>> time = [1;2;3;4];
>> datestr(time./24+datenum('2002-01-01 0:0:0'))
ans =
01-Jan-2002 01:00:00
01-Jan-2002 02:00:00
01-Jan-2002 03:00:00
01-Jan-2002 04:00:00
Look at the Matlab help files associated with the two functions and you can format the date output however you like.

How does Labview save cluster data in a binary file and how do I read it out in MATLAB?

I have a very large number of files that were saved in binary in Labview, where each column is a timestamp cluster followed by a vector of singles.
I read each data file into Matlab r2013a using
fid = fopen(filename);
data = fread(fid,[N M],'*single',0,'b');
fclose(fid);
where I pre-calculate the size of the input array N,M. Since I know what the data is supposed to look like, I have figured out that data(1:5,:) is where the timestamp data is hidden, but it looks like something like this for M = 1:
[0 -842938.0625 -1.19209289550781e-07 0 4.48415508583941e-42]
The first element is always 0, the second element decreases monotonically with a constant step size, the third seems to be bistable, flipping back and forth between two very small values, the fourth is always 0, and the fifth is also constant.
I'm assuming it has something to do with how Labview encodes dates, but my google-fu has not helped me figure that out.
To make this a more general question, then:
How does Labview encode a timestamp cluster when it saves to a binary file, and how can I read it out and translate it into a meaningful number in another programming language, such as Matlab?
EDIT:
For posterity, here is my final code (appended to the code above):
datedata = data(5:-1:1,:);
data(1:5,:) = [];
dms = typecast(reshape(datedata(2:3,:),[],1),'uint64');
dsecs = typecast(reshape(datedata(4:5,:),[],1), 'int64');
timestamp = datenum(1904,1,1) + (double(dsecs) + double(dms)*2^-64)/(3600*24);
In the code #Floris posted from Mathworks, they typecast straight to double, but when I tried that, I got garbage. In order to get the correct date, I had to first convert to integer and then to double. Since my bottleneck is in the fread line (0.3 seconds to read off of an external disk), the extra typecast step is miniscule in the grand scheme of things.
The extra column, 4.5e-42, converts to an integer value of 3200, the number of values in the subsequent vector of singles.
This is not a complete answer, but it should help (I don't have either Labview or Matlab available at home so I can't check this right now).
There is an article at http://www.mathworks.com/matlabcentral/newsreader/view_thread/292060 that describes a similar question. Couple of useful bits of information I extracted from that:
Time stamp is a double (not single)
Need to flip the order of bytes (little vs big endian) to make sense of things
There is a useful comment:
Note that the LabView time convention is miliseconds since Jan 1 1904.
Here is one approach (may contain errors but will point you in the
right direction),
The following code snippet is also given:
%% Read in date information
[ fid, msg ] = fopen(FileName, 'r') ;
NColumns = 60 ; % Number of data columns - probably different for your
dataset!
[a, count] = fread(fid, [ NColumns Inf], '*single') ; % Force data to
be read into Matlab workspace as singles
a = a' ; % Convert to data in columns not rows
% The last two columns of a are the timestamp
b = fliplr(a(:, end-1:end)) ; % Must swap the order of the columns
d = typecast(reshape(b',[],1), 'double') ; % Now we can can convert to
double
time_local = datenum(1904, 1, 1) + d/(24*3600) ; % Convert from
seconds to matlab time format
fclose(fid) ;
It looks believable to me. Let me know if it works - if not, I may be able to help debug in the morning...
A LabVIEW timestamp is a 128-bit type consisting of a signed 64-bit integer measuring the offset in seconds since the LabVIEW epoch (January 1, 1904 00:00:00.00 UTC), and an unsigned 64-bit integer measuring the fractional second. Source: ni.com.
The byte order of the file however may be platform dependent. For example the time 8:02:58.147 AM July 3 2013 EDT may be stored as:
0x 00000000CDF9C372 25AA100000000000 (big/network)
or as
0x 000000000010AA25 72C3F9CD00000000 (little)

Convert binary string greater than 52 bits to single in Matlab?

I'm trying to convert very long binary strings, often greater than 52 bits into numbers. I cannot have a fixed lookahead window because I am doing this to calculate a version of Lempel-Ziv complexity for neural data.
When I try to convert any long string, bin2dec throws and error that the binary string must be 52 bits or less.
Is there a way to get around this size limitation?
dec2bin throws that error because a single is not capable of storing that much precision. Your very question asks an impossibility. You have two choices: store the value in something other than a floating point value, or throw away some precision before you convert.
Or describe more completely what you're trying to accomplish.
EDITING:
Based on your additional information, I am even more certain that converting to floating point is not what you want to do. If you want to reduce the storage size to something more efficient, convert to a vector of bytes (uint8), which is as dense as you can get. Just split the binary string into N rows of 8 digits each, using reshape. This seems to be an accepted approach for biological data.
str = char((rand(1, 100)>0.5) + '0'); % test data
data = uint8(bin2dec(reshape(str(1:end-mod(end,8)), [], 8)));
In this code, I toss any bits that don't divide evenly into 8. Or, skip the uint8 step and just perform your processing on the resulting vector, where each double-precision float represents one 8-bit word from your sequence.
You could roll your own implementation:
len = 60;
string = [];
for i = 1:len
string = [string sprintf('%d', randi([0 1]))];
end
% error
% bin2dec(string);
% roll your own...
value = 0;
for i = length(string):-1:1
value = value + str2num(string(i))*2^(length(string)-i);
end
I'm just looping through the string and adding to some value. At the end, value will contain the decimal value of the string. Does this work for you?
Note: This solution is slow. You can speed it up a bit by preallocating the string, which I did on my own machine. Also, it's going to have issues if your number gets up to 1e6 digits. At that point, you need variable precision arithmetic to keep track of it. And adding that to the calculation really slowed things down. If I were you, I'd strongly consider compiling this from a .mex file if you need the functionality in MATLAB.
credits due to #aardvarkk, but here's a sped up version of his algorithm (+- 100x faster):
N=100;
strbin = char(randi(2,1,N)+'0'-1);
pows2 = 2.^(N-1:-1:0);
value=pows2*(strbin-'0')';
double's range goes only up to 1.79769e+308 which is 2^1024 give or take. From there on, value will be Inf or NaN. So you still need to find another way storing the resulting number.
A final pro on this algorithm: you can cache pows2 for a large number and then use a piece of it for any new strbin of length N:
Nmax = 1e8; % already 700MB for pows2, watch out!
pows2 = 2.^(Nmax-1:-1:0);
and then use
value = pows2(Nmax-N+1:end)*(strbin-'0')';
Solution to matlab's numeric upper bound
There's a tool on the File Exchange called vpi: http://www.mathworks.com/matlabcentral/fileexchange/22725
It allows you to use really big integers (2^5000? no prob). It's only slower (a lot) in calculating everything, I don't suggest using my method above with this. But hey, you can't have everything!
Download the package, addpath it and the following might work:
N=3000;
strbin = char(randi(2,1,N)+'0'-1);
binvals=strbin-'0';
val=0;
twopow=vpi(1);
for ii=1:N
val=val+twopow*binvals(N-ii+1);
twopow=twopow*2;
end

MATLAB - working with timestamps

How can I convert this kind of data 08:00:43.771 given as string into a number specifying the number of milliseconds since midnight corresponding to this time instance?
I generally use the Matlab datenum outputs for timestamping in Matlab. Datenums are the number of days since 0/0/0000, expressed as a double (double precision numbers are precise to about 14 usec for contemporary dates).
Using datenums.
currentDateTime1 = datenum('08:00:43.771'); %Assumes today
currentDateTime2 = datenum('6/8/1975 08:00:43.771'); %Using an explicit date
millisecondsSinceMidnight = mod(currentDateTime1 ,1) *24*60*60*1000; %Mod 1 removes any day component
millisecondsSinceMidnight = mod(currentDateTime2 ,1) *24*60*60*1000; %Then this is just a unit conversion
For unusual string formats, use the extended form of datenum, which can accept a string format specifier.
Use 1000*etime(datevec('08:00:43.771'),datevec('0:00')) to give the number of milliseconds since midnight. etime gives the number of seconds between two date vectors, datevec converts strings to date vectors (assuming Jan 1 this year if only a time is given).