I have some data matrix, the first column is date and the second column is time.
I changed it into a table:
H1 = readtable(filename, 'ReadVariableNames', 0);
H1.Properties.VariableNames =...
{'Date', 'Time', 'Value'};
t = datetime(H1.Date, 'InputFormat', 'yyyy.MM.dd');
H1.Date = t;
clear t
Now, I want to make a sub-table. In this table, the date of the data doesn't matter, but the time of the data must be between 18:00 and 11:00 the next day.
My time format is hh:mm (24 hours).
How could I do it?
I think I could make another column. For each time string, I take only the first 2 characters and convert it to a number (e.g. '05:00' will be changed to '05' and then 5). After that I could index the new column I created.
Nevertheless, I think there must be some other and easier ways to do it.
How?
In addition, if this works, I want to write a function based on this.
I want to give a certain day and time, and plot all the data +-48 hours (or -24 hours and +72 hours) around this day and time.
For example, if I give '2016-06-22', '05:00', then I want to plot all the data ranging from '2016-06-20', '05:00' to '2016-06-24', '05:00'. Any strategy suggestion?
Given that my data size is quite big, speed is a factor that we need to take into consideration.
Thanks!
Related
I am working with a data set of 10,000s of variables which have been repeatedly measured since the 1980s. The first meassurements for each variable are not on the same date and the variables are irregularly measured - sometimes measurements are only a month apart, in a small number of cases they are decades apart.
I want to get the change in each variable per month.
So far I have a cell of dates of measurements,and interpolated rates of change between measurements (each cell represents a single variable in either, and I've only posted the first 5 cells in each array)
DateNumss= {[736614;736641;736669] [736636;736666] 736672 [736631;736659;736685] 736686}
LinearInterpss={[17.7777777777778;20.7142857142857;0] [0.200000000000000;0] 0 [2.57142857142857;2.80769230769231;0]}
How do I get monthly sums of the interpolated change in variable?
i.e.
If the first measurement for a variable is made on the January 1st, and the linearly interpolated change between that an the next measurement is 1 per day; and the next measurement is on Febuary the 5th and the corresponding linearly interpolated change is 2; then January has a total change of 1*31 (31 days at 1) and febuary has a total change of 1*5+2*23 (5 days at 1, 23 days at 2).
You would need the points in the serial dates that correspond with the change of a month.
mat(:,1)=sort(repmat(1980:1989,[1,12]));
mat(:,2)=repmat(1:12,[1,size(mat,1)/12]);
mat(:,3)=1;
monthseps=datenum(mat);
This gives you a list of all 120 changes of months in the eighties.
Now you want, for each month, the change per day, and sum it. If you take the original data it is easier, since you can just interpolate each day's value using matlab. If you only have the "LinearInterpss" you need to map it on the days using interp1 with the method 'previous'.
for ct = 2:length(monthseps)
days = monthseps(ct-1):(monthseps(ct)-1); %days in the month
%now we need each day assigned a certain change. This value depends on your "LinearInterpss". interp1 with method 'previous' searches LineairInterpss for the last value.
vals = interp1(DateNumss,LinearInterpss,days,'previous');
sum(vals); %the sum over the change in each day is the total change in a month
end
This question already has answers here:
Counting values by day/hour with timeseries in MATLAB
(3 answers)
Closed 6 years ago.
I have a year's worth of data, the data is recorded one minute intervals each day of the year.
The date and time was imported from excel (in form 243.981944, then by adding 42004 (so will be for 2015) and formatting to date it becomes 31.8.15 23:34:00).
Importing to MATLAB it becomes
'31/08/2015 23:34:00'
I require the data for each day of the year to be at hourly intervals, so I need to sum the data recorded in each hour and divide that by the number of data recorded for that hour, giving me the hourly average.
For some reason the data in August actually increments in 2 minute intervals, data for every other month increments in one minute intervals.
ie
...
31/07/2015 23:57:00
31/07/2015 23:58:00
31/07/2015 23:59:00
31/08/2015 00:00:00
31/08/2015 00:02:00
31/08/2015 00:04:00
...
I'm not sure how I can find all the values for a specific date and hour in order to work out the averages. I was thinking of using a for loop to find the values on each day, but when I got down to writing code realised this wouldn't work the way I was thinking.
I presume there must be some kind of functions available that would allow for data to be filtered by the date and time?
edit:
So I tried the following but I get these errors.
dates is a 520000x1 cell array containing the dates form = formatIn.
formatIn = 'DD/MM/YYYY HH:MM:SS';
[~,M,D,H] = datevec(dates, formatIn);
Error using cnv2icudf (line 131) Unrecognized minute format.
Format string: DD/MM/YYYY HH:MM:SS.
Error in datevec (line 112) icu_dtformat = cnv2icudf(varargin{isdateformat});`
Assuming your data is in a matrix or cell-array of strings called A, and your other data is in a vector X. Let's say all the data is in the same year (so we can ignore years)
[~,M,D,H] = datevec(A, 'dd/mm/yyyy HH:MM:SS');
mean_A = accumarray([M, D, H+1], X, [], #mean);
Then data from February will be in
mean_A(2,:,:)
To look at the data, you may find the squeeze() function useful, e.g.
squeeze(mean_A(2,1:10,13:24))
shows the average for the hours after midday (by column) for the first ten days (by row) of February.
See also:
Counting values by day/hour with timeseries in MATLAB
I am writing a for loop to average 10 years of hourly measurements made on the hour. The dates of the measurements are recorded as MATLAB datenums.
I am trying to iterate through using 0.0417 as it is the datenum for 1AM 00/00/00 but it is adding in a couple of seconds of error each time I iterate.
Can anyone recommend a better way for me to iterate by hour?
date = a(:,1);
load = a(:,7);
%loop for each hour of the year
for i=0:0.0417:366
%set condition
%condition removes year from current date
c = date(:)-datenum(year(date(:)),0,0)==i;
%evaluate condition on load vector and find mean
X(i,2)=mean(load(c==1));
end
An hour has a duration of 1/24 day, not 0.0417. Use 1/24 and the precision is sufficient high for a year.
For an even higher precision, use something like datenum(y,1,1,1:24*365,0,0) to generate all timestamps.
To avoid error drift entirely, specify the index using integers, and divide the result down inside the loop:
for hour_index=1:365*24
hour_datenum = (hour_index - 1) / 24;
end
I created a diagram presenting limb movement over the time (2.56 seconds). My diagram looks like in the top Picture 1.
My code to get the diagram was:
x=data(1000:1256,2)
Fs=100
Ts=1/Fs
L=length(x)
t = (0:L-1)*Ts;
figure
plot(t,x);
Now Im trying to change time units into real time data (day and time, when the measurements were recorded), I want to get something like in the Picture 2: mark x-axis with date and time. Ideally Id like to have 6 time marks.
Using:
datestr(data(1000),'dd-mm-yyyy HH:MM:SS AM')
I know the first time (row 1000th of my data) is 10-07-2010 11:31:50 PM and the last row (1256th) is 10-07-2010 11:43:42 PM. There was always 50 records recorded per 1 second, BUT the problem is, the measurements were not recorded constantly - I mean, sometimes there was no measurements for a few minutes (when there was no movement).
I ve been trying to use XTick etc but I dont know how to select the real-time data for my x-axis and how to label x-axis with the real time of measurements.
labels=datestr(data);
set(gca,'XTick',1:6; 'XTickLabel',labels);
Anybody could help me?
This should format the tickmarks
datetick('x', 'dd-mm-yyyy HH:MM:SS AM')
It might go after setting the tickmarks
set(gca,'XTick',[ ... ])
Where in the array you should put the datenum values of your times.
Consider this minimal example:
x = linspace(now-2, now, 30);
y = rand(30, 1);
plot(x, y)
datetick('x', 'dd-mmm HH:MM')
I have a data file which contains time data. The list is quite long, 100,000+ points. There is data every 0.1 seconds, and the time stamps are so:
'2010-10-10 12:34:56'
'2010-10-10 12:34:56.1'
'2010-10-10 12:34:56.2'
'2010-10-10 12:34:53.3'
etc.
Not every 0.1 second interval is necessarily present. I need to check whether a 0.1 second interval is missing, then insert this missing time into the date vector. Comparing strings seems unnecessarily complicated. I tried comparing seconds since midnight:
date_nums=datevec(time_stamps);
secs_since_midnight=date_nums(:,4)*3600+date_nums(:,5)*60+date_nums(:,6);
comparison_secs=linspace(0,86400,864000);
res=(ismember(comparison_secs,secs_since_midnight)~=1);
However this approach doesn't work due to rounding errors. Both the seconds since midnight and the linspace of the seconds to compare it to never quite equal up (due to the tenth of a second resolution?). The intent is to later do an fft on the data associated with the time stamps, so I want as much uniform data as possible (the data associated with the missing intervals will be interpolated). I've considered blocking it into smaller chunks of time and just checking the small chunks one at a time, but I don't know if that's the best way to go about it. Thanks!
Multiply your numbers-of-seconds by 10 and round to the nearest integer before comparing against your range.
There may be more efficient ways to do this than ismember. (I don't know offhand how clever the implementation of ismember is, but if it's The Simplest Thing That Could Possibly Work then you'll be taking O(N^2) time that way.) For instance, you could use the timestamps that are actually present (as integer numbers of 0.1-second intervals) as indices into an array.
Since you're concerned with missing data records and not other timing issues such as a drifting time channel, you could check for missing records by converting the time values to seconds, doing a DIFF and finding those first differences that are greater than some tolerance. This would tell you the indices where the missing records should go. It's then up to you to do something about this. Remember, if you're going to use this list of indices to fill the gaps, process the list in descending index order since inserting the records will cause the index list to be unsynchronized with the data.
>> time_stamps = now:.1/86400:now+1; % Generate test data.
>> time_stamps(randi(length(time_stamps), 10, 1)) = []; % Remove 10 random records.
>> t = datenum(time_stamps); % Convert to date numbers.
>> t = 86400 * t; % Convert to seconds.
>> index = find(diff(t) > 1.999 * 0.1)' + 1 % Find missing records.
index =
30855
147905
338883
566331
566557
586423
642062
654682
733641
806963