I have a datetime array that highlights the peaks of a function "datepeak", for every day in one year. I obtained it using a datetime array "date" and the array with the position of the peaks "position".
t1 = datetime(year,1,1,0,0,0);
t2 = datetime(year,12,31,23,59,0);
date = t1:minutes(1):t2;
datepeak=date(position);
I need to take the n number of peaks for the day 1 and transpose this array to the first row of the matrix, and so on.
Since the number of peaks are not constants (min 3 max 4) I tried to initiate the matrix like this:
matrix=NaN(365,4)
Then I override the NaN of every row with this double for loop:
for i=1:365
v=datepeak(day(datepeak,'dayofyear')==i);
for c=1:length(v)
matrix(i,c)=(v(c));
end
end
This loop works (I tried it with the peaks), but with datetime I get an error.
Here's an example to paste:
year=2016;
position=[128 458 950];
t1 = datetime(year,1,1,0,0,0);
t2 = datetime(year,12,31,23,59,0);
date = t1:minutes(1):t2;
datepeak=date(position);
matrix=NaN(365,4);
for i=1:365
v=datepeak(day(datepeak,'dayofyear')==i);
for c=1:length(v)
matrix(i,c)=(v(c));
end
end
The nan array is of class double whereas datepeak is of class datetime so you can't store them in the same array. The way you represent your data should be driven by what you want to do with them later (and what is feasible). In your case, i'll assume that list 365 elements, containing the (any number) peak times of the day is ok.
year=2016;
position=[128 458 950];
t1 = datetime(year,1,1,0,0,0);
t2 = datetime(year,12,31,23,59,0);
date = t1:minutes(1):t2;
datepeak=date(position);
peaktimes_list = cell(365,1);
for i=1:365
peaktimes_list{i} = datepeak(day(datepeak,'dayofyear')==i);
end
EDIT : For a 365x4 cell array, change the last part by :
peaktimes = cell(365,4);
for i=1:365
v = datepeak(day(datepeak,'dayofyear')==i);
nv = numel(v);
peaktimes(i,1:nv) = num2cell(v);
end
When there are less than 4 values, the remaining columns will be empty.
Related
I have the following 606 x 274 table:
see here
Goal:
For every date calculate lower and upper 20% percentiles and, based on the outcome, create 2 new variables, e.g. 'L' for "lower" and 'U' for "upper", which contain the ticker names as seen in the header of the table.
Step by step:
% Replace NaNs with 'empty' for the percentile calculation (error: input to be cell array)
T(cellfun(#isnan,T)) = {[]}
% Change date format
T.Date=[datetime(T.Date, 'InputFormat', 'eee dd-MMM-yyyy')];
% Take row by row
for row=1:606
% If Value is in upper 20% percentile create new variable 'U' that contains the according ticker names.
% If Value is in lower 20% percentile create new variable 'L' that contains the according ticker names.
end;
So far, experimenting with 'prctile' only yielded a numeric outcome, for a single column. Example:
Y = prctile(T.A2AIM,20,2);
Thanks for your help and ideas!
Generally speaking, if you have an array of numbers:
a = [4 2 1 8 -2];
percentiles can be computed by first sorting the array and then attempting to access the index supplied in the percentile. So prctile(a,20)'s functionality could in principle be replaced by
b = sort(a);
ind = round(length(b)*20/100);
if ind==0
ind = 1;
end
b = b(ind);
% b = -2
However, prctile does a bit more of fancy magic interpolating the input vector to get a value that is less affected by array size. However, you can use the idea above to find the percentile splitting columns. If you chose to do it like I said above, what you want to do to get the headers that correspond to the 20% and 80% percentiles is to loop through the rows, remove the NaNs, get the indeces of the sort on the remaining values and get the particular index of the 20% or 80% percentile. Regrettably, I have an old version of Matlab that does not support tables so I couldn't verify if the header names are returned correctly, but the idea should be clear.
L = cell(size(T,1),1);
U = cell(size(T,1),1);
for row=1:size(T,1)
row_values = T{row,:};
row_values = row_values(2:end); % Remove date column
non_nan_indeces = find(~isnan(row_values));
if not(isempty(non_nan_indeces))
[row_values,sorted_indeces] = sort(row_values(non_nan_indeces));
% The +1 is because we removed the date column
L_ind = non_nan_indeces(sorted_indeces(1:round(0.2*length(row_values))))+1;
U_ind = non_nan_indeces(sorted_indeces(round(0.8*length(row_values)):end))+1;
% I am unsure about this part
L{row} = T.Properties.VariableNames(L_ind);
U{row} = T.Properties.VariableNames(U_ind);
else
L{row} = nan;
U{row} = nan;
end
end;
If you want to use matlab's prctile, you would have to find the returned value's index doing something like this:
L = cell(size(T,1),1);
U = cell(size(T,1),1);
for row=1:size(T,1)
row_values = T{row,:};
row_values = row_values(2:end); % Remove date column
non_nan_indeces = find(~isnan(row_values));
if not(isempty(non_nan_indeces))
[row_values,sorted_indeces] = sort(row_values(non_nan_indeces));
L_val = prctile(row_values(non_nan_indeces),20);
U_val = prctile(row_values(non_nan_indeces),80);
% The +1 is because we removed the date column
L_ind = non_nan_indeces(sorted_indeces(find(row_values<=L_val)))+1;
U_ind = non_nan_indeces(sorted_indeces(find(row_values>=U_val)))+1;
% I am unsure about this part
L{row} = T.Properties.VariableNames(L_ind);
U{row} = T.Properties.VariableNames(U_ind);
else
L{row} = nan;
U{row} = nan;
end
end;
I have a 1-by-1000 vector containing datenums for events, and I want to count total events per date (simplified for this example). The dimensions of dates and events agree.
My code is
i = 0
for d = unique(dates)
i = i + 1
result(i) = length(events(d == dates))
end
I get a dimension mismatch for d == dates. I understand why (d is a 1-by-1 vector), but how do I write this properly?
Bonus points: The solution with i is pretty ugly... hints?
Thanks!
edit by request:
dates contains datenums
729028
729028
729028
729028
729028
and events contains floats:
0.1205
0.2932
2.0384
2.0384
1.0411
0.5425
The problem is that unique(dates) is a column vector, and for steps through columns of whatever is on the right-hand side of the equal sign. Thus, d is a vector, not a scalar in your original code.
To get the code what you want to do:
for d = unique(dates)'
To avoid the loop:
d = hist(dates,unique(dates));
You only need to compute how many times each value of dates is repeated. You can do this with bsxfun:
uniqueDates = unique(dates);
count = sum(bsxfun(#eq, uniqueDates(:), dates(:).'),2);
Each entry of count corresponds to the same-index entry of uniqueDates.
Example: for dates = [729028; 729028; 729000; 729028; 729100] the result is
uniqueDates =
729000
729028
729100
count =
1
3
1
I have a matrix in MATLAB of 50572x4 doubles. The last column has datenum format dates, increasing values from 7.3025e+05 to 7.3139e+05. The question is:
How can I split this matrix into sub-matrices, each that cover intervals of 30 days?
If I'm not being clear enough… the difference between the first element in the 4th column and the last element in the 4th column is 7.3139e5 − 7.3025e5 = 1.1376e3, or 1137.6. I would like to partition this into 30 day segments, and get a bunch of matrices that have a range of 30 for the 4th columns. I'm not quite sure how to go about doing this...I'm quite new to MATLAB, but the dataset I'm working with has only this representation, necessitating such an action.
Note that a unit interval between datenum timestamps represents 1 day, so your data, in fact, covers a time period of 1137.6 days). The straightforward approach is to compare each timestamps with the edges in order to determine which 30-day interval it belongs to:
t = A(:, end) - min(A:, end); %// Normalize timestamps to start from 0
idx = sum(bsxfun(#lt, t, 30:30:max(t))); %// Starting indices of intervals
rows = diff([0, idx, numel(t)]); %// Number of rows in each interval
where A is your data matrix, where the last column is assumed to contain the timestamps. rows stores the number of rows of the corresponding 30-day intervals. Finally, you can employ cell arrays to split the original data matrix:
C = mat2cell(A, rows, size(A, 2)); %// Split matrix into intervals
C = C(~cellfun('isempty', C)); %// Remove empty matrices
Hope it helps!
Well, all you need is to find the edge times and the matrix indexes in between them. So, if your numbers are at datenum format, one unit is the same as one day, which means that we can jump from 30 and 30 units until we get as close as we can to the end, as follows:
startTime = originalMatrix(1,4);
endTime = originalMatrix(end,4);
edgeTimes = startTime:30:endTime;
% And then loop though the edges checking for samples that complete a cycle:
nEdges = numel(edgeTimes);
totalMeasures = size(originalMatrix,1);
subMatrixes = cell(1,nEdges);
prevEdgeIdx = 0;
for curEdgeIdx = 1:nEdges
nearIdx=getNearestIdx(originalMatrix(:,4),edgeTimes(curEdgeIdx));
if originalMatrix(nearIdx,4)>edgeTimes(curEdgeIdx)
nearIdx = nearIdx-1;
end
if nearIdx>0 && nearIdx<=totalMeasures
subMatrix{curEdgeIdx} = originalMatrix(prevEdgeIdx+1:curEdgeIdx,:);
prevEdgeIdx=curEdgeIdx;
else
error('For some reason the edge was not inbound.');
end
end
% Now we check for the remaining days after the edges which does not complete a 30 day cycle:
if curEdgeIdx<totalMeasures
subMatrix{end+1} = originalMatrix(curEdgeIdx+1:end,:);
end
The function getNearestIdx was discussed here and it gives you the nearest point from the input values without checking all possible points.
function vIdx = getNearestIdx(values,point)
if isempty(values) || ~numel(values)
vIdx = [];
return
end
vIdx = 1+round((point-values(1))*(numel(values)-1)...
/(values(end)-values(1)));
if vIdx < 1, vIdx = []; end
if vIdx > numel(values), vIdx = []; end
end
Note: This is pseudocode and may contain errors. Please try to adjust it into your problem.
so I have a matrix Data in this format:
Data = [Date Time Price]
Now what I want to do is plot the Price against the Time, but my data is very large and has lines where there are multiple Prices for the same Date/Time, e.g. 1st, 2nd lines
29 733575.459548611 40.0500000000000
29 733575.459548611 40.0600000000000
29 733575.459548612 40.1200000000000
29 733575.45954862 40.0500000000000
I want to take an average of the prices with the same Date/Time and get rid of any extra lines. My goal is to do linear intrapolation on the values which is why I must have only one Time to one Price value.
How can I do this? I did this (this reduces the matrix so that it only takes the first line for the lines with repeated date/times) but I don't know how to take the average
function [ C ] = test( DN )
[Qrows, cols] = size(DN);
C = DN(1,:);
for i = 1:(Qrows-1)
if DN(i,2) == DN(i+1,2)
%n = 1;
%while DN(i,2) == DN(i+n,2) && i+n<Qrows
% n = n + 1;
%end
% somehow take average;
else
C = [C;DN(i+1,:)];
end
end
[C,ia,ic] = unique(A,'rows') also returns index vectors ia and ic
such that C = A(ia,:) and A = C(ic,:)
If you use as input A only the columns you do not want to average over (here: date & time), ic with one value for every row where rows you want to combine have the same value.
Getting from there to the means you want is for MATLAB beginners probably more intuitive with a for loop: Use logical indexing, e.g. DN(ic==n,3) you get a vector of all values you want to average (where n is the index of the date-time-row it belongs to). This you need to do for all different date-time-combinations.
A more vector-oriented way would be to use accumarray, which leads to a solution of your problem in two lines:
[DateAndTime,~,idx] = unique(DN(:,1:2),'rows');
Price = accumarray(idx,DN(:,3),[],#mean);
I'm not quite sure how you want the result to look like, but [DataAndTime Price] gives you the three-row format of the input again.
Note that if your input contains something like:
1 0.1 23
1 0.2 47
1 0.1 42
1 0.1 23
then the result of applying unique(...,'rows') to the input before the above lines will give a different result for 1 0.1 than using the above directly, as the latter would calculate the mean of 23, 23 and 42, while in the former case one 23 would be eliminates as duplicate before and the differing row with 42 would have a greater weight in the average.
Try the following:
[Qrows, cols] = size(DN);
% C is your result matrix
C = DN;
% this will give you the indexes where DN(i,:)==DN(i+1)
i = find(diff(DN(:,2)==0);
% replace C(i,:) with the average
C(i,:) = (DN(i,:)+DN(i+1,:))/2;
% delete the C(i+1,:) rows
C(i,:) = [];
Hope this works.
This should work if the repeated time values come in pairs (the average is calculated between i and i+1). Should you have time repeats of 3 or more then try to rethink how to change these steps.
Something like this would work, but I did not run the code so I can't promise there's no bugs.
newX = unique(DN(:,2));
newY = zeros(1,length(newX));
for ix = 1:length(newX)
allOcurrences = find(DN(:,2)==DN(i,2));
% If there's duplicates, take their mean
if numel(allOcurrences)>1
newY(ix) = mean(DN(allOcurrences,3));
else
% If not, use the only Y value
newY(ix) = DN(ix,3);
end
end
I have 19 cells (19x1) with temperature data for an entire year where the first 18 cells represent 20 days (each) and the last cell represents 5 days, hence (18*20)+5 = 365days.
In each cell there should be 7200 measurements (apart from cell 19) where each measurement is taken every 4 minutes thus 360 measurements per day (360*20 = 7200).
The time vector for the measurements is only expressed as day number i.e. 1,2,3...and so on (thus no decimal day),
which is therefore displayed as 360 x 1's... and so on.
As the sensor failed during some days, some of the cells contain less than 7200 measurements, where one in
particular only contains 858 rows, which looks similar to the following example:
a=rand(858,3);
a(1:281,1)=1;
a(281:327,1)=2;
a(327:328,1)=5;
a(329:330,1)=9;
a(331:498,1)=19;
a(499:858,1)=20;
Where column 1 = day, column 2 and 3 are the data.
By knowing that each day number should be repeated 360 times is there a method for including an additional
amount of every value from 1:20 in order to make up the 360. For example, the first column requires
79 x 1's, 46 x 2's, 360 x 3's... and so on; where the final array should therefore have 7200 values in
order from 1 to 20.
If this is possible, in the rows where these values have been added, the second and third column should
changed to nan.
I realise that this is an unusual question, and that it is difficult to understand what is asked, but I hope I have been clear in expressing what i'm attempting to
acheive. Any advice would be much appreciated.
Here's one way to do it for a given element of the cell matrix:
full=zeros(7200,3)+NaN;
for i = 1:20 % for each day
starti = (i-1)*360; % find corresponding 360 indices into full array
full( starti + (1:360), 1 ) = i; % assign the day
idx = find(a(:,1)==i); % find any matching data in a for that day
full( starti + (1:length(idx)), 2:3 ) = a(idx,2:3); % copy matching data over
end
You could probably use arrayfun to make this slicker, and maybe (??) faster.
You could make this into a function and use cellfun to apply it to your cell.
PS - if you ask your question at the Matlab help forums you'll most definitely get a slicker & more efficient answer than this. Probably involving bsxfun or arrayfun or accumarray or something like that.
Update - to do this for each element in the cell array the only change is that instead of searching for i as the day number you calculate it based on how far allong the cell array you are. You'd do something like (untested):
for k = 1:length(cellarray)
for i = 1:length(cellarray{k})
starti = (i-1)*360; % ... as before
day = (k-1)*20 + i; % first cell is days 1-20, second is 21-40,...
full( starti + (1:360),1 ) = day; % <-- replace i with day
idx = find(a(:,1)==day); % <-- replace i with day
full( starti + (1:length(idx)), 2:3 ) = a(idx,2:3); % same as before
end
end
I am not sure I understood correctly what you want to do but this below works out how many measurements you are missing for each day and add at the bottom of your 'a' matrix additional lines so you do get the full 7200x3 matrix.
nbMissing = 7200-size(a,1);
a1 = nan(nbmissing,3)
l=0
for i = 1:20
nbMissing_i = 360-sum(a(:,1)=i);
a1(l+1:l+nbMissing_i,1)=i;
l = l+nb_Missing_i;
end
a_filled = [a;a1];