improve performance of a double for loop in matlab - matlab

Question

I'm doing some analysis where I'm analysing hundreds of data files, which are being analysed iteratively. Here is an examples of the sort of data that I have:
start_time = datenum('1990-01-01');
end_time = datenum('2009-12-31');
time = start_time:end_time;
datx = rand(length(time),1);
daty = datx-2;
where I have a time variable and two data variables.
After loading the data I then need to pass the data through a function. However, I need to do this by including firstly the data from year 1 only, then from years 1 to 2; 1 to 3, 1 to 4 and so on until I pass the data through the function for the entire series. This can be performed with a loop with the following:
% split into different years
datev = datevec(time);
iyear = datev(:,1);
unique_year = unique(iyear);
for k = 1:length(unique_year);
idx = find(iyear >= unique_year(1) & iyear <= unique_year(k));
% select data for year
d_time = time(idx);
d_datx = datx(idx);
d_daty = daty(idx);
% now select individual years from this subset
datev2 = datevec(d_time);
iyear2 = datev2(:,1);
unique_year2 = unique(iyear2);
for k2 = 1:length(unique_year2);
idx2 = find(iyear2 == unique_year2(k2));
% select data for year
d_time2 = d_time(idx2);
d_datx2 = d_datx(idx2);
d_daty2 = d_daty(idx2);
% pass through some function
mae_out = some_function(d_datx2, d_daty2);
mae(k2) = mae_out;
end
mean_mae(k) = mean(mae);
end
function mae = some_function(datx, daty)
mae = mean(abs(datx - daty));
end
Note here that I'm using a very simple function as an example, and the actual function is more complex.
Having two loops like this takes a long time to run on my actual data. Is there a better/faster way that I can perform the above, possibly without loops?

Answer 1

If you record the previous result, you do not need the inner loop. You are currently computing a total of (20+21)/2 = 210 iterations, but you only need to compute 20. The key here is that mean(a(1:k)) == (mean(a(1:k-1))*(k-1) + a(k)) / k (by the definition of mean). Another optimization is to use logical indexing instead of find. It takes up a bit more space, but is much faster.
% split into different years
datev = datevec(time);
iyear = datev(:,1);
unique_year = unique(iyear);
for k = 1:length(unique_year);
idx = (iyear == unique_year(k));
% select data for year
d_time = time(idx);
d_datx = datx(idx);
d_daty = daty(idx);
mae_out = some_function(d_datx, d_daty);
if k == 1
mean_mae(k) = mean_out;
else
mean_mae(k) = (mean_mae(k-1) * (k-1) + mean(mean_out)) / k;
end
end
function mae = some_function(datx, daty)
mae = mean(abs(datx - daty));
end
As you can see, this should give you approximately 20x or more speedup.