I am working on a dataset with 3750 text files each containing 10240*2 data. The file names are "Data_F_Ind0001 to Data_F_Ind3750". I have written a code to read each column of every file one-by-one and performed empirical mode decomposition (EMD). The EMD produced four variables I1 to I4 and for each of these four some other function (petropy) is performed. The problem with the code is, it's very slow. Can anyone suggest how to make it fast? I appreciate your help. Thank you.
I have give the sample code for processing first 9 files out of 3750. I have used same for loop for remaining files.
clear all;
close all;
l =1;
for k = 1:9
filename = sprintf('Data_F_Ind000%d.txt',k);
% a(:,:,k) = load(filename);
data = load (filename);
x = data(:,1);
y = data (:,2);
alldata = eemd(x,0.01,10);
I1 = alldata (1,:);
I2 = alldata (2,:);
I3 = alldata (3,:);
I4 = alldata (4,:);
imf = {I1, I2, I3, I4};
for j = 1:4
m1(k,j)= petropy(imf{j},3,1,'order');
j=j+1;
l=l+1;
end
end
You don't seem to preallocate memory for m1(k,j). Add m1 = zeros(3750,4) in front of the for loop.
I'm assuming m1() is an array, if it's a stuct or something else, change it accordingly.
Edit:
E.g. like this:
clear all;
close all;
l =1;
m1 = zeros(3750,4);
for k = 1:9
....
It's an important topic when dealing with for loops iteratively generating data - i'd suggest reading this article
These are the things that comes to my mind by looking at your code:
Don't put columns of data in variables x and y. By doing this you are using twice the memory. In you function eemd simply use data(:,1) as input. The same thing applies to I1 to I4 but I guess it has less effect as they are small size variables.
You can try textscan or fscanf instead of sprintf. This should also improve your code.
Here is a more optimised version of your code:
m1 = zeros(3750, 4);
for k = 1:9
filename = sprintf('Data_F_Ind000%d.txt',k);
data = load(filename);
alldata = eemd(data(:, 1),0.01,10);
% for j = 1:4
% m1(k,j)= petropy(alldata(j, :), 3, 1, 'order');
% end
m1(k, :) = arrayfun(#(j) petropy(alldata(j, :), 3, 1, 'order'), 1:4);
end
I have replaced the inner for loop with arrayfun. In case you don't understand it, you could use the for loop that i have commented out
Related
Hi for my code I would like to know how to best save my variable column. column is 733x1. Ideally I would like to have
column1(y)=column, but I obtain the error:
Conversion to cell from logical is not possible.
in the inner loop. I find it difficult to access these stored values in overlap.
for i = 1:7
for y = 1:ydim % ydim = 436
%execute code %code produces different 'column' on each iteration
column1{y} = column; %'column' size 733x1 %altogether 436 sets of 'column'
end
overlap{i} = column1; %iterates 7 times.
end
Ideally I want overlap to store 7 variables saved that are (733x436).
Thanks.
I'm assuming column is calculated using a procedure where each column is dependent on the latter. If not, then there are very likely improvements that can be made to this:
column = zeros(733, 1); % Might not need this. Depends on you code.
all_columns = zeros(xdim, ydim); % Pre-allocate memory (always do this)
% Note that the first dimension is usually called x,
% and the second called y in MATLAB
overlap = cell(7, 1);
overlap(:) = {zeros(xdim, ydim)}; % Pre-allocate memory
for ii = 1:numel(overlap) % numel is better than length
for jj = 1:ydim % ii and jj are better than i and j
% several_lines_of_code_to_calculate_column
column = something;
all_columns(:, jj) = column;
end
overlap{ii} = all_columns;
end
You can access the variables in overlap like this: overlap{1}(1,1);. This will get the first element in the first cell. overlap{2} will get the entire matrix in the second cell.
You specified that you wanted 7 variables. Your code implies that you know that cells are better than assigning it to different variables (var1, var2 ...). Good! The solution with different variables is bad bad bad.
Instead of using a cell array, you could instead use a 3D-array. This might make processing later on faster, if you can vectorize stuff for instance.
This will be:
column = zeros(733, 1); % Might not need this. Depends on you code.
overlap = zeros(xdim, ydim, 7) % Pre-allocate memory for 3D-matrix
for ii = 1:7
for jj = 1:ydim
% several_lines_of_code_to_calculate_column
column = something;
all_column(:, jj, ii) = column;
end
end
I am trying to concatenate several structs. What I take from each struct depends on a function that requires a for loop. Here is my simplified array:
t = 1;
for t = 1:5 %this isn't the for loop I am asking about
a(t).data = t^2; %it just creates a simple struct with 5 data entries
end
Here I am doing concatenation manually:
A = [a(1:2).data a(1:3).data a(1:4).data a(1:5).data] %concatenation function
As you can see, the range (1:2), (1:3), (1:4), and (1:5) can be looped, which I attempt to do like this:
t = 2;
A = [for t = 2:5
a(1:t).data
end]
This results in an error "Illegal use of reserved keyword "for"."
How can I do a for loop within the concatenate function? Can I do loops within other functions in Matlab? Is there another way to do it, other than copy/pasting the line and changing 1 number manually?
You were close to getting it right! This will do what you want.
A = []; %% note: no need to initialize t, the for-loop takes care of that
for t = 2:5
A = [A a(1:t).data]
end
This seems strange though...you are concatenating the same elements over and over...in this example, you get the result:
A =
1 4 1 4 9 1 4 9 16 1 4 9 16 25
If what you really need is just the .data elements concatenated into a single array, then that is very simple:
A = [a.data]
A couple of notes about this: why are the brackets necessary? Because the expressions
a.data, a(1:t).data
don't return all the numbers in a single array, like many functions do. They return a separate answer for each element of the structure array. You can test this like so:
>> [b,c,d,e,f] = a.data
b =
1
c =
4
d =
9
e =
16
f =
25
Five different answers there. But MATLAB gives you a cheat -- the square brackets! Put an expression like a.data inside square brackets, and all of a sudden those separate answers are compressed into a single array. It's magic!
Another note: for very large arrays, the for-loop version here will be very slow. It would be better to allocate the memory for A ahead of time. In the for-loop here, MATLAB is dynamically resizing the array each time through, and that can be very slow if your for-loop has 1 million iterations. If it's less than 1000 or so, you won't notice it at all.
Finally, the reason that HBHB could not run your struct creating code at the top is that it doesn't work unless a is already defined in your workspace. If you initialize a like this:
%% t = 1; %% by the way, you don't need this, the t value is overwritten by the loop below
a = []; %% always initialize!
for t = 1:5 %this isn't the for loop I am asking about
a(t).data = t^2; %it just creates a simple struct with 5 data entries
end
then it runs for anyone the first time.
As an appendix to gariepy's answer:
The matrix concatenation
A = [A k];
as a way of appending to it is actually pretty slow. You end up reassigning N elements every time you concatenate to an N size vector. If all you're doing is adding elements to the end of it, it is better to use the following syntax
A(end+1) = k;
In MATLAB this is optimized such that on average you only need to reassign about 80% of the elements in a matrix. This might not seam much, but for 10k elements this adds up to ~ an order of magnitude of difference in time (at least for me).
Bare in mind that this works only in MATLAB 2012b and higher as described in this thead: Octave/Matlab: Adding new elements to a vector
This is the code I used. tic/toc syntax is not the most accurate method for profiling in MATLAB, but it illustrates the point.
close all; clear all; clc;
t_cnc = []; t_app = [];
N = 1000;
for n = 1:N;
% Concatenate
tic;
A = [];
for k = 1:n;
A = [A k];
end
t_cnc(end+1) = toc;
% Append
tic;
A = [];
for k = 1:n;
A(end+1) = k;
end
t_app(end+1) = toc;
end
t_cnc = t_cnc*1000; t_app = t_app*1000; % Convert to ms
% Fit a straight line on a log scale
P1 = polyfit(log(1:N),log(t_cnc),1); P_cnc = #(x) exp(P1(2)).*x.^P1(1);
P2 = polyfit(log(1:N),log(t_app),1); P_app = #(x) exp(P2(2)).*x.^P2(1);
% Plot and save
loglog(1:N,t_cnc,'.',1:N,P_cnc(1:N),'k--',...
1:N,t_app,'.',1:N,P_app(1:N),'k--');
grid on;
xlabel('log(N)');
ylabel('log(Elapsed time / ms)');
title('Concatenate vs. Append in MATLAB 2014b');
legend('A = [A k]',['O(N^{',num2str(P1(1)),'})'],...
'A(end+1) = k',['O(N^{',num2str(P2(1)),'})'],...
'Location','northwest');
saveas(gcf,'Cnc_vs_App_test.png');
I have a little parfor test script which gives the warning in the title.
The code is this
out = zeros(10, 1);
in = rand(5e8, 10);
tic
parfor i = 1:10
for j = 1:5e8
p = floor(rand(1,1)*5e8);
out(i) = out(i) + in(p, i);
end
end
toc
tot = sum(out)
the warning comes out on line 7 regarding how variable in is accessed.
I don't understand why, slicing should be trivial. Just send each column of in to each worker.
If I change the code to
out = zeros(10, 1);
in = rand(5e8, 10);
tic
parfor i = 1:10
a = in(:,i);
for j = 1:5e8
p = floor(rand(1,1)*5e8);
out(i) = out(i) + a(p);
end
end
toc
tot = sum(out)
the warning disappears but I don't like that assignment to a.
The code was explicitly designed to mess up the cache memory.
Unfortunately, as explained here http://www.mathworks.com/help/distcomp/advanced-topics.html#bq_of7_-1 , MATLAB does not understand how to slice in, hence the code analyser warning. You have to read that page fairly closely to understand why it cannot be sliced. The relevant paragraph is:
Form of Indexing. Within the list of indices for a sliced variable, one of these indices is of the form i, i+k, i-k, k+i, or k-i, where i
is the loop variable and k is a constant or a simple (nonindexed)
broadcast variable; and every other index is a scalar constant, a
simple broadcast variable, colon, or end.
The clause in bold type at the end is the relevant one - in your case, p does not match this constraint.
I have surjective functions created by matching one element in an array MatchesX.trainIdx to one or more elements in a second array MatchesX.queryIdx.
To obtain only the bijective elements of said funciton I run the same function forward
Matches1=Matcher.match(Descriptors1,Descriptors2);
and then backwards
Matches2=Matcher.match(Descriptors2,Descriptors1);
and then look for the elements occuring in both function in following fashion:
k=1;
DoubleMatches=Matches1;
for i=1:length(Matches1)
for j=1:length(Matches2)
if((Matches1(i).queryIdx==Matches2(j).trainIdx)&&(Matches1(i).trainIdx==Matches2(j).queryIdx))
DoubleMatches(k)=Matches1(i);
k=k+1;
end
end
end
DoubleMatches(k:end)=[];
This of course does the work, but it is rather unelegant and seems to bother the JIT accelerator (calc time with accel on and accel off is the same).
Can you think of a way to vectorize this expresion? Is there any other way of avoiding the JIT from "striking"?
Thanks a lot and sorry about the strange structs, I'm working with MEX-functions. Let me know if rewriting the code in "normal" arrays would help
Access to data in multi-dimensional structures is notoriously slow in MATLAB, so transforming your data to an ordinary array will certainly help:
kk = 1;
DoubleMatches = Matches1;
%// transform to regular array
Matches1queryIdx = [Matches1.queryIdx];
Matches1trainIdx = [Matches1.trainIdx];
Matches2queryIdx = [Matches2.queryIdx];
Matches2trainIdx = [Matches2.trainIdx];
%// loop through transformed data instead of structures
for ii = 1:length(Matches1queryIdx)
for jj = 1:length(Matches1queryIdx)
if((Matches1queryIdx(ii)==Matches2trainIdx(jj)) && ...
(Matches1trainIdx(ii)==Matches2queryIdx(jj)))
DoubleMatches(kk) = Matches1(ii);
kk = kk+1;
end
end
end
DoubleMatches(kk:end)=[];
There is also a solution that is almost entirely vectorized:
matches = sum(...
bsxfun(#eq, [Matches1.queryIdx], [Matches2.trainIdx].') & ...
bsxfun(#eq, [Matches1.trainIdx], [Matches2.queryIdx].'));
contents = arrayfun(#(x)..
repmat(Matches1(x),1,matches(x)), 1:numel(matches), ...
'Uniformoutput', false);
DoubleMatches2 = [contents{:}]';
Note that this can be a lot more memory intensive (it has O(N²) peak memory footprint, as opposed to O(N) for the others, although the data type at peak memory is logical and thus 8x smaller than double...). Better do some checks beforehand which one you should use.
A little test. I used the following dummy data:
Matches1 = struct(...
'queryIdx', num2cell(randi(25,1000,1)),...
'trainIdx', num2cell(randi(25,1000,1))...
);
Matches2 = struct(...
'queryIdx', num2cell(randi(25,1000,1)),...
'trainIdx', num2cell(randi(25,1000,1))...
);
and the following test:
%// Your original method
tic
kk = 1;
DoubleMatches = Matches1;
for ii = 1:length(Matches1)
for jj = 1:length(Matches2)
if((Matches1(ii).queryIdx==Matches2(jj).trainIdx) && ...
(Matches1(ii).trainIdx==Matches2(jj).queryIdx))
DoubleMatches(kk) = Matches1(ii);
kk = kk+1;
end
end
end
DoubleMatches(kk:end)=[];
toc
DoubleMatches1 = DoubleMatches;
%// Method with data transformed into regular array
tic
kk = 1;
DoubleMatches = Matches1;
Matches1queryIdx = [Matches1.queryIdx];
Matches1trainIdx = [Matches1.trainIdx];
Matches2queryIdx = [Matches2.queryIdx];
Matches2trainIdx = [Matches2.trainIdx];
for ii = 1:length(Matches1queryIdx)
for jj = 1:length(Matches1queryIdx)
if((Matches1queryIdx(ii)==Matches2trainIdx(jj)) && ...
(Matches1trainIdx(ii)==Matches2queryIdx(jj)))
DoubleMatches(kk) = Matches1(ii);
kk = kk+1;
end
end
end
DoubleMatches(kk:end)=[];
toc
DoubleMatches2 = DoubleMatches;
% // Vectorized method
tic
matches = sum(...
bsxfun(#eq, [Matches1.queryIdx], [Matches2.trainIdx].') & ...
bsxfun(#eq, [Matches1.trainIdx], [Matches2.queryIdx].'));
contents = arrayfun(#(x)repmat(Matches1(x),1,matches(x)), 1:numel(matches), 'Uniformoutput', false);
DoubleMatches3 = [contents{:}]';
toc
%// Check if all are equal
isequal(DoubleMatches1,DoubleMatches2, DoubleMatches3)
Results:
Elapsed time is 6.350679 seconds. %// ( 1×) original method
Elapsed time is 0.636479 seconds. %// (~10×) method with regular array
Elapsed time is 0.165935 seconds. %// (~40×) vectorized
ans =
1 %// indeed, outcomes are equal
Assuming Matcher.match returns array of the same objects as passed to it as arguments you can solve this like this
% m1 are all d1s which have relation to d2
m1 = Matcher.match(d1,d2);
% m2 are all d2s, which have relation to m1
% and all m1 already have backward relation
m2 = Matcher.match(d2,m1);
I have the following code, pasted below. I would like to change it to only average the 10 most recently filtered images and not the entire group of filtered images. The line I think I need to change is: Yout(k,p,q) = (Yout(k,p,q) + (y.^2))/2;, but how do I do it?
j=1;
K = 1:3600;
window = zeros(1,10);
Yout = zeros(10,column,row);
figure;
y = 0; %# Preallocate memory for output
%Load one image
for i = 1:length(K)
disp(i)
str = int2str(i);
str1 = strcat(str,'.mat');
load(str1);
D{i}(:,:) = A(:,:);
%Go through the columns and rows
for p = 1:column
for q = 1:row
if(mean2(D{i}(p,q))==0)
x = 0;
else
if(i == 1)
meanvalue = mean2(D{i}(p,q));
end
%Calculate the temporal mean value based on previous ones.
meanvalue = (meanvalue+D{i}(p,q))/2;
x = double(D{i}(p,q)/meanvalue);
end
%Filtering for 10 bands, based on the previous state
for k = 1:10
[y, ZState{k}] = filter(bCoeff{k},aCoeff{k},x,ZState{k});
Yout(k,p,q) = (Yout(k,p,q) + (y.^2))/2;
end
end
end
% for k = 2:10
% subplot(5,2,k)
% subimage(Yout(k)*5000, [0 100]);
% colormap jet
% end
% pause(0.01);
end
disp('Done Loading...')
The best way to do this (in my opinion) would be to use a circular-buffer to store your images. In a circular-, or ring-buffer, the oldest data element in the array is overwritten by the newest element pushed in to the array. The basics of making such a structure are described in the short Mathworks video Implementing a simple circular buffer.
For each iteration of you main loop that deals with a single image, just load a new image into the circular-buffer and then use MATLAB's built in mean function to take the average efficiently.
If you need to apply a window function to the data, then make a temporary copy of the frames multiplied by the window function and take the average of the copy at each iteration of the loop.
The line
Yout(k,p,q) = (Yout(k,p,q) + (y.^2))/2;
calculates a kind of Moving Average for each of the 10 bands over all your images.
This line calculates a moving average of meanvalue over your images:
meanvalue=(meanvalue+D{i}(p,q))/2;
For both you will want to add a buffer structure that keeps only the last 10 images.
To simplify it, you can also just keep all in memory. Here is an example for Yout:
Change this line: (Add one dimension)
Yout = zeros(3600,10,column,row);
And change this:
for q = 1:row
[...]
%filtering for 10 bands, based on the previous state
for k = 1:10
[y, ZState{k}] = filter(bCoeff{k},aCoeff{k},x,ZState{k});
Yout(i,k,p,q) = y.^2;
end
YoutAvg = zeros(10,column,row);
start = max(0, i-10+1);
for avgImg = start:i
YoutAvg(k,p,q) = (YoutAvg(k,p,q) + Yout(avgImg,k,p,q))/2;
end
end
Then to display use
subimage(Yout(k)*5000, [0 100]);
You would do sth. similar for meanvalue