i want to remove nested for loops from my code?
i can't remove them.
k = 3;
Data = rand(100,5);
m = zeros(size(Data));
N = size(Data,2); % number of features
M = size(Data,1); % number of objects
bound = zeros(N,k+1);
MAX = max(Data);
MIN = min(Data);
for ii = 1:N
bound(ii,:) = linspace(MIN(ii), MAX(ii), k+1);
end
bound(:,end) = bound(:,end)+eps;
tic;
for ii = 1:M
for jj=1:N
for kk=1:k
if bound(jj,kk)<=Data(ii,jj) && Data(ii,jj)<bound(jj,kk+1)
m(ii,jj) = kk;
end
end
end
end
You can do away with nesting upto a certain limit.
At a glance, as the jj index seems to be uniform in the operation within the nested loop, you can replace
for ii = 1:M
for jj=1:N
for kk=1:k
if bound(jj,kk)<=Data(ii,jj) && Data(ii,jj)<bound(jj,kk+1)
m(ii,jj) = kk;
end
end
end
end
by simply
for ii = 1:M
for kk=1:k
m(ii,(bound(:,kk)<=Data(ii,:)' & Data(ii,:)'<bound(:,kk+1))) = kk;
end
end
This would give you the exact same result as before.
Since your longest loop is over ii=1:M, we should prioritise vectorising this one over the others. The smallest loop is over kk=1:k so this one can probably stay without worrying about it too much. You can use bsxfun to great effect in vectorisations of this sort:
for kk = 1:k
ind = bsxfun(#le, bound(:, kk)', Data) & bsxfun(#gt, bound(:, kk+1)', Data);
m(ind) = kk;
end
This gives the same result as your above code.
Another alternative is histc(), which is specifically designed for binning:
for jj = 1:N
[~, m(:,jj)] = histc(Data(:,jj),bound(jj,:));
end
This solution is on par with bsxfun() but it's not a very meaningful comparison because here the loop is across columns while with bsxfun is across bounds. Therefore, as a rule of thumb I would go with histc() if I have less columns than bounds, otherwise bsxfun().
Related
I am currently looking for the most efficient way to shift and rearrange large matrices. Essentially, I have data with some parabolic shift that needs to be corrected in order to shift the "signal" to a linear event.
I have currently tried the following solutions and tried timing them. Is there any other method that may prove to be more efficient?
DATA = ones(100000,501);
DATA(10000,251) = 100;
for i=1:250
DATA(10000+i^2-1000:10000+i^2+1000,251-i) = 100;
DATA(10000+i^2-1000:10000+i^2+1000,251+i) = 100;
end
k = abs(-250:1:250).^2;
d = size(DATA,1);
figure(99)
imagesc(DATA)
t_INDEX = timeit(#()fun_INDEX(DATA,k))
t_SNIPPET = timeit(#()fun_SNIPPET(DATA,k))
t_CIRCSHIFT = timeit(#()fun_CIRCSHIFT(DATA,k))
t_INDEX_clean = timeit(#()fun_INDEX_clean(DATA,k))
t_SPARSE = timeit(#()fun_SPARSE(DATA,k))
t_BSXFUN = timeit(#()fun_BSXFUN(DATA,k))
function fun_INDEX(DATA,k)
DATA_1 = zeros(size(DATA));
for i=1:size(DATA,2)
DATA_1(:,i) = DATA([k(i)+1:end 1:k(i)],i);
end
figure(1)
imagesc(DATA_1)
end
function fun_SNIPPET(DATA,k)
kmax = max(k);
DATA_2 = zeros(size(DATA,1)-kmax,size(DATA,2));
for i=1:size(DATA,2)
DATA_2(:,i) = DATA(k(i)+1:end-kmax+k(i),i);
end
figure(2)
imagesc(DATA_2)
end
function fun_CIRCSHIFT(DATA,k)
DATA_3 = zeros(size(DATA));
for i=1:size(DATA,2)
DATA_3(:,i) = circshift(DATA(:,i),-k(i),1);
end
figure(3)
imagesc(DATA_3)
end
function fun_INDEX_clean(DATA,k)
[m, n] = size(DATA);
k = size(DATA,1)-k;
DATA_4 = zeros(m, n);
for i = (1 : n)
DATA_4(:, i) = [DATA((m - k(i) + 1 : m), i); DATA((1 : m - k(i) ), i)];
end
figure(4)
imagesc(DATA_4)
end
function fun_SPARSE(DATA,k)
[m,n] = size(DATA);
k = -k;
S = full(sparse(mod(k,m)+1,1:n,1,m,n));
DATA_5 = ifft(fft(DATA).*fft(S),'symmetric');
figure(5)
imagesc(DATA_5)
end
function fun_BSXFUN(DATA,k)
DATA = DATA';
k = -k;
[m,n] = size(DATA);
idx0 = mod(bsxfun(#plus,n-k(:),1:n)-1,n);
DATA_6 = DATA(bsxfun(#plus,(idx0*m),(1:m)'));
figure(6)
imagesc(DATA_6)
end
Is there any way to decrease computation time for this kind of problem?
Thanks in advance for any tips!
One option would be to use MATLAB's GPU functions, if your workstation has a GPU. Depending on if the entire data fits on the GPU at once, it will start to outperform CPU circshift at 1000 X 1000 matrix size.
The implementation only requires you to copy your data to the GPU with a single statement, and then operate circshift on the newly created you array.
A small discussion on its performance can be found here: https://www.mathworks.com/matlabcentral/answers/274619-circshift-slower-on-gpu . Especially, the last post describes a much faster GPU implementation if you actually don't need to circularly shift, but can get away with zero passing on one side, which might be relevant.
I am trying to keep a track of all the calculations happening under 3 for loops. The data is too big therefore it is hard to keep the track of the data. Hence, I would like to construct a table which will record the number of iterations taking place inside every for loop.
The code:
for i = 1:4
% Calculations
i
for j = 1:3
% Calculations
j
for k = 1:3
% Calculations
k
end
end
end
So, the tabular output which I am expecting is like this,
Can anybody please help me in achieving this task.
You could use ndgrid to create all permutations of your i, j, and k values and then have a single for loop that loops through all permutations.
[ii, jj, kk] = ndgrid(1:4, 1:3, 1:3);
% Pre-allocate your results matrix
results = zeros(size(ii));
for n = 1:numel(ii)
% Do calculation with ii(n), jj(n), kk(n)
results(n) = ii(n) + jj(n) + kk(n);
end
Now if you want to know what the ii, jj, or kk values were for a particular entry in results, you can just index into all variables the same way.
result_of_interest = results(100);
i_of_interest = ii(100);
j_of_interest = jj(100);
k_of_interest = kk(100);
If you really need tabular output, you can transform ii, jj, and kk into your table.
data = cat(2, ii(:), jj(:), kk(:))';
You can try the following code, where you declare the dimension of each loop at the beginning, and allocate a track matrix.
ni=3
nj=4
nk=5
track = zeros(3,ni*nj*nk);
offset = 1
for i = 1:ni
% Calculations
i
for j = 1:nj
% Calculations
j
for k = 1:nk
% Calculations
k
track(1,offset) = i;
track(2,offset) = j;
track(3,offset) = k;
offset = offset + 1;
end
end
end
Suppose I have a matrix A and I want to apply a function f to each of its elements. I can then use f(A), if f is vectorized or arrayfun(f,A) if it's not.
But what if I had a function that depends on the entry and its indices: f = #(i,j,x) something. How do I apply this function to the matrix A without using a for loop like the following?
for j=1:size(A,2)
for i=1:size(A,1)
fA(i,j) = f(i,j,A(i,j));
end
end
I'd like to consider the function f to be vectorized. Hints on shorter notation for non-vectorized functions are welcome, though.
I have read your answers and I came up with another idea using indexing, which is the fastest way. Here is my test script:
%// Test function
f = #(i,j,x) i.*x + j.*x.^2;
%// Initialize times
tfor = 0;
tnd = 0;
tsub = 0;
tmy = 0;
%// Do the calculation 100 times
for it = 1:100
%// Random input data
A = rand(100);
%// Clear all variables
clear fA1 fA2 fA3 fA4;
%// Use the for loop
tic;
fA1(size(A,1),size(A,2)) = 0;
for j=1:size(A,2)
for i=1:size(A,1)
fA1(i,j) = f(i,j,A(i,j));
end
end
tfor = tfor + toc;
%// Use ndgrid, like #Divakar suggested
clear I J;
tic;
[I,J] = ndgrid(1:size(A,1),1:size(A,2));
fA2 = f(I,J,A);
tnd = tnd + toc;
%// Test if the calculation is correct
if max(max(abs(fA2-fA1))) > 0
max(max(abs(fA2-fA1)))
end
%// Use ind2sub, like #DennisKlopfer suggested
clear I J;
tic;
[I,J] = ind2sub(size(A),1:numel(A));
fA3 = arrayfun(f,reshape(I,size(A)),reshape(J,size(A)),A);
tsub = tsub + toc;
%// Test if the calculation is correct
if max(max(abs(fA3-fA1))) > 0
max(max(abs(fA3-fA1)))
end
%// My suggestion using indexing
clear sA1 sA2 ssA1 ssA2;
tic;
sA1=size(A,1);
ssA1=1:sA1;
sA2=size(A,2);
ssA2=1:sA2;
fA4 = f(ssA1(ones(1,sA2),:)', ssA2(ones(1,sA1,1),:), A); %'
tmy = tmy + toc;
%// Test if the calculation is correct
if max(max(abs(fA4-fA1))) > 0
max(max(abs(fA4-fA1)))
end
end
%// Print times
tfor
tnd
tsub
tmy
I get the result
tfor =
0.6813
tnd =
0.0341
tsub =
10.7477
tmy =
0.0171
Assuming that the function is vectorized ( no dependency or recursions involved), as mentioned in the comments earlier, you could use ndgrid to create 2D meshes corresponding to the two nested loop iterators i and j and of the same size as A. When these are fed to the particular function f, it would operate on the input 2D arrays in a vectorized manner. Thus, the implementation would look something like this -
[I,J] = ndgrid(1:size(A,1),1:size(A,2));
out = f(I,J,A);
Sample run -
>> f = #(i,j,k) i.^2+j.^2+sin(k);
A = rand(4,5);
for j=1:size(A,2)
for i=1:size(A,1)
fA(i,j) = f(i,j,A(i,j));
end
end
>> fA
fA =
2.3445 5.7939 10.371 17.506 26.539
5.7385 8.282 13.538 20.703 29.452
10.552 13.687 18.076 25.804 34.012
17.522 20.684 25.054 32.13 41.331
>> [I,J] = ndgrid(1:size(A,1),1:size(A,2)); out = f(I,J,A);
>> out
out =
2.3445 5.7939 10.371 17.506 26.539
5.7385 8.282 13.538 20.703 29.452
10.552 13.687 18.076 25.804 34.012
17.522 20.684 25.054 32.13 41.331
Using arrayfun(), ind2sub() and reshape() you can create the indexes matching the form of A. This way arrayfun() is applicable. There might be a better version as this feels a little bit like a hack, it should work on vectorized and unvectorized functions though.
[I,J] = ind2sub(size(A),1:numel(A));
fA = arrayfun(f,reshape(I,size(A)),reshape(J,size(A)),A)
Can anyone help vectorize this Matlab code? The specific problem is the sum and bessel function with vector inputs.
Thank you!
N = 3;
rho_g = linspace(1e-3,1,N);
phi_g = linspace(0,2*pi,N);
n = 1:3;
tau = [1 2.*ones(1,length(n)-1)];
for ii = 1:length(rho_g)
for jj = 1:length(phi_g)
% Coordinates
rho_o = rho_g(ii);
phi_o = phi_g(jj);
% factors
fc = cos(n.*(phi_o-phi_s));
fs = sin(n.*(phi_o-phi_s));
Ez_t(ii,jj) = sum(tau.*besselj(n,k(3)*rho_s).*besselh(n,2,k(3)*rho_o).*fc);
end
end
You could try to vectorize this code, which might be possible with some bsxfun or so, but it would be hard to understand code, and it is the question if it would run any faster, since your code already uses vector math in the inner loop (even though your vectors only have length 3). The resulting code would become very difficult to read, so you or your colleague will have no idea what it does when you have a look at it in 2 years time.
Before wasting time on vectorization, it is much more important that you learn about loop invariant code motion, which is easy to apply to your code. Some observations:
you do not use fs, so remove that.
the term tau.*besselj(n,k(3)*rho_s) does not depend on any of your loop variables ii and jj, so it is constant. Calculate it once before your loop.
you should probably pre-allocate the matrix Ez_t.
the only terms that change during the loop are fc, which depends on jj, and besselh(n,2,k(3)*rho_o), which depends on ii. I guess that the latter costs much more time to calculate, so it better to not calculate this N*N times in the inner loop, but only N times in the outer loop. If the calculation based on jj would take more time, you could swap the for-loops over ii and jj, but that does not seem to be the case here.
The result code would look something like this (untested):
N = 3;
rho_g = linspace(1e-3,1,N);
phi_g = linspace(0,2*pi,N);
n = 1:3;
tau = [1 2.*ones(1,length(n)-1)];
% constant part, does not depend on ii and jj, so calculate only once!
temp1 = tau.*besselj(n,k(3)*rho_s);
Ez_t = nan(length(rho_g), length(phi_g)); % preallocate space
for ii = 1:length(rho_g)
% calculate stuff that depends on ii only
rho_o = rho_g(ii);
temp2 = besselh(n,2,k(3)*rho_o);
for jj = 1:length(phi_g)
phi_o = phi_g(jj);
fc = cos(n.*(phi_o-phi_s));
Ez_t(ii,jj) = sum(temp1.*temp2.*fc);
end
end
Initialization -
N = 3;
rho_g = linspace(1e-3,1,N);
phi_g = linspace(0,2*pi,N);
n = 1:3;
tau = [1 2.*ones(1,length(n)-1)];
Nested loops form (Copy from your code and shown here for comparison only) -
for ii = 1:length(rho_g)
for jj = 1:length(phi_g)
% Coordinates
rho_o = rho_g(ii);
phi_o = phi_g(jj);
% factors
fc = cos(n.*(phi_o-phi_s));
fs = sin(n.*(phi_o-phi_s));
Ez_t(ii,jj) = sum(tau.*besselj(n,k(3)*rho_s).*besselh(n,2,k(3)*rho_o).*fc);
end
end
Vectorized solution -
%%// Term - 1
term1 = repmat(tau.*besselj(n,k(3)*rho_s),[N*N 1]);
%%// Term - 2
[n1,rho_g1] = meshgrid(n,rho_g);
term2_intm = besselh(n1,2,k(3)*rho_g1);
term2 = transpose(reshape(repmat(transpose(term2_intm),[N 1]),N,N*N));
%%// Term -3
angle1 = repmat(bsxfun(#times,bsxfun(#minus,phi_g,phi_s')',n),[N 1]);
fc = cos(angle1);
%%// Output
Ez_t = sum(term1.*term2.*fc,2);
Ez_t = transpose(reshape(Ez_t,N,N));
Points to note about this vectorization or code simplification –
‘fs’ doesn’t change the output of the script, Ez_t, so it could be removed for now.
The output seems to be ‘Ez_t’,which requires three basic terms in the code as –
tau.*besselj(n,k(3)*rho_s), besselh(n,2,k(3)*rho_o) and fc. These are calculated separately for vectorization as terms1,2 and 3 respectively.
All these three terms appear to be of 1xN sizes. Our aim thus becomes to calculate these three terms without loops. Now, the two loops run for N times each, thus giving us a total loop count of NxN. Thus, we must have NxN times the data in each such term as compared to when these terms were inside the nested loops.
This is basically the essence of the vectorization done here, as the three terms are represented by ‘term1’,’term2’ and ‘fc’ itself.
In order to give a self-contained answer, I'll copy the original initialization
N = 3;
rho_g = linspace(1e-3,1,N);
phi_g = linspace(0,2*pi,N);
n = 1:3;
tau = [1 2.*ones(1,length(n)-1)];
and generate some missing data (k(3) and rho_s and phi_s in the dimension of n)
rho_s = rand(size(n));
phi_s = rand(size(n));
k(3) = rand(1);
then you can compute the same Ez_t with multidimensional arrays:
[RHO_G, PHI_G, N] = meshgrid(rho_g, phi_g, n);
[~, ~, TAU] = meshgrid(rho_g, phi_g, tau);
[~, ~, RHO_S] = meshgrid(rho_g, phi_g, rho_s);
[~, ~, PHI_S] = meshgrid(rho_g, phi_g, phi_s);
FC = cos(N.*(PHI_G - PHI_S));
FS = sin(N.*(PHI_G - PHI_S)); % not used
EZ_T = sum(TAU.*besselj(N, k(3)*RHO_S).*besselh(N, 2, k(3)*RHO_G).*FC, 3).';
You can check afterwards that both matrices are the same
norm(Ez_t - EZ_T)
I have two matrices A and B, both contain a list of event start and stop times:
A(i,1) = onset time of event i
A(i,2) = offset time of event i
B(j,1) = onset of event j
...
My goal is to get two lists of indecies aIdx and bIdx such that A(aIdx,:) and B(bIdx,:) contain the sets of events that are overlapping.
I've been scratching my head all day trying to figure this one out. Is there a quick, easy, matlaby way to do this?
I can do it using for loops but this seems kind of hacky for matlab:
aIdx = [];
bIdx = []
for i=1:size(A,1)
for j=i:size(B,1)
if overlap(A(i,:), B(j,:)) % overlap is defined elsewhere
aIdx(end+1) = i;
bIdx(end+1) = j;
end
end
end
Here's a zero loop solution:
overlap = #(x, y)y(:, 1) < x(:, 2) & y(:, 2) > x(:, 1)
[tmp1, tmp2] = meshgrid(1:size(A, 1), 1:size(B, 1));
M = reshape(overlap(A(tmp1, :), B(tmp2, :)), size(B, 1), [])';
[aIdx, bIdx] = find(M);
You can do it with one loop:
aIdx = false(size(A,1),1);
bIdx = false(size(B,1),1);
for k = 1:size(B,1)
ai = ( A(:,1) >= B(k,1) & A(:,1) <= B(k,2) ) | ...
( A(:,2) >= B(k,1) & A(:,2) <= B(k,2) );
if any(ai), bIdx(k) = true; end
aIdx = aIdx | ai;
end
There is a way to create a vectorized algorithm. (I wrote a similar function before, but cannot find it right now.) A simply workflow is to (1) combine both matrices, (2) create an index to indicate source of each event, (3) create a matrix indicating start and stop positions, (4) vectorized and sort, (5) find overlaps with diff, cumsum, or combination.
overlap_matrix = zeros(size(A,1),size(B,1))
for jj = 1:size(B,1)
overlap_matrix(:,jj) = (A(:,1) <= B(jj,1)).*(B(jj,1) <= A(:,2));
end
[r,c] = find(overlap_matrix)
% Now A(r(i),:) overlaps with B(c(i),:)
% Modify the above conditional if you want to also check
% whether events in A start in-between the events in B
% as I am only checking the first half of the conditional
% for simplicity.
Completely vectorized code without repmat or reshape (hopefully faster too). I have assumed that the function "overlap" can give a vector of 1's and 0's if complete pairs of A(req_indices,:) and B(req_indices,:) are fed to it. If overlap can return a vector output, then the vectorization can be performed as given below.
Let rows in A matrix be Ra and Rows in B matrix be Rb,
AA=1:Ra;
AAA=AA(ones(Rb,1),:);
AAAA=AAA(:); % all indices of A arranged in desired format, i.e. [11...1,22..2,33...3, ...., RaRa...Ra]'
BB=(1:Rb)';
BBB=BB(:,ones(Ra,1));
BBBB=BBB(:);% all indices of B arranged in desired format, i.e. [123...Rb, 123...Rb,....,123...Rb]'
% Use overlap function
Result_vector = overlap(A(AAAA,:), B(BBBB,:));
Result_vector_without_zeros = find(Result_vector);
aIdx = AAAA(Results_vector_without_zeros);
bIdx = BBBB(Results_vector_without_zeros);
DISADVANTAGE : TOO MUCH RAM CONSUMPTION FOR LARGER MATRICES