I have a big square matrix (refConnect) with approx 500000 elements.
I need to perform this operation:
tmp = find(referenceCluster == 67);
for j=1:length(tmp)
refConnect(tmp(j),tmp)=1;
end
I wonder if there is a simple way to vectorise this so I can avoid the for loop which is taking forever.
Thanks for any help.
Cheers
Seems you can't significantly decrease execution time.
Try evaluate the execution time with this test function.
function test_spped(N, M)
if nargin < 1
N = 707;
end
if nargin < 2
M = 2;
end
refConnectIn = rand(N, N);
referenceCluster = randi(M, N, 1);
refConnectA = refConnectIn;
tic
tmpA = find(referenceCluster == 1);
for j=1:length(tmpA)
refConnectA(tmpA(j),tmpA) = 1;
end
toc
refConnectB = refConnectIn;
tic
tmpB = referenceCluster == 1;
refConnectB(tmpB, tmpB) = 1;
toc
if isequal(refConnectA, refConnectB)
disp('Result are equals');
else
disp('Result are UNEQUALS!');
end
With default parameters you get:
>> test_speed
Elapsed time is 0.002865 seconds.
Elapsed time is 0.001575 seconds.
Result are equals
Note, the execution time of the vectorized code (B case) can be worse for large M:
>> test_speed(707,1000)
Elapsed time is 0.001623 seconds.
Elapsed time is 0.002219 seconds.
Result are equals
Suppose that c is a scalar value, T and W are M-by-N matrices, k is another M-by-N matrix containing values from 1 to M (and there are at least two pairs (i1, j1), (i2, j2) such that k(i1, j1)==k(i2, j2)) and a is a 1-by-M vector. I want to vectorize the following code (hoping that this will speed it up):
T = zeros(M,N);
for j = 1:N
for i = 1:M
T(k(i,j),j) = T(k(i,j),j) + c*W(i,j)/a(i);
end
end
Do you have any tips so that I can vectorize this code (or make it faster in general)?
Thanks in advance!
Since k only ever effects how values are aggregated within a column, but not between columns, you can achieve a slight speedup by reducing the problem to a single loop over columns and using accumarray like so:
T = zeros(M, N);
for col = 1:N
T(:, col) = accumarray(k(:,col), c*W(:, col)./a, [M 1]);
end
I tested each of the solutions (the loop in your question, rahnema's, Divakar's, and mine) by taking the average of 100 iterations using input values initialized as in Divakar's answer. Here's what I got (running Windows 7 x64, 16 GB RAM, MATLAB R2016b):
solution | avg. time (s) | max(abs(err))
---------+---------------+---------------
loop | 0.12461 | 0
rahnema | 0.84518 | 0
divakar | 0.12381 | 1.819e-12
gnovice | 0.09477 | 0
The take-away: loops actually aren't so bad, but if you can simplify them into one it can save you a little time.
Here's an approach with a combination of bsxfun and accumarray -
% Create 2D array of unique IDs along each col to be used as flattened subs
id = bsxfun(#plus,k,M*(0:N-1));
% Compute "c*W(i,j)/a(i)" for all i's and j's
cWa = c*bsxfun(#rdivide,W,a);
% Accumulate final result for all cols
out = reshape(accumarray(id(:),reshape(cWa,[],1),[M*N 1]),[M,N]);
Benchmarking
Approaches as functions -
function out = func1(W,a,c,k,M,N)
id = bsxfun(#plus,k,M*(0:N-1));
cWa = c*bsxfun(#rdivide,W,a);
out = reshape(accumarray(id(:),reshape(cWa,[],1),[M*N 1]),[M,N]);
function T = func2(W,a,c,k,M,N) % #rahnema1's solution
[I J] = meshgrid(1:M,1:N);
idx1 = sub2ind([M N], I ,J);
R = c.* W(idx1) ./ a(I);
T = accumarray([k(idx1(:)) ,J(:)], R(:),[M N]);
function T = func3(W,a,c,k,M,N) % Original approach
T = zeros(M,N);
for j = 1:N
for i = 1:M
T(k(i,j),j) = T(k(i,j),j) + c*W(i,j)/a(i);
end
end
function T = func4(W,a,c,k,M,N) % #gnovice's solution
T = zeros(M, N);
for col = 1:N
T(:, col) = accumarray(k(:,col), c*W(:, col)./a, [M 1]);
end
Machine setup : Kubuntu 16.04, MATLAB 2012a, 4GB RAM.
Timing code -
% Setup inputs
M = 3000;
N = 3000;
W = rand(M,N);
a = rand(M,1);
c = 2.34;
k = randi([1,M],[M,N]);
disp('------------------ With func1')
tic,out = func1(W,a,c,k,M,N);toc
clear out
disp('------------------ With func2')
tic,out = func2(W,a,c,k,M,N);toc
clear out
disp('------------------ With func3')
tic,out = func3(W,a,c,k,M,N);toc
clear out
disp('------------------ With func4')
tic,out = func4(W,a,c,k,M,N);toc
Timing code run -
------------------ With func1
Elapsed time is 0.215591 seconds.
------------------ With func2
Elapsed time is 1.555373 seconds.
------------------ With func3
Elapsed time is 0.572668 seconds.
------------------ With func4
Elapsed time is 0.291552 seconds.
Possible improvements in proposed approach
1] In c*bsxfun(#rdivide,W,a), we are use two stages of broadcasting - One at bsxfun(#rdivide,W,a), where a is broadcasted ; Second one when c is broadcasted to match-up against the 2D output of bsxfun(#rdivide,W,a), though we don't need bsxfun for this one. So, a possible improvement would be if we insert-in c to be divided by a, where c would be only broadcasted to 1D, instead of 2D and then the second level of broadcasting would be1D: c/a to 2D : W just like before. This minor improvement could be timed -
>> tic, c*bsxfun(#rdivide,W,a); toc
Elapsed time is 0.073244 seconds.
>> tic, bsxfun(#times,W,c/a); toc
Elapsed time is 0.041745 seconds.
But, in cases where c and a differ by a lot, the scaling factor c/a would affect the final result by appreciably. So, one need to be careful with this suggestion.
A possible solution:
[I J] = meshgrid(1:M,1:N);
idx1 = sub2ind([M N], I ,J);
R = c.* W(idx1) ./ a(I);
T = accumarray([K(idx1(:)) ,J(:)], R(:),[M N]);
Comparison of different methods in Octave without jit:
------------------ Divakar
Elapsed time is 0.282008 seconds.
------------------ rahnema1
Elapsed time is 1.08827 seconds.
------------------ gnovice
Elapsed time is 0.418701 seconds.
------------------ loop
doesn't complete in 15 seconds.
How can I get the mean(average) of my stimulation time instead of listing all the 100 stimulation time for manual calculation?
function [time] = Babtime(n)
N = 100;
t = zeros(N,1);
for k = 1:N
tic;
Bab(n);
Stimulationtime=toc
Also how can I get the minimum stimulation time without me manually checking for the minimum out of 1000 output time for the code below
function [time] = Haldtime(n)
N = 1000;
t = zeros(N,1);
for k = 1:N
tic;
Hald(n);
Stimulationtime=toc
Thank you
Store the toc values in an array, and then use min. For your code, you already appear to have created the array to store the time values (the array t):
function [time] = Haldtime(n)
N = 1000;
t = zeros(N,1);
for k = 1:N
tic;
Hald(n);
t(k)=toc;
end
disp(min(t));
time=min(t);
Of course, if you want the average (which is far more useful) then replace min with mean
To make it more handy, letting you use any function with any number of input arguments and with variable number of run cycles, you can modify your timer function to the following:
function [times, t_min, t_mean] = myTimer(funcH, sampling, varargin)
times = zeros(sampling, 1);
for ii = 1:sampling
tic;
funcH(varargin{:});
times(ii) = toc;
end
t_min = min(times);
t_mean = mean(times);
Then you can simply use this function to calculate run time of different functions. For your examples it will be:
[~, tmin, tavg] = myTimer(#Bab, 100, n);
or
[T, tmin, tavg] = myTimer(#Hald, 1000, n);
Note that in the second example all of the run times are saved in T, so later you can calculate different statistics besides min and mean.
It might be also useful to take a look at profile.
Now I have a matrix A of dimension N by p, and the other matrix B of dimension N by q. What I want to have is a matrix, say C, of dimension N by pq such that
C(i,:) = kron(A(i,:), B(i,:));
If N is large, loop over N rows may take quite long time. So currently I am augmenting A and B appropriately(combining usage of repmat, permute and reshape) to turn each matrix of dimension N by pq, and then formulating C by something like
C = A_aug .* B_aug;
Any better idea?
Checkout some bsxfun + permute + reshape magic -
out = reshape(bsxfun(#times,permute(A,[1 3 2]),B),size(A,1),[])
Benchmarking & Verification
Benchmarking code -
%// Setup inputs
N = 200;
p = 190;
q = 180;
A = rand(N,p);
B = rand(N,q);
disp('--------------------------------------- Without magic')
tic
C = zeros(size(A,1),size(A,2)*size(B,2));
for i = 1:size(A,1)
C(i,:) = kron(A(i,:), B(i,:));
end
toc
disp('--------------------------------------- With some magic')
tic
out = reshape(bsxfun(#times,permute(A,[1 3 2]),B),size(A,1),[]);
toc
error_val = max(abs(C(:)-out(:)))
Output -
--------------------------------------- Without magic
Elapsed time is 0.524396 seconds.
--------------------------------------- With some magic
Elapsed time is 0.055082 seconds.
error_val =
0
I have a big matrix M (nxm). I am going to sum some elements which are specified by index stored in vector as cell elements. There are many groups of indices so the cell has more than one element. For example
M = rand(2103, 2030);
index{1} = [1 3 2 4 53 5 23 3];
index{2} = [2 3 1 3 23 10234 2032];
% ...
index{2032} = ...;
I am going to sum up all elements at index{1}, sum up all elements at index{2} ..., now I am using a loop
sums = zeros(1, 2032);
for n=1:2032
sums(n) = sum(M(index{n}));
end
I am wondering if there is any way to use one-line command instead of a loop to do that. Using a loop is pretty slow.
Probably a classic use of cellfun
sums = cellfun(#(idx) sum(M(idx)), index);
EDIT: here is a benchmarking for a large case that shows that this approach is slightly slower than a for loop but faster than Eitan T's method
M = rand(2103, 2030);
index = cell(1, 2032);
index{1} = [1 3 2 4 53 5 23 3];
index{2} = [2 3 1 3 23 10234 2032];
for n=3:2032
index{n} = randi(numel(M), 1, randi(10000));
end
N = 1e1;
sums = zeros(1, 2032);
tic
for kk = 1:N
for n=1:2032
sums(n) = sum(M(index{n}));
end
end
toc
tic
for kk = 1:N
sums = cellfun(#(idx) sum(M(idx)), index);
end
toc
tic
for kk = 1:N
sums = cumsum(M([index{:}]));
sums = diff([0, sums(cumsum(cellfun('length', index)))]);
end
toc
results in
Elapsed time is 2.072292 seconds.
Elapsed time is 2.139882 seconds.
Elapsed time is 2.669894 seconds.
Perhaps not as elegant as a cellfun one-liner, but runs more than an order of magnitude faster:
sums = cumsum(M([index{:}]));
sums = diff([0, sums(cumsum(cellfun('length', index)))]);
It even runs approximately 4 or 5 times faster than a JIT-accelerated loop for large inputs. Note that when each cell in index contains a vector with more than ~2000 elements, the performance of this approach begins to deteriorate in comparison with a loop (and cellfun).
Benchmark
M = rand(2103, 2030);
I = ceil(numel(M) * rand(2032, 10));
index = mat2cell(I, ones(size(I, 1), 1), size(I, 2));
N = 100;
tic
for k = 1:N
sums = zeros(1, numel(index));
for n = 1:numel(sums)
sums(n) = sum(M(index{n}));
end
end
toc
tic
for k = 1:N
sums = cellfun(#(idx) sum(M(idx)), index);
end
toc
tic
for k = 1:N
sums = cumsum(M([index{:}]));
sums2 = diff([0, sums(cumsum(cellfun('length', index)))]);
end
toc
When executing this in MATLAB 2012a (Windows Server 2008 R2 running on a 2.27GHz 16-core Intel Xeon processor), I got:
Elapsed time is 0.579783 seconds.
Elapsed time is 1.789809 seconds.
Elapsed time is 0.111455 seconds.