Memory usage by Matlab - matlab

I'm running a Matlab code in the HPC of my university. I have two versions of the code. The second version, despite generating a smaller array, seems to require more memory. I would like your help to understand if this is in fact the case and why.
Let me start from some preliminary lines:
clear
rng default
%Some useful components
n=7^4;
vectors{1}=[1,20,20,20,-1,Inf,-Inf];
vectors{2}=[-19,19,19,19,-20,Inf,-Inf];
vectors{3}=[-19,0,0,0,-20,Inf,-Inf];
vectors{4}=[-19,0,0,0,-20,Inf,-Inf];
T_temp = cell(1,4);
[T_temp{:}] = ndgrid(vectors{:});
T_temp = cat(4+1, T_temp{:});
T = reshape(T_temp,[],4); %all the possible 4-tuples from vectors{1}, ..., vectors{4}
This is the first version 1 of the code: I construct the matrix D1 listing all possible pairs of unordered rows from T
indices_pairs=pairIndices(n);
D1=[T(indices_pairs(:,1),:) T(indices_pairs(:,2),:)];
This is the second version of the code: I construct the matrix D2 listing a random draw of m=10^6 unordered pairs of rows from T
m=10^6;
p=n*(n-1)/2;
random_indices_pairs = randperm(p, m).';
[C1, C2] = myind2ind (random_indices_pairs, n);
indices_pairs=[C1 C2];
D2=[T(indices_pairs(:,1),:) T(indices_pairs(:,2),:)];
My question: when generating D2 the HPC goes out of memory. When generating D1 the HPC works fine, despite D1 being a larger array than D2. Why is that the case?
These are complementary functions used above:
function indices = pairIndices(n)
[y, x] = find(tril(logical(ones(n)), -1)); %#ok<LOGL>
indices = [x, y];
end
function [R , C] = myind2ind(ii, N)
jj = N * (N - 1) / 2 + 1 - ii;
r = (1 + sqrt(8 * jj)) / 2;
R = N -floor(r);
idx_first = (floor(r + 1) .* floor(r)) / 2;
C = idx_first-jj + R + 1;
end

Related

Take a random draw of all possible pairs of indices in Matlab

Consider a Matlab matrix B which lists all possible unordered pairs (without repetitions) from [1 2 ... n]. For example, if n=4,
B=[1 2;
1 3;
1 4;
2 3;
2 4;
3 4]
Note that B has size n(n-1)/2 x 2
I want to take a random draw of m rows from B and store them in a matrix C. Continuing the example above, I could do that as
m=2;
C=B(randi([1 size(B,1)],m,1),:);
However, in my actual case, n=371293. Hence, I cannot create B and, then, run the code above to obtain C. This is because storing B would require a huge amount of memory.
Could you advise on how I could proceed to create C, without having to first store B? Comments on a different question suggest to
Draw at random m integers between 1 and n(n-1)/2.
I=randi([1 n*(n-1)/2],m,1);
Use ind2sub to obtain C.
Here, I'm struggling to implement the second step.
Thanks to the comments below, I wrote this
n=4;
m=10;
coord=NaN(m,2);
R= randi([1 n^2],m,1);
for i=1:m
[cr, cc]=ind2sub([n,n],R(i));
if cr>cc
coord(i,1)=cc;
coord(i,2)=cr;
elseif cr<cc
coord(i,1)=cr;
coord(i,2)=cc;
end
end
coord(any(isnan(coord),2),:) = []; %delete NaN rows from coord
I guess there are more efficient ways to implement the same thing.
You can use the function named myind2ind in this post to take random rows of all possible unordered pairs without generating all of them.
function [R , C] = myind2ind(ii, N)
jj = N * (N - 1) / 2 + 1 - ii;
r = (1 + sqrt(8 * jj)) / 2;
R = N -floor(r);
idx_first = (floor(r + 1) .* floor(r)) / 2;
C = idx_first-jj + R + 1;
end
I=randi([1 n*(n-1)/2],m,1);
[C1 C2] = myind2ind (I, n);
If you look at the odds, for i=1:n-1, the number of combinations where the first value is equal to i is (n-i) and the total number of cominations is n*(n-1)/2. You can use this law to generate the first column of C. The values of the second column of C can then be generated randomly as integers uniformly distributed in the range [i+1, n]. Here is a code that performs the desired tasks:
clc; clear all; close all;
% Parameters
n = 371293; m = 10;
% Generation of C
R = rand(m,1);
C = zeros(m,2);
s = 0;
t = n*(n-1)/2;
for i=1:n-1
if (i<n-1)
ind_i = R>=s/t & R<(s+n-i)/t;
else % To avoid rounding errors for n>>1, we impose (s+n-i)=t at the last iteration (R<(s+n-i)/t=1 always true)
ind_i = R>=s/t;
end
C(ind_i,1) = i;
C(ind_i,2) = randi([i+1,n],sum(ind_i),1);
s = s+n-i;
end
% Display
C
Output:
C =
84333 266452
46609 223000
176395 328914
84865 94391
104444 227034
221905 302546
227497 335959
188486 344305
164789 266497
153603 354932
Good luck!

MATLAB: efficient generating of block matrices using a block vector

Suppose
x = [x1;x2; ...; xn]
where each xi is a column vector with length l(i). We can set L = sum(l), the total length of x. I would like to generate 2 matrices based on x:
Let's call them A and B. For example, when x only as 2 blocks x1 and x2 then:
A = [x1*x1' zeros(l(1),l(2)); zeros(l(2),l(1)), x2*x2'];
B = [x1 zeros(l(1),1);
zeros(l(2),1), x2];
In the notation of the problem, A is always L by L and B is L by n. I can generate A and B given x using loops but it is tedious. Is there a clever (loop-free) way to generate A and B. I am using MATLAB 2018b but you can assume earlier version of MATLAB if necessary.
I think it is both short and fast:
B = x .* (repelem((1:numel(l)).',l)==(1:numel(l)));
A = B * B.';
If you have large data It is better to use sparse matrix:
B = sparse(1:numel(x), repelem(1:numel(l), l), x);
A = B * B.';
The following should work. In this case I do an inefficient conversion to cell arrays so there may be a more efficient implementation possible.
cuml = [0; cumsum(l(:))];
get_x = #(idx) x((1:l(idx))+cuml(idx));
x_cell = arrayfun(get_x, 1:numel(l), 'UniformOutput', false);
B = blkdiag(x_cell{:});
A = B*B';
Edit
After running some benchmarks I found a direct loop based implementation to be about twice as fast as the cell based approach above.
A = zeros(sum(l));
B = zeros(sum(l), numel(l));
prev = 0;
for idx = 1:numel(l)
xidx = (1:l(idx))+prev;
A(xidx, xidx) = x(xidx,1) * x(xidx,1)';
B(xidx, idx) = x(idx,1);
prev = prev + l(idx);
end
Here's an alternative approach:
s = repelem(1:numel(l), l).';
t = accumarray(s, x, [], #(x){x*x'});
A = blkdiag(t{:});
t = accumarray(s, x, [], #(x){x});
B = blkdiag(t{:});

Matlab: how can I perform row operations without brute-force for loop?

I need to do function that works like this :
N1 = size(X,1);
N2 = size(Xtrain,1);
Dist = zeros(N1,N2);
for i=1:N1
for j=1:N2
Dist(i,j)=D-sum(X(i,:)==Xtrain(j,:));
end
end
(X and Xtrain are sparse logical matrixes)
It works fine and passes the tests, but I believe it's not very optimal and well-written solution.
How can I improve that function using some built Matlab functions? I'm absolutely new to Matlab, so I don't know if there really is an opportunity to make it better somehow.
You wanted to learn about vectorization, here some code to study comparing different implementations of this pair-wise distance.
First we build two binary matrices as input (where each row is an instance):
m = 5;
n = 4;
p = 3;
A = double(rand(m,p) > 0.5);
B = double(rand(n,p) > 0.5);
1. double-loop over each pair of instances
D0 = zeros(m,n);
for i=1:m
for j=1:n
D0(i,j) = sum(A(i,:) ~= B(j,:)) / p;
end
end
2. PDIST2
D1 = pdist2(A, B, 'hamming');
3. single-loop over each instance against all other instances
D2 = zeros(m,n);
for i=1:n
D2(:,i) = sum(bsxfun(#ne, A, B(i,:)), 2) ./ p;
end
4. vectorized with grid indexing, all against all
D3 = zeros(m,n);
[x,y] = ndgrid(1:m,1:n);
D3(:) = sum(A(x(:),:) ~= B(y(:),:), 2) ./ p;
5. vectorized in third dimension, all against all
D4 = sum(bsxfun(#ne, A, reshape(B.',[1 p n])), 2) ./ p;
D4 = permute(D4, [1 3 2]);
Finally we compare all methods are equal
assert(isequal(D0,D1,D2,D3,D4))

Correlation coefficients between two matrices to find intercorrelation

I am trying to calculate Pearson coefficients between all pair combinations of my variables of all my samples.
Say i have an m*n matrix where m are the variables and n are the samples
i want to calculate for each variable of my data what is the correlation to every other variable.
So, i managed to do that with nested loops:
X = rand[1000 100];
for i = 1:1000
base = X(i, :);
for j = 1:1000
target = X(j, :);
correlation = corrcoef(base, target);
correlation = correlation(2, 1);
corData(1, j) = correlation
end
totalCor(i, :) = corData
end
and it works, but takes too much time to run
I am trying to find a way to run the corrcoef function on a row basis, meaning maybe to create an additional matrix with repmat of the base values and correlate to the X data using some FUN function.
Could not figure out how to use the fun with inputs from to arrays, running between individuals lines/columns
help will be appreciated
This post involves a bit of hacking, so bear with it!
Stage #0 To start off, we have -
for i = 1:N
base = X(i, :);
for j = 1:N
target = X(j, :);
correlation = corrcoef(base, target);
correlation = correlation(2, 1)
corData(1, j) = correlation;
end
end
Stage #1 From the documentation of corrcoef in its source code :
If C is the covariance matrix, C = COV(X), then CORRCOEF(X) is the
matrix whose (i,j)'th element is : C(i,j)/SQRT(C(i,i)*C(j,j)).
After hacking into the code of covariance, we see that for the default case of one input, the covariance formula is simply -
[m,n] = size(x);
xc = bsxfun(#minus,x,sum(x,1)/m);
xy = (xc' * xc) / (m-1);
Thus, mixing the two definitions and putting them into the problem at hand, we have -
m = size(X,2);
for i = 1:N
base = X(i, :);
for j = 1:N
target = X(j, :);
BT = [base(:) target(:)];
xc = bsxfun(#minus,BT,sum(BT,1)/m);
C = (xc' * xc) / (m-1); %//'
corData = C(2,1)/sqrt(C(2,2)*C(1,1))
end
end
Stage #2 This is the final stage where we use the real fun aka bsxfun to kill all loops, like so -
%// Broadcasted subtract of each row by the average of it.
%// This corresponds to "xc = bsxfun(#minus,BT,sum(BT,1)/m)"
p1 = bsxfun(#minus,X,mean(X,2));
%// Get pairs of rows from X and get the dot product.
%// Thus, a total of "N x N" such products would be obtained.
p2 = sum(bsxfun(#times,permute(p1,[1 3 2]),permute(p1,[3 1 2])),3);
%// Scale them down by "size(X,2)-1".
%// This was for the part : "C = (xc' * xc) / (m-1)".
p3 = p2/(size(X,2)-1);
%// "C(2,2)" and "C(1,1)" are diagonal elements from "p3", so store them.
dp3 = diag(p3);
%// Get "sqrt(C(2,2)*C(1,1))" by broadcasting elementwise multiplication
%// of "dp3". Finally do elementwise division of "p3" by it.
totalCor_out = p3./sqrt(bsxfun(#times,dp3,dp3.'));
Benchmarking
This section compares the original approach against the proposed one and also verifies the output. Here's the benchmarking code -
disp('---------- With original approach')
tic
X = rand(1000,100);
corData = zeros(1,1000);
totalCor = zeros(1000,1000);
for i = 1:1000
base = X(i, :);
for j = 1:1000
target = X(j, :);
correlation = corrcoef(base, target);
correlation = correlation(2, 1);
corData(1, j) = correlation;
end
totalCor(i, :) = corData;
end
toc
disp('---------- With the real fun aka BSXFUN')
tic
p1 = bsxfun(#minus,X,mean(X,2));
p2 = sum(bsxfun(#times,permute(p1,[1 3 2]),permute(p1,[3 1 2])),3);
p3 = p2/(size(X,2)-1);
dp3 = diag(p3);
totalCor_out = p3./sqrt(bsxfun(#times,dp3,dp3.')); %//'
toc
error_val = max(abs(totalCor(:)-totalCor_out(:)))
Output -
---------- With original approach
Elapsed time is 186.501746 seconds.
---------- With the real fun aka BSXFUN
Elapsed time is 1.423448 seconds.
error_val =
4.996e-16

Computing Mahalanobis Distance Between Set of Points and Set of Reference Points

I have an n x p matrix - mX which is composed of n points in R^p.
I have another m x p matrix - mY which is composed of m reference points in R^p.
I would like to create an n x m matrix - mD which is the Mahalanobis Distance matrix.
D(i, j) means the Mahalanobis Distance between point i in mX, mX(i, :) and point j in mY, mY(j, :).
Namely, is computes the following:
mD(i, j) = (mX(i, :) - mY(j, :)) * inv(mC) * (mX(i, :) - mY(j, :)).';
Where mC is the given Mahalanobis Distance PSD Matrix.
It is easy to be done in a loop, is there a way to vectorize it?
Namely, is the a function which its inputs are mX, mY and mC and its output is mD and fully vectorized without using any MATLAB toolbox?
Thank You.
Approach #1
Assuming infinite resources, here's one vectorized solution using bsxfun and matrix-multiplication -
A = reshape(bsxfun(#minus,permute(mX,[1 3 2]),permute(mY,[3 1 2])),[],p);
out = reshape(diag(A*inv(mC)*A.'),n,m);
Approach #2
Here's a comprise solution trying to reduce the loop complexity -
A = reshape(bsxfun(#minus,permute(mX,[1 3 2]),permute(mY,[3 1 2])),[],p);
imC = inv(mC);
out = zeros(n*m,1);
for ii = 1:n*m
out(ii) = A(ii,:)*imC*A(ii,:).';
end
out = reshape(out,n,m);
Sample run -
>> n = 3; m = 4; p = 5;
mX = rand(n,p);
mY = rand(m,p);
mC = rand(p,p);
imC = inv(mC);
>> %// Original solution
for i = 1:n
for j = 1:m
mD(i, j) = (mX(i, :) - mY(j, :)) * inv(mC) * (mX(i, :) - mY(j, :)).'; %//'
end
end
>> mD
mD =
-8.4256 10.032 2.8929 7.1762
-44.748 -4.3851 -13.645 -9.6702
-4.5297 3.2928 0.11132 2.5998
>> %// Approach #1
A = reshape(bsxfun(#minus,permute(mX,[1 3 2]),permute(mY,[3 1 2])),[],p);
out = reshape(diag(A*inv(mC)*A.'),n,m); %//'
>> out
out =
-8.4256 10.032 2.8929 7.1762
-44.748 -4.3851 -13.645 -9.6702
-4.5297 3.2928 0.11132 2.5998
>> %// Approach #2
A = reshape(bsxfun(#minus,permute(mX,[1 3 2]),permute(mY,[3 1 2])),[],p);
imC = inv(mC);
out1 = zeros(n*m,1);
for ii = 1:n*m
out1(ii) = A(ii,:)*imC*A(ii,:).'; %//'
end
out1 = reshape(out1,n,m);
>> out1
out1 =
-8.4256 10.032 2.8929 7.1762
-44.748 -4.3851 -13.645 -9.6702
-4.5297 3.2928 0.11132 2.5998
Instead if you had :
mD(j, i) = (mX(i, :) - mY(j, :)) * inv(mC) * (mX(i, :) - mY(j, :)).';
The solutions would translate to the versions listed next.
Approach #1
A = reshape(bsxfun(#minus,permute(mY,[1 3 2]),permute(mX,[3 1 2])),[],p);
out = reshape(diag(A*inv(mC)*A.'),m,n);
Approach #2
A = reshape(bsxfun(#minus,permute(mY,[1 3 2]),permute(mX,[3 1 2])),[],p);
imC = inv(mC);
out1 = zeros(m*n,1);
for i = 1:n*m
out(i) = A(i,:)*imC*A(i,:).'; %//'
end
out = reshape(out,m,n);
Sample run -
>> n = 3; m = 4; p = 5;
mX = rand(n,p); mY = rand(m,p); mC = rand(p,p); imC = inv(mC);
>> %// Original solution
for i = 1:n
for j = 1:m
mD(j, i) = (mX(i, :) - mY(j, :)) * inv(mC) * (mX(i, :) - mY(j, :)).'; %//'
end
end
>> mD
mD =
0.81755 0.33205 0.82254
1.7086 1.3363 2.4209
0.36495 0.78394 -0.33097
0.17359 0.3889 -1.0624
>> %// Approach #1
A = reshape(bsxfun(#minus,permute(mY,[1 3 2]),permute(mX,[3 1 2])),[],p);
out = reshape(diag(A*inv(mC)*A.'),m,n); %//'
>> out
out =
0.81755 0.33205 0.82254
1.7086 1.3363 2.4209
0.36495 0.78394 -0.33097
0.17359 0.3889 -1.0624
>> %// Approach #2
A = reshape(bsxfun(#minus,permute(mY,[1 3 2]),permute(mX,[3 1 2])),[],p);
imC = inv(mC);
out1 = zeros(m*n,1);
for i = 1:n*m
out1(i) = A(i,:)*imC*A(i,:).'; %//'
end
out1 = reshape(out1,m,n);
>> out1
out1 =
0.81755 0.33205 0.82254
1.7086 1.3363 2.4209
0.36495 0.78394 -0.33097
0.17359 0.3889 -1.0624
Here is one solution that eliminates one loop
function d = mahalanobis(mX, mY)
n = size(mX, 2);
m = size(mY, 2);
data = [mX, mY];
mc = cov(transpose(data));
dist = zeros(n,m);
for i = 1 : n
diff = repmat(mX(:,i), 1, m) - mY;
dist(i,:) = sum((mc\diff).*diff , 1);
end
d = sqrt(dist);
end
You would invoke it as:
d = mahalanobis(transpose(X),transpose(Y))
Reduce to L2
It seems that Mahalanobis Distance can be reduced to ordinary L2 distance if you are allowed to preprocess matrix mC and you are not afraid of numerical differences.
First of all, compute Cholesky decomposition of mC:
mR = chol(mC) % C = R^t * R, where R is upper-triangular
Now we can use these factors to reformulate Mahalanobis Distance:
(Xi-Yj) * inv(C) * (Xi-Yj)^t = || (Xi-Yj) inv(R) ||^2 = ||TXi - TYj||^2
where: TXi = Xi * inv(R)
TYj = Yj * inv(R)
So the idea is to transform points Xi, Yj to TXi, TYj first, and then compute euclidean distances between them. Here is the algorithm outline:
Compute mR - Cholesky factor of covariance matrix mC (takes O(p^3) time).
Invert triangular matrix mR (takes O(p^3) time).
Multiply both mX and mY by inv(mR) on the right (takes O(p^2 (m+n)) time).
Compute squared L2 distances between pairs of points (takes O(m n p) time).
Total time is O(m n p + (m + n) p^2 + p^3) versus original O(m n p^2). It should work faster when 1 << p << n,m. In such case step 4 would takes most of the time and should be vectorized.
Vectorization
I have little experience of MATLAB, but quite a lot of SIMD vectorization on x86 CPUs. In raw computations, it would be enough to vectorize along one sufficiently large array dimension, and make trivial loops for the other dimensions.
If you expect p to be large enough, it may probably be OK to vectorize along coordinates of points, and make two nested loops for i <= n and j <= m. That's similar to what #Daniel posted.
If p is not sufficiently large, you can vectorize along one of the point sequences instead. This would be similar to solution posted by #dpmcmlxxvi: you have to subtract single row of one matrix from all the rows of the second matrix, then compute squared norms of the resulting rows. Repeat n times (
or m times).
As for me, full vectorization (which means rewriting with matrix operations instead of loops in MATLAB) does not sound like a clever performance goal. Most likely partially vectorized solutions would be optimally fast.
I came to the conclusion that vectorizing this problem is not efficient. My best idea for vectorizing this problem would require m x n x p x p working memory, at least if everything is processed at once. This means with n=m=p=152 the code would already require 4GB Ram. At these dimensions, my system can run the loop in less than a second:
mD=zeros(size(mX,1),size(mY,1));
ImC=inv(mC);
for i=1:size(mX,1)
for j=1:size(mY,1)
d=mX(i, :) - mY(j, :);
mD(i, j) = (d) * ImC * (d).';
end
end