Vectorizing sums of different diagonals in a matrix - matlab

I want to vectorize the following MATLAB code. I think it must be simple but I'm finding it confusing nevertheless.
r = some constant less than m or n
[m,n] = size(C);
S = zeros(m-r,n-r);
for i=1:m-r+1
for j=1:n-r+1
S(i,j) = sum(diag(C(i:i+r-1,j:j+r-1)));
end
end
The code calculates a table of scores, S, for a dynamic programming algorithm, from another score table, C.
The diagonal summing is to generate scores for individual pieces of the data used to generate C, for all possible pieces (of size r).
Thanks in advance for any answers! Sorry if this one should be obvious...
Note
The built-in conv2 turned out to be faster than convnfft, because my eye(r) is quite small ( 5 <= r <= 20 ). convnfft.m states that r should be > 20 for any benefit to manifest.

If I understand correctly, you're trying to calculate the diagonal sum of every subarray of C, where you have removed the last row and column of C (if you should not remove the row/col, you need to loop to m-r+1, and you need to pass the entire array C to the function in my solution below).
You can do this operation via a convolution, like so:
S = conv2(C(1:end-1,1:end-1),eye(r),'valid');
If C and r are large, you may want to have a look at CONVNFFT from the Matlab File Exchange to speed up calculations.

Based on the idea of JS, and as Jonas pointed out in the comments, this can be done in two lines using IM2COL with some array manipulation:
B = im2col(C, [r r], 'sliding');
S = reshape( sum(B(1:r+1:end,:)), size(C)-r+1 );
Basically B contains the elements of all sliding blocks of size r-by-r over the matrix C. Then we take the elements on the diagonal of each of these blocks B(1:r+1:end,:), compute their sum, and reshape the result to the expected size.
Comparing this to the convolution-based solution by Jonas, this does not perform any matrix multiplication, only indexing...

I would think you might need to rearrange C into a 3D matrix before summing it along one of the dimensions. I'll post with an answer shortly.
EDIT
I didn't manage to find a way to vectorise it cleanly, but I did find the function accumarray, which might be of some help. I'll look at it in more detail when I am home.
EDIT#2
Found a simpler solution by using linear indexing, but this could be memory-intensive.
At C(1,1), the indexes we want to sum are 1+[0, m+1, 2*m+2, 3*m+3, 4*m+4, ... ], or (0:r-1)+(0:m:(r-1)*m)
sum_ind = (0:r-1)+(0:m:(r-1)*m);
create S_offset, an (m-r) by (n-r) by r matrix, such that S_offset(:,:,1) = 0, S_offset(:,:,2) = m+1, S_offset(:,:,3) = 2*m+2, and so on.
S_offset = permute(repmat( sum_ind, [m-r, 1, n-r] ), [1, 3, 2]);
create S_base, a matrix of base array addresses from which the offset will be calculated.
S_base = reshape(1:m*n,[m n]);
S_base = repmat(S_base(1:m-r,1:n-r), [1, 1, r]);
Finally, use S_base+S_offset to address the values of C.
S = sum(C(S_base+S_offset), 3);
You can, of course, use bsxfun and other methods to make it more efficient; here I chose to lay it out for clarity. I have yet to benchmark this to see how it compares with the double-loop method though; I need to head home for dinner first!

Is this what you're looking for? This function adds the diagonals and puts them into a vector similar to how the function 'sum' adds up all of the columns in a matrix and puts them into a vector.
function [diagSum] = diagSumCalc(squareMatrix, LLUR0_ULLR1)
%
% Input: squareMatrix: A square matrix.
% LLUR0_ULLR1: LowerLeft to UpperRight addition = 0
% UpperLeft to LowerRight addition = 1
%
% Output: diagSum: A vector of the sum of the diagnols of the matrix.
%
% Example:
%
% >> squareMatrix = [1 2 3;
% 4 5 6;
% 7 8 9];
%
% >> diagSum = diagSumCalc(squareMatrix, 0);
%
% diagSum =
%
% 1 6 15 14 9
%
% >> diagSum = diagSumCalc(squareMatrix, 1);
%
% diagSum =
%
% 7 12 15 8 3
%
% Written by M. Phillips
% Oct. 16th, 2013
% MIT Open Source Copywrite
% Contact mphillips#hmc.edu fmi.
%
if (nargin < 2)
disp('Error on input. Needs two inputs.');
return;
end
if (LLUR0_ULLR1 ~= 0 && LLUR0_ULLR1~= 1)
disp('Error on input. Only accepts 0 or 1 as input for second condition.');
return;
end
[M, N] = size(squareMatrix);
if (M ~= N)
disp('Error on input. Only accepts a square matrix as input.');
return;
end
diagSum = zeros(1, M+N-1);
if LLUR0_ULLR1 == 1
squareMatrix = rot90(squareMatrix, -1);
end
for i = 1:length(diagSum)
if i <= M
countUp = 1;
countDown = i;
while countDown ~= 0
diagSum(i) = squareMatrix(countUp, countDown) + diagSum(i);
countUp = countUp+1;
countDown = countDown-1;
end
end
if i > M
countUp = i-M+1;
countDown = M;
while countUp ~= M+1
diagSum(i) = squareMatrix(countUp, countDown) + diagSum(i);
countUp = countUp+1;
countDown = countDown-1;
end
end
end
Cheers

Related

How to programmatically compute this summation

I want to compute the above summation, for a given 'x'. The summation is to be carried out over a block of lengths specified by an array , for example block_length = [5 4 3]. The summation is carried as follows: from -5 to 5 across one dimension, -4 to 4 in the second dimension and -3 to 3 in the last dimension.
The pseudo code will be something like this:
sum = 0;
for i = -5:5
for j = -4:4
for k = -3:3
vec = [i j k];
tv = vec * vec';
sum = sum + 1/(1+tv)*cos(2*pi*x*vec'));
end
end
end
The problem is that I want to find the sum when the number of dimensions are not known ahead of time, using some kind of variable nested loops hopefully. Matlab uses combvec, but it returns all possible combinations of vectors, which is not required as we only compute the sum. When there are many dimensions, combvec returning all combinations is not feasible memory wise.
Appreciate any ideas towards solutions.
PS: I want to do this at high number of dimensions, for example 650, as in machine learning.
Based on https://www.mathworks.com/matlabcentral/answers/345551-function-with-varying-number-of-for-loops I came up with the following code (I haven't tested it for very large number of indices!):
function sum = fun(x, block_length)
sum = 0;
n = numel(block_length); % Number of loops
vec = -ones(1, n) .* block_length; % Index vector
ready = false;
while ~ready
tv = vec * vec';
sum = sum + 1/(1+tv)*cos(2*pi*x*vec');
% Update the index vector:
ready = true; % Assume that the WHILE loop is ready
for k = 1:n
vec(k) = vec(k) + 1;
if vec(k) <= block_length(k)
ready = false;
break; % v(k) increased successfully, leave "for k" loop
end
vec(k) = -1 * block_length(k); % v(k) reached the limit, reset it
end
end
end
where x and block_length should be both 1-x-n vectors.
The idea is that, instead of using explicitly nested loops, we use a vector of indices.
How good/efficient is this when tackling the suggested use case where block_length can have 650 elements? Not much! Here's a "quick" test using merely 16 dimensions and a [-1, 1] range for the indices:
N = 16; tic; in = 0.1 * ones(1, N); sum = fun(in, ones(size(in))), toc;
which yields an elapsed time of 12.7 seconds on my laptop.

Take a random draw of all possible pairs of indices in Matlab

Consider a Matlab matrix B which lists all possible unordered pairs (without repetitions) from [1 2 ... n]. For example, if n=4,
B=[1 2;
1 3;
1 4;
2 3;
2 4;
3 4]
Note that B has size n(n-1)/2 x 2
I want to take a random draw of m rows from B and store them in a matrix C. Continuing the example above, I could do that as
m=2;
C=B(randi([1 size(B,1)],m,1),:);
However, in my actual case, n=371293. Hence, I cannot create B and, then, run the code above to obtain C. This is because storing B would require a huge amount of memory.
Could you advise on how I could proceed to create C, without having to first store B? Comments on a different question suggest to
Draw at random m integers between 1 and n(n-1)/2.
I=randi([1 n*(n-1)/2],m,1);
Use ind2sub to obtain C.
Here, I'm struggling to implement the second step.
Thanks to the comments below, I wrote this
n=4;
m=10;
coord=NaN(m,2);
R= randi([1 n^2],m,1);
for i=1:m
[cr, cc]=ind2sub([n,n],R(i));
if cr>cc
coord(i,1)=cc;
coord(i,2)=cr;
elseif cr<cc
coord(i,1)=cr;
coord(i,2)=cc;
end
end
coord(any(isnan(coord),2),:) = []; %delete NaN rows from coord
I guess there are more efficient ways to implement the same thing.
You can use the function named myind2ind in this post to take random rows of all possible unordered pairs without generating all of them.
function [R , C] = myind2ind(ii, N)
jj = N * (N - 1) / 2 + 1 - ii;
r = (1 + sqrt(8 * jj)) / 2;
R = N -floor(r);
idx_first = (floor(r + 1) .* floor(r)) / 2;
C = idx_first-jj + R + 1;
end
I=randi([1 n*(n-1)/2],m,1);
[C1 C2] = myind2ind (I, n);
If you look at the odds, for i=1:n-1, the number of combinations where the first value is equal to i is (n-i) and the total number of cominations is n*(n-1)/2. You can use this law to generate the first column of C. The values of the second column of C can then be generated randomly as integers uniformly distributed in the range [i+1, n]. Here is a code that performs the desired tasks:
clc; clear all; close all;
% Parameters
n = 371293; m = 10;
% Generation of C
R = rand(m,1);
C = zeros(m,2);
s = 0;
t = n*(n-1)/2;
for i=1:n-1
if (i<n-1)
ind_i = R>=s/t & R<(s+n-i)/t;
else % To avoid rounding errors for n>>1, we impose (s+n-i)=t at the last iteration (R<(s+n-i)/t=1 always true)
ind_i = R>=s/t;
end
C(ind_i,1) = i;
C(ind_i,2) = randi([i+1,n],sum(ind_i),1);
s = s+n-i;
end
% Display
C
Output:
C =
84333 266452
46609 223000
176395 328914
84865 94391
104444 227034
221905 302546
227497 335959
188486 344305
164789 266497
153603 354932
Good luck!

Matlab function for cumulative power

Is there a function in MATLAB that generates the following matrix for a given scalar r:
1 r r^2 r^3 ... r^n
0 1 r r^2 ... r^(n-1)
0 0 1 r ... r^(n-2)
...
0 0 0 0 ... 1
where each row behaves somewhat like a power analog of the CUMSUM function?
You can compute each term directly using implicit expansion and element-wise power, and then apply triu:
n = 5; % size
r = 2; % base
result = triu(r.^max((1:n)-(1:n).',0));
Or, maybe a little faster because it doesn't compute unwanted powers:
n = 5; % size
r = 2; % base
t = (1:n)-(1:n).';
u = find(t>=0);
t = t(u);
result = zeros(n);
result(u) = r.^t;
Using cumprod and triu:
% parameters
n = 5;
r = 2;
% Create a square matrix filled with 1:
A = ones(n);
% Assign the upper triangular part shifted by one with r
A(triu(A,1)==1)=r;
% cumprod along the second dimension and get only the upper triangular part
A = triu(cumprod(A,2))
Well, cumsum accumulates the sum of a vector but you are asking for a specially design matrix, so the comparison is a bit problematic....
Anyway, it might be that there is a function for this if this is a common special case triangular matrix (my mathematical knowledge is limited here, sorry), but we can also build it quite easily (and efficiently=) ):
N = 10;
r = 2;
% allocate arry
ary = ones(1,N);
% initialize array
ary(2) = r;
for i = 3:N
ary(i) = ary(i-1)*r;
end
% build matrix i.e. copy the array
M = eye(N);
for i = 1:N
M(i,i:end) = ary(1:end-i+1);
end
This assumes that you want to have a matrix of size NxN and r is the value that you want calculate the power of.
FIX: a previous version stated in line 13 M(i,i:end) = ary(i:end);, but the assignment needs to start always at the first position of the ary

How can I vectorize the loops of this function in Octave?

I want to be able to vectorize the for-loops of this function to then be able to parallelize it in octave. Can these for-loops be vectorized? Thank you very much in advance!
I attach the code of the function commenting on the start and end of each for-loop and if-else.
function [par]=pem_v(tsm,pr)
% tsm and pr are arrays of N by n. % par is an array of N by 8
tss=[27:0.5:32];
tc=[20:0.01:29];
N=size(tsm,1);
% main-loop
for ii=1:N
% I extract the rows in each loop because each one represents a sample
sst=tsm(ii,:); sst=sst'; %then I convert each sample to column vectors
pre=pr(ii,:); pre=pre';
% main-condition
if isnan(nanmean(sst))==1;
par(ii,1:8)=NaN;
else
% first sub-loop
for k=1:length(tss);
idxx=find(sst>=tss(k)-0.25 & sst<=tss(k)+0.25);
out(k)=prctile(pre(idxx),90);
end
% end first sub-loop
tp90=tss(find(max(out)==out));
% second sub-loop
for j=1:length(tc)
cond1=find(sst>=tc(j) & sst<=tp90);
cond2=find(sst>=tp90);
pem=zeros(length(sst),1);
A=[sst(cond1),ones(length(cond1),1)];
B=regress(pre(cond1),A);
pt90=B(1)*(tp90-tc(j));
AA=[(sst(cond2)-tp90)];
BB=regress(pre(cond2)-pt90,AA);
pem(cond1)=max(0,B(1)*(sst(cond1)-tc(j)));
pem(cond2)=max(0,(BB(1)*(sst(cond2)-tp90))+pt90);
clear A B AA BB;
E(j)=sqrt(nansum((pem-pre).^2)/length(pre));
clear pem;
end
% end second sub-loop
tcc=tc(find(E==min(E)));
% sub-condition
if(isempty(tcc)==1);
par(ii,1:9)=NaN;
else
cond1=find(sst>=tcc & sst<=tp90);
cond2=find(sst>=tp90);
pem=zeros(length(sst),1);
A=[sst(cond1),ones(length(cond1),1)];
B=regress(pre(cond1),A);
pt90=B(1)*(tp90-tcc);
AA=[sst(cond2)-tp90];
BB=regress(pre(cond2)-pt90,AA);
pem(cond1)=max(0,B(1)*(sst(cond1)-tcc));
pem(cond2)=max(0,(BB(1)*(sst(cond2)-tp90))+pt90);
RMSE=sqrt(nansum((pem-pre).^2)/length(pre));
% outputs
par(ii,1)=tcc;
par(ii,2)=tp90;
par(ii,3)=B(1);
par(ii,4)=BB(1);
par(ii,5)=RMSE;
par(ii,6)=nanmean(sst);
par(ii,7)=nanmean(pre);
par(ii,8)=nanmean(pem);
end
% end sub-condition
clear pem pre sst RMSE BB B tp90 tcc
end
% end main-condition
end
% end main-loop
You haven't given any example inputs, so I've created some like so:
N = 5; n = 800;
tsm = rand(N,n)*5+27; pr = rand(N,n);
Then, before you even consider vectorising your code, you should keep 4 things in mind...
Avoid calulating the same thing (like the size of a vector) every loop, instead do it before looping
Pre-allocate arrays where possible (declare them as zeros/NaNs etc)
Don't use find to convert logical indices into linear indices, there is no need and it will slow down your code
Don't repeatedly use clear, especially many times within loops. It is slow! Instead, use pre-allocation to ensure the variables are as you expect each loop.
Using the above random inputs, and taking account of these 4 things, the below code is ~65% quicker than your code. Note: this is without even doing any vectorising!
function [par]=pem_v(tsm,pr)
% tsm and pr are arrays of N by n.
% par is an array of N by 8
tss=[27:0.5:32];
tc=[20:0.01:29];
N=size(tsm,1);
% Transpose once here instead of every loop
tsm = tsm';
pr = pr';
% Pre-allocate memory for output 'par'
par = NaN(N, 8);
% Don't compute these every loop, do it before the loop.
% numel simpler than length for vectors, and size is clearer still
ntss = numel(tss);
nsst = size(tsm,1);
ntc = numel(tc);
npr = size(pr, 1);
for ii=1:N
% Extract the columns in each loop because each one represents a sample
sst=tsm(:,ii);
pre=pr(:,ii);
% main-condition. Previously isnan(nanmean(sst))==1, but that's only true if all(isnan(sst))
% We don't need to assign par(ii,1:8)=NaN since we initialised par to a matrix of NaNs
if ~all(isnan(sst));
% first sub-loop, initialise 'out' first
out = zeros(1, ntss);
for k=1:ntss;
% Don't use FIND on an indexing vector. Use the logical index raw, it's quicker
idxx = (sst>=tss(k)-0.25 & sst<=tss(k)+0.25);
% We need a check that some values of idxx are true, otherwise prctile will error.
if nnz(idxx) > 0
out(k) = prctile(pre(idxx), 90);
end
end
% Again, no need for FIND, just reduces speed. This is a theme...
tp90=tss(max(out)==out);
for jj=1:ntc
cond1 = (sst>=tc(jj) & sst<=tp90);
cond2 = (sst>=tp90);
% Use nnz (numer of non-zero) instead of length, since cond1 is now a logical vector of all elements
A = [sst(cond1),ones(nnz(cond1),1)];
B = regress(pre(cond1), A);
pt90 = B(1)*(tp90-tc(jj));
AA = [(sst(cond2)-tp90)];
BB = regress(pre(cond2)-pt90,AA);
pem=zeros(nsst,1);
pem(cond1) = max(0, B(1)*(sst(cond1)-tc(jj)));
pem(cond2) = max(0, (BB(1)*(sst(cond2)-tp90))+pt90);
E(jj) = sqrt(nansum((pem-pre).^2)/npr);
end
tcc = tc(E==min(E));
if ~isempty(tcc);
cond1 = (sst>=tcc & sst<=tp90);
cond2 = (sst>=tp90);
A = [sst(cond1),ones(nnz(cond1),1)];
B = regress(pre(cond1),A);
pt90 = B(1)*(tp90-tcc);
AA = [sst(cond2)-tp90];
BB = regress(pre(cond2)-pt90,AA);
pem = zeros(length(sst),1);
pem(cond1) = max(0, B(1)*(sst(cond1)-tcc));
pem(cond2) = max(0, (BB(1)*(sst(cond2)-tp90))+pt90);
RMSE = sqrt(nansum((pem-pre).^2)/npr);
% Outputs, which we might as well assign all at once!
par(ii,:)=[tcc, tp90, B(1), BB(1), RMSE, ...
nanmean(sst), nanmean(pre), nanmean(pem)];
end
end
end

Calculating overlap in Mx2 and Nx2 matrices

I have two matrices A and B, both contain a list of event start and stop times:
A(i,1) = onset time of event i
A(i,2) = offset time of event i
B(j,1) = onset of event j
...
My goal is to get two lists of indecies aIdx and bIdx such that A(aIdx,:) and B(bIdx,:) contain the sets of events that are overlapping.
I've been scratching my head all day trying to figure this one out. Is there a quick, easy, matlaby way to do this?
I can do it using for loops but this seems kind of hacky for matlab:
aIdx = [];
bIdx = []
for i=1:size(A,1)
for j=i:size(B,1)
if overlap(A(i,:), B(j,:)) % overlap is defined elsewhere
aIdx(end+1) = i;
bIdx(end+1) = j;
end
end
end
Here's a zero loop solution:
overlap = #(x, y)y(:, 1) < x(:, 2) & y(:, 2) > x(:, 1)
[tmp1, tmp2] = meshgrid(1:size(A, 1), 1:size(B, 1));
M = reshape(overlap(A(tmp1, :), B(tmp2, :)), size(B, 1), [])';
[aIdx, bIdx] = find(M);
You can do it with one loop:
aIdx = false(size(A,1),1);
bIdx = false(size(B,1),1);
for k = 1:size(B,1)
ai = ( A(:,1) >= B(k,1) & A(:,1) <= B(k,2) ) | ...
( A(:,2) >= B(k,1) & A(:,2) <= B(k,2) );
if any(ai), bIdx(k) = true; end
aIdx = aIdx | ai;
end
There is a way to create a vectorized algorithm. (I wrote a similar function before, but cannot find it right now.) A simply workflow is to (1) combine both matrices, (2) create an index to indicate source of each event, (3) create a matrix indicating start and stop positions, (4) vectorized and sort, (5) find overlaps with diff, cumsum, or combination.
overlap_matrix = zeros(size(A,1),size(B,1))
for jj = 1:size(B,1)
overlap_matrix(:,jj) = (A(:,1) <= B(jj,1)).*(B(jj,1) <= A(:,2));
end
[r,c] = find(overlap_matrix)
% Now A(r(i),:) overlaps with B(c(i),:)
% Modify the above conditional if you want to also check
% whether events in A start in-between the events in B
% as I am only checking the first half of the conditional
% for simplicity.
Completely vectorized code without repmat or reshape (hopefully faster too). I have assumed that the function "overlap" can give a vector of 1's and 0's if complete pairs of A(req_indices,:) and B(req_indices,:) are fed to it. If overlap can return a vector output, then the vectorization can be performed as given below.
Let rows in A matrix be Ra and Rows in B matrix be Rb,
AA=1:Ra;
AAA=AA(ones(Rb,1),:);
AAAA=AAA(:); % all indices of A arranged in desired format, i.e. [11...1,22..2,33...3, ...., RaRa...Ra]'
BB=(1:Rb)';
BBB=BB(:,ones(Ra,1));
BBBB=BBB(:);% all indices of B arranged in desired format, i.e. [123...Rb, 123...Rb,....,123...Rb]'
% Use overlap function
Result_vector = overlap(A(AAAA,:), B(BBBB,:));
Result_vector_without_zeros = find(Result_vector);
aIdx = AAAA(Results_vector_without_zeros);
bIdx = BBBB(Results_vector_without_zeros);
DISADVANTAGE : TOO MUCH RAM CONSUMPTION FOR LARGER MATRICES