I have two matrices A and B, both contain a list of event start and stop times:
A(i,1) = onset time of event i
A(i,2) = offset time of event i
B(j,1) = onset of event j
...
My goal is to get two lists of indecies aIdx and bIdx such that A(aIdx,:) and B(bIdx,:) contain the sets of events that are overlapping.
I've been scratching my head all day trying to figure this one out. Is there a quick, easy, matlaby way to do this?
I can do it using for loops but this seems kind of hacky for matlab:
aIdx = [];
bIdx = []
for i=1:size(A,1)
for j=i:size(B,1)
if overlap(A(i,:), B(j,:)) % overlap is defined elsewhere
aIdx(end+1) = i;
bIdx(end+1) = j;
end
end
end
Here's a zero loop solution:
overlap = #(x, y)y(:, 1) < x(:, 2) & y(:, 2) > x(:, 1)
[tmp1, tmp2] = meshgrid(1:size(A, 1), 1:size(B, 1));
M = reshape(overlap(A(tmp1, :), B(tmp2, :)), size(B, 1), [])';
[aIdx, bIdx] = find(M);
You can do it with one loop:
aIdx = false(size(A,1),1);
bIdx = false(size(B,1),1);
for k = 1:size(B,1)
ai = ( A(:,1) >= B(k,1) & A(:,1) <= B(k,2) ) | ...
( A(:,2) >= B(k,1) & A(:,2) <= B(k,2) );
if any(ai), bIdx(k) = true; end
aIdx = aIdx | ai;
end
There is a way to create a vectorized algorithm. (I wrote a similar function before, but cannot find it right now.) A simply workflow is to (1) combine both matrices, (2) create an index to indicate source of each event, (3) create a matrix indicating start and stop positions, (4) vectorized and sort, (5) find overlaps with diff, cumsum, or combination.
overlap_matrix = zeros(size(A,1),size(B,1))
for jj = 1:size(B,1)
overlap_matrix(:,jj) = (A(:,1) <= B(jj,1)).*(B(jj,1) <= A(:,2));
end
[r,c] = find(overlap_matrix)
% Now A(r(i),:) overlaps with B(c(i),:)
% Modify the above conditional if you want to also check
% whether events in A start in-between the events in B
% as I am only checking the first half of the conditional
% for simplicity.
Completely vectorized code without repmat or reshape (hopefully faster too). I have assumed that the function "overlap" can give a vector of 1's and 0's if complete pairs of A(req_indices,:) and B(req_indices,:) are fed to it. If overlap can return a vector output, then the vectorization can be performed as given below.
Let rows in A matrix be Ra and Rows in B matrix be Rb,
AA=1:Ra;
AAA=AA(ones(Rb,1),:);
AAAA=AAA(:); % all indices of A arranged in desired format, i.e. [11...1,22..2,33...3, ...., RaRa...Ra]'
BB=(1:Rb)';
BBB=BB(:,ones(Ra,1));
BBBB=BBB(:);% all indices of B arranged in desired format, i.e. [123...Rb, 123...Rb,....,123...Rb]'
% Use overlap function
Result_vector = overlap(A(AAAA,:), B(BBBB,:));
Result_vector_without_zeros = find(Result_vector);
aIdx = AAAA(Results_vector_without_zeros);
bIdx = BBBB(Results_vector_without_zeros);
DISADVANTAGE : TOO MUCH RAM CONSUMPTION FOR LARGER MATRICES
Related
I want to be able to vectorize the for-loops of this function to then be able to parallelize it in octave. Can these for-loops be vectorized? Thank you very much in advance!
I attach the code of the function commenting on the start and end of each for-loop and if-else.
function [par]=pem_v(tsm,pr)
% tsm and pr are arrays of N by n. % par is an array of N by 8
tss=[27:0.5:32];
tc=[20:0.01:29];
N=size(tsm,1);
% main-loop
for ii=1:N
% I extract the rows in each loop because each one represents a sample
sst=tsm(ii,:); sst=sst'; %then I convert each sample to column vectors
pre=pr(ii,:); pre=pre';
% main-condition
if isnan(nanmean(sst))==1;
par(ii,1:8)=NaN;
else
% first sub-loop
for k=1:length(tss);
idxx=find(sst>=tss(k)-0.25 & sst<=tss(k)+0.25);
out(k)=prctile(pre(idxx),90);
end
% end first sub-loop
tp90=tss(find(max(out)==out));
% second sub-loop
for j=1:length(tc)
cond1=find(sst>=tc(j) & sst<=tp90);
cond2=find(sst>=tp90);
pem=zeros(length(sst),1);
A=[sst(cond1),ones(length(cond1),1)];
B=regress(pre(cond1),A);
pt90=B(1)*(tp90-tc(j));
AA=[(sst(cond2)-tp90)];
BB=regress(pre(cond2)-pt90,AA);
pem(cond1)=max(0,B(1)*(sst(cond1)-tc(j)));
pem(cond2)=max(0,(BB(1)*(sst(cond2)-tp90))+pt90);
clear A B AA BB;
E(j)=sqrt(nansum((pem-pre).^2)/length(pre));
clear pem;
end
% end second sub-loop
tcc=tc(find(E==min(E)));
% sub-condition
if(isempty(tcc)==1);
par(ii,1:9)=NaN;
else
cond1=find(sst>=tcc & sst<=tp90);
cond2=find(sst>=tp90);
pem=zeros(length(sst),1);
A=[sst(cond1),ones(length(cond1),1)];
B=regress(pre(cond1),A);
pt90=B(1)*(tp90-tcc);
AA=[sst(cond2)-tp90];
BB=regress(pre(cond2)-pt90,AA);
pem(cond1)=max(0,B(1)*(sst(cond1)-tcc));
pem(cond2)=max(0,(BB(1)*(sst(cond2)-tp90))+pt90);
RMSE=sqrt(nansum((pem-pre).^2)/length(pre));
% outputs
par(ii,1)=tcc;
par(ii,2)=tp90;
par(ii,3)=B(1);
par(ii,4)=BB(1);
par(ii,5)=RMSE;
par(ii,6)=nanmean(sst);
par(ii,7)=nanmean(pre);
par(ii,8)=nanmean(pem);
end
% end sub-condition
clear pem pre sst RMSE BB B tp90 tcc
end
% end main-condition
end
% end main-loop
You haven't given any example inputs, so I've created some like so:
N = 5; n = 800;
tsm = rand(N,n)*5+27; pr = rand(N,n);
Then, before you even consider vectorising your code, you should keep 4 things in mind...
Avoid calulating the same thing (like the size of a vector) every loop, instead do it before looping
Pre-allocate arrays where possible (declare them as zeros/NaNs etc)
Don't use find to convert logical indices into linear indices, there is no need and it will slow down your code
Don't repeatedly use clear, especially many times within loops. It is slow! Instead, use pre-allocation to ensure the variables are as you expect each loop.
Using the above random inputs, and taking account of these 4 things, the below code is ~65% quicker than your code. Note: this is without even doing any vectorising!
function [par]=pem_v(tsm,pr)
% tsm and pr are arrays of N by n.
% par is an array of N by 8
tss=[27:0.5:32];
tc=[20:0.01:29];
N=size(tsm,1);
% Transpose once here instead of every loop
tsm = tsm';
pr = pr';
% Pre-allocate memory for output 'par'
par = NaN(N, 8);
% Don't compute these every loop, do it before the loop.
% numel simpler than length for vectors, and size is clearer still
ntss = numel(tss);
nsst = size(tsm,1);
ntc = numel(tc);
npr = size(pr, 1);
for ii=1:N
% Extract the columns in each loop because each one represents a sample
sst=tsm(:,ii);
pre=pr(:,ii);
% main-condition. Previously isnan(nanmean(sst))==1, but that's only true if all(isnan(sst))
% We don't need to assign par(ii,1:8)=NaN since we initialised par to a matrix of NaNs
if ~all(isnan(sst));
% first sub-loop, initialise 'out' first
out = zeros(1, ntss);
for k=1:ntss;
% Don't use FIND on an indexing vector. Use the logical index raw, it's quicker
idxx = (sst>=tss(k)-0.25 & sst<=tss(k)+0.25);
% We need a check that some values of idxx are true, otherwise prctile will error.
if nnz(idxx) > 0
out(k) = prctile(pre(idxx), 90);
end
end
% Again, no need for FIND, just reduces speed. This is a theme...
tp90=tss(max(out)==out);
for jj=1:ntc
cond1 = (sst>=tc(jj) & sst<=tp90);
cond2 = (sst>=tp90);
% Use nnz (numer of non-zero) instead of length, since cond1 is now a logical vector of all elements
A = [sst(cond1),ones(nnz(cond1),1)];
B = regress(pre(cond1), A);
pt90 = B(1)*(tp90-tc(jj));
AA = [(sst(cond2)-tp90)];
BB = regress(pre(cond2)-pt90,AA);
pem=zeros(nsst,1);
pem(cond1) = max(0, B(1)*(sst(cond1)-tc(jj)));
pem(cond2) = max(0, (BB(1)*(sst(cond2)-tp90))+pt90);
E(jj) = sqrt(nansum((pem-pre).^2)/npr);
end
tcc = tc(E==min(E));
if ~isempty(tcc);
cond1 = (sst>=tcc & sst<=tp90);
cond2 = (sst>=tp90);
A = [sst(cond1),ones(nnz(cond1),1)];
B = regress(pre(cond1),A);
pt90 = B(1)*(tp90-tcc);
AA = [sst(cond2)-tp90];
BB = regress(pre(cond2)-pt90,AA);
pem = zeros(length(sst),1);
pem(cond1) = max(0, B(1)*(sst(cond1)-tcc));
pem(cond2) = max(0, (BB(1)*(sst(cond2)-tp90))+pt90);
RMSE = sqrt(nansum((pem-pre).^2)/npr);
% Outputs, which we might as well assign all at once!
par(ii,:)=[tcc, tp90, B(1), BB(1), RMSE, ...
nanmean(sst), nanmean(pre), nanmean(pem)];
end
end
end
I am trying to keep a track of all the calculations happening under 3 for loops. The data is too big therefore it is hard to keep the track of the data. Hence, I would like to construct a table which will record the number of iterations taking place inside every for loop.
The code:
for i = 1:4
% Calculations
i
for j = 1:3
% Calculations
j
for k = 1:3
% Calculations
k
end
end
end
So, the tabular output which I am expecting is like this,
Can anybody please help me in achieving this task.
You could use ndgrid to create all permutations of your i, j, and k values and then have a single for loop that loops through all permutations.
[ii, jj, kk] = ndgrid(1:4, 1:3, 1:3);
% Pre-allocate your results matrix
results = zeros(size(ii));
for n = 1:numel(ii)
% Do calculation with ii(n), jj(n), kk(n)
results(n) = ii(n) + jj(n) + kk(n);
end
Now if you want to know what the ii, jj, or kk values were for a particular entry in results, you can just index into all variables the same way.
result_of_interest = results(100);
i_of_interest = ii(100);
j_of_interest = jj(100);
k_of_interest = kk(100);
If you really need tabular output, you can transform ii, jj, and kk into your table.
data = cat(2, ii(:), jj(:), kk(:))';
You can try the following code, where you declare the dimension of each loop at the beginning, and allocate a track matrix.
ni=3
nj=4
nk=5
track = zeros(3,ni*nj*nk);
offset = 1
for i = 1:ni
% Calculations
i
for j = 1:nj
% Calculations
j
for k = 1:nk
% Calculations
k
track(1,offset) = i;
track(2,offset) = j;
track(3,offset) = k;
offset = offset + 1;
end
end
end
I am doing a very large calculation (atmospheric absorption) that has a lot of individual narrow peaks that all get added up at the end. For each peak, I have pre-calculated the range over which the value of the peak shape function is above my chosen threshold, and I am then going line by line and adding the peaks to my spectrum. A minimum example is given below:
X = 1:1e7;
K = numel(a); % count the number of peaks I have.
spectrum = zeros(size(X));
for k = 1:K
grid = X >= rng(1,k) & X <= rng(2,k);
spectrum(grid) = spectrum(grid) + peakfn(X(grid),a(k),b(k),c(k)]);
end
Here, each peak has some parameters that define the position and shape (a,b,c), and a range over which to do the calculation (rng). This works great, and on my machine it benchmarks at around 220 seconds to do a complete data set. However, I have a 4 core machine and I would eventually like to run this on a cluster, so I'd like to parallelize it and make it scaleable.
Because each loop relies on the results of the previous iteration, I cannot use parfor, so I am taking my first step into learning how to use spmd blocks. My first try looked like this:
X = 1:1e7;
cores = matlabpool('size');
K = numel(a);
spectrum = zeros(size(X),cores);
spmd
n = labindex:cores:K
N = numel(n);
for k = 1:N
grid = X >= rng(1,n(k)) & X <= rng(2,n(k));
spectrum(grid,labindex) = spectrum(grid,labindex) + peakfn(X(grid),a(n(k)),b(n(k)),c(n(k))]);
end
end
finalSpectrum = sum(spectrum,2);
This almost works. The program crashes at the last line because spectrum is of type Composite, and the documentation for 2013a is spotty on how to turn Composite data into a matrix (cell2mat does not work). This also does not scale well because the more cores I have, the larger the matrix is, and that large matrix has to get copied to each worker, which then ignores most of the data. Question 1: how do I turn a Composite data type into a useable array?
The second thing I tried was to use a codistributed array.
spmd
spectrum = codistributed.zeros(K,cores);
disp(size(getLocalPart(spectrum)))
end
This tells me that each worker has a single vector of size [K 1], which I believe is what I want, but when I try to then meld the above methods
spmd
spectrum = codistributed.zeros(K,cores);
n = labindex:cores:K
N = numel(n);
for k = 1:N
grid = X >= rng(1,n(k)) & X <= rng(2,n(k));
spectrum(grid) = spectrum(grid) + peakfn(X(grid),a(n(k)),b(n(k)),c(n(k))]); end
finalSpectrum = gather(spectrum);
end
finalSpectrum = sum(finalSpectrum,2);
I get Matrix dimensions must agree errors. Since it's in a parallel block, I can't use my normal debugging crutch of stepping through the loop and seeing what the size of each block is at each point to see what's going on. Question 2: what is the proper way to index into and out of a codistributed array in an spmd block?
Regarding question#1, the Composite variable in the client basically refers to a non-distributed variant array stored on the workers. You can access the array from each worker by {}-indexing using its corresponding labindex (e.g: spectrum{1}, spectrum{2}, ..).
For your code that would be: finalSpectrum = sum(cat(2,spectrum{:}), 2);
Now I tried this problem myself using random data. Below are three implementations to compare (see here to understand the difference between distributed and nondistributed arrays). First we start with the common data:
len = 100; % spectrum length
K = 10; % number of peaks
X = 1:len;
% random position and shape parameters
a = rand(1,K); b = rand(1,K); c = rand(1,K);
% random peak ranges (lower/upper thresholds)
ranges = sort(randi([1 len], [2 K]));
% dummy peakfn() function
fcn = #(x,a,b,c) x+a+b+c;
% prepare a pool of MATLAB workers
matlabpool open
1) Serial for-loop:
spectrum = zeros(size(X));
for i=1:size(ranges,2)
r = ranges(:,i);
idx = (r(1) <= X & X <= r(2));
spectrum(idx) = spectrum(idx) + fcn(X(idx), a(i), b(i), c(i));
end
s1 = spectrum;
clear spectrum i r idx
2) SPMD with Composite array
spmd
spectrum = zeros(1,len);
ind = labindex:numlabs:K;
for i=1:numel(ind)
r = ranges(:,ind(i));
idx = (r(1) <= X & X <= r(2));
spectrum(idx) = spectrum(idx) + ...
feval(fcn, X(idx), a(ind(i)), b(ind(i)), c(ind(i)));
end
end
s2 = sum(vertcat(spectrum{:}));
clear spectrum i r idx ind
3) SPMD with co-distributed array
spmd
spectrum = zeros(numlabs, len, codistributor('1d',1));
ind = labindex:numlabs:K;
for i=1:numel(ind)
r = ranges(:,ind(i));
idx = (r(1) <= X & X <= r(2));
spectrum(labindex,idx) = spectrum(labindex,idx) + ...
feval(fcn, X(idx), a(ind(i)), b(ind(i)), c(ind(i)));
end
end
s3 = sum(gather(spectrum));
clear spectrum i r idx ind
All three results should be equal (to within an acceptably small margin of error)
>> max([max(s1-s2), max(s1-s3), max(s2-s3)])
ans =
2.8422e-14
I am applying an ML estimate of a Bernoulli random variable. I have initially the following code:
muBern = 0.75;
bernoulliSamples = rand(1, N);
bernoulliSamples(bernoulliSamples < muBern) = 1;
bernoulliSamples(bernoulliSamples > muBern & bernoulliSamples ~= 1) = 0;
bernoulliSamples; % 1xN matrix of Bernoulli measurements, 1's and 0's
estimateML = zeros(1,N);
for n = 1:N
estimateML(n) = (1/n)*sum(bernoulliSamples(1:n)); % The ML estimate for muBern
end
This works fairly well, but every run of the code is only one possible result of taking N=100 observations. I want to repeat this experiment I=100 times and take the average of all the results, to get a solution that accurately represents the experiment.
muBern = 0.75;
bernoulliSamples = rand(I, N);
bernoulliSamples(bernoulliSamples < muBern) = 1;
bernoulliSamples(bernoulliSamples > muBern & bernoulliSamples ~= 1) = 0;
bernoulliSamples; % IxN matrix of Bernoulli measurements, 1's and 0's
estimateML = zeros(I,N);
for n = 1:N
estimateML(n,:) = (1/n)*sum(bernoulliSamples(1:n,2)); % The ML estimate for muBern
end
I am wondering if this for loop is doing what I want it to: each row represents a completely different experiment. Is the second code instance doing the same thing as the first one, only with 100 different results as a cause of 100 different experiments?
You don't need any loops. In the single-experiment case, replace the loop by this, which does the same thing:
estimateML = cumsum(bernoulliSamples) ./ (1:N);
In the multiple-experiment case, use this:
estimateML = bsxfun(#rdivide, cumsum(bernoulliSamples,2), 1:N);
Came up with the answer, I was just overthinking it, if anyone is interested the following is what I was looking for:
for n = 1:N
estimateML(:,n) = (1/n)*sum(bernoulliSamples(:,1:n),2); % The ML estimate for muBern
end
I want to vectorize the following MATLAB code. I think it must be simple but I'm finding it confusing nevertheless.
r = some constant less than m or n
[m,n] = size(C);
S = zeros(m-r,n-r);
for i=1:m-r+1
for j=1:n-r+1
S(i,j) = sum(diag(C(i:i+r-1,j:j+r-1)));
end
end
The code calculates a table of scores, S, for a dynamic programming algorithm, from another score table, C.
The diagonal summing is to generate scores for individual pieces of the data used to generate C, for all possible pieces (of size r).
Thanks in advance for any answers! Sorry if this one should be obvious...
Note
The built-in conv2 turned out to be faster than convnfft, because my eye(r) is quite small ( 5 <= r <= 20 ). convnfft.m states that r should be > 20 for any benefit to manifest.
If I understand correctly, you're trying to calculate the diagonal sum of every subarray of C, where you have removed the last row and column of C (if you should not remove the row/col, you need to loop to m-r+1, and you need to pass the entire array C to the function in my solution below).
You can do this operation via a convolution, like so:
S = conv2(C(1:end-1,1:end-1),eye(r),'valid');
If C and r are large, you may want to have a look at CONVNFFT from the Matlab File Exchange to speed up calculations.
Based on the idea of JS, and as Jonas pointed out in the comments, this can be done in two lines using IM2COL with some array manipulation:
B = im2col(C, [r r], 'sliding');
S = reshape( sum(B(1:r+1:end,:)), size(C)-r+1 );
Basically B contains the elements of all sliding blocks of size r-by-r over the matrix C. Then we take the elements on the diagonal of each of these blocks B(1:r+1:end,:), compute their sum, and reshape the result to the expected size.
Comparing this to the convolution-based solution by Jonas, this does not perform any matrix multiplication, only indexing...
I would think you might need to rearrange C into a 3D matrix before summing it along one of the dimensions. I'll post with an answer shortly.
EDIT
I didn't manage to find a way to vectorise it cleanly, but I did find the function accumarray, which might be of some help. I'll look at it in more detail when I am home.
EDIT#2
Found a simpler solution by using linear indexing, but this could be memory-intensive.
At C(1,1), the indexes we want to sum are 1+[0, m+1, 2*m+2, 3*m+3, 4*m+4, ... ], or (0:r-1)+(0:m:(r-1)*m)
sum_ind = (0:r-1)+(0:m:(r-1)*m);
create S_offset, an (m-r) by (n-r) by r matrix, such that S_offset(:,:,1) = 0, S_offset(:,:,2) = m+1, S_offset(:,:,3) = 2*m+2, and so on.
S_offset = permute(repmat( sum_ind, [m-r, 1, n-r] ), [1, 3, 2]);
create S_base, a matrix of base array addresses from which the offset will be calculated.
S_base = reshape(1:m*n,[m n]);
S_base = repmat(S_base(1:m-r,1:n-r), [1, 1, r]);
Finally, use S_base+S_offset to address the values of C.
S = sum(C(S_base+S_offset), 3);
You can, of course, use bsxfun and other methods to make it more efficient; here I chose to lay it out for clarity. I have yet to benchmark this to see how it compares with the double-loop method though; I need to head home for dinner first!
Is this what you're looking for? This function adds the diagonals and puts them into a vector similar to how the function 'sum' adds up all of the columns in a matrix and puts them into a vector.
function [diagSum] = diagSumCalc(squareMatrix, LLUR0_ULLR1)
%
% Input: squareMatrix: A square matrix.
% LLUR0_ULLR1: LowerLeft to UpperRight addition = 0
% UpperLeft to LowerRight addition = 1
%
% Output: diagSum: A vector of the sum of the diagnols of the matrix.
%
% Example:
%
% >> squareMatrix = [1 2 3;
% 4 5 6;
% 7 8 9];
%
% >> diagSum = diagSumCalc(squareMatrix, 0);
%
% diagSum =
%
% 1 6 15 14 9
%
% >> diagSum = diagSumCalc(squareMatrix, 1);
%
% diagSum =
%
% 7 12 15 8 3
%
% Written by M. Phillips
% Oct. 16th, 2013
% MIT Open Source Copywrite
% Contact mphillips#hmc.edu fmi.
%
if (nargin < 2)
disp('Error on input. Needs two inputs.');
return;
end
if (LLUR0_ULLR1 ~= 0 && LLUR0_ULLR1~= 1)
disp('Error on input. Only accepts 0 or 1 as input for second condition.');
return;
end
[M, N] = size(squareMatrix);
if (M ~= N)
disp('Error on input. Only accepts a square matrix as input.');
return;
end
diagSum = zeros(1, M+N-1);
if LLUR0_ULLR1 == 1
squareMatrix = rot90(squareMatrix, -1);
end
for i = 1:length(diagSum)
if i <= M
countUp = 1;
countDown = i;
while countDown ~= 0
diagSum(i) = squareMatrix(countUp, countDown) + diagSum(i);
countUp = countUp+1;
countDown = countDown-1;
end
end
if i > M
countUp = i-M+1;
countDown = M;
while countUp ~= M+1
diagSum(i) = squareMatrix(countUp, countDown) + diagSum(i);
countUp = countUp+1;
countDown = countDown-1;
end
end
end
Cheers