How can I access different rows from multiple pages in a 3D array while avoiding a for-loop?
Let's assume I have a 10x5x3 matrix (mat1) and I would like to copy different individual rows from the three pages (such as the 4th, 2nd, and 5th row of the 1st, 2nd, and 3rd page) into the first row of another 10x5x3 matrix (mat2).
My solution uses a for-loop. What about vectorization?
mat1 = randi(100, 10, 5, 3)
mat2 = nan(size(mat1))
rows_to_copy = [4, 2, 5]
for i = 1 : 3
mat2(1, :, i) = mat1(rows_to_copy(i), :, i)
end
Any vectorized solution is likely not going to be as simple as your for loop solution, and might actually be less efficient (edit: see timing tests below). However, if you're curious, vectorizing an indexing operation like this generally involves converting your desired indices from subscripts to linear indices. Normally you can do this using sub2ind, but since you're selecting entire rows it may be more efficient to calculate the index yourself.
Here's a solution that takes advantage of implicit expansion in newer versions of MATLAB (R2016b and later):
[R, C, D] = size(mat1);
index = rows_to_copy+R.*(0:(C-1)).'+R*C.*(0:(D-1));
mat2(1, :, :) = reshape(mat1(index), 1, C, D);
Note that if you don't really need all the extra space full of NaN values in mat2, you can make your result more compact by just concatenating all the rows into a 2-D matrix instead:
>> mat2 = mat1(index).'
mat2 =
95 41 2 19 44
38 31 93 27 27
49 10 72 91 49
And if you're still using an older version of MATLAB without implicit expansion, you can use bsxfun instead:
index = bsxfun(#plus, rows_to_copy+R*C.*(0:(D-1)), R.*(0:(C-1)).');
Timing
I ran some tests using timeit (R2018a, Windows 7 64-bit) to see how the loop and indexing solutions compared. I tested 3 different scenarios: increasing row size, increasing column size, and increasing page size (third dimension) for mat1. The rows_to_copy was randomly selected and always had the same number of elements as the page size of mat1. Here are the results, showing the ratio of the loop time versus the indexing time:
Aside from some transient noise, there are some clear patterns. Increasing either the number of rows or columns (blue or red lines) doesn't appreciably change the time ratio, which hovers in the range of 0.7 to 0.9, meaning the for loop is slightly faster on average. Increasing the number of pages (yellow line) means the for loop has to iterate more times, and the indexing solution quickly starts to win out, reaching an 8 times speedup when the page size exceeds about 150.
Related
I have a matrix A and a vector b. I don't know their sizes, the size varies because it is the output of another function. What I want to do is filter A by a column (let's say jth column) which has at least one value that is in b.
How do I do this without measuring the size of b and concatenating every filtered result. Right now, the code is like this (assume j is a given value)
bsize=size(b,1);
for i=1:bsize
if i==1
a=A(A(:,j)==b(i),:);
else
a=[a; A(A(:,j)==b(i),:)];
end
end
I want to code a faster solution.
I am adding a numerical example just to make it clear. Let's say
A=[2 4
7 14
11 13
15 14]
and b=[4 14]
What I'm trying to do is filter to obtain the A matrix whose values are 4 and 14 in the second column, the elements of b to obtain the following output.
A=[2 4
7 14
15 14]
In my data A has more than 12000 rows and b has more than 100 elements. It doesn't always have to be the second column, sometimes the column index changes but that's not the problem now.
Use the ismember function to create a logical index based on column j=2 of A and vector b, and use that index into the rows of A:
output = A(ismember(A(:,j), b), :);
I have a 164 x 246 matrix called M. M is data for time series containing 246 time points of 164 brain regions. I want to work on only specific blocks of the time series, not the whole thing. To do so, I created a vector called onsets containing the time onset of each block.
onsets = [7;37;82;112;145;175;190;220];
In this example, there are 8 blocks total (though this number can vary), each blocks containing 9 time points. So for instance, the first block would contain time point 7, 8, 9,..., 15; the second block would contain time point 37, 38, 39,..., 45. I would like to extract the time points for these 8 blocks from M and concatenate 8 these blocks. Thus, the output should be a 164 x 72 matrix (i.e., 164 regions, 8 blocks x 9 time points/per block).
This seems like a very simple indexing problem but I'm struggling to do this efficiently. I've tried indexing each block in M (for intance, vertcat(M(onsets(1,1):onsets(1,1)+8,:));) then use vertcat but this seems very clumsy. Can anyone help?
Try this:
% create sample data
M = rand(164,246);
% create index vector
idx = false(1,size(M,2));
onsets = [7;37;82;112;145;175;190;220];
for i=1:numel(onsets)
idx(onsets(i):onsets(i)+8) = true;
end
% create output matrix
MM = M(:,idx);
You seem to have switched the dimensions somehow, i.e. you try to operate on the rows of M whilst according to your description you need to operate on the columns. Hope this helps.
I have a Problem. I have a Matrix A with integer values between 0 and 5.
for example like:
x=randi(5,10,10)
Now I want to call a filter, size 3x3, which gives me the the most common value
I have tried 2 solutions:
fun = #(z) mode(z(:));
y1 = nlfilter(x,[3 3],fun);
which takes very long...
and
y2 = colfilt(x,[3 3],'sliding',#mode);
which also takes long.
I have some really big matrices and both solutions take a long time.
Is there any faster way?
+1 to #Floris for the excellent suggestion to use hist. It's very fast. You can do a bit better though. hist is based on histc, which can be used instead. histc is a compiled function, i.e., not written in Matlab, which is why the solution is much faster.
Here's a small function that attempts to generalize what #Floris did (also that solution returns a vector rather than the desired matrix) and achieve what you're doing with nlfilter and colfilt. It doesn't require that the input have particular dimensions and uses im2col to efficiently rearrange the data. In fact, the the first three lines and the call to im2col are virtually identical to what colfit does in your case.
function a=intmodefilt(a,nhood)
[ma,na] = size(a);
aa(ma+nhood(1)-1,na+nhood(2)-1) = 0;
aa(floor((nhood(1)-1)/2)+(1:ma),floor((nhood(2)-1)/2)+(1:na)) = a;
[~,a(:)] = max(histc(im2col(aa,nhood,'sliding'),min(a(:))-1:max(a(:))));
a = a-1;
Usage:
x = randi(5,10,10);
y3 = intmodefilt(x,[3 3]);
For large arrays, this is over 75 times faster than colfilt on my machine. Replacing hist with histc is responsible for a factor of two speedup. There is of course no input checking so the function assumes that a is all integers, etc.
Lastly, note that randi(IMAX,N,N) returns values in the range 1:IMAX, not 0:IMAX as you seem to state.
One suggestion would be to reshape your array so each 3x3 block becomes a column vector. If your initial array dimensions are divisible by 3, this is simple. If they don't, you need to work a little bit harder. And you need to repeat this nine times, starting at different offsets into the matrix - I will leave that as an exercise.
Here is some code that shows the basic idea (using only functions available in FreeMat - I don't have Matlab on my machine at home...):
N = 100;
A = randi(0,5*ones(3*N,3*N));
B = reshape(permute(reshape(A,[3 N 3 N]),[1 3 2 4]), [ 9 N*N]);
hh = hist(B, 0:5); % histogram of each 3x3 block: bin with largest value is the mode
[mm mi] = max(hh); % mi will contain bin with largest value
figure; hist(B(:),0:5); title 'histogram of B'; % flat, as expected
figure; hist(mi-1, 0:5); title 'histogram of mi' % not flat?...
Here are the plots:
The strange thing, when you run this code, is that the distribution of mi is not flat, but skewed towards smaller values. When you inspect the histograms, you will see that is because you will frequently have more than one bin with the "max" value in it. In that case, you get the first bin with the max number. This is obviously going to skew your results badly; something to think about. A much better filter might be a median filter - the one that has equal numbers of neighboring pixels above and below. That has a unique solution (while mode can have up to four values, for nine pixels - namely, four bins with two values each).
Something to think about.
Can't show you a mex example today (wrong computer); but there are ample good examples on the Mathworks website (and all over the web) that are quite easy to follow. See for example http://www.shawnlankton.com/2008/03/getting-started-with-mex-a-short-tutorial/
I'm working with a 30*26000 size matrix that has NaNs at the beginning and at the end. NaNs are also sprinkled throughout each row. I can fill in the NaNs with linear interpolation but that will leave NaNs at the beginning and end of each row. Extrapolating to replace these NaNs at the ends is not ideal for my data set.
I want to just trim the matrix. Take for example a 3 by 6 matrix:
NaN NaN 1 2 3 NaN
NaN 1 2 3 NaN NaN
1 NaN 2 3 4 5
Cut off the left most and right most columns such that no row begins or ends with a NaN.
1 2
2 3
2 3
So we are left with a 3 by 2 matrix.
How can I do this in Matlab? (speed-optimized; I will need to apply this to a million size matrix)
Thanks!
For your example you can do the following:
let a your matrix with NaN and numerical values.
ind1 = sum(isnan(a),1); % count the NaN values along columns
s = find(ind1 == 0, 1, 'first'); % find the first column without any NaN
e = find(ind1 == 0, 1, 'last'); % find the last column without any NaN
So now just keep this part of the matrix from s-th to e-th column:
b = a(:,s:e);
Additional check may be needed for the case no column is clear of NaNs.
Firstly, the vectorized solution of argyris will work perfectly well (+1). I'm only posting this because you emphasized that you wanted a speed optimized solution. Well, the downside of argyris solution is that the sum and isnan operation are performed on the entire matrix. This will be optimal if you have to come a long way in on either side to find the first non-NaN column. But what if you don't? A loop-based solution that exploits the fact that you may only need to come in a few columns may do better (particularly given how good the JIT accelerator is getting at executing single loops quickly). I've put together a speed test that includes both argyris and my solution:
%#Set up an example case using the matrix size you indicated in the question
T = 30;
N = 26000;
X = rand(T, N);
TrueL = 8;
TrueR = N - 8;
X(:, 1:TrueL) = NaN;
X(:, TrueR:end) = NaN;
%#argyris solution
tic
I1 = sum(isnan(X));
argL = find(I1 == 0, 1, 'first');
argR = find(I1 == 0, 1, 'last');
Soln1 = X(:, argL:argR);
toc
%#My loop based solution (faster if TrueL and TrueR are small)
tic
for n = 1:N
if ~any(isnan(X(:, n)))
break
end
end
ColinL = n;
for n = N:-1:1
if ~any(isnan(X(:, n)))
break
end
end
ColinR = n;
Soln2 = X(:, ColinL:ColinR);
toc
In the above example, the solution will need to get rid of the first 8 and last 8 columns. The outcome of the speed test?
Elapsed time is 0.002919 seconds. %#argyris solution
Elapsed time is 0.001007 seconds. %#My solution
The loop based solution is almost 3 times faster. Okay, now let's up the number of columns that we need to get rid of on either side to 100:
Elapsed time is 0.002769 seconds. %#argyris solution
Elapsed time is 0.001999 seconds. %#My solution
Still ahead. What about 1000 columns on either side?
Elapsed time is 0.003597 seconds. %#argyris solution
Elapsed time is 0.003719 seconds. %#My solution
So we've found our tipping point (on my machine at least - Quad core i7, Linux Mint v12, Matlab R2012b). Once we need to come in about 1000 columns on either side, we're better off using the vectorized solution.
One final note of CAUTION: If the routine is occurring inside another (possibly unrelated) loop, then speed comparisons should be re-done. This is because my solution will now involve a double loop. Even if the loops are unrelated, the JIT accelerator is not so good with double loops. I did some quick tests on my machine, and my solution still comes out ahead for small TrueL and TrueR (ie less than 100), but the advantage is not as large as it was when the outer loop was not present.
Anyway, hope this proves useful to you or anyone else who comes a-reading.
Cheers!
EDIT: I've done a few speed tests incorporating angainor's very neat one-liner (+1). It performs almost as well as my loop based solution when the number of columns to be removed is small. Suprisingly, it didn't scale that well when the number of columns to be removed is large, unlike argyris's solution. That may have something to do with the computer I'm on now though: work Windows machine - I've never really trusted it fully :-)
Both earlier proposed solutions are great, I am posting this one-liner for completeness:
A(:,isfinite(sum(A)))
ans =
1 2
2 3
2 3
It avoids going through the matrix entries twice (what Colin pointed out) by first calculating the row sums and after that calling isfinite. I also removed the find calls - they are not necessary since you can use logical indexing instead.
I do not have my computer here, so I leave out the performance tests.
I have a matrix in MATLAB, lets say:
a = [
89 79 96
72 51 74
94 88 87
69 47 78
]
I want to subtract from each element the average of its column and divide by the column's standard deviation. How can I do it in a way which could be implemented to any other matrix without using loops.
thanks
If your version supports bsxfun (which is probably the case unless you have very old matlab version), you should use it, it's much faster than repmat, and consumes much less memory.
You can just do: result = bsxfun(#rdivide,bsxfun(#minus,a,mean(a)),std(a))
You can use repmat to make your average/std vector the same size as your original matrix, then use direct computation like so:
[rows, cols] = size(a); %#to get the number of rows
avgc= repmat(avg(a),[rows 1]); %# average by column, vertically replicated by number of rows
stdc= repmat(std(a),[rows 1]); %# std by column, vertically replicated by number of rows
%# Here, a, avgc and stdc are the same size
result= (a - avgc) ./ stdc;
Edit:
Judging from a mathworks blog post,bsxfun solution is faster and consumes less memory (see acai answer). For moderate size matrices, I personally prefer repmat that makes code easier to read and debug (for me).
You could also use the ZSCORE function from the Statistics Toolbox:
result = zscore(a)
In fact, it calls BSXFUN underneath, but it is careful not to divide by a zero standard deviation (you can look at the source code yourself: edit zscore)