I'm working with a 30*26000 size matrix that has NaNs at the beginning and at the end. NaNs are also sprinkled throughout each row. I can fill in the NaNs with linear interpolation but that will leave NaNs at the beginning and end of each row. Extrapolating to replace these NaNs at the ends is not ideal for my data set.
I want to just trim the matrix. Take for example a 3 by 6 matrix:
NaN NaN 1 2 3 NaN
NaN 1 2 3 NaN NaN
1 NaN 2 3 4 5
Cut off the left most and right most columns such that no row begins or ends with a NaN.
1 2
2 3
2 3
So we are left with a 3 by 2 matrix.
How can I do this in Matlab? (speed-optimized; I will need to apply this to a million size matrix)
Thanks!
For your example you can do the following:
let a your matrix with NaN and numerical values.
ind1 = sum(isnan(a),1); % count the NaN values along columns
s = find(ind1 == 0, 1, 'first'); % find the first column without any NaN
e = find(ind1 == 0, 1, 'last'); % find the last column without any NaN
So now just keep this part of the matrix from s-th to e-th column:
b = a(:,s:e);
Additional check may be needed for the case no column is clear of NaNs.
Firstly, the vectorized solution of argyris will work perfectly well (+1). I'm only posting this because you emphasized that you wanted a speed optimized solution. Well, the downside of argyris solution is that the sum and isnan operation are performed on the entire matrix. This will be optimal if you have to come a long way in on either side to find the first non-NaN column. But what if you don't? A loop-based solution that exploits the fact that you may only need to come in a few columns may do better (particularly given how good the JIT accelerator is getting at executing single loops quickly). I've put together a speed test that includes both argyris and my solution:
%#Set up an example case using the matrix size you indicated in the question
T = 30;
N = 26000;
X = rand(T, N);
TrueL = 8;
TrueR = N - 8;
X(:, 1:TrueL) = NaN;
X(:, TrueR:end) = NaN;
%#argyris solution
tic
I1 = sum(isnan(X));
argL = find(I1 == 0, 1, 'first');
argR = find(I1 == 0, 1, 'last');
Soln1 = X(:, argL:argR);
toc
%#My loop based solution (faster if TrueL and TrueR are small)
tic
for n = 1:N
if ~any(isnan(X(:, n)))
break
end
end
ColinL = n;
for n = N:-1:1
if ~any(isnan(X(:, n)))
break
end
end
ColinR = n;
Soln2 = X(:, ColinL:ColinR);
toc
In the above example, the solution will need to get rid of the first 8 and last 8 columns. The outcome of the speed test?
Elapsed time is 0.002919 seconds. %#argyris solution
Elapsed time is 0.001007 seconds. %#My solution
The loop based solution is almost 3 times faster. Okay, now let's up the number of columns that we need to get rid of on either side to 100:
Elapsed time is 0.002769 seconds. %#argyris solution
Elapsed time is 0.001999 seconds. %#My solution
Still ahead. What about 1000 columns on either side?
Elapsed time is 0.003597 seconds. %#argyris solution
Elapsed time is 0.003719 seconds. %#My solution
So we've found our tipping point (on my machine at least - Quad core i7, Linux Mint v12, Matlab R2012b). Once we need to come in about 1000 columns on either side, we're better off using the vectorized solution.
One final note of CAUTION: If the routine is occurring inside another (possibly unrelated) loop, then speed comparisons should be re-done. This is because my solution will now involve a double loop. Even if the loops are unrelated, the JIT accelerator is not so good with double loops. I did some quick tests on my machine, and my solution still comes out ahead for small TrueL and TrueR (ie less than 100), but the advantage is not as large as it was when the outer loop was not present.
Anyway, hope this proves useful to you or anyone else who comes a-reading.
Cheers!
EDIT: I've done a few speed tests incorporating angainor's very neat one-liner (+1). It performs almost as well as my loop based solution when the number of columns to be removed is small. Suprisingly, it didn't scale that well when the number of columns to be removed is large, unlike argyris's solution. That may have something to do with the computer I'm on now though: work Windows machine - I've never really trusted it fully :-)
Both earlier proposed solutions are great, I am posting this one-liner for completeness:
A(:,isfinite(sum(A)))
ans =
1 2
2 3
2 3
It avoids going through the matrix entries twice (what Colin pointed out) by first calculating the row sums and after that calling isfinite. I also removed the find calls - they are not necessary since you can use logical indexing instead.
I do not have my computer here, so I leave out the performance tests.
Related
This is my code:
variables=1000;
t=20;
x=zeros(t,t,3);
y=rand(variables,3);
z=rand(t,t,variables);
e=rand(variables,1);
for c=1:variables
x(:,:,1)=x(:,:,1)+y(c,1).*((z(:,:,c)-e(c)).^2);
x(:,:,2)=x(:,:,2)+y(c,2).*((z(:,:,c)-e(c)).^2);
x(:,:,3)=x(:,:,3)+y(c,3).*((z(:,:,c)-e(c)).^2);
end
How can I improve calculation speed on this loop? I think that the problem is the for loop with a large c.
It's a myth, but alas a persistent one, that loops are slow in MATLAB. As you've written your for loop, it goes sequentially through the last dimension of your variables. That pretty much translates to a FORTRAN loop directly, leaving little room for improvement using vectorisation. The below does vectorise your output as much as possible, but doesn't improve performance much, even though reshape() is almost free, and severely degrades readability.
In each iteration, all you're doing is calculating y(c,1).*((z(:,:,c)-e(c)).^2), which is added to the total. If we are able to vectorise that expression, we can sum over the dimension of c to get rid of the loop.
z(:,:,c)-e(c) can be vectorised by adding two singleton dimensions to e: reshape(e, [1 1 numel(e)]), then subtract and power by 2 as usual.
Multiplication by y(c,1) also works, if we add two singleton dimensions to y(:,1):, reshape(y(:,1), [1 1 numel(e)]), then multiply again as usual.
Finally, we just need to sum over our 3rd dimension and we end up with our t -by- t result: sum(tmp2, 3).
All that's left are the hardcoded three dimensions in x, which I've left be in a loop.
The working code on R2007b:
variables=10;
t=2;
x=zeros(t,t,3);
y=rand(variables,3);
z=rand(t,t,variables);
e=rand(variables,1);
for ii = 1:size(x, 3)
x(:, :, ii) = sum(bsxfun(#times, reshape(y(:,1), [1 1 numel(e)]), bsxfun(#minus, z, reshape(e, [1 1 numel(e)])).^2), 3);
end
I wasn't sure what to do with the hardcoded dimension of 3, so I just left a loop over that. The rest is vectorised away, thanks to a few reshape() calls to arrange the dimensions for the bsxfun() expansion.
Code for >R2016b with implicit expansion:
for ii = 1:size(x, 3)
x(:, :, ii) = sum(reshape(y(:,ii), [1 1 numel(e)]) .* (z - reshape(e, [1 1 numel(e)])).^2, 3)
end
A quick timing comparison shows that this is roughly 2x faster than your original loop:
Elapsed time is 0.780516 seconds. Original code
Elapsed time is 0.397369 seconds. My bsxfun() solution
Elapsed time is 0.305160 seconds. My implicit expansion
Note that in the above a 100 loops were ran for each code version, i.e. timings are 8ms, 4ms and 3ms per version.
For an introduction to reshape() you can refer to this answer of mine.
The documentation article on implicit broadcasting is rather good, as is this blog.
How can I access different rows from multiple pages in a 3D array while avoiding a for-loop?
Let's assume I have a 10x5x3 matrix (mat1) and I would like to copy different individual rows from the three pages (such as the 4th, 2nd, and 5th row of the 1st, 2nd, and 3rd page) into the first row of another 10x5x3 matrix (mat2).
My solution uses a for-loop. What about vectorization?
mat1 = randi(100, 10, 5, 3)
mat2 = nan(size(mat1))
rows_to_copy = [4, 2, 5]
for i = 1 : 3
mat2(1, :, i) = mat1(rows_to_copy(i), :, i)
end
Any vectorized solution is likely not going to be as simple as your for loop solution, and might actually be less efficient (edit: see timing tests below). However, if you're curious, vectorizing an indexing operation like this generally involves converting your desired indices from subscripts to linear indices. Normally you can do this using sub2ind, but since you're selecting entire rows it may be more efficient to calculate the index yourself.
Here's a solution that takes advantage of implicit expansion in newer versions of MATLAB (R2016b and later):
[R, C, D] = size(mat1);
index = rows_to_copy+R.*(0:(C-1)).'+R*C.*(0:(D-1));
mat2(1, :, :) = reshape(mat1(index), 1, C, D);
Note that if you don't really need all the extra space full of NaN values in mat2, you can make your result more compact by just concatenating all the rows into a 2-D matrix instead:
>> mat2 = mat1(index).'
mat2 =
95 41 2 19 44
38 31 93 27 27
49 10 72 91 49
And if you're still using an older version of MATLAB without implicit expansion, you can use bsxfun instead:
index = bsxfun(#plus, rows_to_copy+R*C.*(0:(D-1)), R.*(0:(C-1)).');
Timing
I ran some tests using timeit (R2018a, Windows 7 64-bit) to see how the loop and indexing solutions compared. I tested 3 different scenarios: increasing row size, increasing column size, and increasing page size (third dimension) for mat1. The rows_to_copy was randomly selected and always had the same number of elements as the page size of mat1. Here are the results, showing the ratio of the loop time versus the indexing time:
Aside from some transient noise, there are some clear patterns. Increasing either the number of rows or columns (blue or red lines) doesn't appreciably change the time ratio, which hovers in the range of 0.7 to 0.9, meaning the for loop is slightly faster on average. Increasing the number of pages (yellow line) means the for loop has to iterate more times, and the indexing solution quickly starts to win out, reaching an 8 times speedup when the page size exceeds about 150.
Given a matrix, it's easy to compute the value and index of the min value:
A = rand(10);
[value, index] = min(A(:));
However I would also like to recover the second min value (idem for max).
I can of course take any of this two approaches:
Converting A to a vector and sorting it.
PROS: I can then recover the second, third... n minimum value
CONS: If A is large, sorting is expensive
Once the min location of A is located, I can replace this value by a large one (eg: Inf) and then run min again.
PROS: Cheaper than sort
CONS: I must modify my matrix (and save the modified value in an aux variable). Also re-running min is costly on a large matrix.
I'm wondering if there is a better solution:
When computing min the algorithm has to keep track of the min value found so far, until a new value has a lower value (then we update the value).
If instead we keep track of the last n min values found so far will allow to recover the minimum n values.
I can implement this, but I'm wondering if it's the best approach or if it's already implemented.
I don't know in which case it would be less expensive than sorting, but an easy, but not so fast way would be to use the following code. I may be wrong, but I don't think you can get faster with build-in functions if you just want the first and the second min.
A = rand(10);
[firstMin, firstMinIndex] = min(A(:));
secondMin = min(A(A~=firstMin));
secondMinIndex = find(A==secondMin); % slow, but use only if you need the index
Here, you go through the matrix two times more, one for the boolean operation, and one for the second min.
After some testing on 2000x2000 and 4000x4000 random matrix, it seems that this code snipset is around 3.5 time faster than the sort function applied on the same matrix.
If you really need more efficiency, you'd have to write your own mex routine, with which you can theoretically get the two values in n+log n-2 comparison, as explained in the link provided by #luismendotomas.
Hope this help !
In a single pass:
a = [53 53 49 49 97 75 4 22 4 37];
first = Inf;
second = Inf;
for i = 1:1:numel(a)
if (a(i) < first)
second = first;
first = a(i);
elseif (a(i) < second && a(i) ~= first)
second = a(i);
end
end
fprintf('First smallest %d\n', first);
fprintf('Second smallest %d\n', second);
You can remove the a(i) ~= first condition if you rather have 4, 4 as output instead of 4, 23
Also, see this SO question
As already mentioned I suppose the best (read: "most efficient") method is to implement the methods from #luismendotomas link.
However, if you want to avoid doing too much programming yourself, then you could apply some k-nearest neighbours algorithm, given you have a lower bound on your data, e.g. if all your data points are positive, you can find the 2 nearest neighbours to 0. Though I am not sure whether this is faster than your initial suggestions or not.
For one k-nearest neighbour algorithm see e.g. this
beesleep has already pointed out that method 2 (by computing the minimum twice) is more efficient that method 1 (by sorting). However the implementation provided in the answer to compute the index of the second minimum via find is, as mentioned, very inefficient.
In fact, to get the index of the second minimum, it is ca. 10x faster to set the first minimum value to inf (as suggested in the question) and then get the index of the second minimum from the min function (as opposed to using find)
[firstMin, firstMinIndex] = min(A(:));
A(firstMinIndex) = inf;
[secondMin, secondMinIndex] = min(A(:));
Here is the code which I used to compare this implementation to the one suggested by beesleep:
for i = 1:10
A = rand(10000);
tic
[firstMin, firstMinIndex] = min(A(:));
secondMin = min(A(A~=firstMin));
secondMinIndex = find(A==secondMin); % slow, but use only if you need the index
t1(i) = toc;
tic
[firstMin, firstMinIndex] = min(A(:));
A(firstMinIndex) = inf;
[secondMin, secondMinIndex] = min(A(:));
t2(i) = toc;
end
disp(mean(t1) / mean(t2))
I'm wondering, what is faster for addressing a single Element of a vector:
1) direct access via
result = a(index)
or
2) access an element via a matrix multiplication e.g
a = [1 2 3 4]';
b = [0 0 1 0];
result = b*a; % Would return 3
In my oppinion (which comes from "classic" programming like C++) the first method must be more performant, because of the direct access...the second method would need a iteration through both vectors(?).
The reason why I'm asking is, that matlab is very performant on matrix and vector operations, maybe I am missing any aspect and the second method is more effective...
A quick test:
function [] = fun1()
a = [1 2 3 4]';
b = [0 0 1 0];
tic;
for i=1:1000000
r = a(3);
end
toc;
end
Elapsed time: 0.006 seconds
Change a(3) to b*a
Elapsed time: 0.9 seconds
The performance difference is quite obvious(, and you should have done that yourself before asking this question).
Reason behind that:
No matter how efficient MATLAB's calculation is, MATLAB still needs to fetch the number 1 by 1, and do multiplication 1 by 1, and sum up. There is no hope to be faster than a single access.
In your special case, there are all 0's except 1, but it is useless to do optimization for single special case in my opinion, and the best optimization I can come up with still needs to access all the elements for at least once each.
EDIT:
It seems I am in quite good mood today....
Change a(3) to a(1)*b(1)+a(2)*b(2)+a(3)*b(3)+a(4)*b(4)
Elapsed time: 0.02 seconds
It seems that boundary checking (and/or other errands) take more time than the access and calculation.
Why would you think that multiplying a lot of numbers by zeros would be at all efficient? Even if MATLAB could be smart enough to do a test first before the multiply, it must then still do many tests.
I'm asking this question to make a point, that the dot product cannot possibly be at all efficient. Even if MATLAB were smart enough to know that there was only one element that was non-zero, to know that, it would need to do a search for the non-zero element. And how would MATLAB be smart enough to know that what you have written as a vector*vector dot product is actually intended just to access a single element, instead of a true dot product for nefarious purposes unknown to it?
How about
3) access an element by a boolean index matrix:
a = [1 2 3 4]';
b = [0 0 1 0];
result = a(b)
It's almost certainly going to be faster than (2), slower than (1).
Is there a way to extend a vector by making it repeat itself?
>v = [1 2];
>v10 = v x 5; %x represents some function. Something like "1 2" x 5 in perl
Then v10 would be:
>v10
1 2 1 2 1 2 1 2 1 2
This should work for the general case, not just for [1 2]
The function you're looking for is repmat().
v10 = repmat(v, 1, 5)
Obviously repmat is the way to go if you know in which direction you want to expand the vector.
However, if you want a general solution that always repeats the vector in the longest direction, this combination of repmat and indexing should do the trick:
v10=v(repmat(1:length(v),1,5))
Although late, I am posting this because this turned out to be the most popular answer to a similar question here.
This is a Faster Method Than repmat or reshape by an Order of Magnitude
One of the best methods for doing such things is Using Tony's Trick. I came across this trick in one of the Electrical Engineering course lectures notes of Columbia University. Repmat and Reshape are usually found to be slower than Tony's trick as it directly uses Matlabs inherent indexing. To answer you question,
Lets say, you want to tile the row vector r=[1 2 3] N times like r=[1 2 3 1 2 3 1 2 3...], then,
c=r'
cc=c(:,ones(N,1));
r_tiled = cc(:)';
This method has significant time savings against reshape or repmat for large N's.
I conducted a small Matlab test to check the speed differential between repmat and tony's trick. Using the code mentioned below, I calculated the times for constructing the same tiled vector from a base vector A=[1:N]. The results show that YES, Tony's-Trick is FASTER BY AN ORDER of MAGNITUDE, especially for larger N. People are welcome to try it themselves. This much time differential can be critical if such an operation has to be performed in loops. Here is the small script I used;
N= 10 ;% ASLO Try for values N= 10, 100, 1000, 10000
% time for tony_trick
tic;
A=(1:N)';
B=A(:,ones(N,1));
C=B(:)';
t_tony=toc;
clearvars -except t_tony N
% time for repmat
tic;
A=(1:N);
B=repmat(A,1,N);
t_repmat=toc;
clearvars -except t_tony t_repmat N
The Times (in seconds) for both methods are given below;
N=10, time_repmat = 8e-5 , time_tony = 3e-5
N=100, time_repmat = 2.9e-4 , time_tony = 6e-5
N=1000, time_repmat = 0.0302 , time_tony = 0.0058
N=10000, time_repmat = 2.9199 , time_tony = 0.5292
My RAM didn't permit me to go beyond N=10000. I am sure, the time difference between the two methods will be even more significant for N=100000. I know, these times might be different for different machines, but the relative difference in order-of-magnitude of times will stand. Also, I know, the avg of times could have been a better metric, but I just wanted to show the order of magnitude difference in time consumption between the two approaches. My machine/os details are given below :
Relevant Machine/OS/Matlab Details : Athlon i686 Arch, Ubuntu 11.04 32 bit, 3gb ram, Matlab 2011b