Preamble:
My framework is Matlab. I have a very large data matrix M (size(M) = 30 20 30 20 51 300 ) and I need to manipulate this matrix (to calculate some correlations, mean, shift it circularly, interpolate it and so on).
!Important! : most of the elements of this matrix are zeros or ones !!
My question: Since it is very time consuming to work with such a huge matrix, is it possible to perform the same manipulations, but on the sparse form of this matrix? Of course, one should not loose any information about zeros or ones (for example, for calculations of averages or correlations between different elements).
Is there any other way to handle such matrices? (huge and mostly 0's and 1's)
Thanks in advance!
You can use sparse matrices.
The only issue with sparse matrices is, that they only come in two dimensions, so the straight-forward way to represent your matrix would be to wrap it into a sparse matrix, of size [N 1] where N = prod([ 30 20 30 20 51 300]) in your case.
I've done this for N-dimension histograms (which sounds similar to your application) and it works fine.
You'll lose the possibility to use all the smart indexing though.
So using mean/sum etc. on single dimensions will become somewhat more complicated since you'll have to convert subscript-indices to linear indices and vice versa.
For that, you should have a look at sub2ind and ind2sub.
(Sounds like a fun project on wrapping the builtin sparse matrix into a n-dimensional sparse matrix...)
Related
Suppose that for some needed transformation I have a mxn matrix A that consists of a few 1's and many 0's. If I were to transform a nx1 vector by A, is this considered a vectorized implementation?
While the matrix A mainly consists of 0's, does this still cause the same amount of FLOPs to occur?
Would it be wiser and more optimized to do the transformation needed another way? One that won't cause needless calculations such as 0*c?
The matrix is sparse. The number of operations for various things is lower when you store it properly and use the right routines.
I have a 40x43 matrix and I would like to use this matrix a building block to generate larger matrix.
I want to generate a structure like the image attached and the building block is the 40x43 matrix. I tried using [A zeros(20,43); zeros(20,43) A] but as I had guessed, the horzcat did't work. I would ideally like to use this block 1000 times to extend the structure of matrix. Could anyone tell me an efficient way to concatenate the small matrix?
Try using kron. This performs what is known as the Kronecker product such that for two matrices A and B, the result is:
In this case, we can replicate what you want exactly by setting A to be the identity matrix of size 1000 x 1000 and B to be the matrix you want to replicate. However, to promote computational savings and memory usage, make sure you use the sparse version of the identity matrix. This will convert the output matrix to sparse form. If you want to replicate this 1000 times, you are creating a 40000 x 43000 matrix and this requires 13.76 GB of memory and you probably don't have enough memory available for this matrix. Since most of the elements are zero, use the sparse version instead:
N = 1000;
B = kron(speye(N), A);
I have a sparse 5018x5018 matrix in MATLAB, which has about 100k values set to 1 (i.e., about 99.6% empty).
I'm trying to flip roughly 5% of those zeros to ones (i.e., about 1.25m entries). I have the x and y indices in the matrix I want to flip.
Here is what I have done:
sizeMat=size(network);
idxToReplace=sub2ind(sizeMat,x_idx, y_idx);
network(idxToReplace) = 1;
This is incredibly slow, in particular the last line. Is there any way to make this operation run noticeably faster, preferably without using mex files?
This should be faster:
idxToReplace=sparse(x_idx,y_idx,ones(size(x_idx),size(matrix,1),size(matrix,2)); % Create a sparse with ones at locations
network=network+idxToReplace; % Add the two matrices
I think your solution is very slow because you create a 1.26e6 logical array with your points and then store them in the sparse matrix. In my solution, you only create a sparse matrix and just sum the two.
I need to convolve a matrix with many other matrices with few calls to convn.
for example: I have size(MyMat)=[fm, fm ,1, bSize] and size(masks)=[s, s, maskNum]
I want res(:,:,k,:) to be the product of convolving masks(:,:,k) with MyMat
res(:,:,k,:)=convn(MyMat,masks(:,:,k));
since the convolution takes up over 80% of the running time for my script and is called hundreds of thousands of times, I don't want to use a loop.
I'm looking for the fastest way to do this. basically, you could say I have bSize matrices, and I want to apply convolution masks masks to all of them with as few calls as possible to convolution.
The matrices are all small,non-sparse, fft-based convolution will probably slow it down (as a commentor here verified :) )
(The reason I have a 1 in the size of MyMat is because I actually have more elements in that dimension, but I compute the convolution for each element in that dimension in a loop)
The main goal is simply to eliminate the need for the following loop, or make it parallel with very little overhead, if possible:
for i=1:length
res(:,:,:,i)=convn(MyArray,convMask(:,:,i));
end
parallelizing for the GPU would be great if there's a way to do this with less overhead than the usual parfor
Thank you!
I assume that you are preallocating the array res correctly? Without a simple demo of what your doing and an idea of the size of fm, s, etc., one can only make guesses to help you. If the sizes of your matrices are large enough you might look into FFT-based convolution methods (there are some for convn on the Matlab File Exchange). If the data is sparse (> 50% zeros), you could try converting this to matrix multiplication and use sparse data types. You could also try gpuArray/convn if you have a decent one.
preface: As the matlab guiderules state, Usually, when one wants to efficiently populate a sparse matrix in matlab, he should create a vector of indexes into the matrix and a vector of values he wants to assign, and then concentrate all the assignments into one atomic operation, so as to allow matlab to "prepare" the matrix in advance and optimize the assignment speed. A simple example:
A=sparse([]);
inds=some_index_generating_method();
vals=some_value_generating_method();
A(inds)=vals;
My question: what can I do in the case where inds contain overlapping indexes, i.e inds=[4 17 8 17 9] where 17 repeats twice.
In this case, what I would want to happen is that the matrix would be assigned the addition of all the values that are mapped to the same index, i.e for the previous example
A(17)=vals(2)+vals(4) %as inds(2)==inds(4)
Is there any straightforward and, most importantly, fast way to achieve this? I have no way of generating the indexes and values in a "smarter" way.
This might help:
S = sparse(i,j,s,m,n,nzmax) uses vectors i, j, and s to generate an m-by-n sparse matrix such that S(i(k),j(k)) = s(k), with space allocated for nzmax nonzeros. Vectors i, j, and s are all the same length. Any elements of s that are zero are ignored, along with the corresponding values of i and j. Any elements of s that have duplicate values of i and j are added together.
See more at MATLAB documentation for sparse function