I have a sparse 5018x5018 matrix in MATLAB, which has about 100k values set to 1 (i.e., about 99.6% empty).
I'm trying to flip roughly 5% of those zeros to ones (i.e., about 1.25m entries). I have the x and y indices in the matrix I want to flip.
Here is what I have done:
sizeMat=size(network);
idxToReplace=sub2ind(sizeMat,x_idx, y_idx);
network(idxToReplace) = 1;
This is incredibly slow, in particular the last line. Is there any way to make this operation run noticeably faster, preferably without using mex files?
This should be faster:
idxToReplace=sparse(x_idx,y_idx,ones(size(x_idx),size(matrix,1),size(matrix,2)); % Create a sparse with ones at locations
network=network+idxToReplace; % Add the two matrices
I think your solution is very slow because you create a 1.26e6 logical array with your points and then store them in the sparse matrix. In my solution, you only create a sparse matrix and just sum the two.
Related
I have a 40x43 matrix and I would like to use this matrix a building block to generate larger matrix.
I want to generate a structure like the image attached and the building block is the 40x43 matrix. I tried using [A zeros(20,43); zeros(20,43) A] but as I had guessed, the horzcat did't work. I would ideally like to use this block 1000 times to extend the structure of matrix. Could anyone tell me an efficient way to concatenate the small matrix?
Try using kron. This performs what is known as the Kronecker product such that for two matrices A and B, the result is:
In this case, we can replicate what you want exactly by setting A to be the identity matrix of size 1000 x 1000 and B to be the matrix you want to replicate. However, to promote computational savings and memory usage, make sure you use the sparse version of the identity matrix. This will convert the output matrix to sparse form. If you want to replicate this 1000 times, you are creating a 40000 x 43000 matrix and this requires 13.76 GB of memory and you probably don't have enough memory available for this matrix. Since most of the elements are zero, use the sparse version instead:
N = 1000;
B = kron(speye(N), A);
I am filling a sparse matrix P (230k,290k) with values coming from a text file which I read line by line, here is the (simplified) code
while ...
C = textscan(text_line,'%d','delimiter',',','EmptyValue', 0);
line_number = line_number+1;
P(line_number,:)=C{1};
end
the problem I have is that while at the beginning the
P(line_number,:)=C{1};
statement is fast, after a few thousands lines become exterely slow, I guess because Matlab need to find the memory space to allocate every time. Is there a way to pre-allocate memory with sparse matrixes? I don't think so but maybe I am missing something. Any other advise which can speed up the operation (e.g. having a lot of free RAM can make the difference?)
There's a sixth input argument to sparse that tells the number of nonzero elements in the matrix. That's used by Matlab to preallocate:
S = sparse(i,j,s,m,n,nzmax) uses vectors i, j, and s to generate an
m-by-n sparse matrix such that S(i(k),j(k)) = s(k), with space
allocated for nzmax nonzeros.
So you could initiallize with
P = sparse([],[],[],230e3,290e3,nzmax);
You can make a guess about the number of nonzeros (perhaps checking file size?) and use that as nzmax. If it turns you need more nonzero elements in the end, Matlab will preallocate on the fly (slowly).
By far the fastest way to generate a sparse matrix wihtin matlab is to load all the values in at once, then generate the sparse matrix in one call to sparse. You have to load the data and arrange it into vectors defining the row and column indices and values for each filled cell. You can then call sparse using the S = sparse(i,j,s,m,n) syntax.
Preamble:
My framework is Matlab. I have a very large data matrix M (size(M) = 30 20 30 20 51 300 ) and I need to manipulate this matrix (to calculate some correlations, mean, shift it circularly, interpolate it and so on).
!Important! : most of the elements of this matrix are zeros or ones !!
My question: Since it is very time consuming to work with such a huge matrix, is it possible to perform the same manipulations, but on the sparse form of this matrix? Of course, one should not loose any information about zeros or ones (for example, for calculations of averages or correlations between different elements).
Is there any other way to handle such matrices? (huge and mostly 0's and 1's)
Thanks in advance!
You can use sparse matrices.
The only issue with sparse matrices is, that they only come in two dimensions, so the straight-forward way to represent your matrix would be to wrap it into a sparse matrix, of size [N 1] where N = prod([ 30 20 30 20 51 300]) in your case.
I've done this for N-dimension histograms (which sounds similar to your application) and it works fine.
You'll lose the possibility to use all the smart indexing though.
So using mean/sum etc. on single dimensions will become somewhat more complicated since you'll have to convert subscript-indices to linear indices and vice versa.
For that, you should have a look at sub2ind and ind2sub.
(Sounds like a fun project on wrapping the builtin sparse matrix into a n-dimensional sparse matrix...)
I have to perform this operation:
N = A'*P*A
The structure of the P matrix is block diagonal while the A matrix is largely sparse (also in a banded structure). The multiplication is performed in blocks. But the problem is storage.
The N matrix is too huge to store in full (out of memory when trying to allocate). So, I want to store in a sparse fashion. While the sparse command generates only the values in row,column format, can it be applied to store banded matrices with the row column as the index of the block?
I have tried spalloc given in the this question but it hasnt helped storing the row and index of the block.
Thank you.
Image for A P A' formation
The problem lies in the blocks. The blocks are themselves sparse. So is it possible to make blocks as sparse matrices themselves while saving.
So, if a block has a row = 1 and col = 1, then can this be done?
N(row,col) = sparse(A'*P*A)
There may be some additional tricks to play but the first thing to try is to make sure the full matrix N is never created in memory. The immediate problem is that if you call sparse(A'*P*A) then you multiple A'*P then (A'*P)*A and only then do you make it sparse and take out the zeros. Right before making it sparse, the entire non-sparse matrix representation of N is in memory. To force MATLAB to be smarter do the following:
SA = sparse(A);
N = SA'*sparse(P)*SA;
whos N
You should see that N is sparse but, more importantly, each multiplication result is sparse as well because you are multiplying a sparse matrix times a sparse matrix.
preface: As the matlab guiderules state, Usually, when one wants to efficiently populate a sparse matrix in matlab, he should create a vector of indexes into the matrix and a vector of values he wants to assign, and then concentrate all the assignments into one atomic operation, so as to allow matlab to "prepare" the matrix in advance and optimize the assignment speed. A simple example:
A=sparse([]);
inds=some_index_generating_method();
vals=some_value_generating_method();
A(inds)=vals;
My question: what can I do in the case where inds contain overlapping indexes, i.e inds=[4 17 8 17 9] where 17 repeats twice.
In this case, what I would want to happen is that the matrix would be assigned the addition of all the values that are mapped to the same index, i.e for the previous example
A(17)=vals(2)+vals(4) %as inds(2)==inds(4)
Is there any straightforward and, most importantly, fast way to achieve this? I have no way of generating the indexes and values in a "smarter" way.
This might help:
S = sparse(i,j,s,m,n,nzmax) uses vectors i, j, and s to generate an m-by-n sparse matrix such that S(i(k),j(k)) = s(k), with space allocated for nzmax nonzeros. Vectors i, j, and s are all the same length. Any elements of s that are zero are ignored, along with the corresponding values of i and j. Any elements of s that have duplicate values of i and j are added together.
See more at MATLAB documentation for sparse function