preface: As the matlab guiderules state, Usually, when one wants to efficiently populate a sparse matrix in matlab, he should create a vector of indexes into the matrix and a vector of values he wants to assign, and then concentrate all the assignments into one atomic operation, so as to allow matlab to "prepare" the matrix in advance and optimize the assignment speed. A simple example:
A=sparse([]);
inds=some_index_generating_method();
vals=some_value_generating_method();
A(inds)=vals;
My question: what can I do in the case where inds contain overlapping indexes, i.e inds=[4 17 8 17 9] where 17 repeats twice.
In this case, what I would want to happen is that the matrix would be assigned the addition of all the values that are mapped to the same index, i.e for the previous example
A(17)=vals(2)+vals(4) %as inds(2)==inds(4)
Is there any straightforward and, most importantly, fast way to achieve this? I have no way of generating the indexes and values in a "smarter" way.
This might help:
S = sparse(i,j,s,m,n,nzmax) uses vectors i, j, and s to generate an m-by-n sparse matrix such that S(i(k),j(k)) = s(k), with space allocated for nzmax nonzeros. Vectors i, j, and s are all the same length. Any elements of s that are zero are ignored, along with the corresponding values of i and j. Any elements of s that have duplicate values of i and j are added together.
See more at MATLAB documentation for sparse function
Related
I have a sparse 5018x5018 matrix in MATLAB, which has about 100k values set to 1 (i.e., about 99.6% empty).
I'm trying to flip roughly 5% of those zeros to ones (i.e., about 1.25m entries). I have the x and y indices in the matrix I want to flip.
Here is what I have done:
sizeMat=size(network);
idxToReplace=sub2ind(sizeMat,x_idx, y_idx);
network(idxToReplace) = 1;
This is incredibly slow, in particular the last line. Is there any way to make this operation run noticeably faster, preferably without using mex files?
This should be faster:
idxToReplace=sparse(x_idx,y_idx,ones(size(x_idx),size(matrix,1),size(matrix,2)); % Create a sparse with ones at locations
network=network+idxToReplace; % Add the two matrices
I think your solution is very slow because you create a 1.26e6 logical array with your points and then store them in the sparse matrix. In my solution, you only create a sparse matrix and just sum the two.
I am filling a sparse matrix P (230k,290k) with values coming from a text file which I read line by line, here is the (simplified) code
while ...
C = textscan(text_line,'%d','delimiter',',','EmptyValue', 0);
line_number = line_number+1;
P(line_number,:)=C{1};
end
the problem I have is that while at the beginning the
P(line_number,:)=C{1};
statement is fast, after a few thousands lines become exterely slow, I guess because Matlab need to find the memory space to allocate every time. Is there a way to pre-allocate memory with sparse matrixes? I don't think so but maybe I am missing something. Any other advise which can speed up the operation (e.g. having a lot of free RAM can make the difference?)
There's a sixth input argument to sparse that tells the number of nonzero elements in the matrix. That's used by Matlab to preallocate:
S = sparse(i,j,s,m,n,nzmax) uses vectors i, j, and s to generate an
m-by-n sparse matrix such that S(i(k),j(k)) = s(k), with space
allocated for nzmax nonzeros.
So you could initiallize with
P = sparse([],[],[],230e3,290e3,nzmax);
You can make a guess about the number of nonzeros (perhaps checking file size?) and use that as nzmax. If it turns you need more nonzero elements in the end, Matlab will preallocate on the fly (slowly).
By far the fastest way to generate a sparse matrix wihtin matlab is to load all the values in at once, then generate the sparse matrix in one call to sparse. You have to load the data and arrange it into vectors defining the row and column indices and values for each filled cell. You can then call sparse using the S = sparse(i,j,s,m,n) syntax.
I have to perform this operation:
N = A'*P*A
The structure of the P matrix is block diagonal while the A matrix is largely sparse (also in a banded structure). The multiplication is performed in blocks. But the problem is storage.
The N matrix is too huge to store in full (out of memory when trying to allocate). So, I want to store in a sparse fashion. While the sparse command generates only the values in row,column format, can it be applied to store banded matrices with the row column as the index of the block?
I have tried spalloc given in the this question but it hasnt helped storing the row and index of the block.
Thank you.
Image for A P A' formation
The problem lies in the blocks. The blocks are themselves sparse. So is it possible to make blocks as sparse matrices themselves while saving.
So, if a block has a row = 1 and col = 1, then can this be done?
N(row,col) = sparse(A'*P*A)
There may be some additional tricks to play but the first thing to try is to make sure the full matrix N is never created in memory. The immediate problem is that if you call sparse(A'*P*A) then you multiple A'*P then (A'*P)*A and only then do you make it sparse and take out the zeros. Right before making it sparse, the entire non-sparse matrix representation of N is in memory. To force MATLAB to be smarter do the following:
SA = sparse(A);
N = SA'*sparse(P)*SA;
whos N
You should see that N is sparse but, more importantly, each multiplication result is sparse as well because you are multiplying a sparse matrix times a sparse matrix.
I have to two evenly sized very large vectors (columns) A and B. I would like to divide vector A by vector B. This will give me a large matrix AxB filled with zeros, except the last column. This column contains the values I'm interested in. When I simple divide the vectors in a Matlab script, I run out of memory. Probably because the matrix AxB becomes very large. Probably I can prevent this from happening by repeating the following:
calculating the first row of matrix AxB
filter the last value and put it into another vector C.
delete the used row of matrix AxB
redo step 1-4 for all rows in vector A
How can I make a loop which does this?
You're question doesn't make it clear what you are trying to do, although it sounds like you want to do an element wise division.
Try:
C = A./B
"Matrix product AxB" and "dividing vectors" are distinct operations.
If we understood this correctly, what you do want to calculate is "C = last column from AxB", such that:
lastcolsel=zeros(size(B,2),1)
C=(A*B)*lastcolsel
If that code breaks your memory limit, recall that matrix product is associative (MxN)xP = Mx(NxP). Simplifying your example, we get:
lastcolsel=zeros(size(B,2),1)
simplifier=B*lastcolsel
C=A*simplifier
I have to create a very big 3D matrix (such as: 500000x60x60). Is there any way to do this in matlab?
When I try
omega = zeros(500000,60,60,'single');
I get an out-of-memory error.
The sparse function is no option since it is only meant for 2D matrices. So is there any alternative to that for higher dimensional matrices?
Matlab only has support for sparse matrices (2D). For 3D tensors/arrays, you'll have to use a workaround. I can think of two:
linear indexing
cell arrays
Linear indexing
You can create a sparse vector like so:
A = spalloc(500000*60*60, 1, 100);
where the last entry (100) refers to the amount of non-zeros eventually to be assigned to A. If you know this amount beforehand it makes memory usage for A more efficient. If you don't know it beforehand just use some number close to it, it'll still work, but A can consume more memory in the end than it strictly needs to.
Then you can refer to elements as if it is a 3D array like so:
A(sub2ind(size(A), i,j,k))
where i, j and k are the indices to the 1st, 2nd and 3rd dimension, respectively.
Cell arrays
Create each 2D page in the 3D tensor/array as a cell array:
a = cellfun(#(x) spalloc(500000, 60, 100), cell(60,1), 'UniformOutput', false);
The same story goes for this last entry into spalloc. Then concatenate in 3D like so:
A = cat(3, a{:});
then you can refer to individual elements like so:
A{i,j,k}
where i, j and k are the indices to the 1st, 2nd and 3rd dimension, respectively.
Since your matrix is sparse, try to use ndsparse (N-dimensional sparse arrays FEX)