I have a dataset in which I have a 25,000 by 25,000 matrix for each timepoint with around 60% of the cells being zeroes. I need to be able to take the eigenvalues of my matrices and reorder the columns and rows. Currently, I am running into memory errors and am considering implementing sparse matrices to save memory. Is my dataset sparse enough to do this? Would I simply use the 'sparse' and 'eigs' commands to manipulate these matrices? Thank you in advance!
Related
Suppose that for some needed transformation I have a mxn matrix A that consists of a few 1's and many 0's. If I were to transform a nx1 vector by A, is this considered a vectorized implementation?
While the matrix A mainly consists of 0's, does this still cause the same amount of FLOPs to occur?
Would it be wiser and more optimized to do the transformation needed another way? One that won't cause needless calculations such as 0*c?
The matrix is sparse. The number of operations for various things is lower when you store it properly and use the right routines.
i need to generate a large square binary sparse matrix in MATLAB (about 100k x 100k). but i get the "out of memory" error.
Can anybody help?
A 100,000 x 100,000 matrix contains 10,000,000,000 doubles. At 8 bytes each, that's 80,000,000,000 bytes, i.e. about 74.5058 Gb.
I seriously doubt you have 80Gb of RAM (let alone, allocated only to matlab), so presumably you'll have to find another way to process your data in chunks.
EDIT Apologies, I only just noticed the sparse bit.
If you try to initialise your sparse matrix as sparse (zeros( 100000,100000)), this will fail for the above reason (i.e. you're asking octave / matlab to first store a 75Gb matrix of zeros, and only then convert it to a sparse matrix).
Instead, you should initialise your 100,000x100,000 sparse matrix like so:
s = sparse(100000,100000);
and then proceed to fill in its contents.
Assuming the number of nonzero elements in your sparse matrix is low enough that they can be handled easily with your system's memory, and that you have a way of filling in the necessary values you have in mind without allocating a big bad matrix first, then this should work fine.
Have a look at the sparse function for other ways of initialising a sparse matrix from data.
Try increasing the size of the swap file of your system.
I have a sparse 5018x5018 matrix in MATLAB, which has about 100k values set to 1 (i.e., about 99.6% empty).
I'm trying to flip roughly 5% of those zeros to ones (i.e., about 1.25m entries). I have the x and y indices in the matrix I want to flip.
Here is what I have done:
sizeMat=size(network);
idxToReplace=sub2ind(sizeMat,x_idx, y_idx);
network(idxToReplace) = 1;
This is incredibly slow, in particular the last line. Is there any way to make this operation run noticeably faster, preferably without using mex files?
This should be faster:
idxToReplace=sparse(x_idx,y_idx,ones(size(x_idx),size(matrix,1),size(matrix,2)); % Create a sparse with ones at locations
network=network+idxToReplace; % Add the two matrices
I think your solution is very slow because you create a 1.26e6 logical array with your points and then store them in the sparse matrix. In my solution, you only create a sparse matrix and just sum the two.
Preamble:
My framework is Matlab. I have a very large data matrix M (size(M) = 30 20 30 20 51 300 ) and I need to manipulate this matrix (to calculate some correlations, mean, shift it circularly, interpolate it and so on).
!Important! : most of the elements of this matrix are zeros or ones !!
My question: Since it is very time consuming to work with such a huge matrix, is it possible to perform the same manipulations, but on the sparse form of this matrix? Of course, one should not loose any information about zeros or ones (for example, for calculations of averages or correlations between different elements).
Is there any other way to handle such matrices? (huge and mostly 0's and 1's)
Thanks in advance!
You can use sparse matrices.
The only issue with sparse matrices is, that they only come in two dimensions, so the straight-forward way to represent your matrix would be to wrap it into a sparse matrix, of size [N 1] where N = prod([ 30 20 30 20 51 300]) in your case.
I've done this for N-dimension histograms (which sounds similar to your application) and it works fine.
You'll lose the possibility to use all the smart indexing though.
So using mean/sum etc. on single dimensions will become somewhat more complicated since you'll have to convert subscript-indices to linear indices and vice versa.
For that, you should have a look at sub2ind and ind2sub.
(Sounds like a fun project on wrapping the builtin sparse matrix into a n-dimensional sparse matrix...)
Over the course of finite difference discretization of an elliptic equation
and application of Neumann BCs on all sides of the 2D domain I have a large
sparse matrix. I need to find null space of its transpose to enforce
consistency condition on either side. For a computational domain of 50X50
and 100X100 I am able to do this with the available RAM of 32 GB in
Mathematica and MATLAB with the available full matrix commands of NullSpace and null respectively.
If the computational domain is 500X250 (which is of
the order of what I would have in general) the RAM required for the storage of the
matrix of size (500X250)X(500X250) is 125 GB and is highly prohibitive. I use sparse matrices to store this super-matrix and I don't have have the space constraint anymore. But I can't use the 'null" command on this as it is for only full matrices. MATLAB suggests to use "SVDS" command on sparse matrices. SVDS(A) gives only the first 6 singular values and singular vectors. There is another command SVDS(A,k,sigma) which gives "k" singular values and vectors around the scalar singular value of "sigma". When I use sigma=0 so as to find the singular vector corresponding to the the "zero" value, which would be the vector from null space basis I get an error that the "sigma" is too close to the eigen value of the matrix.
My matrix is singular and hence one of its Eigen values is "zero". How do I circumvent this error? Or Is there a better way to do this with the tools at hand. I have both MATLAB and Mathematica at hand.
Thanks in advance for your help and suggestions.
Best
Trinath
I guess you can try with some sort of decomposition.
http://www.mathworks.co.uk/matlabcentral/fileexchange/11120-null-space-of-a-sparse-matrix
Have you tried this?
Or maybe this?
http://www.mathworks.co.uk/matlabcentral/newsreader/view_thread/249467
I am confident they should work, but I have not tried them myself. Another way of proceeding would be to get into QR decomposition (which would give you the permutation of the first k independent columns, if k is the rank of your matrix. Then the vectors from k+1 to n would provide a basis for your null space).
Hope this helps.
Cheers,
GL