I have a sparse matrix in Matlab 43916x43916, which is calculated by this equation:
B=(speye(nV,nV)-alpha*NeMatrix+beta*NeMatrix*NeMatrix);
being nVa int, alphaa int, NeMatrix a sparse matrix and beta a int.
I can't do inv(B) because it increases the use of RAM till it crashes. I've tried LU already with no success.
How can I alternatively calculate this inverse matrix?
The inverse will be a dense matrix. Thus you should check, whether you can handle a matrix of this size. Try, e.g., to set up ones(nV,nV) ...
If you have enough storage, you may consider to compute the inverse column wise. The i-th column would be B\ei, where ei is the i-th unit vector.
HOWEVER, in numerical computations you hardly ever need the inverse of a matrix B. Most times B\v is enough, where v is a vector. So you better check, whether you really need the full inverse...
Related
Consider A an nxn matrix. its not a special matrix and in the worst case all of its entries are non-zero. I am looking for a way to compute AA^T using matrix-vector operation. The total number of flops are (2n-1)*(n(n+1))/2 because all I have to do for a symmetric matrix like C=AA^T is to compute the diagonal entires which are C(i,i)=A(i,:)^T * A(i,:). What I want now is to compute the lower triangular part and then when I am done I just say that the upper triangular part is the same as the lower triangular part. The problem is can I do it in matrix-vector multiplication or will it force me to perform unnecessary multiplication (like multiplying elements in the upper part)? It is clear that a scalar computation would work but I am interested to know if a matrix-vector computation would work or not.
MATLAB is already smart enough to do this for you. When MATLAB encounters an expression like A*A.', it will recognize that the two operands are the same and will call a symmetric BLAS library function in the background to do the calculation. This symmetric function does exactly what you want ... only does about 1/2 the operations and generates an exact symmetric result.
I have a matrix like M = K x N ,where k is 49152 and is the dimension of the problem and N is 52 and is the number of observations.
I have tried to use [U,S,V]=SVD(M) but doing this I get less memory space.
I found another code which uses [U,S,V]=SVD(COV(M)) and it works well. My questions are what is the meaning of using the COV(M) command inside the SVD and what is the meaning of the resultant [U,S,V]?
Finding the SVD of the covariance matrix is a method to perform Principal Components Analysis or PCA for short. I won't get into the mathematical details here, but PCA performs what is known as dimensionality reduction. If you like a more formal treatise on the subject, you can read up on my post about it here: What does selecting the largest eigenvalues and eigenvectors in the covariance matrix mean in data analysis?. However, simply put dimensionality reduction projects your data stored in the matrix M onto a lower dimensional surface with the least amount of projection error. In this matrix, we are assuming that each column is a feature or a dimension and each row is a data point. I suspect the reason why you are getting more memory occupied by applying the SVD on the actual data matrix M itself rather than the covariance matrix is because you have a significant amount of data points with a small amount of features. The covariance matrix finds the covariance between pairs of features. If M is a m x n matrix where m is the total number of data points and n is the total number of features, doing cov(M) would actually give you a n x n matrix, so you are applying SVD on a small amount of memory in comparison to M.
As for the meaning of U, S and V, for dimensionality reduction specifically, the columns of V are what are known as the principal components. The ordering of V is in such a way where the first column is the first axis of your data that describes the greatest amount of variability possible. As you start going to the second columns up to the nth column, you start to introduce more axes in your data and the variability starts to decrease. Eventually when you hit the nth column, you are essentially describing your data in its entirety without reducing any dimensions. The diagonal values of S denote what is called the variance explained which respect the same ordering as V. As you progress through the singular values, they tell you how much of the variability in your data is described by each corresponding principal component.
To perform the dimensionality reduction, you can either take U and multiply by S or take your data that is mean subtracted and multiply by V. In other words, supposing X is the matrix M where each column has its mean computed and the is subtracted from each column of M, the following relationship holds:
US = XV
To actually perform the final dimensionality reduction, you take either US or XV and retain the first k columns where k is the total amount of dimensions you want to retain. The value of k depends on your application, but many people choose k to be the total number of principal components that explains a certain percentage of your variability in your data.
For more information about the link between SVD and PCA, please see this post on Cross Validated: https://stats.stackexchange.com/q/134282/86678
Instead of [U, S, V] = svd(M), which tries to build a matrix U that is 49152 by 49152 (= 18 GB 😱!), do svd(M, 'econ'). That returns the “economy-class” SVD, where U will be 52 by 52, S is 52 by 52, and V is also 52 by 52.
cov(M) will remove each dimension’s mean and evaluate the inner product, giving you a 52 by 52 covariance matrix. You can implement your own version of cov, called mycov, as
function [C] = mycov(M)
M = bsxfun(#minus, M, mean(M, 1)); % subtract each dimension’s mean over all observations
C = M' * M / size(M, 1);
(You can verify this works by looking at mycov(randn(49152, 52)), which should be close to eye(52), since each element of that array is IID-Gaussian.)
There’s a lot of magical linear algebraic properties and relationships between the SVD and EVD (i.e., singular value vs eigenvalue decompositions): because the covariance matrix cov(M) is a Hermitian matrix, it’s left- and right-singular vectors are the same, and in fact also cov(M)’s eigenvectors. Furthermore, cov(M)’s singular values are also its eigenvalues: so svd(cov(M)) is just an expensive way to get eig(cov(M)) 😂, up to ±1 and reordering.
As #rayryeng explains at length, usually people look at svd(M, 'econ') because they want eig(cov(M)) without needing to evaluate cov(M), because you never want to compute cov(M): it’s numerically unstable. I recently wrote an answer that showed, in Python, how to compute eig(cov(M)) using svd(M2, 'econ'), where M2 is the 0-mean version of M, used in the practical application of color-to-grayscale mapping, which might help you get more context.
I am using IPOPT in MATLAB to run an optimization and I am running into some issues where it says:
Hessian must be an n x n sparse, symmetric and lower triangular matrix
with row indices in increasing order, where n is the number of variables.
After looking at my Hessian Matrix, I found that the non-symmetric elements it is complaining about are very close, here is an example:
H(k,j) = 2.956404205984938
H(j,k) = 2.956404205984939
Obviously these elements are close enough and there are some numerical round-off issues or something of the like. Also, when I call MATLABs issymmetric function with H as an input, I get false. Is there a way to forget about these very small differences in symmetry?
A little more info:
I am using an optimized matlabFunction to actually calculate the entire hessian (H), then I did some postprocessing before passing it to IPOPT:
H = tril(H);
H = sparse(H);
The tril command generates a lower triangular matrix, so these numeral differences should not come into play. So, the issue might be that it is complaining that the sparse command passes back increasing column indices and not increasing row indices. Is there a way to change this so that it passes back the sparse matrix in increasing row indices?
If H is very close to symmetric but not quite, and you need to force it to be exactly symmetric, a standard way to do this would be to say H = (H+H')./2.
I have a matrix problem in Matlab.
I have a 1million x 1million sparse matrix, and I keep using null. Usually, the problem is that I run out of memory. I tried svds (which is used for svd for sparse matrix), but my issue is I run out of memory as well. Is there a possible work around for large sparse matrices for the null() function in Matlab?
In general, the null space of a matrix, or the unitary matrices (U and V) of the singular values decomposition are NOT sparse even if the input matrix is sparse. Therefore, if you try to work with a 1M-by-1M matrix, even though it is sparse, the outputs of your operations are NOT and therefore you run out of memory.
What can you do?
If your input matrix has a certain structure to it (in addition to its sparseness) you might find some algebraic method to take advantage of this structure.
Another path you should consider, is why you need to compute the null space of the matrix? Can you achieve the same goal without explicitly estimating the null space?
I have been doing this using an svd computation
[U, S, V] = svd(A)
wherein I use the last column of A as my null space approximation. Since A gets really large, I realized that this is slowing down my computation.
For null(A), the documentation seems to suggest that it does an SVD anyways. Also, it does not work if A is full rank. An SVD proceeds by finding the largest singular value, then the next one and so on whereas I just need the smallest one.
This seems to be a big bottleneck. Will really appreciate help on this.
Am using MATLAB.
Thanks.
This Wikipedia article describes three methods for the numerical computation of the null space: reduction (Gaussian elimination), SVD, and QR decomposition. In brief, (1) reduction is "not suitable for a practical computation of the null space because of numerical accuracy problems in the presence of rounding errors", (2) SVD is the "state-of-the art approach", but it "generally costs about the same as several matrix-matrix multiplications with matrices of the same size", and (3) the numerical stability and the cost of QR decomposition are "between those of the SVD and the reduction approaches".
So if SVD is too slow, you could give a chance to QR decomposition. The algorithm with your notations is as follows: "A is a 4xN matrix with 4<N. Using the QR factorization of A', we can find a matrix such that A'*P = Q*R = [Q1 Q2]*R, where where P is a permutation matrix, Q is NxN and R is Nx4. Matrix Q1 is Nx4 and consists of the first 4 columns of Q. Matrix Q2 is Nx(N-4) and is made up of the last N-4 columns of Q. Since A*Q2 = 0, the columns of Q2 span the null space of A."
Matlab implementation: [Q, R, P] = qr(A', 'matrix'); The columns of matrix Q2 = Q(:, 5:end); give the null space of A.
This answers builds on your comment that what you actually want to do is to solve Ax = 0. For this purpose, a complete nullspace computation is usually inefficient. If you want a least-squares approximation to x, have a look into the matlab operator \ (see help mldivide).
In other cases, an "economic" SVD via svd(A,0) might be helpful for non-square matrices (it does not compute the full S, but only the non-zero block).
If all points are from a plane, call SVD with just a sample.