I have a feature vector(FV1) of size 1*n. Now I subtract mean of all feature vectors from the feature vector FV1 Now I take transpose of that(FV1_Transpose) which is n*1. Now I add do matrix multiplication (FV1_Transpose * FV1) to get covariance matrix which is n*n.
But my problem is that I dont get a positive definite matrix. I read everywhere that covariance matrix should be symmetric positive definite.
FV1 after subtraction of mean = -17.7926788,0.814089298,33.8878059,-17.8336430,22.4685001;
Covariance matrix =
316.579407, -14.4848289, -602.954834, 317.308289, -399.774811
-14.4848289, 0.662741363, 27.5876999, -14.5181780, 18.2913647
-602.954834, 27.5876999, 1148.38342, -604.343018, 761.408142
317.308289, -14.5181780, -604.343018, 318.038818, -400.695221
-399.774811, 18.2913647, 761.408142, -400.695221, 504.833496
This covariance matrix is not positive definite. Any ideawhy is it so?
Thanks in advance.
Are you sure the matrix is not positive definite? I did the following in octave.
A = [ 316.579407, -14.4848289, -602.954834, 317.308289, -399.774811 -14.4848289, 0.662741363, 27.5876999, -14.5181780, 18.2913647 -602.954834, 27.5876999, 1148.38342, -604.343018, 761.408142 317.308289, -14.5181780, -604.343018, 318.038818, -400.695221 -399.774811, 18.2913647, 761.408142, -400.695221, 504.833496]
A = reshape(A, 5, 5)
svd(A)
The eigen values of A as obtained from svd were.
2.2885e+03
5.4922e-05
1.5958e-05
1.3636e-05
1.1507e-08
Please note that all the eigen values are positive.
Now, A is symmetric (being a co-variance matrix), To verify,
A - A'
would give you a 5 x 5 zero matrix
A symmetric matrix which has positive eigen values should be positive definite.
reference
Related
Given a random vector, Y=[y1,y2,...,yn];, its covariance matrix looks like this:
How can I calculate the covariance matrix in MATLAB?
the covariance matrix can be computed with the cov() function. But be aware: the covariance matrix of a vector will always be a 1-by-1 matrix, because there are no cross-variances in a single variable.
% random vector of length 10
vec = rand(10,1);
% covariance matrix
cov(vec)
I have a 1500x1500 covariance matrix of which I am trying to calculate the determinant for EM-ML method. The covariance matrix is obtained by finding the SIGMA matrix and then passing it into the nearestSPD library (Link) to make the matrix positive definite . In this case the matrix is always singular. Another method I tried was of manually generating a positive definite matrix using A'*A technique. (A was taken as a 1600x1500 matrix). This always gives me the determinant as infinite. Any idea on how I can get a positive definite matrix with a finite determinant?
Do you actually need the determinant, or the log of the determinant?
For example if you are computing a log likelihood of gaussians then what enters into the log likelihood is the log of the determinant. In high dimensions determinants mey not fit in a double, but its log most likely will.
If you perform a cholesky factorisation of the covariance C, with (lower triangular) factor L say so that
C = L*L'
then
det C = det(L) * det( L') = det(L) * det(L)
But the determinant of a lower triangular matrix is the product of its diagonal elements, so, taking logs above we get:
log det C = 2*Sum{ i | log( L[i,i])}
(In response to a comment)
Even if you need to calculate a gaussian pdf, it is better to calculate the log of that and exponentiate only when you need to. For example a d dimenions gaussian with covariance C (which has a cholesky factor L) and mean 0 (purely to save typing) is:
p(x) = exp( -0.5*x'*inv(C)*x) /( sqrt( pow(2pi,d) * det(C))
so
log p(x) = -0.5*x'*inv(C)*x - 0.5*d*log(2pi) - 0.5*log(det(C))
which can also be written
log p(x) = -0.5*y'*y - 0.5*d*log(2pi) - log(det(L))
where
y = inv(L)*x
Hi everybody I have this problem:
I have Dataset of n vectors each has D dimensions.
I also have a covariance matrix of size D*D, Let It be C.
I perform the following action:
I choose K vectors from the dataset, and also choose E dimensions randomly. Let M be the sample covariance of the selected data on the selected dimensions.so M is a E*E matrix.
let P be the partial covariance matrix corresponding to the dimensions E of C, ie. C(E,E) in matlab
is the following matrix positive semi definite?:
X = (1-a)P + aM
where a is a constant like 0.2.
I sometimes get the following error when using mvnrnd(mean,X) :
SIGMA must be a symmetric positive semi-definite matrix
My code is:
%%%Dims are randomly choosen dimensions
%%%Inds are randomly choosen Indexes form {1, 2, ...,n}
%%% PP are n D dimensional vectors, composing my data set PP is n*D
%%% Sigmaa is a D*D covariance matrix
co = cov(PP(Inds,Dims));
me = mean(PP(Inds,Dims));
Bettaa = 0.2;
sigmaaDims = sigmaa(Dims,Dims);
sigmaaDims = (1-Bettaa)*sigmaaDims + (co)*Bettaa;
Tem = mvnrnd(me,sigmaaDims);
Simply looking at the matrix dimensions It is not possible to tell if a matrix is positive semi-definite.
To find out if a given matrix is positive semi-definite, you must check if It's eigenvalues are non-negative and it's symmetry:
symmetry = issymmetric(X);
[~,D]=eig(X);
eigenvalues = diag(D);
if all(eigenvalues>0) & symmetry
disp('Positive semi-definite matrix.')
else
disp('Non positive semi-definite matrix.')
end
Where X is the matrix you are interested in.
Note that if you use the weaker definition of a positive definite matrix (see Extention for non symmetric matrices section), X does not need to be symmetric and you would end up with:
[~,D]=eig(X);
eigenvalues = diag(D);
if all(eigenvalues>=0)
disp('Positive semi-definite matrix.')
else
disp('Non positive semi-definite matrix.')
end
I'm trying to figure out Eigenvalues/Eigenvectors for large datasets in order to compute
the PCA. I can calculate the Eigenvalues and Eigenvectors for 2x2, 3x3 etc..
The problem is, I have a dataset containing 451x128 I compute the covariance matrix which
gives me 128x128 values from this. This, therefore looks like the following:
A = [ [1, 2, 3,
2, 3, 1,
..........,
= 128]
[5, 4, 1,
3, 2, 1,
2, 1, 2,
..........
= 128]
.......,
128]
Computing the Eigenvalues and vectors for a 128x128 vector seems really difficult and
would take a lot of computing power. However, if I allow for each of the blocks in A to be a 2-dimensional (3xN) I can then compute the covariance matrix which will give me a 3x3 matrix.
My question is this: Would this be a good or reasonable assumption for solving the eigenvalues and vectors? Something like this:
A is a 2-dimensional vector containing 128x451,
foreach of the blocks compute the eigenvalues and eigenvectors of the covariance vector,
like so:
Eig1 = eig(cov(A[0]))
Eig2 = eig(cov(A[1]))
This would then give me 128 Eigenvalues (for each of the blocks inside the 128x128 vector)..
If this is not correct, how does MATLAB handle such large dimensional data?
Have you tried svd()
Do the singular value decomposition
[U,S,V] = svd(X)
U and V are orthogonal matrices and S contains the eigen values. Sort U and V in descending order based on S.
As kkuilla mentions, you can use the SVD of the original matrix, as the SVD of a matrix is related to the Eigenvalues and Eigenvectors of the covariance matrix as I demonstrate in the following example:
A = [1 2 3; 6 5 4]; % A rectangular matrix
X = A*A'; % The covariance matrix of A
[V, D] = eig(X); % Get the eigenvectors and eigenvalues of the covariance matrix
[U,S,W] = svd(A); % Get the singular values of the original matrix
V is a matrix containing the eigenvectors, and D contains the eigenvalues. Now, the relationship:
SST ~ D
U ~ V
As to your own assumption, I may be misreading it, but I think it is false. I can't see why the Eigenvalues of the blocks would relate to the Eigenvalues of the matrix as a whole; they wouldn't correspond to the same Eigenvectors, as the dimensionality of the Eigenvectors wouldn't match. I think your covariances would be different too, but then I'm not completely clear on how you are creating these blocks.
As to how Matlab does it, it does use some tricks. Perhaps the link below might be informative (though it might be a little old). I believe they use (or used) LAPACK and a QZ factorisation to obtain intermediate values.
https://au.mathworks.com/company/newsletters/articles/matlab-incorporates-lapack.html
Use the word
[Eigenvectors, Eigenvalues] = eig(Matrix)
I have a 512x512x3 matrix that stores 512x512 there-dimensional vectors. What is the best way to normalize all those vectors, so that my result are 512x512 vectors with length that equals 1?
At the moment I use for loops, but I don't think that is the best way in MATLAB.
If the vectors are Euclidean, the length of each is the square root of the sum of the squares of its coordinates. To normalize each vector individually so that it has unit length, you need to divide its coordinates by its norm. For that purpose you can use bsxfun:
norm_A = sqrt(sum(A .^ 2, 3)_; %// Calculate Euclidean length
norm_A(norm_A < eps) == 1; %// Avoid division by zero
B = bsxfun(#rdivide, A, norm_A); %// Normalize
where A is your original 3-D vector matrix.
EDIT: Following Shai's comment, added a fix to avoid possible division by zero for null vectors.