i wanted to apply HOTELLING TRANSFORMATION to the give vectors and to make practice my self, that why i have wrote following code in matlab
function [Y covariance_matrix]=hotteling_trasform(X)
% this function take X1,X2,X3,,Xn as a matrix and apply hottleing
%transformation to get new set of vectors y1, y2,..ym so that covariance
%matrix of matrix consiist by yi vectors are almost diagonal
%% determine size of given matrix
[m n]=size(X);
%% compute mean of columns of given matrix
means=mean(X);
%% substract mean from given matrix
centered=X-repmat(means,m,1);
%% calculate covariance matrix
covariance=(centered'*centered)/(m-1);
%% Apply eigenvector decomposition
[V,D]=eig(covariance);
%% determine dimension of V
[m1 n1]=size(V);
%% arrange matrix so that eigenvectors are as rows,create matrix with size n1 m1
A1=zeros(n1,m1);
for ii=1:n1
A1(ii,:)=V(:,ii);
end
%% applying hoteling transformation
Y=A1*centered; %% because centered matrix is original -means
%% calculate covariance matrix
covariance_matrix=cov(Y);
then i have tested it to the given matrix
A
A =
4 6 10
3 10 13
-2 -6 -8
and after running code
[Y covariance_matrix]=hotteling_trasform(A);
covariance_matrix
covariance_matrix =
8.9281 22.6780 31.6061
22.6780 66.5189 89.1969
31.6061 89.1969 120.8030
definitely this is not diagonal matrix, so what is wrong? thanks in advance
As you're dealing with row vectors instead of column vectors you need to adjust for it in the eigenvalue/eigenvector-decomposiiton. Instead of Y=A1*centered you need Y=centered*V. Then you'll get
covariance_matrix =
0.0000 -0.0000 0.0000
-0.0000 1.5644 -0.0000
0.0000 -0.0000 207.1022
So you'll get two nonzero components which is what you could expect from only three points in the 3D-space. (They can only form a plane, but not a volume.)
Related
How can I implement the function lu(A) in MATLAB so that L*U is directly A and I also get the real L matrix?
When I use [L,U] = lu(A), MATLAB doesn't give me the right L matrix. When I use [L,U,P] = lu(A), I need to implement P*A = L*U, but I only want to multiply L*U to receive A.
MATLAB's lu always performs pivoting by default. If you had for example a diagonal coefficient that was equal to 0 when you tried to do the conventional LU decomposition algorithm, it will not work as the diagonal coefficients are required when performing the Gaussian elimination to create the upper triangular matrix U so you would get a divide by zero error. Pivoting is required to ensure that the decomposition is stable.
However, if you can guarantee that the diagonal coefficients of your matrix are non-zero, it is very simple but you will have to write this on your own. All you have to do is perform Gaussian elimination on the matrix and reduce the matrix into reduced echelon form. The result reduced echelon form matrix is U while the coefficients required to remove the lower triangular part of L in Gaussian elimination would be placed in the lower triangular half to make U.
Something like this could work, assuming your matrix is stored in A. Remember that I'm assuming a square matrix here. The implementation of the non-pivoting LU decomposition algorithm is placed in a MATLAB function file called lu_nopivot:
function [L, U] = lu_nopivot(A)
n = size(A, 1); % Obtain number of rows (should equal number of columns)
L = eye(n); % Start L off as identity and populate the lower triangular half slowly
for k = 1 : n
% For each row k, access columns from k+1 to the end and divide by
% the diagonal coefficient at A(k ,k)
L(k + 1 : n, k) = A(k + 1 : n, k) / A(k, k);
% For each row k+1 to the end, perform Gaussian elimination
% In the end, A will contain U
for l = k + 1 : n
A(l, :) = A(l, :) - L(l, k) * A(k, :);
end
end
U = A;
end
As a running example, suppose we have the following 3 x 3 matrix:
>> rng(123)
>> A = randi(10, 3, 3)
A =
7 6 10
3 8 7
3 5 5
Running the algorithm gives us:
>> [L,U] = lu_nopivot(A)
L =
1.0000 0 0
0.4286 1.0000 0
0.4286 0.4474 1.0000
U =
7.0000 6.0000 10.0000
0 5.4286 2.7143
0 0 -0.5000
Multiplying L and U together gives:
>> L*U
ans =
7 6 10
3 8 7
3 5 5
... which is the original matrix A.
You could use this hack (though as already mentioned, you might lose numerical stability):
[L, U] = lu(sparse(A), 0)
You might want to consider doing LDU decomposition instead of unpivoted LU. See, LU without pivoting is numerically unstable - even for matrices that are full rank and invertible. The simple algorithm provided above shows why - there is division by each diagonal element of the matrix involved. Thus, if there is a zero anywhere on the diagonal, decomposition fails, even though the matrix could still be non-singular.
Wikipedia talks a little about LDU decomposition here:
https://en.wikipedia.org/wiki/LU_decomposition#LDU_decomposition
without citing an algorithm. It cites the following textbook for proof of existence:
Horn, Roger A.; Johnson, Charles R. (1985), Matrix Analysis, Cambridge University Press, ISBN 978-0-521-38632-6. See Section 3.5.
LDU is guaranteed to exist (at least for an invertible matrix), it is numerically stable, and it is also unique (provided that both L and U are constrained to have unit elements on the diagonal).
Then, if for any reason "D" gets in your way, you can absorb the diagonal matrix D into either L (L:=LD) or U (U:=DU), or split it symmetrically between L and U (such as L:=L*sqrt(D) and U:=sqrt(D)*U), or however you want to do it. There is an infinite number of ways to split LDU into LU, and this is why LU decomposition is not unique.
I am trying to implement Principal Component Analysis (PCA) to extract the features from the image in MATLAB. I have implemented the following code.
[Rows, Columns] = size(x); % find size of input matrix
m=mean(x); % find mean of input matrix
y=x-ones(size(x,1),1)*m; % normalise by subtracting mean
c=cov(y); % find covariance matrix
[V,D]=eig(c); % find eigenvectors (V) and eigenvalues (D) of covariance matrix
[D,idx] = sort(diag(D)); % sort eigenvalues in descending order by first diagonalising eigenvalue matrix, idx stores order to use when ordering eigenvectors
D = D(end:-1:1)';
V = V(:,idx(end:-1:1)); % put eigenvectors in order to correspond with eigenvalues
V2d=V(:,1:200); % (significant Principal Components we use, OutputSize is input variable)
prefinal=V2d'*y';
final=prefinal'; % final is normalised data projected onto eigenspace
imshow(final);
I want to know that how can I check the 1st Eigen faces,2nd Eigen faces.. etc
EDIT:
Here is the Input Image and the Eigen Image is Eigen Image
The first eigenface is the first eigenvector!
My guess, is that with your code:
eigenface1=reshape(V(:,1),rows,cols);
as, if your code is right, each eigenvector should be the same size as your input images, but unrolled. I am assuming that rows and cols are the size of the image.
I am doing some Fourier transforms of audio (.wav) data using the FFT command in MATLAB. The input values are numbers between -1.0 and 1.0
My understanding is that after taking the absolute value (modulus) of the output of the FFT, I should get values that have units of amplitude, but the actual values are on the order of thousands. This doesn't make sense as theoretically I should be able to sum the Fourier components to get the original signal back. I feel like the output should then also be between 0 and 1, so what's up here? My guess is that the FFT algorithm blows it out of proportion but I'm not sure what value to use to scale it back.
The FFT is an algorithm for computing the Discrete Fourier Transform (DFT). The inverse DFT (IDFT) has a 1/N scaling factor in its definition. Perhaps that's what's confusing you. From the Wikipedia:
DFT (from finite sequence x to Fourier coefficients X):
IDFT (from X back to x):
So, just apply ifft to the result of fft and you'll get the original result. For example:
>> x = linspace(-1,1,5)
x =
-1.0000 -0.5000 0 0.5000 1.0000
>> y = fft(x)
y =
0 -1.2500 + 1.7205i -1.2500 + 0.4061i -1.2500 - 0.4061i -1.2500 - 1.7205i
>> abs(y)
ans =
0 2.1266 1.3143 1.3143 2.1266 %// note values greater than 1
>> ifft(y)
ans =
-1.0000 -0.5000 0.0000 0.5000 1.0000
In fact, the IDFT can be expressed in terms of the DFT applying complex conjugation and the referred scaling factor. Denoting the DFT by F, the IDFT by F-1 and complex conjugate by *,
In the above example,
>> 1/numel(y) * conj(fft(conj(y)))
ans =
-1.0000 -0.5000 0.0000 0.5000 1.0000
In Matlab use the following code to scale from 1 to (roughly) 0.
dataDFT=abs(fft(data)); % Take the complex magnitude of the fft of your data
dataDFTScaled=dataDFT/max(dataDFT); % Divide by the maximum value
You don't want it to scale to zero because that would make it impossible to view on a log plot.
I have this RGB matrix of a set of different pixels. (N pixels => n rows, RGB => 3 columns). I have to calculate the minimum RGB distance between any two pixels from this matrix. I tried the loop approach, but because the set is too big (let's say N=24000), it looks like it will take forever for the program to finish. Is there another approach? I read about pdist, but the RGB Euclidean distance cannot be used with it.
k=1;
for i = 1:N
for j = 1:N
if (i~=j)
dist_vect(k)=RGB_dist(U(i,1),U(j,1),U(i,2),U(j,2),U(i,3),U(j,3))
k=k+1;
end
end
end
Euclidean distance between two pixels:
So, pdist syntax would be like this: D=pdist2(U,U,#calc_distance());, where U is obtained like this:
rgbImage = imread('peppers.png');
rgb_columns = reshape(rgbImage, [], 3)
[U, m, n] = unique(rgb_columns, 'rows','stable');
But if pdist2 does the loops itself, how should I enter the parameters for my function?
function[distance]=RGB_dist(R1, R2, G1, G2, B1, B2),
where R1,G1,B1,R2,G2,B2 are the components of each pixel.
I made a new function like this:
function[distance]=RGB_dist(x,y)
distance=sqrt(sum(((x-y)*[3;4;2]).^2,2));
end
and I called it D=pdist(U,U,#RGB_dist); and I got 'Error using pdist (line 132)
The 'DISTANCE' argument must be a
string or a function.'
Testing RGB_dist new function alone, with these input set
x=[62,29,64;
63,31,62;
65,29,60;
63,29,62;
63,31,62;];
d=RGB_dist(x,x);
disp(d);
outputs only values of 0.
Contrary to what your post says, you can use the Euclidean distance as part of pdist. You have to specify it as a flag when you call pdist.
The loop you have described above can simply be computed by:
dist_vect = pdist(U, 'euclidean');
This should compute the L2 norm between each unique pair of rows. Seeing that your matrix has a RGB pixel per row, and each column represents a single channel, pdist should totally be fine for your application.
If you want to display this as a distance matrix, where row i and column j corresponds to the distance between a pixel in row i and row j of your matrix U, you can use squareform.
dist_matrix = squareform(dist_vect);
As an additional bonus, if you want to find which two pixels in your matrix share the smallest distance, you can simply do a find search on the lower triangular half of dist_matrix. The diagonals of dist_matrix are going to be all zero as any vector whose distance to itself should be 0. In addition, this matrix is symmetric and so the upper triangular half should be equal to the lower triangular half. Therefore, we can set the diagonal and the upper triangular half to Inf, then search for the minimum for those elements that are remaining. In other words:
indices_to_set = true(size(dist_matrix));
indices_to_set = triu(indices_to_set);
dist_matrix(indices_to_set) = Inf;
[v1,v2] = find(dist_matrix == min(dist_matrix(:)), 1);
v1 and v2 will thus contain the rows of U where those RGB pixels contained the smallest Euclidean distance. Note that we specify the second parameter as 1 as we want to find just one match, as what your post has stated as a requirement. If you wish to find all vectors who match the same distance, simply remove the second parameter 1.
Edit - June 25th, 2014
Seeing as how you want to weight each component of the Euclidean distance, you can define your own custom function to calculate distances between two RGB pixels. As such, instead of specifying euclidean, you can specify your own function which can calculate the distances between two vectors within your matrix by calling pdist like so:
pdist(x, #(XI,XJ) ...);
#(XI,XJ)... is an anonymous function that takes in a vector XI and a matrix XJ. For pdist you need to make sure that the custom distance function takes in XI as a 1 x N vector which is a single row of pixels. XJ is then a M x N matrix that contains multiple rows of pixels. As such, this function needs to return a M x 1 vector of distances. Therefore, we can achieve your weighted Euclidean distance as so:
weights = [3;4;2];
weuc = #(XI, XJ, W) sqrt(bsxfun(#minus, XI, XJ).^2 * W);
dist_matrix = pdist(double(U), #(XI, XJ) weuc(XI, XJ, weights));
bsxfun can handle that nicely as it will replicate XI for as many rows as we need to, and it should compute this single vector with every single element in XJ by subtracting. We thus square each of the differences, weight by weights, then take the square root and sum. Note that I didn't use sum(X,2), but I used vector algebra to compute the sum. If you recall, we are simply computing the dot product between the square distance of each component with a weight. In other words, x^{T}y where x is the square distance of each component and y are the weights for each component. You could do sum(X,2) if you like, but I find this to be more elegant and easy to read... plus it's less code!
Now that I know how you're obtaining U, the type is uint8 so you need to cast the image to double before we do anything. This should achieve your weighted Euclidean distance as we talked about.
As a check, let's put in your matrix in your example, then run it through pdist then squareform
x=[62,29,64;
63,31,62;
65,29,60;
63,29,62;
63,31,62];
weights = [3;4;2];
weuc = #(XI, XJ, W) sqrt(bsxfun(#minus,XI,XJ).^2 * W);
%// Make sure you CAST TO DOUBLE, as your image is uint8
%// We don't have to do it here as x is already a double, but
%// I would like to remind you to do so!
dist_vector = pdist(double(x), #(XI, XJ) weuc(XI, XJ, weights));
dist_matrix = squareform(dist_vector)
dist_matrix =
0 5.1962 7.6811 3.3166 5.1962
5.1962 0 6.0000 4.0000 0
7.6811 6.0000 0 4.4721 6.0000
3.3166 4.0000 4.4721 0 4.0000
5.1962 0 6.0000 4.0000 0
As you can see, the distance between pixels 1 and 2 is 5.1962. To check, sqrt(3*(63-62)^2 + 4*(31-29)^2 + 2*(64-62)^2) = sqrt(3 + 16 + 8) = sqrt(27) = 5.1962. You can do similar checks among elements within this matrix. We can tell that the distance between pixels 5 and 2 is 0 as you have made these rows of pixels the same. Also, the distance between each of themselves is also 0 (along the diagonal). Cool!
Im going to do dimensionality reduction by using PCA/SVD for my extracted features.
Suppose if I want to do classification using SIFT as the features and SVM as the classifier.
I have 3 images for training and I arrange them in a different row..
So 1st row for 1st images, 2nd row for second images and 3rd row for the 3rd image...
And the columns represents the features
A=[ 1 2 3 4 5
4 5 6 7 8
0 0 1 9 0]
To do dimensionality reduction (from my 3x5 matrix/non square matrix), we have to do A*EigenVector
Now I have to extract the eigen vector from my A matrix And from what I understand, that SVD is for non square matrix, so to perform PCA (eigs function), I need to make it square by multiplying it with it's transpose)
Here is if I do SVD directly
[u1,s1,v1] = svd(A);
I got
u1 =
-0.4369 0.1426 0.8882
-0.8159 0.3530 -0.4580
-0.3788 -0.9247 -0.0379
v1 =
-0.2229 0.2206 -0.7088 -0.6070 -0.1754
-0.2984 0.2910 -0.3857 0.4705 0.6754
-0.3966 0.2301 -0.0910 0.5382 -0.7012
-0.6547 -0.7495 0.0045 -0.0598 0.0779
-0.5248 0.5020 0.5836 -0.3419 0.1233
and when I use PCA (eigs function) {as I arrange the feat of images in different row, so I need to do A*A'}, I got
c=A*A'
[e1 e2]=eigs(c);
e1 =
0.4369 0.1426 0.8882
0.8159 0.3530 -0.4580
0.3788 -0.9247 -0.0379
My question is:
is that right that I used it in SVD or in the PCA (by converting t into A*A' matrix) will give me he same eig vectoe (e1 and u1)??
As I arrange my images in different rows and the features for each images in different column. and PCA/SVD is suing to extract the eig vector which represent the relation between the variable.. So in this case the variable would be the row (images) or the columns (features)??
Do I have to convert my matrix into covariance matrix by using (Cov function) if I want to use eigs function?? Or it will be done by eigs function manually??
I do really appriciate any answer
Suppose you have n-dimension samples and you want to reduce it to d-dimension data by PCA.
Suppose your data are stored in matrix AnxN (N is the number of samples(here images)).
here n=3 and N=5.
We define a correlation matrix R = A*A' (nxn). You can use the covariance matrix instead.
Calculate the eignen vectors of R and corresponding eigen values:
R = A*A';
[eigVec, eigVal] = eig(R)
eigVec =
0.8882 0.1426 0.4369
-0.4580 0.3530 0.8159
-0.0379 -0.9247 0.3788
eigVal =
1.7728 0 0
0 49.6457 0
0 0 275.5815
Note that columns of eigVec are the eigen vectors of R.
Some of the eigen values will be zero or if not, you can take a threshold. So you can eliminate the corresponding eigen vectors:
T = eigVec(:, 2:3)
T =
0.1426 0.4369
0.3530 0.8159
-0.9247 0.3788
now T is a nxd matrix.
For any row vector X1xn the X1xn*Tnxd will result a Y1xd output.
The final answer;
B = A'*T;