How to display the similarity between classes? - matlab

Question

I have handwritten samples from two writers. I am using a feature extractor to extract features from both.
I want to display the similarity between the classes. As to show how identical both are and how difficult it can be for a classifier to classify them correctly.
I have read papers that use PCA to demonstrate this. I tried with PCA but I dont think I'm implementing this correctly. I'm using this to display the similarity.
[COEFF,SCORE] = princomp(features_extracted);
plot(COEFF,'.')
But for every class and every sample I get exactly the same plot. I mean they should be similar not exactly the same. What am I doing wrong?

Answer 1

You will struggle to show anything significant with only 10 samples per class, and over 4000 features.
Nevertheless, the following code will calculate PCA and show the relationship between the first two principal components (the components that contain 'most' variance).
% Truly indistinguishable data
dummy_data = randn(20, 4000);
% Uncomment this to make the data distinguishable
%dummy_data(1:10, :) = dummy_data(1:10, :) - 0.5;
% Normalise the data - this isn't technically required for the dummy data
% above, but is included for completeness.
dummy_data_normalised = dummy_data;
for f = 1:size(a, 2)
dummy_data_normalised(:, f) = dummy_data_normalised(:, f) - nanmean(dummy_data_normalised(:, f));
dummy_data_normalised(:, f) = dummy_data_normalised(:, f) / nanstd(dummy_data_normalised(:, f));
end
% Generate vector of 10 0's and 10 1's
class_labels = reshape(repmat([0 1], 10, 1), 20, 1);
% Perform PCA
pca_coeffs = pca(dummy_data_normalised);
% Calculate transformed data
dummy_data_pca = dummy_data_normalised * pca_coeffs;
figure;
hold on;
for class = unique(class_labels)'
% Plot first two components of first class
scatter(dummy_data_pca(class_labels == class, 1), dummy_data_pca(class_labels == class, 2), 'filled')
end
legend(strcat({'Class '},int2str(unique(class_labels)))')
For indistinguishable data, this will show a scatter plot similar to the following:
Clearly it is not possible to draw a separation boundary between the two classes.
If you uncomment the fifth line to make the data distinguishable, then the plot will instead come out as follows:
However, to repeat what I wrote in my comment, PCA does not necessarily find the components that give the best separation. It is an unsupervised method and only finds the components with the largest variance. In some applications, this is also the components that give good separation. With only 10 samples per class, you will not be able to demonstrate anything statistically significant. Also have a look at this question for more details on PCA and the number of samples per class.
EDIT: This also extends naturally to having more classes:
numer_of_classes = 10;
samples_per_class = 20;
% Truly indistinguishable data
dummy_data = randn(numer_of_classes * samples_per_class, 4000);
% Make the data distinguishable
for i = 1:numer_of_classes
ixd = (((i - 1) * samples_per_class) + 1):(i * samples_per_class);
dummy_data(ixd, :) = dummy_data(ixd, :) - (0.5 * (i - 1));
end
% Normalise the data
dummy_data_normalised = dummy_data;
for f = 1:size(a, 2)
dummy_data_normalised(:, f) = dummy_data_normalised(:, f) - nanmean(dummy_data_normalised(:, f));
dummy_data_normalised(:, f) = dummy_data_normalised(:, f) / nanstd(dummy_data_normalised(:, f));
end
% Generate vector of classes (1 to numer_of_classes)
class_labels = reshape(repmat(1:numer_of_classes, samples_per_class, 1), numer_of_classes * samples_per_class, 1);
% Perform PCA
pca_coeffs = pca(dummy_data_normalised);
% Calculate transformed data
dummy_data_pca = dummy_data_normalised * pca_coeffs;
figure;
hold on;
for class = unique(class_labels)'
% Plot first two components of first class
scatter(dummy_data_pca(class_labels == class, 1), dummy_data_pca(class_labels == class, 2), 'filled')
end
legend(strcat({'Class '},int2str(unique(class_labels)))')