I would like to perform simple LDA on my small data set (65x8). I have 65 instances (samples) , 8 features (attributes) and 4 classes. Any matlab code for LDA , as I know Matlab Toolbox does not have LDA function So I need to write own code. Any help?
I find on web this code
load /Data;
All_data= Data(:,1:8);
All_data_label= Data(:,9);
testing_ind = [];
for i = 1:length(Data)
if rand>0.8
testing_ind = [testing_ind, i];
end
end
training_ind = setxor(1:length(Data), testing_ind);
[ldaClass,err,P,logp,coeff] = classify(Data(testing_ind,:),...
Data((training_ind),:),Data_label(training_ind,:),'linear');
[ldaResubCM,grpOrder] = confusionmat(All_data_label(testing_ind,:),ldaClass)
Then I got this results
ldaClass =
3
2
3
2
1
4
3
3
1
2
1
1
2
err =
0.2963
P =
0.0001 0.0469 0.7302 0.2229
0.1178 0.5224 0.3178 0.0419
0.0004 0.2856 0.4916 0.2224
0.0591 0.6887 0.1524 0.0998
0.8327 0.1637 0.0030 0.0007
0.0002 0.1173 0.3897 0.4928
0.0000 0.0061 0.7683 0.2255
0.0000 0.0241 0.5783 0.3976
0.9571 0.0426 0.0003 0.0000
0.2719 0.5569 0.1630 0.0082
0.9999 0.0001 0.0000 0.0000
0.9736 0.0261 0.0003 0.0000
0.0842 0.6404 0.2634 0.0120
coeff =
4x4 struct array with fields:
type
name1
name2
const
linear
ldaResubCM =
4 0 0 0
0 3 1 0
0 1 1 0
0 0 2 1
grpOrder =
1
2
3
4
So I have 65 Instances, 8 Attributes and 4 classes (1,2,3,4). So dont know how to interpret these results. Any help?
The interpretation of the results derives directly from the documentation of classify.
classify trains a classifier based on the training data and labels (second and third argument), and applies the classifier to the test data (first argument).
ldaClass gives the classes chosen for the test data points, based on the classifier that has been trained using the training data points and labels.
err is the training error rate, the fraction of training data points that are incorrectly classified using the classifier which was trained using that data. The training error rate underestimates the error to be expected on independent test data.
P gives the posterior probabilities. I.e. for each test data point (rows) it gives for each class (columns) the probability that the data point belongs to that class. Probabilities sum to 1 across classes (for each row). The definite classification in ldaClass derives from the posterior probabilities such that for each test data point the class with the highest probability is chosen: [~, ind] = max(P') results in ind = ldaClass'.
coeff contains details about the trained classifier. In order to use this, you have to study in detail how the classifier works.
confusionmat compares the classes assigned by the classifier to the test data with the known true classes, and makes a table of the results, a confusion matrix. Each row corresponds to the true class of a test data point, each column to the class assigned by the classifier. Numbers on the diagonal indicate correct classifications; in your result, you have a test error of 1 - sum(diag(confusionmat)) / sum(confusionmat(:)) of 0.308. In particular, the confusion matrix shows you that of the 4 test data points that belong to class two, three have been classified correctly and 1 incorrectly (as belonging to class three).
grpOrder just gives the explicit class labels for the four classes numbered 1 to 4; in your case, indices and labels are identical.
Related
Could you give an example of classification of 4 classes using Support Vector Machines (SVM) in matlab something like:
atribute_1 atribute_2 atribute_3 atribute_4 class
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3
SVMs were originally designed for binary classification. They have then been extended to handle multi-class problems. The idea is to decompose the problem into many binary-class problems and then combine them to obtain the prediction.
One approach called one-against-all, builds as many binary classifiers as there are classes, each trained to separate one class from the rest. To predict a new instance, we choose the classifier with the largest decision function value.
Another approach called one-against-one (which I believe is used in LibSVM), builds k(k-1)/2 binary classifiers, trained to separate each pair of classes against each other, and uses a majority voting scheme (max-win strategy) to determine the output prediction.
There are also other approaches such as using Error Correcting Output Code (ECOC) to build many somewhat-redundant binary-classifiers, and use this redundancy to obtain more robust classifications (uses the same idea as Hamming codes).
Example (one-against-one):
%# load dataset
load fisheriris
[g gn] = grp2idx(species); %# nominal class to numeric
%# split training/testing sets
[trainIdx testIdx] = crossvalind('HoldOut', species, 1/3);
pairwise = nchoosek(1:length(gn),2); %# 1-vs-1 pairwise models
svmModel = cell(size(pairwise,1),1); %# store binary-classifers
predTest = zeros(sum(testIdx),numel(svmModel)); %# store binary predictions
%# classify using one-against-one approach, SVM with 3rd degree poly kernel
for k=1:numel(svmModel)
%# get only training instances belonging to this pair
idx = trainIdx & any( bsxfun(#eq, g, pairwise(k,:)) , 2 );
%# train
svmModel{k} = svmtrain(meas(idx,:), g(idx), ...
'BoxConstraint',2e-1, 'Kernel_Function','polynomial', 'Polyorder',3);
%# test
predTest(:,k) = svmclassify(svmModel{k}, meas(testIdx,:));
end
pred = mode(predTest,2); %# voting: clasify as the class receiving most votes
%# performance
cmat = confusionmat(g(testIdx),pred);
acc = 100*sum(diag(cmat))./sum(cmat(:));
fprintf('SVM (1-against-1):\naccuracy = %.2f%%\n', acc);
fprintf('Confusion Matrix:\n'), disp(cmat)
Here is a sample output:
SVM (1-against-1):
accuracy = 93.75%
Confusion Matrix:
16 0 0
0 14 2
0 1 15
MATLAB does not support multiclass SVM at the moment. You could use svmtrain (2-classes) to achieve this, but it would be much easier to use a standard SVM package.
I have used LIBSVM and can confirm that it's very easy to use.
%%# Your data
D = [
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3];
%%# For clarity
Attributes = D(:,1:4);
Classes = D(:,5);
train = [1 3 5 6];
test = [2 4];
%%# Train
model = svmtrain(Classes(train),Attributes(train,:),'-s 0 -t 2');
%%# Test
[predict_label, accuracy, prob_estimates] = svmpredict(Classes(test), Attributes(test,:), model);
I'm using HOG in order to extract a set of features trough an Image A.
the HOG returns a features' vector of 1xN elements.
However the linear SVM accept only 2 features for each sample i.e the training data matrix's size is Mx2. so how i can adapt the HOG vector to be trained on linear SVM.
Please help me.
Thanks
What do you mean by "the linear SVM accept only 2 features for each sample"? You may be confused on how the SVM function accepts its training data. Here's a quick example of how I use it:
First, lets train an SVM model using fitcsvm using 500 samples of random data (the rows in the training data matrix), each with 1000 elements (the columns in the training data matrix), where the first 250 samples are in class 1 (first 250 rows of training labels), and the last 250 samples are in class 0 (last 250 rows of training labels):
>> training_data = rand(500, 1000);
>> training_labels = [ones(250,1); zeros(250,1)];
>>
>> svm_model = fitcsvm(training_data, testing_data)
svm_model =
ClassificationSVM
PredictorNames: {1x1000 cell}
ResponseName: 'Y'
ClassNames: [0 1]
ScoreTransform: 'none'
NumObservations: 500
Alpha: [418x1 double]
Bias: 2.3217
KernelParameters: [1x1 struct]
BoxConstraints: [500x1 double]
ConvergenceInfo: [1x1 struct]
IsSupportVector: [500x1 logical]
Solver: 'SMO'
Properties, Methods
We can generate some random test data for 10 test samples, each with 1000 elements, and create some predictions from it:
>> test_data = rand(10, 1000);
>> predicted_classes = predict(svm_model, test_data)
predicted_classes =
1
1
1
1
1
0
0
0
1
0
Does that answer your question?
I am training a one vs all svm classifier. I used a 200 by 459 matrix to train the classifier using VLFeat svm classifier. (http://www.vlfeat.org/matlab/vl_svmtrain.html)
[W B] = vl_svmtrain(train_image_feats', tmp', .00001);
where train_image_feats' is a 200 by 459 matrix, and tmp' is the label matrix which is 1 by 459 vector.
The above command trains the svm with no problem, but then to classify the scores obtained on the test matrix I get an error. The test matrix is obviously not of the same size as that of the training matrix.
scores(i, :) = W'*test_image_feats' + B;
Where test_image_feats' is a 200 by 90 matrix. scores is a 9 by 459 matrix. 9 Because there are 9 categories(labels) to classify and 459 are the number of training images.
The above command gives the error:
Subscripted assignment dimension mismatch.
Error in svm_classify (line 56) scores(i, :) = W'*test_image_feats'
+ B;
Edit: Full code added..
categories = unique(train_labels);
num_categories = length(categories);
scores = zeros([num_categories size(train_labels, 1)]); %train_labels is 459 by 1 size
for i=1:num_categories %there are 9 categories
tmp = strcmp(train_labels, categories{i});
tmp = tmp - (1-tmp);
[W B] = vl_svmtrain(train_image_feats', tmp', .00001);
scores(i, :) = W'*test_image_feats' + B;
end
predicted_categories = cell(size(train_labels));
parfor i=1:size(test_image_feats,1)
image_scores = scores(:, i);
label_index = find(image_scores==max(image_scores));
predicted_categories{i}=categories(label_index);
end
Conceptually you are training a model with 459 training samples to predict the scores of 90 test samples.
scores = zeros([num_categories size(train_labels, 1)]);
isn't right as it will be the size of the training set. In fact you don't have to care at all about the size of the training set, you could train the model with 20 or 20000 images the prediction step shouldn't be any different.
scores have to be defined with the test case in mind
scores = zeros([num_categories size(test_labels, 1)]);
When you used 459 for both it only worked because size(test_labels, 1) was equal to size(train_labels, 1)
The problem is not with your right hand side of the assignment, but with score(i,:): you are trying to assign a 9-by-90 size matrix into a single row of score - this simply won't fit.
I'm using LIBSVM toolbox for Matlab. My problem is a binary classification problem with labels of 1 (True) and 0 (False). When I train my system with this function :
svmstruct = svmtrain(TrainTargets, TrainInputs,['-t 2 ' '-g ' SIGMA ' -c ' P ' -q' ' -b 1']);
and test my test accuracy with this function :
[TestOutputs, ~, ~] = svmpredict(TestTargets, TestInputs, svmstruct,'-b 1 -q');
Now I want use desined SVM model for out sample data. So I use this function :
[OUT, ~, Prob_Out] = svmpredict(zeros(size(Outsample_DATA,1),1), Outsample_DATA, svmstruct,'-q -b 1');
For my first trained model (I have trained SVM model with different parameters) I have this Output (Out sample data set is same in both cases) : [Prob_Out OUT]
0.8807 0.1193 0
0.8717 0.1283 0
0.0860 0.9140 1.0000
0.7846 0.2154 0
0.7685 0.2315 0
0.7916 0.2084 0
0.0326 0.9674 1.0000
0.7315 0.2685 0
0.3550 0.6450 1.0000
for second one I have this :
0.4240 0.5760 0
0.4090 0.5910 0
0.7601 0.2399 1.0000
0.5000 0.5000 1.0000
0.4646 0.5354 0
0.4589 0.5411 0
Suppose that I want find class 1 with these probabilities. In first group of data when column 2 is larger than column 1 this sample belongs to class 1 but in second group when column 1 is larger than column 2 the sample belongs to class 1.
The structure of these two out sample data is same. What is the problem?
Thanks.
PS.
When I check SVMstruct parameters after training the model in on of these models Label is [0;1] and in another label is [1;0] !
As you have already noticed, the difference is due to the different mapping of the labels.
LIBSVM uses its own labels internally and therefore needs a mapping between the internal labels and the labels you provided.
The labels in this mapping are generated using the order the labels appear in the training data. So if the label of the first element in your training data changes, the label mapping also changes.
Could you give an example of classification of 4 classes using Support Vector Machines (SVM) in matlab something like:
atribute_1 atribute_2 atribute_3 atribute_4 class
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3
SVMs were originally designed for binary classification. They have then been extended to handle multi-class problems. The idea is to decompose the problem into many binary-class problems and then combine them to obtain the prediction.
One approach called one-against-all, builds as many binary classifiers as there are classes, each trained to separate one class from the rest. To predict a new instance, we choose the classifier with the largest decision function value.
Another approach called one-against-one (which I believe is used in LibSVM), builds k(k-1)/2 binary classifiers, trained to separate each pair of classes against each other, and uses a majority voting scheme (max-win strategy) to determine the output prediction.
There are also other approaches such as using Error Correcting Output Code (ECOC) to build many somewhat-redundant binary-classifiers, and use this redundancy to obtain more robust classifications (uses the same idea as Hamming codes).
Example (one-against-one):
%# load dataset
load fisheriris
[g gn] = grp2idx(species); %# nominal class to numeric
%# split training/testing sets
[trainIdx testIdx] = crossvalind('HoldOut', species, 1/3);
pairwise = nchoosek(1:length(gn),2); %# 1-vs-1 pairwise models
svmModel = cell(size(pairwise,1),1); %# store binary-classifers
predTest = zeros(sum(testIdx),numel(svmModel)); %# store binary predictions
%# classify using one-against-one approach, SVM with 3rd degree poly kernel
for k=1:numel(svmModel)
%# get only training instances belonging to this pair
idx = trainIdx & any( bsxfun(#eq, g, pairwise(k,:)) , 2 );
%# train
svmModel{k} = svmtrain(meas(idx,:), g(idx), ...
'BoxConstraint',2e-1, 'Kernel_Function','polynomial', 'Polyorder',3);
%# test
predTest(:,k) = svmclassify(svmModel{k}, meas(testIdx,:));
end
pred = mode(predTest,2); %# voting: clasify as the class receiving most votes
%# performance
cmat = confusionmat(g(testIdx),pred);
acc = 100*sum(diag(cmat))./sum(cmat(:));
fprintf('SVM (1-against-1):\naccuracy = %.2f%%\n', acc);
fprintf('Confusion Matrix:\n'), disp(cmat)
Here is a sample output:
SVM (1-against-1):
accuracy = 93.75%
Confusion Matrix:
16 0 0
0 14 2
0 1 15
MATLAB does not support multiclass SVM at the moment. You could use svmtrain (2-classes) to achieve this, but it would be much easier to use a standard SVM package.
I have used LIBSVM and can confirm that it's very easy to use.
%%# Your data
D = [
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3];
%%# For clarity
Attributes = D(:,1:4);
Classes = D(:,5);
train = [1 3 5 6];
test = [2 4];
%%# Train
model = svmtrain(Classes(train),Attributes(train,:),'-s 0 -t 2');
%%# Test
[predict_label, accuracy, prob_estimates] = svmpredict(Classes(test), Attributes(test,:), model);