I am trying to do a 5 fold cross validation with libsvm (matlab) using a precomputed kernel, but, I get the following error message :
Undefined function 'ge' for input arguments of type 'struct'.
this is because the Libsvm return a structure instead of a value in cross validation, How can I solve this problem, this is my code:
load('iris.dat')
data=iris(:,1:4);
class=iris(:,5);
% normalize the data
range=repmat((max(data)-min(data)),size(data,1),1);
data=(data-repmat(min(data),size(data,1),1))./range;
% train
tr_data=[data(1:5,:);data(52:56,:);data(101:105,:)];
tr_lbl=[ones(5,1);2*ones(5,1);3*ones(5,1)];
% kernel computation
sigma=.8
rbfKernel = #(X,Y,sigma) exp((-pdist2(X,Y,'euclidean').^2)./(2*sigma^2));
Ktr=[(1:15)',rbfKernel(tr_data,tr_data,sigma)];
kts=[ (1:150)',rbfKernel(data,tr_data,sigma)];
% svmptrain
bestcv = 0;
for log2c = -1:3
cmd = ['Ktr -t 4 -v 5 -c ', num2str(2^log2c)];
cv = svmtrain2(tr_lbl,tr_data, cmd);
if (cv >= bestcv)
bestcv = cv;
bestc = 2^log2c;
end
end
cmd=['-s 0 -c ', num2str(bestc), 'Ktr -t 4']
model=svmtrain2(tr_lbl,tr_data,cmd)
% svm predict
labels=svmpredict(class,data,model,kts)
The function svmtrain2 you are using is not part of standard MATLAB and also the output of the function is not a structure. But if you insist to use that, you can calculate an score for data using the other existing function:
[f,K] = svmeval(X_eval,varargin)
that evaluates the trained svm using the outputs from svmtrain2. But I prefer to use first the standard functions embedded in MATLAB. In standard MATLAB library there is:
SVMStruct = svmtrain(Training,Group)
that returns a structure, SVMStruct, containing information about the trained support vector machine (SVM) classifier. or
SVMModel = fitcsvm(X,Y)
that returns a support vector machine classifier SVMModel, trained by predictors X and class labels Y for one- or two-class classification. and then you can get some score for each prediction using:
[label,Score] = predict(SVMModel,X)
that returns class likelihood measures, i.e., either scores or posterior probabilities.
You get that error because you are trying to compare a struct and a number.
If what you want is to find the best performance in the training set (as it seems from you comparison), I don't think you can get it directly from the structure returned from svmtrain. You should first use svmpredict with the training set and the trained model, and you can get the accuracy from the resulting structure.
Related
I'm trying to apply cross validated LDA using matlab cross validation method. To do this I put the crossval() in a loop and in each loop I extract corresponding train and test labels and feature matrix (trFV, tsFV). It's like the example presented in Matlab cvpartition class:
cvp = cvpartition(labelCell,'KFold',kFoldCV)
for i = 1:cvp.NumTestSets
trFV = f(cvp.training(i), :);
tsFV = f(cvp.test(i), :);
% calculate LDA projection matrix:
[~, W] = LDA(trFV, featureMat(cvp.training(i), end));
% apply W to both train and test:
trFVW = trFV * W(:, 1:numel(classes)-1);
tsFVW = tsFV * W(:, 1:numel(classes)-1);
fW = [trFVW; tsFVW];
labels = [labelCell(cvp.training(i)); labelCell(cvp.test(i))];
Mdl = fitcecoc(fW, labels, 'Coding', 'onevsall',...
'Learners', learnerTemplate,...
'ClassNames', classes);
CVMdl = crossval(Mdl, 'CVPartition', cvp);
% Other stuff
end
This implementation is very inefficient since I just need the result for one of the folds (not entire folds). I process each fold once in a loop while crossval process whole loops in each loop. Hence in current implementation instead of cvp.NumTestSets times it perform the cross validation cvp.NumTestSets^2 times. I need something like this:
CVMdl = crossval(Mdl, 'CVPartition', cvp, 'compute just for partition i and not all partitions');
Update
The code above have some problem regarding cross validation. However I'm still interested if Matlab built-in LDA (linear discriminant analysis) could be used to reduce dimension.
I use svm in Rand matlab with the same dataset.
My R code works fine, which gives me some reasonable predictions.
matdat <- readMat(con = "data.mat")
svm.model <- svm(x = matdat$normalize.X, y = matdat$Yt)
pred <- predict(svm.model, newdata = matdat$normalize.X)
pred <- sapply(pred, function(x){ifelse(x > 0, 1, -1)})
sum(pred == matdat$Yt)/length(matdat$Yt)
But, my matlab code gives me all 1 prediction on the training data.
load('data.mat')
model2 = svmtrain(Yt, normalize_X,'-s 3 -c 1 -t 2 -p 0.1');
[predicted_label,accuracy, decision_values] = svmpredict(Yt, normalize_X, model2);
I have checked the default parameters of svm{e1071}, which in my opinion agrees with the matlab version.
I use the e1071 package with verion 1.6-7 in R. And the latest libsvm from the official page.
So, what can I do to find the reason, any ideas?
==== update====
Before feeding the data to libsvm in data, I apply mapstd to normalize the data which is automatically done in R. Then I got the same trained model in both R and Matlab.
In Matlab you use the -s 3 option which is regression, not classification.
As a starting point, don't assume anything about default parameters, just specify parameters explicitly in both R and Matlab.
I am using libsvm on Matlab. I want to build a model and use this model for prediction.
It is wired that the returns of svmpredict ([predict_label, accuracy_all, prob_values]) are empty. Here is my simple code:
svm_model = svmtrain([train_label],[train],'-t 2, -c 100 -q');
[predict_label, accuracy_all, prob_values] = svmpredict(testlabels,testdata,svm_model,'-q, -b 1');
[predict_label, accuracy_all, prob_values] are 0x0 matrix. And also Matlab also shows some warning information:
Usage: [predicted_label, accuracy, decision_values/prob_estimates] = svmpredict(testing_label_vector, testing_instance_matrix, model, 'libsvm_options')
Parameters:
model: SVM model structure from svmtrain.
libsvm_options:
-b probability_estimates: whether to predict probability estimates, 0 or 1 (default 0); one-class SVM not supported yet
Returns:
predicted_label: SVM prediction output vector.
accuracy: a vector with accuracy, mean squared error, squared correlation coefficient.
prob_estimates: If selected, probability estimate vector.
Can anyone help me?
What is q in the SVM model ? where is it's value ?
there is two parameters in the SVM which need to be well defined c and g , you have put 100 as value of c but ther is no value for q (or must be g which is called gamma)
this what you need
cmd = ['-t 2 -c ',num2str(C), ' -g ',num2str(gamma) ];
model = svmtrain2(trainClass, trainData, cmd);
[predClass, acc, decVals] = svmpredict(testClass, testData, model);
Also i think that svmtrain must be renamed svmtrain2 to avoid the confusion with svmtrain function of matlab.
I m trying to simulate out of sample prediction of a binary classifier using libsvm in matlab. My target variable (ie my label) is binary (-1 +1). Therefore, in my test set there are series for which i don t know the label. I created a new label for these observations (this label is 747). I found that in my predicted_label_test vector (see code below), this 747 label is included. So it means the prediction I get is influenced by the labels of the data included the test set, which is what I m supposed to predict? - The mistake may be in the way I use Libsvm read and write functions but i can t find it - many thanks!!
%%%%%%%%%% GET DATA FROM THE CSV FILE AND CONVERT THEM TO LIBSVM
addpath('C:\libsvm1\matlab'); %indicate position of the CSV file
ALLDATA = csvread('DATACSV.csv'); % read a csv file
labels = ALLDATA(:, 1); % labels are included in the first column of data
labels_sparse = sparse (labels); %? needed
features = ALLDATA(:, 4:end); % features start at 4th column
features_sparse = sparse(features); % features must be in a sparse matrix
libsvmwrite('TTT.train', labels_sparse, features_sparse); % write the file to libsvm format
[label_vector, predictors_matrix] = libsvmread('C:\libsvm1\matlab\TTT.train'); % read the file that was recorded in Libsvm format
%%%%% DEFINE VECTOR AND MATRIX SIZE
label_vector_train = label_vector (1:143,:);
predictors_matrix_train = predictors_matrix (1:143,:);
label_vector_test = label_vector (144:193,:);
predictors_matrix_test = predictors_matrix (144:193,:);
%PREDICTION
param = ['-q -c 2 -g 3'];
bestModel = svmtrain(label_vector_test, predictors_matrix_test, param);
[predicted_label_test, accuracy, prob_values] = svmpredict(label_vector_test, predictors_matrix_test, bestModel);
You are training a svm model with test data, when you should train it with training data:
bestModel = svmtrain(label_vector_test, predictors_matrix_test, param);
should be:
bestModel = svmtrain(label_vector_train, predictors_matrix_train, param);
I know that Cross validation is used for selecting good parameters. After finding them, i need to re-train the whole data without the -v option.
But the problem i face is that after i train with -v option, i get the cross-validation accuracy( e.g 85%). There is no model and i can't see the values of C and gamma. In that case how do i retrain?
Btw i applying 10 fold cross validation.
e.g
optimization finished, #iter = 138
nu = 0.612233
obj = -90.291046, rho = -0.367013
nSV = 165, nBSV = 128
Total nSV = 165
Cross Validation Accuracy = 98.1273%
Need some help on it..
To get the best C and gamma, i use this code that is available in the LIBSVM FAQ
bestcv = 0;
for log2c = -6:10,
for log2g = -6:3,
cmd = ['-v 5 -c ', num2str(2^log2c), ' -g ', num2str(2^log2g)];
cv = svmtrain(TrainLabel,TrainVec, cmd);
if (cv >= bestcv),
bestcv = cv; bestc = 2^log2c; bestg = 2^log2g;
end
fprintf('(best c=%g, g=%g, rate=%g)\n',bestc, bestg, bestcv);
end
end
Another question : Is that cross-validation accuracy after using -v option similar to that we get when we train without -v option and use that model to predict? are the two accuracy similar?
Another question : Cross-validation basically improves the accuracy of the model by avoiding the overfitting. So, it needs to have a model in place before it can improve. Am i right? Besides that, if i have a different model, then the cross-validation accuracy will be different? Am i right?
One more question: In the cross-validation accuracy, what is the value of C and gamma then?
The graph is something like this
Then the values of C are 2 and gamma = 0.0078125. But when i retrain the model with the new parameters. The value is not the same as 99.63%. Could there be any reason?
Thanks in advance...
The -v option here is really meant to be used as a way to avoid the overfitting problem (instead of using the whole data for training, perform an N-fold cross-validation training on N-1 folds and testing on the remaining fold, one at-a-time, then report the average accuracy). Thus it only returns the cross-validation accuracy (assuming you have a classification problem, otherwise mean-squared error for regression) as a scalar number instead of an actual SVM model.
If you want to perform model selection, you have to implement a grid search using cross-validation (similar to the grid.py helper python script), to find the best values of C and gamma.
This shouldn't be hard to implement: create a grid of values using MESHGRID, iterate overall all pairs (C,gamma) training an SVM model with say 5-fold cross-validation, and choosing the values with the best CV-accuracy...
Example:
%# read some training data
[labels,data] = libsvmread('./heart_scale');
%# grid of parameters
folds = 5;
[C,gamma] = meshgrid(-5:2:15, -15:2:3);
%# grid search, and cross-validation
cv_acc = zeros(numel(C),1);
for i=1:numel(C)
cv_acc(i) = svmtrain(labels, data, ...
sprintf('-c %f -g %f -v %d', 2^C(i), 2^gamma(i), folds));
end
%# pair (C,gamma) with best accuracy
[~,idx] = max(cv_acc);
%# contour plot of paramter selection
contour(C, gamma, reshape(cv_acc,size(C))), colorbar
hold on
plot(C(idx), gamma(idx), 'rx')
text(C(idx), gamma(idx), sprintf('Acc = %.2f %%',cv_acc(idx)), ...
'HorizontalAlign','left', 'VerticalAlign','top')
hold off
xlabel('log_2(C)'), ylabel('log_2(\gamma)'), title('Cross-Validation Accuracy')
%# now you can train you model using best_C and best_gamma
best_C = 2^C(idx);
best_gamma = 2^gamma(idx);
%# ...
If you use your entire dataset to determine your parameters, then train on that dataset, you are going to overfit your data. Ideally, you would divide the dataset, do the parameter search on a portion (with CV), then use the other portion to train and test with CV. Will you get better results if you use the whole dataset for both? Of course, but your model is likely to not generalize well. If you want determine true performance of your model, you need to do parameter selection separately.