libsvm: Evaluating an SVM using leave-one-out - matlab

Question

I am trying to use libsvm with MATLAB to evaluate a one-vs-all SVM, the only issue is that my dataset is not big enough to warrant selecting a specific test set. Thus, I want to evaluate my classifiers using leave-one-out.
I am not particularly experienced in using SVMs, so forgive me if I am a little bit confused as to what to do. I need to generate precision vs recall curves, and confusion matrices for my classifiers but I have no idea where to start.
I've given it a go and came up with the following as a rough start to do the leave on out training but I'm not sure how to do evaluation.
function model = do_leave_one_out(labels, data)
acc = [];
bestC = [];
bestG = [];
for ii = 1:length(data)
% Training data for this iteration
trainData = data;
trainData(ii) = [];
looLabel = labels(ii);
trainLabels = labels;
trainLabels(ii) = [];
% Do grid search to find the best parameters?
acc(ii) = bestReportedAccuracy;
bestC(ii) = bestValueForC;
bestG(ii) = bestValueForG;
end
% After this I am not sure how to train and evaluate the final model
end

Answer 1

I'm trying to provide some modules that you may be interested in, and you can incorporate them into your function. Hope it helps.
Leave-one-out:
scrambledList = randperm(totalNumberOfData);
trainingData = Data(scrambledList(1:end-1),:);
trainingLabel = Label(scrambledList(1:end-1));
testData = Data(scrambledList(end),:);
testLabel = Label(scrambledList(end));
Grid Search (Dual-class case):
acc = 0;
for log2c = -1:3,
for log2g = -4:1,
cmd = ['-v 5 -c ', num2str(2^log2c), ' -g ', num2str(2^log2g)];
cv = svmtrain(trainingLabel, trainingData, cmd);
if (cv >= acc),
acc = cv; bestC = 2^log2c; bestG = 2^log2g;
end
end
end
One-vs-all (Used for Multi-class case):
model = cell(NumofClass,1);
for k = 1:NumofClass
model{k} = svmtrain(double(trainingLabel==k), trainingData, '-c 1 -g 0.2 -b 1');
end
%% calculate the probability of different labels
pr = zeros(1,NumofClass);
for k = 1:NumofClass
[~,~,p] = svmpredict(double(testLabel==k), testData, model{k}, '-b 1');
pr(:,k) = p(:,model{k}.Label==1); %# probability of class==k
end
%% your label prediction will be the one with highest probability:
[~,predctedLabel] = max(pr,[],2);