I need to use a SVM to distinguish 28x28 matrices into 9 classes. There are 60,000 training inputs and 10,000 testing inputs.
My current program is as follows:
clear;
load mnist.mat
xtest = xtest ./ 255; <--- Normalizing the data
xtrain = xtrain ./ 255;
SVMModels = cell(9,1);
classes = unique(ytrain);
rng(1); % For reproducibility
blah = fitcsvm(xtrain, ytrain);
for j = 1:numel(classes);
indx = strcmp(ytrain,classes(j)); % Create binary classes for each classifier
SVMModels{j} = fitcsvm(xtrain,indx, 'KernelFunction','rbf','BoxConstraint',1);
end
I believe the problem is due either to the fact that the inputs are 28x28. How do I fix this?
Additional info:
xtest: 28x28x10000
ytest = 1x10000
xtrain = 28x28x60000
ytrain = 1x60000
You are correct. fitcsvm requires that the input training examples is a N x P matrix where N is the total number of samples and P is the total number of features. What you have to do in your case is reshape your array so that xtrain and xtest are 60000 x 784. The 784 is due to 28 x 28. Specifically, you must unroll each slice of your 3D matrix so that it fits into a single vector. Similarly the class labels must be N x 1, so you just need to transpose ytrain and ytest.
To achieve the desired reshaping, you use reshape like so:
xtrain_final = reshape(xtrain, 784, 60000).'; %'
xtest_final = reshape(xtest, 784, 60000).'; %'
ytrain_final = ytrain.'; %'
ytest_final = ytest.'; %'
Now the reshaping of the training and testing examples is a bit odd. How MATLAB works when reshaping is that it performs this on a column major basis. This means that when you reshape, it takes columns at a time to produce your results. As such, because your matrix is 28 x 28 x 60000, each slice of your 3D matrix is 28 x 28. Therefore, to facilitate the column major ordering, you take each 2D slice and fit it to a single column. You would thus have 60000 columns corresponding to 60000 training examples. The last thing you need to do now is transpose this result to get what is required for fitcsvm.
Now that this is done, you can train your model.
Related
What are the difference between the following two functions?
prepTransform.m
function [mu trmx] = prepTransform(tvec, comp_count)
% Computes transformation matrix to PCA space
% tvec - training set (one row represents one sample)
% comp_count - count of principal components in the final space
% mu - mean value of the training set
% trmx - transformation matrix to comp_count-dimensional PCA space
% this is memory-hungry version
% commented out is the version proper for Win32 environment
tic;
mu = mean(tvec);
cmx = cov(tvec);
%cmx = zeros(size(tvec,2));
%f1 = zeros(size(tvec,1), 1);
%f2 = zeros(size(tvec,1), 1);
%for i=1:size(tvec,2)
% f1(:,1) = tvec(:,i) - repmat(mu(i), size(tvec,1), 1);
% cmx(i, i) = f1' * f1;
% for j=i+1:size(tvec,2)
% f2(:,1) = tvec(:,j) - repmat(mu(j), size(tvec,1), 1);
% cmx(i, j) = f1' * f2;
% cmx(j, i) = cmx(i, j);
% end
%end
%cmx = cmx / (size(tvec,1)-1);
toc
[evec eval] = eig(cmx);
eval = sum(eval);
[eval evid] = sort(eval, 'descend');
evec = evec(:, evid(1:size(eval,2)));
% save 'nist_mu.mat' mu
% save 'nist_cov.mat' evec
trmx = evec(:, 1:comp_count);
pcaTransform.m
function [pcaSet] = pcaTransform(tvec, mu, trmx)
% tvec - matrix containing vectors to be transformed
% mu - mean value of the training set
% trmx - pca transformation matrix
% pcaSet - output set transforrmed to PCA space
pcaSet = tvec - repmat(mu, size(tvec,1), 1);
%pcaSet = zeros(size(tvec));
%for i=1:size(tvec,1)
% pcaSet(i,:) = tvec(i,:) - mu;
%end
pcaSet = pcaSet * trmx;
Which one is actually doing PCA?
If one is doing PCA, what is the other one doing?
The first function prepTransform is actually doing the PCA on your training data where you are determining the new axes to represent your data onto a lower dimensional space. What it does is that it finds the eigenvectors of the covariance matrix of your data and then orders the eigenvectors such that the eigenvector with the largest eigenvalue appears in the first column of the eigenvector matrix evec and the eigenvector with the smallest eigenvalue appears in the last column. What's important with this function is that you can define how many dimensions you want to reduce the data down to by keeping the first N columns of evec which will allow you to reduce your data down to N dimensions. The discarding of the other columns and keeping only the first N is what is set as trmx in the code. The variable N is defined by the prep_count variable in prepTransform function.
The second function pcaTransform finally transforms data that is defined within the same domain as your training data but not necessarily the training data itself (it could be if you wish) onto the lower dimensional space that is defined by the eigenvectors of the covariance matrix. To finally perform the reduction of dimensions, or dimensionality reduction as it is popularly known, you simply take your training data where each feature is subtracted from its mean and you multiply your training data by the matrix trmx. Note that prepTransform outputting the mean of each feature in the vector mu is important in order to mean subtract your data when you finally call pcaTransform.
How to use these functions
To use these functions effectively, first determine the trmx matrix, which contain the principal components of your data by first defining how many dimensions you want to reduce your data down to as well as the mean of each feature stored in mu:
N = 2; % Reduce down to two dimensions for example
[mu, trmx] = prepTransform(tvec, N);
Next you can finally perform dimensionality reduction on your data that is defined within the same domain as tvec (or even tvec if you wish, but it doesn't have to be) by:
pcaSet = pcaTransform(tvec, mu, trmx);
In terms of vocabulary, pcaSet contain what are known as the principal scores of your data, which is the term used for the transformation of your data to the lower dimensional space.
If I can recommend something...
Finding PCA through the eigenvector approach is known to be unstable. I highly recommend you use the Singular Value Decomposition via svd on the covariance matrix where the V matrix of the result already gives you the eigenvectors sorted which correspond to your principal components:
mu = mean(tvec, 1);
[~,~,V] = svd(cov(tvec));
Then perform the transformation by taking the mean subtracted data per feature and multiplying by the V matrix, once you subset and grab the first N columns of V:
N = 2;
X = bsxfun(#minus, tvec, mu);
pcaSet = X*V(:, 1:N);
X is the mean subtracted data which performs the same thing as doing pcaSet = tvec - repmat(mu, size(tvec,1), 1);, but you are not explicitly replicating the mean vector over each training example but letting bsxfun do that for you internally. However, taking advantage of MATLAB R2016b, this repeating can be done without the explicit call to bsxfun:
X = tvec - mu;
Further Reading
If you fully want to understand the code that was written and the theory behind what it's doing, I recommend the following two Stack Overflow posts that I have written that talk about the topic:
What does selecting the largest eigenvalues and eigenvectors in the covariance matrix mean in data analysis?
How to use eigenvectors obtained through PCA to reproject my data?
The first post brings the code you presented into light which performs PCA using the eigenvector approach. The second post touches base on how you'd do it using the SVD towards the end of the answer. This answer I've written here is a mix between the two posts above.
I am trying to perform cross-validation on images for my SVM, where I have 3 categories of labels for the classification, "Good", "Ok" and "Bad".
For my data set, I have a 120 * 20 cell array, mainly 19 columns of features and with the last column being the class label for 120 distinct images.
The SVM train is performed using 2 different train label as such
SVMStruct = svmtrain(normalizedTrainingSet , train_label, 'kernel_function', 'linear');
SVMStruct1 = svmtrain(normalizedTrainingSet , train_label1, 'kernel_function', 'linear');
Where "normalizedTrainingSet" is the numeric matrix for my data set. train_label is the label for Bad vs Normal&Good; train_label1 is the label for Good vs Normal&Bad, and I performed some if else statements to sort them out.
I want to perform cross-validation with 5 folds, and during each fold, I want to split the images equally for each category. For example, 4 for testing, and 16 for training during each fold, equally for all 3 categories.
Below is my code for the cross-validation.
K = 5; % The number of folds
N = size(DataSet, 1);
idx = crossvalind('Kfold', N, K);
cp = classperf(train_label3); %train_label3 is the combination of all 3 categories in one array.
for i = 1:K
Data_Set = DataSet(idx ~= i, :); % data to train on, 90% of the total.
training_label = train_label3(idx ~= i, :); % class labels of training data.
Test_Set = DataSet(idx == i, :); % data to test on, 10% of the total.
testing_label = train_label3(idx == i, :); % class labels of test data.
I am stucked trying to perform the cross-validation and need some help on how to continue.
I am a beginner in matlab and I have a particular z matrix of size m×1 with values 0,1,3,5,2 etc..with above values repeating. Now I have 4 other column matrix x1,x2,x3 and y and I want to do regression.
I have used lm = LinearModel.fit(x,y,'linear') specifying columns.Now I want to do regression only for values in matrix x1,x2,x3 and y for those corresponding to z matrix with value of 1 and neglect the other rows.How do I do it?
That's very simple. I'm going to assume that your matrix of predictor variables and outputs are also of size m (number of samples). All you have to do is find the locations within z that are 1, subset your 3 column matrix of x1,x2,x3 and y, then use LinearModel.fit to fit your data. Assuming your matrix of predictors is stored in X, and your outputs are stored in y, you would do this:
ind = z == 1;
xOut = X(ind,:);
yOut = y(ind);
lm1 = LinearModel.fit(xOut, yOut, 'linear');
BTW, these are very simple subsetting operations in MATLAB. Suggest you read a tutorial before asking any further questions here.
I have the following code for calculating the result of a linear combination of Gaussian functions. What I'd really like to do is to vectorize this somehow so that it's far more performant in Matlab.
Note that y is a column vector (output), x is a matrix where each column corresponds to a data point and each row corresponds to a dimension (i.e. 2 rows = 2D), variance is a double, gaussians is a matrix where each column is a vector corresponding to the mean point of the gaussian and weights is a row vector of the weights in front of each gaussian. Note that the length of weights is 1 bigger than gaussians as weights(1) is the 0th order weight.
function [ y ] = CalcPrediction( gaussians, variance, weights, x )
basisFunctions = size(gaussians, 2);
xvalues = size(x, 2);
if length(weights) ~= basisFunctions + 1
ME = MException('TRAIN:CALC', 'The number of weights should be equal to the number of basis functions plus one');
throw(ME);
end
y = weights(1) * ones(xvalues, 1);
for xIdx = 1:xvalues
for i = 1:basisFunctions
diff = x(:, xIdx) - gaussians(:, i);
y(xIdx) = y(xIdx) + weights(i+1) * exp(-(diff')*diff/(2*variance));
end
end
end
You can see that at the moment I simply iterate over the x vectors and then the gaussians inside 2 for loops. I'm hoping that this can be improved - I've looked at meshgrid but that seems to only apply to vectors (and I have matrices)
Thanks.
Try this
diffx = bsxfun(#minus,x,permute(gaussians,[1,3,2])); % binary operation with singleton expansion
diffx2 = squeeze(sum(diffx.^2,1)); % dot product, shape is now [XVALUES,BASISFUNCTIONS]
weight_col = weights(:); % make sure weights is a column vector
y = exp(-diffx2/2/variance)*weight_col(2:end); % a column vector of length XVALUES
Note, I changed diff to diffx since diff is a builtin. I'm not sure this will improve performance as allocating arrays will offset increase by vectorization.
I am working on thumb recognition system. I need to implement KNN algorithm to classify my images. according to this, it has only 2 measurements, through which it is calculating the distance to find the nearest neighbour but in my case I have 400 images of 25 X 42, in which 200 are for training and 200 for testing. I am searching for few hours but I am not finding the way to find the distance between the points.
EDIT:
I have reshaped 1st 200 images in to 1 X 1050 and stored them in a matrix trainingData of 200 X 1050. similarly I made testingData.
Here is an illustration code for k-nearest neighbor classification (some functions used require the Statistics toolbox):
%# image size
sz = [25,42];
%# training images
numTrain = 200;
trainData = zeros(numTrain,prod(sz));
for i=1:numTrain
img = imread( sprintf('train/image_%03d.jpg',i) );
trainData(i,:) = img(:);
end
%# testing images
numTest = 200;
testData = zeros(numTest,prod(sz));
for i=1:numTest
img = imread( sprintf('test/image_%03d.jpg',i) );
testData(i,:) = img(:);
end
%# target class (I'm just using random values. Load your actual values instead)
trainClass = randi([1 5], [numTrain 1]);
testClass = randi([1 5], [numTest 1]);
%# compute pairwise distances between each test instance vs. all training data
D = pdist2(testData, trainData, 'euclidean');
[D,idx] = sort(D, 2, 'ascend');
%# K nearest neighbors
K = 5;
D = D(:,1:K);
idx = idx(:,1:K);
%# majority vote
prediction = mode(trainClass(idx),2);
%# performance (confusion matrix and classification error)
C = confusionmat(testClass, prediction);
err = sum(C(:)) - sum(diag(C))
If you want to compute the Euclidean distance between vectors a and b, just use Pythagoras. In Matlab:
dist = sqrt(sum((a-b).^2));
However, you might want to use pdist to compute it for all combinations of vectors in your matrix at once.
dist = squareform(pdist(myVectors, 'euclidean'));
I'm interpreting columns as instances to classify and rows as potential neighbors. This is arbitrary though and you could switch them around.
If have a separate test set, you can compute the distance to the instances in the training set with pdist2:
dist = pdist2(trainingSet, testSet, 'euclidean')
You can use this distance matrix to knn-classify your vectors as follows. I'll generate some random data to serve as example, which will result in low (around chance level) accuracy. But of course you should plug in your actual data and results will probably be better.
m = rand(nrOfVectors,nrOfFeatures); % random example data
classes = randi(nrOfClasses, 1, nrOfVectors); % random true classes
k = 3; % number of neighbors to consider, 3 is a common value
d = squareform(pdist(m, 'euclidean')); % distance matrix
[neighborvals, neighborindex] = sort(d,1); % get sorted distances
Take a look at the neighborvals and neighborindex matrices and see if they make sense to you. The first is a sorted version of the earlier d matrix, and the latter gives the corresponding instance numbers. Note that the self-distances (on the diagonal in d) have floated to the top. We're not interested in this (always zero), so we'll skip the top row in the next step.
assignedClasses = mode(neighborclasses(2:1+k,:),1);
So we assign the most common class among the k nearest neighbors!
You can compare the assigned classes with the actual classes to get an accuracy score:
accuracy = 100 * sum(classes == assignedClasses)/length(classes);
fprintf('KNN Classifier Accuracy: %.2f%%\n', 100*accuracy)
Or make a confusion matrix to see the distribution of classifications:
confusionmat(classes, assignedClasses)
yes, there is a function for knn : knnclassify
Play around with the number of neighbors you want to keep in order to get the best result (use a confusion matrix). This function takes care of the distance, of course.