I am new to MATLAB and to LIBSVM. I got the fact that to use precomputed kernel, you must include sample serial number as the first column of the training and testing data. But importantly what I don't understand is: why would I use precomputed kernels? I am following the script described in LIBSVM and got results for linear and RBF but do not understand the need to use precomputed kernels.
Whether you need precomputed kernels depends on your metrics of input vectors. Sometimes you may need a different similarity function other than the norm-2 Euclidean distance or the radial basis function between a given feature to all the features from training set.
It is also possible that you need a multiplicative kernel (for example the product of two different kernel functions), which is not feasible by choosing one kernel type from Libsvm. Then writing a precomputed kernel is a good option.
Related
I am new to machine learning and I am trying to do unsupervised learning with k-means clustering (even if I read that k-means cannot work very well with categorical data). I encoded my categorical variables and tried to apply kernel PCA since I have a categorical feature (it is gender). I noticed that there are several values for the kernel parameter which are 'linear', 'poly', 'rbf', 'sigmoid', 'cosine' and 'precomputed'.
I searched on internet but I couldn't find any proper explanation on these. I could not be sure if the usage of kernel at PCA and SVM are the same either. Is there anyone who can explain what they are, when they should be used and/or how to choose the correct one for our dataset? Since we cannot visualize our dataset with more than 3 dimensions, how will we decide its shape to choose the correct parameter? Part of the code is below just to show where the parameter is used:
# Applying Kernel PCA
from sklearn.decomposition import KernelPCA
kpca = KernelPCA(n_components = 2, kernel = 'linear')
X = kpca.fit_transform(X)
Thank you in advance.
None of these predefined kernels supports mixed data either. They are vector kernels.
Linear kennel should give the same result as non-kernel PCA, just a lot slower.
There is not much relationship to SVM except the use of kernels. And kernels like rbf make much more sense when you can do hyperparameter optimization in a supervised classification task. Since choosing such parameters is hard, making good use of KernelPCA is difficult except for toy problems.
I want to use SVM with polynomial kernel for my binary classification problem? Can SVM with polynomial kernel be used for binary classification?
The Kernel Functions are independent from multi-class classification. There purpose is to transform non-linearly seperatable data into an higher dimensional feature space. This allows the SVM to learn non-linear decision boundaries. So, as stated before, the answer is yes. You can use the RBF for binary classification. SVMs are inherently binary classifiers.
It's interesting that you want to choose the Kernel Function before proper evaluation.
I'd recommend you to test all the functions and see which one performs best on your data. It's hard to determine this by simply viewing the data...depending on your dimensions.
I am trying to find a way to visualize the data with high-dimensional input for two-class classification in SVM, before analysis to decide which kernel to use. In documents online, the visualization of data is given only for two dimensional inputs (I mean two attributes).
Another question rises: What if I have multi-class and more than two attributes?
To visualize, the data should be represented by 3 or less dimension.
Simply PCA can be applied to reduce dimension.
use pre-image using MDS.
refer to a paper The pre-image problem in kernel methods and its matlab code in http://www.cse.ust.hk/~jamesk/publication.html
I have a data set to classify.By using KNN algo i am getting an accuracy of 90% but whereas by using SVM i just able to get over 70%. Is SVM not better than KNN. I know this might be stupid to ask but, what are the parameters for SVM which will give nearly approximate results as KNN algo. I am using libsvm package on matlab R2008
kNN and SVM represent different approaches to learning. Each approach implies different model for the underlying data.
SVM assumes there exist a hyper-plane seperating the data points (quite a restrictive assumption), while kNN attempts to approximate the underlying distribution of the data in a non-parametric fashion (crude approximation of parsen-window estimator).
You'll have to look at the specifics of your scenario to make a better decision as to what algorithm and configuration are best used.
It really depends on the dataset you are using. If you have something like the first line of this image ( http://scikit-learn.org/stable/_images/plot_classifier_comparison_1.png ) kNN will work really well and Linear SVM really badly.
If you want SVM to perform better you can use a Kernel based SVM like the one in the picture (it uses a rbf kernel).
If you are using scikit-learn for python you can play a bit with code here to see how to use the Kernel SVM http://scikit-learn.org/stable/modules/svm.html
kNN basically says "if you're close to coordinate x, then the classification will be similar to observed outcomes at x." In SVM, a close analog would be using a high-dimensional kernel with a "small" bandwidth parameter, since this will cause SVM to overfit more. That is, SVM will be closer to "if you're close to coordinate x, then the classification will be similar to those observed at x."
I recommend that you start with a Gaussian kernel and check the results for different parameters. From my own experience (which is, of course, focused on certain types of datasets, so your mileage may vary), tuned SVM outperforms tuned kNN.
Questions for you:
1) How are you selecting k in kNN?
2) What parameters have you tried for SVM?
3) Are you measuring accuracy in-sample or out-of-sample?
I have a dataset for text classification ready to be used in MATLAB. Each document is a vector in this dataset and the dimensionality of this vector is extremely high. In these cases peopl usually do some feature selection on the vectors like the ones that you have actually find the WEKA toolkit. Is there anything like that in MATLAB? if not can u suggest and algorithm for me to do it...?
thanks
MATLAB (and its toolboxes) include a number of functions that deal with feature selection:
RANDFEATURES (Bioinformatics Toolbox): Generate randomized subset of features directed by a classifier
RANKFEATURES (Bioinformatics Toolbox): Rank features by class separability criteria
SEQUENTIALFS (Statistics Toolbox): Sequential feature selection
RELIEFF (Statistics Toolbox): Relief-F algorithm
TREEBAGGER.OOBPermutedVarDeltaError, predictorImportance (Statistics Toolbox): Using ensemble methods (bagged decision trees)
You can also find examples that demonstrates usage on real datasets:
Identifying Significant Features and Classifying Protein Profiles
Genetic Algorithm Search for Features in Mass Spectrometry Data
In addition, there exist third-party toolboxes:
Matlab Toolbox for Dimensionality Reduction
LIBGS: A MATLAB Package for Gene Selection
Otherwise you can always call your favorite functions from WEKA directly from MATLAB since it include a JVM...
Feature selection depends on the specific task you want to do on the text data.
One of the simplest and crudest method is to use Principal component analysis (PCA) to reduce the dimensions of the data. This reduced dimensional data can be used directly as features for classification.
See the tutorial on using PCA here:
http://matlabdatamining.blogspot.com/2010/02/principal-components-analysis.html
Here is the link to Matlab PCA command help:
http://www.mathworks.com/help/toolbox/stats/princomp.html
Using the obtained features, the well known Support Vector Machines (SVM) can be used for classification.
http://www.mathworks.com/help/toolbox/bioinfo/ref/svmclassify.html
http://www.autonlab.org/tutorials/svm.html
You might consider using the independent features technique of Weiss and Kulikowski to quickly eliminate variables which are obviously unimformative:
http://matlabdatamining.blogspot.com/2006/12/feature-selection-phase-1-eliminate.html