I post here because I don't know how to improve the performance of my binary KNN.
The problem is that I have 99.8% Specificity and only 82% Sensitivity, but I'd rather have more Sensitivity than Specificity.
I'm new to this field, and I've only been working on this for 1 month.
In my study I've used an anomaly detector trained with only 1 class, and in that case, in order to raise the Sensitivity of the knn classifier I increased the value of the threshold...
Now that I have to compare my anomaly detector with a 2 class classifier it seems that KNN works better in the first case... The geometric mean of Sensitivity and Specificity (√Se*Sp) is 0,95 in the one-class classifier, and only 0,91 in the two-classes because of the low Sensitivity. I expected the exact opposite...
Can anyone help me?
Related
I am working on Multi-Class classification problem, where I have designed
SVM classifier
for
31 different classes
After the classification process I got the predicted labels now I want to calculate the accuracy rate for each distinct class. Although I have calculated the overall accuracy of model but I want more detail.
Can some one provide me Matlab code for this purpose or guide me to calculate the desired result.
I am solving a classification problem. I train my unsupervised neural network for a set of entities (using skip-gram architecture).
The way I evaluate is to search k nearest neighbours for each point in validation data, from training data. I take weighted sum (weights based on distance) of labels of nearest neighbours and use that score of each point of validation data.
Observation - As I increase the number of epochs (model1 - 600 epochs, model 2- 1400 epochs and model 3 - 2000 epochs), my AUC improves at smaller values of k but saturates at the similar values.
What could be a possible explanation of this behaviour?
[Reposted from CrossValidated]
To cross check if imbalanced classes are an issue, try fitting a SVM model. If that gives a better classification(possible if your ANN is not very deep) it may be concluded that classes should be balanced first.
Also, try some kernel functions to check if this transformation makes data linearly separable?
As Known, there are classifiers that have a training or a learning step, like SVM or Random Forest. On the other hand, KNN does not have.
Can KNN be better than these classifiers?
If no, why?
If yes, when, how and why?
The main answer is yes, it can due to no free lunch theorem implications. FLT can be loosley stated as (in terms of classification)
There is no universal classifier which is consisntenly better at any task than others
It can also be (not very strictly) inverted
For each (well defined) classifier there exists a dataset where it is the best one
And in particular - kNN is well-defined classifier, in particular it is consistent with any distibution, which means that given infinitely many training points it converges to the optimal, Bayesian separator.
So can it be better than SVM or RF? Obviously! When? There is no clear answer. First of all in supervised learning you often actually get just one training set and try to fit the best model. In such scenario any model can be the best one. When statisticians/theoretical ML try to answer whether one model is better than another, we actually try to test "what would happen if we would have ifinitely many training sets" - so we look at the expected value of the behaviour of the classifiers. In such setting, we often show that SVM/RF is better than KNN. But it does not mean that they are always better. It only means, that for randomly selected dataset you should expect KNN to work worse, but this is only probability. And as you can always win in a lottery (no matter the odds!) you can also always win with KNN (just to be clear - KNN has bigger chances of being a good model than winning a lottery :-)).
What are particular examples? Let us for example consider a rotated XOR problem.
If the true decision boundaries are as above, and you only have this four points. Obviously 1NN will be much better than SVM (with dot, poly or rbf kernel) or RF. It should also be true once you include more and more training points.
"In general kNN would not be expected to exceed SVM or RF. When kNN does, that says something very interesting about the training data. If many doublets are present i the data set, a nearest neighbor algorithm works very well."
I heard the argument something like as written by Claudia Perlich in this podcast:
http://www.thetalkingmachines.com/blog/2015/6/18/working-with-data-and-machine-learning-in-advertizing
My intuitive understanding of why RF and SVM is better kNN in generel: All algorithms basicly assume some local similarity, such that samples very alike gets classified alike. kNN can only choose the most similar samples by distance(or some other global kernel). So the samples which could influence a prediction on kNN would exists within a hyper sphere for the Euclidean distance kernel. RF and SVM can learn other definitions of locality which could stretch far by some features and short by others. Also the propagation of locality could take up many learned shapes, and these shapes can differ through out the feature space.
I have a data set to classify.By using KNN algo i am getting an accuracy of 90% but whereas by using SVM i just able to get over 70%. Is SVM not better than KNN. I know this might be stupid to ask but, what are the parameters for SVM which will give nearly approximate results as KNN algo. I am using libsvm package on matlab R2008
kNN and SVM represent different approaches to learning. Each approach implies different model for the underlying data.
SVM assumes there exist a hyper-plane seperating the data points (quite a restrictive assumption), while kNN attempts to approximate the underlying distribution of the data in a non-parametric fashion (crude approximation of parsen-window estimator).
You'll have to look at the specifics of your scenario to make a better decision as to what algorithm and configuration are best used.
It really depends on the dataset you are using. If you have something like the first line of this image ( http://scikit-learn.org/stable/_images/plot_classifier_comparison_1.png ) kNN will work really well and Linear SVM really badly.
If you want SVM to perform better you can use a Kernel based SVM like the one in the picture (it uses a rbf kernel).
If you are using scikit-learn for python you can play a bit with code here to see how to use the Kernel SVM http://scikit-learn.org/stable/modules/svm.html
kNN basically says "if you're close to coordinate x, then the classification will be similar to observed outcomes at x." In SVM, a close analog would be using a high-dimensional kernel with a "small" bandwidth parameter, since this will cause SVM to overfit more. That is, SVM will be closer to "if you're close to coordinate x, then the classification will be similar to those observed at x."
I recommend that you start with a Gaussian kernel and check the results for different parameters. From my own experience (which is, of course, focused on certain types of datasets, so your mileage may vary), tuned SVM outperforms tuned kNN.
Questions for you:
1) How are you selecting k in kNN?
2) What parameters have you tried for SVM?
3) Are you measuring accuracy in-sample or out-of-sample?
I am trying to use LibSVM for regression. I am trying to detect faces (10 classes of different faces). I labeled 1-10 as face class and 11 is for non face. I want to develop a script usig LibSVM which will give me a continuous score between 0-1 if the test image falls to any of the 10 face class, otherwise it will give me -1 (non-face). From this score, I can be able to predict my calss. If the test image is matched with 1st class, the score should be around .1. Similarly if the test image is matched with class 10, the score should be around 1 (any continuous value close to 1). I am trying to use SVR using LibSVM to solve this problem. I can easily get the predicted class through classification. But I want a continuous score value which I can get through regression. Now, I was looking in the net for the function or parameters in function for SVR using LibSVM, but I couldn't find anything. Can anybody please help me in this regard?
This is not a regression problem. Solving it through regression will not yield good results.
You are dealing with a multiclass classification problem. The best way to approach this is to construct 10 one-vs-all classifiers with probabilistic output. To get probabilistic output (e.g. in the interval [0,1]), you can train and predict with the -b 1 option for C-SVC (-s 0).
If any of the 10 classifiers yields a sufficiently large probability for its positive class, you use that probability (which is close to 1). If none of the 10 classifiers yield a positive label with high enough confidence you can default to -1.
So in short: make a multiclass classifier containing one-vs-all classifiers with probabilistic output. Subsequently post-process the predictions as I described, using a probability threshold of your choice (for example 0.7).