MATLAB Train Interface LIBLINEAR - matlab

In the LIBLINEAR docs, we have
matlab> model = train(training_label_vector, training_instance_matrix [,'liblinear_options', 'col']);
-training_label_vector:
An m by 1 vector of training labels. (type must be double)
-training_instance_matrix:
An m by n matrix of m training instances with n features.
It must be a sparse matrix. (type must be double)
-liblinear_options:
A string of training options in the same format as that of LIBLINEAR.
-col:
if 'col' is set, each column of training_instance_matrix is a data instance. Otherwise each row is a data instance.
However, even after reading the homepage and looking at the docs, I can't find out what the options are for liblinear_options.
Is this listed somewhere but I am clearly missing it?
Futhermore, since I am unable to find liblinear_options listed anywhere, I am stuck with the following question:
Does the train method use a linear SVM to develop a model?

Liblinear is a linear classifier. Besides SVM, it also included logistic regression based classifier. And yes, as its name indicates, the linear kernel is applied in SVM.
You may check their github page for the liblinear_options. I copied them here as well:
"liblinear_options:\n"
"-s type : set type of solver (default 1)\n"
" 0 -- L2-regularized logistic regression (primal)\n"
" 1 -- L2-regularized L2-loss support vector classification (dual)\n"
" 2 -- L2-regularized L2-loss support vector classification (primal)\n"
" 3 -- L2-regularized L1-loss support vector classification (dual)\n"
" 4 -- multi-class support vector classification by Crammer and Singer\n"
" 5 -- L1-regularized L2-loss support vector classification\n"
" 6 -- L1-regularized logistic regression\n"
" 7 -- L2-regularized logistic regression (dual)\n"
"-c cost : set the parameter C (default 1)\n"
"-e epsilon : set tolerance of termination criterion\n"
" -s 0 and 2\n"
" |f'(w)|_2 <= eps*min(pos,neg)/l*|f'(w0)|_2,\n"
" where f is the primal function and pos/neg are # of\n"
" positive/negative data (default 0.01)\n"
" -s 1, 3, 4 and 7\n"
" Dual maximal violation <= eps; similar to libsvm (default 0.1)\n"
" -s 5 and 6\n"
" |f'(w)|_1 <= eps*min(pos,neg)/l*|f'(w0)|_1,\n"
" where f is the primal function (default 0.01)\n"
"-B bias : if bias >= 0, instance x becomes [x; bias]; if < 0, no bias term added (default -1)\n"
"-wi weight: weights adjust the parameter C of different classes (see README for details)\n"
"-v n: n-fold cross validation mode\n"
"-q : quiet mode (no outputs)\n"

Possibly some new developments since this was posted. Running train in the matlab prompt will give you all the options. At least on R2020b with the version of liblinear that I just downloaded.
>> train
Usage: model = train(training_label_vector, training_instance_matrix, 'liblinear_options', 'col');
liblinear_options:
-s type : set type of solver (default 1)
for multi-class classification
0 -- L2-regularized logistic regression (primal)
1 -- L2-regularized L2-loss support vector classification (dual)
2 -- L2-regularized L2-loss support vector classification (primal)
3 -- L2-regularized L1-loss support vector classification (dual)
4 -- support vector classification by Crammer and Singer
5 -- L1-regularized L2-loss support vector classification
6 -- L1-regularized logistic regression
7 -- L2-regularized logistic regression (dual)
for regression
11 -- L2-regularized L2-loss support vector regression (primal)
12 -- L2-regularized L2-loss support vector regression (dual)
13 -- L2-regularized L1-loss support vector regression (dual)
for outlier detection
21 -- one-class support vector machine (dual)
-c cost : set the parameter C (default 1)
-p epsilon : set the epsilon in loss function of SVR (default 0.1)
-n nu : set the parameter nu of one-class SVM (default 0.5)
-e epsilon : set tolerance of termination criterion
-s 0 and 2
|f'(w)|_2 <= eps*min(pos,neg)/l*|f'(w0)|_2,
where f is the primal function and pos/neg are # of
positive/negative data (default 0.01)
-s 11
|f'(w)|_2 <= eps*|f'(w0)|_2 (default 0.0001)
-s 1, 3, 4, 7, and 21
Dual maximal violation <= eps; similar to libsvm (default 0.1 except 0.01 for -s 21)
-s 5 and 6
|f'(w)|_1 <= eps*min(pos,neg)/l*|f'(w0)|_1,
where f is the primal function (default 0.01)
-s 12 and 13
|f'(alpha)|_1 <= eps |f'(alpha0)|,
where f is the dual function (default 0.1)
-B bias : if bias >= 0, instance x becomes [x; bias]; if < 0, no bias term added (default -1)
-R : not regularize the bias; must with -B 1 to have the bias; DON'T use this unless you know what it is
(for -s 0, 2, 5, 6, 11)
-wi weight: weights adjust the parameter C of different classes (see README for details)
-v n: n-fold cross validation mode
-C : find parameters (C for -s 0, 2 and C, p for -s 11)
-q : quiet mode (no outputs)
col:
if 'col' is setted, training_instance_matrix is parsed in column format, otherwise is in row format

Related

Am I thinking about the linear regression model correctly?

I have the following model:
<bound method Model.summary of Class : LinearRegression
Schema
------
Number of coefficients : 18
Number of examples : 21613
Number of feature columns : 17
Number of unpacked features : 17
Hyperparameters
---------------
L1 penalty : 10000000000.0
L2 penalty : 0.0
Training Summary
----------------
Solver : fista
Solver iterations : 10
Solver status : Completed (Iteration limit reached).
Training time (sec) : 1.2776
Settings
--------
Residual sum of squares : 2842629034369063.5
Training RMSE : 364204.5762
Highest Positive Coefficients
-----------------------------
(intercept) : 274873.056
bathrooms : 8468.5311
grade : 842.068
sqft_living_sqrt : 350.0606
sqft_living : 24.4207
Lowest Negative Coefficients
----------------------------
No Negative Coefficients :
Does this mean that my equation would be:
Prediction = 274873.056 + 8468.5311[bathroom] + 842.068[grade]^2 + 350.0606[sqft_living_sqrt]^3 + 24.4207[sqft_living]^4
If that is correct, then how does the model know which features belong to the power 2, power 3, etc. ? If I change the order of the features will the coefficients change?
I am not sure I follow you with the ^2 ... ^3... ^4.. ?
RMSE does ^2 ... ^2 ... ( etc )
This is wat RMSE Does:
Root Mean Squared Error
Find out the difference between original and predicted values.
Square the differences
Sum all squared differences
Take the average of the sum.
Take the square root of the average
You can see my example over here as Math SE
https://math.stackexchange.com/questions/3650442/simple-calculation-from-formula-rmse/3843518#3843518
Just, substitute your values with the RMSE Formula. And calculate the RMSE.
Regards,
//Will

Understanding of threshold value in a neural network

Consider the hypothetical neural network here
$o_1$ is the output of neuron 1.
$o_2$ is the output of neuron 2.
$w_1$ is the weight of connection between 1 and 3.
$w_2$ is the weight of connection between 2 and 3.
So the input to neuron 3 is $i =o_1w_1 +o_2w_2$
Let the activation function of neuron 3 be sigmoid function.
$f(x) = \dfrac{1}{1+e^{-x}}$ and the threshold value of neuron 3 be $\theta$.
Therefore, output of neuron 3 will be $f(i)$ if $i\geq\theta$ and $0$ if $i\lt\theta$.
Am I correct?
Thresholds are used for binary neurons (I forget the technical name), whereas biases are used for sigmoid (and pretty much all modern) neurons. Your understanding of the threshold is correct, but again this is used in neurons where the output is either 1 or 0, which is not very useful for learning (optimization). With a sigmoid neuron, you would simply add the bias (previously the threshold but moved to the other side of the equation), so you're output would be f(weight * input + bias). All the sigmoid function is doing (for the most part) is limiting your output to a value between 0 and 1
I do not think it is the place to ask this sort of questions. You will find lot of NN ressources online. For your simple case, each link has a weight, so basicly the input of neuron 3 is :
Neuron3Input = Neuron1Output * WeightOfLinkNeuron1To3 + Neuron2Output * WeightOfLinkNeuron2To3
+ bias.
Then, to get the output, just use the activation function. Neuron3Output = F_Activation(Neuron3Input)
O3 = F(O1 * W1 + O2 * W2 + Bias)

How to predict a label in svm for real data? [duplicate]

Could you give an example of classification of 4 classes using Support Vector Machines (SVM) in matlab something like:
atribute_1 atribute_2 atribute_3 atribute_4 class
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3
SVMs were originally designed for binary classification. They have then been extended to handle multi-class problems. The idea is to decompose the problem into many binary-class problems and then combine them to obtain the prediction.
One approach called one-against-all, builds as many binary classifiers as there are classes, each trained to separate one class from the rest. To predict a new instance, we choose the classifier with the largest decision function value.
Another approach called one-against-one (which I believe is used in LibSVM), builds k(k-1)/2 binary classifiers, trained to separate each pair of classes against each other, and uses a majority voting scheme (max-win strategy) to determine the output prediction.
There are also other approaches such as using Error Correcting Output Code (ECOC) to build many somewhat-redundant binary-classifiers, and use this redundancy to obtain more robust classifications (uses the same idea as Hamming codes).
Example (one-against-one):
%# load dataset
load fisheriris
[g gn] = grp2idx(species); %# nominal class to numeric
%# split training/testing sets
[trainIdx testIdx] = crossvalind('HoldOut', species, 1/3);
pairwise = nchoosek(1:length(gn),2); %# 1-vs-1 pairwise models
svmModel = cell(size(pairwise,1),1); %# store binary-classifers
predTest = zeros(sum(testIdx),numel(svmModel)); %# store binary predictions
%# classify using one-against-one approach, SVM with 3rd degree poly kernel
for k=1:numel(svmModel)
%# get only training instances belonging to this pair
idx = trainIdx & any( bsxfun(#eq, g, pairwise(k,:)) , 2 );
%# train
svmModel{k} = svmtrain(meas(idx,:), g(idx), ...
'BoxConstraint',2e-1, 'Kernel_Function','polynomial', 'Polyorder',3);
%# test
predTest(:,k) = svmclassify(svmModel{k}, meas(testIdx,:));
end
pred = mode(predTest,2); %# voting: clasify as the class receiving most votes
%# performance
cmat = confusionmat(g(testIdx),pred);
acc = 100*sum(diag(cmat))./sum(cmat(:));
fprintf('SVM (1-against-1):\naccuracy = %.2f%%\n', acc);
fprintf('Confusion Matrix:\n'), disp(cmat)
Here is a sample output:
SVM (1-against-1):
accuracy = 93.75%
Confusion Matrix:
16 0 0
0 14 2
0 1 15
MATLAB does not support multiclass SVM at the moment. You could use svmtrain (2-classes) to achieve this, but it would be much easier to use a standard SVM package.
I have used LIBSVM and can confirm that it's very easy to use.
%%# Your data
D = [
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3];
%%# For clarity
Attributes = D(:,1:4);
Classes = D(:,5);
train = [1 3 5 6];
test = [2 4];
%%# Train
model = svmtrain(Classes(train),Attributes(train,:),'-s 0 -t 2');
%%# Test
[predict_label, accuracy, prob_estimates] = svmpredict(Classes(test), Attributes(test,:), model);

interpret Wilkinson Notation linear regression model matlab

I fitted the following model to my data.
Linear regression model:
NNSB ~ 1 + Gender + Age*MMRC
Estimated Coefficients:
Estimate SE tStat pValue
(Intercept) 1.8004 1.027 1.7531 0.079978
Age 0.014051 0.01529 0.91898 0.35839
Gender_Male 0.43134 0.099535 4.3335 1.6559e-05
MMRC_MMRC 1 -0.64548 1.2465 -0.51785 0.60471
MMRC_MMRC 2 2.5536 1.2689 2.0124 0.044513
MMRC_MMRC 3 2.1066 1.3638 1.5447 0.12283
MMRC_MMRC 4 2.07 1.5724 1.3164 0.18841
Age:MMRC_MMRC 1 0.012023 0.018482 0.65052 0.51555
Age:MMRC_MMRC 2 -0.034328 0.018843 -1.8218 0.06886
Age:MMRC_MMRC 3 -0.026653 0.020307 -1.3125 0.18973
Age:MMRC_MMRC 4 -0.016567 0.023175 -0.71488 0.47489
Could you help me in understanding the meaning of the last 4 coefficients?
How should I read the ":" simbol?
MMRC is a categorical variable that can assume the values 0, 1, 2,3 and 4
Matlab uses what I think is called Wilkinson notation for defining models. This is where the tilde (~) sign comes from too.
When you have a a*b term in your model, Matlab actually also includes any lower order terms. For example: y ~ a*b in Wilkinson notation actually corresponds to y = a + b + a*b in standard notation. If you just want a product in Wilkinson notation without any lower order terms (aka an 'interaction'), this is expressed as a:b. So, y ~ a*b is equivalent to y ~ a + b + a:b.
In your case, the first four MMRC terms correspond to the MMRC's on their own. The final four terms are the interactions between Age and MMRC.

support vector machines in matlab

Could you give an example of classification of 4 classes using Support Vector Machines (SVM) in matlab something like:
atribute_1 atribute_2 atribute_3 atribute_4 class
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3
SVMs were originally designed for binary classification. They have then been extended to handle multi-class problems. The idea is to decompose the problem into many binary-class problems and then combine them to obtain the prediction.
One approach called one-against-all, builds as many binary classifiers as there are classes, each trained to separate one class from the rest. To predict a new instance, we choose the classifier with the largest decision function value.
Another approach called one-against-one (which I believe is used in LibSVM), builds k(k-1)/2 binary classifiers, trained to separate each pair of classes against each other, and uses a majority voting scheme (max-win strategy) to determine the output prediction.
There are also other approaches such as using Error Correcting Output Code (ECOC) to build many somewhat-redundant binary-classifiers, and use this redundancy to obtain more robust classifications (uses the same idea as Hamming codes).
Example (one-against-one):
%# load dataset
load fisheriris
[g gn] = grp2idx(species); %# nominal class to numeric
%# split training/testing sets
[trainIdx testIdx] = crossvalind('HoldOut', species, 1/3);
pairwise = nchoosek(1:length(gn),2); %# 1-vs-1 pairwise models
svmModel = cell(size(pairwise,1),1); %# store binary-classifers
predTest = zeros(sum(testIdx),numel(svmModel)); %# store binary predictions
%# classify using one-against-one approach, SVM with 3rd degree poly kernel
for k=1:numel(svmModel)
%# get only training instances belonging to this pair
idx = trainIdx & any( bsxfun(#eq, g, pairwise(k,:)) , 2 );
%# train
svmModel{k} = svmtrain(meas(idx,:), g(idx), ...
'BoxConstraint',2e-1, 'Kernel_Function','polynomial', 'Polyorder',3);
%# test
predTest(:,k) = svmclassify(svmModel{k}, meas(testIdx,:));
end
pred = mode(predTest,2); %# voting: clasify as the class receiving most votes
%# performance
cmat = confusionmat(g(testIdx),pred);
acc = 100*sum(diag(cmat))./sum(cmat(:));
fprintf('SVM (1-against-1):\naccuracy = %.2f%%\n', acc);
fprintf('Confusion Matrix:\n'), disp(cmat)
Here is a sample output:
SVM (1-against-1):
accuracy = 93.75%
Confusion Matrix:
16 0 0
0 14 2
0 1 15
MATLAB does not support multiclass SVM at the moment. You could use svmtrain (2-classes) to achieve this, but it would be much easier to use a standard SVM package.
I have used LIBSVM and can confirm that it's very easy to use.
%%# Your data
D = [
1 2 3 4 0
1 2 3 5 0
0 2 6 4 1
0 3 3 8 1
7 2 6 4 2
9 1 7 10 3];
%%# For clarity
Attributes = D(:,1:4);
Classes = D(:,5);
train = [1 3 5 6];
test = [2 4];
%%# Train
model = svmtrain(Classes(train),Attributes(train,:),'-s 0 -t 2');
%%# Test
[predict_label, accuracy, prob_estimates] = svmpredict(Classes(test), Attributes(test,:), model);