I am using polynomial SVM in MATLAB for CIFAR-10 dataset using HOG features for data extraction. I wanted to know how I can tune the regularization parameters for 'fitcecoc' to avoid overfitting the training set.
template = templateSVM(...
'KernelFunction', 'polynomial', ...
'PolynomialOrder', 2, ...
'KernelScale', 'auto', ...
'BoxConstraint', 1, ...
'Standardize', true);
SVM_model = fitcecoc(...
X_train, ...
Y_train, ...
'Learners', template, ...
'Coding', 'onevsone', ...
'Verbose', 2,...
'ClassNames', [1; 2; 3; 4; 5; 6; 7; 8; 9; 10]);
Related
I have trained R-CNN network models on a custom dataset and got the results as expected in the end. But I couldn't find where to set the number of iterations before starting the train process and the training continues without any sign of when it's going to stop. Is there a way to set the number of iterations beforehand, so it would stop after specified steps?
This is the code of training the rcnn:
%%%%%%%%%%%%%%%%%%%%%% Define Inputs
imagePath = 'D:\Thesis\Data\VEDAI\vedai\train_images\';
sampleImage = '00000000.png';
objectClasses = {'car','truck','tractor','campingcar','van','other', 'pickup', 'boat', 'plane'};
imageTable = vedaiTrain;
smallestObjectSize = [32, 32, 3];
%%%%%%%%%%%%%%%%%%%%%% Calculations
numClassesPlusBackground = numel(objectClasses) + 1;
t = num2cell(smallestObjectSize);
[height, width, numChannels] = deal(t{:});
imageSize = [height width numChannels];
%%%%%%%%%%%%%%%%%%%%%% Network Layers
%%%%% inputLayer
inputLayer = imageInputLayer(imageSize);
%%%%% middleLayer
filterSize = [5 5];
numFilters = 32;
middleLayers = [
convolution2dLayer(filterSize, numFilters, 'Padding', 2)
reluLayer()
maxPooling2dLayer(3, 'Stride', 2)
convolution2dLayer(filterSize, numFilters, 'Padding', 2)
reluLayer()
maxPooling2dLayer(3, 'Stride',2)
convolution2dLayer(filterSize, 2 * numFilters, 'Padding', 2)
reluLayer()
maxPooling2dLayer(3, 'Stride',2)
]
%%%%% finalLayer
finalLayers = [
fullyConnectedLayer(64)
reluLayer
fullyConnectedLayer(numClassesPlusBackground)
softmaxLayer
classificationLayer
]
Layers = [
inputLayer
middleLayers
finalLayers
]
layers(2).Weights = 0.0001 * randn([filterSize numChannels numFilters]);
%%%%%%%%%%%%%%%%%%%%%% training options
options = trainingOptions('sgdm', ...
'Momentum', 0.9, ...
'InitialLearnRate', 0.001, ...
'LearnRateSchedule', 'piecewise', ...
'LearnRateDropFactor', 0.1, ...
'LearnRateDropPeriod', 8, ...
'L2Regularization', 0.004, ...
'MaxEpochs', 40, ...
'MiniBatchSize', 128, ...
'Verbose', true);
%%%%%%%%%%%%%%%%%%%%%% Train an R-CNN object detector
rcnn = trainRCNNObjectDetector(imageTable,Layers, options, ...
'NegativeOverlapRange', [0 0.3], 'PositiveOverlapRange',[0.5 1]);
It keeps training for iterations until some time, which I don't know how it decides.
In the file train_faster_rcnn_alt_opt.py file, set the max_iters = [80000, 40000, 80000, 40000] parameter to the number of iterations you want at each stage.
Trying to implement a very basic XOR FFNN in TensorFlow. I may just be misunderstanding the code but can anyone see an obvious reason why this won't work-- blows up to NaNs and starts with loss of $0$.
Toggles are on works/ doesn't work if you want to mess around with it.
Thanks!
import math
import tensorflow as tf
import numpy as np
HIDDEN_NODES = 10
x = tf.placeholder(tf.float32, [None, 2])
W_hidden = tf.Variable(tf.truncated_normal([2, HIDDEN_NODES]))
b_hidden = tf.Variable(tf.zeros([HIDDEN_NODES]))
hidden = tf.nn.relu(tf.matmul(x, W_hidden) + b_hidden)
#-----------------
#DOESN"T WORK
W_logits = tf.Variable(tf.truncated_normal([HIDDEN_NODES, 1]))
b_logits = tf.Variable(tf.zeros([1]))
logits = tf.add(tf.matmul(hidden, W_logits),b_logits)
#WORKS
# W_logits = tf.Variable(tf.truncated_normal([HIDDEN_NODES, 2]))
# b_logits = tf.Variable(tf.zeros([2]))
# logits = tf.add(tf.matmul(hidden, W_logits),b_logits)
#-----------------
y = tf.nn.softmax(logits)
#-----------------
#DOESN"T WORK
y_input = tf.placeholder(tf.float32, [None, 1])
#WORKS
#y_input = tf.placeholder(tf.float32, [None, 2])
#-----------------
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, y_input)
loss = tf.reduce_mean(cross_entropy)
loss = cross_entropy
train_op = tf.train.GradientDescentOptimizer(0.1).minimize(loss)
init_op = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init_op)
xTrain = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
#-----------------
#DOESN"T WORK
yTrain = np.array([[0], [1], [1], [0]])
# WORKS
#yTrain = np.array([[1, 0], [0, 1], [0, 1], [1, 0]])
#-----------------
for i in xrange(500):
_, loss_val,logitsval = sess.run([train_op, loss,logits], feed_dict={x: xTrain, y_input: yTrain})
if i % 10 == 0:
print "Step:", i, "Current loss:", loss_val,"logits",logitsval
print sess.run(y,feed_dict={x: xTrain})
TL;DR: For this to work, you should use
loss = tf.nn.l2_loss(logits - y_input)
...instead of tf.nn.softmax_cross_entropy_with_logits.
The tf.nn.softmax_cross_entropy_with_logits operator expects the logits and labels inputs to be a matrix of size batch_size by num_classes. Each row of logits is an unscaled probability distribution across the classes; and each row of labels is a one-hot encoding of the true class for each example in the batch. If the inputs do not match these assumptions, the training process may diverge.
In this code, the logits are batch_size by 1, which means that there is only a single class, and the softmax outputs a prediction of class 0 for all of the examples; the labels are not one-hot. If you look at the implementation of the operator, the backprop value for tf.nn.softmax_cross_entropy_with_logits is:
// backprop: prob - labels, where
// prob = exp(logits - max_logits) / sum(exp(logits - max_logits))
This will be [[1], [1], [1], [1]] - [[0], [1], [1], [0]] in every step, which clearly does not converge.
I used ADAPT for incremental training a simple network, and i know that ADAPT changes weights and biases,i used this:
clc
clear all
net = linearlayer([0 1 2]);
pi = {[1; 1] [2;2]};
p = {[3 ;4] [5; 6] [7;8]};
t={[40; 50; 60] [10 ;20; 30] [70;60;50]};
net=configure(net,p,t);
net.inputweights{1}.learnparam.lr=0.001
net.adaptParam.passes = 10;
for i=1:1
[net,y,E,pf,af] = adapt(net,p,t,pi);
end
after that i simulate that network with the same input:
y1=sim(net,p,pi);
I expect that y =y1, but the results y1 and y are not equal!!
Why there is differnce betweet network output training with ADAPT(y) and the output of the trained network(y1)!?
What does ADAPT do?
there you have it:
net = linearlayer([0 1 2]);
pi = {[1; 1] [2;2]};
p = {[3 ;4] [5; 6] [7;8]};
t = {[40; 50; 60] [10 ;20; 30] [70;60;50]};
net = configure(net,p,t);
net.inputweights{1}.learnparam.lr = 0.001;
net.adaptParam.passes = 10;
view(net)
[net,y,E,pf,af] = train(net,p,t);
tout = net(p);
You would use adapt() for post training applications. The matlab documentation specifically says that you use adapt after it has been trained and the network adapts as it is simulated (http://www.mathworks.com/help/nnet/ref/adapt.html)
THANK YOU BRIAN.
So i shoud use ADAPT after training? like this code?
net = linearlayer([0 1 2]);
pi = {[1; 1] [2;2]};
p = {[3 ;4] [5; 6] [7;8]};
t = {[40; 50; 60] [10 ;20; 30] [70;60;50]};
net = configure(net,p,t);
net.inputweights{1}.learnparam.lr = 0.001;
net.adaptParam.passes = 10;
view(net)
[net,y,E,pf,af] = train(net,p,t);
tout = net(p);
for i=1:100
[net,y,E,pf,af] = adapt(net,p,t,pi);
end
If yes,is this incremental training!?
I would like to write a matlab function to find an equation of a linear classifier for 2 separable sets of points using one single-layer perceptron. I have got 2 files:
script file - run.m:
x_1 = [3, 3, 2, 4, 5];
y_1 = [3, 4, 5, 2, 2];
x_2 = [6, 7, 5, 9, 8];
y_2 = [3, 3, 4, 2, 5];
target_array = [0 0 0 0 0 1 1 1 1 1];
[ func ] = classify_perceptron([x_1 x_2; y_1 y_2], target_array);
x = -2:10;
y = arrayfun(func, x);
plot(x_1, y_1, 'o', x_2, y_2, 'X', x, y);
axis([-2, 10, -2, 10]);
classify_perceptron.m
function [ func ] = classify_perceptron( points, target )
% points - matrix of x,y coordinates
% target - array of expected results
% func - function handler which appropriately classifies a point
% given by x, y arguments supplied to this function
target_arr = target;
weights = rand(1, 2);
translation = rand();
for i=1:size(points, 2)
flag = true;
while flag
result = weights * points(:, i) + translation;
y = result > 0;
e = target_arr(1, i) - y;
if e ~= 0
weights = weights + (e * points(:, i))';
translation = translation + e;
else
flag = false;
end
end
end
func = #(x)(-(translation + (weights(1, 1) * x)) / weights(1, 2));
return
end
The problem is that I don't know where I am making the mistake that leads to incorrect result. It looks like the slope of the line is right, however translation should be a bit bigger. I would be really thankful for pointing me in the right direction. The result I get is presented in the picture below:
Ok, so I have made a significant progress. In case someone runs into the same problem I present to you the solution. The problem has been solved by adding a variable learning_rate = 0.1 and packing the loop iterating over points into another loop iterating as many times as specified in the variable epochs (e.g. 300) .
I'm trying to compare the result of newff with different number of hidden layer but the result is the same. I used 1 hidden layer and 2 hidden layers to compare.
net = newff( minmax( pn ), [5 1], {'tansig' 'purelin'}, 'trainlm');
net = newff( minmax( pn ), [5 5 1], {'tansig' 'tansig' 'purelin'}, 'trainlm');
code:
load data.txt;
P = data(1:20,1:3);
T = data(1:20,4);
[a,minp,maxp,b,mint,maxt] = premnmx(P',T');
net = newff( minmax( pn ), [5 1], {'tansig' 'purelin'}, 'trainlm');
net.trainParam.epochs = 10000;
net.trainParam.show = 5;
net = train(net,a,b);
y = sim(net,a)
x = postmnmx(y',mint,maxt);
plot(x, 'r');
hold
plot(T);
What is the problem here?
May I suggest you to use a GUI based matlab command nprtools for neural networks.