MATLAB - How to change "Validation Check" count - matlab

How can I change "Validation Checks" value from 6 to higher or lower values using code?
I have following code:
% Create a Pattern Recognition Network
hiddenLayerSize = ns;
net = patternnet(hiddenLayerSize);
net.divideParam.trainRatio = trRa/100;
net.divideParam.valRatio = vaRa/100;
net.divideParam.testRatio = teRa/100;
% Train the Network
[net,tr] = train(net,inputs,targets);
% Test the Network
outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs);
% Recalculate Training, Validation and Test Performance
trainTargets = targets .* tr.trainMask{1};
valTargets = targets .* tr.valMask{1};
testTargets = targets .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,outputs);
valPerformance = perform(net,valTargets,outputs);
testPerformance = perform(net,testTargets,outputs);
I could not find a clue at http://www.mathworks.com/help/nnet/ug/train-and-apply-multilayer-neural-networks.html

TL;DL: net.trainParam.max_fail = 8;
I've used the example provided in the page you linked to get a working instance of nntraintool.
When you open nntraintool.m you see a small piece of documentation that says (among else):
% net.trainParam.showWindow = false;
This hinted that some properties are stored within net.trainParam. When querying it to see what it holds you get:
ans =
Function Parameters for 'trainlm'
Show Training Window Feedback showWindow: true
Show Command Line Feedback showCommandLine: false
Command Line Frequency show: 25
Maximum Epochs epochs: 1000
Maximum Training Time time: Inf
Performance Goal goal: 0
Minimum Gradient min_grad: 1e-07
Maximum Validation Checks max_fail: 6
Mu mu: 0.001
Mu Decrease Ratio mu_dec: 0.1
Mu Increase Ratio mu_inc: 10
Maximum mu mu_max: 10000000000
Here you can see how the Maximum Validation Checks is stored: in a field named max_fail. Now it was only a case of testing if it was a read-only field or not, which could easily be tested using net.trainParam.max_fail = 8; train(net,...); - which correctly changed the default value from 6 to 8.

Related

Correct practice and approach for reporting the training and generalization performance

I am trying to learn the correct procedure for training a neural network for classification. Many tutorials are there but they never explain how to report for the generalization performance. Can somebody please tell me if the following is the correct method or not. I am using first 100 examples from the fisheriris data set that has labels 1,2 and call them as X and Y respectively. Then I split X into trainData and Xtest with a 90/10 split ratio. Using trainData I trained the NN model. Now the NN internally further splits trainData into tr,val,test subsets. My confusion is which one is usually used for generalization purpose when reporting the performance of the model to unseen data in conferences/Journals?
The dataset can be found in the link: https://www.mathworks.com/matlabcentral/fileexchange/71468-simple-neural-networks-with-k-fold-cross-validation-manner
rng('default')
load iris.mat;
X = [f(1:100,:) l(1:100)];
numExamples = size(X,1);
indx = randperm(numExamples);
X = X(indx,:);
Y = X(:,end);
split1 = cvpartition(Y,'Holdout',0.1,'Stratify',true); %90% trainval 10% test
istrainval = training(split1); % index for fitting
istest = test(split1); % indices for quality assessment
trainData = X(istrainval,:);
Xtest = X(istest,:);
Ytest = Y(istest);
numExamplesXtrainval = size(trainData,1);
indxXtrainval = randperm(numExamplesXtrainval);
trainData = trainData(indxXtrainval,:);
Ytrain = trainData(:,end);
hiddenLayerSize = 10;
% data format = rows = number of dim, column = number of examples
net = patternnet(hiddenLayerSize);
net = init(net);
net.performFcn = 'crossentropy';
net.trainFcn = 'trainscg';
net.trainParam.epochs=50;
[net tr]= train(net,trainData', Ytrain');
Trained = sim(net, trainData'); %outputs predicted labels
train_predict = net(trainData');
performanceTrain = perform(net,Ytrain',train_predict)
lbl_train=grp2idx(Ytrain);
Yhat_train = (train_predict >= 0.5);
Lbl_Yhat_Train = grp2idx(Yhat_train);
[cmMatrixTrain]= confusionmat(lbl_train,Lbl_Yhat_Train )
accTrain=sum(lbl_train ==Lbl_Yhat_Train)/size(lbl_train,1);
disp(['Training Set: Total Train Acccuracy by MLP = ',num2str(100*accTrain ), '%'])
[confTest] = confusionmat(lbl_train(tr.testInd),Lbl_Yhat_Train(tr.testInd) )
%unknown test
test_predict = net(Xtest');
performanceTest = perform(net,Ytest',test_predict);
Yhat_test = (test_predict >= 0.5);
test_lbl=grp2idx(Ytest);
Lbl_Yhat_Test = grp2idx(Yhat_test);
[cmMatrix_Test]= confusionmat(test_lbl,Lbl_Yhat_Test )
This is the output.
Problem1: There seems to be no prediction for the other class. Why?
Problem2: Do I need a separate dataset like the one I created as Xtest for reporting generalization error or is it the practice to use the data trainData(tr.testInd,:) as the generalization test set? Did I create an unnecessary subset?
performanceTrain =
2.2204e-16
cmMatrixTrain =
45 0
45 0
Training Set: Total Train Acccuracy by MLP = 50%
confTest =
9 0
5 0
cmMatrix_Test =
5 0
5 0
There are a few issues with the code. Let's deal with them before answering your question. First, you set a threshold of 0.5 for making decisions (Yhat_train = (train_predict >= 0.5);) while all points of your net prediction are above 0.5. This means you only get zeros in your confusion matrices. You can plot the scores to choose a better threshold:
figure;
plot(train_predict(Ytrain == 1),'.b')
hold on
plot(train_predict(Ytrain == 2),'.r')
legend('label 1','label 2')
cvpartition gave me an error. It ran successfully as split1 = cvpartition(Y,'Holdout',0.1); In any case, artificial neural networks usuallly manage partitioning within the training process, so you feed in X and Y and some parameters for how to do it. See here for example: link where you set
net.divideParam.trainRatio = .4;
net.divideParam.valRatio = .3;
net.divideParam.testRatio = .3;
So how to report the results? Only for the test data. The train data will suffer from overfit, and will show false, too good results. If you use validation data (you havn't), then you cannot show results for it because it will also suffer from overfit. If you let the training do validation for you your test results will be safe from overfit.

Error using Zeros in Matlab while training

I am Trying to train my Neural Network using vInputs and vTargets to create a model net
But I keep getting this irritating Error no matter what I try to correct in my code:
ERROR: Error using zeros.Maximum variable size allowed by the program is exceeded.
I dont find any such big data created by my code and I have a RAM of 4GB
I have 230 images which I need to classify in 3 parts 'Sedan', 'SUV', 'Hatchback'
image size [15x26]
I used this code to convert single image's features extracted by gabor filter to Feature Vector
function IMVECTOR = im2vec (img)
load Gabor;
img = adapthisteq(img);
Features75x208 = cell(5,8);
for s = 1:5
for j = 1:8
Features75x208{s,j} = mminmax(abs(ifft2(G{s,j}.*fft2(double(img),32,32),15,26)));
end
end
Features25x70 = cell2mat(Features75x208);
Features25x70 (3:3:end,:)=[];
Features25x70 (2:2:end,:)=[];
Features25x70 (:,3:3:end)=[];
Features25x70 (:,2:2:end)=[];
IMVECTOR = reshape (Features25x70,[1750 1]);
Now after the feature vector is created I made vInputs using same function for 230 images.
Thus I got
1. vInputs=[1750x230].
2. vTargets=[4x230].
But whenever I run the Project I get error in function.
function net = create_pr_net(inputs,targets)
%CREATE_PR_NET Creates and trains a pattern recognition neural network.
% Create Network
numHiddenNeurons = 35;
net = newpr(inputs,targets,numHiddenNeurons);
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
% Train and Apply Network
[net,tr] = train(net,inputs,targets);% <== ERROR OCCURS HERE<==
outputs = sim(net,inputs);
% Plot
plotperf(tr);
plotconfusion(targets,outputs);
save net net
Here is the complete Error:
Error using zeros
Maximum variable size allowed by the program is exceeded.
Error in nnMex2.codeHints (line 117)
hints.TEMP =
zeros(1,ceil(tempSize/8),'double');
Error in nncalc.setup2 (line 13)
calcHints = calcMode.codeHints(calcHints);
Error in network/train (line 306)
[calcLib,calcNet] = nncalc.setup2(calcMode,calcNet,calcData,calcHints);
Error in create_pr_net (line 14)
[net,tr] = train(net,inputs,targets);
Error in executeMe (line 22)
net=create_pr_net(vInputs,vTargets);
Please Help me and ask me if I missed anything and need to specify some more details.
value of tempSize:
EDIT: Well I figured it out that as I am using 32bit system so I can address a maximum of 2^32 = 4.294967e+09 at a time.
While if I was using 64bit I could be able to address about 2^64 = 9.22337e+18 address at a time.
So can you guys give me some idea about how to make it work on my system.

Out of Memory errors in MATLAB NN Toolbox and data division problems in training datasets

I am facing problem in running huge data set in matlab NN Toolbox- the problem is-> when i use trainlm algorithm, NN Toolbox fails to run the data and shows Out of memory error, but for other algorithms there is no memory problem. Why is this so? Moreover when i put hidden neuron more than 15 it also shows out of memory. How to solve this kind of problems?
One more thing: i put 10, 45, 45 % data division for training -validation and testing, but after running the codes i found that in the workspace it executed 25% data for training, 37% data for validation, and 37% data for testing purpose. How to resolve this issue?
Do anybody have idea how to solve this kind of problems? I will be glad to have the comments and any kind of suggestion. Thanks.
I am using R2010b version of MATLAB in my laptop which is running in Windows 7.
Here is the code i used for training the dataset
EX_355 = xlsread('Training Dataset.xlsx','B2:B435106');
EX_532 = xlsread('Training Dataset.xlsx','C2:C435106');
BA_355 = xlsread('Training Dataset.xlsx','D2:D435106');
BA_532 = xlsread('Training Dataset.xlsx','E2:E435106');
BA_1064 = xlsread('Training Dataset.xlsx','F2:F435106');
Reff = xlsread('Training Dataset.xlsx','G2:G435106');
Input(1,:) = EX_355;
Input(2,:) = EX_532;
Input(3,:) = BA_355;
Input(4,:) = BA_532;
Input(5,:) = BA_1064;
Target(1,:) = Reff;
net = feedforwardnet;
net = configure(net,Input,Target);
net = init(net);
inputs = Input;
targets = Target;
hiddenLayerSize = 10;
net = fitnet(hiddenLayerSize);
net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
net.outputs{2}.processFcns = {'removeconstantrows','mapminmax'};
net.divideFcn = 'dividerand';
net.divideMode = 'sample';
net.divideParam.trainRatio = 10/100;
net.divideParam.valRatio = 45/100;
net.divideParam.testRatio = 45/100;
net.trainFcn = 'trainlm';
net.performFcn = 'mse';
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ... 'plotregression', 'plotfit'};
[net,tr] = train(net,inputs,targets);
outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs)
trainTargets = targets .* tr.trainMask{1};
valTargets = targets .* tr.valMask{1};
testTargets = targets .* tr.testMask{1};
net.trainParam.epochs;
net.trainParam.time;
net.trainParam.goal;
net.trainParam.min_grad;
net.trainParam.mu_max;
net.trainParam.max_fail;
net.trainParam.show;
paste this before "train"
net.efficiency.memoryReduction = NUMBER;
change this number till the code run
you can increment from 1 --> inf
more description is available # http://www.mathworks.com/help/nnet/ug/train-the-network.html
You can check this link http://www.mathworks.co.uk/help/nnet/ug/optimize-neural-network-training-speed-and-memory.html
This is a quote from MathWorks:
If MATLAB is being used and memory limitations are a problem, the amount of temporary storage needed can be reduced by a factor of N, in exchange for performing the computations N times sequentially on each of N subsets of the data.
net2 = train(net1,x,t,'reduction',N);
This is called memory reduction.
Keep increasing the value of N till your code runs.

Problems with real-valued input deep belief networks (of RBMs)

I am trying to recreate the results reported in Reducing the dimensionality of data with neural networks of autoencoding the olivetti face dataset with an adapted version of the MNIST digits matlab code, but am having some difficulty. It seems that no matter how much tweaking I do on the number of epochs, rates, or momentum the stacked RBMs are entering the fine-tuning stage with a large amount of error and consequently fail to improve much at the fine-tuning stage. I am also experiencing a similar problem on another real-valued dataset.
For the first layer I am using a RBM with a smaller learning rate (as described in the paper) and with
negdata = poshidstates*vishid' + repmat(visbiases,numcases,1);
I'm fairly confident I am following the instructions found in the supporting material but I cannot achieve the correct errors.
Is there something I am missing? See the code I'm using for real-valued visible unit RBMs below, and for the whole deep training. The rest of the code can be found here.
rbmvislinear.m:
epsilonw = 0.001; % Learning rate for weights
epsilonvb = 0.001; % Learning rate for biases of visible units
epsilonhb = 0.001; % Learning rate for biases of hidden units
weightcost = 0.0002;
initialmomentum = 0.5;
finalmomentum = 0.9;
[numcases numdims numbatches]=size(batchdata);
if restart ==1,
restart=0;
epoch=1;
% Initializing symmetric weights and biases.
vishid = 0.1*randn(numdims, numhid);
hidbiases = zeros(1,numhid);
visbiases = zeros(1,numdims);
poshidprobs = zeros(numcases,numhid);
neghidprobs = zeros(numcases,numhid);
posprods = zeros(numdims,numhid);
negprods = zeros(numdims,numhid);
vishidinc = zeros(numdims,numhid);
hidbiasinc = zeros(1,numhid);
visbiasinc = zeros(1,numdims);
sigmainc = zeros(1,numhid);
batchposhidprobs=zeros(numcases,numhid,numbatches);
end
for epoch = epoch:maxepoch,
fprintf(1,'epoch %d\r',epoch);
errsum=0;
for batch = 1:numbatches,
if (mod(batch,100)==0)
fprintf(1,' %d ',batch);
end
%%%%%%%%% START POSITIVE PHASE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
data = batchdata(:,:,batch);
poshidprobs = 1./(1 + exp(-data*vishid - repmat(hidbiases,numcases,1)));
batchposhidprobs(:,:,batch)=poshidprobs;
posprods = data' * poshidprobs;
poshidact = sum(poshidprobs);
posvisact = sum(data);
%%%%%%%%% END OF POSITIVE PHASE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
poshidstates = poshidprobs > rand(numcases,numhid);
%%%%%%%%% START NEGATIVE PHASE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
negdata = poshidstates*vishid' + repmat(visbiases,numcases,1);% + randn(numcases,numdims) if not using mean
neghidprobs = 1./(1 + exp(-negdata*vishid - repmat(hidbiases,numcases,1)));
negprods = negdata'*neghidprobs;
neghidact = sum(neghidprobs);
negvisact = sum(negdata);
%%%%%%%%% END OF NEGATIVE PHASE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
err= sum(sum( (data-negdata).^2 ));
errsum = err + errsum;
if epoch>5,
momentum=finalmomentum;
else
momentum=initialmomentum;
end;
%%%%%%%%% UPDATE WEIGHTS AND BIASES %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
vishidinc = momentum*vishidinc + ...
epsilonw*( (posprods-negprods)/numcases - weightcost*vishid);
visbiasinc = momentum*visbiasinc + (epsilonvb/numcases)*(posvisact-negvisact);
hidbiasinc = momentum*hidbiasinc + (epsilonhb/numcases)*(poshidact-neghidact);
vishid = vishid + vishidinc;
visbiases = visbiases + visbiasinc;
hidbiases = hidbiases + hidbiasinc;
%%%%%%%%%%%%%%%% END OF UPDATES %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
end
fprintf(1, '\nepoch %4i error %f \n', epoch, errsum);
end
dofacedeepauto.m:
clear all
close all
maxepoch=200; %In the Science paper we use maxepoch=50, but it works just fine.
numhid=2000; numpen=1000; numpen2=500; numopen=30;
fprintf(1,'Pretraining a deep autoencoder. \n');
fprintf(1,'The Science paper used 50 epochs. This uses %3i \n', maxepoch);
load fdata
%makeFaceData;
[numcases numdims numbatches]=size(batchdata);
fprintf(1,'Pretraining Layer 1 with RBM: %d-%d \n',numdims,numhid);
restart=1;
rbmvislinear;
hidrecbiases=hidbiases;
save mnistvh vishid hidrecbiases visbiases;
maxepoch=50;
fprintf(1,'\nPretraining Layer 2 with RBM: %d-%d \n',numhid,numpen);
batchdata=batchposhidprobs;
numhid=numpen;
restart=1;
rbm;
hidpen=vishid; penrecbiases=hidbiases; hidgenbiases=visbiases;
save mnisthp hidpen penrecbiases hidgenbiases;
fprintf(1,'\nPretraining Layer 3 with RBM: %d-%d \n',numpen,numpen2);
batchdata=batchposhidprobs;
numhid=numpen2;
restart=1;
rbm;
hidpen2=vishid; penrecbiases2=hidbiases; hidgenbiases2=visbiases;
save mnisthp2 hidpen2 penrecbiases2 hidgenbiases2;
fprintf(1,'\nPretraining Layer 4 with RBM: %d-%d \n',numpen2,numopen);
batchdata=batchposhidprobs;
numhid=numopen;
restart=1;
rbmhidlinear;
hidtop=vishid; toprecbiases=hidbiases; topgenbiases=visbiases;
save mnistpo hidtop toprecbiases topgenbiases;
backpropface;
Thanks for your time
Silly me, I had forgotten to change the back-propagation fine-tuning script (backprop.m). One has to change the output layer (where the faces get reconstructed) to be for real-valued units. I.e.
dataout = w7probs*w8;

How to set output size in Matlab newff method

Summary:
I'm trying to do classification of some images depending on the angles between body parts.
I assume that human body consists of 10 parts(as rectangles) and find the center of each part and calculate the angle of each part by reference to torso.
And I have three action categories:Handwave-Walking-Running.
My goal is to find which test images fall into which action category.
Facts:
TrainSet:1057x10 feature set,1057 stands for number of image.
TestSet:821x10
I want my output to be 3x1 matrice each row showing the percentage of classification for action category.
row1:Handwave
row2:Walking
row3:Running
Code:
actionCat='H';
[train_data_hw train_label_hw] = tugrul_traindata(TrainData,actionCat);
[test_data_hw test_label_hw] = tugrul_testdata(TestData,actionCat);
actionCat='W';
[train_data_w train_label_w] = tugrul_traindata(TrainData,actionCat);
[test_data_w test_label_w] = tugrul_testdata(TestData,actionCat);
actionCat='R';
[train_data_r train_label_r] = tugrul_traindata(TrainData,actionCat);
[test_data_r test_label_r] = tugrul_testdata(TestData,actionCat);
Train=[train_data_hw;train_data_w;train_data_r];
Test=[test_data_hw;test_data_w;test_data_r];
Target=eye(3,1);
net=newff(minmax(Train),[10 3],{'logsig' 'logsig'},'trainscg');
net.trainParam.perf='sse';
net.trainParam.epochs=50;
net.trainParam.goal=1e-5;
net=train(net,Train);
trainSize=size(Train,1);
testSize=size(Test,1);
if(trainSize > testSize)
pend=-1*ones(trainSize-testSize,size(Test,2));
Test=[Test;pend];
end
x=sim(net,Test);
Question:
I'm using Matlab newff method.But my output is always an Nx10 matrice not 3x1.
My input set should be grouped as 3 classes but they are grouped to 10 classes.
Thanks
%% Load data : I generated some random data instead
Train = rand(1057,10);
Test = rand(821,10);
TrainLabels = randi([1 3], [1057 1]);
TestLabels = randi([1 3], [821 1]);
trainSize = size(Train,1);
testSize = size(Test,1);
%% prepare the input/output vectors (1-of-N output encoding)
input = Train'; %'matrix of size numFeatures-by-numImages
output = zeros(3,trainSize); % matrix of size numCategories-by-numImages
for i=1:trainSize
output(TrainLabels(i), i) = 1;
end
%% create net: one hidden layer with 10 nodes (output layer size is infered: 3)
net = newff(input, output, 10, {'logsig' 'logsig'}, 'trainscg');
net.trainParam.perf = 'sse';
net.trainParam.epochs = 50;
net.trainParam.goal = 1e-5;
view(net)
%% training
net = init(net); % initialize
[net,tr] = train(net, input, output); % train
%% performance (on Training data)
y = sim(net, input); % predict
%[err cm ind per] = confusion(output, y);
[maxVals predicted] = max(y); % predicted
cm = confusionmat(predicted, TrainLabels);
acc = sum(diag(cm))/sum(cm(:));
fprintf('Accuracy = %.2f%%\n', 100*acc);
fprintf('Confusion Matrix:\n');
disp(cm)
%% Testing (on Test data)
y = sim(net, Test');
Note how I converted from category label for each instance (1/2/3) to a 1-to-N encoding vector ([100]: 1, [010]: 2, [001]: 3)
Also note that the test set is currently not being used, since by default the input data is divided into train/test/validation. You could achieve your manual division by setting net.divideFcn to the divideind function, and setting the corresponding net.divideParam parameters.
I showed the testing on the same training data, but you could do the same for the Test data.