I have a 10 by 57300 matrix as an input, and a 1 by 57300 matrix as an output that only includes 0 and 1.I tried to train neural network with feed-forward back propagation and layer recurrent back propagation structures. I tried those structures with one hidden layer and 40 neurons in hidden layer.In the best case the performance stopped at point 0.133. I simulated the network with new inputs but it did not give me the result that I wanted. And the results were not even close to what I trained the network with. Do you have any suggestion to improve the performance of the network?
inputs = input;
targets = output;
% Create a Fitting Network
hiddenLayerSize = 50;
net = fitnet(hiddenLayerSize);
% Choose Input and Output Pre/Post-Processing Functions
% For a list of all processing functions type: help nnprocess
net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
net.outputs{2}.processFcns = {'removeconstantrows','mapminmax'};
% Setup Division of Data for Training, Validation, Testing
% For a list of all data division functions type: help nndivide
net.divideFcn = 'divideind';
net.divideParam.trainInd=1:28650; % The first 94 inputs are for training.
net.divideParam.valInd=28651:42977; % The first 94 inputs are for validation.
net.divideParam.testInd=42978:57300; % The last 5 inputs are for testing the network.
% For help on training function 'trainlm' type: help trainlm
% For a list of all training functions type: help nntrain
net.trainFcn = 'trainlm'; % Levenberg-Marquardt
% Choose a Performance Function
% For a list of all performance functions type: help nnperformance
net.performFcn = 'mse'; % Mean squared error
% Choose Plot Functions
% For a list of all plot functions type: help nnplot
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
'plotregression', 'plotfit'};
% Train the Network
[net,tr] = train(net,inputs,targets);
% Test the Network
outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs)
% Recalculate Training, Validation and Test Performance
trainTargets = targets .* tr.trainMask{1};
valTargets = targets .* tr.valMask{1};
testTargets = targets .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,outputs)
valPerformance = perform(net,valTargets,outputs)
testPerformance = perform(net,testTargets,outputs)
% View the Network
view(net)
% Plots
% Uncomment these lines to enable various plots.
%figure, plotperform(tr)
%figure, plottrainstate(tr)
%figure, plotfit(net,inputs,targets)
%figure, plotregression(targets,outputs)
%figure, ploterrhist(errors)
This is the code that I used for training neural network. My 57300 input divides to 300X191 groups of data. I mean each set of input is a 10 by 191 group. So that's why I've used "divideind". I have normalized the input and output matrix in [-1 1] range because I use tansig transfer function. But I still do not get the result that I want form the network.
it is usually complicate to tell how to improve a network with such few information. In some cases a performance of 0.133 could be a good result, in others not so much. Nonetheless, general ideas to improve networks could be normalizing the inputs to [0,1] range, performing feature selection, maybe adding a rbm layer and performing non-supervised training before the supervised backpropagation learning scheme (see Deep belief networks), increasing the data for learning, or using cross-validation for choosing the free parameters and to early stop.
Edit v1:
I see that you have 94 inputs for testing, 94 for validation and 5 for testing. This ratio seems a bit error prone.
First of all, 94 input sets for a feature vector of 10 is very few. For a feature vector of 100 is futile :). So basically the problem is that you don't have enough data to train 40 neurons. If you can't generate more data, i'd recommend a new splitting:
150 Training
10 Validation
30 Testing
This information is rather more general approach to improve NN performance:
The training methodology
How many epochs? It is possible that the network is over-trained to your input set.
Parameters of the BP algorithm (momentum, adaptation factors, etc.)
The feature extraction algorithm
Often, this is the main problem. The algorithm is not able to really extract the specific features of an input.
I suggest plotting all the inputs visually and see if you are seeing any pattern and are able to determine the separation visually. After all, the neural network is a bit more complex statistic system.
Good luck!
I am not sure if you are training your network for classification purpose or regression. But if you directly want to improve the performance I will suggest you to do following things :
1- Increase the number of nodes in the hidden layer further. Try out something bigger to see if the neural network is doing better(it may be because of overtraining, but ignore it for now).
2- If their is very high non-linearity in the data space then add one more hidden layer. But adding many of them will not work in the case of simple neural network.
Related
I am training a neural network to learn a function. Everything is going great so far.
I have input matrix of 4x10000 and output matrix of 3x10000. I have much more data points than 10000. But not all of them can be fit at once so I have decided to feed pack of 10000-10000 data points and train same neural network on it.
There are three layers and 7 units in hidden layer.
So what I do is, I train the network with 10000 data points randomly and then again train on another random 10000 data points and so on.
So for this I store CheckPoints (in-built functionality of neural net toolkit). But what happens is that the network, which is being trained, is stored as struct in CheckPoints rather than network type itself. So when I load the checkpoint next time I run the program, it shows error something as below.
Undefined function 'train' for input arguments of type 'struct'
I am using fitnet network.
% Create a Fitting Network
hiddenLayerSize = 7;
net = fitnet(hiddenLayerSize,'trainlm');
% Setup Division of Data for Training, Validation, Testing
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;
load('Highlights_Checkpoint.mat');
existanceOfCheckpoint = exist('checkpoint', 'var');
if existanceOfCheckpoint==0
else
net = (checkpoint.net);
end
% Train the Network
[net,tr] = train(net,x,t,'useParallel', 'yes','showResources','yes', 'CheckpointFile','Highlights_Checkpoint.mat');
Well solution to this problem was quite easy.
All I had to do was the following:
net = network(checkpoint.net);
And all was set. :D
I'm trying to use a neural network for a classification problem, but the result of the training produce very bad performance. The classification problem:
I have more than 300,000 training samples
Each input is a vector of 32 values (real values)
Each output is a vector of 32 values (0 or 1)
This is how I train the network:
DNN_SIZE = [1000, 1000];
% Initialize DNN
net = feedforwardnet(DNN_SIZE, 'traingda');
net.performParam.regularization = 0.2;
%Set activation functions
for i=1:length(DNN_SIZE)
net.layers{i}.transferFcn = 'poslin';
end
net.layers{end}.transferFcn = 'logsig';
net = train(net, train_inputs, train_outputs);
Note: I have tried different values for DNN_SIZE including larger and smaller values, for hidden layers and less, but it didn't make a difference.
Note 2: I have tried training the same network using a data set from Matlab's examples (simpleclass_dataset) and I still got bad performance.
The performance of the trained network is very bad- Its output is basically 0.5 in every output for every input vector (when the target outputs during training are always 0 or 1). What am I doing wrong, and how can I fix it?
Thanks.
In case, there are 2 inputs (X1 and X2) and 1 target output (t) to be estimated by neural network (each nodes has 6 samples):
X1 = [2.765405915 2.403146899 1.843932529 1.321474515 0.916837222 1.251301467];
X2 = [84870 363024 983062 1352580 804723 845200];
t = [-0.12685144347197 -0.19172223428950 -0.29330584684934 -0.35078062276141 0.03826908777226 0.06633047875487];
I was trying to find the best fit of t prediction by using multiple linear regression (Ordinary Least Squares or OLS) manually and the outcomes were pretty good.
I intend to find the a, b, c (regression coefficients) from this equation:
t = a + b*X1 + c*X2
Since the equation is the basic form of multiple linear regression equation with two regressors, of course I could find the value of a, b, c by doing the OLS.
The problem is: I've tried to find the regression coefficients by using neural network (with MATLAB nftool and train it by Levenberg-Marquardt Backpropagation or lmtrain) but have no idea how to find them, though the outcomes were showing less error than the OLS.
Then, several questions that comes:
Is it possible to find the regression coefficient by using neural network?
If it's possible, What kind of ANN algorithm that could solve this kind of problem and how to build it manually?
If you have any ideas how to solve it, please help. I really need your help!
This is the script that generated by MATLAB nftool that I used to fit the output estimation:
% Solve an Input-Output Fitting problem with a Neural Network
% Script generated by NFTOOL
% Created Fri Jun 05 06:26:36 ICT 2015
%
% This script assumes these variables are defined:
%
% x - input data.
% t - target data.
x = [2.765405915 2.403146899 1.843932529 1.321474515 0.916837222 1.251301467; 84870 363024 983062 1352580 804723 845200];
t = [-0.12685144347197 -0.19172223428950 -0.29330584684934 -0.35078062276141 0.03826908777226 0.06633047875487];
inputs = x;
targets = t;
% Create a Fitting Network
hiddenLayerSize = 10;
net = fitnet(hiddenLayerSize);
% Setup Division of Data for Training, Validation, Testing
net.divideParam.trainRatio = 90/100;
net.divideParam.valRatio = 5/100;
net.divideParam.testRatio = 5/100;
% Train the Network
[net,tr] = train(net,inputs,targets);
% Test the Network
outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs)
% View the Network
view(net)
% Plots
% Uncomment these lines to enable various plots.
%figure, plotperform(tr)
%figure, plottrainstate(tr)
%figure, plotfit(net,inputs,targets)
%figure, plotregression(targets,outputs)
%figure, ploterrhist(errors)
A neural network will generally not find or encode a formula like t = a + b*X1 + c*X2, unless you built a really simple one with no hidden layers and linear output. If you did then you could read the values [a,b,c] from the weights attached to bias, input 1 and input 2. However, such a network offers no advantage over linear regression (essentially it is linear regression using NN tools to build it and comparatively slow gradient descent to find the lowest least-square error when it can be done in a single pass in OLS).
What you have built instead is a more complex non-linear function. Most likely the error is low because you have over-fit your data, which is very easy to do with a neural net. With your input data as shown, it should be possible to get an training error of 0, but that is not as good as it seems - it just means the neural network has found a complex surface that connects all your examples, which is probably of limited use as a predictive model.
m1= xlsread('NSL_KDD_TRAIN.xlsx','A2:AO67344'); % m1=input
m= xlsread('NSL_KDD_TRAIN.xlsx','AP2:AP67344'); % m=output
net=newff(m1',m',8);
net=train(net,m1',m');
y=sim(net,m1'); %training data output
y3=round(y); % making nonfraction number
n1= xlsread('NSL_KDD_TRAIN.xlsx','A67345:AO79000'); % n1=input
n= xlsread('NSL_KDD_TRAIN.xlsx','AP67345:AP79000'); % n=output
net=newff(n1',n',8);
net=train(net,n1',n');
y1=sim(net,n1'); %training data output
y31=round(y1);
i want to use the outputs of feed forward neural networks as input for training another same kind of neural network. how can i do that using built-in function?,from the above code i need to train a new newff where my input arguments will be y3 and y31. in m1 and n1 a have 41 columns and i need all 41 columns presence in the new network. how can i do that using built-in function?
Why not to use two layers and train just one network if you assume architecture is the same?
net = newff(input,target,[Layer_1 Layer_2],{'tansig' 'tansig' 'tansig'});
You can specify the size of Layer_1 and Layer_2 as much as you want as well as you can change activation function between each of the layers.
Tutorial to create custom network from Mathworks
Hope this will help
I have created a GRNN using command:
net = newgrnn(inputs, output);
How do I train existing GRNN on new inputs in MATLAB? In other words, how to train net on other inputs?
Ok so the problem is that newgrnn is defined based on the training data. Therefore you cannot retrain the network unless you do it from the scratch. What I would suggest you is to use a different radial basis network that can be retrain such as the newrb. I added a simulation of such networks for a regression problem. Take into consideration that this is a complex regression as you have only partial coverage in the training data, and you should not expect much from either network. You can also use this simulation to try different architectures such as MLP.
CODE:
%% simulate data
X=linspace(0,20*pi,1250);
Y=sin(X)-X/20+rand(1,1250)*0.2;
Xtrain=X([250:500,550,600,650,700,750:1000]);
Ytrain=Y([250:500,550,600,650,700,750:1000]);
%% create a radial neural network through newrbe
spread=0.6;
net = newrbe(Xtrain,Ytrain,spread);
view (net)
Y_net = net(X);
%% create a radial neural network through newgrnn
spread=0.6;
net1 = newgrnn(Xtrain,Ytrain,spread);
view (net1)
Y_net1 = net1(X);
%% create a radial neural network through newrb and retrain with bayesian
spread=0.6;
net2 = newrb(Xtrain,Ytrain,0,spread,40);
view (net2)
Y_net2 = net2(X);
net3=net2;
net3.trainFcn='trainbr';
net3.trainParam.epochs = 1000;
net3 = train(net3,Xtrain,Ytrain);
view (net3)
Y_net3 = net3(X);
%% plot networks
h=figure;
plot(X,Y,'black'),hold on
plot(Xtrain,Ytrain,'blacko'),hold on
plot(X,Y_net1,'Color',[1,0,1])
plot(X,Y_net,'r')
plot(X,Y_net2,'g')
plot(X,Y_net3,'b')
axis([X(1),X(end),-5,2])
legend('original data','train data','newrbe','newgrnn','newrb','newrb-retrain')
OUTPUT:
GRNN can not be trained in the sense of backpropagation. It is single pass learning. It will assign new neuron for every input pattern so in order to train it just include the data point in the command newgrnn(1:n,1:m);