Stabilizing Neural Network - matlab

Question

I am trying to build a neural network and have the following code:
for i = 1:num_samples-num_reserved
% Getting the sample and transposing it so it can be multiplied
sample = new_data_a(:,i)';
% Normalizing the input vector
sample = sample/norm(sample);
% Calculating output
outputs = sample*connections;
% Neuron that fired the hardest's index (I) and its output (output)
[output, I] = max(outputs);
% Conections leading to this neuron
neuron_connections = connections(:,I);
% Looping through input components
for j = 1:num_features
% Value of this input component
component_input = sample(j);
% Updating connection weights
delta = 0.7*(component_input - neuron_connections(j));
neuron_connections(j) = neuron_connections(j) + delta;
end
% Copying new connection weights into original matrix
connections(:,I) = neuron_connections/norm(neuron_connections);
if(rem(i,100) == 0)
if(I == 1)
delta_track = [delta_track connections(2,I)];
end
end
% Storing current connections
end
I think I'm doing everything right. This loop is repeated about 600 times so as to progressively update connections. The function to update weights I am using is the standard one I found in textbooks.
However, when I look at the values stored in delta_track These keep oscillating forming a regular pattern.
Any advice?

Answer 1

You can reduce the feedback factor. Then the network may require more time to learn but is less likely to oscillate.
Another common technique is to add a decay, i.e. reducing the factor each iteration.
In general neural networks have the same stability rules as control systems have (because as long as a NN is learning it is a control system) Therefore similar approaches work as for e.g. PID controllers.