Back Propagation Neural Network Hidden Layer all output is 1 - matlab

everyone I have created a neural network with 1600 input, one hidden layer with different number of neurons nodes and 24 output neurons.
My code shown that I can decrease the error each epoch, but the output of hidden layer always is 1. Due to this reason, the weight adjusted always produce same result for my testing data.
I try different number of neuron nodes and learning rate in the ANN and also randomly initialize my initial weight. I use sigmoid function as my activate function since my output is either 1 or 0 in different output.
May I know that what is the main reason that causes the output of hidden layer always is 1 and how should i solve it?
My purpose for this neural network is to recognize 24 hand shape for alphabet, I try intensities data in my first phase of project.
I have try 30 hidden neural nodes also 100 neural nodes even 1000 neural nodes but the output of hidden layer still is 1. Due to this reason, all of the outcome in testing data is always similar.
I added the code for my network
Thanks
g = inline('logsig(x)');
[row, col] = size(input);
numofInputNeurons = col;
weight_input_hidden = rand(numofInputNeurons, numofFirstHiddenNeurons);
weight_hidden_output = rand(numofFirstHiddenNeurons, numofOutputNeurons);
epochs = 0;
errorMatrix = [];
while(true)
if(totalEpochs > 0 && epochs >= totalEpochs)
break;
end
totalError = 0;
epochs = epochs + 1;
for i = 1:row
targetRow = zeros(1, numofOutputNeurons);
targetRow(1, target(i)) = 1;
hidden_output = g(input(1, 1:end)*weight_input_hidden);
final_output = g(hidden_output*weight_hidden_output);
error = abs(targetRow - final_output);
error = sum(error);
totalError = totalError + error;
if(error ~= 0)
delta_final_output = learningRate * (targetRow - final_output) .* final_output .* (1 - final_output);
delta_hidden_output = learningRate * (hidden_output) .* (1-hidden_output) .* (delta_final_output * weight_hidden_output');
for m = 1:numofFirstHiddenNeurons
for n = 1:numofOutputNeurons
current_changes = delta_final_output(1, n) * hidden_output(1, m);
weight_hidden_output(m, n) = weight_hidden_output(m, n) + current_changes;
end
end
for m = 1:numofInputNeurons
for n = 1:numofFirstHiddenNeurons
current_changes = delta_hidden_output(1, n) * input(1, m);
weight_input_hidden(m, n) = weight_input_hidden(m, n) + current_changes;
end
end
end
end
totalError = totalError / (row);
errorMatrix(end + 1) = totalError;
if(errorThreshold > 0 && totalEpochs == 0 && totalError < errorThreshold)
break;
end
end

I see a few obvious errors that need fixing in your code:
1) You have no negative weights when initialising. This is likely to get the network stuck. The weight initialisation should be something like:
weight_input_hidden = 0.2 * rand(numofInputNeurons, numofFirstHiddenNeurons) - 0.1;
2) You have not implemented bias. That will severely limit the ability of the network to learn. You should go back to your notes and figure that out, it is usually implemented as an extra column of 1's inserted into input and activation vectors/matrix before determining the activations of each layer, and there should be a matching additional column of weights.
3) Your delta for output layer is wrong. This line
delta_final_output = learningRate * (targetRow - final_output) .* final_output .* (1 - final_output);
. . . is not the delta for the output layer activations. It has some extra unwanted factors.
The correct delta for logloss objective function and sigmoid activation in output layer would be:
delta_final_output = (final_output - targetRow);
There are other possibilities, depending on your objective function, which is not shown. You original code is close to correct for mean squared error, which would probably still work if you changed the sign and removed the factor of learningRate
4) Your delta for hidden layer is wrong. This line:
delta_hidden_output = learningRate * (hidden_output) .* (1-hidden_output) .* (delta_final_output * weight_hidden_output');
. . . is not the delta for the hidden layer activations. You have multiplied by the learningRate for some reason (combined with the other delta that means you have a factor of learningRate squared).
The correct delta would be:
delta_hidden_output = (hidden_output) .* (1-hidden_output) .* (delta_final_output * weight_hidden_output');
5) Your weight update step needs adjusting to match fixes to (3) and (4). These lines:
current_changes = delta_final_output(1, n) * hidden_output(1, m);
would need to be adjusted to get correct sign and learning rate multiplier
current_changes = -learningRate * delta_final_output(1, n) * hidden_output(1, m);
That's 5 bugs from looking through the code, I may have missed some. But I think that's more than enough for now.

Related

MATLAB 2-Layer Neural Network from Scratch

Currently, I'm working on a simple two Layer NN (25 input - sigmoid, 199 outputs - softmax) from scratch for debug reasons - Precisely, I want to track some values.
My input are batches or generally speaking matrices of dimension (rows x 25) in order to fit the input layer structure. Regarding my weight matrices: the first but last rows are the weights w_ij. The last row includes the biases.
The forward method seems to work correctly but I think I have a wrong backpropagation.
My backpropagation code snippet:
%Error gradient for the softmax output
error = single(output) - single(targets);
%Error for the input layer - W21 includes w_ij
error_out_to_input = error*(W21.');
gradient_outputLayer = single(zeros(26,199));
gradient_outputLayer = single(first_layerout_zerofilled.')*single(error);
biasGrad = single(sum(error,1));
gradient_outputLayer(26:26,:) = single(biasGrad);
%InputLayer
%derivative of sigmoid o(1-o)
%1
grad = single(1);
%1-o
grad = single(grad) - single(first_layerout_zerofilled);
%o(1-o)
grad = single(first_layerout_zerofilled) .* single((grad));
%final error
grad = single(grad) .* single(error_out_to_input);
gradient_inputLayer = single(zeros(26,25));
gradient_inputLayer = single(inputs.')*single(grad);
biasGrad = single(sum(grad,1));
gradient_inputLayer(26:26,:) = single(biasGrad);
%Update
W1 = W1-gradient_inputLayer * learning_rate;
W2 = W2-gradient_outputLayer * learning_rate;
This is not a question of efficiency. I just want to be sure that my backprogation calculates the correct gradients. I hope someone can review.

Calculate convolutional layer in CNN implementation

I am trying to train a Convolutional Neural Network using Sparse autoenconders in order to compute the filters for the convolution layer. I am using UFLDL code in order to construct patches and to train the CNN network. My code is the following:
===========================================================================
imageDim = 30; % image dimension
imageChannels = 3; % number of channels (rgb, so 3)
patchDim = 10; % patch dimension
numPatches = 100000; % number of patches
visibleSize = patchDim * patchDim * imageChannels; % number of input units
outputSize = visibleSize; % number of output units
hiddenSize = 400; % number of hidden units
epsilon = 0.1; % epsilon for ZCA whitening
poolDim = 10; % dimension of pooling region
optTheta = zeros(2*hiddenSize*visibleSize+hiddenSize+visibleSize, 1);
ZCAWhite = zeros(visibleSize, visibleSize);
meanPatch = zeros(visibleSize, 1);
load patches_16_1
===========================================================================
% Display and check to see that the features look good
W = reshape(optTheta(1:visibleSize * hiddenSize), hiddenSize, visibleSize);
b = optTheta(2*hiddenSize*visibleSize+1:2*hiddenSize*visibleSize+hiddenSize);
displayColorNetwork( (W*ZCAWhite));
stepSize = 100;
assert(mod(hiddenSize, stepSize) == 0, stepSize should divide hiddenSize);
load train.mat % loads numTrainImages, trainImages, trainLabels
load train.mat % loads numTestImages, testImages, testLabels
% size 30x30x3x8862
numTestImages = 8862;
numTrainImages = 8862;
pooledFeaturesTrain = zeros(hiddenSize, numTrainImages, floor((imageDim - patchDim + 1) / poolDim), floor((imageDim - patchDim + 1) / poolDim) );
pooledFeaturesTest = zeros(hiddenSize, numTestImages, ...
floor((imageDim - patchDim + 1) / poolDim), ...
floor((imageDim - patchDim + 1) / poolDim) );
tic();
testImages = trainImages;
for convPart = 1:(hiddenSize / stepSize)
featureStart = (convPart - 1) * stepSize + 1;
featureEnd = convPart * stepSize;
fprintf('Step %d: features %d to %d\n', convPart, featureStart, featureEnd);
Wt = W(featureStart:featureEnd, :);
bt = b(featureStart:featureEnd);
fprintf('Convolving and pooling train images\n');
convolvedFeaturesThis = cnnConvolve(patchDim, stepSize, ...
trainImages, Wt, bt, ZCAWhite, meanPatch);
pooledFeaturesThis = cnnPool(poolDim, convolvedFeaturesThis);
pooledFeaturesTrain(featureStart:featureEnd, :, :, :) = pooledFeaturesThis;
toc();
clear convolvedFeaturesThis pooledFeaturesThis;
fprintf('Convolving and pooling test images\n');
convolvedFeaturesThis = cnnConvolve(patchDim, stepSize, ...
testImages, Wt, bt, ZCAWhite, meanPatch);
pooledFeaturesThis = cnnPool(poolDim, convolvedFeaturesThis);
pooledFeaturesTest(featureStart:featureEnd, :, :, :) = pooledFeaturesThis;
toc();
clear convolvedFeaturesThis pooledFeaturesThis;
end
I have problems calculating the convolution and pooling layers. I am getting pooledFeaturesTrain(featureStart:featureEnd, :, :, :) = pooledFeaturesThis; subscripted assignment dimension mismatch. The pathces have normally calculated and they are:
I am trying to understand what exactly the convPart variable is doing and what pooledFeaturesThis. Secondly I notice that my problem is a mismatch in this line pooledFeaturesTrain(featureStart:featureEnd, :, :, :) = pooledFeaturesThis;
where I got the message that the variables is mismatching. THe size of pooledFeaturesThis is 100x3x2x2 where the size of pooledFeaturesTrain is 400x8862x2x2. What exactly pooledFeaturesTrain represents? Is the 2x2 result for every filter? CnnConvolve could be found here :
EDIT: I have changed a little bit my code and it works. However I a little bit concerned about the comprehension of the code.
Ok so in this line you are setting the pooling region.
poolDim = 10; % dimension of pooling region
This part means that for each kernel in each layer you are taking the image and pooling and area of 10x10 pixels. From your code it looks like you are applying a mean function, which means that it a patch and computes the mean and outputs this in the next layer... aka, takes the image from say 100x100 to 10x10. In your network you are repeating convolution+pooling until you get down to a 2x2 image, based on this output (btw, this is not generally good practice in my experience).
400x8862x2x2
Anyways back to your code. Notice that at the beginning of your training you do the following initialization:
pooledFeaturesTrain = zeros(hiddenSize, numTrainImages, floor((imageDim - patchDim + 1) / poolDim), floor((imageDim - patchDim + 1) / poolDim) );
So your error is quite simple and correct - the size of the matrix which holds the output of the convolution+pooling is not the size of the matrix you initialized.
The question is now how to fix it. I supposed a lazy man's way to fix it is to take out the initialization. It will drastically slow down your code, and is not guaranteed to work if you have more than 1 layer.
I suggest you instead have pooledFeaturesTrain be a struct of 3 dimensional array. So instead of this
pooledFeaturesTrain(featureStart:featureEnd, :, :, :) = pooledFeaturesThis;
you'd do something more along the lines of this:
pooledFeaturesTrain{n}(:, :, :) = pooledFeaturesThis;
where n is the current layer.
CNN nets aren't as easy as they're cracked up to be- and even when they don't crash getting them to train well is a feat. I highly suggest reading up on the theory of CNNs - it will make coding and debugging much easier.
Good luck with it ! :)

Neural Network convergence speed (Levenberg-Marquardt) (MATLAB)

I was trying to approximate a function (single input and single output) with an ANN. Using MATLAB toolbox I could see that with 5 or more neurons in the hidden layer, I can achieve a very nice result. So I am trying to do it manually.
Calculations:
As the network has only one input and one output, the partial derivative of the error (e=d-o, where 'd' is the desired output and 'o' is the actual output) in respect to a weigth which connects a hidden neuron j to the output neuron, will be -hj (where hj is the output of a hidden neuron j);
The partial derivative of the error in respect to output bias will be -1;
The partial derivative of the error in respect to a weight which connects the input to a hidden neuron j will be -woj*f'*i, where woj is the hidden neuron j output weigth, f' is the tanh() derivative and 'i' is the input value;
Finally, the partial derivative of the error in respect to hidden layer bias will be the same as above (in respect to input weight) except that here we dont have the input:
-woj*f'
The problem is:
the MATLAB algorithm always converge faster and better. I can achieve the same curve as MATLAB does, but my algorithm requires much more epochs.
I've tried to remove pre and postprocessing functions from MATLAB algorithm. It still converges faster.
I've also tried to create and configure the network, and extract weight/bias values before training so I could copy them to my algorithm to see if it converges faster but nothing changed (is the weight/bias initialization inside create/configure or train function?).
Does the MATLAB algorithm have some kind of optimizations inside the code?
Or may be this difference only in the organization of the training set and weight/bias initialization?
In case one wants to look my code, here is the main loop which makes the training:
Err2 = N;
epochs = 0;
%compare MSE of error2
while ((Err2/N > 0.0003) && (u < 10000000) && (epochs < 100))
epochs = epochs+1;
Err = 0;
%input->hidden weight vector
wh = w(1:hidden_layer_len);
%hidden->output weigth vector
wo = w((hidden_layer_len+1):(2*hidden_layer_len));
%hidden bias
bi = w((2*hidden_layer_len+1):(3*hidden_layer_len));
%output bias
bo = w(length(w));
%start forward propagation
for i=1:N
%take next input value
x = t(i);
%propagate to hidden layer
neth = x*wh + bi;
%propagate through neurons
ij = tanh(neth)';
%propagate to output layer
neto = ij*wo + bo;
%propagate to output (purelin)
output(i) = neto;
%calculate difference from target (error)
error(i) = yp(i) - output(i);
%Backpropagation:
%tanh derivative
fhd = 1 - tanh(neth').*tanh(neth');
%jacobian matrix
J(i,:) = [-x*wo'.*fhd -ij -wo'.*fhd -1];
%SSE (sum square error)
Err = Err + 0.5*error(i)*error(i);
end
%calculate next error with updated weights and compare with old error
%start error2 from error1 + 1 to enter while loop
Err2 = Err+1;
%while error2 is > than old error and Mu (u) is not too large
while ((Err2 > Err) && (u < 10000000))
%Weight update
w2 = w - (((J'*J + u*eye(3*hidden_layer_len+1))^-1)*J')*error';
%New Error calculation
%New weights to propagate
wh = w2(1:hidden_layer_len);
wo = w2((hidden_layer_len+1):(2*hidden_layer_len));
%new bias to propagate
bi = w2((2*hidden_layer_len+1):(3*hidden_layer_len));
bo = w2(length(w));
%calculate error2
Err2 = 0;
for i=1:N
%forward propagation again
x = t(i);
neth = x*wh + bi;
ij = tanh(neth)';
neto = ij*wo + bo;
output(i) = neto;
error2(i) = yp(i) - output(i);
%Error2 (SSE)
Err2 = Err2 + 0.5*error2(i)*error2(i);
end
%compare MSE from error2 with a minimum
%if greater still runing
if (Err2/N > 0.0003)
%compare with old error
if (Err2 <= Err)
%if less, update weights and decrease Mu (u)
w = w2;
u = u/10;
else
%if greater, increment Mu (u)
u = u*10;
end
end
end
end
It's not easy to know the exact implementation of the Levenberg Marquardt algorithm in Matlab. You may try to run the algorithm one iteration at a time, and see if it is identical to your algorithm. You can also try other implementations, such as, http://www.mathworks.com/matlabcentral/fileexchange/16063-lmfsolve-m--levenberg-marquardt-fletcher-algorithm-for-nonlinear-least-squares-problems, to see if the performance can be improved. For simple learning problems, convergence speed may be a matter of learning rate. You might simply increase the learning rate to get faster convergence.

Shifting the FFT of a signal in Matlab

I'm trying to implement a spectral correlation function to plot with the surf function.
I think I understand the idea of the SCF as described in a paper I read, but I'm having trouble implementing my function in Matlab. I've been following these instructions:
I'm mostly having trouble shifting my pieces of the data properly. Is there an easy way to achieve step 3?
Here's what I tried in my code:
function [output] = spectral(x, N)
% This function does cyclostationary spectral analysis
% on a data set and returns some features
t = length(x);
samplesPerFrame = floor(t / N);
count = 1;
for alpha = -1:0.01:1
% Split up the samples into frames
% Have to leave some samples out if unevenly split
for i = 1:N+1
frange = ((i - 1) * samplesPerFrame + 1):(i * samplesPerFrame);
if i == N+1
break;
end
xFrame(i, :) = x(frange);
ts = [1:length(xFrame(i,:))];
shiftLeft = fft(xFrame(i, :) .* exp(-1 * 2 * pi * 1i * (alpha / 2) .* ts));
shiftRight = fft(xFrame(i, :).* exp(2 * pi * 1i * (alpha / 2) .* ts));
S(i,:) = (1 / samplesPerFrame) .* shiftLeft .* conj(shiftRight);
end
Savg(count, :) = mean(S, 1);
Ssmooth(count, :) = smooth(Savg(count,:), 'moving');
count = count + 1;
end
output = Ssmooth;
end
It looks good actually.
You may also try circshift(fft(xFrame(i, :)),[1,a]) to achieve shiftRight, and circshift(fft(xFrame(i, :)),[1,-a]) to get shiftLeft. Please note here a is integer, indicates the elements in xFrame(i, :) that you wish to move, and corresponds to Fs*a in frequency domain where Fs is your sampling rate.
The method of spectral correlation estimation you are attempting is something I refer to as the Time-Smoothing Method of spectral correlation estimation, or the TSM. The code you posted cannot provide the correct answer except in some trivial cases such as alpha = 0. The reason is that you need to adjust the cyclic periodogram for each frame by a complex phase factor to compensate for the fact that each data block is a delayed version of the one preceding it.
If you replace the line
S(i,:) = (1 / samplesPerFrame) .* shiftLeft .* conj(shiftRight);
with the two lines
S(i,:) = (1 / samplesPerFrame) .* shiftLeft .* conj(shiftRight);
S(i, :) = S(i, :) * exp(-1i * 2 * pi * alpha * i * samplesPerFrame);
you'll be able to estimate the SCF. I confirmed this by applying your original code and the modified code to a BPSK signal with bit rate (normalized) of 1/10. In this case, one of your alpha values in the loop over alpha will exactly coincide with the true cycle frequency of 1/10. Only the modified code gives the correct SCF for the bit-rate cycle frequency.
Please see my blog cyclostationary.wordpress.com for more detail and examples. In particular, I have a post on the TSM at
http://cyclostationary.blog/2015/12/18/csp-estimators-the-time-smoothing-method. (Corrected this link 5/2/17.)

Removing duplicate entries in a vector, when entries are complex and rounding errors are causing problems

I want to remove duplicate entries from a vector on Matlab. The problem I'm having is that rounding errors are stopping the inbuilt Matlab function 'unique' from working properly. Ideally I'd like a way to set some sort of tolerance on the 'unique' function, or a small procedure that will remove the duplicates otherwise. If both the real and imaginary parts of two entries differ by less than 0.0001, then I'm happy to consider them equal. How can I do this?
Any help will be greatly appreciated. Thanks
A simple approximation would be to round the numbers and the use the indices returned by unique:
X = ... (input vector)
[b, i] = unique(round(X / (tolerance * (1 + i))));
output = X(i);
(you can probably replace b with ~ depending on your Matlab version).
it won't quite have the behaviour you want, since it is possible that two numbers are very close but will be rounded differently. I think you could mitigate this by doing:
X = ... (input vector)
[b, ind] = unique(round(X / (tolerance * (1 + i))));
X = X(ind);
[b, ind] = unique(round(X / (tolerance * (1 + i)) + 0.5 * (1 + i)));
X = X(ind);
This will round them twice, so any numbers that are exactly on a rounding boundary will be caught by the second unique.
There is still some messiness in this - some numbers will be affected as though the tolerance was doubled. But it might be sufficient for your needs.
The alternative is probably a for loop:
X = sort(X);
last = X(1);
indices = ones(numel(X), 1);
for j=2:numel(X)
if X(j) > last + tolerance * (1 + i)
last = X(j) + tolerance * (1 + i) / 2;
else
indices(j) = 0;
end
end
X = X(logical(indices));
I think this has the best behaviour you can expect (because you want to represent the vector by as few unique values as possible - when there are lots of numbers that differ by less than the tolerance level, there may be multiple ways of splitting them. This algorithm does so greedily, starting with the smallest).
I'm almost certain the below ill always assume any values closer than 1e-8 are equal. Simply replace 1e-8 with whatever value you want.
% unique function that assumes 1e-8 is equal
function [out, I] = unique(input, first_last)
threshold = 1e-8;
if nargin < 2
first_last = 'last';
end
[out, I] = sort(input);
db = diff(out);
k = find(abs(db) < threshold);
if strcmpi(first_last, 'last')
k2 = min(I(k), I(k+1));
elseif strcmpi(first_last, 'first')
k2 = max(I(k), I(k+1));
else
error('unknown flag option for unique, must be first or last');
end
k3 = true(1, length(input));
k3(k2) = false;
out = out(k3(I));
I = I(k3(I));
end
The following might serve your purposes. Given X, an array of complex doubles, it sorts them, then checks whether the absolute value differences between elements is within the complex tolerance, real_tol and imag_tol. It removes elements that satisfy this tolerance.
function X_unique = unique_complex_with_tolerance(X,real_tol,imag_tol)
X_sorted = sort(X); %Sorts by magnitude first, then imaginary part.
dX_sorted = diff(X_sorted);
dX_sorted_real = real(dX_sorted);
dX_sorted_imag = imag(dX_sorted);
remove_idx = (abs(dX_sorted_real)<real_tol) & (abs(dX_sorted_imag)<imag_tol);
X_unique = X_sorted;
X_unique(remove_idx) = [];
return
Note that this code will remove all elements which satisfy this difference tolerance. For example, if X = [1+i,2+2i,3+3i,4+4i], real_tol = 1.1, imag_tol = 1.1, then this function will return only one element, X_unique = [4+4i], even though you might consider, for example, X_unique = [1+i,4+4i] to also be a valid answer.