Getting NaN values in neural network weight matrices - matlab

Question

**I am trying to develop a feedforward NN in MATLAB. I have a dataset of 12 inputs and 1 output with 46998 samples. I have some NaN values in last rows of Matrix, because some inputs are accelerations & velocities which are 1 & 2 steps less respectively than displacements.
With this current data set I am getting w1_grad & w2_grad as NaN matrices. I tried to remove them using `Heave_dataset(isnan(Heave_dataset))=[];, but my dataset is getting converted into a column matrix of (1*610964).
can anyone help me with this ?
%
%% Clear Variables, Close Current Figures, and Create Results Directory
clc;
clear all;
close all;
mkdir('Results//'); %Directory for Storing Results
%% Configurations/Parameters
load 'Heave_dataset'
% Heave_dataset(isnan(Heave_dataset))=[];
nbrOfNeuronsInEachHiddenLayer = 24;
nbrOfOutUnits = 1;
unipolarBipolarSelector = -1; %0 for Unipolar, -1 for Bipolar
learningRate = 0.08;
nbrOfEpochs_max = 50000;
%% Read Data
Input = Heave_dataset(:, 1:length(Heave_dataset(1,:))-1);
TargetClasses = Heave_dataset(:, length(Heave_dataset(1,:)));
%% Calculate Number of Input and Output NodesActivations
nbrOfInputNodes = length(Input(1,:)); %=Dimention of Any Input Samples
nbrOfLayers = 2 + length(nbrOfNeuronsInEachHiddenLayer);
nbrOfNodesPerLayer = [nbrOfInputNodes nbrOfNeuronsInEachHiddenLayer nbrOfOutUnits];
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Forward Pass %%%%%%%%%%%
%% Adding the Bias to Input layer
Input = [ones(length(Input(:,1)),1) Input];
%% Weights leading from input layer to hidden layer is w1
w1 = rand(nbrOfNeuronsInEachHiddenLayer,(nbrOfInputNodes+1));
%% Input & output of hidde layer
hiddenlayer_input = Input*w1';
hiddenlayer_output = -1 + 2./(1 + exp(-(hiddenlayer_input)));
%% Adding the Bias to hidden layer
hiddenlayer_output = [ones(length(hiddenlayer_output(:,1)),1) hiddenlayer_output];
%% Weights leading from input layer to hidden layer is w1
w2 = rand(nbrOfOutUnits,(nbrOfNeuronsInEachHiddenLayer+1));
%% Input & output of hidde layer
outerlayer_input = hiddenlayer_output*w2';
outerlayer_output = outerlayer_input;
%% Error Calculation
TotalError = 0.5*(TargetClasses-outerlayer_output).^2;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Backward Pass %%%%%%%%%%%
d3 = outerlayer_output - TargetClasses;
d2 = (d3*w2).*hiddenlayer_output.*(1-hiddenlayer_output);
d2 = d2(:,2:end);
D1 = d2' * Input;
D2 = d3' * hiddenlayer_output;
w1_grad = D1/46998 + learningRate*[zeros(size(w1,1),1) w1(:,2:end)]/46998;
w2_grad = D2/46998 + learningRate*[zeros(size(w2,1),1) w2(:,2:end)]/46998;

Answer 1

You should try vectorize your algorithm. First arrange your data in a 46998x12 matrix X.Add bias to X like X=[ones(46998,1 X]. Then the weights leading from input layer to first hidden layer must be arranged in a matrix W1 with dimensions numberofneuronsinfirsthiddenlayer(24)x(input + 1). Then XW1' is what you feed in your neuron function (either is it sigmoid or whatever it is). The result (like sigmoid(XW') is the output of neurons at hidden level 1. You add bias like before and multiply by weight matrix W2 (the weights that lead from hidden layer 1 to hidden layer 2) and so on. Hope this helps to get you started vectorizing your code at least for the feedforward part. The back-propagation part is a little trickier but luckily involves the same matrices.
I will shortly recite the feedforward process so that we use same language talking about backpropagation.
There is the data called X.(dimensions 46998x12)
A1 = [ones(46998,1 X] is the input including bias. (46998x13)
Z2 = A1*W1' (W1 is the weight matrix that leads from input to hidden layer 1)
A2 = sigmoid(Z2);
A2 = [ones(m,1) A2]; adding bias again
Z3 = A2 * W2';
A3 = sigmoid(Z3);
Supposing you only have one hidden layer feedforward stops here. I'll start backwards now and you can generalize as appropriate.
d3 = A3 - Y; (Y must is part of your data, the actual values of the data with which you train your nn)
d2 = (d3 * W2).* A2 .* (1-A2); ( Sigmod function has a nice property that d(sigmoid(z))/dz = sigmoid(z)*(1-sigmoid(z)).)
d2 = d2(:,2:end);(You dont need the first column that corresponds in the bias)
D1 = d2' * A1;
D2 = d3' * A2;
W1_grad = D1/m + lambda*[zeros(size(W1,1),1) W1(:,2:end)]/m; (lamda is the earning rate, m is 46998)
W2_grad = D2/m + lambda*[zeros(size(W2,1),1) W2(:,2:end)]/m;
Everything must be in place now except for the vectorized cost function which have to be minimized. Hope this helps a bit...