Some of you may be familiar with the simple handwritten digit classification NN that's part of Andrew Ng's ML course on coursera. To improve my understanding of the theory I'm trying to modify the implementation such that it outputs one float instead of 10 classification labels.
It has only one hidden layer. The code below is my attempt but the backprop produces wrong gradients. They don't match at all with analytical gradient comparisons. Because it only outputs one number, the activation of that output node is a simple linear function f(x) = x. Due to this, the error in the output unit is simply proportionally distributed backwards to the hidden layer based on weights. Perhaps someone can spot what I'm missing here, why the gradients are wrong.
m is the training data count, y is the correct output in a vector, X holds the training data, one row each.
%average cost over all training data
X = [ones(m, 1) X]; %insert bias column
a2 = tanh(X * Theta1'); %hidden layer activation
a2 = [ones(m, 1) a2];
a3 = a2 * Theta2' %linear combination for final output
Cost = sum((y-a3).^2)/m; %mean squared error
backprop, iterating over m to accumulate gradients before averaging them:
for i = 1:m
a1 = [1 X(i, :)]'; %get current training row
z2 = Theta1 * a1;
a2 = tanh(z2);
a2 = [1; a2];
z3 = Theta2 * a2;
a3 = z3; %no activation function for final output
delta3 = (a3-y(i))^2; %cost of current training row
delta2 = Theta2' * delta3; %proportionally distribute error backwards, because no activation function was used?
delta2 = delta2(2:end); %cut out bias element
Theta2_grad = Theta2_grad + delta3 * a2'; %accumulate gradients
Theta1_grad = Theta1_grad + delta2 * a1';
endfor
Theta1_grad = (1/m) * Theta1_grad; %average gradients
Theta2_grad = (1/m) * Theta2_grad;
I think you are wrong delta3=a3-y not (a3-y)**2
I am having trouble with my code that is meant to provide a cost function for my neural network. The cost function (J) is defined as the cost function. Given sample inputs, the cost function returns a negative value that is about 5 times less than the expected value. I have worked on this issue for a few hours but still cannot get the desired value.
Thanks in advance.
function [J grad] = nnCostFunction(nn_params, ...
input_layer_size, ...
hidden_layer_size, ...
num_labels, ...
X, y, lambda)
%NNCOSTFUNCTION Implements the neural network cost function for a two layer
%neural network which performs classification
% [J grad] = NNCOSTFUNCTON(nn_params, hidden_layer_size, num_labels, ...
% X, y, lambda) computes the cost and gradient of the neural network. The
% parameters for the neural network are "unrolled" into the vector
% nn_params and need to be converted back into the weight matrices.
%
% The returned parameter grad should be a "unrolled" vector of the
% partial derivatives of the neural network.
%
% Reshape nn_params back into the parameters Theta1 and Theta2, the weight matrices
% for our 2 layer neural network
Theta1 = reshape(nn_params(1:hidden_layer_size * (input_layer_size + 1)), ...
hidden_layer_size, (input_layer_size + 1));
Theta2 = reshape(nn_params((1 + (hidden_layer_size * (input_layer_size + 1))):end), ...
num_labels, (hidden_layer_size + 1));
% Setup some useful variables
m = size(X, 1);
% You need to return the following variables correctly
J = 0;
Theta1_grad = zeros(size(Theta1));
Theta2_grad = zeros(size(Theta2));
% ====================== YOUR CODE HERE ======================
% Instructions: You should complete the code by working through the
% following parts.
%
% Part 1: Feedforward the neural network and return the cost in the
one=ones(1,10);
temp=one
one=transpose(temp);
sizeTheta1=size(Theta1);
sizeTheta2=size(Theta2);
for i=1:m
if y(i)==1
yVec=[1,0,0,0,0,0,0,0,0,0];
end
if y(i)==2
yVec=[0,1,0,0,0,0,0,0,0,0];
end
if y(i)==3
yVec=[0,0,1,0,0,0,0,0,0,0];
end
if y(i)==4
yVec=[0,0,0,1,0,0,0,0,0,0];
end
if y(i)==5
yVec=[0,0,0,0,1,0,0,0,0,0];
end
if y(i)==6
yVec=[0,0,0,0,0,1,0,0,0,0];
end
if y(i)==7
yVec=[1,0,0,0,0,0,1,0,0,0];
end
if y(i)==8
yVec=[1,0,0,0,0,0,0,1,0,0];
end
if y(i)==9
yVec=[0,0,0,0,0,0,0,0,1,0];
end
if y(i)==10
yVec=[0,0,0,0,0,0,0,0,0,1];
end
xVec=transpose(X(i,:));
term1=transpose(-yVec).*(log(sigmoid(Theta2(1:10,1:sizeTheta2(2)-1))*(sigmoid(Theta1(1:25,1:sizeTheta1(2)-1)*xVec))));
term2=(one-transpose(yVec)).*(log(one-(sigmoid(Theta2(1:10,1:sizeTheta2(2)-1)*(sigmoid(Theta1(1:25,1:sizeTheta1(2)-1)*xVec))))));
J=J+(term1-term2);
end
regTheta1=0;
regTheta2=0;
J=sum(sum(J))*(1/m);
regTheta1=(sum(sum(Theta1.*Theta1)));
regTheta2=(sum(sum(Theta2.*Theta2)));
J=J+((lambda)*(regTheta1+regTheta2))/(2*m);
% -------------------------------------------------------------
% =========================================================================
% Unroll gradients
grad = [Theta1_grad(:) ; Theta2_grad(:)];
end
I am currently taking Machine Learning on the Coursera platform and I am trying to implement Logistic Regression. To implement Logistic Regression, I am using gradient descent to minimize the cost function and I am to write a function called costFunctionReg.m that returns both the cost and the gradient of each parameter evaluated at the current set of parameters.
The problem is better described below:
My cost function is working, but the gradient function is not. Please note that I would prefer to implement this using looping, rather than element-by-element operations.
I am computing theta[0] (in MATLAB, theta(1)) separately as it is not being regularized, i.e. we do not use the first term (with lambda).
function [J, grad] = costFunctionReg(theta, X, y, lambda)
%COSTFUNCTIONREG Compute cost and gradient for logistic regression with regularization
% J = COSTFUNCTIONREG(theta, X, y, lambda) computes the cost of using
% theta as the parameter for regularized logistic regression and the
% gradient of the cost w.r.t. to the parameters.
% Initialize some useful values
m = length(y); % number of training examples
n = length(theta); %number of parameters (features)
% You need to return the following variables correctly
J = 0;
grad = zeros(size(theta));
% ====================== YOUR CODE HERE ======================
% Instructions: Compute the cost of a particular choice of theta.
% You should set J to the cost.
% Compute the partial derivatives and set grad to the partial
% derivatives of the cost w.r.t. each parameter in theta
% ----------------------1. Compute the cost-------------------
%hypothesis
h = sigmoid(X * theta);
for i = 1 : m
% The cost for the ith term before regularization
J = J - ( y(i) * log(h(i)) ) - ( (1 - y(i)) * log(1 - h(i)) );
% Adding regularization term
for j = 2 : n
J = J + (lambda / (2*m) ) * ( theta(j) )^2;
end
end
J = J/m;
% ----------------------2. Compute the gradients-------------------
%not regularizing theta[0] i.e. theta(1) in matlab
j = 1;
for i = 1 : m
grad(j) = grad(j) + ( h(i) - y(i) ) * X(i,j);
end
for j = 2 : n
for i = 1 : m
grad(j) = grad(j) + ( h(i) - y(i) ) * X(i,j) + lambda * theta(j);
end
end
grad = (1/m) * grad;
% =============================================================
end
What am I doing wrong?
The way you are applying regularization is incorrect. You add regularization after you sum over all training examples but instead you are adding regularization after each example. If you left your code as it was before the correction, you are inadvertently making the gradient step larger and will eventually overshoot the solution. This overshooting will accumulate and will inevitably give you a gradient vector of Inf or -Inf for all components (except for the bias term).
Simply put, place your lambda*theta(j) statement after the second for loop terminates:
for j = 2 : n
for i = 1 : m
grad(j) = grad(j) + ( h(i) - y(i) ) * X(i,j); % Change
end
grad(j) = grad(j) + lambda * theta(j); % Change
end
I'm new with Matlab and Machine Learning and I tried to make a gradient descent function without using matrix.
m is the number of example on my training set
n is the number of feature for each example
The function gradientDescentMulti takes 5 arguments:
X mxn Matrix
y m-dimensional vector
theta : n-dimensional vector
alpha : a real number
nb_iters : a real number
I already have a solution using matrix multiplication
function theta = gradientDescentMulti(X, y, theta, alpha, num_iters)
for iter = 1:num_iters
gradJ = 1/m * (X'*X*theta - X'*y);
theta = theta - alpha * gradJ;
end
end
The result after iterations:
theta =
1.0e+05 *
3.3430
1.0009
0.0367
But now, I tried to do the same without matrix multiplication, this is the function:
function theta = gradientDescentMulti(X, y, theta, alpha, num_iters)
m = length(y); % number of training examples
n = size(X, 2); % number of features
for iter = 1:num_iters
new_theta = zeros(1, n);
%// for each feature, found the new theta
for t = 1:n
S = 0;
for example = 1:m
h = 0;
for example_feature = 1:n
h = h + (theta(example_feature) * X(example, example_feature));
end
S = S + ((h - y(example)) * X(example, n)); %// Sum each feature for this example
end
new_theta(t) = theta(t) - alpha * (1/m) * S; %// Calculate new theta for this example
end
%// only at the end of the function, update all theta simultaneously
theta = new_theta'; %// Transpose new_theta (horizontal vector) to theta (vertical vector)
end
end
The result, all the theta are the same :/
theta =
1.0e+04 *
3.5374
3.5374
3.5374
If you look at the gradient update rule, it may be more efficient to actually compute the hypothesis of all of your training examples first, then subtract this with the ground truth value of each training example and store these into an array or vector. Once you do this, you can then compute the update rule very easily. To me, it doesn't appear that you're doing this in your code.
As such, I rewrote the code, but I have a separate array that stores the difference in the hypothesis of each training example and ground truth value. Once I do this, I compute the update rule for each feature separately:
for iter = 1 : num_iters
%// Compute hypothesis differences with ground truth first
h = zeros(1, m);
for t = 1 : m
%// Compute hypothesis
for tt = 1 : n
h(t) = h(t) + theta(tt)*X(t,tt);
end
%// Compute difference between hypothesis and ground truth
h(t) = h(t) - y(t);
end
%// Now update parameters
new_theta = zeros(1, n);
%// for each feature, find the new theta
for tt = 1 : n
S = 0;
%// For each sample, compute products of hypothesis difference
%// and the right feature of the sample and accumulate
for t = 1 : m
S = S + h(t)*X(t,tt);
end
%// Compute gradient descent step
new_theta(tt) = theta(tt) - (alpha/m)*S;
end
theta = new_theta'; %// Transpose new_theta (horizontal vector) to theta (vertical vector)
end
When I do this, I get the same answers as using the matrix formulation.
Cost function
I am trying to code the above expression in Matlab. Unfortunately I seem to be getting a cost of 10.441460 instead of 0.287629 so I'm out by a factor of over 36!
As for each of the symbols:
m is the number of training examples. [a scalar number]
K is the number of output nodes. [a scalar number]
y is the vector of training outputs [an m by 1 vector]
y^{(i)}_{k} is the ith training output (target) for the kth output node.
[a scalar number]
x^{(i)} is the ith training input. [a column vector for all the input
nodes]
h_{\theta}(x^{(i)})_{k} is the value of the hypothesis at output k, with
weights theta, and training input i. [a scalar number]
note: h_{\theta}(x^{(i)}) will be a column vector with K rows.
My attempt for the cost function:
Theta1 = [ones(1,size(Theta1,2));Theta1];
X = [ones(m,1) , X]; %Add a column of 1's to X
R=zeros(m,1);
for i = 1:m
a = y(i) == [10 1:9];
R(i) = -(a*(log(sigmoid(Theta2*(sigmoid(Theta1*X(i,:)'))))) + (1-a)*(log(1-sigmoid(Theta2*(sigmoid(Theta1*X(i,:)'))))))/m;
end
J = sum(R);
This will probably be useful for reference:
function [J grad] = nnCostFunction(nn_params, ...
input_layer_size, ...
hidden_layer_size, ...
num_labels, ...
X, y, lambda)
Theta1 = reshape(nn_params(1:hidden_layer_size * (input_layer_size + 1)), ...
hidden_layer_size, (input_layer_size + 1));
Theta2 = reshape(nn_params((1 + (hidden_layer_size * (input_layer_size + 1))):end), ...
num_labels, (hidden_layer_size + 1));
% Setup some useful variables
m = size(X, 1);
% ====================== YOUR CODE HERE ======================
% Instructions: You should complete the code by working through the
% following parts.
%
% Part 1: Feedforward the neural network and return the cost in the
% variable J. After implementing Part 1, you can verify that your
% cost function computation is correct by verifying the cost
% computed in ex4.m
%
% =========================================================================
end