matlab fminunc not quitting (running indefinitely) - matlab

I have been trying to implement logistic regression in matlab for a while now. I have done it already, but for reasions unknown to me, I am unable to perform a single iteration using fminunc. When the function it called, the program just go in to wait mode indefinitely. Is there something wrong with code, or is my data set to large?
function [theta J] = logisticReg(initial_theta, X, y, lambda, iter)
% Set Options
options = optimset('GradObj', 'on', 'MaxIter', iter);
% Optimize
[theta, J, exit_flag] = fminunc(#(t)(costFunctionReg(t, X, y, lambda)), initial_theta, options);
end
where
X is a [676,6251] matrix
y is a [676,1] vector
lambda = 1
initial_theta is [6251, 1] vector of zeros
iter = 1
Any 'pointing in the right direction' will be greatly appreciated!
P.S. and I was able to run costFunctionReg. So am assuming it is this function.
as requested the costFunctionReg
function [J, grad] = costFunctionReg(theta, X, y, lambda)
m = length(y); % number of training examples
J = 0;
grad = zeros(size(theta));
hyp = sigmoid(X * theta);
cost_reg = (lambda / (2*m)) * sum(theta(2:end).^2);
J = 1/m * sum((-y .* log(hyp)) - ((1-y) .* log(1-hyp))) + cost_reg;
grad(1) = 1/m * sum((hyp - y)' * X(:,1));
grad(2:end) = (1/m * ((hyp - y)' * X(:,2:end))) + (lambda/m)*theta(2:end)';
to answer #Rasman question:
Cost at initial theta: NaN
press any key to continue
Performing Logistic Regrestion
Error using sfminbx (line 28)
Objective function is undefined at initial point. fminunc cannot continue.
Error in fminunc (line 406)
[x,FVAL,~,EXITFLAG,OUTPUT,GRAD,HESSIAN] = sfminbx(funfcn,x,l,u, ...
Error in logisticReg (line 8)
[theta, J, exit_flag] = fminunc(#(t)(costFunctionReg(t, X, y, lambda)),
initial_theta, options);
Error in main (line 40)
[theta J] = logisticReg(initial_theta, X, y, lambda, iter);
The first line is me running costFunctionReg with initial_theta.

It's possible that you already tried searching this link:
http://www.mathworks.com/matlabcentral/newsreader/view_thread/290418
The general arc (I've copied and pasted/made edits to the text from the above site) is:
The error message indicates that your objective function "obj" either
errors, or returns and invalid value such as Inf, NaN or a complex
number when evaluated at the point x0.
You may want to evaluate your function at x0 before calling fmincon (or in your case, fminunc) to see if it's well defined: something like
costFunctionReg(initial_theta)
And if your function (costFunctionReg) is returning a complex-valued result, then use real() to strip it away.

Related

Trouble computing cost in logistic regression

I am taking the course from Andrew Ng on Machine Learning on Coursera. In this assginment, I am working to calculate the cost function using logistic regression in MatLab, but am receiving "Error using sfminbx (line 27)
Objective function is undefined at initial point. fminunc cannot continue.".
I should add that the cost J within the costFunction function below is NaN because the log(sigmoid(X * theta)) is a -Inf vector. I'm sure this is related to the exception. Can you please help?
My cost function looks like the following:
function [J, grad] = costFunction(theta, X, y)
m = length(y); % number of training examples
J = 0;
grad = zeros(size(theta));
h = sigmoid(theta * X);
J = - (1 / m) * ((log(h)' * y) + (log(1 - h)' * (1 - y)));
grad = (1 / m) * X' * (h - y);
end
My code that calls this function looks like the following:
data = load('ex2data1.txt');
X = data(:, [1, 2]); y = data(:, 3);
[m, n] = size(X);
% Add intercept term to x and X_test
X = [ones(m, 1) X];
% Initialize fitting parameters
initial_theta = zeros(n + 1, 1);
% Compute and display initial cost and gradient
[cost, grad] = costFunction(initial_theta, X, y);
fprintf('Cost at initial theta (zeros): %f\n', cost);
fprintf('Expected cost (approx): 0.693\n');
fprintf('Gradient at initial theta (zeros): \n');
fprintf(' %f \n', grad);
fprintf('Expected gradients (approx):\n -0.1000\n -12.0092\n -11.2628\n');
% Compute and display cost and gradient with non-zero theta
test_theta = [-24; 0.2; 0.2];
[cost, grad] = costFunction(test_theta, X, y);
fprintf('\nCost at test theta: %f\n', cost);
fprintf('Expected cost (approx): 0.218\n');
fprintf('Gradient at test theta: \n');
fprintf(' %f \n', grad);
fprintf('Expected gradients (approx):\n 0.043\n 2.566\n 2.647\n');
fprintf('\nProgram paused. Press enter to continue.\n');
pause;
%% ============= Part 3: Optimizing using fminunc =============
% In this exercise, you will use a built-in function (fminunc) to find the
% optimal parameters theta.
% Set options for fminunc
options = optimset('GradObj', 'on', 'MaxIter', 400, 'Algorithm', 'trust-
region');
% Run fminunc to obtain the optimal theta
% This function will return theta and the cost
[theta, cost] = ...
fminunc(#(t)(costFunction(t, X, y)), initial_theta, options);
end
The dataset looks like the following:
34.62365962451697,78.0246928153624,0
30.28671076822607,43.89499752400101,0
35.84740876993872,72.90219802708364,0
60.18259938620976,86.30855209546826,1
79.0327360507101,75.3443764369103,1
45.08327747668339,56.3163717815305,0
61.10666453684766,96.51142588489624,1
75.02474556738889,46.55401354116538,1
76.09878670226257,87.42056971926803,1
84.43281996120035,43.53339331072109,1
95.86155507093572,38.22527805795094,0
75.01365838958247,30.60326323428011,0
82.30705337399482,76.48196330235604,1
69.36458875970939,97.71869196188608,1
39.53833914367223,76.03681085115882,0
53.9710521485623,89.20735013750205,1
69.07014406283025,52.74046973016765,1
67.94685547711617,46.67857410673128,0
70.66150955499435,92.92713789364831,1
76.97878372747498,47.57596364975532,1
67.37202754570876,42.83843832029179,0
89.67677575072079,65.79936592745237,1
50.534788289883,48.85581152764205,0
34.21206097786789,44.20952859866288,0
77.9240914545704,68.9723599933059,1
62.27101367004632,69.95445795447587,1
80.1901807509566,44.82162893218353,1
93.114388797442,38.80067033713209,0
61.83020602312595,50.25610789244621,0
38.78580379679423,64.99568095539578,0
61.379289447425,72.80788731317097,1
85.40451939411645,57.05198397627122,1
52.10797973193984,63.12762376881715,0
52.04540476831827,69.43286012045222,1
40.23689373545111,71.16774802184875,0
54.63510555424817,52.21388588061123,0
33.91550010906887,98.86943574220611,0
64.17698887494485,80.90806058670817,1
74.78925295941542,41.57341522824434,0
34.1836400264419,75.2377203360134,0
83.90239366249155,56.30804621605327,1
51.54772026906181,46.85629026349976,0
94.44336776917852,65.56892160559052,1
82.36875375713919,40.61825515970618,0
51.04775177128865,45.82270145776001,0
62.22267576120188,52.06099194836679,0
77.19303492601364,70.45820000180959,1
97.77159928000232,86.7278223300282,1
62.07306379667647,96.76882412413983,1
91.56497449807442,88.69629254546599,1
79.94481794066932,74.16311935043758,1
99.2725269292572,60.99903099844988,1
90.54671411399852,43.39060180650027,1
34.52451385320009,60.39634245837173,0
50.2864961189907,49.80453881323059,0
49.58667721632031,59.80895099453265,0
97.64563396007767,68.86157272420604,1
32.57720016809309,95.59854761387875,0
74.24869136721598,69.82457122657193,1
71.79646205863379,78.45356224515052,1
75.3956114656803,85.75993667331619,1
35.28611281526193,47.02051394723416,0
56.25381749711624,39.26147251058019,0
30.05882244669796,49.59297386723685,0
44.66826172480893,66.45008614558913,0
66.56089447242954,41.09209807936973,0
40.45755098375164,97.53518548909936,1
49.07256321908844,51.88321182073966,0
80.27957401466998,92.11606081344084,1
66.74671856944039,60.99139402740988,1
32.72283304060323,43.30717306430063,0
64.0393204150601,78.03168802018232,1
72.34649422579923,96.22759296761404,1
60.45788573918959,73.09499809758037,1
58.84095621726802,75.85844831279042,1
99.82785779692128,72.36925193383885,1
47.26426910848174,88.47586499559782,1
50.45815980285988,75.80985952982456,1
60.45555629271532,42.50840943572217,0
82.22666157785568,42.71987853716458,0
88.9138964166533,69.80378889835472,1
94.83450672430196,45.69430680250754,1
67.31925746917527,66.58935317747915,1
57.23870631569862,59.51428198012956,1
80.36675600171273,90.96014789746954,1
68.46852178591112,85.59430710452014,1
42.0754545384731,78.84478600148043,0
75.47770200533905,90.42453899753964,1
78.63542434898018,96.64742716885644,1
52.34800398794107,60.76950525602592,0
94.09433112516793,77.15910509073893,1
90.44855097096364,87.50879176484702,1
55.48216114069585,35.57070347228866,0
74.49269241843041,84.84513684930135,1
89.84580670720979,45.35828361091658,1
83.48916274498238,48.38028579728175,1
42.2617008099817,87.10385094025457,1
99.31500880510394,68.77540947206617,1
55.34001756003703,64.9319380069486,1
74.77589300092767,89.52981289513276,1
The only problem I see is that you should have written h = sigmoid(X * theta) instead of h = sigmoid(theta * X). I am getting the same answer from your code after changing this as I was getting from my code for the same assignment.

DerivativeCheck fails in fsolve function (Jacobian)

I am trying to find a solution for a non-linear system in MATLAB using the fsolve function. I have an idea of the region the solution is, so my initial points are a random variation inside this area. The code follows:
%MATLAB parameters
digits = 32;
format shortEng
%Initialization of solving process
min_norm = realmax;
options = optimoptions('fsolve', 'Algorithm', 'trust-region-dogleg',...
'Display', 'off', 'FunValCheck', 'on', 'Jacobian', 'on',...
'DerivativeCheck', 'on');
%Solving loop
while 1
%Refreshes seed after 100000 iterations
rng('shuffle');
for n=1:100000
%Initial points are randomly placed in solving region
x_0 = zeros(1, 3);
x_0(1) = 20*(10^-3)+ abs(3*(10^-3)*randn);
x_0(2) = abs(10^(-90) + 10^-89*randn);
x_0(3) = abs(10*randn);
%Solving
[x, fval] = fsolve(#fbnd, x_0, options);
norm_fval = norm(fval);
%If norm is minimum, result is displayed
if all(x > 0.0) && (norm_fval < min_norm)
iter_solu = x;
display(norm_fval);
display(iter_solu);
min_norm = norm_fval;
end
end
end
The function to be optimized and its jacobian follow:
function [F, J] = fbnd(x)
F(1) = x(1) - x(2)*(exp(7.56/(1.5*0.025))-1);
F(2) = x(1) - x(2)*(exp((x(3)*0.02)/(1.5*0.025))-1)-0.02;
F(3) = x(1) - x(2)*(exp((6.06+x(3)*0.018)/(1.5*0.025))-1)-0.018;
J = [[1.0, -3.5790482371355382991651020082214*10^87, 0.0];...
[1.0, 1.0-exp((8/15)*x(3)),-(8/15)*x(2)*exp((8/15)*x(3))];...
[1.0, 1.0-exp(0.48*x(3)+161.6), -0.48*x(2)*exp(0.48*x(3)+161.6)]];
end
When I turn DerivativeCheck on, I get the following error:
Objective function derivatives:
Maximum relative difference between user-supplied
and finite-difference derivatives = 1.
User-supplied derivative element (1,1): 1
Finite-difference derivative element (1,1): 0
For some reason, it things dF(1)/dx(1) should be zero instead of 1, when it is obvious it is 1. So I went ahead and replaced the J(1,1) by 0, and now I get the same problem with J(3,1):
Objective function derivatives:
Maximum relative difference between user-supplied
and finite-difference derivatives = 1.
User-supplied derivative element (3,1): 1
Finite-difference derivative element (3,1): 0
So, I replaced J(3,1) by 0 and it catches only minor precision errors. Is there an explanation why it thinks J(1,1) and J(3,1) should be zero? It seems really strange to me.

How do I plot an "indefite-sum" function in matlab?

My function is just a simple summation from -inf to inf: f(x) = sum(pi * exp(-x+2pi*j), j = -inf to in);
I tried doing this, but got error saying "A numeric or double convertible argument is expected"
x = linspace(-7, 7, 1000);
h = 10;
syms j;
v_hat = symsum(pi * exp(x + 2*pi*j),-inf, inf); %v_hat is a function of x: v_hat(x)
plot(x, v_hat);
I see nothing wrong with your code programming-wise, but there is a fatal mathematical flaw -- the sum is infinite for every x. Note that it's just sum of exp(2*pi*j) multiplied by pi*exp(x). The values exp(2*pi*j) are large when j is positive, and you try to add infinitely many of them...
When I change the formula to a series that actually converges, the code works, although it takes a long while.
x = linspace(-7, 7, 1000);
h = 10;
syms j;
v_hat = symsum(pi * exp(-abs(x+ 2*pi*j/h) ),-inf, inf);
plot(x, v_hat);
Note that the quality isn't great despite using 1000 data points. You will get better quality much faster by abandoning symbolic sum. The contribution of the terms with abs(j)>100 is tiny here, so drop them and use numeric sum over the rest.
h = 10;
x = linspace(-7,7,10000);
[X,j] = meshgrid(x, -100:100);
v_hat = sum(pi * exp(-abs(X+ 2*pi*j/h) ), 1);
plot(x, v_hat);
Also, realize that the function is periodic with period 2*pi/h. So you can plot
one or two periods of the function, and imagine the rest.
h = 10;
x = linspace(-2*pi/h, 2*pi/h, 101);
syms j;
v_hat = symsum(pi * exp(-abs(x+ 2*pi*j/h) ),-inf, inf);
plot(x, v_hat);

Use Matlab/Maple to find roots of a nonlinear equation

I am having difficulty in finding roots of a nonlinear equation. I have tried Matlab and Maple both, and both give me the same error which is
Error, (in RootFinding:-NextZero) can only handle isolated zeros
The equation goes like
-100 + 0.1335600000e-5*H + (1/20)*H*arcsinh(2003.40/H)
The variable is H in the equation.
How do I find the roots (or the approximate roots) of this equation?
Matlab Code:
The function file:
function hor_force = horizontal(XY, XZ, Lo, EAo, qc, VA)
syms H
equation = (-1*ZZ) + (H/qc)*(cosh((qc/H)*(XZ- XB))) - H/qc + ZB;
hor_force = `solve(equation);`
The main file:
EAo = 7.5*10^7;
Lo = 100.17;
VA = 2002;
XY = 0;
ZY = 0;
XB = 50;
ZB = -2;
XZ = 100;
ZZ = 0;
ql = 40;
Error which Matlab shows:
Error using sym/solve (line 22)
Error using maplemex
Error, (in RootFinding:-NextZero) can only handle isolated zeros
Error in horizontal (line 8)
hor_force = solve(equation);
Error in main (line 34)
h = horizontal(XY, XZ, Lo, EAo, ql, VA)
http://postimg.org/image/gm93z3b7z/
You don't need the symbolic toolbox for this:
First, create an anonymous function that can take vectors at input (use .* and ./:
equation = #(H) ((-1*ZZ) + (H./qc).*(cosh((qc./H).*(XZ- XB))) - H./qc + ZB);
Second, create a vector that you afterwards insert into the equation to find approximately when the sign of the function changes. In the end, use fzero with x0 as the second input parameter.
H = linspace(1,1e6,1e4);
x0 = H(find(diff(sign(equation(H))))); %// Approximation of when the line crosses zero
x = fzero(equation, x0) %// Use fzero to find the crossing point, using the initial guess x0
x =
2.5013e+04
equation(x)
ans =
0
To verify:
You might want to check out this question for more information about how to find roots of non-polynomials.
In Maple, using the expression from your question,
restart:
ee := -100 + 0.1335600000e-5*H + (1/20)*H*arcsinh(2003.40/H):
Student:-Calculus1:-Roots(ee, -1e6..1e6);
[ 5 ]
[-1.240222868 10 , -21763.54830, 18502.23816]
#plot(ee, H=-1e6..1e6, view=-1..1);

Create flexible function handle

I am using numerical integration in MATLAB, with one varibale to integrate over but the function also contains a variable number of terms depending on the dimension of my data. Right now this looks like the following for the 2-dimensional case:
for t = 1:T
fxt = #(u) exp(-0.5*(x(t,1)-theta*norminv(u,0,1)).^2) .* ...
exp(-0.5*(x(t,2) -theta*norminv(u,0,1)).^2);
f(t) = integral(fxt,1e-4,1-1e-4,'AbsTol',1e-3);
end
I would like to have this function flexible in the sense that there could be any number of data points in, each in the following term:
exp(-0.5*(x(t,i) -theta*norminv(u,0,1)).^2);
I hope this is understandable.
If x and u have a valid dimension match (vector-vector or array-scalar) for the subtraction, you can put the whole matrix x into the handle and pass it to the integral function using the name-parameter pair ('ArrayValued',true):
fxt = #(u) exp(-0.5*(x - theta*norminv(u,0,1)).^2) .* ...
exp(-0.5*(x - theta*norminv(u,0,1)).^2);
f = integral(fxt,1e-4,1-1e-4,'AbsTol',1e-3,'ArrayValued',true);
[Documentation]
You may need a loop if integral ever passes a vector u into the handle.
But in looking at how the integral function is written, the integration nodes are entered as scalars for array-valued functions, so the loop shouldn't be necessary unless some weird dimension-mismatch error is thrown.
Array-Valued Output
In response to the comments below, you could try this function handle:
fx = #(u,t,k) prod(exp(-0.5*(x(t,1:k)-theta*norminv(u,0,1)).^2),2);
Then your current loop would look like
fx = #(u,t,k) prod(exp(-0.5*(x(t,1:k)-theta*norminv(u,0,1)).^2),2);
k = 2;
for t = 1:T
f(t) = integral(#(u)fx(u,t,k),1e-4,1-1e-4,'AbsTol',1e-3,'ArrayValued',true);
end
The ArrayValued flag is needed since x and u will have a dimension mismatch.
In this form, another loop would be needed to sweep through the k indexes.
However, we can improve this function by skipping the loop altogether since each iterate of the loop is independent by using the ArrayValued mode:
fx = #(u,k) prod(exp(-0.5*(x(:,1:k)-theta*norminv(u,0,1)).^2),2);
k = 2;
f = integral(#(u)fx(u,k),1e-4,1-1e-4,'AbsTol',1e-3,'ArrayValued',true);
Vector-Valued Output
If ArrayValued is not desired, which may be the case if the integration requires a lot of subdivisions and a vector-valued u is preferable, you can also try a recursive version of the handle using cell arrays:
% x has size [T,K]
fx = cell(K,1);
fx{1} = #(u,t) exp(-0.5*(x(t,1) - theta*norminv(u,0,1)).^2);
for k = 2:K
fx{k} = #(u,t) fx{k-1}(u,t).*exp(-0.5*(x(t,k) - theta*norminv(u,0,1)).^2);
end
f(T) = 0;
k = 2;
for t = 1:T
f(t) = integral(#(u)fx{k}(u,t),1e-4,1-1e-4,'AbsTol',1e-3);
end
ThanksTroy but now I run into the follwing:
x = [0.3,0.8;1.5,-0.7];
T = size(x,1);
k = size(x,2);
theta= 1;
fx = #(u,t,k) prod(exp(-0.5*(x(t,1:k) - theta*norminv(u,0,1))^2));
for t = 1,T
f(t) = integral(#(u)fx(u,t,k),1e-4,1-1e-4,'AbsTol',1e-3);
end
Error using -
Matrix dimensions must agree.
Error in #(u,t,k)prod(exp(-0.5*(x(t,1:k)-theta*norminv(u,0,1))^2))
Error in #(u)fx(u,t,k)
Error in integralCalc/iterateScalarValued (line 314)
fx = FUN(t);
Error in integralCalc/vadapt (line 133)
[q,errbnd] = iterateScalarValued(u,tinterval,pathlen);
Error in integralCalc (line 76)
[q,errbnd] = vadapt(#AtoBInvTransform,interval);
Error in integral (line 89)
Q = integralCalc(fun,a,b,opstruct);