I want to minimize a function like below:
Here, n can be 5,10,50 etc. I want to use Matlab and want to use Gradient Descent and Quasi-Newton Method with BFGS update to solve this problem along with backtracking line search. I am a novice in Matlab. Can anyone help, please? I can find a solution for a similar problem in that link: https://www.mathworks.com/help/optim/ug/unconstrained-nonlinear-optimization-algorithms.html .
But, I really don't know how to create a vector-valued function in Matlab (in my case input x can be an n-dimensional vector).
You will have to make quite a leap to get where you want to be -- may I suggest to go through some basic tutorial first in order to digest basic MATLAB syntax and concepts? Another useful read is the very basic example to unconstrained optimization in the documentation. However, the answer to your question touches only basic syntax, so we can go through it quickly nevertheless.
The absolute minimum to invoke the unconstraint nonlinear optimization algorithms of the Optimization Toolbox is the formulation of an objective function. That function is supposed to return the function value f of your function at any given point x, and in your case it reads
function f = objfun(x)
f = sum(100 * (x(2:end) - x(1:end-1).^2).^2 + (1 - x(1:end-1)).^2);
end
Notice that
we select the indiviual components of the x vector by matrix indexing, and that
the .^ notation effects that the operand is to be squared elementwise.
For simplicity, save this function to a file objfun.m in your current working directory, so that you have it available from the command window.
Now all you have to do is to call the appropriate optimization algorithm, say, the quasi Newton method, from the command window:
n = 10; % Use n variables
options = optimoptions(#fminunc,'Algorithm','quasi-newton'); % Use QM method
x0 = rand(n,1); % Random starting guess
[x,fval,exitflag] = fminunc(#objfun, x0, options); % Solve!
fprintf('Final objval=%.2e, exitflag=%d\n', fval, exitflag);
On my machine I see that the algorithm converges:
Local minimum found.
Optimization completed because the size of the gradient is less than
the default value of the optimality tolerance.
Final objval=5.57e-11, exitflag=1
I'm using ode45 to solve/plot a second-order differential equation in Matlab. My tspan is from 0 to 0.25. But the initial conditions near zero are ill-defined (slope goes to infinity, complex values). The conditions near 0.25 are well defined (both slope and value are zero).
Questions:
Can I reverse tspan, and use the "final conditions" as initial conditions?
Well, I know I can do it (see code below), and I get a plot that looks like what I expect, but is this a valid thing to do in general? Am I getting lucky in this one case?
ode45 provides numerical solutions and is not exact. Am I likely to have larger error after reversing tspan?
Here's my code, which should run standalone:
function ReverseTspan()
% solve diff-eq backward from tspan end to tspan start using ode45()
% - Good initial conditions at the end, but not start.
% - Is this a valid thing to do?
% clean slate
clc; clear all; close all;
% tspan - reversed!
R = 0.25:-0.001:0;
% initial values
hinits=[0.0000001;0];
% solve
[R,v] = ode45(#equ7,R,hinits);
% strip imaginary values (can't plot 'em)
v(find(real(v)~=v)) = NaN;
% plot first column
plot(R,v(:,1));
function vprime = equ7(R,v);
% Solve second order non-linear differential equation 7:
% v''(R) + 2/R*v'(R) = K_plus/(R^2)*( v^(-1/2) - lamda_plus*(1-v)^(-1/2)
%
% Matlab ode45 only likes first derivatives, so let:
% v_1(R) = v(R)
% v_2(R) = v'(R)
%
% And create a system of first order diff eqs:
% v_1'(R) = v_2(R)
% v_2'(R) = -2/R*v_2(R) + K_plus/(R^2)*( v_1(R)^(-1/2) - lamda_plus*(1-v_1(R))^(-1/2)
%
% Constant Parameters:
K_plus = 0.0859;
lambda_plus = 3.7;
% Build result in pieces for easier debugging of problematic terms
int1 = 1 - v(1);
int2 = int1^(-1/2);
int3 = v(1)^(-1/2);
int4 = K_plus/(R^2);
vprime2 = -2/R*v(2);
vprime2 = vprime2 + int4*( int3 - lambda_plus*(int2) );
vprime=[v(2); vprime2 ];
Numerically, you're never going to be able to start at (or get to) R=0 with that ODE. NaN will be returned for the first step after your initial condition, thus corrupting any future results. It might be possible to rescale your system to get rid of the singularity, but that's a mathematics issue. You could also try starting your integration at a time just a bit larger than zero, provided you specify initial conditions that are valid (satisfy your ODE).
Can I reverse tspan, and use the "final conditions" as initial conditions?
Yes. This is a standard procedure. In particular, it's often used to find unstable manifolds and unstable equilibria.
Well, I know I can do it (see code below), and I get a plot that looks like what I expect, but is this a valid thing to do in general? Am I getting lucky in this one case?
It's hard to generalize across all possible systems. There are likely cases where this won't work for reasons of stability or numerical error.
ode45 provides numerical solutions and is not exact. Am I likely to have larger error after reversing tspan?
I can't see why it would necessarily be larger, but again this will depend on the system and solver used. Try using other solvers (e.g., ode15s or ode113) and/or varying the absolute and relative tolerances via odeset.
Suppose I have a function in Matlab calc(x,a,b) which outputs a scalar. a and b are constants, x is treated as multivariate. How do I minimize calc(x,a,b) with respect to x in Matlab?
edit: The content of the function creates a vector $v(x)$ and a matrix $A(x)$ and then computes $v(x)'*A(x)^(-1)*v(x)$
This is a fairly general question with a million possible responses depending on what calc is. (For instance, Can you provide gradients for calc? Does x need to take on values in a specific range?)
But, as a start, go for fminunc. It is for functions where you have no gradient information available and you want to find an unconstrained minimum.
Sample Code:
Suppose you want to minimize dot(x,x).
calc = #(x,a,b) dot(x,x)
calc_to_pass_to_fminunc = #(x) calc(x,1,2)
X = fminunc(calc_to_pass_to_fminunc,ones(3,1))
Gives:
Warning: Gradient must be provided for trust-region algorithm;
using line-search algorithm instead.
> In fminunc at 383
Local minimum found.
Optimization completed because the size of the gradient is less than
the default value of the function tolerance.
<stopping criteria details>
X =
0
0
0
The easy answer is: if a and b are constants, and x is a one-dimensional variable, it's a 1-D optimization problem.
The previous answer suggests to usefminunc, which is part of the MATLAB Optimization Toolbox. If you don't have it, you can use fminbnd instead of it, which works just well in case of 1-D optimization in a given interval.
As example, let's say your calc function is:
function [y] = calc(x,a,b)
y = x.^3-2*x-5+a-b;
end
This is what you should do to find the minimum in the interval x1 < x < x2:
% constants
a = 1;
b = 2;
% boundaries of search interval
x1 = 0;
x2 = 2;
x = fminbnd(#(x)calc(x,a,b), x1, x2);
% value of function at the minimum
y = calc(x,y,a);
In the case of the x variable not being a scalar, you could use the analogous of fminbnd for a multidimensional variable: fminsearch, which performs an unconstrained search for the minimum of a multivariate function.
Addendum
fminbnd is a nice tool, but sometimes it's hard to make it behave as you expect. Of course you can specify the desired accuracy and a maximum number of iterations for converging in the options, but in my experience fminbnd might have problems with highly non-linear functions.
In these situations it's desirable to have a finer control on the optimization procedure, and especially on how it's defined the search interval. Given the search interval, arrayfun provides an elegant way to iterate over an array for finding a minimum of the function. Sample code:
% constants
a = 1;
b = 2;
% search interval
xi = linspace(0,2,1000);
yi = arrayfun(#(x)calc(x,a,b), xi);
% value of function at the minimum
[y, idx_m] = min(yi);
% location of minimum
x = xi(idx_m);
The drawback of this approach is that, in order to achieve a high accuracy, you might need a very long array xi. Good thing is that there are several ways to mitigate this issue: for instance, one could use a vector of log-spaced sampling points, or perform a multi-step minimization narrowing and increasing the sampling frequency at each step until the desired accuracy is achieved.
I have a multivariable function that I wish to minimize. The function has two input arguments, a vector c and a scalar \theta.
Using fmincon in MATLAB to solve the optimization problem for both c and \theta is complicated because certain values of \theta causes numerical errors. However, fixing theta, c can be easily obtained via fmincon without any errors.
So the plan now is to do a brute force approach, i.e. compute c for each value of \theta in the range 1:100 (although the true constraint for \theta is \theta \ge 0) and choose \theta (and the corresponding c) for which the objective value is minimized simply by plugging the estimated parameters back to the objective function.
Now this doesn't sound very efficient to me and I'm wondering if I can employ a bisection method-esque approach so that I would not have to go over all possible values of \theta in the range specified above.
Thanks a lot!
You should be able to let fmincon do you work for you on both c and theta. If it has problems getting a decent result when theta is included, it is likely because the elements in c and theta are of very different scales. You should scale your equations so that all of the variables end up around a value of 1.0. This greatly improves the performance (ie, speed) and accuracy of nearly any numerical optimization code.
So, if you suspect that the final values of c might end up being [1.0 0.001 10.0] and you suspect that theta might end up as [10.0], you would formulate your problem as
>>>>>>>>>>>>>>>>>>>>> in your main program prior to invoking fmincon
c = [1.0 0.001 10.0]
theta = 10.0
foo_x = [c(:);thetha];
scale_fac = [1.0 1000.0 0.1 0.1];
x = foo_x .* scale_fac; %scale your seed values to be near 1.0
>>>>>>>>>>>>>> inside your function
function err = myfunction(x,scale_fac)
foo_x = x ./ scale_fac; %de-scale to get back to correct magnitude
c = foo_x(1:3);
theta = foo_x (4);
...rest of your code
Bisection search over theta will only work if the objective function is convex (or quasiconvex) in theta. Otherwise, you risk finding a local min instead of a global min.
Doing a nested fmincon, as #chipaudette suggests, should work if you choose a solver capable of solving nonconvex optimization problems. (The MATLAB help on this topic is a little vague, but I think the SQP solver should be OK.) But I suspect it will be more efficient just to enumerate over the relevant range of theta, rather than using fmincon for it.
I am using the global maximization toolbox to maximize the following function:
function x = NameOfFunction (w1, w2, w3, a, b, c, Structure1, Structure2, Structure3)
where I am minimizing x by changing the values of w1, w2, and w3. The remaining parameters are constants and structures containing data. The value of x, as well as the three w variables depend on the data that is fed into the function via the structures.
The function returns x which is the mean of 180 values that are calculated in the process of running NameOfFunction.
I am wondering how I could add a constraint on the standard deviation of the 180 values. I am not interested in minimizing both the mean and the standard deviation, but rather to minimize x(the mean), while allowing standard deviation to be no greater than some specific value. I know how to add constraints to the decision variables (ie. w1, w2, w3), but have no idea how to do so for a value like the standard deviation.
EDIT: More detail, per Werner's suggestion:
%the functions is f(w) rather than f(x)
%constraints:
Aeq = [1 1 1];
beq = 1;
lb = .10 * [1 1 1];
ub = .8 * [1 1 1];
w = [weight1, weight2, weight3];
wstart = randn(3,1);
options = optimset('Algorithm','interior-point');
% function handle for the objective function (note that variables
% aa through hh are additional parameters that the solver does not modify):
h = #(w)NameOfFunction(w(1),w(2),w(3), aa, bb, cc, dd, ee, ff, gg, hh);
% problem structure:
problem = createOptimProblem('fmincon','x0',wstart,'objective',h,...'
'Aeq',Aeq,'beq',beq,'options',options,'lb',lb,'ub',ub);
gs = GlobalSearch;
run(gs,problem)
I'm running a GlobalSearch using fmincon.
7/16/2013, After implementing nonlcon I was able to achieve what I tried to do. (I have a follow-up question, which I put on the bottom of this post). Here's what I did:
I added another function (StdConstraintFunction) as discussed. So now I have the following:
stdMax = 0.01;
h = #(w)NameOfFunction(w(1),w(2),w(3),aa, bb, cc, dd, ee, ff, gg);
StdConstraint = #(w)StdConstraintFunction(w(1),w(2),w(3),aa, bb, cc, dd, ee, ff, gg,stdMax);
where StdConstraintFunction is a modified version of NameOfFunction that calculates the standard deviation rather than the mean.
The last line in the two functions is the only thing that is different in the body of the functions.
In NameOfFunction, the last line is:
ReturnVariable = -1 * (nanmedian([vect1]));
%note: I added the -1 multiplication to search for the maximum rather than minimum
The last line in StdConstraintFunction is:
ReturnVariable = (std([vect1]) - stdMax);
ceq = [];
%ceq is a required variable that is supposed to return the equality non-linear
%constraint; here it is blank because I don't have one. The optimization
%would produce an error if I exclude it
and my problem setup is:
problem = createOptimProblem('fmincon','x0',xstart,'objective',h,'Aeq',Aeq,'beq',beq,'options',options,'lb',lb,'ub',ub,'nonlcon',StdConstraint);
#Werner: If you want to post this as the answer to the question, I will gladly accept it as the official answer. Thanks so much for all your help!
Solving optimizations problems with non-linear conditions depending only on variables being optimized
Using matlab fmincon documentation:
x = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon,options)
Where:
nonlcon:The function that computes the nonlinear inequality constraints c(x)≤ 0 and the nonlinear equality constraints ceq(x) = 0. nonlcon accepts a vector x and returns the two vectors c and ceq. c is a vector that contains the nonlinear inequalities evaluated at x, and ceq is a vector that contains the nonlinear equalities evaluated at x. nonlcon should be specified as a function handle to a file or to an anonymous function, such as mycon:
x = fmincon(#myfun,x0,A,b,Aeq,beq,lb,ub,#mycon)
where mycon is a MATLAB function such as
function [c,ceq] = mycon(x)
c = ... % Compute nonlinear inequalities at x.
ceq = ... % Compute nonlinear equalities at x.
If the gradients of the constraints can also be computed and the GradConstr option is 'on', as set by
options = optimoptions('fmincon','GradConstr','on')
then nonlcon must also return, in the third and fourth output arguments, GC, the gradient of c(x), and GCeq, the gradient of ceq(x). GC and GCeq can be sparse or dense. If GC or GCeq is large, with relatively few nonzero entries, save running time and memory in the interior-point algorithm by representing them as sparse matrices. For more information, see Nonlinear Constraints.
So, what you will need to do in order to add the non linear constrain is to use the nonlcon function that will return c with the standard deviation from w's values. It may be accomplished using an anonymous function:
nonlcon = #(x) std(x) - std_lim;
which means std(x) <= std_lim, where x are the variables passed to matlab to be optimized, in this case the wstart variable, but at the kth iteration. Of course, instead of std you may use whatever you may want, i.e x(1)^2 + x(2)^3 - sin(x(3)), supposing you have three variables being optimized.
And then change your code to:
problem = createOptimProblem('fmincon','x0',wstart,'objective',h,...
'Aeq',Aeq,'beq',beq,'options',options,'lb',lb,'ub',ub,'nonlcon',nonlcon);
Note: If you don't have one or more of the above extra variables, i.e. linear lower boundary lb, just don't add it to createOptimProblem.
Solving optimizations problems with non-linear conditions also depending on variables that are not being optimized
In this particular problem, the problem variables to be optimized (w) are not the only variables needed to calculate the standard deviation as noticed by #Mr. Kinn, so there is a need to feed the matlab non linear conditions function with extra variables that are not being optimized in each iteration. In order to do so, we alter our anonymous function handle to:
StdConstraint = #(w)StdConstraintFunction(w(1),w(2),w(3),aa, bb, cc, dd, ee, ff, gg,stdMax);
which is a function handle with one input variable, called w, fed via matlab internal code with the variables being optimized. In this presented solution, this variable is fed as three arguments to a matlab file function called StdConstrainFunction, which will also receive variables aa,bb,cc,dd,ee,ff,gg,stdMax from the environment where the StdConstraint handle function was created, so they will not be modified by the matlab internal routines during optimization.
There, the variables are used to calculate the non linear condition to be respected, remembering that, as said by the matlab documentation, the values returned by the non linear condition function must be two: c,ceq. The first returned output, c, are the conditions which must be lesser than zero when they are being respected, o.c. when out of bounds. The second output ceqare the non linear conditions equation that must be respected.
You may adapt this particular solution for your problem, just by changing the arguments passed to the handle function used as non linear constrain.
Consider also seeing this question.