I am using fmincon to curve fitting by minimizing residual sum of squares. When I do not have very many data points, fmincon usually finds local minima that are not close the the global minima which will lead to a good fit. Is there a way to use a gradient scanning method with fmincon to avoid these local minima?
Since your problem has local minima, these simple optimization procedures will result in different answers for different initial condition. Try initializing with a reasonable guess, or initialize with multiple random values and choose the one with smallest error.
We can help you better, if you fully describe your problem (What your curve is, and possibly a code snippet.)
Related
I am performing a numerical optimization where I try to find the parameters of a statistical model that best match certain moments of the data. I have 6 parameters in total I need to find. I have written a matlab function which takes the parameters as input and gives the sum of squared deviations from the empirical moments as output. I use the fminsearch function to find the parameters and it gives me a solution.
However, I am unsure if this is really a global minimum. What type of checks I could do to ensure the numerical solution is correct? Plotting the function is challenging due to high dimensionality. Any general advice in solving this type of problem is also appreciated.
You are describing the difficulties of a global optimization problem.
As mentioned in one of the comments, fminsearch() and related function fminunc() will return a local minimum. It provides no guarantee that you will get a global minimum.
A simple way to check if the answer you get really is a global minimum, would be to run the function multiple times from various starting points. If the answer all converges to the same value, it might be a global minimum. If you find an answer with lower error values, then the last answer was not the global minimum.
The only way to be perfectly sure that you have the global minima, is to know whether or not your function is convex (i.e. your function has only a single minima.) This will have to be done analytically.
If it is not possible to be done analytically, there are many global optimization methods you may want to consider, including some available as this MATLAB toolbox.
Is it possible to force MATLAB's fmincon to generate candidate points from a specific set of points that I supply? I ask this because if this were possible, it would greatly reduce computation complexity, and knock out a couple of constraints from the optimization.
Edited to add more info: Essentially, I'm looking for a set of points in a high dimension space which satisfy a particular function (belonging to that space), along with a couple of other constraints. The objective function minimizes the total length of the path formed by these points. Think generalized geodesic problem, attempted via non-linear optimization. While I manage to get near a solution while using fmincon, it is painfully slow and is prone to get stuck in local minima. I already have a sizeable set of data-points which satisfy this high-dimensional function, and have no compulsion that this new set of points not belong to this pre-existing set.
I am trying to solve a "linearized" linear-system-of-equations, which requires two parameters to be estimated by iteration because of linearization. The actual problem is nonlinear actually, but using fourier series method, it iss linearized.
I have been solving linear system by just matrices and SVDs which takes not much time but these matrices depend on the two parameters that are to be iteratively solved. At the end I just need to make sure that one of the parameters I solve iteratively matches the response I get in the system. This is the criteria to be minimized.
I have been using "fmincon" and "multi-start" to solve for two parameters and I get some results, but it is taking longer than what I expect. There is local minima issue too, so I had to include "multi-start".
Anyone has an idea if any other method would be easier to solve this problem?
I really appreciate it.
A global optimization method that one may use is Simulated annealing.
May be MATLAB has a relevant routine.
There is free Simulated annealing software that you may also try.
I got an improvement in my problem, and I just replied it in comments but I think it is worth putting it in here since what I did emerged something unexpected:
So I ran a monte carlo sim for two variables to be iteratively solved, and plotted how the error changes with respect to input variables. I realized that there are tons of local minima in the error of the response and that's why fmincon was not able to solve itself because it was quickly jumping into one of those local minima holes, and I needed a very refined multi-start for fmincon so that I could get global minimum. This is very interesting observation because I wasn't expecting that rough error distribution with respect to two parameters.
Is there any efficient solver/optimizer in matlab that you know of, to get the global minimum in cases where there are many local minima? Or any other method?
Thanks,
I fitted a computational model with scikit's gaussian process. I would like to find the minimum of the fitted gaussian process.
Davoud
A gaussian process (GP) is a probability distribution over an infinite number functions with an infinite number minima. That is the beauty of a GP, although perhaps in your view a downfall.
What I believe you mean to say is what is the minima of the mean function of a GP. This, unfortunately, is not something we can write down in closed from and hence is probably why scikit would not have it built into the framework.
If you really want to find it I suggest taking a linespace over your input space to sample and try to find a minima (if your input space is low dimensional) or expand the expression for the mean function and differentiate with respect to your input space to try and find the extreme points and global minima (not much use when you have many observations).
I am trying to solve a system of non linear equations using fsolve; lets say
F(x;lambda) = 0, where lambda is a vector of parameters, and x the vector I want to solve for.
I am using Matlab's fsolve.
I have 2 values of the parameter lambda, that I want to solve the system for. For the one value of lambda I get a solution, which seems alright.
For the other value of lambda I get a solution again (matlab exits with a flag of 1. However I know this is not an actual solution For example I know that some of the dimensions of x have to be equal to each other, and this is not the case in the solution I get from fsolve.
I have tried both trust-region and the levenberg-marquardt algorithm, and I am not getting any better results. (explicitly enforcing those x's to be the same, still seems to give solutions that are not consistent with what I would be expecting from the properties of the system)
My question is: do the algorithms used by fsolve depend on any kind of stability of the system? Could it be that changing the parameter lambda in the second case I mention above, I make the system unstable, and could that make fsolve having a hard time to solve it correctly?
Thank you, George
fsolve isn't "failing" - as commented by jucestain, it's giving you a local minimum, which is not necessarily a global minimum. This is what it's designed to do.
To improve your chances of obtaining a global minimum you need to either:
Know that your initial guess is good
Run the optimisation several times with a grid of initial guesses, and pick the best result
Add constraints to prevent the solver straying into areas you know to have local minima
Modify your cost function to remove local minima
If you ever come across a non-linear solver that can guarantee a global minimum, do let us know!