I am fairly new to Mixed Integer Linear Programming and I was hoping someone could clarify a performance question for me. Basically I am performing a calculation with about 34 decision variables and my calculation time is around 5 seconds. I would like to ideally get the calculation time down into the sub 1 second range.
Currently I am using the CBC solver & MATLAB, but as I understand it this is a single-threaded solver. Most of the MILP solvers I've seen seem to pride themselves on large project performance with 1k+ variables and knocking compute time down from days to hours but only a few of the expensive ones are even multi-threaded. Processor speed would seem to only go so far with such a problem so there has to be something that could be done on the software side.
In a situation like I have what factors play a role in the calculation time? In theory would a solution like Gurobi be able to ramp up and make a discernible difference over CBC on such a small problem?
When solving MIPs, multi-threading usually doesn't help that much, compared to other applications. One thing more sophisticated solvers do better than CBC is presolving. It may very well happen, that all decision variables can be eliminated/fixed in the root node. In general, it is not possible to predict the solving time of a MIP based on its size. The solving process is simply too complex.
To answer your question: I do suppose that you will see a significant speedup when trying another solver (Gurobi, Xpress, CPLEX, SCIP, etc.). It is not guaranteed though, and you might also witness a slowdown.
Most of the commercial MILP solvers would probably consider most problems with only 1000 variables to be tiny. It is very common to solve problems with hundreds of thousands or millions of variables. Note however, that there are also some really small problems that are considered to be very hard or still open in that nobody has solved them to a proven optimal solution. Basically, the time taken to solve these problems is enormously problem dependent.
If you want to see how fast another solver can solve your problem, try submitting it as e.g. an MPS file to the NEOS solver service at http://www.neos-server.org/neos/
Related
In Dymola, I often meet a nonlinear system initialization failure or maybe a stiff system that is hard to solve in the large thermo-fluid system, but for a simple system, there wouldn't be this kind of problem. My questions are:
So I am wondering how much is the largest capability of solving a nonlinear system model? For example, how many nonlinear equations I could include in my model at most?
Is there any setting in Dymola which allows increasing the capability of solving nonlinear system?
How could I decrease the number of nonlinear equations in the model without damage to the accuracy of the model?
These are pretty difficult questions to be answered in a generally valid fashion. Still I'll try to share some of my experience with Dymola and non-linear systems.
There is no hard number which will limit the size. It depends more on how strongly non-linear the equations are than on their number. I have simulated models with non-linear systems of size 150, which are pretty stable while others of size 10 can brake...
There are multiple perspectives to this
I have worked on some models that made the C-Compiler run out of memory during compilation. If you have this sort of problem, forcing 64Bit compilation by setting Advanced.CompileWith64=2 can help. Then you shouldn't run out of memory any more. This only refers to the size only.
Performance for non-linear systems can be improved by activating DAE-mode by setting Advanced.Define.DAEsolver=true. This does not work with all solvers though.
Additionally to the above it can help to set Advanced.MoveEquationsToDynamics=true, for which the manual states: "It forces the integrator to solve the nonlinear
equations each integrator step and thereby it also updates the initial guesses more often."
As mentioned by Erik, the homotopy()-operator can be very important as it helps the solver converging in case of difficult initialization.
This is very specific to the model. Decoupling can help, e.g. by splitting the system in smaller systems by adding energy storing elements/states. This can be done based on physics of the system and is the preferable solution if possible. As an (more artificial) alternative filter/delays can be added. Usually this has a negative effect on accuracy.
I very much agree with Markus's advice but would also like to remind you about Modelica's homotopy operator. A well-chosen simplified model can greatly help Dymola to initialize a model with a large and difficult non-linear system.
In general good initial guesses are very important when solving non-linear systems. Using homotopy is simply an implicit way to provide these good guesses.
When running the GlobalSearch solver on a nonlinear constrained optimization problem I have, I often get very different solutions each run. For the cases that I have an analytical solution, the numerical results are less dispersed than the non-analytical cases but are still different each run. It would be nice to get the same results at least for these analytical cases so that I know the optimization routine is working properly. Is there a good explanation of this in the Global Optimization Toolbox User Guide that I missed?
Also, why does GlobalSearch use a different number of local solver runs each run?
Thanks!
A full description of how the GlobalSearch algorithm works can be found Here.
In summary the GlobalSearch method iteratively performs convex optimization. Basically it starts out by using fmincon to search for a local minimum near the initial conditions you have provided. Then a bunch of "trial points", based on how good the initial result was, are generated using the "scatter search algorithm." Then there is some more convex optimization and rating of "how good" the minima around these points are.
There are a couple of things that can cause the algorithm give you different answers:
1. Changing the initial conditions you give it
2. The scatter search algorithm itself
The fact that you are getting different answers each time likely means that your function is highly non-convex. The best thing that I know of that you can do in this scenario is just to try the optimization algorithm at several different initial conditions and see what result you get back the most frequently.
It also looks like there is something called the 'PlotFcns' property which would allow you get a better idea what the functions the solver is generating for you look like.
You can use the ga or gamulti objective functions within the GlobalSearch api. I would recommend this. Convex optimizers wont be able to solve a non-linear problem. Even then Genetic Algorithms dont gaurantee the solution. If you run the ga and then use its final minimum as the start of your fmincon search then it should result in the same answer consistently. There may be better ones but if the search space is unknown you may never know.
I am constructing a finite volume model in Dymola which evolves in time and space. The spatial discretization is hard coded in the equations section, the time evolution is implemented with a term consisting of der(phi).
Is the time integration of Dymola always numerically stable when using a variable step size algorithm? If not, can I do something about that?
Is the Euler integration algorithm from Dymola the explicit or implicit Euler method?
The Dymola Euler solver by default is explicit (if an in-line sovler is not selected).
The stability of time integration is going to depend on your integrator. Generally speaking, implicit methods are going to be much better than explicit ones.
But since you mention spatial and time discretization, I think it is worth pointing out that for certain classes of problems things can get pretty sticky. In general, I think elliptic and parabolic PDEs are pretty safe to solve in this way. But hyperbolic PDEs can get very tricky.
For example, the Courant-Friedrichs-Lewy condition will affect the overall stability of the solution method. But by discretizing in space first, you leave the solver with information only regarding time and it cannot check or conform to the CFL condition. My guess is that a variable time step integrator will detect the error being introduced by not following the CFL condition but that it will struggle to identify the proper time step and probably also end up permitting an unacceptably unstable solution.
Are there any faster and more efficient solvers other than fmincon? I'm using fmincon for a specific problem and I run out of memory for modest sized vector variable. I don't have any supercomputers or cloud computing options at my disposal, either. I know that any alternate solution will still run out of memory but I'm just trying to see where the problem is.
P.S. I don't want a solution that would change the way I'm approaching the actual problem. I know convex optimization is the way to go and I have already done enough work to get up until here.
P.P.S I saw the other question regarding the open source alternatives. That's not what I'm looking for. I'm looking for more efficient ones, if someone faced the same problem adn shifted to a better solver.
Hmmm...
Without further information, I'd guess that fmincon runs out of memory because it needs the Hessian (which, given that your decision variable is 10^4, will be 10^4 x numel(f(x1,x2,x3,....)) large).
It also takes a lot of time to determine the values of the Hessian, because fmincon normally uses finite differences for that if you don't specify derivatives explicitly.
There's a couple of things you can do to speed things up here.
If you know beforehand that there will be a lot of zeros in your Hessian, you can pass sparsity patterns of the Hessian matrix via HessPattern. This saves a lot of memory and computation time.
If it is fairly easy to come up with explicit formulae for the Hessian of your objective function, create a function that computes the Hessian and pass it on to fmincon via the HessFcn option in optimset.
The same holds for the gradients. The GradConstr (for your non-linear constraint functions) and/or GradObj (for your objective function) apply here.
There's probably a few options I forgot here, that could also help you. Just go through all the options in the optimization toolbox' optimset and see if they could help you.
If all this doesn't help, you'll really have to switch optimizers. Given that fmincon is the pride and joy of MATLAB's optimization toolbox, there really isn't anything much better readily available, and you'll have to search elsewhere.
TOMLAB is a very good commercial solution for MATLAB. If you don't mind going to C or C++...There's SNOPT (which is what TOMLAB/SNOPT is based on). And there's a bunch of things you could try in the GSL (although I haven't seen anything quite as advanced as SNOPT in there...).
I don't know on what version of MATLAB you have, but I know for a fact that in R2009b (and possibly also later), fmincon has a few real weaknesses for certain types of problems. I know this very well, because I once lost a very prestigious competition (the GTOC) because of it. Our approach turned out to be exactly the same as that of the winners, except that they had access to SNOPT which made their few-million variable optimization problem converge in a couple of iterations, whereas fmincon could not be brought to converge at all, whatever we tried (and trust me, WE TRIED). To this day I still don't know exactly why this happens, but I verified it myself when I had access to SNOPT. Once, when I have an infinite amount of time, I'll find this out and report this to the MathWorks. But until then...I lost a bit of trust in fmincon :)
I find genetic algorithm simulations like this to be incredibly entrancing and I think it'd be fun to make my own. But the problem with most simulations like this is that they're usually just hill climbing to a predictable ideal result that could have been crafted with human guidance pretty easily. An interesting simulation would have countless different solutions that would be significantly different from each other and surprising to the human observing them.
So how would I go about trying to create something like that? Is it even reasonable to expect to achieve what I'm describing? Are there any "standard" simulations (in the sense that the game of life is sort of standardized) that I could draw inspiration from?
Depends on what you mean by interesting. That's a pretty subjective term. I once programmed a graph analyzer for fun. The program would first let you plot any f(x) of your choice and set the bounds. The second step was creating a tree holding the most common binary operators (+-*/) in a random generated function of x. The program would create a pool of such random functions, test how well they fit to the original curve in question, then crossbreed and mutate some of the functions in the pool.
The results were quite cool. A totally weird function would often be a pretty good approximation to the query function. Perhaps not the most useful program, but fun nonetheless.
Well, for starters that genetic algorithm is not doing hill-climbing, otherwise it would get stuck at the first local maxima/minima.
Also, how can you say it doesn't produce surprising results? Look at this vehicle here for example produced around generation 7 for one of the runs I tried. It's a very old model of a bicycle. How can you say that's not a surprising result when it took humans millennia to come up with the same model?
To get interesting emergent behavior (that is unpredictable yet useful) it is probably necessary to give the genetic algorithm an interesting task to learn and not just a simple optimisation problem.
For instance, the Car Builder that you referred to (although quite nice in itself) is just using a fixed road as the fitness function. This makes it easy for the genetic algorithm to find an optimal solution, however if the road would change slightly, that optimal solution may not work anymore because the fitness of a solution may have grown dependent on trivially small details in the landscape and not be robust to changes to it. In real, cars did not evolve on one fixed test road either but on many different roads and terrains. Using an ever changing road as the (dynamic) fitness function, generated by random factors but within certain realistic boundaries for slopes etc. would be a more realistic and useful fitness function.
I think EvoLisa is a GA that produces interesting results. In one sense, the output is predictable, as you are trying to match a known image. On the other hand, the details of the output are pretty cool.