I have an optimization problem and I am using or-tools cp_sat solver. The number of variables is around 3500 (all boolean) but the number of constraints is huge (~750000). Out of 3500 variables, ~3000 are directly dependent on the other 500.
There are 2 scenarios I tested:
With a simple objective function depending on ~3000 constraint variables.
With a complex objective function depending on ~3000*3000 new variables, where each new variable is pairwise logical_and of the variables in (1).
For each case, we seed the solver with hints for ~500 variables.
For 1, it cannot find an optimal solution in reasonable time. After around 30-45 minutes of runtime, the improvement to the objective function is negligible, but the solutions are satisfactory.
For 2, behavior is weird. Around half of the time, it claims that the problem is INFEASIBLE, half of the time, claims that it found OPTIMAL solution, but only returns back the solution implied by the hints. Only rarely (less than a couple percent of the runs), it does some optimization and returns FEASIBLE.
In addition, case 1 uses around 4-6 GB of memory but case 2 uses 100-120 GB of memory.
Is the behavior in case 2 expected? How should I approach debugging this?
For case 2, the problem become very big. You are creating 9M Boolean variables.
Are you using multithreading ?
Can you try reducing the size of the model and see if this is still flaky ?
Is the problem creation deterministic ?
Are you using large coefficient ? Is it possible you are hitting an integer overflow error ?
Thanks
Related
I have a fairly large scale optimization problem although the problem itself is fairly simple. It is just quadratic + linear objective, with linear constraints. So the problem is solvable with cplexqp. The scale of the problem is around 1300 variables, but I need to solve ~200 independent problems.
If I just loop over 200 times and call cplexqp as usual, it takes about 16 minutes to solve all the problems. I considered using parallel computing, so I changed the loop to parfor, and it now takes around 14 minutes. I would have thought we would get much bigger speedup factor, considering that we have 12 cores and 12 workers.
I made sure that the parallel worker is already initialized (so MATLAB does not have to spend time initializing them). I also verified that all 12 worker threads were active in task manager, and they all were using non trivial amount of CPU each.
My question is: do you think cplexqp has a locking mechanism, as in it can't be called with more than one problem at a given time (from different threads?) What if I have different MATLAB processes? (For example I can save the inputs to a file, and start up several MATLAB sessions to consume the file and each session would know which index of problems to solve).
16 minutes is not bad, but we may need to do this several times a day (with potentially different inputs), so I was wondering if we can speed up the process even more.
TIA
The problem is that by default CPLEX will use all cores on your machine to solve one problem. So if you attempt to solve multiple problems in parallel then you are heavily oversubscribing the CPUs. This is likely to result in an overall slowdown.
So you should carefully select how many models you solve in parallel and how many cores you allow for each solve. If you use parfor then you should use the Cplex.Param.threads parameter to limit he number of cores for a single solve, or alternatively, select the simplex algorithm to solve your QPs.
Whether this whole parallelization gives you an overall speedup depends on how much slowdown you will observe for the individual models by limiting the thread counts.
Some general Modelica advice?
We've built a model with ~2000 equations and three vectors of input from measured data. Using OpenModelica, attempts at simulation have begun to hang in the translation stage (which runs for hours where it used to take less than a minute) and now I regularly "lose connection to omc.exe." Is there perhaps something cumulative occurring that's degrading translation/compilation performance?
In general, are there any good rules of thumb for keeping simulations lighter and faster? I realize that, depending on the couplings, additional equations could be exponentially increasing the size of the resulting system of equations - could this be a problem?
Thanks for your thoughts!
It shouldn't take that long. Seems like a bug.
You can report this bug here:
https://trac.openmodelica.org/OpenModelica (New Ticket).
If your model is public you can post it there, if not you can contact the OpenModelica team privately.
I did some cleaning in the code; and got the part that repeats 12x (the module) down to ~180 equations; in the process I reduced the size of my input vectors (and also a 2D look-up table the module refers to) by quite a bit - they're both down to a few hundred values. It's working now--simulations run in reasonable time, a few minutes each.
Since all these tables were defined within Modelica functions (as you pointed out, Mr. Tiller) perhaps shrinking them helped to improve the performance. I had assumed that all that data just got spread out in a memory array, without going through any real processing, but maybe that's not the case...time to know more about what's going on under the hood in this environment (as always).
Thanks for the help!
I am using MATLAB gamultiobj optimization
as I have 6 to 12 objective functions; the gamultiobj function inefficiently handling the problem, always terminated because the number of generations exceeded, not because the changes of the objective functions become smaller
I looked at the gamultiobj options documentations, but it didn't help
http://www.mathworks.com/help/gads/examples/multiobjective-genetic-algorithm-options.html
1- how can I increase the capability of gamultiobj function to handle this number of objective functions?
2- are there a better way at all (using MATLAB)?
Well,
this is my update:
1- I increased the number of generations, the population size, and assigned proper initial population using the common ga options, it worked better (I didn't know that they are working with gamultiobj too, but I knew, it isn't stated anywhere in the documentation explicitly).
2- after running and inspecting the results I realized that gamultiobj can handle many objective functions efficiently providing that they are independent. As long as the objective functions are strongly dependent (which is the case of my problem, unfortunately) the gamultiobj solver's efficiency dramatically decreases.
thanks !
You should increase the number of generations, possibly play with the options such as crossover, mutation, the constraint bounds in which you're going to get the solution.
The bounds are to specified correctly. and the initial population is also pretty much needed to get it to the correct set of parameters that you want to optimize
Is it a good practice to manually set numbers with large negative exponential like 1e-300 to zero to avoid underflow in Matlab?
If not how can we avoid harm of underflow when implementing functions like log(1+exp(x))?
Typically, you get trouble when adding very large and very small values, because this can lead to a high relative error. Get rid of this summation (1+exp(x)), it quickly runs out of the range of double values when x is large.
log(1+exp(x))
log(1+1/exp(x))*exp(x))
log(1+1/exp(x))+log(exp(x))
log(1+1/exp(x))+x
An alternative is the use of vpa:
log(1+exp(vpa(10^6)))
very slow, but you get a result with the configured precision.
I never saw a case where manually setting small values to zero was a good solution, typically comparing with a tolerance is better.
I've read some similar posts, while none of them actually tackled my problem.
I need to do a series of multiplication-similar operations for A, B, specifically calculating kernel matrices, on Windows Platform. While, the problem is both of A, B could be really large, let us say, 20000-by-360000. While, my server can only provide 96 GB memory. It may seem infeasible to have them in memory at the same time and do the calculation. So is there any good way to efficiently handle such a large multiplication? Btw, The size of result, which is 20000-by-20000, is much less than the multiplier and can fit in the memory properly.
Because I do it on Windows, it may be not feasible to call functions like mmap2.
I wonder whether converting them into sparse matrix is a good option. However, it may heavily depend on the properties of data.
Another solution I've come up with is to partition the origin matrix into blocks. Then do the calculation block-by-block.
Is there any other better solution? Any practical suggestions would be really appreciated.
Best regards,
Peiyun
If I where you I'd look into the block processing function:
B = blockproc(filename,[M N],fun)
and use the Destination parameter to allow saving the results without overflowing your memory.