I am using both Cplex and Gurobi for an LP program whose inequality constraint matrix A can become truly large -- around 5 to 10GB. When I want to use one of those solvers, I have to create a separate struct with all the problem constraints. This means that I have the matrix A in my workspace, and the matrix A in my solver struct at the same time. Even if I clear it in my Workspace as fast as possible, there is still a time when both exist and my RAM is overloaded.
I am asking if there is some clever method to deliver the matrix A into the model without both existing at the same time. The only thing I can think of right now is delivering it in small chunks...
MATLAB using copy-on-write, or lazy copying. This means that, as long as you don't modify one of the copies, all copies of a matrix share the same data:
A = randn(10000);
B = A; % does not take up extra memory
myfunc(B);
function myfunc(matrix)
C = matrix; % does not take up extra memory.
For reference, see for example on Loren's blog and Undocumented Matlab.
Related
I have a matrix X, size 40-by-60000
while writing the SVM, I need to form a linear kernel: K = X'*X
And of course I would get an error
Requested 60000x60000 (26.8GB) array exceeds maximum array size preference.
How is it usually done? The data set is Mnist, so someone must have done this before. In this case rank(K) <= 40, I need a way to store K and later pass it to quadprog.
How is it usually done?
Usually kernel matrices for big datasets are not precomputed. Since optimisation methods used (like SMO or gradient descent) do only need access to a subset of samples in each iteration, you simply need a data structure which is a lazy kernel matrix, in other words - each time an optimiser requests K[i,j] you literally compute K(xi,xj) then. Often, there are also caching mechanisms to make sure that often requested kernel values are already prepared etc.
If you're willing to commit to a linear kernel (or any other kernel whose corresponding feature transformation is easily computed) you can avoid allocating O(N^2) memory by using a primal optimization method, which does not construct the full kernel matrix K.
Primal methods represent the model using a weighted sum of the training samples' features, and so will only take O(NxD) memory, where N and D are the number of training samples and their feature dimension.
You could also use liblinear (if you resolve the C++ issues).
Note this comment from their website: "Without using kernels, one can quickly train a much larger set via a linear classifier."
This problem occurs due to the large size of your data set, thus it exceeds the amount of RAM available in your system. In 64-bit systems data processing performs better than in 32-bit, so you'll want to check which of the two your system is.
I am facing the following problem: I have a system of 160000 linear equations with 160000 variables. I am going to write two programs on conjugate gradient method and steepest descent method to solve it. The matrix is block tridiagonal with only 5 non zero diagonals, thus it's not necessary to create and store the matrix. But I am having the following problem: when I go to the iterarion stepe, there must be dot product of vectors involved. I have tried the following commands: dot(u,v), u'*v, which are commonly used. But when I run the program, MATLAB told me the data size is too large for the memory.
To resolve this problem, I tried to decompose the huge vector into sparse vectors with small support, then calculate the dot products of small vectors and finally glue them together. But it seems that this method is more complicated and not very efficient, and it is easy (especially for beginners like me) to make mistakes. I wonder if there're any more efficient ways to deal with this problem. Thanks in advance.
In my code at each time step sparse matrix A is built as follows:
1< DX <120000
A = sparse(i,j,s,DX,DX,6*DX)
b = (1,DX)
The problem that I am dealing is a sort of discretization problem. I have maximum 120000 nodes. Each of these nodes are having special charachters and I choose only the ones that meet my criteria. the number of these chosen ones is DX and is completely dependent on the physical process.
I am using backslash in x = A\b. But as the size of A could become quite big, the computational time rises drastically (more than 10e5 time steps are having (DX > 6e4).
As far as I know, backslash operation is already well optimized in MATLAB but I would like to know:
Would it make sense to use codegen and convert the code to C?
Does any one know an alternative method instead of backslash, so that the computational time decreases (maybe an iterative method)?
While x = A\b works well for many systems, you can run into memory issues causing slowdowns. Alternately, MATLAB has a number of built in functions for iteratively solving Ax=b for sparse matrices, such as pcq(), bigcg(), cgs(), etc. See the MATLAB Documentation on interative methods for solving systems of linear equations
I have a linear program with order N^4 variables and order N^4 constraints. If I want to solve this in AMPL, I define the constraints one by one without having to bother about the exact coefficient matrices. No memory issues arises. When using the standard LP-solver in Matlab however, I need to define the matrices explicitly.
When I have variables with four subscripts, this will lead to a massively sparse matrix of dimension order N^4 x N^4. This matrix won't even fit in memory for non trivial problem sizes.
Is there a way to get around this problem using Matlab, apart from various column generation/cutting plane techniques? Since AMPL manages to solve it, I suppose they're either automating some kind of decomposition, or they somehow solve the LP without explicitly working with this sparse monster matrix.
Apart from sparse mentioned by m.s. you can also use AMPL API for MATLAB. It is especially useful if you already have an AMPL model and want to work with it from MATLAB.
Converting my comment into an answer:
MATLAB supports sparse matrices using the sparse command which allows you to build your constraint matrix without exceeding memory limits.
I understand how using vectorization in a language like MATLAB speeds up the code by removing the overhead of maintaining a loop variable, but how does the vectorization actually take place in the assembly / machine code? I mean there still has to be a loop somewhere, right?
Matlab 'vectorization' concept is completely different than the vector instructions concept, such as SSE. This is a common misunderstanding between two groups of people: matlab programmers and C/asm programmers. Matlab 'vectorization', as the word is commonly used, is only about expressing loops in the form of (vectors of) matrix indices, and sometimes about writing things in terms of basic matrix/vector operations (BLAS), instead of writing the loop itself. Matlab 'vectorized' code is not necessarily expressed as vectorized CPU instructions. Consider the following code:
A = rand(1000);
B = (A(1:2:end,:)+A(2:2:end,:))/2;
This code computes mean values for two adjacent matrix rows. It is a 'vectorized' matlab expression. However, since matlab stores matrices column-wise (columns are contiguous in memory), this operation is not trivially changed into operations on SSE vectors: since we perform the operations row-wise the data you need to load into the vectors is not stored contiguously in the memory.
This code on the other hand
A = rand(1000);
B = (A(:,1:2:end)+A(:,2:2:end))/2;
can take advantage of SSE instructions and streaming instructions, since we operate on two adjacent columns at a time.
So, matlab 'vectorization' is not equivalent to using CPU vector instructions. It is just a word used to signify the lack of a loop implemented in MATLAB. To add to the confusion, sometimes people even use the word to say that some loop has been implemented using a built-in function, such as arrayfun, or bsxfun. Which is even more misleading since those functions might be significantly slower than native matlab loops. As robince said, not all loops are slow in matlab nowadays, though you do need to know when they work, and when they don't.
And in any way you always need a loop, it is just implemented in matlab built-in functions / BLAS instead of the users matlab code.
Yes there is still a loop. But it is able to loop directly in compiled code. Loops in Fortran (on which Matlab was originally based) C or C++ are not inherently slow. That they are slow in Matlab is a property of dynamic runtime (they are also slower in other dynamic languages like Python).
Since Matlab has introduced a Just-In-Time compiler loop performance has actually increased dramatically - so the old guidelines to avoid loops are less important with recent versions than they once were.