I have several "out of memory" problems using MATLAB. I don't understand exactly if Matlab can use (or not) all the ram memory of my computer. This is the problem: my computer has 4gb of ram memory and 2 gb for Swap memory (my OS is Linux/Ubuntu 12.10), but Matlab only uses up to 2.6 gb and then shows the warning: "out of memory".
Is it possible to fix this and allow Matlab to use all the "available" memory?
Thanks.
It sounds like you're running 32bit linux and or 32bit MATLAB.
If you allow for enough swap, a process can take up to its virtual memory address space worth of memory.
Generally for 32bit linux you're limited to 3gb of address space for any process (the last gb is kernel memory space). It's entirely possible, depending on usage patterns, that at 2.6gb the next request for memory can't complete because there isn't enough /contiguous/ memory to satisfy it. This is especially common when growing large arrays.
Upgrading to a 64bit version of linux/windows/macOS with 64bit MATLAB should solve this problem but even so, using 3gb+ of virtual address space on a system with 4gb of ram is probably going to start making things very slow.
Some googling brought up this:
MATLAB will use as much memory as your OS lets it use;
the only way to increase the amount of memory MATLAB can use is to reduce
the amount of memory occupied by other applications or to give the OS more
memory to partition out to the applications.
So no, there's no easy way to tell matlab to use more memory.
You either have to buy more memory, optimize your code, run your scripts/functions with less output to store at once or reduce memory usage by other procedures that are running.
Here are some helpful links though:
memory management functions
memory allocation
related discussion on the mathworks forum
Related
I would like to dedicate 8 GB of RAM instead of the full (12) for a very long computation, in order to use the remainder for another operation. Is it possible?
Is there maybe a MATLAB command that forces the maximum limit of memory usage?
I would like to work with 2 separate editors.
See here for a possible solution on "limit the memory of a process on windows":
Set Windows process (or user) memory limit
Matlab has no command to limit the memory usage, it will aquire as much memory as needed to do the computation. On some operating systems you can limit the memory usage, for example using ulimit on Linux. But be aware, when Matlab needs more than 8gb it will not be slow when reaching the limit, it will throw an exception and stop computing.
I was wondering why Matlab doesn't use swap, but instead throws the error "Out of memory"?
Shouldn't Matlab just slow down instead of throwing an "Out of memory"?
Is this Java related?
added:
I know "out of memory" means it's out of contiguous memory. Doesn't swap have contiguous memory, or? I'm confused...
It is not about MATLAB. What happens when you try allocate more memory than exists in your hardware is an OS specific behavior.
On Linux, by default the OS will 'optimistically' allocate almost anything you want, i.e. swap space is also counted as allocatable memory. You will get what you want - no OOM error, but slow computations with swap-allocated data. This 'feature' is called overcommit. You can change this behavior by modifying the overcommit settings in Linux (have a look e.g. here for a brief summary).
Overcommit is probably not the best idea to use this for solving larger problems in MATLAB, since the entire OS starts to work really slow. It can definitely not be compared to optimized 'out-of-core' implementations that consciously use the hard disk in computations.
This is how it is on Linux. I do not know how to change the memory allocation behavior on Windows, but I doubt you really want to do that. You need more RAM.
And you do confuse things - swap has nothing to do with contiguous memory. Memory allocated by the OS is 'virtual memory', which is contiguous regardless of whether the underlying physical memory can be mapped to contiguous pages.
Edit For transparent 'out-of-core' operations on large matriices using disk space as extra memory you might want to have look at VVAR fileexchange project. This class pretends to be a usual MATLAB class, but it operates on an underlying HDD file. Note that the usual array size limitations of MATLAB still apply.
I am suffering from the out of memory problem in mablab.
Is is possible to build a large memory system for matlab(e.g. 64GB ram)?
If yes, what do I need?
#Itamar gives good advice about how MATLAB requires contiguous memory to store arrays, and about good practices in memory management such as chunking your data. In particular, the technical note on memory management that he links to is a great resource. However much memory your machine has, these are always sensible things to do.
Nevertheless, there are many applications of MATLAB that will never be solved by these tips, as the datasets are just too large; and it is also clearly true that having a machine with much more RAM can address these issues.
(By the way, it's also sometimes the case that it's cheaper to just buy a new machine with more RAM than it is to pay the MATLAB developer to make all the memory optimizations they could - but that's for you to decide).
It's not difficult to access large amounts of memory with MATLAB. If you have a Windows or Linux machine with 64GB (or more) - it will obviously need to be running a 64-bit OS - MATLAB will be able to access it. I've come across plenty of MATLAB users who are doing this. If you know what you're doing you can build your own machine, or nowadays you can just buy a machine that size of the shelf from Dell.
Another option (depending on your application) would be to look into getting a small cluster, and using Parallel Computing Toolbox together with MATLAB Distributed Computing Server.
When you try to allocate an array in Matlab, Matlab must have enough contiguous memory the size of the array, and if not enough contiguous memory is available, you will get out of memory error, no matter how much RAM you have on your computer.
From my experience, the solution will not come from dealing directly with memory-related properties of your hardware, but from writing your code in a way that prevents allocation of too large arrays (cutting data to chunks, etc.). If you can describe your code and the task you try to solve, it might be possible to guide you in that direction.
You can read more here:http://www.mathworks.com/support/tech-notes/1100/1106.html
To increase the timeliness of my programs matlab, I got Windows 7 (64bit) and 64bit Matlab. and I've installed on a partition of the hard disk. Unfortunately, I was shocked to see that the execution time of the program is longer with 64bit Matlab. I do not know what's the problem. knowing that I have a core 2 Quad processor and 3GB of RAM.
In general, 64-bit does not make code faster. It just lets you access more memory. Your code will only speed up if it was memory constrained in a 32-bit process. In Matlab, this would usually cause Out Of Memory errors, not slowdowns. And since you only have 3 GB, you probably weren't hitting the 32-bit limit of 4 GB. So you probably shouldn't expect a speedup. A slowdown is surprising, though.
Are you using object-oriented Matlab, especially the old (pre-MCOS) style? There is a known bug in 64-bit Matlab on Windows that increases the overhead of method dispatch. OO code will run slower in 64-bit Matlab than 32-bit Matlab, with the slowdown increasing with the density of method calls. In my codebase (heavily OO), it's about a 2x slowdown. That's about the magnitude you're seeing.
See Is MATLAB OOP slow or am I doing something wrong?. (It's discussed tangentially there.)
You can still run 32-bit Matlab on 64-bit Windows. (Though it's not officially supported.) This arrangement does not suffer from the method dispatch slowdown, plus it gets 4 GB of virtual memory instead of the 2 GB it would under a 32-bit OS. (Probably only useful if you have >=4GB RAM.) If the 32-bit does run faster on the exact same machine, you should report it as a bug to MathWorks; the more users that mention it, the more likely it is to get fixed.
Matlab has a built-in profiler, which is a tool that tells you how many times each function is called and how much time it takes to execute. You should use the profiler to find out where the bottle-neck is, i. e. what parts of your program take the most time.
If you do this on both the 32-bit and the 64-bit platforms, you may find out why the 64-bit version is slower.
When I run a sample script in MATLAB, it says:
Out of memory. Type HELP MEMORY for your options.
When I type "memory", it reports:
Maximum possible array: 156 MB (1.638e+008 bytes) *
Memory available for all arrays: 740 MB (7.756e+008 bytes) **
Memory used by MATLAB: 1054 MB (1.105e+009 bytes)
Physical Memory (RAM): 3070 MB (3.219e+009 bytes)
* Limited by contiguous virtual address space available.
** Limited by virtual address space available.
Is there any way to get around this error? I'm using Windows XP x32 with MATLAB 2009a.
pack does a memory defragmentation. It might help you a bit as far as the contiguous memory available.
Remember, when MATLAB says it's out of memory, it means it's out of contiguous memory, so rebooting or restarting MATLAB may work.
But, I'd recommend optimizing your code and identifying how you're eating up so much memory. It could be an ill-designed recursive loop, or a bad indexing function (using doubles instead of logicals to index a huge matrix).
I practically lived with memory errors for a while since I was dealing with huge datasets, but there's always a workaround, ask specific questions and you'll be surprised.
Problem fixed.
Under Windows XP x32, I managed to almost double the amount of memory available to MATLAB by editing boot.ini to add the switch /3GB /USERVA=3030
[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect /3GB /USERVA=3030
Together with reducing our array sizes, this completely fixed the problem :)
I could have also fixed the problem by upgrading to Windows x64 or Windows 7 x64. This act also doubles the amount of memory available to MATLAB, even if you stick with MATLAB x32 and don't upgrade to MATLAB x64. Windows x64 is just far more memory efficient, even with systems that only have 4 GB of physical RAM installed.
Try this, it works well for me.
Go to Home -> Preference icon -> General -> Java Heap Memory -> Allocate what size of memory you want
In Preference window, go to "Workspace" (out of Java heap memory level) -> See "Matlab Array size limit"
Make sure uncheck the 'Limit the maximum array size to a percentage of RAM'. Because you want to extend memory
so we don't need this feature.
Done.
What are you attempting to allocate when it runs out of memory (OOM)? Do you have code to reproduce? A wide range of problems can cause out of memory errors.
To diagnose, use "dbstop if all error" to set a breakpoint on errors. The out of memory will trigger this, and you can use dbup, dbdown, and whos() to see what's consuming memory. Often an OOM is caused by a bad array size or index calculation, not just by big data structures. E.g. this will trigger an OOM in pretty much any 32-bit MATLAB.
>> x = 1;
>> x(2^30) = 2
??? Out of memory. Type HELP MEMORY for your options.
I faced a similar error while running an (old) C file in MATLAB using mex.
I found my solution at this issue on GitLab.
First, uncheck the option "Limit the maximum Array Size to a % of RAM" located under Preferences -> Workspace, as also indicated in this earlier answer.
Once applied, run your C file in the command window using
mex filename.c -compatibleArrayDims