I am using Matlab 2014a under Windows7. I am running a loop that reads very big xlsx files (~40MB each). After I am done with a file I use 'clear' in order to free the memory taken by reading the file. The thing is that every once in a while the script is stops and giving me an error message:
Error using xlsread (line 247)
Error: Not enough storage is available to complete this operation.
I want to emphasis that after each time I am finishing with a file I clear all the variables, so each iteration only one file is loaded. If I restart Matlab the script may work again - making me believe that some how 'clear' command doesn't free all the memory that was allocated. is there a way to really free the memory that once was allocated in matlab?
thank you very much
Ariel
If restarting Matlab is not an option, the "pack" function should help. Otherwise you could also use matlab without the gui and write a shell script that starts and matlab for each file.
Related
I'm running a short code to open one by one a list of files and saving back only one of the variables contained in the files. The process seems to me much slower than I expected and getting slower with time, I don't fully understand why and how I could make it run faster. I always struggle with optimization. I'd appreciate if you have suggestions.
The code is the following (the ... substitute the actual path just for example):
main_dir=dir(strcat('\\storage2-...\Raw\DAQ5\'));
filename={};
for m=7:size(main_dir,1)
m
second_dir=dir([main_dir(m).folder '\' main_dir(m).name '\*.mat']);
for mm=1:numel(second_dir)
filename{end+1}=[second_dir(mm).folder '\' second_dir(mm).name];
for mmm=1:numel(filename)
namefile=sprintf(second_dir(mm,1).name);
load(string(filename(1,mmm)));
save(['\\storage2-...\DAQ5\Ch1_',namefile(end-18:end-4),'.mat'], 'Ch_1_y')
end
end
end
The original file is about 17 MB and once the single variable is saved it is about 6 MB in size.
The Matlab load function takes an optional additional argument to specify just a selected variable to read from the input file.
s = load('path/to/file.mat', 'Ch_1_y');
That way you don't have to spend time loading in all the other variables from those input .mat files that you're just going to immediately throw away.
And using save to save MAT-files over SMB shares can be slow. You might want to call save to write it to a temporary local file first, and then copy the completed file to the final destination. Sounds like more I/O, but it can actually be a net win, depending on your particular system and network. Measure it both ways to see if it's a win in your particular situation.
I can load a matrix from text file:
load mydata.txt
The problem is my matrix file is about 250Mb and after several such loads I have no memory to work with next files.
How could unload it and free resources for further use?
Use clear, or clearvars. By default, MATLAB will create a variable called mydata as a result of your statement, so
clear mydata
Find the variables in your workspace that contain the large data sets and in your script or from the console type
clear whateverVariableName
To clear all memory use
clear all
You can even right-click individual variables in he workspace editor and delete them using the IDE if you wish.
What you have to do clear mydata and then issue pack . The first command says to Matlab that the reference to the memory held for mydata is not needed anymore. The second command instruct the Matlab to free unused memory. If you don't issue pack, then memory will be deallocated when the Matlab memory manager decides to.
I'm using MATLAB and calling an .exe via the system command.
[status,cmdout] = system(command_s);
where command_s is a command string that is formatted earlier in my script to pass all the desired options to the .exe. The .exe would normally write to a .csv file via the > redirection operator in Windows/DOS. Instead, this output is going to cmdout where I use it later in the MATLAB script. It is working correctly and as expected. I'm doing it this way so that the process just uses memory and does not write a very large file to the disk, which would then have to be read from the disk and then deleted after I'm done with it. In the end, it saves a .mat file that's usually in hundreds of KB instead of 10s/100s of MBs as the .csv file would be (some unneeded data is thrown out in the end).
The issue I'm having is since I'm dealing with large files, the executable can take a significant amount of time. I typically have to wait about 2 minutes after executing this command. In the meantime, I have no feedback to know it is progressing and that my system hasn't froze. I know I could add the & symbol to the end of my string, command_s, and run MATLAB code while this is running in the background (or asynchronously as some would say), but that brings up an external window AND makes cmdout empty - so I cannot use the output - forcing me to sit there for 2 minutes wondering each time it executes.
Is there any way to run in the background AND get the stdout from the command?
Maybe you could try system(command_s,'-echo')?
I have a matrix cube which I load in my program to read data from. The size of this .mat file is 2.8 GB. I am not being able to load it with the error of 'running out of memory'. Is there a way to fix this?
You can use the matfile class to work on ranges within variables inside MatLab files. See
Load and save parts of variables in MAT-files
Here's some additional discussion that discloses that this feature is new with R2011b.
If the size of the data exceeds the available memory on your machine, then you are in trouble - this is unavoidable. However, if you only want certain variables inside the .mat file you can try to load just those variables using the
load(filename, variables)
version of the load function. It really depends on the contents of your .mat file. If the file is 2.8GB and you need ALL of the variables in the file and your machine does not have enough memory to cope, your only option is to buy more RAM.
EDIT Apparently this answer is incorrect if you are running R2011b and above as explained in the answer of Ben Voight
I have noticed that MATLAB (R2011b on Windows 7, 64 bit) tends to slow down if I am in debugging mode for a long period of time (e.g. 3 hours). I don't recall this happening on previous versions of MATLAB.
The slow down is small, but significant enough to have an impact on my productivity (sometimes MATLAB needs to wait for up to 1 sec before I can type on the command line or on the editor).
I usually spend hours on debugging mode (e.g. after stopping at a keyboard statement) coding full projects in this mode. I find working on debugging mode convenient to organically grow my code while inspecting my code anytime in execution time.
The odd thing is my machine has 16 GB of RAM and the total size of all workspaces while in debugging mode is usually less than 4 GB. I don't have any other large process running in the background, and my system reports ~8GB of free RAM.
Also, unfortunately MATLAB does not let me call pack from debugging mode; it complains with :
Warning: PACK can only be used from the MATLAB command line.
I have reproduced this behavior after restarting MATLAB, rebooting my system, and on different days. With this, my question/s are:
Has anybody else noticed this? Is there anything I could do to prevent this slowdown without exiting debugging mode?
Are there any technical notes or statements from Mathworks addressing this issue?
In case it matters, my code is on a network drive, so I added the following on my startup.m file, which should alleviate any impact on performance resulting from it:
system_dependent('RemoteCWDPolicy', 'None');
system_dependent('RemotePathPolicy', 'None');
system_dependent('DirChangeHandleWarn','Never');
I have experienced some similar issues. The problem ended up being that Mathworks changed how Matlab caches files. For some users, it is now storing data in the TMP folder as defined by the environment variables. This folder was being scanned by anti virus and causing a lot of performance problem. Of course, IT wouldn't let us exclude the TMP folder from scans. So we added a line to our start up script that changes the environment variable of TMP to some other location within an excluded folder.
You don't have to worry about changing the variable back or messing up other programs. When applications launch, they copy the environment variables into their own local instance of them. Any changes made to them only change the local copy of those variables, not the system copy.
Here is the function you will need.
setenv('TEMP', 'C:\TEMP');
I'm not sure if it was TMP or TEMP. Check your environment variables to be sure.
I am using MATLAB R2011 on linux 10, windows 7 (32 bit).
I experienced MATLAB slowing down while printing simple variables in command window.
It turned that there was one .m file loaded in my Editor.
It was a big file with 10000 lines. These lines were simple data that should have been saved as mat file. When i closed this file, the editor was back to its normal speed.