I have this script in Matlab
struct = svmTraining(feature_train,class_final_train);
svmclassify(struct,feature_test);
but, after 5 seconds the following message appears
??? Error using ==> svmclassify at 117
An error was encountered during classification.
Out of memory. Type HELP MEMORY for your options.
Help me, Thanks
I was able to solve this same problem for myself by calling the svmclassify() function on successive subsets of the test data. For some reason it needs an enormous amount of memory if you give it a large array of test data.
So here is something that worked for me
numExemplars = size(testData,1);
chunkSize = 1000;
j=1:chunkSize:numExemplars;
classifications = zeros(numExemplars,1); %initialize
for i=1:length(j)-1;
index1 = j(i);
index2 = j(i+1)-1;
fprintf('classifying exemplars %d to %d\n', index1, index2 );
chunk = testData(index1:index2,:);
classifications(index1:index2) = svmclassify(SVM_struct,chunk);
end
% last bit of data
chunk = testData(j(end):numExemplars,:);
classifications(j(end):numExemplars) = svmclassify(SVM_struct,chunk);
The error means you do not have enough memory available on your machine to carry out the classification.
Firstly, try repeating the commands with a freshly started MATLAB, without creating any more variables than necessary, and with no other applications running.
If that doesn't work, then essentially you will need to either work with a smaller dataset, or get more memory for your machine.
Related
This is sort of a weird question but hopefully someone will be able to help me.
I have a matlab code where due to the parallelized nature of the code i need to work with struct arrays.
After running the parfor loop I want to transform those structure arrays into three-dimensional arrays.
At the moment I am using the following code:
for k = 1:nsim
ksim(:,:,k) = st(k).ksim;
Msim(:,k) = st(k).Msim;
Vsim(:,:,k) = st(k).Vsim;
Psim(:,:,k) = st(k).Psim;
end
clearvars st
However this seems to be extremely inefficient as momentarily matlab needs to double all the matrices thus almos doubling memory use.
Any smarter way of achieving this without increasing that much memory use?
I do not consider this as the answer you are looking for; But it is an improvement.
Define the new arrays and remove fields one by one. Since there seems to be three huge outputs, this will decrease the peak of memory usage to ~130% instead of 200%.
for k = 1:nsim
ksim(:,:,k) = st(k).ksim;
end
st = rmfield(st , 'ksim');
for k = 1:nsim
Msim(:,k) = st(k).Msim;
end
st = rmfield(st , 'Msim');
and so on.
I'm learning nn toolbox with matlab examples and i've got all time error
Out of memory. Type HELP MEMORY for your
options. Error in test2 (line 10) xTest = zeros(inputSize,numel(xTestImages));
Here is my simply code
% Get the number of pixels in each image
imageWidth = 28;
imageHeight = 28;
inputSize = imageWidth*imageHeight;
% Load the test images
[xTestImages, outputs] = digittest_dataset;
% Turn the test images into vectors and put them in a matrix
xTest = zeros(inputSize,numel(xTestImages));
for i = 1:numel(xTestImages)
xTest(:,i) = xTestImages{i}(:);
end
code is written according to
mathwork example (but im trying to do my own custom network). I reinstall matlab, make maximum java RAM storage, clean some disk space and delate rest of neural network. Still not working. Any ideas how to fix this problem?
As written above, the line:
xTest = zeros(inputSize,numel(xTestImages)); # xTestImages is 1x5000
would yield a matrix of size 28^2*5000= 3,920e6 elements. Every element has a double precision (8byte), hence the matrix would consume around 30mb...
You stated, that the command memory shows the following:
Maximum possible array: 29 MB (3.054e+07 bytes)
* Memory available for all arrays: 467 MB (4.893e+08 bytes)
** Memory used by MATLAB: 624 MB (6.547e+08 bytes)
Physical Memory (RAM): 3067 MB (3.216e+09 bytes)
So the first line shows the limitation for ONE single array.
So a few things to consider:
I guess clear all or quitting some other running applications does not improve the situation!?
Do you use a 64 or 32bit OS? And/or MATLAB 32/64 bit?
Did you try to change the Java Heap Settings? https://de.mathworks.com/help/matlab/matlab_external/java-heap-memory-preferences.html
I know this won't fix the problem, but maybe it will help you to keep on working in the meanwhile: You could create the matrix with single precision which should work for your testcase. Simply pass single as second option while creating the matrix.
Out of memory was created by Levenberg–Marquardt algorithm - it's create huge Jacobian matrix for calculations when data is big.
I am trying to create a piece of parallel code to speed up the processing of a very large (couple of hundred million rows) array. In order to parallelise this, I chopped my data into 8 (my number of cores) pieces and tried sending each worker 1 piece. Looking at my RAM usage however, it seems each piece is send to each worker, effectively multiplying my RAM usage by 8. A minimum working example:
A = 1:16;
for ii = 1:8
data{ii} = A(2*ii-1:2*ii);
end
Now, when I send this data to workers using parfor it seems to send the full cell instead of just the desired piece:
output = cell(1,8);
parfor ii = 1:8
output{ii} = data{ii};
end
I actually use some function within the parfor loop, but this illustrates the case. Does MATLAB actually send the full cell data to each worker, and if so, how to make it send only the desired piece?
In my personal experience, I found that using parfeval is better regarding memory usage than parfor. In addition, your problem seems to be more breakable, so you can use parfeval for submitting more smaller jobs to MATLAB workers.
Let's say that you have workerCnt MATLAB workers to which you are gonna handle jobCnt jobs. Let data be a cell array of size jobCnt x 1, and each of its elements corresponds to a data input for function getOutput which does the analysis on data. The results are then stored in cell array output of size jobCnt x 1.
in the following code, jobs are assigned in the first for loop and the results are retrieved in the second while loop. The boolean variable doneJobs indicates which job is done.
poolObj = parpool(workerCnt);
jobCnt = length(data); % number of jobs
output = cell(jobCnt,1);
for jobNo = 1:jobCnt
future(jobNo) = parfeval(poolObj,#getOutput,...
nargout('getOutput'),data{jobNo});
end
doneJobs = false(jobCnt,1);
while ~all(doneJobs)
[idx,result] = fetchnext(future);
output{idx} = result;
doneJobs(idx) = true;
end
Also, you can take this approach one step further if you want to save up more memory. What you could do is that after fetching the results of a done job, you can delete the corresponding member of future. The reason is that this object stores all the input and output data of getOutput function which probably is going to be huge. But you need to be careful, as deleting members of future results index shift.
The following is the code I wrote for this porpuse.
poolObj = parpool(workerCnt);
jobCnt = length(data); % number of jobs
output = cell(jobCnt,1);
for jobNo = 1:jobCnt
future(jobNo) = parfeval(poolObj,#getOutput,...
nargout('getOutput'),data{jobNo});
end
doneJobs = false(jobCnt,1);
while ~all(doneJobs)
[idx,result] = fetchnext(future);
furure(idx) = []; % remove the done future object
oldIdx = 0;
% find the index offset and correct index accordingly
while oldIdx ~= idx
doneJobsInIdxRange = sum(doneJobs((oldIdx + 1):idx));
oldIdx = idx
idx = idx + doneJobsInIdxRange;
end
output{idx} = result;
doneJobs(idx) = true;
end
The comment from #m.s is correct - when parfor slices an array, then each worker is sent only the slice necessary for the loop iterations that it is working on. However, you might well see the RAM usage increase beyond what you originally expect as unfortunately copies of the data are required as it is passed from the client to the workers via the parfor communication mechanism.
If you need the data only on the workers, then the best solution is to create/load/access it only on the workers if possible. It sounds like you're after data parallelism rather than task parallelism, for which spmd is indeed a better fit (as #Kostas suggests).
I would suggest to use the spmd command of MATLAB.
You can write code almost as it would be for a non-parallel implementation and also have access to the current worker by the labindex "system" variable.
Have a look here:
http://www.mathworks.com/help/distcomp/spmd.html
And also at this SO question about spmd vs parfor:
SPMD vs. Parfor
I am doing parallel computations with MATALB parfor. The code structure looks pretty much like
%%% assess fitness %%%
% save communication overheads
bitmaps = pop(1, new_indi_idices);
porosities = pop(2, new_indi_idices);
mid_fitnesses = zeros(1, numel(new_indi_idices));
right_fitnesses = zeros(1, numel(new_indi_idices));
% parallelization starts
parfor idx = 1:numel(new_indi_idices) % only assess the necessary
bitmap = bitmaps{idx};
if porosities{idx}>POROSITY_MIN && porosities{idx}<POROSITY_MAX
[mid_dsp, right_dsp] = compute_displacement(bitmap, ['1/' num2str(PIX_NO_PER_SIDE)]);
mid_fitness = 100+mid_dsp;
right_fitness = 100+right_dsp;
else % porosity not even qualified
mid_fitness = 0;
right_fitness = 0;
end
mid_fitnesses(idx) = mid_fitness;
right_fitnesses(idx) = right_fitness;
fprintf('Done.\n');
pause(0.01); % for break
end
I encountered the following weird error.
Error using parallel.internal.pool.deserialize (line 9)
Bad version or endian-key
Error in distcomp.remoteparfor/getCompleteIntervals (line 141)
origErr =
parallel.internal.pool.deserialize(intervalError);
Error in nsga2 (line 57)
parfor idx = 1:numel(new_indi_idices) % only assess the necessary
How should I fix it? A quick Google search returns no solution.
Update 1
The weirder thing is the following snippet works perfectly under the exactly same settings and the same HPC. I think there might be some subtle differences between them two, causing one to work and the other to fail. The working snippet:
%%% assess fitness %%%
% save communication overheads
bitmaps = pop(1, new_indi_idices);
porosities = pop(2, new_indi_idices);
fitnesses = zeros(1, numel(new_indi_idices));
% parallelization starts
parfor idx = 1:numel(new_indi_idices) % only assess the necessary
bitmap = bitmaps{idx};
if porosities{idx}>POROSITY_MIN && porosities{idx}<POROSITY_MAX
displacement = compute_displacement(bitmap, ['1/' num2str(PIX_NO_PER_SIDE)]);
fitness = 100+displacement;
else % porosity not even qualified
fitness = 0;
end
fitnesses(idx) = fitness;
%fprintf('Done.\n', gen, idx);
pause(0.01); % for break
end
pop(3, new_indi_idices) = num2cell(fitnesses);
Update 2
Suspecting [mid_dsp, right_dsp] = compute_displacement(bitmap, ['1/' num2str(PIX_NO_PER_SIDE)]); causes me trouble, I replace it with
mid_dsp = rand();
right_dsp = rand();
Then, it works! This proves that this is indeed caused by this particular line. However, I do have tested the function, and it returns two numbers correctly! Since the function returns value just as rand() does, I can't see any difference. This confuses me more.
I had the same issue and it came out that Matlab 2015 is reserving all necessary memory resources for each of the loops in the parfor resulting in memory break shortage.
The error message is tricky. After fine tuning the code in the loop and providing 120GB of RAM from the SSD through system setting in Pagefile in Windows 10, the parfor executed beautifully.
After working a while on my own similar code block, I've decided that this is actually a memory issue.
I'm using a 6 core 4GHz CPU and 8 gigs of RAM and seen this issue (on MATLAB 2014b) when I set the worker count high, and did not have any problems with low worker counts.
When I use 6 or more workers (which is not ideal I know), memory consumption is high and this error message pops out sporadically. Also I have seen various out of memory errors in my tests.
I havent seen the error when I use 5 or less workers thus far, and I'm pretty sure some memory limit (possibly inside a java code block) is causing this issue by preventing some of the results' integrity (or existance)
Hope you can resolve this issue by reducing the worker count.
I have one code like below-
W = 3;
i = 4;
s = fullfact(ones(1,i)*(W + 1)) - 1;
p2 = unique(sort(s(sum(s,2) == i,:),2),'rows');
I can run this code only upto "i=11" but i want to run this code for upto "i=25".When i run these code for i=12 it shows error message "Out of Memory".
I need to keep these code as it is.How can i modify these code for larger value of "i"?
Matlab experts need your valuable suggestion.
Just wanting to do silly things is not enough. You are generating arrays that are simply too large to fit into memory.
See that the size of the matrix s is a function of i here. size(s) will be 2^(2*i) by i. (By the way, some will argue it is a bad idea to use i as a variable, which is normally sqrt(-1), for such variables.)
So when i = 4, s is only 256x4.
When i = 11, s is 4194304x11. This array takes 369098752 bytes of space, so 370 megabytes.
When i = 25, the array will be of size
2^50*25
ans =
2.8147e+16
Multiply that by 8 to get the memory needed. Something like 224 petabytes of memory! If you have that much memory, then send me a few terabytes of RAM. You won't miss them.
Yes, there are times when MATLAB runs out of memory. You can get the amount of memory available at any point of time by executing the following:
memory
However, I would suggest follow one of the strategies to reduce memory usage available here. Also, you might want to clear the variables which are not required in every iteration by
clear variable_name