I have to execute this for cycle:
load('Y');
X_test = ...;
Y_test = ...;
X_train = ...;
Y_train = ...;
for i=1:length(Y.Y)
if Y.Y(i,1) == l
current_test_data = [current_test_data; X_test(i,:)];
current_test_labes = [current_test_labes; Y_test(i,:)];
else
current_train_data = [current_train_data; X_train(i,:)];
current_train_labes = [current_train_labes; Y_train(i,:)];
end
end
But length(Y.Y) is 2300250 so this execution takes a long time. There is a faster way to do that?
What you are doing is indeed not great in terms of performance.
The first issue is the loop. Matlab does not handle them very fast; when possible, vectorized operations should be preferred as they are well optimized. For instance, it is much faster to execute A=B.*C than for ii=1:length(B), A(ii)=B(ii)*C(ii);end
The second issue is that you are concatenating arrays within the loop. current_test_data starts as a small array that grows over time. Each time some data is appended, memory needs to be reallocated. The data may have to be moved to another place in the memory. Since Matlab stores data in column major order, adding an extra row also means that all samples but the first column have to be moved (while adding an extra column is just appending data at the end). All of this conspires to make a terrible performance. With small arrays, this may not be noticeable; but as you start moving megabytes or more of data in memory at each iteration, performance will plummet
The usual solution, when the final size of the array is known, is to pre-allocate arrays, for instance current_test_data = zeros(expected_rows,expected_columns);, and put the data straight away where it belongs: current_test_data(jj,:) = some_matrix(ii,:);. No more memory allocation, no more memory moves, no more shuffling-samples-around.
But then, in your specific case, the solution lies with the first point: using vectorized notation is the solution. It will pre-allocate the arrays to the right size and copy data efficiently.
sel = Y.Y(:,1)==1; % Builds a logical vector
% Selects data based on logical vector
current_test_data = X_test(sel,:);
current_test_labes = Y_test(sel,:);
current_train_data = X_train(~sel,:);
current_train_labes = Y_train(~sel,:);
Related
I haven't found a good solution for this problem, which takes forever and seems to be mainly due to storage of the data in a cell array (as far as I see).
I process movie data in this format:
[data{1:4}] = deal(int16(randi([0 255],200,400,100))); %200px, 200px, 100 frames, 4 similar movies
Where data represents 4 different, but similar movies, in a cell array. Now I would like to take the average of the 4 variables data{1:4}, frame by frame. This is what I came up with:
for frame = 1:size(data{ind},3)
tmp = zeros(200,400,'int16');
for ind = 1:4
tmp = tmp + data{ind}(:,:,frame);
end
data_avg(:,:,frame) = tmp./4;
end
is there a more efficient (faster performing, without doubling the RAM usage) way to do this (I haven't found any)?
the fastest approach will be to do:
data_avg= (data{1}+data{2}+data{3}+data{4})/4;
no need for a for loop.
This is slower: mean(double(cat(4,data{:})),4); because matlabs mean has an overhead inefficiency. your for loop is in between.
More of a blue skies question here - if I have some code that is like
A = [1,2,3,4,5,6]; %input data
B = sort(A); %step one
C = B(1,1) + 10; %step two
Is there a line of code I can use to remove "B" to save memory before doing something else with C?
clear B
This will remove the variable B from memory.
See the documentation here for more info.
There is no need to assign each result to a new variable. For example, you could write:
A = [1,2,3,4,5,6]; %input data
A = sort(A); %step one
A = A(1,1) + 10; %step two
Especially if A is large, it is much more efficient to write A = sort(A) than B = sort(A), because then sort can work in-place, avoiding the need to create a secondary array. The same is true for many other functions. Working in-place means that the cache can be used more effectively, speeding up operations. The reduced memory usage is also a plus for very large arrays, and in-place operations tend to avoid memory fragmentation.
In contrast, things like clear B tend to slow down the interpreter, as they make things more complicated for the JIT. Furthermore, as can be seen in the documentation,
On UNIX® systems, clear does not affect the amount of memory allocated to the MATLAB process.
That is, the variable is cleared from memory, but the memory itself is not returned to the system.
As an aside, as #obchardon said in a comment, your code can be further simplified by realizing that min does the same thing as keeping only the first value of the result of sort (but much more efficiently).
As an example, I've put three operations in a row that can work in-place, and used timeit to time the execution time of these two options: using a different variable every time and clearing them when no longer needed, or assigning into the same variable.
N = 1000;
A = rand(1,N);
disp(timeit(#()method1(A)))
disp(timeit(#()method2(A)))
function D = method1(A)
B = sort(A);
clear A
C = cumsum(B);
clear B
D = cumprod(C);
end
function A = method2(A)
A = sort(A);
A = cumsum(A);
A = cumprod(A);
end
Using MATLAB Online I see these values:
different variables + clear: 5.8806e-05 s
re-using same variable: 4.4185e-05 s
MATLAB Online is not the best way for timing tests, as so many other things happen on the server at the same time, but it gives a good indication. I've ran the test multiple times and seen similar values most of those times.
I'm putting together a hypothesis tree type algorithm in MATLAB and it is being slowed terribly by memory issues. The profiler shows all time being spent just writing into arrays.
The algorithm keeps a list of hypotheses with information about them in an array of structs. The issue is related to 3D arrays (not big) within the hypothesis:
H(x).someInfo(a,b,c)
Each iteration, some hypotheses are discarded:
H = H(keepIndices);
And the ones that remain are expanded and updated:
Hin = H;
H(N*length(H)) = H(1); % Pre-alloc?
count = 0;
for x = 1:length(Hin)
for y = 1:N
count = count + 1;
H(count) = Hin(x);
... % Computations
H(count).someInfo(:,:,a) = M; % Much time spent here
end
end
The profiler indicates huge amounts of time spent just doing the write (note comment). "someInfo" is preallocated so is not itself growing dynamically, but it is getting copied around.
Can anyone suggest a way to achieve this type of functionality without getting crossways with inefficiencies in the way MATLAB deals with memory? Not blaming MATLAB, but its flexibility makes this harder than it would be in C++.
If the access pattern to someInfo is always the same, you could turn it into a cell array of 2D matrices. You'll find that
H(count).someInfo{a} = M;
is faster than
H(count).someInfo(:,:,a) = M;
because the array data is not copied over, only the reference to the data is.
...and if that is the case, you might want to do
H{count,a} = M;
Note that the fewer levels of indexing (you have 3!), the faster it is.
Any trick to avoid an out of memory error in matlab?
I am assuming that the reason it shows up is because matlab is very inefficient in using horzcat and actually needs to temporarily duplicate matrices.
I have a matrix A with size 108977555 x 25. I want to merge this with three vectors d, m and y with size 108977555 x 1 each.
My machine has 32GB ram, and the above matrice + vectors occupy 18GB.
Now I want to run the following command:
A = [A(:,1:3), d, m, y, A(:,5:end)];
But that yields the error:
Error using horzcat
Out of memory. Type HELP MEMORY for your options.
Any trick to do this merge?
Working with Large Data Sets. If you are working with large data sets, you need to be careful when increasing the size of an array to avoid getting errors caused by insufficient memory. If you expand the array beyond the available contiguous memory of its original location, MATLAB must make a copy of the array and set this copy to the new value. During this operation, there are two copies of the original array in memory.
Restart matlab, I often find it doesn't fully clean up its memory or it get's fragmented, leading to lower maximal array sizes.
Change your datatype (if you can). E.g. if you're only dealing with numbers 0 - 255, use uint8, the memory size will reduce by a factor 8 compared to an array of doubles
Start of with A already large enough (i.e. 108977555x27 instead of 108977555x25 and insert in place:
A(:, 4) = d;
clear d
A(:, 5) = m;
clear m
A(:, 6) = y;
Merge the data in one datatype to reduce total memory requirement, eg a date easily fits into one uint32.
Leave the data separated, think about why you want the data in one matrix in the first place and if that is really necessary.
Use C-code to do the data allocation yourself (only if you're really desperate)
Further reading: https://nl.mathworks.com/help/matlab/matlab_prog/memory-allocation.html
Even if you could make it using Gunther's suggestions, it will just occupy memory. Right now it takes more than half of available memory. So, what are you planning to do then? Even simple B = A+1 doesn't fit. The only thing you can do is stuff like sum, or operations on part of array.
So, you should consider going to tall arrays and other related big data concepts, which are exactly meant to work with such large datasets.
https://www.mathworks.com/help/matlab/tall-arrays.html
You can first try the efficient memory management strategies as mentioned on the official mathworks site : https://in.mathworks.com/help/matlab/matlab_prog/strategies-for-efficient-use-of-memory.html
Use Single (4 bytes) or some other smaller data type instead of Double (8 bytes) if your code can work with that.
If possible use block processing (like rows or columns) i.e. store blocks as separate mat files and load and access only those parts of the matrix which are required.
Use matfile command for loading large variables in parts. Perhaps something like this :
save('A.mat','A','-v7.3')
oldMat = matfile('A.mat');
clear A
newMat = matfile('Anew.mat','Writeable',true) %Empty matfile
for i=1:27
if (i<4), newMat.A(:,i) = oldMat.A(:,i); end
if (i==4), newMat.A(:,i) = d; end
if (i==5), newMat.A(:,i) = m; end
if (i==6), newMat.A(:,i) = y; end
if (i>6), newMat.A(:,i) = oldMat.A(:,i-2); end
end
Is there a way to rewrite my code to make it faster?
for i = 2:length(ECG)
u(i) = max([a*abs(ECG(i)) b*u(i-1)]);
end;
My problem is the length of ECG.
You should pre-allocate u like this
>> u = zeros(size(ECG));
or possibly like this
>> u = NaN(size(ECG));
or maybe even like this
>> u = -Inf(size(ECG));
depending on what behaviour you want.
When you pre-allocate a vector, MATLAB knows how big the vector is going to be and reserves an appropriately sized block of memory.
If you don't pre-allocate, then MATLAB has no way of knowing how large the final vector is going to be. Initially it will allocate a short block of memory. If you run out of space in that block, then it has to find a bigger block of memory somewhere, and copy all the old values into the new memory block. This happens every time you run out of space in the allocated block (which may not be every time you grow the array, because the MATLAB runtime is probably smart enough to ask for a bit more memory than it needs, but it is still more than necessary). All this unnecessary reallocating and copying is what takes a long time.
There are several several ways to optimize this for loop, but, surprisingly memory pre-allocation is not the part that saves the most time. By far. You're using max to find the largest element of a 1-by-2 vector. On each iteration you build this vector. However, all you're doing is comparing two scalars. Using the two argument form of max and passing it two scalar is MUCH faster: 75+ times faster on my machine for large ECG vectors!
% Set the parameters and create a vector with million elements
a = 2;
b = 3;
n = 1e6;
ECG = randn(1,n);
ECG2 = a*abs(ECG); % This can be done outside the loop if you have the memory
u(1,n) = 0; % Fast zero allocation
for i = 2:length(ECG)
u(i) = max(ECG2(i),b*u(i-1)); % Compare two scalars
end
For the single input form of max (not including creation of random ECG data):
Elapsed time is 1.314308 seconds.
For my code above:
Elapsed time is 0.017174 seconds.
FYI, the code above assumes u(1) = 0. If that's not true, then u(1) should be set to it's value after preallocation.