Array of structures importing - memory prealocation problemMatLab - matlab

Question

I have a few .mat files, each of them has an array of structures (of unknown length) called DATA. I want to import all these structures in a single array, but I don't want to use this code:
FileNames = strcat('file',num2str((1:N)''),'.mat');
DATATemp = [];
for int = 1:length(FileNames)
load(FileNames(int,:));
DATATemp=[DATATemp DATA];
end
DATA = DATATemp;
because it does not prealocate the memory for the array.
Are there any clever ways of doing that?

Answer 1

If the length is short enough that you can over-allocate memory, you can do something like this: Pick an array size that is way larger than what you will ever see, and then trim it back down after you are done.
FileNames = strcat('file',num2str((1:N)''),'.mat');
DATATemp = zeros(1e6,1);
idx = 1;
for int = 1:length(FileNames)
load(FileNames(int,:));
idx_end = idx + length(DATA) - 1;
DATATemp(idx:idx_end) = DATA;
idx = idx_end + 1;
end
DATA = DATATemp(1:idx_end);
However, if you are talking about a LOT of data, or just want to cover all your bases, a more rigorous solution is to allocate in chunks
FileNames = strcat('file',num2str((1:N)''),'.mat');
CHUNK_SIZE = 1e6;
DATATemp = zeros(INIT_SIZE,1);
idx = 1;
for int = 1:length(FileNames)
load(FileNames(int,:));
idx_end = idx + length(DATA) - 1;
if idx_end > length(DATATemp)
DATATemp = [DATATemp zeros(INIT_SIZE ,1);
DATATemp(idx:idx_end) = DATA;
idx = idx_end + 1;
end
DATA = DATATemp(1:idx_end);
Just make sure that your CHUNK_SIZE is significantly larger than the size of a typical individual file. I picked 1e6 here. That's what I would pick if I was loading ~20 files with an average size of 1e5 each. That way, although I'm still concatenating more space, it's much less often. This may not be very clever, but I hope it helps.
Also note that if you are loading files from a network drive that will slow things down immensely.