Workaround for indexing limitations of matfile command - matlab

Question

I have a very large number of large data files. I would like to be able to categorize the data in each file, and then save the filename to a cell array, such that at the end I'll have one cell array of filenames for each category of data, which I could then save to a mat file so that I can then come back later and run analysis on each category. It might look something like this:
MatObj = matfile('listOfCategorizedFilenames.mat');
MatObj.boring = {};
MatObj.interesting = {};
files = dir(directory);
K = numel(files);
for k=1:K
load(files(k).name,'data')
metric = testfunction(data)
if metric < threshold
MatObj.boring{end+1} = files(k).name;
else
MatObj.interesting{end+1} = files(k).name;
end
end
Because the list of files is very long, and testfunction can be slow, I'd like to set this to run unattended overnight or over the weekend (this is a stripped down version, metric might return one of several different categories), and in case of crashes or unforeseen errors, I'd like to save the data on the fly rather than populating a cell array in memory and dumping to disk at the end.
The problem is that using matfile will not allow cell indexing, so the save step throws an error. My question is, is there a workaround for this limitation? Is there better way to incrementally write the filenames to a list that would be easy to retrieve later?

Answer 1

I have no experience with matfile, so I cannot help you with that. As a quick and dirty solution, I would just write the filenames to two different text-files. Quick testing suggests that the data is flushed to disk straight away and that the text-files are OK even if you close matlab without doing a fclose (to simulate a crash). Untested code:
files = dir(directory);
K = numel(files);
boring = fopen('boring.txt', 'w');
interesting = fopen('interesting.txt', 'w');
for k=1:K
load(files(k).name,'data')
metric = testfunction(data)
if metric < threshold
fprintf(boring, '%s\n', files(k).name);
else
fprintf(interesting, '%s\n', files(k).name);
end
end
%be nice and close files
fclose(boring);
fclose(interesting);
Processing the boring/interesting text files afterwards should be trivial. If you would also write the directory listing to a separate file before starting the loop, it should be pretty easy (either by hand or automatically) to figure out where to continue in case of a crash.

Answer 2

Mat files are probably the most efficient way to store lists of files, but I guess whenever I've had this problem, I make a cell array and save it using xlswrite or fprintf into a document that I can just reload later.
You said the save step throws an error, so I assume this part is okay, right?
for k=1:K
load(files(k).name,'data')
metric = testfunction(data)
if metric < threshold
MatObj.boring{end+1} = files(k).name;
else
MatObj.interesting{end+1} = files(k).name;
end
end
Personally, I just then write,
xlswrite('name.xls', MatObj.interesting, 1, 'A1');
[~, ~, list] = xlsread('name.xls'); % later on
Or if you prefer text,
% I'm assuming here that it's just a single list of text strings.
fid = fopen('name.txt', 'w');
for row=1:nrows
fprintf(fid, '%s\n', MatObj.interesting{row});
end
fclose(fid);
And then later open with fscanf. I just use the xlswrite. I've never had a problem with it, and it's not noticeably slow enough to detract me from using it. I know my answer is just a workaround instead of a real solution, but I hope it helps.