I have to process a batch of images (around 30000) in a for loop. For this I read one image with every execution of the loop.
When reaching a certain index (the exact value varies but usually it is around 2000) the progress slows dramatically (factor 2 or 3). How can this be?
Here is a minimal code example that shows this behavior:
imgFolder = [uigetdir, '/'];
files = dir(fullfile(imgFolder, '*.tiff'));
filenames = sort_nat({files.name});
imshow(imread([imgFolder, '/', filenames{1}]))
roi = roipoly;
meansRGB = zeros(3,length(files));
tic;
for i = 1:size(files,1)
img = imread([imgFolder, '/', filenames{i}]);
% for j=1:3
% a = regionprops(roi,img(:,:,j),'PixelValues');
% meansRGB(j,i) = mean(a.PixelValues);
% end
% show remaining time
dt = 500; % display remaining time every dt'th step
if(mod(i,dt)) == 0
disp(i)
elapsed = toc;
remainingSeconds = elapsed*(length(files)-i)/dt;
[hours, minutes, seconds] = sec2hms(remainingSeconds);
str = sprintf('remaining time: %d:%d:%d', hours, minutes, round(seconds));
disp(str);
tic;
end
end
Note that all the important parts that I thought would consume the most time are already commented out and are not the reason for the decelerating loop. Also there is plenty of RAM and processor cycles left, this shouldn't be the problem.
Swapping?
Could it be that the first few hundred images fill up your RAM?
Any other accesses to memory would trigger swapping and writing to the hard drive, which is easily orders of magnitude slower...
Is there any chance that imread() is leaving file handles open? (Though I'd be rather surprised if that were the case?)
You could maybe try sticking an fclose('all') in the loop to check?
Related
I created a MATLAB code using Psychtoolbox to make an experiment.
Everything works as I intended but it seems the initial loading of the experiment takes too long. The task is a simple yes/no response task whether the target word('probe') appeared in the previous set of word stimuli.
I put basic intro text as an image and then wait for any keypress to start the experiment but it will take about 40 seconds to actually begin the first trial after any keystroke. I want to make it work without any delay. It should start its first trial immediately after any keystroke.
I checked the timestops with GetSecs() on numerous positions in the code and it was not anything to do with loading stimuli or initial setting of the experiment before the for loop I attached below.
To make things look simpler, I changed some of the variables into actual numbers I used. I can gurantee that it is not due to large stimuli size since it is only 1500 words. Once the for loop starts, it goes smoothly but it takes 40 seconds to actually start the first trial so I think it is something to do with a specific function in the for loop or the way I built it.
Please let me know if anything is too vague or unclear. I will do my best to make things read better.
Edit: I minimalized the code leaving only the function names used in Psychtoolbox. I left the functions I used in between loops to let you know if they could cause any delay. It will not be possible to run this without Psychtoolbox installed so I guess you can briefly examine the structure of the code.
for trial = 1:250
for i = 1:6
DrawFormattedText();
Screen();
WaitSecs(0.5);
end
DrawFormattedText();
flipTime = Screen();
WaitSecs(0.5);
DrawFormattedText();
flipTime = Screen();
rt = 0;
resp = 0;
while GetSecs - flipTime < 3
clear keyCode;
RestrictKeysForKbCheck();
[keyIsDown,secs,keyCode] = KbCheck;
respTime = GetSecs;
pressedKeys = find(keyCode);
% ESC key quits the experiment
if keyCode(KbName('ESCAPE')) == 1
clear all
close all
sca
return
end
% Check for response keys
if ~isempty(pressedKeys)
for i = 1:2
if KbName(i) == pressedKeys(1)
resp = i;
rt = respTime - flipTime;
end
end
end
% Exit loop once a response is recorded
if rt > 0
break;
end
end
if rt == 0 || rt > 3 % 3 second limit for subjects to react to the probe stimuli
DrawFormattedText();
Screen();
WaitSecs(1);
end
Screen();
vbl = Screen();
WaitSecs(1);
% Record the trial data into output data matrix
respMat{1, trial} = trial;
respMat{2, trial} = resp;
respMat{3, trial} = rt;
end
I have some pretty massive data files (256 channels, on the order of 75-100 million samples = ~40-50 GB or so per file) in int16 format. It is written in flat binary format, so the structure is something like: CH1S1,CH2S1,CH3S1 ... CH256S1,CH1S2,CH2S2,...
I need to read in each channel separately, filter and offset correct it, then save. My current bottleneck is loading each channel, which takes about 7-8 minutes... scale that up 256 times, and I'm looking at nearly 30 hours just to load the data! I am trying to intelligently use fread, to skip bytes as I read each channel; I have the following code in a loop over all 256 channels to do this:
offset = i - 1;
fseek(fid,offset*2,'bof');
dat = fread(fid,[1,nSampsTotal],'*int16',(nChan-1)*2);
Reading around, this is typically the fastest way to load parts of a large binary file, but is the file simply too large to do this any faster?
I'm not loading that much data... the test file I'm working with is 37GB, for one of the 256 channels, I'm only loading 149MB for the entire trace... maybe the 'skip' functionality of fread is suboptimal?
System details: MATLAB 2017a, Windows 7, 64bit, 32GB RAM
#CrisLuengo's idea was much faster: essentially, chunking the data, loading each chunk and then splitting that out to separate channel files to save RAM.
Here is some code for just the loading part which is fast, less than 1 minute:
% fake raw data
disp('building... ');
nChan = 256;
nSampsTotal = 10e6;
tic; DATA = rand(nChan,nSampsTotal); toc;
fid = fopen('rawData.dat','w');
disp('writing flat binary file... ');
tic; fwrite(fid,DATA(:),'int16'); toc;
fclose(fid);
% compute the number of samples and chunks
chunkSize = 1e6;
nChunksTotal = ceil(nSampsTotal/chunkSize);
%% load by chunks
t1 = tic;
fid = fopen('rawData.dat','r');
dat = zeros(nChan,chunkSize,'int16');
chunkCnt = 1;
while 1
tic
if chunkCnt <= nChunksTotal
% load the data
fprintf('Chunk %02d/%02d: loading... ',chunkCnt,nChunksTotal);
dat = fread(fid,[nChan,chunkSize],'*int16');
else
break;
end
toc;
chunkCnt = chunkCnt + 1;
end
t = toc(t1); fprintf('Total time: %4.2f secs.\n\n\n',t);
% Total time: 55.07 secs.
fclose(fid);
On the other hand, loading by channel by skipping through the file takes about 20x longer, a little over 20 minutes:
%% load by channels (slow)
t1 = tic;
fid = fopen('rawData.dat','r');
dat = zeros(1,nSampsTotal);
for i = 1:nChan
tic;
fprintf('Channel %03d/%03d: loading... ');
offset = i-1;
fseek(fid,offset*2,'bof');
dat = fread(fid,[1,nSampsTotal],'*int16',(nChan-1)*2);
toc;
end
t = toc(t1); fprintf('Total time: %4.2f secs.\n\n\n',t);
% Total time: 1133.48 secs.
fclose(fid);
I'd also like to thank OCDER on the Matlab forums for their help: link
I'm looping in parallel and changing a variable if a condition is met. Super idiomatic code that I'm sure everyone has written a hundred times:
trials = 100;
greatest_so_far = 0;
best_result = 0;
for trial_i = 1:trials
[amount, result] = do_work();
if amount > greatest_so_far
greatest_so_far = amount;
best_result = result;
end
end
If I wanted to replace for by parfor, how can I ensure that there aren't race conditions when checking whether we should replace greatest_so_far? Is there a way to lock this variable outside of the check? Perhaps like:
trials = 100;
greatest_so_far = 0;
best_result = 0;
parfor trial_i = 1:trials
[amount, result] = do_work();
somehow_lock(greatest_so_far);
if amount > greatest_so_far
greatest_so_far = amount;
best_result = result;
end
somehow_unlock(greatest_so_far);
end
Skewed answer. It does not exactly solve your problem, but it might help you avoiding it.
If you can afford the memory to store the outputs of your do_work() in some vectors, then you could simply run your parfor on this function only, store the result, then do your scoring at the end (outside of the loop):
amount = zeros( trials , 1 ) ;
result = zeros( trials , 1 ) ;
parfor trial_i = 1:trials
[amount(i), result(i)] = do_work();
end
[ greatest_of_all , greatest_index ] = max(amount) ;
best_result = result(greatest_index) ;
Edit/comment : (wanted to put that in comment of your question but it was too long, sorry).
I am familiar with .net and understand completely your lock/unlock request. I myself tried many attempts to implement a kind of progress indicator for very long parfor loop ... to no avail.
If I understand Matlab classification of variable correctly, the mere fact that you assign greatest_so_far (in greatest_so_far=amount) make Matlab treat it as a temporary variable, which will be cleared and reinitialized at the beginning of every loop iteration (hence unusable for your purpose).
So an easy locked variable may not be a concept we can implement simply at the moment. Some convoluted class event or file writing/checking may do the trick but I am afraid the timing would suffer greatly. If each iteration takes a long time to execute, the overhead might be worth it, but if you use parfoor to accelerate a high number of short execution iterations, then the convoluted solutions would slow you down more than help ...
You can have a look at this stack exchange question, you may find something of interest for your case: Semaphores and locks in MATLAB
The solution from Hoki is the right way to solve the problem as stated. However, as you asked about race conditions and preventing them when loop iterations depend on each other you might want to investigate spmd and the various lab* functions.
You need to use SPMD to do this - SPMD allows communication between the workers. Something like this:
bestResult = -Inf;
bestIndex = NaN;
N = 97;
spmd
% we need to round up the loop range to ensure that each
% worker executes the same number of iterations
loopRange = numlabs * ceil(N / numlabs);
for idx = 1:numlabs:loopRange
if idx <= N
local_result = rand(); % obviously replace this with your actual function
else
local_result = -Inf;
end
% Work out which index has the best result - use a really simple approach
% by concatenating all the results this time from each worker
% allResultsThisTime will be 2-by-numlabs where the first row is all the
% the results this time, and the second row is all the values of idx from this time
allResultsThisTime = gcat([local_result; idx]);
% The best result this time - consider the first row
[bestResultThisTime, labOfBestResult] = max(allResultsThisTime(1, :));
if bestResultThisTime > bestResult
bestResult = bestResultThisTime;
bestIndex = allResultsThisTime(2, labOfBestResult);
end
end
end
disp(bestResult{1})
disp(bestIndex{1})
I run a lot of long simulations in Matlab, typically taking from a couple of minutes to a couple of hours, so to speed things up I decided to run the simulations simultaneously using a parfor loop.
arglist = [arg1, arg2, arg3, arg4];
parfor ii = 1:size(arglist, 2)
myfun(arglist(ii));
end
Everything worked just fine, except for one thing: the progress printing. Since each simulation takes a lot of time, I usually print progress using something like
prevlength = 0;
for ii = 1:tot_iter
% Some calculations here
msg = sprintf('Working on %d of %d, %.2f percent done', ii, tot_iter, ii/tot_iter);
fprintf(repmat('\b', 1, prevlength))
fprintf(msg);
prevlength = numel(msg);
end
but, as could be expected, when doing this inside a parfor loop, you get chaos.
I have googled a lot in search of a solution and have found a bunch of "parfor progress printers" like this one. However, all of them print the progress of the entire parfor loop instead of showing how far each of the individual iterations have come. Since I only have about 4-8 iterations in the parfor loop, but each iteration takes about an hour, this approach isn't very helpful to me.
The ideal solution for me would be something that looks like this
Working on 127 of 10000, 1.27 percent done
Working on 259 of 10000, 2.59 percent done
Working on 3895 of 10000, 38.95 percent done
Working on 1347 of 10000, 13.47 percent done
that is, each simulation gets one line showing how far it has run. I'm not sure though if this is possible at all, I can at least not imagine any way to do this.
Another way would be to do something like this
Sim 1: 1.27% Sim 2: 2.59% Sim 3: 38.95% Sim 4: 13.47%
that is, show all the progresses on the same line. To do this, you would need to keep track of what position on the line each simulation is to write on and write there, without erasing the other progresses. I can't figure out how this would be done, is this possible to do?
If there is some other solution to my problem (showing the progress of each individual iteration) that I haven't thought of, I'd be happy to hear about it.
Since this is the first time I ask a question here on SO it is quite possible that there is something that I have missed; if so, please feel free to comment below.
Edit
After receiving this answer, I thought that I should share how I used it to solve my problem since I didn't use it exactly as in the answer, in case someone else encounters the same problem.
Here is a small test program with basically the same structure as my program, making use of the progress bar (parfor_progress) mentioned in the answer:
function parfor_progress_test()
cpus = feature('numCores');
matlabpool('open', cpus);
cleaner = onCleanup(#mycleaner);
args = [1000, 1000, 1000, 1000];
m = sum(args);
parfor_progress(m);
parfor ii = 1:size(args,2)
my_fun(args(ii));
end
parfor_progress(0);
end
function my_fun(N)
for ii = 1:N
pause(rand*0.01);
parfor_progress;
end
end
function mycleaner
matlabpool close;
fclose all;
end
Simple Progress Bar
Something like a progress bar could be done similar to this...
Before the parfor loop :
fprintf('Progress:\n');
fprintf(['\n' repmat('.',1,m) '\n\n']);
And during the loop:
fprintf('\b|\n');
Here we have m is the total number of iterations, the . shows the total number of iterations and | shows the number of iterations completed. The \n makes sure the characters are printed in the parfor loop.
Progress Bar and Percentage Completion
Otherwise you could try this: http://www.mathworks.com/matlabcentral/fileexchange/32101-progress-monitor--progress-bar--that-works-with-parfor
It will display a progress bar and percentage completion but can be easily modified to just include the percentage completion or progress bar.
This function amends a character to a file on every iteration and then reads the number of characters written to that file which indicates the number of iterations completed. This file accessing method is allowed in parfor's.
Assuming you correctly add the above to your MATLAB path somehow, you can then use the following:
arglist = [arg1, arg2, arg3, arg4];
parfor_progress(size(arglist, 2)); % Set the total number of iterations
parfor ii = 1:size(arglist, 2)
myfun(arglist(ii));
parfor_progress; % Increment the progress counter
end
parfor_progress(0); % Reset the progress counter
Time to Completion and Percentage Completion
There is also a function called showTimeToCompletion() which is available from: https://www.soundzones.com/software/sound-zone-tools/
and works alongside parfor_progress. This function allows you to print a detailed summary of the progress of a parfor loop (or any loop for that matter) which contains the start time, length of time running, estimated finish time and percentage completion. It makes smart use of the \b (backspace) character so that the command window isn't flooded with text. Although not strictly a progress 'bar' it is perhaps more informative.
The third example in the header of the function file,
fprintf('\t Completion: ');
showTimeToCompletion; startTime=tic;
len=1e2;
p = parfor_progress( len );
parfor i = 1:len
pause(1);
p = parfor_progress;
showTimeToCompletion( p/100, [], [], startTime );
end
outputs the following to the command window:
Completion: 31.00%
Remaining: 00:00:23
Total: 00:00:33
Expected Finish: 3:30:07PM 14-Nov-2017
Useful for estimating the completion of a running simulation, especially one that may take hours or days.
Starting in R2013b, you can use PARFEVAL to evaluate your function asynchronously and have the client display progress updates. (Obviously this approach is not quite as simple as adding stuff to your PARFOR loop). There's an example here.
The Diary property of the Future returned by PARFEVAL is updated continuously while processing, so that might also be useful if you have a small number of large tasks.
Starting in R2017a, you can use parallel.pool.DataQueue and afterEach to implement a waitbar for parfor, like so:
if isempty(gcp('nocreate'))
parpool('local', 3);
end
dq = parallel.pool.DataQueue;
N = 10;
wb = waitbar(0, 'Please wait...');
% Use the waitbar's UserData to track progress
wb.UserData = [0 N];
afterEach(dq, #(varargin) iIncrementWaitbar(wb));
afterEach(dq, #(idx) fprintf('Completed iteration: %d\n', idx));
parfor idx = 1:N
pause(rand());
send(dq, idx);
end
close(wb);
function iIncrementWaitbar(wb)
ud = wb.UserData;
ud(1) = ud(1) + 1;
waitbar(ud(1) / ud(2), wb);
wb.UserData = ud;
end
after exploring #Edric answer I found that there is an example in the Matlab documentation that exactly implements a waitbar for a pareval loop. Check help FetchNext
N = 100;
for idx = N:-1:1
% Compute the rank of N magic squares
F(idx) = parfeval(#rank, 1, magic(idx));
end
% Build a waitbar to track progress
h = waitbar(0, 'Waiting for FevalFutures to complete...');
results = zeros(1, N);
for idx = 1:N
[completedIdx, thisResult] = fetchNext(F);
% store the result
results(completedIdx) = thisResult;
% update waitbar
waitbar(idx/N, h, sprintf('Latest result: %d', thisResult));
end
% Clean up
delete(h)
I am running a parfor loop in Matlab that takes a lot of time and I would like to know how many iterations are left. How can I get that info?
I don't believe you can get that information directly from MATLAB, short of printing something with each iteration and counting these lines by hand.
To see why, recall that each parfor iteration executes in its own workspace: while incrementing a counter within the loop is legal, accessing its "current" value is not (because this value does not really exist until completion of the loop). Furthermore, the parfor construct does not guarantee any particular execution order, so printing the iterator value isn't helpful.
cnt = 0;
parfor i=1:n
cnt = cnt + 1; % legal
disp(cnt); % illegal
disp(i); % legal ofc. but out of order
end
Maybe someone does have a clever workaround, but I think that the independent nature of the parfor iterations belies taking a reliable count. The restrictions mentioned above, plus those on using evalin, etc. support this conclusion.
As #Jonas suggested, you could obtain the iteration count via side effects occurring outside of MATLAB, e.g. creating empty files in a certain directory and counting them. This you can do in MATLAB of course:
fid = fopen(['countingDir/f' num2str(i)],'w');
fclose(fid);
length(dir('countingDir'));
Try this FEX file: http://www.mathworks.com/matlabcentral/fileexchange/32101-progress-monitor--progress-bar--that-works-with-parfor
You can easily modify it to return the iteration number instead of displaying a progress bar.
Something like a progress bar could be done similar to this...
Before the parfor loop :
fprintf('Progress:\n');
fprintf(['\n' repmat('.',1,m) '\n\n']);
And during the loop:
fprintf('\b|\n');
Here we have m is the total number of iterations, the . shows the total number of iterations and | shows the number of iterations completed. The \n makes sure the characters are printed in the parfor loop.
With Matlab 2017a or later you can use a data queue or a pollable data queue to achieve this. Here's the MathWorks documentation example of how to do a progress bar from the first link :
function a = parforWaitbar
D = parallel.pool.DataQueue;
h = waitbar(0, 'Please wait ...');
afterEach(D, #nUpdateWaitbar);
N = 200;
p = 1;
parfor i = 1:N
a(i) = max(abs(eig(rand(400))));
send(D, i);
end
function nUpdateWaitbar(~)
waitbar(p/N, h);
p = p + 1;
end
end
End result :
If you just want to know how much time is left approximately, you can run the program once record the max time and then do this
tStart = tic;
parfor i=1:n
tElapsed = toc(tStart;)
disp(['Time left in min ~ ', num2str( ( tMax - tElapsed ) / 60 ) ]);
...
end
I created a utility to do this:
http://www.mathworks.com/matlabcentral/fileexchange/48705-drdan14-parforprogress