I am running a small script with a parfor loop. The script starts with the followin line:
parfor i=1:length(vX)
fprintf('%d/%d\n',i,length(X));
...
So apparently I should see printouts right away. When I run it with a pool of two workers, opened by matlabpool(2), I see no printout. When I close the workers, keeping the parfor loop, I see printouts but only when I hit ctrl-c. When I change the parfor into a regular for I see the output. Note that I never saw the loop run to completion as it is quite long, but the printout is the second line the script and should happen immediately, unless there is some buffer-flushing issue I am not aware of in matlab.
What is happening??
I suspect that the delay is due to the overhead of setting up a parfor loop.
In my experience it can take several, up to 15, seconds just to initialize the loop. This is particularly true when the code in my a loop uses variables with large memory footprints, as the variables must be copied to the workers.
The copying of data between workers is done over the network and a basic network activity monitor should reveal this activity. I find it useful to determine how much of my parfor execution time is getting eaten up initializing the workers.
I use fprintf all the time in my parfor loops; however, I try to be creative about the generated output as the loops iterations are not sequentially.
If all you are looking for is a progress indicator check out: http://www.mathworks.com/matlabcentral/fileexchange/32101-progress-monitor-progress-bar-that-works-with-parfor
I had the same problem and tried for months to find a solution. But in the end I just resorted to using
disp(['text ',num2str(variable));
Related
I have an embarrassingly-parallel Monte Carlo code for which I am using parfor. Each parfor loop iteration does around 100 seconds of calculation into a temporary array, before adding that array to the master array, which is an reduction variable. I'm timing using a sliced variable.
Like this:
master_array=zeros(1000,1000,50)
iteration_times=zeros(500,1);
parfor i=1:500
iteration_start=datetime;
temp_array=zeros(1000,1000,50);
**do about 100 seconds of work building temp_array**
master_array=master_array+temp_array;
iteration_times(i)=seconds(datetime-iteration_start);
end
Sometimes when the code is run, but not all the time, the code takes far longer. When it does, it seems that some loop iterations are taking way way longer than they should, as shown in the graph iteration number vs runtime. (This is not caused by variations in the amount of 'work done' - the variation disappears without the parfor.) This was running with a pool of 4 workers.
The arrays are large, and can be 100MB or more. I don't seem to be running out of memory - the hard drive is not being used, and I get this problem still with smaller arrays.
'Slow' iteration times seem oddly quantised, like some loop interations are waiting for others to complete before completing themselves.
Any ideas what i can check?
When I have the parallel computing toolbox installed and use parfor in my code, MATLAB starts the pool automatically once it reaches the parfor loop. This however makes it difficult to debug at times, which is why I would like to prevent MATLAB from opening a pool in certain situations. So, how can I tell MATLAB not to open a pool? Obviously I could go through my code and remove all parfor loops and replace them with normal for loops, but this is tedious and I might forget to undo my changes.
edit: To specify, I ideally would like the parfor loop to behave exactly like a for when setting a control or variable or something. That is, I should for example also be able to place breakpoints in the for-loop.
Under Home->parallel->parallel preferences you can deselect the check box "Automatically create a parallel pool (if one doesn't already exist) when parallel keywords are executed." This makes all the parfor loops behave as a normal for loop.
I'll get back to you if I figure out a way to do this in the code as opposed to using the check box.
Update turns out it is indeed possible to change the settings through code, although I would not recommend this, as it involves changing MATLAB's preference file. This is taken from the Undocumented MATLAB blog by Yair Altman.
ps = parallel.Settings;
ps.Pool
ans =
PoolSettings with properties:
AutoCreate: 1
RestartOnClusterChange: 1
RestartOnPreferredNumWorkersChange: 1
IdleTimeout: 30
PreferredNumWorkers: 12
where you need to change the AutoCreate switch to 0.
As alternative I'd suggest wrapping everything inside your parfor in a function, thus calling
parfor 1:N
output = function(..)
end
Now modify your script/function to have a Parallel switch on top:
if Parallel
parfor 1:N
output = function(..)
end
else
for 1:N
output = function(..)
end
end
You can edit and debug the function itself and set your switch on top of your program to execute in parallel or serial.
As well as the normal syntax
parfor i = 1:10
you can also use
parfor (i = 1:10, N)
where N is the maximum number of workers to be used in the loop. N can be a variable set by other parts of the code, so you can effectively turn on and off parallelism by setting the variable N to 1 or 0.
Edit: to be clear, this only controls the number of workers on which the code is executed (and if N is zero, whether a pool is started at all). If no pool exists, the code will execute on the client. Nevertheless, the code remains a parfor loop, which does not have the same semantics as a for loop - there are restrictions on the loop code for parfor loops that do not exist for for loops, and there is no guarantee on the order in which the loop iterations are executed.
When you use parfor, you're doing more than just saying "speed this up please". You're saying to MATLAB "I can guarantee to you that the iterations of this loop are independent, and can be executed in any order, so you will be OK if you try to parallelize it". Because you've guaranteed that, MATLAB is able to speed things up by using different semantics than it would do for a for loop.
The only way to completely get for loop behaviour is to use for, and if you need to switch back and forth for debugging purposes you'll need to comment and uncomment the for/parfor (or perhaps use an if/else block, switching between a for and a parfor depending on some variable).
I think that the way to go here, is not to disable the parfor, but rather to let it behave like a simple for.
This should be possible by setting the number of workers to 1.
parpool(1)
Depending on your code you may be able to just do this once before you run the code, or perhaps you need to do this (conditionally) each time when you set the number of workers anywhere in your code.
I run a pretty computationally expensive genetic algorithm with MATLAB. The code has been running for 3 whole days, and I am pretty sure it gets stuck somewhere, because it is not printing out the progress information for debugging purpose.
I now wish to stop it. I did CTRL+C, but no luck. The bottom left of the window still displays "Busy".
I cannot simply quit the whole MATLAB, because I need to find out where it gets stuck by inspecting the variables in the variable window.
Given the CTRL+C is not working, how can I
stop the execution, OR
save the variables for inspection purpose?
Sometimes ctrl-C stops working if you have a memory over-allocation problem -- if you are trying to allocate a matrix that doesn't fit in memory, and so virtual memory begins thrashing.
It's also likely that crtl-C won't work while execution is passed to COMSOL.
I think you have little choice now but to kill matlab and try to debug by either stepping through the code or inserting fprintf statements.
I'm running a large code so I would like to ensure everything is correct when it starts to run. One thing I do is to plot some data to see if they make sense. I had several such plots (figure (1), figure (2) ...) and all of them are buried in different 'if' sentences. But I found only some of them pop up during the code is running, the others just don't show up until the code finished running. I have checked all the if statements are true.
Since my code is a little large I can't put it here. Can someone tell me the possible reasons that affect when figures pop up (during the code is running or after it's finished)? Thanks a lot!
You can use the drawnow function to update your figures while code is still running. If you want your figure window to "pop up" at any given time, you may also need to explicitly call figure(figNum).
Note that while it can be fun to watch your figures update in real time, you may incur a severe performance penalty by doing so. If you have long-running loops, you might consider only showing every N-th update, where N is 10, 100, 1000, or some other appropriate value:
for iter = 1:1e6
plot(x,y);
if ~mod(f0,1000); drawnow; end
end
If you are just interested in plotting right at the beginning as a sanity check, you might run the first iteration of your code outside of a for loop, and run drawnow to make sure it plots. Then enter into the for loop starting at the second iteration. The advantage over testing iter==1 is that you don't have to waste cycles on a conditional at every iteration.
I have noticed that the first time I run a script, it takes considerably more time than the second and third time1. The "warm-up" is mentioned in this question without an explanation.
Why does the code run faster after it is "warmed up"?
I don't clear all between calls2, but the input parameters change for every function call. Does anyone know why this is?
1. I have my license locally, so it's not a problem related to license checking.
2. Actually, the behavior doesn't change if I clear all.
One reason why it would run faster after the first time is that many things are initialized once, and their results are cached and reused the next time. For example in the M-side, variables can be defined as persistent in functions that can be locked. This can also occur on the MEX-side of things.
In addition many dependencies are loaded after the first time and remain so in memory to be re-used. This include M-functions, OOP classes, Java classes, MEX-functions, and so on. This applies to both builtin and user-defined ones.
For example issue the following command before and after running your script for the first run, then compare:
[M,X,C] = inmem('-completenames')
Note that clear all does not necessarily clear all of the above, not to mention locked functions...
Finally let us not forget the role of the accelerator. Instead of interpreting the M-code every time a function is invoked, it gets compiled into machine code instructions during runtime. JIT compilation occurs only for the first invocation, so ideally the efficiency of running object code the following times will overcome the overhead of re-interpreting the program every time it runs.
Matlab is interpreted. If you don't warm up the code, you will be losing a lot of time due to interpretation instead of the actual algorithm. This can skew results of timings significantly.
Running the code at least once will enable Matlab to actually compile appropriate code segments.
Besides Matlab-specific reasons like JIT-compilation, modern CPUs have large caches, branch predictors, and so on. Warming these up is an issue for benchmarking even in assembly language.
Also, more importantly, modern CPUs usually idle at low clock speed, and only jump to full speed after several milliseconds of sustained load.
Intel's Turbo feature gets even more funky: when power and thermal limits allow, the CPU can run faster than its sustainable max frequency. So the first ~20 seconds to 1 minute of your benchmark may run faster than the rest of it, if you aren't careful to control for these factors.
Another issue not mensioned by Amro and Marc is memory (pre)allocation.
If your script does not pre-allocate its memory it's first run would be very slow due to memory allocation. Once it completed its first iteration all memory is allocated, so consecutive invokations of the script would be more efficient.
An illustrative example
for ii = 1:1000
vec(ii) = ii; %// vec grows inside loop the first time this code is executed only
end