I am having trouble using struct arrays in Matlab's parfor loop. The following code has 2 problems I do not understand:
s=struct('a',{},'b',{});
if matlabpool('size')==0
matlabpool open local 2
end
for j = 1:2
parfor k=1:4
fprintf('[%d,%d]\n',k,j)
s(j,k).a = k;
s(j,k).b = j;
end
end
matlabpool close
It fails with an error Error using parallel_function (line 589)
Insufficient number of outputs from right hand side of equal sign to satisfy assignment.
On output, variable s is a vector, not an array (as it should be, even if the code breaks before finishing).
EDIT the problem is solved if I initialize the struct arrays to the correct size, by:
s=struct('a',cell(2,4),'b',cell(2,4));
However, I would still be happy to get insights about the problem (e.g is it rally a bug, as suggested by Oleg Komarov)
It was originally working fine for me but then I don't know what happens. In general you need to be careful with parfor loops and there are ample documentation on how to align everything. Two different words of advice.
First and more importantly, the parfor loop is on the outside loop:
function s = foo
s=struct('a',{},'b',{});
parfor j = 1:2
for k=1:4
fprintf('[%d,%d]\n',k,j)
s(j,k).a = k;
s(j,k).b = j;
end
end
Two, Matlab gets very picky about writing the main exit variable (i.e. the variable contained in the parfor loop which is indexed to the loop, in your case, s). You first want to create a dummy variable that holds all the innerloop information, and then writes to it once at the end of the loops. Example:
function s = khal
s=struct('a',{},'b',{});
parfor j = 1:2
dummy=struct('a',{},'b',{});
for k=1:4
fprintf('[%d,%d]\n',k,j)
dummy(k).a = k;
dummy(k).b = j;
end
s(j,:) = dummy;
end
You don't have a problem here, but it can get complicated in other instances
Related
I have a code in MATLAB in which I'm running monte-carlo simulations using parfor instead of simple for loop to convert the code from sequential to parallel. Following is the piece of code which is inside the parfor loop.
But MATLAB gives an error saying "Valid indices for local_Q_mega_sub_seed are restricted in parfor loop". Suggested action says to "Fix the index" and it suggests to use "Sliced Variables". I've been struggling to use this concept. I have read https://blogs.mathworks.com/loren/2009/10/02/using-parfor-loops-getting-up-and-running/#12 and https://www.mathworks.com/matlabcentral/answers/123922-sliced-variables-in-parfor-loop-restricted-indexing along with MATLAB documentation https://www.mathworks.com/help/distcomp/sliced-variable.html and https://www.mathworks.com/help/distcomp/parfor.html but I'm not getting it right.
Could anyone please let me know how can I use sliced variables in the given piece of code so that I could get an idea?
index_f = 1;
subseed_step = (sub_seed_transmitted_at/fs_local)*sintablen_mega_frequency;
for i = 1 : fs_local
local_Q_mega_sub_seed(i) = SINTAB(round(index_f));
local_I_mega_sub_seed(i) = COSTAB(round(index_f));
index_f = index_f + subseed_step;
if index_f>sintablen_mega_frequency
index_f = index_f - sintablen_mega_frequency;
end
You're not showing enough context here, but I bet the problem here is similar to this one:
parfor ii = 1:10
for jj = 1:10
tmp(jj) = rand
end
out(ii) = sum(tmp);
end
In this case, the parfor machinery cannot categorically prove that the way tmp is being used is independent of the order of iterations of the parfor loop. This is because it appears as though values assigned to tmp in one iteration of the parfor loop are still being used in the next iteration.
Fortunately, there's a very simple workaround for this case - convince parfor that you are not doing anything dependent on the order of evaluation of the iterations of the loop by resetting the variable. In the simple case above, this means:
parfor ii = 1:10
% reset 'tmp' at the start of each parfor loop iteration
tmp = [];
for jj = 1:10
tmp(jj) = rand
end
out(ii) = sum(tmp);
end
I have a Matlab script including a for loop which loos like the following:
for k = 1:10
c = myfun(k,a,b);
result{k} = c;
end
Right now, the problem is that during the for loop, sometimes myfun() may have errors and stop. After fixing the error in myfun(), how can I continue to run with the existing value of variables? The reason is that myfun() will take a very long time to get the result and the previous results are right.
For example, if a error happens when k == 4, then I save all the variables in the current workspace. I set a breakpoint at c = myfun(k,a,b); and restore the saved variables, but I find that in the next loop, k will be 2 instead of 5 as I want. Matlab is not allowed to modify the value of k during the for loop I think. I have tested this for a few times.
How can I continue the for loop with some existing data?
You cannot change your for loop iterator programmatically inside of the loop.
For example:
for ii = 1:3
disp(ii)
ii = 3;
end
Prints:
1
2
3
If you're going to be modifying code based on errors received, dbstop if error is not going to be beneficial because it will not reflect changes in your code until the debugger is exited and your code executed again (unless you execute manually in the debugger). If you're not modifying code you could potentially use a try/except clause to catch fixable issues.
If you're loading data for a later index and then restarting, you can change where your for loop begins, or use a while loop (if appropriate).
For example:
% Load data here
for ii = 3:3
disp(ii)
end
Prints 3.
Where the while interpretation would be:
% Load data here
ii = 3
while ii <= 3
disp(ii)
ii = ii + 1;
end
For the same result.
On solution can be first catch the exception likes the following and pass from them:
bug = [];
for k = 1:10
try
c = myfun(k,a,b);
result{k} = c;
catch
warning('some bug for the following values:');
display([k a b]);
bug = [bug; k a b];
result{k} = NaN;
end
end
Then iterate over bug to compute missing information after debugging. This solution works when your algorithm is not dependent on the previous value of the result (or is not recursive).
I have a little parfor test script which gives the warning in the title.
The code is this
out = zeros(10, 1);
in = rand(5e8, 10);
tic
parfor i = 1:10
for j = 1:5e8
p = floor(rand(1,1)*5e8);
out(i) = out(i) + in(p, i);
end
end
toc
tot = sum(out)
the warning comes out on line 7 regarding how variable in is accessed.
I don't understand why, slicing should be trivial. Just send each column of in to each worker.
If I change the code to
out = zeros(10, 1);
in = rand(5e8, 10);
tic
parfor i = 1:10
a = in(:,i);
for j = 1:5e8
p = floor(rand(1,1)*5e8);
out(i) = out(i) + a(p);
end
end
toc
tot = sum(out)
the warning disappears but I don't like that assignment to a.
The code was explicitly designed to mess up the cache memory.
Unfortunately, as explained here http://www.mathworks.com/help/distcomp/advanced-topics.html#bq_of7_-1 , MATLAB does not understand how to slice in, hence the code analyser warning. You have to read that page fairly closely to understand why it cannot be sliced. The relevant paragraph is:
Form of Indexing. Within the list of indices for a sliced variable, one of these indices is of the form i, i+k, i-k, k+i, or k-i, where i
is the loop variable and k is a constant or a simple (nonindexed)
broadcast variable; and every other index is a scalar constant, a
simple broadcast variable, colon, or end.
The clause in bold type at the end is the relevant one - in your case, p does not match this constraint.
In MATLAB, I have a variable proba and I have a parfor loop as showing below:
parfor f = 1:N
proba = (1/M)*ones(1, M);
% rest of the code
end
pi_proba = proba;
MATLAB said that: "The temporary variable 'proba' is used after the PARFOR loop, but its value is nondeterministic"
I do not understand how to correct this error. I need to use a parallel loop and I need proba after the loop. How to do this?
When using parfor the classes are classified according to these categories. Make sure every variable matches one of these categories. For non-writing access to proba a Broadcast-Variable would be the best choice:
proba = (1/M)*ones(1, M);
parfor f = 1:N
% rest of the code
end
pi_proba = proba;
In case of writing access within the loop, a sliced variable is nessecary:
proba=cell(1,N)
parfor f = 1:N
%now use proba{f} inside the loop
proba{f}=(1/M)*ones(1, M);
% rest of the code
end
%get proba from whatever iteration you want
pi_proba = proba{N};
I am running a parfor loop in Matlab that takes a lot of time and I would like to know how many iterations are left. How can I get that info?
I don't believe you can get that information directly from MATLAB, short of printing something with each iteration and counting these lines by hand.
To see why, recall that each parfor iteration executes in its own workspace: while incrementing a counter within the loop is legal, accessing its "current" value is not (because this value does not really exist until completion of the loop). Furthermore, the parfor construct does not guarantee any particular execution order, so printing the iterator value isn't helpful.
cnt = 0;
parfor i=1:n
cnt = cnt + 1; % legal
disp(cnt); % illegal
disp(i); % legal ofc. but out of order
end
Maybe someone does have a clever workaround, but I think that the independent nature of the parfor iterations belies taking a reliable count. The restrictions mentioned above, plus those on using evalin, etc. support this conclusion.
As #Jonas suggested, you could obtain the iteration count via side effects occurring outside of MATLAB, e.g. creating empty files in a certain directory and counting them. This you can do in MATLAB of course:
fid = fopen(['countingDir/f' num2str(i)],'w');
fclose(fid);
length(dir('countingDir'));
Try this FEX file: http://www.mathworks.com/matlabcentral/fileexchange/32101-progress-monitor--progress-bar--that-works-with-parfor
You can easily modify it to return the iteration number instead of displaying a progress bar.
Something like a progress bar could be done similar to this...
Before the parfor loop :
fprintf('Progress:\n');
fprintf(['\n' repmat('.',1,m) '\n\n']);
And during the loop:
fprintf('\b|\n');
Here we have m is the total number of iterations, the . shows the total number of iterations and | shows the number of iterations completed. The \n makes sure the characters are printed in the parfor loop.
With Matlab 2017a or later you can use a data queue or a pollable data queue to achieve this. Here's the MathWorks documentation example of how to do a progress bar from the first link :
function a = parforWaitbar
D = parallel.pool.DataQueue;
h = waitbar(0, 'Please wait ...');
afterEach(D, #nUpdateWaitbar);
N = 200;
p = 1;
parfor i = 1:N
a(i) = max(abs(eig(rand(400))));
send(D, i);
end
function nUpdateWaitbar(~)
waitbar(p/N, h);
p = p + 1;
end
end
End result :
If you just want to know how much time is left approximately, you can run the program once record the max time and then do this
tStart = tic;
parfor i=1:n
tElapsed = toc(tStart;)
disp(['Time left in min ~ ', num2str( ( tMax - tElapsed ) / 60 ) ]);
...
end
I created a utility to do this:
http://www.mathworks.com/matlabcentral/fileexchange/48705-drdan14-parforprogress