while using Matlab parfor I came across the following behaviour
parpool(2)
parfor j=1:100
v = j+1;
clear v
end
> Error in ==> parallel_function>make_general_channel/channel_general at 886
> Transparency violation error.
I looked into it, and indeed one is not allowed to use clear within parfor.
My question is why. v is created inside every specific worker, and so it does not interfere with other workers.
Matlab uses static code analyzer to understand how the body of parfor loop interacts with main workspace, i.e. which variables need to be transferred to workers and back. A number of functions, such as eval, evalc, evalin, assignin (with the workspace argument specified as 'caller'), load (unless the output is assigned to a variable), save and clear can modify workspace in ways that cannot be predicted by the static analyzer. There is no way to ensure integrity of the workspace when multiple workers are operating on it, and such functions are used.
Important thing to realize is that when you use a command syntax to invoke a function, such as clear v, the argument is passed as a string literal, meaning there is no way for the static analyzer to understand which variable you are trying to clear, hence no way to figure out the effect the command will have on the workspace.
As suggested in documentation, the workaround to free up most of the memory used by a variable inside parfor is: v = [];
Related
The following loop results in an error in C_mat and B_mat:
%previously defined
N_RIPETIZIONI=2;
K=201;
parfor n=1:N_RIPETIZIONI*K
[r,k]=ind2sub([N_RIPETIZIONI,K],n);
B=B_mat{r};
C=C_mat{r};
end
The warning says:
The entire array or structure B_mat is a broadcast variable. This might result in unnecessary communication overhead.
The same for C_mat.
How can I fix it so that the indices of B_mat and C_mat are no more broadcast variables?
The issue is that the way you index B_mat (i.e. not using n), every thread in the parfor requires the entirety of B_mat to run. The big bottleneck in parfor code is transferring copies of the data to each node.
MATLAB is basically telling you that if you were to do this, you may actually have slower code than otherwise. Its not that B_mat is some type of variable called "broadcast", its that the way you wrote the code, each n in parfor requires a copy of B_mat.
I assume this is not your real code, so we can't really help you fix it, but hopefully this explains it.
I am trying to use a table within a parfor loop in MATLAB. This gives me "Transparency violation error. See Parallel Computing Toolbox about transparency" I'm trying to build this table so I can make a prediction using a trained classifier from the MATLAB classification learner app (trainedClassifier.prefictFcn(T))...so either I need to build a table within the parfor loop or need some alternative to a table that I can still feed into the classifier.
parfor i=1:100
acheck=1;
bcheck=2;
ccheck=3;
T=table(acheck,bcheck,ccheck);
end
This solution works for your particular problem:
parfor i=1:100
acheck=1;
bcheck=2;
ccheck=3;
T(i,:)=table([acheck,bcheck,ccheck]);
end
Note that in your original program you just overwrite existing values and end up with a one row table. I assumed that that was not intended. Actually, that would be the outcome of a for.
Also, since this is a parfor and T is created inside the loop (as well as acheck, etc.) using just T creates nothing at all. The variable is a temporary one, visible to each process locally and destroyed in global scope (more can be found here).
To fix both overwriting and accessibility the program assigns the each set of variables to each row of T. If square brackets are omitted, the program throws a transparency error. Unfortunately, I do not know why is that but it may be that the operations done by the table data-structure cause that. Maybe someone else will know the answer, for now this seem to solve your problem though.
I have a couple questions regarding MATLAB workspaces:
When does MATLAB decide to change the workspace and what conditions prompt it?
Something strange is happening in the following example. I run it with a breakpoint at line 4, then step using F10 to watch the workspace variables. Clearly, I see that m is deleted within the first iteration of the inner loop, but somehow MATLAB still knows to go to the next iteration in the outer loop!
Example:
something = 2;
somethingelse = 3;
for m = 1 : something
for n = 1 : somethingelse
%do something
clearvars -except n something somethingelse % This clears m, but it still functions
end
end
The only thing I can think of is that MATLAB probably has not updated the memory locations given that the variable does not show on the list.
In MATLAB there are generally two types of workspaces: the base workspace and function workspaces. The base workspace is what you use when you enter commands at the command line, or when you run scripts (all of which share the base workspace). Conversely, each function has its own unique workspace. Unlike C or C++ (or a number of other languages) you don't have any scoping of variable within loops or conditional structures, just one unique workspace for each instance of a function.
The issue you're seeing in your example isn't really related to this, it's just an artifact of how for loops behave in MATLAB. Taken from the "Tips" section:
Avoid assigning a value to the index variable within the loop statements. The for statement overrides any changes made to index within the loop.
In other words, once an iteration of the loop completes and returns to the beginning, MATLAB ignores any changes to the loop variable and simply increments it to the next value.
If you'd like to learn more details about MATLAB workspaces and scoping, I'd check out these links:
Share Data Between Workspaces
Nested functions
Local functions
global variables
persistent variables
Function handles
Matlab changes workspace to the current scope.
You've only cleared the value of m within the scope of the second loop.
Try adding p = m+n after the clearvars command within the second loop. Since you've cleared m only within the scope of the n loop, you cannot use it. However, you did not remove m from the scope of the m loop.
Also, since the m for loop exists within the scope of your base workspace, you can clear m within the m for loop all you want, the loop will always have access to it. That's why, if you remove the clearvars line, when you return to the base workspace, you can see m and n equal to something and somethingelse respectively.
What I think you're looking for is a better explanation of Matlab's memory management, which you can find here: MATLAB's Garbage Collector?
I would like to run a complex Simulink model in a parfor loop on many cores with different data.
However, I didn't have success yet so I created a simple model and tried to run it parallel with the same data.
The model looks like this, adding two signals:
With the code:
function Result = Add (X, Y)
Result = X + Y;
The script to run the model is the following:
if matlabpool('size') == 0
matlabpool('open',4);
end
parfor i = 1:4
data=ones(1,20);
X=timeseries(data);
Y=timeseries(data);
output = sim('model_test','StopTime','5');
Result = output.get('Res');
end
However, the following error occurs:
I don't understand why the variables are not existing. I know that paralell computing is always critical in terms of variable access, but I didn't have success with simulink parallel running yet. Can you please explain the error to me and how to solve it?
Thank you very much!
Answer to am304: Thank you, the answer helped me in the way that I now know how to change constants with set_param in the parfor loop and I understand why it doesn't work for timeseries.
However for timeseries I am still struggling.
I tried several versions, also this one:
if matlabpool('size') == 0
matlabpool('open',4);
end
data=ones(1,20);
X=timeseries(data);
Ybase=timeseries(data);
parfor i = 1:4
Y = evalin('base', 'Ybase');
output = sim('model_test','StopTime','5');
Result{i} = output.get('Res');
end
The variable Ybase exists in the workspace, but the following error occurs:
As you see, the variable Ybase exists in the base workspace. Do you know how to use evalin or assignin to access properly?
Thanks and regards!
I suspect it's because your data data only exists in the workspace of the main MATLAB, not in any of the instances fired up by matlabpool on the workers. Have a look at Workspace Access Issues in the documentation for more details on how to resolve this, with some examples illustrating the two approaches:
The MATLAB workers, however, do not have access to the workspace of
the MATLAB client session where the model and its associated workspace
variables have been loaded. Hence, if you load a model and define its
associated workspace variables outside of and before a parfor loop,
then neither is the model loaded, nor are the workspace variables
defined in the MATLAB worker sessions where the parfor iterations are
executed. This is typically the case when you define model parameters
or external inputs in the base workspace of the client session. These
scenarios constitute workspace access issues.
[...]
Resolving Workspace Access Issues
When a Simulink model is loaded into memory in a MATLAB client session, it is only visible and
accessible in that MATLAB session; it is not accessible in the memory
of the MATLAB worker sessions. Similarly, the workspace variables
associated with a model that are defined in a MATLAB client session
(such as parameters and external inputs) are not automatically
available in the worker sessions. You must therefore ensure that the
model is loaded and that the workspace variables referenced in the
model are defined in the MATLAB worker session by using the following
two methods.
In the parfor loop, use the sim command to load the model and to set parameters that change with each iteration. (Alternative:
load the model and then use the g(s)et_param command(s) to set the
parameters in the parfor loop)
In the parfor loop, use the MATLAB evalin and assignin commands to assign data values to variables. Alternatively, you can
simplify the management of workspace variables by defining them in
the model workspace. These variables will then be automatically
loaded when the model is loaded into the worker sessions. There
are, however, limitations to this method. For example, you cannot
have tunable parameters in a model workspace. For a detailed
discussion on the model workspace, see Model Workspaces.
EDIT
Here's what I would do in your particular example:
if matlabpool('size') == 0
matlabpool('open',4);
end
data=ones(1,20);
X=timeseries(data);
Y=timeseries(data);
parfor i = 1:4
assignin('base','Y',Y);
output = sim('model_test','StopTime','5');
Result{i} = output.get('Res');
end
Another option is to include X and Y in the model workspace so that the model is self-contained.
I am calling external functions in my parfor loop as follows.
parfor idx = 1:2
import com.comsol.model.*
import com.comsol.model.util.*
model = ModelUtil.create('Model');
model.modelNode.create('comp1');
model.geom.create('geom1', 2);
model.geom('geom1').feature.create('sq1', 'Square');
model.geom('geom1').feature('sq1').set('size', '0.03125');
model.geom('geom1').feature('sq1').setIndex('pos', '0', 0);
model.geom('geom1').feature('sq1').setIndex('pos', '0', 1);
model.geom('geom1').run;
end
Error: MATLAB cannot determine whether "ModelUtil" refers to a function or variable.
See Parallel for Loops in MATLAB, "Unambiguous Variable Names".
After reading the "Unambiguous Variable Names" part in the MATLAB parfor documentation, I pretty much understand why this error occurs. However, I have no idea how to fix it.
I encountered the same problem with .Net objects. Like you, I found it works perfectly in non-parallel mode and the problem only occurs with parfor.
This is not a question of paths, this is purely a Matlab syntax/parser problem.
As a workaround, I wrapped the operations in a separate Matlab function so that within the scope of the parfor it is no longer ambiguous. Within the (otherwise unnecessary) function it has no trouble resolving the class, even when called from within parfor.
Vector's solution didn't work for me. The thing is that the different workers don't see the library you are using: you have to update the javapath in the parfor with javaaddpath.