How to fix conflicting simulink simulation accelerated artificats issue when running test in parallel mode? - matlab

Question

My goal is to optimize the time it takes to run a set of simulation test cases. I'm having issues running test with parallel processing and accelerated simulation features. https://www.mathworks.com/help/simulink/ug/how-the-acceleration-modes-work.html
Context:
I have 29 Simulink files that are called inside of a parametrize Matlab Unit Test. The Simulink files have a lot of reference models. Before running a 20 second simulation for each simulink, the simulinks have to load all reference models and create a lot of simulation artifacts in a work folder. A lot of reference model are shared in-between simulink projects. The simulink projects have 64/187 reference model that run in accelerated mode. Normal mode generates .mexw64, and accelerated mode generates .slxc and .mexw64 in the work folder.
Action:
I run 1 test once . All in normal mode. My tests Succeed.
I run 29 test sequentially. All in normal mode. My tests Succeed.
I run 1 test once, then I run the 29 test in parallel clusters.All in normal mode.My tests Succeed.(**See Ref #1)
I run 1 test once, All in Accelerator Mode. My tests Succeed.
I run 29 test sequentially. All in Accelerator mode. My tests Succeed.
I run 1 test once, then I run the 29 test in parallel clusters,All in Accelerator Mode.My tests Fails.
Expected & Results:
I expected my simulation running in accelerator/parallel mode to have same positive results as the normal/parallel mode. But :
I'm having read/write/building issue using shared resources on
parallel tread when I run 2 test in parallel.
My parallel thread fails when I try to run 29 at the
same time.
Any idea how to fix this?
I tried different build configuration in simulation and build settings, I tried reduce the number of accelerated targets, and reading online resources.
Error:
#1
Building with 'Microsoft Visual C++ 2015 (C)'.
LINK : fatal error LNK1104: cannot open file 'D:\GIT\***\***\work\sim_artifacts\***_src_msf.mexw64'
NMAKE : fatal error U1077: 'C:\PROGRA~1\MATLAB\R2017b\bin\win64\mex.EXE' : return code '0xffffffff'
Stop.
The make command returned an error of 2
'An_error_occurred_during_the_call_to_make' is not recognized as an internal or external command,
operable program or batch file.
### Build procedure for model: '***_src' aborted due to an error.
#2
The client lost connection to worker 4. This might be due to network problems,
or the interactive communicating job might have errored.
Ref:
How to fix missing simulink simulation artificats issue when running test in parallel mode?
https://www.mathworks.com/help/simulink/gui/rebuild.html
https://www.mathworks.com/help/simulink/ug/model-callbacks.html#
https://www.mathworks.com/help/simulink/ug/reuse-simulation-builds-for-faster-simulations.html
https://www.mathworks.com/help/matlab/ref/matlab.unittest.testrunner.runinparallel.html

Answer 1

My problem was a data concurrency issues.
I found the solution.
Solution:
Now I run a test once in accelerator mode to generate cached folder (simulation targets and accelerated artifacts). Then I copy the cache folder 29 times. then I assign each folder to 1 parallel test run as input. Hence, all parallel test read and write in different folder hence they don't conflict with each other anymore.
Ref:
https://www.youtube.com/watch?v=cKK3qpBjixA
https://www.mathworks.com/help/simulink/ug/model-reference-simulation-targets-1.html
https://www.mathworks.com/help/simulink/ug/not-recommended-using-sim-function-within-parfor.html
you need to address any workspace access issues and data concurrency issues to produce useful results. Specifically, the simulations need to create separately named output files and workspace variables. Otherwise, each simulation overwrites the same workspace variables and files, or can have collisions trying to write variables and files simultaneously.
Code:
function setParCacheFolder()
%% get Simulink Files names
originalPath = path;
addpath(genpath('***\***\***\***'));
file=getFileNames('FOOTEST_*.slx','***\***\***\***');
for i = 1:length(file)
%% open/load simulink system
[~,NAME,~]=fileparts(file(i));
sys{i}=NAME;
open_system(sys{i});
%% copy Cache Folder
copyfile(fullfile(pwd,'\***\work\sim_artifacts'), fullfile(pwd,strcat('\***\work\sim_artifacts',sys{i}{1})));
%% get specific cache folder
get_param(0, 'CacheFolder')
%% set specific cache folder
set_param(0, 'CacheFolder', fullfile(pwd,strcat('\***\work\sim_artifacts',sys{i}{1})));
%% save simulink project
save_system(sys{i}{1},[],'SaveDirtyReferencedModels','on','OverwriteIfChangedOnDisk',true);
bdclose('all');
end
path(originalPath);
end