I'm trying to accelerate our test environment by using the ParralelToolbox of Mathworks. However I am unable to start several Matlab instances in parallel (up to now we run our tests sequentially and each one is starting a new Matlab instance via an ActX server).
So when I run the following code below
ML=ver('Matlab');
ML_Path=matlabroot;
ML_Ver=ML.Version;
parfor i = 1:3
NewMatlab = actxserver(['matlab.application.single.',ML_Ver])
Answer = NewMatlab.Feval('test',1);
NewMatlab.Quit;
NewMatlab.release;
end
the Matlab instances are called sequentially (test is just a very simple script that sums up a few numbers).
However if I start a new Matlab via command line
ML=ver('Matlab');
ML_Path=matlabroot;
ML_Ver=ML.Version;
parfor i = 1:3
dos('matlab -nodesktop -minimize -wait -batch "test"');
end
it works. I see that these two methods are quite different in the handling of starting Matlab, but the first approach would be
If you want each iteration of your test to run in a completely separate MATLAB instance, you could use the batch function, like this:
for i = 1:3
j(i) = batch(#test, nOut, {argsIn...});
end
% Later, collect results
for i = 1:3
wait(j(i)), fetchOutputs(j(i))
end
Or, you could simply use parfor directly
parpool() % If necessary
parfor i = 1:3
out{i} = test(...)
end
(You only need to call parpool if no pool is currently open, and you have your preferences set so that a pool is not automatically created when you hit the parfor).
Related
I have a code from matlab 2010a that I want to run it in matlab 2019a, I'm using parallelism.
matlabpool open 4 %prepares matlab to run in 4 parallel procesors
j1 = batch('parallel1', 'matlabpool', 0);
pause(1)
j2 = batch('parallel2', 'matlabpool', 0);
pause(1)
j3 = batch('parallel3', 'matlabpool', 0);
pause(1)
j4 = batch('parallel4', 'matlabpool', 0);
matlabpool close
But, the code dosen't run in this version of matlab, because I have to use parpool.
So, I'm asking to someone who know how to convert or how to change this part of the code to run in my new matlab version.
The literal translation of your code is to do this:
parpool(4) % Creates a parallel pool with 4 workers
j1 = batch('parallel1', 'Pool', 0) % creates a batch job with no pool
... % etc.
However, I'm curious as to whether this is actually what you want to do. The parpool(4) command launches 4 worker processes to be used by your desktop MATLAB - for when you use parfor, spmd, or parfeval. Each batch command spawns an additional worker process, which cannot access the workers from the parallel pool.
First step is to check the original documentation, since 2010a is no longer online here the corresponding 2013a documentation. It still has matlabpool explained:
'Matlabpool' — An integer specifying the number of workers to make into a MATLAB pool for the job in addition to the worker running the batch job itself. The script or function uses this pool for execution of statements such as parfor and spmd that are inside the batch code. Because the MATLAB pool requires N workers in addition to the worker running the batch, there must be at least N+1 workers available on the cluster. You do not have to have a MATLAB pool already running to execute batch; and the new pool that batch opens is not related to a MATLAB pool you might already have open. (See Run a Batch Parallel Loop.) The default value is 0, which causes the script or function to run on only the single worker without a MATLAB pool.
In current MATLAB-Versions this option is replaced by the pool parameter. 0 Is still the default behavior, you can use:
j1 = batch('parallel1');
I have written a program in matlab and my algorithm depends on few parameters let's say a and b with a=1:10 and b=1:10 I want to find those values of a and b which gives me the best results.
My main code is as follows:
a= 0.1:0.1:1;
b= 1:1:10;
arr_mat = zeros(length(a),length(b));
for i=1:length(a)
for j=1:length(b)
disp(['loop no = ',num2str(i),' & ',num2str(j)]);
knn = a(j);
eta = b(i);
arr_mat(i,j) = called_function(knn,eta);
end
end
The program runs but is very computationally expensive. I was wondering if there is any matlab inbuilt parallel processing toolbox which will be helpful for me in this case. I was thinking along these lines :
Divide my main program into parts:
a1= 0.1:0.1:0.5;
b1= 1:1:5;
arr_mat1 = zeros(length(a1),length(b1));
for i1=1:length(a)
for j1=1:length(b)
disp(['loop no = ',num2str(i1),' & ',num2str(j1)]);
knn1 = a1(j);
eta1 = b1(i);
arr_mat1(i,j) = called_function(knn1,eta1);
end
end
a2= 0.6:0.1:1;
b2= 6:1:10;
arr_mat2 = zeros(length(a2),length(b2));
for i2=1:length(a2)
for j2=1:length(b2)
disp(['loop no = ',num2str(i2),' & ',num2str(j2)]);
knn2 = a2(j);
eta2 = b2(i);
arr_mat2(i,j) = called_function(knn2,eta2);
end
end
Run the codes in parallel. My system configuration is : Intel Core i7-3770 # 3.40 GHz with 32.0 GB RAM. I have MATLAB 2013b installed.
I have consulted this question and have tried to write my own code in that format:
clc;clear all;close all;
% run ixmas for different modifications
a= 0.1:0.1:1;
b= 1:1:10;
arr_mat = zeros(length(a),length(b));
matlabpool open local 2
parfor i=1:length(a)
for j=1:length(b)
disp(['loop no = ',num2str(i),' & ',num2str(j)]);
knn = a(j);
eta = b(i);
recog = 0;
for k=1:5
recog(k) = ixmas(knn,eta);
end
arr_mat(i,j) = mean(recog);
end
end
end
matlabpool close
This code throws up the error : "Illegal use of reserved keyword "end"."
My queries :
Kindly please tell me where I am going wrong.
I tried to introduce a parfor in the inner loop but it throws up an error?
What does the statement matlabpool open local 2 mean ?
How many threads can I use in parallel for my system configuration ? How can I even check those limits and optimally use them to the full capacity ?
Does parallel processing means utilizing different cores of my own machine or utilizing resources of other machines ? Is there any way either options can be selected and how does one go about doing so?
matlabpool does not open a statement that needs a closing end. In the linked question the end is closing the spmd statement.
The meaning of the statement matlabpool open local 2 is:
matlabpool open or closes a pool of matlab parallel processing worker processes.
open tells matlab to run these workers. (this is the default)
local tells matlab to open these workers on the local machine (this is the default)
2 tells matlab to run 2 such workers (default is number of cpus)
You can just matlabpool without any parameters.
in order to configure the local profile go to the matlab home tab and choose 'manage clusters' under the parallel menu.
Also, for some strange reason matlab can't split the array between the workers when the parfor is not in the inner most loop. change the inner for to parfor and it will work.
I have figured out some awesome ways of speeding up my MATLAB code: vectorizing, arrayfun, and basically just getting rid of for loops (not using parfor). I want to take it to the next step.
Suppose I have 2 function calls that are computationally intensive.
x = fun(a);
y = fun(b);
They are completely independent, and I want to run them in parallel rather than serially. I dont have the parallel processing toolbox. Any help is appreciated.
thanks
If I am optimistic I think you ask "How Can I simply do parallel processing in Matlab". In that case the answer would be:
Parallel processing can most easily be done with the parallel computing toolbox. This gives you access to things like parfor.
I guess you can do:
parfor t = 1:2
if t == 1, x = fun(a); end
if t == 2, y = fun(b); end
end
Of course there are other ways, but that should be the simplest.
The MATLAB interpreter is single-threaded, so the only way to achieve parallelism across MATLAB functions is to run multiple instances of MATLAB. Parallel Computing Toolbox does this for you, and gives you a convenient interface in the form of PARFOR/SPMD/PARFEVAL etc. You can run multiple MATLAB instances manually, but you'll probably need to do a fair bit of work to organise the work that you want to be done.
The usual examples involve parfor, which is probably the easiest way to get parallelism out of MATLAB's Parallel Computing Toolbox (PCT). The parfeval function is quite easy, as demonstrated in this other post. A less frequently discussed functionality of the PCT is the system of jobs and tasks, which are probably the most appropriate solution for your simple case of two completely independent function calls. Spoiler: the batch command can help to simplify creation of simple jobs (see bottom of this post).
Unfortunately, it is not as straightforward to implement; for the sake of completeness, here's an example:
% Build a cluster from the default profile
c = parcluster();
% Create an independent job object
j = createJob(c);
% Use cells to pass inputs to the tasks
taskdataA = {field1varA,...};
taskdataB = {field1varB,...};
% Create the task with 2 outputs
nTaskOutputs = 2;
t = createTask(j, #myCoarseFunction, nTaskOutputs, {taskdataA, taskdataB});
% Start the job and wait for it to finish the tasks
submit(j); wait(j);
% Get the ouptuts from each task
taskoutput = get(t,'OutputArguments');
delete(j); % do not forget to remove the job or your APPDATA folder will fill up!
% Get the outputs
out1A = taskoutput{1}{1};
out1B = taskoutput{2}{1};
out2A = taskoutput{1}{2};
out2B = taskoutput{2}{2};
The key here is the function myCoarseFunction given to createTask as the function to evaluate in the task objects to creates. This can be your fun or a wrapper if you have complicated inputs/outputs that might require a struct container.
Note that for a single task, the entire workflow above of creating a job and task, then starting them with submit can be simplified with batch as follows:
c = parcluster();
jobA = batch(c, #myCoarseFunction, 1, taskdataA,...
'Pool', c.NumWorkers / 2 - 1, 'CaptureDiary', true);
Also, keep in mind that as with matlabpool(now called parpool), using parcluster requires time to startup the MATLAB.exe processes that will run your job.
I have Matlab .m script that sets and trains Neural network ("nn") using Matlab's Neural network toolbox. The script launches some GUI that shows trainig progress etc. The training of nn usually takes long time.
I'm doing these experiments on computer with 64 processor cores. I want to train several networks at the same time without having to run multiple Matlab sessions.
So I want to:
Start training of neural network
Modify script that creates network to create different one
Start training of modified network
Modify script to create yet another network...
Repeat steps 1-4 several times
The problem is that when I run the scrip it blocks Matlab terminal so I cannot do anything else until the script executes its last command - and that takes long. How can I run all those computations in parallel? I do have Matlab parallel toolbox.
EDIT: Matlab bug??
Update: This problem seems to happen only on R2012a, looks like fixed on R2012b.
There is very strange error when I try command sequence recommended in Edric's answer.
Here is my code:
>> job = batch(c, #nn, 1, {A(:, 1:end -1), A(:, end)});
>> wait(job);
>> r = fetchOutputs(job)
Error using parallel.Job/fetchOutputs (line 677)
An error occurred during execution of Task with ID 1.
Caused by:
Error using nntraintool (line 35)
Java is not available.
Here are the lines 27-37 of nntraintool (part of Matlab's Neural networks toolkit) where error originated:
if ~usejava('swing')
if (nargin == 1) && strcmp(command,'check')
result = false;
result2 = false;
return
else
disp('java used');
error(message('nnet:Java:NotAvailable'));
end
end
So it looks like the problem is that GUI (because Swing is not available) cannot be used when job is executed using batch command. The strange thing is that the nn function does not launch any GUI in it's current form. The error is caused by train that launches GUI by default but in nn I have switched that off:
net.trainParam.showWindow = false;
net = train(net, X, y);
More interestingly if the same nn function is launched normally (>> nn(A(:, 1:end -1), A(:, end));) it never enters the outer if-then statement of nntraintool on line 27 (I have checked that using debugger). So using the same function, the same arguments expression ~usejava('swing') evaluates to 0 when command is launched normally but to 1 when launched using batch.
What do you think about this? It looks like ugly Matlab or Neural networks toolbox bug :(((
With Parallel Computing Toolbox, you can run up to 12 'local workers' to execute your scripts (to run more than that, you'd need to purchase additional MATLAB Distributed Computing Server licences). Given your workflow, the best thing might be to use the BATCH command to submit a series of non-interactive jobs. Note that you will not be able to see any GUI from the workers. You might do something like this (using R2012a+ syntax):
c = parcluster('local'); % get the 'local' cluster object
job = batch(c, 'myNNscript'); % submit script for execution
% now edit 'myNNscript'
job2 = batch(c, 'myNNscript'); % submit script for execution
...
wait(job); load(job) % get the results
Note that the BATCH command automatically attaches a copy of the script to run to the job, so that you are free to make changes to it after submission.
I have a problem using the Parallel Toolbox from Matlab. Indeed, I want to untar a series of archive in a parfor loop, and it seems that neither untar nor system are working. They do not cause an error, they simply do not produce any result.
The very same code works without any problem as soon as I deactivate the parallelism.
Is there a reference that lists the functions that cannot be used in parfor loops? I couldn't find it easily in the parallel toolbox documentation.
system should work correctly inside a PARFOR loop - providing that the executable you invoke does not require user input.
>> matlabpool('size')
ans =
3
>> parfor ii=1:2, system('pwd'), end
/tmp
ans =
0
/tmp
ans =
0
The main constraints on functions which cannot be used directly inside a PARFOR loop body relate to "workspace transparency" - you cannot use functions that modify the workspace such as assignin, load, clear etc. See this page for more on that. (You can of course call a function from the body of a PARFOR loop which calls load etc.)
You can try something like this.
function parallel_stuff
parfor i = 1:10
b = my_untar(a)
end
end
function b = my_untar(a)
b = untar(a)
end