I am a beginner in MATLAB software and I use MATLAB online .The problem is in initializing a variable called 'estModel' in a parallel loop . I want to use parallel processing because the calculations take a long time to complete but this problem occurs when I use 'threads' in parpool instead of 'local' . In 'local' mode this problem does not exist and the code is executed correctly and the results are correct . But in the case of 'threads', the variable 'estModel' is not initialize in the parfor loop , and give this error : Unrecognized function or variable 'estModel' . In 'local' mode the number of workers are 2 but in 'threads' mode the number of workers are 8 . So I use 'threads' instead of 'local' . I do not know about parpool structure completely . Is there a problem in this part ( parpool : local or threads or ... and chooses 'threads' based on the number of workers ) ? Finally the three variables 'kh', 'model' and 'DATA' are completely initializing before parallel loop and the variable 'model' uses the arima function .
Can anyone help me to solve this problem?
code :
clear all;
close all;
clc;
DATA=xlsread('test'); % The data for modeling and forecasting is read from an excel file called test, which is a column vector.
size_of_DATA=length(DATA);
p=20; % nonseasonal autoregressive polynomial degree range in arima model.
q=10; % nonseasonal moving average polynomial degree range in arima model.
D=2; % degree of nonseasonal integration range in arima model.
h=0; % counter
for k=1:D
for j=1:q
for i=1:p
if i+j+k-2<=size_of_DATA
h=h+1;
model(h)=arima(i-1,k-1,j-1);
end
end
end
end
poolobj=parpool('threads');
parfor i=1:h
estModel(i)=estimate(model(i),DATA);
end
delete(poolobj);
The excel file called 'test' is an arbitrary time series that is a column vector with 500 datas (of course, its length can vary). you can use file in this link for test : ln5.sync.com/dl/9e3913140/ctiyajcx-mgmnis6z-8mgyaigq-p2a9hpxu
Related
I am using a simulated annealing algorithm to optimize my problem, I have to do it for 100 different input variables and save the output for all variables in order. the problem is that I don't know how to implement spmd in my code to do parallel computing so that each input run on one CPU core and the final results stored in a matrix with 100 rows. I've tried to put it before the first for loop but it only returns a composite consists of 4 elements, since my CPU has 4 cores. Here is my code
spmd
for v=1:100
posmat=loading_param(Matrix,v);
nvar=size(posmat,2);
popsize=50;
maxiter=20;
T0=1000;
Tf=1;
Tdamp=((T0-Tf)/maxiter);
nn=5;
T=T0;
%% initial population
tic
emp.var=[];
emp.fit=inf;
pop=repmat(emp,popsize,1);
for i=1:popsize
pop(i).var=randperm(nvar);
pop_double=pop(i).var;
posmat_new=tabdil(nvar,pop_double,posmat);
dis=cij(posmat_new);
pop(i).fit=fittness(dis);
end
[value,index]=min([pop.fit]);
gpop=pop(index);
%% algorithm main loop
BEST=zeros(maxiter,1);
for iter=1:maxiter
for i=1:popsize
bnpop=emp;
for j=1:nn
npop=create_new_pop(pop(j),nvar,posmat);
if npop.fit<bnpop.fit
bnpop=npop;
end
end
if bnpop.fit<pop(i).fit
pop(i)=bnpop;
else
E=bnpop.fit-pop(i).fit;
pr=exp(-E/T);
if rand<pr
pop(i)=bnpop;
end
end
end
T=T-Tdamp;
[value,index]=min([pop.fit]);
if value<gpop.fit
gpop=pop(index);
BEST(iter)=gpop.fit;
disp([ 'iter= ' num2str(iter) 'BEST=' num2str(BEST(iter))])
end
end
%% algorithm results
disp([ ' Best solution=' num2str(gpop.var)])
disp([ ' Best fittness=' num2str(gpop.fit)])
disp([ ' Best time=' num2str(toc)])
bnpop_all(d,:)=bnpop.var;
d=d+1;
end %end of main for loop
end % end of spmd
From the documentation on spmd:
Values returning from the body of an spmd statement are converted to Composite objects on the MATLAB client. A Composite object contains references to the values stored on the remote MATLAB workers, and those values can be retrieved using cell-array indexing. The actual data on the workers remains available on the workers for subsequent spmd execution, so long as the Composite exists on the client and the parallel pool remains open.
Thus the output is a composite with 4 elements, since you have 4 CPU cores, so output{1} gives you the first element, output{2} the second etc. Just concatenate those to get your output in a single matrix.
Your code at this point just runs four times, one complete 100 iteration for loop per worker. An easier way to solve this, is to use parfor instead of spmd, as you can leave your loop the same. If you want to use spmd, first cut your v into four pieces (of 25 elements each), then on each worker iterate over just those 25 elements.
Seeing your code, with its three nested loops, I suggest not parallellising now, but instead try to profile your code, find out where the bottlenecks are, and try to speed up those. Probably trying to vectorise your nested loops will improve a lot already.
I am experimenting with MATLAB SPDM. However, I have the following problem to solve:
I am running a quite long algorithm and I would like to save the progress along the way in case the power gets cut, someone unplugs the power plug or memory error.
The loop has 144 iterations that take each around 30 minutes to complete => 72h. A lot of problems can occur in that interval.
Of course, I have the distributed computing toolbox on my machine. The computer has 4 physical cores. I run MATLAB R2016a.
I do not really want to use a parfor loop because I concatenate results and have dependency across iterations. I think SPMD is the best choice for what I want to do.
I'll try to describe what I want as best as I can:
I want to be able to save at a set iteration of the loop the results so far, and I want to save the results by worker.
Below is a Minimum (non)-Working Example. The last four lines should be put in a different .m file. This function, called within a parfor loop, allows to save intermediate iterations. It is working properly in other routines that I use. The error is at line 45 (output_save). Somehow, I would like to "pull" the composite object into a "regular" object (cell/structure).
My hunch is that I do not quite understand how Composite objects work and especially how they can be saved into "regular" objects (cells, structures, etc).
% SPMD MWE
% Clear necessary things
clear output output2 output_temp iter kk
% Useful thing that will be used later on
Rorder=perms(1:4);
% Stem of the file to save the data to
stem='MWE_MATLAB_spmd';
% Create empty cells where the results of the kk loop will be stored
output1{1,1}=[];
output2{1,2}=[];
% Start the parpool
poolobj=gcp;
% Define which worker/lab will do which iteration
iterperworker=ceil(size(Rorder,1)/poolobj.NumWorkers);
for i=1:poolobj.NumWorkers
if i<poolobj.NumWorkers
itertodo{1,i}=1+(iterperworker)*(i-1):iterperworker*i;
else
itertodo{1,i}=1+(iterperworker)*(i-1):size(Rorder,1);
end
end
%Start the spmd
% try
spmd
iter=1;
for kk=itertodo{1,labindex}
% Print which iteration is done at the moment
fprintf('\n');
fprintf('Ordering %d/%d \r',kk,size(Rorder,1));
for j=1:size(Rorder,2)
output_temp(1,j)=Rorder(kk,j).^j; % just to populate a structure
end
output.output1{1,1}=cat(2,output.output1{1,1},output_temp); % Concatenate the results
output.output2{1,2}=cat(2,output.output1{1,2},0.5*output_temp); % Concatenate the results
labindex_save=labindex;
if mod(iter,2)==0
output2.output=output; % manually put output in a structure
dosave(stem,labindex_save,output2); % Calls the function that allows me to save in parallel computing
end
iter=iter+1;
end
end
% catch me
% end
% Function to paste in another m-file
% function dosave(stem,i,vars)
% save(sprintf([stem '%d.mat'],i),'-struct','vars')
% end
A Composite is created only outside an spmd block. In particular, variables that you define inside an spmd block exist as a Composite outside that block. When the same variable is used back inside an spmd block, it is transformed back into the original value. Like so:
spmd
x = labindex;
end
isa(x, 'Composite') % true
spmd
isa(x, 'Composite') % false
isequal(x, labindex) % true
end
So, you should not be transforming output using {:} indexing - it is not a Composite. I think you should simply be able to use
dosave(stem, labindex, output);
I wanted to bootstrap with matlab using the built in command "bootstrp". What I noticed was that the procedure makes N+1 iterations when I am only asking for N iterations. Why is that? When I build a manual loop to do the bootstrapping, so that it really just runs N times, then it is faster. Here is a minimal example of the problem:
clear all
global iterationcounter
tic
iterationcounter=0;
data=unifrnd(0,1,1,1000); %draw vector of 1000 random numbers
bootstat = bootstrp(100,#testmean,data); %evaluate function for 100 bootstrap samples
toc
which uses the function
function [ m ] = testmean( data )
global iterationcounter
m=mean(data);
iterationcounter=iterationcounter+1
end
The function should evaluate 100 samples, yet when I run the script, it will evaluate the function 101 times:
...
iterationcounter =
101
Elapsed time is 0.102291 seconds.
So why should one use this build-in Matlab function that appears to waste time?
bootstrp makes a call to bootfun (the function argument) for sanity checks (from the source code, in MATLAB 2015b, bootstrp.m, l.167 ff) :
% Sanity check bootfun call and determine dimension and type of result
try
% Get result of bootfun on actual data, force to a row.
bootstat = feval(bootfun,bootargs{:});
bootstat = bootstat(:)';
catch ME
m = message('stats:bootstrp:BadBootFun');
MEboot = MException(m.Identifier,'%s',getString(m));
ME = addCause(ME,MEboot);
rethrow(ME);
end
I would think that in a realistic application, N>>100, so the extra overhead is (much) less than a percent of he total runtime (not taking into account speed gains from possible parallelisation), so that should not matter that much?
i have 8 feature from a mat file
each of this feature divided 4 part (X_train , Y_train , X_test,Y_test)
for 10 times randomly obtained this parameter
now i should classify this feature according KNN
my code is here
kk=7;
bb=1;
mdl1= ClassificationKNN.fit([X1_train{bb};X2_train{bb};X3_train{bb};X4_train{bb};X5_train{bb};X6_train{bb};X7_train{bb};X8_train{bb};X9_train{bb};X10_train{bb};X11_train{bb};X12_train{bb}],[Y1_train{bb};Y2_train{bb};Y3_train{bb};Y4_train{bb};Y5_train{bb};Y6_train{bb};Y7_train{bb};Y8_train{bb};Y9_train{bb};Y10_train{bb};Y11_train{bb};Y12_train{bb}],'NumNeighbors',kk);
.
.
.
bb=10;
mdl10= ClassificationKNN.fit([X1_train{bb};X2_train{bb};X3_train{bb};X4_train{bb};X5_train{bb};X6_train{bb};X7_train{bb};X8_train{bb};X9_train{bb};X10_train{bb};X11_train{bb};X12_train{bb}],[Y1_train{bb};Y2_train{bb};Y3_train{bb};Y4_train{bb};Y5_train{bb};Y6_train{bb};Y7_train{bb};Y8_train{bb};Y9_train{bb};Y10_train{bb};Y11_train{bb};Y12_train{bb}],'NumNeighbors',kk);
as you seen this functions repeat 10 times for evaluate the 10 mdl
in the following i write this code to simplify the project
for j=1:10
for h=1:12
mdl{j}{h}=ClassificationKNN.fit([X_train{j}{h}],[Y_train{j}{h}]);
end
end
this code work proerly without (mdl{j}{h}) but if this sentence is used i have this error message ((Cell contents assignment to a non-cell array object))
anybode know what shall i do to fix this problem
thanks
at first you should define the mdl variable size
mdll= cell(10, 8);
then form this for loop
for j=1:10
mdll{j}= ClassificationKNN.fit([X_train{j}{1};X_train{j}{2};X_train{j}{3};X_train{j}{4};X_train{j}{5};X_train{j}{6};X_train{j}{7};X_train{j}{8};X_train{j}{9};X_train{j}{10};X_train{j}{11};X_train{j}{12}],[Y_train{j}{1};Y_train{j}{2};Y_train{j}{3};Y_train{j}{4};Y_train{j}{5};Y_train{j}{6};Y_train{j}{7};Y_train{j}{8};Y_train{j}{9};Y_train{j}{10};Y_train{j}{11};Y_train{j}{12}],'NumNeighbors',kk);
end
i checked it and work correctly
I have a basic code like this :
parfor i=1:8
[t,y]=ode15s(#rate,tspan,cin,options,i); % the option i is evaluated in the rate function
figure(1)
subplot(3,3,i+1)
plot(t,y)
hold on
end
Will any conflict arise because the variable name y is same in all iterations ?
No, every worker has its unique namespace.
However, a worker cannot open figures that display on the client (thanks for the reminder, #Edric), so everything after the call to ode15s will not produce any useful result.
To move the plotting outside the parfor loop, you can do the following (there are more efficient solutions, this one will work for sure):
tCell = cell(8,1);
yCell = cell(8,1);
parfor i=1:8
[tCell{i},yCell{i}]=ode15s(#rate,tspan,cin,options,i); % the option i is evaluated in the rate function
end
figure(1)
for i=1:8
subplot(3,3,i+1)
plot(tCell{i},yCell{i})
hold on
end
Following on from #Jonas' answer, just a note to point out that if you're using R2013b or later, and you wish to display graphics while waiting for your parallel computations to complete, you could use PARFEVAL, like in this example: http://www.mathworks.co.uk/help/distcomp/examples/parfeval-blackjack.html .