I am trying to compute the pairwise distances between two struct objects. This distance is symmetric. I have about N = 8000, such objects in an array.
So I need to compute N * (N+1)/2 distances only. How can I parallelize this computation, since each computation is independent ?
Here my objects are stored in Array X and I want to store the distances in Array A which is of size N*(N+1)/2. BDHMM() is a function which returns the distance between two objects.
I have tried the following Matlab Code.
N = 8000;
load inithmm.mat
size = N*(N+1)/2;
A = zeros(size,1);
matlabpool open local 4
parfor i = 1:N-1
i
T = [];
for j = i:N
if(j == i)
temp = 0;
else
temp = BDHMM(X(i),X(j));
end
T = [T; temp];
end
beg = size - (N + 1 - i)*(N + 2 - i)/2 + 1;
l = length(T);
A(beg:beg+l-1, 1) = T;
end
matlabpool close
I am getting the following error:
Error: The variable A in a parfor cannot be classified.
Please help.
You cannot assassin to indexes you calculate withing the parfor, Matlab needs to know in advance what sections of the matrix will be assassin by witch iteration. this makes sense if you think about it.
this should solve it:
N = 800;
size = N*(N+1)/2;
A = cell(N,1);
matlabpool open local 4
parfor i = 1:N-1
i
T = zeros(N-i+1,1);
for j = i:N
if(j == i)
T(j-i+1) = 0;
else
T(j-i+1) = BDHMM(X(i),X(j));
end
end
A{i, 1} = T;
end
matlabpool close
B=vertcat(A{:})
Related
I am trying to convert my code over to run with parfor, since as it is it takes a long time to run on its own. However I keep getting this error. I have search around on the website and have read people with similar problems, but none of those answers seem to fix my problem. This is my code:
r = 5;
Mu = 12.57e-9;
Nu = 12e6;
I = 1.8;
const = pi*Nu*Mu*r*I;
a = 55;
b = 69;
c = 206;
[m,n,p] = size(Lesion_Visible);
A = zeros(m,n,p);
parpool(2)
syms k
parfor J = 1:m
for I = 1:n
for K = 1:p
if Lesion_Visible(J,I,K) ~= 0
Theta = atand((J-b)/(I-a));
Rho = abs((I-a)/cosd(Theta))*0.05;
Z = abs(c-K)*0.05;
E = vpa(const*int(abs(besselj(0,Rho*k)*exp(-Z*k)*besselj(0,r*k)),0,20),5);
A (J,I,K) = E;
end
end
end
end
I'm trying to calculate the electric field in specific position on an array and matlab give me the error "The variable A in a parfor cannot be classified". I need help. Thanks.
As classification of variables in parfor loop is not permitted, you should try to save the output of each loop in a variable & then save the final output into the desired variable, A in your case!
This should do the job-
parfor J = 1:m
B=zeros(n,p); %create a padding matrix of two dimension
for I = 1:n
C=zeros(p); %create a padding matrix of one dimension
for K = 1:p
if Lesion_Visible(J,I,K) ~= 0
Theta = atand((J-b)./(I-a));
Rho = abs((I-a)./cosd(Theta))*0.05;
Z = abs(c-K).*0.05;
E = vpa(const.*int(abs(besselj(0,Rho.*k).*exp(-Z.*k).*besselj(0,r.*k)),0,20),5);
C(K) = E; %save output of innnermost loop to the padded matrix C
end
end
B(I,:)=C; % save the output to dim1 I of matrix B
end
A(J,:,:)=B; save the output to dim1 J of final matrix A
end
Go through the following for better understanding-
http://www.mathworks.com/help/distcomp/classification-of-variables-in-parfor-loops.html
http://in.mathworks.com/help/distcomp/sliced-variable.html
I'm using a code that calculates expectation value of probabilities. This code contains a while-loop that finds all possible combinations and adds up products of probability combinations. However, when the number of elements becomes large(over 40) it takes too much time, and I want to make the code faster.
The code is as follow-
function pcs = combsum(N,K,prbv)
nprbv=1-prbv; %prbv: probability vector
WV = 1:K; % Working vector.
lim = K; % Sets the limit for working index.
inc = 0; % Controls which element of WV is being worked on.
pcs = 0;
stopp=0;
while stopp==0
if logical((inc+lim)-N)
stp = inc; % This is where the for loop below stops.
flg = 0; % Used for resetting inc.
else
stp = 1;
flg = 1;
end
for jj = 1:stp
WV(K + jj - inc) = lim + jj; % Faster than a vector assignment.
end
PV=nprbv;
PV(WV)=prbv(WV);
pcs=prod(PV)+pcs;
inc = inc*flg + 1; % Increment the counter.
lim = WV(K - inc + 1 ); % lim for next run.
if (inc==K)&&(lim==N-K)
stopp=1;
WV = (N-K+1):N;
PV=nprbv;
PV(WV)=prbv(WV);
pcs=prod(PV)+pcs;
end
end
Is there a way to reduce calculation time? I wonder if parallel computing using GPU would help.
I tried to remove dependent variables in a loop for parallel computing, and I made a matrix of possible combinations using 'combnk' function. This worked faster.
nprbv=1-prbv; %prbv : a probability vector
N = 40;
K = 4;
n_combnk = size(combnk(1:N,K),1);
PV_mat = repmat(nprbv,n_combnk,1);
cnt = 0;
tic;
for i = 1:N-K+1
for j = i+1:N-K+2
for k = j+1:N-K+3
for l = k+1:N-K+4
cnt = cnt+1;
PV_mat(cnt,i) = prbv(i);
PV_mat(cnt,j) = prbv(j);
PV_mat(cnt,k) = prbv(k);
PV_mat(cnt,l) = prbv(l);
end
end
end
end
toc;
tic;
pcs_rr = sum(prod(PV_mat,2));
toc;
However, when K value gets larger, an out-of-memory problem happens in building a combination matrix(PV_mat). How can I break up the big matrix into small ones to avoid memory problem?
I have a vector named signal consisting of 300001 values. In each iteration of the for loop, I want to pick up 2000 consecutive values from this vector and store it in another vector X (X is 1*2000 vector)
The code is as follows:
D = 1:300001;
A = zeros(1,2000);
r=1;
n=0;
m=1;
for i=1:300001
for p = (1+(2000*n)):(r*2000)
while m<2000
A(1,m)= signal(1,p);
%disp (m);
m = m+1;
end
end
r = r+1;
n = n+1;
m = 1;
end
But it gives me the error "Index exceeds matrix dimensions.
Can somebody help me out with a better way to do it?
this would work
signal = ones(1,30000);
index1= 1:2000:length(signal);
index2= 2000:2000:length(signal);
for i=1:length(index1)
A = signal(index1(i):index2(i));
end
or this
signal = ones(1,30000);
temp = reshape(signal,2000,[]);
for i = 1:size(temp,2)
A=temp(:,i);
end
I'm attempting to create a loop that reads through a matrix (A) and stores the non-zero values into a new matrix (w). I'm not sure what is wrong with my code.
function [d,w] = matrix_check(A)
[nrow ncol] = size(A);
total = 0;
for i = 1:nrow
for j = 1:ncol
if A(i,j) ~= 0
total = total + 1;
end
end
end
d = total;
w = [];
for i = 1:nrow
for j = 1:ncol
if A(i,j) ~= 0
w = [A(i,j);w];
end
end
end
The second loop is not working (at at least it is not printing out the results of w).
You can use nonzeros and nnz:
w = flipud(nonzeros(A)); %// flipud to achieve the same order as in your code
d = nnz(A);
The second loop is working. I'm guessing you're doing:
>> matrix_check(A)
And not:
>> [d, w] = matrix_check(A)
MATLAB will only return the first output unless otherwise specified.
As an aside, you can accomplish your task utilizing MATLAB's logical indexing and take advantage of the (much faster, usually) array operations rather than loops.
d = sum(sum(A ~= 0));
w = A(A ~= 0);
I am doing a very large calculation (atmospheric absorption) that has a lot of individual narrow peaks that all get added up at the end. For each peak, I have pre-calculated the range over which the value of the peak shape function is above my chosen threshold, and I am then going line by line and adding the peaks to my spectrum. A minimum example is given below:
X = 1:1e7;
K = numel(a); % count the number of peaks I have.
spectrum = zeros(size(X));
for k = 1:K
grid = X >= rng(1,k) & X <= rng(2,k);
spectrum(grid) = spectrum(grid) + peakfn(X(grid),a(k),b(k),c(k)]);
end
Here, each peak has some parameters that define the position and shape (a,b,c), and a range over which to do the calculation (rng). This works great, and on my machine it benchmarks at around 220 seconds to do a complete data set. However, I have a 4 core machine and I would eventually like to run this on a cluster, so I'd like to parallelize it and make it scaleable.
Because each loop relies on the results of the previous iteration, I cannot use parfor, so I am taking my first step into learning how to use spmd blocks. My first try looked like this:
X = 1:1e7;
cores = matlabpool('size');
K = numel(a);
spectrum = zeros(size(X),cores);
spmd
n = labindex:cores:K
N = numel(n);
for k = 1:N
grid = X >= rng(1,n(k)) & X <= rng(2,n(k));
spectrum(grid,labindex) = spectrum(grid,labindex) + peakfn(X(grid),a(n(k)),b(n(k)),c(n(k))]);
end
end
finalSpectrum = sum(spectrum,2);
This almost works. The program crashes at the last line because spectrum is of type Composite, and the documentation for 2013a is spotty on how to turn Composite data into a matrix (cell2mat does not work). This also does not scale well because the more cores I have, the larger the matrix is, and that large matrix has to get copied to each worker, which then ignores most of the data. Question 1: how do I turn a Composite data type into a useable array?
The second thing I tried was to use a codistributed array.
spmd
spectrum = codistributed.zeros(K,cores);
disp(size(getLocalPart(spectrum)))
end
This tells me that each worker has a single vector of size [K 1], which I believe is what I want, but when I try to then meld the above methods
spmd
spectrum = codistributed.zeros(K,cores);
n = labindex:cores:K
N = numel(n);
for k = 1:N
grid = X >= rng(1,n(k)) & X <= rng(2,n(k));
spectrum(grid) = spectrum(grid) + peakfn(X(grid),a(n(k)),b(n(k)),c(n(k))]); end
finalSpectrum = gather(spectrum);
end
finalSpectrum = sum(finalSpectrum,2);
I get Matrix dimensions must agree errors. Since it's in a parallel block, I can't use my normal debugging crutch of stepping through the loop and seeing what the size of each block is at each point to see what's going on. Question 2: what is the proper way to index into and out of a codistributed array in an spmd block?
Regarding question#1, the Composite variable in the client basically refers to a non-distributed variant array stored on the workers. You can access the array from each worker by {}-indexing using its corresponding labindex (e.g: spectrum{1}, spectrum{2}, ..).
For your code that would be: finalSpectrum = sum(cat(2,spectrum{:}), 2);
Now I tried this problem myself using random data. Below are three implementations to compare (see here to understand the difference between distributed and nondistributed arrays). First we start with the common data:
len = 100; % spectrum length
K = 10; % number of peaks
X = 1:len;
% random position and shape parameters
a = rand(1,K); b = rand(1,K); c = rand(1,K);
% random peak ranges (lower/upper thresholds)
ranges = sort(randi([1 len], [2 K]));
% dummy peakfn() function
fcn = #(x,a,b,c) x+a+b+c;
% prepare a pool of MATLAB workers
matlabpool open
1) Serial for-loop:
spectrum = zeros(size(X));
for i=1:size(ranges,2)
r = ranges(:,i);
idx = (r(1) <= X & X <= r(2));
spectrum(idx) = spectrum(idx) + fcn(X(idx), a(i), b(i), c(i));
end
s1 = spectrum;
clear spectrum i r idx
2) SPMD with Composite array
spmd
spectrum = zeros(1,len);
ind = labindex:numlabs:K;
for i=1:numel(ind)
r = ranges(:,ind(i));
idx = (r(1) <= X & X <= r(2));
spectrum(idx) = spectrum(idx) + ...
feval(fcn, X(idx), a(ind(i)), b(ind(i)), c(ind(i)));
end
end
s2 = sum(vertcat(spectrum{:}));
clear spectrum i r idx ind
3) SPMD with co-distributed array
spmd
spectrum = zeros(numlabs, len, codistributor('1d',1));
ind = labindex:numlabs:K;
for i=1:numel(ind)
r = ranges(:,ind(i));
idx = (r(1) <= X & X <= r(2));
spectrum(labindex,idx) = spectrum(labindex,idx) + ...
feval(fcn, X(idx), a(ind(i)), b(ind(i)), c(ind(i)));
end
end
s3 = sum(gather(spectrum));
clear spectrum i r idx ind
All three results should be equal (to within an acceptably small margin of error)
>> max([max(s1-s2), max(s1-s3), max(s2-s3)])
ans =
2.8422e-14