Hello I want to use parallelize my MATLAB code to run High computing server. It is code to make image database for Deep learning. To parallelize the code I found the I have to for parfor loop. But I used that with the first loop or with the second loop it shows me error parfor cannot be run due to the variable imdb and image_counter. Anyone please help me to change the code to work with parfor
for i = 1:length(cur_images)
X = sprintf('image Numb: %d ',i);
disp(X)
cur_image = load(cur_images{i,:});
cur_image=(cur_image.Image.crop);
%----------------------------------------------
cur_image = imresize(cur_image, image_size);
if(rgb < 1)
imdb.images.data(:,:,1,image_counter) = cur_image;
else
imdb.images.data(:,:,1,image_counter) = cur_image(:,:,1);
imdb.images.data(:,:,2,image_counter) = cur_image(:,:,2);
imdb.images.data(:,:,3,image_counter) = cur_image(:,:,3);
imdb.images.data(:,:,4,image_counter) = cur_image(:,:,4);
imdb.images.data(:,:,5,image_counter) = cur_image(:,:,5);
imdb.images.data(:,:,6,image_counter) = cur_image(:,:,6);
imdb.images.data(:,:,7,image_counter) = cur_image(:,:,7);
imdb.images.data(:,:,8,image_counter) = cur_image(:,:,8);
imdb.images.data(:,:,9,image_counter) = cur_image(:,:,9);
imdb.images.data(:,:,10,image_counter) = cur_image(:,:,10);
end
imdb.images.set( 1,image_counter) = set;
image_counter = image_counter + 1;
end
The main problem here is that you can't assign to fields of a structure inside parfor in the way that you're trying to do. Also, your outputs need to be indexed by the loop variable to qualify as "sliced" - i.e. don't use image_counter. Putting this together, you need something more like:
% Make a numeric array to store the output.
data_out = zeros([image_size, 10, length(cur_images)]);
parfor i = 1:length(cur_images)
cur_image = load(cur_images{i, :});
cur_image=(cur_image.Image.crop);
cur_image = imresize(cur_image, image_size);
% Now, assign into 'data_out'. A little care needed
% here.
if rgb < 1
data_tmp = zeros([image_size, 10]);
data_tmp(:, :, 1) = cur_image;
else
data_tmp = cur_image;
end
data_out(:, :, :, i) = data_tmp;
end
imdb.images.data = data_out;
Related
I have a list of parameters and I need to evaluate my method over this list. Right now, I am doing it this way
% Parameters
params.corrAs = {'objective', 'constraint'};
params.size = {'small', 'medium', 'large'};
params.density = {'uniform', 'non-uniform'};
params.k = {3,4,5,6};
params.constraintP = {'identity', 'none'};
params.Npoints_perJ = {2, 3};
params.sampling = {'hks', 'fps'};
% Select the current parameter
for corrAs_iter = params.corrAs
for size_iter = params.size
for density_iter = params.density
for k_iter = params.k
for constraintP_iter = params.constraintP
for Npoints_perJ_iter = params.Npoints_perJ
for sampling_iter = params.sampling
currentParam.corrAs = corrAs_iter;
currentParam.size = size_iter;
currentParam.density = density_iter;
currentParam.k = k_iter;
currentParam.constraintP = constraintP_iter;
currentParam.Npoints_perJ = Npoints_perJ_iter;
currentParam.sampling = sampling_iter;
evaluateMethod(currentParam);
end
end
end
end
end
end
end
I know it looks ugly and if I want to add a new type of parameter, I have to write another for loop. Is there any way, I can vectorize this? Or maybe use 2 for loops instead of so many.
I tried the following but, it doesn't result in what I need.
for i = 1:numel(fields)
% if isempty(params.(fields{i}))
param.(fields{i}) = params.(fields{i})(1);
params.(fields{i})(1) = [];
end
What you need is all combinations of your input parameters. Unfortunately, as you add more parameters the storage requirements will grow quickly (and you'll have to use a large indexing matrix).
Instead, here is an idea which uses linear indicies of a (never created) n1*n2*...*nm matrix, where ni is the number of elements in each field, for m fields.
It is flexible enough to cope with any amount of fields being added to params. Not performance tested, although as with any "all combinations" operation you should be wary of the non-linear increase in computation time as you add more fields to params, note prod(sz)!
The code I've shown is fast, but the performance will depend entirely on which operations you do in the loop.
% Add parameters here
params.corrAs = {'objective', 'constraint'};
params.size = {'small', 'medium', 'large'};
params.density = {'uniform', 'non-uniform'};
% Setup
f = fieldnames( params );
nf = numel(f);
sz = NaN( nf, 1 );
% Loop over all parameters to get sizes
for jj = 1:nf
sz(jj) = numel( params.(f{jj}) );
end
% Loop for every combination of parameters
idx = cell(1,nf);
for ii = 1:prod(sz)
% Use ind2sub to switch from a linear index to the combination set
[idx{:}] = ind2sub( sz, ii );
% Create currentParam from the combination indices
currentParam = struct();
for jj = 1:nf
currentParam.(f{jj}) = params.(f{jj}){idx{jj}};
end
% Do something with currentParam here
% ...
end
Asides:
I'm using dynamic field name references for indexing the fields
I'm passing multiple outputs into a cell array from ind2sub, so you can handle a variable number of field names when ind2sub has one output for each dimension (or field in this use-case).
Here is a vectorized solution :
names = fieldnames(params).';
paramGrid = cell(1,numel(names));
cp = struct2cell(params);
[paramGrid{:}] = ndgrid(cp{:});
ng = [names;paramGrid];
st = struct(ng{:});
for param = st(:).'
currentParam = param;
end
Instead of nested loops we can use ndgrid to create the cartesian product of the cell entries so we can find all combinations of cell entries without loop.
I am doing the following to generate random matrices with eigenvalues in a specific range:
function mat = randEig(dim, rReal)
D=diff(rReal).*rand(dim,1)+rReal(1);
P=rand(dim);
mat=P*diag(D)/P;
end
But I also want to be able to generate random real matrices with complex (conjugate) eigenvalues. How would one do that? The similarity transformation trick would return complex matrices.
EDIT: Okay I managed to do it by piggybacking on MATLAB's cdf2rdf function (which is basically the second function below).
function mat = randEig(dim, rangeEig, nComplex)
if 2*nComplex > dim
error('Cannot happen');
end
if nComplex
cMat=diff(rangeEig).*rand(dim-2*nComplex,1)+rangeEig(1);
for k=1:nComplex
rpart=(diff(rangeEig).*rand(1,1)+rangeEig(1))*ones(2,1);
ipart=(diff(rangeEig).*rand(1,1)+rangeEig(1))*i;
ipart=[ipart; -ipart];
cMat=[cMat; rpart+ipart];
end
else
cMat=diff(rangeEig).*rand(dim,1)+rangeEig(1);
end
D=cMat;
realDform = comp2rdf(diag(D));
P=rand(dim);
mat=P*realDform/P;
end
function dd = comp2rdf(d)
i = find(imag(diag(d))');
index = i(1:2:length(i));
if isempty(index)
dd=d;
else
if (max(index)==size(d,1)) | any(conj(d(index,index))~=d(index+1,index+1))
error(message('Complex conjugacy not satisfied'));
end
j = sqrt(-1);
t = eye(length(d));
twobytwo = [1 1;j -j];
for i=index
t(i:i+1,i:i+1) = twobytwo;
end
dd=t*d/t;
end
end
But the code is ugly, mainly the way rand is called multiple times is annoying). If someone wants to post an answer that calls rand once and manages to do the trick I will surely accept and upvote.
I made it either a single call or two calls with this:
function mat = randEig(dim, rangeEig, nComplex)
if 2*nComplex > dim
error('Cannot happen');
end
if nComplex
cMat=diff(rangeEig).*rand(2*nComplex,1)+rangeEig(1);
cPart=cMat(1:nComplex)*i;
cMat(1:nComplex)=[];
cPart=upsample(cPart,2);
cPart=cPart+circshift(-cPart,1);
cMat=upsample(cMat,2);
cMat=cMat+circshift(cMat,1);
cMat=cMat+cPart;
cMat=[diff(rangeEig).*rand(dim-2*nComplex,1)+rangeEig(1); cMat];
else
cMat=diff(rangeEig).*rand(dim,1)+rangeEig(1);
end
D=cMat;
realDform = comp2rdf(diag(D));
P=rand(dim);
mat=P*realDform/P;
end
function dd = comp2rdf(d)
i = find(imag(diag(d))');
index = i(1:2:length(i));
if isempty(index)
dd=d;
else
if (max(index)==size(d,1)) | any(conj(d(index,index))~=d(index+1,index+1))
error(message('Complex conjugacy not satisfied'));
end
j = sqrt(-1);
t = eye(length(d));
twobytwo = [1 1;j -j];
for i=index
t(i:i+1,i:i+1) = twobytwo;
end
dd=t*d/t;
end
end
If someone can make it a single call or shorter/more elegant code they are welcome to post an answer.
Im trying to make a loop for doing the same operation to a lot of .mov files in matlab. The code i have right now looks like this:
close all
clear all
clc
movFiles = dir('*.mov');
numFiles = length(movFiles);
mydata = cell(1,numFiles);
% mydata = zeros(numFiles);
for k = 1:numFiles
mydata{1,k} = VideoReader(movFiles(k).name);
end
for k=1:numFiles
bk_downsample = 5; %The downsample factor for frame averaging
%disp('Opening video...') %lower number =longer computation time
vob = mydata;
frame = vob.read(inf); %Reads to end = vob knows the number of frames
vidHeight = vob.Height;
vidWidth = vob.Width;
nFrames = vob.NumberOfFrames;
%% First-iteration background frame
background_frame = double(frame*0);
disp('Calculating background...')
for k = 1:bk_downsample:nFrames
background_frame = background_frame + double(read(vob, k));
disp(k/(nFrames)*100)
end
%background_frame = uint8(bk_downsample*background_frame/(nFrames));
background_frame = bk_downsample*background_frame/(nFrames);
%imshow(background_frame)
%% Second-iteration background frame
%This section re-calculates the background frame while attempting to
%minimize the effect of moving objects in the calculation
background_frame2 = double(frame*0);
pixel_sample_density = im2bw(double(frame*0));
diff_frame = double(frame*0);
stream_frame = diff_frame(:,:,1);
bk_downsample = 10;
figure
hold on
for k = 1:bk_downsample:nFrames
diff_frame = imabsdiff(double(read(vob, k)), background_frame);
diff_frame = 1-im2bw(uint8(diff_frame),.25);
pixel_sample_density = pixel_sample_density + diff_frame;
stream_frame = stream_frame + (1-diff_frame)/(nFrames/bk_downsample);
nonmoving = double(read(vob, k));
nonmoving(:,:,1) = nonmoving(:,:,1).*diff_frame;
nonmoving(:,:,2) = nonmoving(:,:,2).*diff_frame;
nonmoving(:,:,3) = nonmoving(:,:,3).*diff_frame;
background_frame2 = background_frame2 + nonmoving;
%pause
disp(k/(nFrames)*100)
end
background_frame2(:,:,1) = background_frame2(:,:,1)./pixel_sample_density;
background_frame2(:,:,2) = background_frame2(:,:,2)./pixel_sample_density;
background_frame2(:,:,3) = background_frame2(:,:,3)./pixel_sample_density;
imshow(uint8(background_frame2))
%imshow(stream_frame)
filename = ['Ring_' num2str(k) '_background_' num2str(img) '.jpg'];
imwrite((uint8(background_frame2)),filename)
end
I know that the error starts with vob=mydata; but im not sure how to correct it, hope that someone is able to help me since it would save me a lot of time in my data-analysis.
Have a great day! :)
Your code doesn't make much sense... You're creating a cell array:
mydata = cell(1,numFiles);
%// . . .
mydata{1,k} = . . .
but however you try to access it like a structure:
vob = mydata;
frame = vob.read(inf);
If I'd guess, then your error stems from you forgetting to index in the cell array, i.e.:
vob = mydata{k};
Other programming oddity I noticed in your code is the fact you're using the same looping variable k, in two nested for lops, the outer one being on k=1:numFiles and the inner ones being on k=1:bk_downsample:nFrames; don't do that unless you're trying to drive yourself crazy while figuring out why your for loop executes only once. Name them k1 for the outer loop and k2 for the inner loops and you'll be happier.
I'm no expert in video processing, but for me it looks like your line should be:
vob=mydata{1,k};
That's why that error shows up, because you are treating a cell of structs as if it was a single struct.
I'm unable to execute the following code:
parallel_mode = true; %To process multiple images at the same time
parallel_degree = 4; %Number of threads that will be created
if parallel_mode
if matlabpool('size') == 0
matlabpool(parallel_degree);
elseif matlabpool('size') ~= parallel_degree
matlabpool close;
matlabpool(parallel_degree);
end
end
%Loading dictionary
try
load(dictionary_path);
catch
error(['Was impossible to load the dictionary in path: ' dictionary_path ', Please, check the path. ' ...
'Maybe you should use dictionary_training function to create it.'])
end
%% Processing test images
test_images = dir([test_im_path, pattern]);
num_test_images = size(test_images,1);
%Pre-allocating memory to speed-up
estimated_count = zeros(1,num_test_images);
true_count = zeros(1,num_test_images);
estimated_upper_count = zeros(1,num_test_images);
estimated_lower_count = zeros(1,num_test_images);
true_upper_count = zeros(1,num_test_images);
true_lower_count = zeros(1,num_test_images);
%Calculating dimensions of the image subregion where we can count
im_test = imread([test_im_path test_images(1).name]);
dis = round(patch_size/2);
dim_x = dis:size(im_test,2)-dis+1;
dim_y = dis:size(im_test,1)-dis+1;
toGaussian = fspecial('gaussian', hsize, sigma);
parfor a=1:num_test_images
disp(['Processing image #' num2str(a) ' of ' num2str(num_test_images) '...']);
im_test = imread([test_im_path test_images(a).name]);
[~, name, extension] = fileparts(test_images(a).name);
im_ground_truth = imread([ground_truth_path name 'dots' extension]);
disp('Extracting features...');
features = extract_features(im_test, features_type, dic_signal, sparsity, patch_size, mean_rem_flag);
features = full(features); %ND-sparse arrays are not supported.
%Re-arranging features
features = reshape(features', size(dim_y,2), size(dim_x,2), dic_size);
%Normalizing features
max_factors_3D = repmat(max_factors_depth, [size(features,1), size(features,2)]);
max_offset_3D = repmat(max_offset_depth, [size(features,1), size(features,2)]);
features = (features-max_offset_3D)./max_factors_3D;
%%Some stuff
....
end
When i execute it i get:
Undefined function or variable 'dic_signal'.
when it arrives to the extract_features function. However, in the single thread version (with for instead of parfor) it works correctly.
Someone can give me any hint?.
Thank you.
EDIT:
dic_signal is defined and correctly loaded in load(dictionary_path);
I suspect the load command doesn't load the variables in the workers workspace, only on your MATLAB instance workspace, which is why it works in a normal for loop, but not with a parfor loop. You might want to try instead:
pctRunOnAll load(dictionary_path)
to ensure the correct data is loaded into the workspace of each of the workers.
I am trying to parallize two of my for-loops and run it on a remote cluster.
I am using matlabpool open local 12 at the beginning with matlabpool close at the end. The problem I am running into is that my parfor-loop cannot use my matric properly and I am not sure how I would rewrite it so that it works.
H = hadamard(n);
H = [H;-H];
P = setdiff(P,H,'rows');
[r,c] = size(P);
A = zeros(n,r);
parfor i=1:r
for j=1:n
d = P(i,:) + H(j,:);
A(j,i) = sum(d(:) ~= 0);
end
end
and:
u2Had = cell(2,r);
parfor i =1:r
u2Had{1,i} = min(A(:,i));
MinHadIndex = find(A(:,i) == u2Had{1,i});
u2Had{2,i} = MinHadIndex;
end
Those are the two segments of the code I am trying to parallize. Any help is much appreciated and if I need to add anymore information please ask.
I don't know what your problem is in the first part as it works fine (perhaps if you defined P better)
regarding the second part, you can only send information to and from parloops in narrow cases.
Here change your code to the following:
u2HadT = cell(1,r);
parfor i =1:r
temp = min(A(:,i));
MinHadIndex = find(A(:,i) == temp);
u2HadT{i} = {temp;MinHadIndex};
end
u2Had = cell(2,r);
for i =1:r
u2Had{1,i} = u2HadT{i}(1);
u2Had{2,i} = u2HadT{i}(2);
end