I want to train my network using matlab and matconvnet-1.0-beta25.
My problem is regression and I use pdist as loss function to get mse.
The inputs data is 56*56*64*6000 and the targets data is 56*56*64*6000 and network architecture is as follows:
opts.networkType = 'simplenn' ;
opts = vl_argparse(opts, varargin) ;
lr = [.01 2] ;
% Define network CIFAR10-quick
net.layers = {} ;
% Block 1
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.01*randn(5,5,64,32, 'single'), zeros(1, 32, 'single')}}, ...
'learningRate', lr, ...
'stride', 1, ...
'pad', 2) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.05*randn(5,5,32,16, 'single'), zeros(1,16,'single')}}, ...
'learningRate', .1*lr, ...
'stride', 1, ...
'pad', 2) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.01*randn(5,5,16,8, 'single'), zeros(1, 8, 'single')}}, ...
'learningRate', lr, ...
'stride', 1, ...
'pad', 2) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.05*randn(5,5,8,16, 'single'), zeros(1,16,'single')}}, ...
'learningRate', .1*lr, ...
'stride', 1, ...
'pad', 2) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.01*randn(5,5,16,32, 'single'), zeros(1, 32, 'single')}}, ...
'learningRate', lr, ...
'stride', 1, ...
'pad', 2) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.05*randn(5,5,32,64, 'single'), zeros(1,64,'single')}}, ...
'learningRate', .1*lr, ...
'stride', 1, ...
'pad', 2) ;
net.layers{end+1} = struct('type', 'relu') ;
% Loss layer
net.layers{end+1} = struct('type', 'pdist') ;
% Meta parameters
net.meta.inputSize = [56 56 64] ;
net.meta.trainOpts.learningRate = [0.0005*ones(1,30) 0.0005*ones(1,10) 0.0005*ones(1,5)] ;
net.meta.trainOpts.weightDecay = 0.0001 ;
net.meta.trainOpts.batchSize = 100 ;
net.meta.trainOpts.numEpochs = numel(net.meta.trainOpts.learningRate) ;
% Fill in default values
net = vl_simplenn_tidy(net) ;
I change getSimpleNNBatch(imdb, batch) function in ncnn_train (the name of mine) as follows:
function [images, labels] = getSimpleNNBatch(imdb, batch)
images = imdb.images.data(:,:,:,batch) ;
labels = imdb.images.labels(:,:,:,batch) ;
if rand > 0.5, images=fliplr(images) ;
end
because my label is multi-dimensional.
Also I change errorFunction in cnn_train from multiclasses to none:
opts.errorFunction = 'none' ;
and change the error variable from:
% accumulate errors
error = sum([error, [...
sum(double(gather(res(end).x))) ;
reshape(params.errorFunction(params, labels, res),[],1) ; ]],2) ;
to:
% accumulate errors
error = sum([error, [...
mean(mean(mean(double(gather(res(end).x))))) ;
reshape(params.errorFunction(params, labels, res),[],1) ; ]],2) ;
My first question is why the res(end).x third dimension in above command is one instead of 64? this is 56*56*1*100 (100 is the batch).
Have I made a mistake?
here is the results:
train: epoch 01: 2/ 40: 10.1 (27.0) Hz objective: 21360.722
train: epoch 01: 3/ 40: 13.0 (30.0) Hz objective: 67328685.873
...
train: epoch 01: 39/ 40: 29.7 (29.6) Hz objective: 5179175.587
train: epoch 01: 40/ 40: 29.8 (30.6) Hz objective: 5049697.440
val: epoch 01: 1/ 10: 87.3 (87.3) Hz objective: 49.512
val: epoch 01: 2/ 10: 88.9 (90.5) Hz objective: 50.012
...
val: epoch 01: 9/ 10: 88.2 (88.2) Hz objective: 49.936
val: epoch 01: 10/ 10: 88.1 (87.3) Hz objective: 49.962
train: epoch 02: 1/ 40: 30.2 (30.2) Hz objective: 49.650
train: epoch 02: 2/ 40: 30.3 (30.4) Hz objective: 49.704
...
train: epoch 02: 39/ 40: 30.2 (31.6) Hz objective: 49.739
train: epoch 02: 40/ 40: 30.3 (31.0) Hz objective: 49.722
val: epoch 02: 1/ 10: 91.8 (91.8) Hz objective: 49.687
val: epoch 02: 2/ 10: 92.0 (92.2) Hz objective: 49.831
...
val: epoch 02: 9/ 10: 92.0 (88.5) Hz objective: 49.931
val: epoch 02: 10/ 10: 91.9 (91.1) Hz objective: 49.962
train: epoch 03: 1/ 40: 31.7 (31.7) Hz objective: 49.014
train: epoch 03: 2/ 40: 31.2 (30.8) Hz objective: 49.237
...
here is my network schema image
Two inputs of pdist have got nxmx64x100 size as below and as this mentioned, the output of pdist has got the same height and width, but depth equal to one. About the correctness of error definition, you should debug and check the size and definition accurately.
Related
This is a radar plot script for MATLAB I copied from Google Search. I was hoping to use radar plot with different scales with 4 or 5 axes.
How would I fill the graph of this radar plot? Should I use patch or fill? I know this shouldn't take a long time to do this. My math isn't that great.
Thanks in advance,
Gary
function [f, ca, o] = spider(data,tle,rng,lbl,leg,f)
% create a spider plot for ranking the data
% function [f, ca, o] = spider(data,tle,rng,lbl,leg,f)
%
% inputs 6 - 5 optional
% data input data (NxM) (# axes (M) x # data sets (N)) class real
% tle spider plot title class char
% rng peak range of the data (Mx1 or Mx2) class real
% lbl cell vector axes names (Mxq) in [name unit] pairs class cell
% leg data set legend identification (1xN) class cell
% f figure handle or plot handle class real
%
% outptus 3 - 3 optional
% f figure handle class integer
% x axes handle class real
% o series object handles class real
%
% michael arant - jan 30, 2008
%
% to skip any parameter, enter null []
%
% examples
%
% spider([1 2 3; 4 5 6; 7 8 9; 10 11 12; 13 14 15;16 17 18; ...
% 19 20 21; 22 23 24; 25 26 27]','test plot');
%
% spider([1 2 3 4; 4 5 6 7; 7 8 9 10; 10 11 12 13; 13 14 15 16; ...
% 16 17 18 19; 19 20 21 22; 22 23 24 25; 25 26 27 28],'test plot', ...
% [[0:3:24]' [5:3:29]'],[],{'Case 1' 'Case 2' 'Case 3' 'Case 4'});
%
% spider([1 2 3 4; 4 5 6 7; 7 8 9 10; 10 11 12 13; 13 14 15 16; ...
% 16 17 18 19; 19 20 21 22; 22 23 24 25; 25 26 27 28],'test plot', ...
% [],[],{'Case 1' 'Case 2' 'Case 3' 'Case 4'});
%
% figure; clf; set(gcf,'color','w'); s = zeros(1,4);
% for ii = 1:4; s(ii) = subplot(2,2,ii); end
%
% spider([1 2 3; 4 5 6; 7 8 9; 10 11 12; 13 14 15;16 17 18; ...
% 19 20 21; 22 23 24; 25 26 27]','test plot 1',[],[],[],s(1));
%
% spider([1 2 3; 4 5 6; 7 8 9; 10 11 12; 13 14 15;16 17 18; ...
% 19 20 21; 22 23 24; 25 26 27],'test plot 2',[0 30],[],[],s(2));
%
% spider([1 2 3 4; 4 5 6 7; 7 8 9 10; 10 11 12 13; 13 14 15 16; ...
% 16 17 18 19; 19 20 21 22; 22 23 24 25; 25 26 27 28]','test plot 3', ...
% [0 30],{'Label 1' 'Unit 1'; 'Label 2' 'Unit 2'; 'Label 3' 'Unit 3'; ...
% 'Label 4' 'Unit 4'},{'Case 1' 'Case 2' 'Case 3' 'Case 4' 'Case 5' ...
% 'Case 6' 'Case 7' 'Case 8' 'Case 9'},s(3));
%
% spider([1 2 3 4; 4 5 6 7; 7 8 9 10; 10 11 12 13; 13 14 15 16; ...
% 16 17 18 19; 19 20 21 22; 22 23 24 25; 25 26 27 28],'test plot 4', ...
% [[0:3:24]' [5:3:29]'],[],{'Case 1' 'Case 2' 'Case 3' 'Case 4'},s(4));
% data check
if nargin < 1; help spider; error('Need data to plot'); end
% size segments and number of cases
[r c] = size(data);
% exit for too few axes
if r < 3
errordlg('Must have at least three measuremnt axes')
error('Program Termination: Must have a minimum of three axes')
end
% title
if ~exist('tle','var') || isempty(tle) || ~ischar(tle)
tle = 'Spider Plot';
end
% check for maximum range
if ~exist('rng','var') || isempty(rng) || ~isreal(rng)
% no range given or range is in improper format
% define new range
rng = [min([min(data,[],2) zeros(r,1)],[],2) max(data,[],2)];
% check for negative minimum values
if ~isempty(ismember(-1,sign(data)))
% negative value found - adjust minimum range
for ii = 1:r
% negative range for axis ii - set new minimum
if min(data(ii,:)) < 0
rng(ii,1) = min(data(ii,:)) - ...
0.25 * (max(data(ii,:)) - min(data(ii,:)));
end
end
end
elseif size(rng,1) ~= r
if size(rng,1) == 1
% assume that all axes have commom scale
rng = ones(r,1) * rng;
else
% insuffent range definition
uiwait(msgbox(char('Range size must be Mx1 - number of axes x 1', ...
sprintf('%g axis ranges defined, %g axes exist',size(rng,1),r))))
error(sprintf('%g axis ranges defined, %g axes exist',size(rng,1),r))
end
elseif size(rng,2) == 1
% assume range is a maximum range - define minimum
rng = sort([min([zeros(r,1) min(data,[],2) - ...
0.25 * (max(data,[],2) - min(data,[],2))],[],2) rng],2);
end
% check for axis labels
if ~exist('lbl','var') || isempty(lbl)
% no labels given - define a default lable
lbl = cell(r,1); for ii = 1:r; lbl(ii) = cellstr(sprintf('Axis %g',ii)); end
elseif size(lbl,1) ~= r
if size(lbl,2) == r
lbl = lbl';
else
uiwait(msgbox(char('Axis labels must be Mx1 - number of axes x 1', ...
sprintf('%g axis labels defined, %g axes exist',size(lbl,1),r))))
error(sprintf('%g axis labels defined, %g axes exist',size(lbl,1),r))
end
elseif ischar(lbl)
% check for charater labels
lbl = cellstr(lbl);
end
if ~exist('leg','var') || isempty(leg)
% no data legend - define default legend
leg = cell(1,c); for ii = 1:c; leg(ii) = cellstr(sprintf('Set %g',ii)); end
elseif numel(leg) ~= c
uiwait(msgbox(char('Data set label must be 1XN - 1 x number of sets', ...
sprintf('%g data sets labeled, %g exist',numel(leg),c))))
error(sprintf('%g data sets labeled, %g exist',numel(leg),c))
end
% check for figure or axes
if ~exist('f','var')
% no figure or axes requested - generate new ones
f = figure; ca = gca(f); cla(ca); hold on; set(f,'color','w')
elseif ismember(f,get(0,'children')')
% existing figure - clear and set up
ca = gca(f); hold on;
elseif isint(f)
% generating a new figure
figure(f); ca = gca(f); cla(ca); hold on
else
% may be an axes - may be garbage
try
%is this an axes?
if ismember(get(f,'parent'),get(0,'children')')
% existing figure axes - use
ca = f; f = get(f,'parent'); hold on
end
catch
% make new figure and axes
disp(sprintf('Invalid axes handle %g passed. Generating new figure',f))
f = figure; ca = gca(f); cla(ca); hold on
end
end
% set the axes to the current text axes
axes(ca)
% set to add plot
set(ca,'nextplot','add');
% clear figure and set limits
set(ca,'visible','off'); set(f,'color','w')
set(ca,'xlim',[-1.25 1.25],'ylim',[-1.25 1.25]); axis(ca,'equal','manual')
% title
text(0,1.3,tle,'horizontalalignment','center','fontweight','bold');
% define data case colors
col = color_index(c);
% scale by range
angw = linspace(0,2*pi,r+1)';
mag = (data - rng(:,1) * ones(1,c)) ./ (diff(rng,[],2) * ones(1,c));
% scale trimming
mag(mag < 0) = 0; mag(mag > 1) = 1;
% wrap data (close the last axis to the first)
ang = angw(1:end-1); angwv = angw * ones(1,c); magw = [mag; mag(1,:)];
% make the plot
% define the axis locations
start = [zeros(1,r); cos(ang')]; stop = [zeros(1,r); sin(ang')];
% plot the axes
plot(ca,start,stop,'color','k','linestyle','-'); axis equal
% plot axes markers
inc = 0.25:.25:1; mk = .025 * ones(1,4); tx = 4 * mk; tl = 0:.25:1;
% loop each axis ang plot the line markers and labels
% add axes
for ii = 1:r
% plot tick marks
tm = plot(ca,[[cos(ang(ii)) * inc + sin(ang(ii)) * mk]; ...
[cos(ang(ii)) * inc - sin(ang(ii)) * mk]], ...
[[sin(ang(ii)) * inc - cos(ang(ii)) * mk] ;
[sin(ang(ii)) * inc + cos(ang(ii)) * mk]],'color','k');
% label the tick marks
for jj = 1:4
% temp = text([cos(ang(ii)) * inc(jj) + sin(ang(ii)) * tx(jj)], ...
% [sin(ang(ii)) * inc(jj) - cos(ang(ii)) * tx(jj)], ...
% num2str(chop(rng(ii,1) + inc(jj)*diff(rng(ii,:)),2)), ...
% 'fontsize',8);
temp = text([cos(ang(ii)) * inc(jj) + sin(ang(ii)) * tx(jj)], ...
[sin(ang(ii)) * inc(jj) - cos(ang(ii)) * tx(jj)], ...
num2str(rd(rng(ii,1) + inc(jj)*diff(rng(ii,:)),-2)), ...
'fontsize',8);
% flip the text alignment for lower axes
if ang(ii) >= pi
set(temp,'HorizontalAlignment','right')
end
end
% label each axis
temp = text([cos(ang(ii)) * 1.1 + sin(ang(ii)) * 0], ...
[sin(ang(ii)) * 1.1 - cos(ang(ii)) * 0], ...
char(lbl(ii,:)));
% flip the text alignment for right side axes
if ang(ii) > pi/2 && ang(ii) < 3*pi/2
set(temp,'HorizontalAlignment','right')
end
end
% plot the data
o = polar(ca,angw*ones(1,c),magw);
% set color of the lines
for ii = 1:c; set(o(ii),'color',col(ii,:),'linewidth',1.5); end
% apply the legend
temp = legend(o,leg,'location','best');
return
function [v] = rd(v,dec)
% quick round function (to specified decimal)
% function [v] = rd(v,dec)
%
% inputs 2 - 1 optional
% v number to round class real
% dec decimal loaction class integer
%
% outputs 1
% v result class real
%
% positive dec shifts rounding location to the right (larger number)
% negative dec shifts rounding location to the left (smaller number)
%
% michael arant
% Michelin Maericas Research and Development Corp
if nargin < 1; help rd; error('I/O error'); end
if nargin == 1; dec = 0; end
v = v / 10^dec;
v = round(v);
v = v * 10^dec;
function [val] = color_index(len)
% get unique colors for each item to plot
% function [val] = color_index(len)
%
% inputs 1
% len number of objects class integer
%
% outputs 1
% val color vector class real
%
% michael arant
if nargin < 1 || nargout < 1; help color_index; error('I / O error'); end
if len == 1
val = [0 0 0];
else
% initial color posibilities (no white)
% default color scale
col = [ 0 0 0
0 0 1
0 1 1
0 1 0
1 1 0
1 0 1
1 0 0];
% reduce if fewer than 6 items are needed (no interpolation needed)
switch len
case 1, col([2 3 4 5 6 7],:) = [];
case 2, col([2 3 4 5 6],:) = [];
case 3, col([3 4 5 6],:) = [];
case 4, col([3 5 6],:) = [];
case 5, col([5 6],:) = [];
case 6, col(6,:) = [];
end
% number of requested colors
val = zeros(len,3); val(:,3) = linspace(0,1,len)';
% interpolate to fill in colors
val(:,1) = interp1q(linspace(0,1,size(col,1))',col(:,1),val(:,3));
val(:,2) = interp1q(linspace(0,1,size(col,1))',col(:,2),val(:,3));
val(:,3) = interp1q(linspace(0,1,size(col,1))',col(:,3),val(:,3));
end
function [res] = isint(val)
% determines if value is an integer
% function [res] = isint(val)
%
% inputs 1
% val value to be checked class real
%
% outputs 1
% res result (1 is integer, 0 is not) class integer
%
% michael arant may 15, 2004
if nargin < 1; help isint; error('I / O error'); end
% numeric?
if ~isnumeric(val); error('Must be numeric'); end
% check for real number
if isreal(val) & isnumeric(val)
% check for integer
if round(val) == val
res = 1;
else
res = 0;
end
else
res = 0;
end
I have an issue I can't quite figure out. For a simulation I generate artificial data randomly, with randomly drawn variance and a mean of 0. To acheive this I first create a vector of possible variances and then randomly draw the index for the vector, like in the following example
%% Covariance Matrix
% Variances of explanatory variables
var1 = 0.1:0.1:100;
var2 = 0.1:0.1:100;
var3 = 0.1:0.1:100;
%% Randomly selecting variances
% if exist('s','var')
% rng(s) % Loading Random generator settings for replication
% else
% s=rng; % Saving Random generator settings for replication
% end
ind_1=randi([0 1000]);
ind_2=randi([0 1000]);
ind_3=randi([0 1000]);
var_11=var1(ind_1);
var_22=var2(ind_2);
var_33=var3(ind_3);
For some reason the random number generator seems to give me the same numbers in the first (ind_1=815, ind_2=906, ind_3=127) and in the second run (ind_1=914, ind_2=632, ind_3=97) after restarting matlab, if I generate the vector of variances first. I've been able to replicate that on different PC as well. Is there a feature that I'm overlooking or am I making, and I would imagine I am, a crucial mistake? (I am well aware that there are only pseudo random numbers in matlab, but this seems too pseudo for my taste.)
This is because the seed generator in matlab, when you start matlab is always the same take a look at this
>> rng('default')
>> ind_1=randi([0 1000])
ind_2=randi([0 1000])
ind_3=randi([0 1000])
ind_1=randi([0 1000])
ind_2=randi([0 1000])
ind_3=randi([0 1000])
rng('default')
ind_1=randi([0 1000])
ind_2=randi([0 1000])
ind_3=randi([0 1000])
ind_1 =
815
ind_2 =
906
ind_3 =
127
ind_1 =
914
ind_2 =
632
ind_3 =
97
ind_1 =
815
ind_2 =
906
ind_3 =
127
So the only thing that you have to do is change the initial seed every time you generate new numbers.
Execute before rng('shuffle'), it reseeds the generator using a different seed based on the current time.
>> rng('default')
>> [randi([0 1000]), randi([0 1000]), randi([0 1000])]
ans =
815 906 127
>> rng('shuffle')
>> [randi([0 1000]), randi([0 1000]), randi([0 1000])]
ans =
404 10 838
>> [randi([0 1000]), randi([0 1000]), randi([0 1000])]
ans =
31 459 534
>> rng('shuffle')
>> rng('shuffle')
>> [randi([0 1000]), randi([0 1000]), randi([0 1000])]
ans =
708 963 21
>> rng('default')
>> [randi([0 1000]), randi([0 1000]), randi([0 1000])]
ans =
815 906 127
>> [randi([0 1000]), randi([0 1000]), randi([0 1000])]
ans =
914 632 97
>> rng('default')
>> [randi([0 1000]), randi([0 1000]), randi([0 1000])]
ans =
815 906 127
Solved: Previously my dataset had around 1000 images. I increased it to 50 000 and now the neural network learns and works.
I have created a convolutional neural network for recognizing three emotions from facial expression(positive, neutral, negative emotion). Somehow, my error function does not get any better(error image). Training and validation error are constant for 100 epochs. What could be the reason?
Why the error is constant?
Here's my code:
function training(varargin)
setup ;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
rngNum = 1; % rng number for random weight initialization, e.g., 1,2,3
num_fcHiddenNeuron =1024; % # neurons in the fully-connected hidden layer
prob_fcDropout = 0.5; % dropout probability in the fully-connected hidden layer,
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% input data for training deep CNNs
imdb1 = load(['trainingdata']) ;
imdb2 = load(['testdata']) ;
imdb.images.data = cat(4, imdb1.images.data, imdb2.images.data);
imdb.images.labels = cat(2, imdb1.images.labels, imdb2.images.labels);
imdb.images.set = cat(2, imdb1.images.set, imdb2.images.set);
imdb.meta = imdb1.meta;
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
trainOpts.batchSize = 200 ;
trainOpts.numEpochs = 100 ;
trainOpts.gpus = [] ;
trainOpts.continue = true ;
trainOpts.learningRate = [0.004*ones(1,25), 0.002*ones(1,25), 0.001*ones(1,25), 0.0005*ones(1,25)];
trainOpts = vl_argparse(trainOpts, varargin);
%% Training Deep CNNs
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CNN configuration
net.layers = {} ;
% %
% % %% Conv1 - MaxPool1
rng(rngNum) %control random number generation
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.01*randn(3,3,1,32, 'single'), 0.1*ones(1, 32, 'single')}}, ...
'stride', 1, ...
'pad', 1, ...
'filtersLearningRate', 1, ...
'biasesLearningRate', 1, ...
'filtersWeightDecay', 1/5, ...
'biasesWeightDecay', 0) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'pool', ...
'method', 'max', ...
'pool', [2 2], ...
'stride', 2, ...
'pad', 0) ;
% %%% Conv2 - MaxPool2
rng(rngNum)
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.01*randn(3,3,32,32, 'single'), 0.1*ones(1, 32, 'single')}}, ...
'stride', 1, ...
'pad', 0, ...
'filtersLearningRate', 1, ...
'biasesLearningRate', 1, ...
'filtersWeightDecay', 1/5, ...
'biasesWeightDecay', 0) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'pool', ...
'method', 'max', ...
'pool', [2 2], ...
'stride', 2, ...
'pad', [1, 0, 1, 0]) ;
% %%% Conv3 - MaxPool3
rng(rngNum)
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.01*randn(3,3,32,64, 'single'), 0.1*ones(1, 64, 'single')}}, ...
'stride', 1, ...
'pad', 1, ...
'filtersLearningRate', 1, ...
'biasesLearningRate', 1, ...
'filtersWeightDecay', 1/5, ...
'biasesWeightDecay', 0) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'pool', ...
'method', 'max', ...
'pool', [2 2], ...
'stride', 2, ...
'pad', 0) ;
% %%% Fc Hidden
rng(rngNum)
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{0.001*randn(5,5,64,num_fcHiddenNeuron, 'single'), 0.01*ones(1, num_fcHiddenNeuron, 'single')}}, ...
'stride', 1, ...
'pad', 0, ...
'filtersLearningRate', 1, ...
'biasesLearningRate', 1, ...
'filtersWeightDecay', 1/5, ...
'biasesWeightDecay', 0) ;
net.layers{end+1} = struct('type', 'relu') ;
net.layers{end+1} = struct('type', 'dropout', ...
'rate', prob_fcDropout) ;
%
% %%% Fc Output
rng(rngNum)
net.layers{end+1} = struct('type', 'conv', ...
'weights', {{zeros(1,1,num_fcHiddenNeuron, 3, 'single'), zeros(1, 3, 'single')}}, ...
'stride', 1, ...
'pad', 0, ...
'filtersLearningRate', 1, ...
'biasesLearningRate', 1, ...
'filtersWeightDecay', 4, ...
'biasesWeightDecay', 0) ;
net.layers{end+1} = struct('type', 'softmaxloss') ;
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% starting to train deep CNN
[net,info] = cnn_train(net, imdb, getBatch(opts), trainOpts, 'val', find(imdb.images.set == 2)) ;
net.layers(end) = [] ;
function fn = getBatch(opts)
% -------------------------------------------------------------------------
fn = #(x,y) getSimpleNNBatch(x,y) ;
end
% -------------------------------------------------------------------------
function [images, labels] = getSimpleNNBatch(imdb, batch)
% -------------------------------------------------------------------------
images = imdb.images.data(:,:,:,batch) ;
labels = imdb.images.labels(1,batch) ;
end
I have the following code:
tic;
H = rand(100, 1000);
F = rand(1, 1000);
net = newff(H, F, [30, 10], { 'tansig' 'tansig'}, 'traingdx', 'learngdm', 'mse');
net.trainParam.epochs = 400;
net.performParam.regularization = 0.05;
net.divideParam.trainRatio = 1;
net.divideParam.valRatio = 0;
net.divideParam.testRatio = 0;
net.trainParam.showWindow = 0;
net.trainParam.showCommandLine = 0;
% net = train(net, H, F, 'useGPU', 'yes', 'showResources', 'yes'); % line 1
net = train(net, H, F, 'showResources', 'yes'); % line 2
toc;
with line 2 uncommented I get
Computing Resources:
GPU device #1, GeForce 800M
Elapsed time is 5.084222 seconds.
and with line 1 uncommented I get
Computing Resources:
MEX2
Elapsed time is 1.870803 seconds.
Why is GPU slower than CPU?
My GPU properties:
CUDADevice with properties:
Name: 'GeForce 800M'
Index: 1
ComputeCapability: '2.1'
SupportsDouble: 1
DriverVersion: 6
ToolkitVersion: 5
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [65535 65535 65535]
SIMDWidth: 32
TotalMemory: 2.1475e+09
FreeMemory: 1.9886e+09
MultiprocessorCount: 1
ClockRateKHz: 1475000
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1`
I have gpu
>> d = gpuDevice
d =
CUDADevice with properties:
Name: 'GeForce 800M'
Index: 1
ComputeCapability: '2.1'
SupportsDouble: 1
DriverVersion: 6
ToolkitVersion: 5
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [65535 65535 65535]
SIMDWidth: 32
TotalMemory: 2.1475e+09
FreeMemory: 1.9886e+09
MultiprocessorCount: 1
ClockRateKHz: 1475000
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1`
I try use gpu in neuralnetwork train, but my gpu slower than cpu.
If I try use gpuArray, gpu is faster than cpu, but I haven't speed acceleration in neural network training.
For example
>> a1 = rand(1000); b1 = rand(1000); tic; c1 = a1 * b1; toc;
Elapsed time is 0.044095 seconds.
>> a2 = gpuArray(rand(1000)); b2 = gpuArray(rand(1000)); tic; c2 = a2 * b2; toc;
Elapsed time is 0.000416 seconds.
But in code
net = newff(H, F, Layers, { 'tansig' 'tansig'}, 'traingdx', 'learngdm', 'mse');
net.trainParam.epochs = Epochs;
net.trainParam.show = 500;
net.trainParam.time = 495;
net.trainParam.goal = 2.0000e-11;
net.trainParam.max_fail = 200000;
net.trainParam.min_grad = 1.0000e-050;
net.performParam.regularization = 0.05;
net.divideParam.trainRatio = 1;
net.divideParam.valRatio = 0;
net.divideParam.testRatio = 0;
net.trainParam.showWindow = 0;
net.trainParam.showCommandLine = 0;
if Gpu1 == 1
net = train(net, H, F, 'useGPU', 'yes', 'showResources','yes');
else
net = train(net, H, F, 'showResources','yes');
end;
tic; net = net_example(300, [23, 9], rand(100, 1000), rand(1, 1000), 1); toc;
Computing Resources:
GPU device #1, GeForce 800M
works slower than
tic; net = net_example(300, [23, 9], rand(100, 1000), rand(1, 1000), 0); toc;
Computing Resources:
MEX2