Related
I'm trying to use SVM-RFE with libsvm library to run on the gene expression dataset. My algorithm is written in Matlab. The particular dataset able to produce 80++% of classification accuracy under the 5-fold CV without applied Feature selection. When I tried to apply svm-rfe on this dataset (same svm parameter setting and use 5-fold CV), the classification result become worse, can only achieve 60++% classification accuracy.
Here is my matlab coding, appreciate if anyone can shed some light on what's wrong with my codes. Thank you in advance.
[label, data] = libsvmread('libsvm_data.scale');
[N D] = size(data);
numfold=5;
indices = crossvalind ('Kfold',label, numfold);
cp = classperf(label);
for i= 1:numfold
disp(strcat('Fold-',int2str(i)));
testix = (indices == i); trainix = ~testix;
test_data = data(testix,:); test_label = label(testix);
train_data = data(trainix,:); train_label = label(trainix);
model = svmtrain(train_label, train_data, sprintf('-s 0 -t 0); %'
s = 1:D;
r = [];
iter = 1;
while ~isempty(s)
X = train_data(:,s);
fs_model = svmtrain(train_label, X, sprintf('-s 0 -t %f -c %f -g %f -b 1', kernel, cost, gamma));
w = fs_model.SVs' * fs_model.sv_coef; %'
c = w.^2;
[c_minvalue, f] = min(c);
r = [s(f),r];
ind = [1:f-1, f+1:length(s)];
s = s(ind);
iter = iter + 1;
end
predefined = 100;
important_feat = r(:,D-predefined+1:end);
for l=1:length(important_feat)
testdata(:,l) = test_data (:,important_feat(l));
end
[predict_label_itest, accuracy_itest, prob_values] = svmpredict(test_label, testdata, model,'-b 1');
acc_itest_fs (:,i) = accuracy_itest(1);
clear testdata;
end
Mean_itest_fs = mean((acc_itest_fs),2);
Mean_bac_fs = mean (bac_fs,2);
After applying RFE to the traindata, you get a subset of the traindata. So when you use the traindata to train a model, I think you should use the subset of the traindata to train this model.
I need to keep track of the F1-scores while tuning C & Sigma in SVM,
For example the following code keeps track of the Accuracy, I need to change it to F1-Score but I was not able to do that…….
%# read some training data
[labels,data] = libsvmread('./heart_scale');
%# grid of parameters
folds = 5;
[C,gamma] = meshgrid(-5:2:15, -15:2:3);
%# grid search, and cross-validation
cv_acc = zeros(numel(C),1);
for i=1:numel(C)
cv_acc(i) = svmtrain(labels, data, ...
sprintf('-c %f -g %f -v %d', 2^C(i), 2^gamma(i), folds));
end
%# pair (C,gamma) with best accuracy
[~,idx] = max(cv_acc);
%# now you can train you model using best_C and best_gamma
best_C = 2^C(idx);
best_gamma = 2^gamma(idx);
%# ...
I have seen the following two links
Retraining after Cross Validation with libsvm
10 fold cross-validation in one-against-all SVM (using LibSVM)
I do understand that I have to first find the best C and gamma/sigma parameters over the training data, then use these two values to do a LEAVE-ONE-OUT crossvalidation classification experiment,
So what I want now is to first do a grid-search for tuning C & sigma.
Please I would prefer to use MATLAB-SVM and not LIBSVM.
Below is my code for LEAVE-ONE-OUT crossvalidation classification.
... clc
clear all
close all
a = load('V1.csv');
X = double(a(:,1:12));
Y = double(a(:,13));
% train data
datall=[X,Y];
A=datall;
n = 40;
ordering = randperm(n);
B = A(ordering, :);
good=B;
input=good(:,1:12);
target=good(:,13);
CVO = cvpartition(target,'leaveout',1);
cp = classperf(target); %# init performance tracker
svmModel=[];
for i = 1:CVO.NumTestSets %# for each fold
trIdx = CVO.training(i);
teIdx = CVO.test(i);
%# train an SVM model over training instances
svmModel = svmtrain(input(trIdx,:), target(trIdx), ...
'Autoscale',true, 'Showplot',false, 'Method','ls', ...
'BoxConstraint',0.1, 'Kernel_Function','rbf', 'RBF_Sigma',0.1);
%# test using test instances
pred = svmclassify(svmModel, input(teIdx,:), 'Showplot',false);
%# evaluate and update performance object
cp = classperf(cp, pred, teIdx);
end
%# get accuracy
accuracy=cp.CorrectRate*100
sensitivity=cp.Sensitivity*100
specificity=cp.Specificity*100
PPV=cp.PositivePredictiveValue*100
NPV=cp.NegativePredictiveValue*100
%# get confusion matrix
%# columns:actual, rows:predicted, last-row: unclassified instances
cp.CountingMatrix
recallP = sensitivity;
recallN = specificity;
precisionP = PPV;
precisionN = NPV;
f1P = 2*((precisionP*recallP)/(precisionP + recallP));
f1N = 2*((precisionN*recallN)/(precisionN + recallN));
aF1 = ((f1P+f1N)/2);
i have changed the code
but i making some mistakes and i am getting errors,
a = load('V1.csv');
X = double(a(:,1:12));
Y = double(a(:,13));
% train data
datall=[X,Y];
A=datall;
n = 40;
ordering = randperm(n);
B = A(ordering, :);
good=B;
inpt=good(:,1:12);
target=good(:,13);
k=10;
cvFolds = crossvalind('Kfold', target, k); %# get indices of 10-fold CV
cp = classperf(target); %# init performance tracker
svmModel=[];
for i = 1:k
testIdx = (cvFolds == i); %# get indices of test instances
trainIdx = ~testIdx;
C = 0.1:0.1:1;
S = 0.1:0.1:1;
fscores = zeros(numel(C), numel(S)); %// Pre-allocation
for c = 1:numel(C)
for s = 1:numel(S)
vals = crossval(#(XTRAIN, YTRAIN, XVAL, YVAL)(fun(XTRAIN, YTRAIN, XVAL, YVAL, C(c), S(c))),inpt(trainIdx,:),target(trainIdx));
fscores(c,s) = mean(vals);
end
end
end
[cbest, sbest] = find(fscores == max(fscores(:)));
C_final = C(cbest);
S_final = S(sbest);
.......
and the function.....
.....
function fscore = fun(XTRAIN, YTRAIN, XVAL, YVAL, C, S)
svmModel = svmtrain(XTRAIN, YTRAIN, ...
'Autoscale',true, 'Showplot',false, 'Method','ls', ...
'BoxConstraint', C, 'Kernel_Function','rbf', 'RBF_Sigma', S);
pred = svmclassify(svmModel, XVAL, 'Showplot',false);
cp = classperf(YVAL, pred)
%# get accuracy
accuracy=cp.CorrectRate*100
sensitivity=cp.Sensitivity*100
specificity=cp.Specificity*100
PPV=cp.PositivePredictiveValue*100
NPV=cp.NegativePredictiveValue*100
%# get confusion matrix
%# columns:actual, rows:predicted, last-row: unclassified instances
cp.CountingMatrix
recallP = sensitivity;
recallN = specificity;
precisionP = PPV;
precisionN = NPV;
f1P = 2*((precisionP*recallP)/(precisionP + recallP));
f1N = 2*((precisionN*recallN)/(precisionN + recallN));
fscore = ((f1P+f1N)/2);
end
So basically you want to take this line of yours:
svmModel = svmtrain(input(trIdx,:), target(trIdx), ...
'Autoscale',true, 'Showplot',false, 'Method','ls', ...
'BoxConstraint',0.1, 'Kernel_Function','rbf', 'RBF_Sigma',0.1);
put it in a loop that varies your 'BoxConstraint' and 'RBF_Sigma' parameters and then uses crossval to output the f1-score for that iterations combination of parameters.
You can use a single for-loop exactly like in your libsvm code example (i.e. using meshgrid and 1:numel(), this is probably faster) or a nested for-loop. I'll use a nested loop so that you have both approaches:
C = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10, 30, 100, 300] %// you must choose your own set of values for the parameters that you want to test. You can either do it this way by explicitly typing out a list
S = 0:0.1:1 %// or you can do it this way using the : operator
fscores = zeros(numel(C), numel(S)); %// Pre-allocation
for c = 1:numel(C)
for s = 1:numel(S)
vals = crossval(#(XTRAIN, YTRAIN, XVAL, YVAL)(fun(XTRAIN, YTRAIN, XVAL, YVAL, C(c), S(c)),input(trIdx,:),target(trIdx));
fscores(c,s) = mean(vals);
end
end
%// Then establish the C and S that gave you the bet f-score. Don't forget that c and s are just indexes though!
[cbest, sbest] = find(fscores == max(fscores(:)));
C_final = C(cbest);
S_final = S(sbest);
Now we just have to define the function fun. The docs have this to say about fun:
fun is a function handle to a function with two inputs, the training
subset of X, XTRAIN, and the test subset of X, XTEST, as follows:
testval = fun(XTRAIN,XTEST) Each time it is called, fun should use
XTRAIN to fit a model, then return some criterion testval computed on
XTEST using that fitted model.
So fun needs to:
output a single f-score
take as input a training and testing set for X and Y. Note that these are both subsets of your actual training set! Think of them more like a training and validation SUBSET of your training set. Also note that crossval will split these sets up for you!
Train a classifier on the training subset (using your current C and S parameters from your loop)
RUN your new classifier on the test (or validation rather) subset
Compute and output a performance metric (in your case you want the f1-score)
You'll notice that fun can't take any extra parameters which is why I've wrapped it in an anonymous function so that we can pass the current C and S values in. (i.e. all that #(...)(fun(...)) stuff above. That's just a trick to "convert" our six parameter fun into the 4 parameter one required by crossval.
function fscore = fun(XTRAIN, YTRAIN, XVAL, YVAL, C, S)
svmModel = svmtrain(XTRAIN, YTRAIN, ...
'Autoscale',true, 'Showplot',false, 'Method','ls', ...
'BoxConstraint', C, 'Kernel_Function','rbf', 'RBF_Sigma', S);
pred = svmclassify(svmModel, XVAL, 'Showplot',false);
CP = classperf(YVAL, pred)
fscore = ... %// You can do this bit the same way you did earlier
end
I found the only problem with target(trainIdx). It's a row vector so I just replaced target(trainIdx) with target(trainIdx) which is a column vector.
In cwt() I can specify which wavelet function to use. How does that impact the speed of cwt()?
Here is a benchmark, which I run with the -singleCompThread option when starting MATLAB to force it to use a single computational thread. cwt() was passed a 1,000,000-sample signal and asked to compute scales 1 to 10. My CPU is an i7-3610QM.
Code used:
clear all
%% Benchmark parameters
results_file_name = 'results_scale1-10.csv';
number_of_random_runs = 10;
scales = 1:10;
number_of_random_samples = 1000000;
%% Construct a cell array containing all the wavelet names
wavelet_haar_names = {'haar'};
wavelet_db_names = {'db1'; 'db2'; 'db3'; 'db4'; 'db5'; 'db6'; 'db7'; 'db8'; 'db9'; 'db10'};
wavelet_sym_names = {'sym2'; 'sym3'; 'sym4'; 'sym5'; 'sym6'; 'sym7'; 'sym8'};
wavelet_coif_names = {'coif1'; 'coif2'; 'coif3'; 'coif4'; 'coif5'};
wavelet_bior_names = {'bior1.1'; 'bior1.3'; 'bior1.5'; 'bior2.2'; 'bior2.4'; 'bior2.6'; 'bior2.8'; 'bior3.1'; 'bior3.3'; 'bior3.5'; 'bior3.7'; 'bior3.9'; 'bior4.4'; 'bior5.5'; 'bior6.8'};
wavelet_rbior_names = {'rbio1.1'; 'rbio1.3'; 'rbio1.5'; 'rbio2.2'; 'rbio2.4'; 'rbio2.6'; 'rbio2.8'; 'rbio3.1'; 'rbio3.3'; 'rbio3.5'; 'rbio3.7'; 'rbio3.9'; 'rbio4.4'; 'rbio5.5'; 'rbio6.8'};
wavelet_meyer_names = {'meyr'};
wavelet_dmeyer_names = {'dmey'};
wavelet_gaus_names = {'gaus1'; 'gaus2'; 'gaus3'; 'gaus4'; 'gaus5'; 'gaus6'; 'gaus7'; 'gaus8'};
wavelet_mexh_names = {'mexh'};
wavelet_morl_names = {'morl'};
wavelet_cgau_names = {'cgau1'; 'cgau2'; 'cgau3'; 'cgau4'; 'cgau5'};
wavelet_shan_names = {'shan1-1.5'; 'shan1-1'; 'shan1-0.5'; 'shan1-0.1'; 'shan2-3'};
wavelet_fbsp_names = {'fbsp1-1-1.5'; 'fbsp1-1-1'; 'fbsp1-1-0.5'; 'fbsp2-1-1'; 'fbsp2-1-0.5'; 'fbsp2-1-0.1'};
wavelet_cmor_names = {'cmor1-1.5'; 'cmor1-1'; 'cmor1-0.5'; 'cmor1-1'; 'cmor1-0.5'; 'cmor1-0.1'};
% Concatenate all wavelet names into a single cell array
wavelet_categories_names = who('wavelet*names');
wavelet_names = {};
for wavelet_categories_number=1:size(wavelet_categories_names,1)
temp = wavelet_categories_names(wavelet_categories_number);
temp = eval(temp{1});
wavelet_names = vertcat(wavelet_names, temp);
end
%% Prepare data
random_signal = rand(number_of_random_runs,number_of_random_samples);
%% Run benchmarks
result_file_ID = fopen(results_file_name, 'w');
for wavelet_number = 1:size(wavelet_names,1)
wavelet_name = wavelet_names(wavelet_number,:)
% Compute wavelet on a random signal
tic
for run = 1:number_of_random_runs
cwt(random_signal(run, :),scales,wavelet_name{1});
end
run_time_random_test = toc
fprintf(result_file_ID, '%s,', wavelet_name{1})
fprintf(result_file_ID, '%d\n', run_time_random_test)
end
size(wavelet_names,1)
fclose(result_file_ID);
If you want to see the impact of the choice of the scale:
Code used:
clear all
%% Benchmark parameters
results_file_name = 'results_sym2_change_scale.csv';
number_of_random_runs = 10;
scales = 1:10;
number_of_random_samples = 10000000;
% wavelet_names = {'sym2', 'sym3'}%, 'sym4'};
output_directory = 'output';
wavelet_names = get_all_wavelet_names();
%% Prepare data
random_signal = rand(number_of_random_runs,number_of_random_samples);
%% Prepare result folder
if ~exist(output_directory, 'dir')
mkdir(output_directory);
end
%% Run benchmarks
result_file_ID = fopen(results_file_name, 'w');
for wavelet_number = 1:size(wavelet_names,1)
wavelet_name = wavelet_names{wavelet_number}
if wavelet_number > 1
fprintf(result_file_ID, '%s\n', '');
end
fprintf(result_file_ID, '%s', wavelet_name)
run_time_random_test_scales = zeros(size(scales,2),1);
for scale_number = 1:size(scales,2)
scale = scales(scale_number);
% Compute wavelet on a random signal
tic
for run = 1:number_of_random_runs
cwt(random_signal(run, :),scale,wavelet_name);
end
run_time_random_test = toc
fprintf(result_file_ID, ',%d', run_time_random_test)
run_time_random_test_scales(scale_number) = run_time_random_test;
end
figure
bar(run_time_random_test_scales)
title(['Run time on random signal for ' wavelet_name])
xlabel('Scale')
ylabel('Run time (seconds)')
save_figure( fullfile(output_directory, ['run_time_random_test_' wavelet_name]) )
close all
end
size(wavelet_names,1)
fclose(result_file_ID);
With 3 functions:
get_all_wavelet_names.m:
function [ wavelet_names ] = get_all_wavelet_names( )
%GET_ALL_WAVELET_NAMES Get a list of available wavelet functions
%% Construct a cell array containing all the wavelet names
wavelet_haar_names = {'haar'};
wavelet_db_names = {'db1'; 'db2'; 'db3'; 'db4'; 'db5'; 'db6'; 'db7'; 'db8'; 'db9'; 'db10'};
wavelet_sym_names = {'sym2'; 'sym3'; 'sym4'; 'sym5'; 'sym6'; 'sym7'; 'sym8'};
wavelet_coif_names = {'coif1'; 'coif2'; 'coif3'; 'coif4'; 'coif5'};
wavelet_bior_names = {'bior1.1'; 'bior1.3'; 'bior1.5'; 'bior2.2'; 'bior2.4'; 'bior2.6'; 'bior2.8'; 'bior3.1'; 'bior3.3'; 'bior3.5'; 'bior3.7'; 'bior3.9'; 'bior4.4'; 'bior5.5'; 'bior6.8'};
wavelet_rbior_names = {'rbio1.1'; 'rbio1.3'; 'rbio1.5'; 'rbio2.2'; 'rbio2.4'; 'rbio2.6'; 'rbio2.8'; 'rbio3.1'; 'rbio3.3'; 'rbio3.5'; 'rbio3.7'; 'rbio3.9'; 'rbio4.4'; 'rbio5.5'; 'rbio6.8'};
wavelet_meyer_names = {'meyr'};
wavelet_dmeyer_names = {'dmey'};
wavelet_gaus_names = {'gaus1'; 'gaus2'; 'gaus3'; 'gaus4'; 'gaus5'; 'gaus6'; 'gaus7'; 'gaus8'};
wavelet_mexh_names = {'mexh'};
wavelet_morl_names = {'morl'};
wavelet_cgau_names = {'cgau1'; 'cgau2'; 'cgau3'; 'cgau4'; 'cgau5'};
wavelet_shan_names = {'shan1-1.5'; 'shan1-1'; 'shan1-0.5'; 'shan1-0.1'; 'shan2-3'};
wavelet_fbsp_names = {'fbsp1-1-1.5'; 'fbsp1-1-1'; 'fbsp1-1-0.5'; 'fbsp2-1-1'; 'fbsp2-1-0.5'; 'fbsp2-1-0.1'};
wavelet_cmor_names = {'cmor1-1.5'; 'cmor1-1'; 'cmor1-0.5'; 'cmor1-1'; 'cmor1-0.5'; 'cmor1-0.1'};
% Concatenate all wavelet names into a single cell array
wavelet_categories_names = who('wavelet*names');
wavelet_names = {};
for wavelet_categories_number=1:size(wavelet_categories_names,1)
temp = wavelet_categories_names(wavelet_categories_number);
temp = eval(temp{1});
wavelet_names = vertcat(wavelet_names, temp);
end
end
save_figure.m:
function [ ] = save_figure( output_graph_filename )
% Record aa figure as PNG and fig files
% Create the folder if it doesn't exist already.
[pathstr, name, ext] = fileparts(output_graph_filename);
if ~exist(pathstr, 'dir')
mkdir(pathstr);
end
h = gcf;
set(0,'defaultAxesFontSize',18) % http://www.mathworks.com/support/solutions/en/data/1-8XOW94/index.html?solution=1-8XOW94
boldify(h);
print('-dpng','-r600', [output_graph_filename '.png']);
print(h,[output_graph_filename '.pdf'],'-dpdf','-r600')
saveas(gcf,[output_graph_filename '.fig'], 'fig')
end
and boldify.m:
function boldify(h,g)
%BOLDIFY Make lines and text bold for standard viewgraph style.
% BOLDIFY boldifies the lines and text of the current figure.
% BOLDIFY(H) applies to the graphics handle H.
%
% BOLDIFY(X,Y) specifies an X by Y inch graph of the current
% figure. If text labels have their 'UserData' data property
% set to 'slope = ...', then the 'Rotation' property is set to
% account for changes in the graph's aspect ratio. The
% default is MATLAB's default.
% S. T. Smith
% The name of this function does not represent an endorsement by the author
% of the egregious grammatical trend of verbing nouns.
if nargin < 1, h = gcf;, end
% Set (and get) the default MATLAB paper size and position
set(gcf,'PaperPosition','default');
units = get(gcf,'PaperUnits');
set(gcf,'PaperUnits','inches');
fsize = get(gcf,'PaperPosition');
fsize = fsize(3:4); % Figure size (X" x Y") on paper.
psize = get(gcf,'PaperSize');
if nargin == 2 % User specified graph size
fsize = [h,g];
h = gcf;
end
% Set the paper position of the current figure
set(gcf,'PaperPosition', ...
[(psize(1)-fsize(1))/2 (psize(2)-fsize(2))/2 fsize(1) fsize(2)]);
fsize = get(gcf,'PaperPosition');
fsize = fsize(3:4); % Graph size (X" x Y") on paper.
set(gcf,'PaperUnits',units); % Back to original
% Get the normalized axis position of the current axes
units = get(gca,'Units');
set(gca,'Units','normalized');
asize = get(gca,'Position');
asize = asize(3:4);
set(gca,'Units',units);
ha = get(h,'Children');
for i=1:length(ha)
% if get(ha(i),'Type') == 'axes'
% changed by B. A. Miller
if strcmp(get(ha(i), 'Type'), 'axes') == 1
units = get(ha(i),'Units');
set(ha(i),'Units','normalized');
asize = get(ha(i),'Position'); % Axes Position (normalized)
asize = asize(3:4);
set(ha(i),'Units',units);
[m,j] = max(asize); j = j(1);
scale = 1/(asize(j)*fsize(j)); % scale*inches -normalized units
set(ha(i),'FontWeight','Bold');
set(ha(i),'LineWidth',2);
[m,k] = min(asize); k = k(1);
if asize(k)*fsize(k) > 1/2
set(ha(i),'TickLength',[1/8 1.5*1/8]*scale); % Gives 1/8" ticks
else
set(ha(i),'TickLength',[3/32 1.5*3/32]*scale); % Gives 3/32" ticks
end
set(get(ha(i),'XLabel'),'FontSize',18); % 14-pt labels
set(get(ha(i),'XLabel'),'FontWeight','Bold');
set(get(ha(i),'XLabel'),'VerticalAlignment','top');
set(get(ha(i),'YLabel'),'FontSize',18); % 14-pt labels
set(get(ha(i),'YLabel'),'FontWeight','Bold');
%set(get(ha(i),'YLabel'),'VerticalAlignment','baseline');
set(get(ha(i),'Title'),'FontSize',18); % 16-pt titles
set(get(ha(i),'Title'),'FontWeight','Bold');
% set(get(ha(i), 'FontSize',20, 'XTick',[]));
end
hc = get(ha(i),'Children');
for j=1:length(hc)
chtype = get(hc(j),'Type');
if chtype(1:4) == 'text'
set(hc(j),'FontSize',17); % 12 pt descriptive labels
set(hc(j),'FontWeight','Bold');
ud = get(hc(j),'UserData'); % User data
if length(ud) 8
if ud(1:8) == 'slope = ' % Account for change in actual slope
slope = sscanf(ud,'slope = %g');
slope = slope*(fsize(2)/fsize(1))/(asize(2)/asize(1));
set(hc(j),'Rotation',atan(slope)/pi*180);
end
end
elseif chtype(1:4) == 'line'
set(hc(j),'LineWidth',2);
end
end
end
Bonus: correlation between all wavelets on a random signal with 1000000 samples with the first 10 scales:
Code used:
%% PRE-REQUISITE: You need to download http://www.mathworks.com/matlabcentral/fileexchange/24253-customizable-heat-maps , which gives the function heatmap()
%% Benchmark parameters
scales = 1:10;
number_of_random_samples = 1000000;
% wavelet_names = {'sym2'; 'sym3'; 'sym4'; 'sym5'; 'sym6'; 'sym7'; 'sym8'};
% wavelet_names = {'cgau1'; 'cgau2'; 'cgau3'; 'cgau4'; 'cgau5'};
wavelet_names = {'db2'; 'sym2'};
OUTPUT_FOLDER = 'output_corr';
% wavelet_names = get_all_wavelet_names(); % WARNING: you need to remove all complex wavelets, viz. cgau1, shan, fbsp and cmor, and the heatmap will be pissed to see complex values coming to her.
%% Prepare data
random_signal = rand(1,number_of_random_samples);
results = zeros(size(wavelet_names,1), number_of_random_samples);
%% Prepare result folder
if ~exist(OUTPUT_FOLDER, 'dir')
mkdir(OUTPUT_FOLDER);
end
%% Run benchmarks
for scale_number = 1:size(scales,2)
scale = scales(scale_number);
for wavelet_number = 1:size(wavelet_names,1)
wavelet_name = wavelet_names{wavelet_number}
% Compute wavelet on a random signal
run = 1;
results(wavelet_number, :) = cwt(random_signal(run, :),scale,wavelet_name);
if wavelet_number == 999
break
end
end
correlation_results = corrcoef(results')
heatmap(correlation_results, [], [], '%0.2f', 'MinColorValue', -1.0, 'MaxColorValue', 1.0, 'Colormap', 'jet',...
'Colorbar', true, 'ColorLevels', 64, 'UseFigureColormap', false);
title(['Correlation matrix for scale ' num2str(scale)]);
xlabel(['Wavelet 1 to ' num2str(size(wavelet_names,1)) ' for scale ' num2str(scale)]);
ylabel(['Wavelet 1 to ' num2str(size(wavelet_names,1)) ' for scale ' num2str(scale)]);
snapnow
print('-dpng','-r600',fullfile(OUTPUT_FOLDER, ['scalecorr' num2str(scale) '.png']))
end
Correlation for each wavelet between different scales (1 to 100):
Code used:
%% PRE-REQUISITE: You need to download http://www.mathworks.com/matlabcentral/fileexchange/24253-customizable-heat-maps , which gives the function heatmap()
%% Benchmark parameters
scales = 1:100;
number_of_random_samples = 1000000;
% wavelet_name = 'gaus2';
% wavelet_names = {'sym2', 'sym3'}%, 'sym4'};
OUTPUT_FOLDER = 'output_corr';
wavelet_names = get_all_wavelet_names(); % WARNING: you need to remove all complex wavelets, viz. cgau1, shan, fbsp and cmor, and the heatmap will be pissed to see complex values coming to her.
%% Prepare data
random_signal = rand(1,number_of_random_samples);
results = zeros(size(scales,2), number_of_random_samples);
%% Prepare result folder
if ~exist(OUTPUT_FOLDER, 'dir')
mkdir(OUTPUT_FOLDER);
end
%% Run benchmarks
for wavelet_number = 1:size(wavelet_names,1)
wavelet_name = wavelet_names{wavelet_number}
run_time_random_test_scales = zeros(size(scales,2),1);
for scale_number = 1:size(scales,2)
scale = scales(scale_number);
run = 1;
% Compute wavelet on a random signal
results(scale_number, :) = cwt(random_signal(run, :),scale,wavelet_name);
end
correlation_results = corrcoef(results')
heatmap(correlation_results, [], [], '%0.2f', 'MinColorValue', -1.0, 'MaxColorValue', 1.0, 'Colormap', 'jet',...
'Colorbar', true, 'ColorLevels', 64, 'UseFigureColormap', false);
title(['Correlation matrix for wavelet ' wavelet_name]);
xlabel(['Scales 1 to ' num2str(max(scales)) ' for wavelet ' wavelet_name]);
ylabel(['Scales 1 to ' num2str(max(scales)) ' for wavelet ' wavelet_name]);
snapnow
print('-dpng','-r600',fullfile(OUTPUT_FOLDER, [wavelet_name '_scalecorr_scale1to' num2str(max(scales)) '.png']))
end
Intro: I'm using MATLAB's Neural Network Toolbox in an attempt to forecast time series one step into the future. Currently I'm just trying to forecast a simple sinusoidal function, but hopefully I will be able to move on to something a bit more complex after I obtain satisfactory results.
Problem: Everything seems to work fine, however the predicted forecast tends to be lagged by one period. Neural network forecasting isn't much use if it just outputs the series delayed by one unit of time, right?
Code:
t = -50:0.2:100;
noise = rand(1,length(t));
y = sin(t)+1/2*sin(t+pi/3);
split = floor(0.9*length(t));
forperiod = length(t)-split;
numinputs = 5;
forecasted = [];
msg = '';
for j = 1:forperiod
fprintf(repmat('\b',1,numel(msg)));
msg = sprintf('forecasting iteration %g/%g...\n',j,forperiod);
fprintf('%s',msg);
estdata = y(1:split+j-1);
estdatalen = size(estdata,2);
signal = estdata;
last = signal(end);
[signal,low,high] = preprocess(signal'); % pre-process
signal = signal';
inputs = signal(rowshiftmat(length(signal),numinputs));
targets = signal(numinputs+1:end);
%% NARNET METHOD
feedbackDelays = 1:4;
hiddenLayerSize = 10;
net = narnet(feedbackDelays,[hiddenLayerSize hiddenLayerSize]);
net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
signalcells = mat2cell(signal,[1],ones(1,length(signal)));
[inputs,inputStates,layerStates,targets] = preparets(net,{},{},signalcells);
net.trainParam.showWindow = false;
net.trainparam.showCommandLine = false;
net.trainFcn = 'trainlm'; % Levenberg-Marquardt
net.performFcn = 'mse'; % Mean squared error
[net,tr] = train(net,inputs,targets,inputStates,layerStates);
next = net(inputs(end),inputStates,layerStates);
next = postprocess(next{1}, low, high); % post-process
next = (next+1)*last;
forecasted = [forecasted next];
end
figure(1);
plot(1:forperiod, forecasted, 'b', 1:forperiod, y(end-forperiod+1:end), 'r');
grid on;
Note:
The function 'preprocess' simply converts the data into logged % differences and 'postprocess' converts the logged % differences back for plotting. (Check EDIT for preprocess and postprocess code)
Results:
BLUE: Forecasted Values
RED: Actual Values
Can anyone tell me what I'm doing wrong here? Or perhaps recommend another method to achieve the desired results (lagless prediction of sinusoidal function, and eventually more chaotic timeseries)? Your help is very much appreciated.
EDIT:
It's been a few days now and I hope everyone has enjoyed their weekend. Since no solutions have emerged I've decided to post the code for the helper functions 'postprocess.m', 'preprocess.m', and their helper function 'normalize.m'. Maybe this will help get the ball rollin.
postprocess.m:
function data = postprocess(x, low, high)
% denormalize
logdata = (x+1)/2*(high-low)+low;
% inverse log data
sign = logdata./abs(logdata);
data = sign.*(exp(abs(logdata))-1);
end
preprocess.m:
function [y, low, high] = preprocess(x)
% differencing
diffs = diff(x);
% calc % changes
chngs = diffs./x(1:end-1,:);
% log data
sign = chngs./abs(chngs);
logdata = sign.*log(abs(chngs)+1);
% normalize logrets
high = max(max(logdata));
low = min(min(logdata));
y=[];
for i = 1:size(logdata,2)
y = [y normalize(logdata(:,i), -1, 1)];
end
end
normalize.m:
function Y = normalize(X,low,high)
%NORMALIZE Linear normalization of X between low and high values.
if length(X) <= 1
error('Length of X input vector must be greater than 1.');
end
mi = min(X);
ma = max(X);
Y = (X-mi)/(ma-mi)*(high-low)+low;
end
I didn't check you code, but made a similar test to predict sin() with NN. The result seems reasonable, without a lag. I think, your bug is somewhere in synchronization of predicted values with actual values.
Here is the code:
%% init & params
t = (-50 : 0.2 : 100)';
y = sin(t) + 0.5 * sin(t + pi / 3);
sigma = 0.2;
n_lags = 12;
hidden_layer_size = 15;
%% create net
net = fitnet(hidden_layer_size);
%% train
noise = sigma * randn(size(t));
y_train = y + noise;
out = circshift(y_train, -1);
out(end) = nan;
in = lagged_input(y_train, n_lags);
net = train(net, in', out');
%% test
noise = sigma * randn(size(t)); % new noise
y_test = y + noise;
in_test = lagged_input(y_test, n_lags);
out_test = net(in_test')';
y_test_predicted = circshift(out_test, 1); % sync with actual value
y_test_predicted(1) = nan;
%% plot
figure,
plot(t, [y, y_test, y_test_predicted], 'linewidth', 2);
grid minor; legend('orig', 'noised', 'predicted')
and the lagged_input() function:
function in = lagged_input(in, n_lags)
for k = 2 : n_lags
in = cat(2, in, circshift(in(:, end), 1));
in(1, k) = nan;
end
end
After getting my testlabel and trainlabel, i implemented SVM on libsvm and i got an accuracy of 97.4359%. ( c= 1 and g = 0.00375)
model = svmtrain(TrainLabel, TrainVec, '-c 1 -g 0.00375');
[predict_label, accuracy, dec_values] = svmpredict(TestLabel, TestVec, model);
After i find the best c and g,
bestcv = 0;
for log2c = -1:3,
for log2g = -4:1,
cmd = ['-v 5 -c ', num2str(2^log2c), ' -g ', num2str(2^log2g)];
cv = svmtrain(TrainLabel,TrainVec, cmd);
if (cv >= bestcv),
bestcv = cv; bestc = 2^log2c; bestg = 2^log2g;
end
fprintf('%g %g %g (best c=%g, g=%g, rate=%g)\n', log2c, log2g, cv, bestc, bestg, bestcv);
end
end
c = 8 and g = 0.125
I implement the model again:
model = svmtrain(TrainLabel, TrainVec, '-c 8 -g 0.125');
[predict_label, accuracy, dec_values] = svmpredict(TestLabel, TestVec, model);
I get an accuracy of 82.0513%
How is it possible for the accuracy to decrease? shouldn't it increase? Or am i making any mistake?
The accuracies that you were getting during parameter tuning are biased upwards because you were predicting the same data that you were training. This is often fine for parameter tuning.
However, if you wanted those accuracies to be accurate estimates of the true generalization error on your final test set, then you have to add an additional wrap of cross validation or other resampling scheme.
Here is a very clear paper that outlines the general issue (but in a similar context of feature selection): http://www.pnas.org/content/99/10/6562.abstract
EDIT:
I usually add cross validation like:
n = 95 % total number of observations
nfold = 10 % desired number of folds
% Set up CV folds
inds = repmat(1:nfold, 1, mod(nfold, n))
inds = inds(randperm(n))
% Loop over folds
for i = 1:nfold
datapart = data(inds ~= i, :)
% do some stuff
% save results
end
% combine results
To do cross validation, you are supposed to split your training data. Here you test on training data to find your best set of parameter. That is not a good measure. You should use the following pseudo code:
for param = set of parameter to test
[trainTrain,trainVal] = randomly split (trainSet); %%% you can repeat that several times and take the mean accuracy
model = svmtrain(trainTrain, param);
acc = svmpredict(trainVal, model);
if accuracy is the best
bestPAram = param
end
end