how to make the prediction of a qos with a lstm network under matlab - matlab

I am trying to make a QOS prediction on the QWS dataset but I have the following error:
Error using trainNetwork (line 170)
Too many input arguments.
Error in lstm (line 63)
net =
trainNetwork(x_train,y_train,layers,options);
Caused by:
Error using
trainNetwork>iParseInputArguments
(line 326)
Too many input arguments.
data = readtable('C:\Users\Etudiant FST\Documents\études\mini_pjt\d\qws1\qws1.txt');
%test_data = readtable('C:\Users\Etudiant FST\Documents\études\mini_pjt\d\qws2\qws2.txt');
data = data(:,1:10);
x = [];
y = [];
delta_x = 1;
delta_y = 1;
pas = 1;
while (height(data) >= delta_x + delta_y)
x = [x; data(1:delta_x,:)];
y = [y; data(delta_x + 1:delta_x + delta_y,:)];
data(1:pas,:) = [];
end
%numObservations = height(data);
%idxTrain = 1:floor(0.8*numObservations);
%idxTest = floor(0.8*numObservations)+1:numObservations;
%dataTrain = data(idxTrain,:);
%dataTest = data(idxTest,:);
%%for n = 1:numel(dataTrain)
%X = dataTrain{n};
% xt{n} = X(:,1:end-1);
% tt{n} = X(:,2:end);
%%end
height_x = height(x);
split = fix(height_x*0.8);
x_train = x(1:split,:);
x_test = x(split:height_x,:);
y_train = y(1:split,:);
y_test = y(split:height_x,:);
layers = [
sequenceInputLayer(10)
lstmLayer(128,'OutputMode','sequence')
fullyConnectedLayer(10)
regressionLayer];
options = trainingOptions('adam', ...
'MaxEpochs',maxEpochs, ...
'MiniBatchSize',miniBatchSize, ...
'InitialLearnRate',0.01, ...
'GradientThreshold',1, ...
'Shuffle','never', ...
'Plots','training-progress',...
'Verbose',0);
net = trainNetwork(x_train,y_train,layers,options);
enter image description here
I would like it to give me a prediction of the new QOS from the old ones
thank you.

As the error message suggests, MATLAB isn't able to detect the correct trainNetwork function to use (since the function is overloaded). This is because the correct function is selected based on the numbers of inputs and the input (types) passed to it.
If you look at the example for LSTM on the documentation for trainNetwork, you will see that XTrain is a 270 by 1 'cell array' with every cell containing a N x M array while YTrain is a 270 by 1 'categorical array'.
Shaping your Xtrain and Ytrain to these data shape and types should solve the problem. Everything else on the code seems okay to me.

Related

Unknown error on optimizing CNN on Matlab

I want to optimize my CNN as this example using Bayesian optimization and Matlab 2019a. Instead of showing whole my codes in a single part, I split it to three parts to show which lines causes error. First part is about loading data, dividing for training and validation and defining optimization variables. Second part is where error is. Third part is remaining part of codes. Note that buildLayer is also written by me to make network layers.
My questions are: how can I fix aforementioned error and why it occurred?
First part is (loading data):
folder = 'C:\Users\X';
imds = imageDatastore(folder, ...
"IncludeSubfolders",true, ...
"LabelSource","foldernames");
num = 20;
per = randperm(1000, num);
figure;
for i=1 : num
subplot(4, 5, i);
imshow(imds.Files{per(i)});
end
img = readimage(imds,1);
[trainData, validData] = splitEachLabel(imds,0.8);
% defining optimizable variables
optVars = [
optimizableVariable('depth', [1 5], "Type","integer")
optimizableVariable('initLearningRate', [0.001, 1], "Transform","log")
optimizableVariable('initFilterNum', [8 32], "Type","integer")
optimizableVariable('momentum', [0.7 0.98])
optimizableVariable('l2Reg', [1e-10 1e-2],'Transform','log')]
objFcn = makeObjFcn(trainData,validData);
I get this error:
Array formation and parentheses-style indexing with objects of class
'matlab.io.datastore.ImageDatastore' is not allowed. Use objects of
class 'matlab.io.datastore.ImageDatastore' only as scalars or use a
cell array.
Error in unique>uniqueR2012a (line 191)
a = a(:);
Error in unique (line 103)
[varargout{1:nlhs}] = uniqueR2012a(varargin{:});
Error in main>makeObjFcn/valErrorFun (line 47)
numClasses = numel(unique(vaData));
Error in BayesianOptimization/callObjNormally (line 2560)
[Objective, ConstraintViolations, UserData] = this.ObjectiveFcn(conditionalizeX(this, X));
Error in BayesianOptimization/callObjFcn (line 467)
= callObjNormally(this, X);
Error in BayesianOptimization/runSerial (line 1989)
ObjectiveFcnObjectiveEvaluationTime, ObjectiveNargout] = callObjFcn(this, this.XNext);
Error in BayesianOptimization/run (line 1941)
this = runSerial(this);
Error in BayesianOptimization (line 457)
this = run(this);
Error in bayesopt (line 323) Results = BayesianOptimization(Options);
on second part (the lines cause error):
bayesObj = bayesopt(...
objFcn, optVars, ...
"MaxTime", 1* 3600,...,
"IsObjectiveDeterministic",false,...
"UseParallel",false);
and third part (remaining of codes):
estIdx = bayesObj.IndexOfMinimumTrace(end);
fileName = bayesObj.UserDataTrace{bestIdx};
savedStruct = load(fileName);
valError = savedStruct.valError;
function ObjFcn = makeObjFcn(trData,vaData)
ObjFcn = #valErrorFun;
function [valError,cons,fileName] = valErrorFun(optVars)
% making layers
imageSize = [32 32 3];
numClasses = numel(unique(vaData));
layers = buidLayers(...
imageSize,...
optVars.initFilterNum,...
optVars.depth,...
numClasses);
miniBatchSize = 256;
validationFrequency = floor(numel(vaData)/miniBatchSize);
options = trainingOptions('sgdm', ...
'InitialLearnRate',optVars.initLearningRate, ...
'Momentum',optVars.momentum1, ...
'MaxEpochs',60, ...
'LearnRateSchedule','piecewise', ...
'LearnRateDropPeriod',40, ...
'LearnRateDropFactor',0.1, ...
'MiniBatchSize',miniBatchSize, ...
'L2Regularization',optVars.l2Reg, ...
'Shuffle','every-epoch', ...
'Verbose',false, ...
'Plots','training-progress', ...
'ValidationData',vaData, ...
'ValidationFrequency',validationFrequency);
pixelRange = [-4 4];
imageAugmenter = imageDataAugmenter( ...
'RandXReflection',true, ...
'RandXTranslation',pixelRange, ...
'RandYTranslation',pixelRange);
datasource = augmentedImageDatastore(...
imageSize,...
trData,...
'DataAugmentation',imageAugmenter);
trainedNet = trainNetwork(datasource,layers,options);
%close(findall(groot,'Tag','NNET_CNN_TRAININGPLOT_FIGURE'))
YPredicted = classify(trainedNet,XValidation);
valError = 1 - mean(YPredicted == YValidation);
fileName = num2str(valError) + ".mat";
save(fileName,'trainedNet','valError','options')
cons = [];
end
end

Transfer Learning for Regression in Matlab

I am trying to implement a model that takes an image as the input and gives a vector of 26 numbers. I am using VGG-16 at this time through the following Matlab code:
analyzeNetwork(net);
NUM_OUTPUT = 26;
layers = net.Layers;
%output = fullyConnectedLayer(NUM_OUTPUT, ...
% 'Name','output_layer', ...
% 'WeightLearnRateFactor',10, ...
% 'BiasLearnRateFactor',10);
layers = [
layers(1:38)
fullyConnectedLayer(NUM_OUTPUT)
regressionLayer];
%layers(1:67) = freezeWeights(layers(1:67));
miniBatchSize = 5;
validationFrequency = floor(numel(YTrain)/miniBatchSize);
options = trainingOptions('sgdm',...
'InitialLearnRate',0.001, ...
'ValidationData',{XValidation,YValidation},...
'Plots','training-progress',...
'Verbose',false);
net = trainNetwork(XTrain,YTrain,layers,options);
YPred = predict(net,XValidation);
predictionError = YValidation - YPred;
thr = 10;
numCorrect = sum(abs(predictionError) < thr);
numImagesValidation = numel(YValidation);
accuracy = numCorrect/numImagesValidation;
rmse = sqrt(mean(predictionError.^2));
The shape of XTrain and YTrain are as follows:
XTrain: 224 224 3 140
YTrain: 26 140
By running the code above (it is a part of the code not the whole of it) I get the following error:
Error using trainNetwork (line 170)
Number of observations in X and Y disagree.
I would appreciate it if somebody could help me to figure out what is the problem because as far as I know the number of samples in both are equal and there is no necessity for the rest of the dimensions to be equal.
Transpose YTrain to be 140x26.
Name your new layers, and make them layerGraph
Regression can easly go unstable so decrease learning rate or increase batch size if you get some nans.
net = vgg16 ; % analyzeNetwork(net);
LAYERS_FREEZE_UNTIL=35;
LAYERS_COPY_UNTIL=38;
NUM_TRAIN_SAMPLES = size(YTrain,1);
NUM_OUTPUT = size(YTrain,2);
my_layers =layerGraph([
freezeWeights(net.Layers(1:LAYERS_FREEZE_UNTIL))
net.Layers(LAYERS_FREEZE_UNTIL+1:LAYERS_COPY_UNTIL)
fullyConnectedLayer(NUM_OUTPUT*2,'Name','my_fc1')
fullyConnectedLayer(NUM_OUTPUT,'Name','my_fc2')
regressionLayer('Name','my_regr')
]);
% figure; plot(my_layers), ylim([0.5,6.5])
% analyzeNetwork(my_layers);
MINI_BATCH_SIZE = 16;
options = trainingOptions('sgdm', ...
'MiniBatchSize',MINI_BATCH_SIZE, ...
'MaxEpochs',20, ...
'InitialLearnRate',1e-4, ...
'Shuffle','every-epoch', ...
'ValidationData',{XValidation,YValidation}, ...
'ValidationFrequency',floor(NUM_TRAIN_SAMPLES/MINI_BATCH_SIZE), ...
'Verbose',true, ...
'Plots','training-progress');
my_net = trainNetwork(XTrain,YTrain,my_layers,options);

Neural Network Backpropagation Algorithm Implementation

I implemented a Neural Network Back propagation Algorithm in MATLAB, however is is not training correctly. The training data is a matrix X = [x1, x2], dimension 2 x 200 and I have a target matrix T = [target1, target2], dimension 2 x 200. The first 100 columns in T can be [1; -1] for class 1, and the second 100 columns in T can be [-1; 1] for class 2.
theta = 0.1; % criterion to stop
eta = 0.1; % step size
Nh = 10; % number of hidden nodes
For some reason the total training error is always 1.000, it never goes close to the theta, so it runs forever.
I used the following formulas:
The total training error:
The code is well documented below. I would appreciate any help.
clear;
close all;
clc;
%%('---------------------')
%%('Generating dummy data')
%%('---------------------')
d11 = [2;2]*ones(1,70)+2.*randn(2,70);
d12 = [-2;-2]*ones(1,30)+randn(2,30);
d1 = [d11,d12];
d21 = [3;-3]*ones(1,50)+randn([2,50]);
d22 = [-3;3]*ones(1,50)+randn([2,50]);
d2 = [d21,d22];
hw5_1 = d1;
hw5_2 = d2;
save hw5.mat hw5_1 hw5_2
x1 = hw5_1;
x2 = hw5_2;
% step 1: Construct training data matrix X=[x1,x2], dimension 2x200
training_data = [x1, x2];
% step 2: Construct target matrix T=[target1, target2], dimension 2x200
target1 = repmat([1; -1], 1, 100); % class 1
target2 = repmat([-1; 1], 1, 100); % class 2
T = [target1, target2];
% step 3: normalize training data
training_data = training_data - mean(training_data(:));
training_data = training_data / std(training_data(:));
% step 4: specify parameters
theta = 0.1; % criterion to stop
eta = 0.1; % step size
Nh = 10; % number of hidden nodes, actual hidden nodes should be 11 (including a biase)
Ni = 2; % dimension of input vector = number of input nodes, actual input nodes should be 3 (including a biase)
No = 2; % number of class = number of out nodes
% step 5: Initialize the weights
a = -1/sqrt(No);
b = +1/sqrt(No);
inputLayerToHiddenLayerWeight = (b-a).*rand(Ni, Nh) + a
hiddenLayerToOutputLayerWeight = (b-a).*rand(Nh, No) + a
J = inf;
p = 1;
% activation function
% f(net) = a*tanh(b*net),
% f'(net) = a*b*sech2(b*net)
a = 1.716;
b = 2/3;
while J > theta
% step 6: randomly choose one training sample vector from X,
% together with its target vector
k = randi([1, size(training_data, 2)]);
input_X = training_data(:,k);
input_T = T(:,k);
% step 7: Calculate net_j values for hidden nodes in layer 1
% hidden layer output before activation function applied
netj = inputLayerToHiddenLayerWeight' * input_X;
% step 8: Calculate hidden node output Y using activation function
% apply activation function to hidden layer neurons
Y = a*tanh(b*netj);
% step 9: Calculate net_k values for output nodes in layer 2
% output later output before activation function applied
netk = hiddenLayerToOutputLayerWeight' * Y;
% step 10: Calculate output node output Z using the activation function
% apply activation function to the output layer neurons
Z = a*tanh(b*netk);
% step 11: Calculate sensitivity delta_k = (target - Z) * f'(Z)
% find the error between the expected_output and the neuron output
% we got using the weights
% delta_k = (expected - output) * activation(output)
delta_k = [];
for i=1:size(Z)
yi = Z(i,:);
expected_output = input_T(i,:);
delta_k = [delta_k; (expected_output - yi) ...
* a*b*(sech(b*yi)).^2];
end
% step 12: Calculate sensitivity
% delta_j = Sum_k(delta_k * hidden-to-out weights) * f'(net_j)
% error = (weight_k * error_j) * activation(output)
delta_j = [];
for j=1:size(Y)
yi = Y(j,:);
error = 0;
for k=1:size(delta_k)
error = error + delta_k(k,:)*hiddenLayerToOutputLayerWeight(j, k);
end
delta_j = [delta_j; error * (a*b*(sech(b*yi)).^2)];
end
% step 13: update weights
%2x10
inputLayerToHiddenLayerWeight = [];
for i=1:size(input_X)
xi = input_X(i,:);
wji = [];
for j=1:size(delta_j)
wji = [wji, eta * xi * delta_j(j,:)];
end
inputLayerToHiddenLayerWeight = [inputLayerToHiddenLayerWeight; wji];
end
inputLayerToHiddenLayerWeight
%10x2
hiddenLayerToOutputLayerWeight = [];
for j=1:size(Y)
yi = Y(j,:);
wjk = [];
for k=1:size(delta_k)
wjk = [wjk, eta * delta_k(k,:) * yi];
end
hiddenLayerToOutputLayerWeight = [hiddenLayerToOutputLayerWeight; wjk];
end
hiddenLayerToOutputLayerWeight
% Mean Square Error
J = 0;
for j=1:size(training_data, 2)
X = training_data(:,j);
t = T(:,j);
netj = inputLayerToHiddenLayerWeight' * X;
Y = a*tanh(b*netj);
netk = hiddenLayerToOutputLayerWeight' * Y;
Z = a*tanh(b*netk);
J = J + immse(t, Z);
end
J = J/size(training_data, 2)
p = p + 1;
if p == 4
break;
end
end
% testing neural network using the inputs
test_data = [[2; -2], [-3; -3], [-2; 5], [3; -4]];
for i=1:size(test_data, 2)
end
Weight decay isn't essential for Neural Network training.
What I did notice was that your feature normalization wasn't correct.
The correct algorthim for scaling data to the range of 0 to 1 is
(max - x) / (max - min)
Note: you apply this for every element within the array (or vector). Data inputs for NN need to be within the range of [0,1]. (Technically they can be a little bit outside of that ~[-3,3] but values furthur from 0 make training difficult)
edit*
I am unaware of this activation function
a = 1.716;
b = 2/3;
% f(net) = a*tanh(b*net),
% f'(net) = a*b*sech2(b*net)
It sems like a variation on tanh.
Could you elaborate what it is?
If you're net still doesn't work give me an update and I'll look at your code more closely.

Training neural network for image segmentation

I have one set of original image patches (101x101 matrices) and another corresponding set of image patches (same size 101x101) in binary which are the 'answer' for training the neural network. I wanted to train my neural network so that it can learn, recognize the shape that it's trained from the given image, and produce the image (in same matrix form 150x10201 maybe?) at the output matrix (as a result of segmentation).
Original image is on the left and the desired output is on the right.
So, as for pre-processing stage of the data, I reshaped the original image patches into vector matrices of 1x10201 for each image patch. Combining 150 of them i get a 150x10201 matrix as my input, and another 150x10201 matrix from the binary image patches. Then I provide these input data into the deep learning network. I used Deep Belief Network in this case.
My Matlab code for setup and train DBN as below:
%train a 4 layers 100 hidden unit DBN and use its weights to initialize a NN
rand('state',0)
%train dbn
dbn.sizes = [100 100 100 100];
opts.numepochs = 5;
opts.batchsize = 10;
opts.momentum = 0;
opts.alpha = 1;
dbn = dbnsetup(dbn, train_x, opts);
dbn = dbntrain(dbn, train_x, opts);
%unfold dbn to nn
nn = dbnunfoldtonn(dbn, 10201);
nn.activation_function = 'sigm';
%train nn
opts.numepochs = 1;
opts.batchsize = 10;
assert(isfloat(train_x), 'train_x must be a float');
assert(nargin == 4 || nargin == 6,'number ofinput arguments must be 4 or 6')
loss.train.e = [];
loss.train.e_frac = [];
loss.val.e = [];
loss.val.e_frac = [];
opts.validation = 0;
if nargin == 6
opts.validation = 1;
end
fhandle = [];
if isfield(opts,'plot') && opts.plot == 1
fhandle = figure();
end
m = size(train_x, 1);
batchsize = opts.batchsize;
numepochs = opts.numepochs;
numbatches = m / batchsize;
assert(rem(numbatches, 1) == 0, 'numbatches must be a integer');
L = zeros(numepochs*numbatches,1);
n = 1;
for i = 1 : numepochs
tic;
kk = randperm(m);
for l = 1 : numbatches
batch_x = train_x(kk((l - 1) * batchsize + 1 : l * batchsize), :);
%Add noise to input (for use in denoising autoencoder)
if(nn.inputZeroMaskedFraction ~= 0)
batch_x = batch_x.*(rand(size(batch_x))>nn.inputZeroMaskedFraction);
end
batch_y = train_y(kk((l - 1) * batchsize + 1 : l * batchsize), :);
nn = nnff(nn, batch_x, batch_y);
nn = nnbp(nn);
nn = nnapplygrads(nn);
L(n) = nn.L;
n = n + 1;
end
t = toc;
if opts.validation == 1
loss = nneval(nn, loss, train_x, train_y, val_x, val_y);
str_perf = sprintf('; Full-batch train mse = %f, val mse = %f',
loss.train.e(end), loss.val.e(end));
else
loss = nneval(nn, loss, train_x, train_y);
str_perf = sprintf('; Full-batch train err = %f', loss.train.e(end));
end
if ishandle(fhandle)
nnupdatefigures(nn, fhandle, loss, opts, i);
end
disp(['epoch ' num2str(i) '/' num2str(opts.numepochs) '. Took ' num2str(t) ' seconds' '. Mini-batch mean squared error on training set is ' num2str(mean(L((n-numbatches):(n-1)))) str_perf]);
nn.learningRate = nn.learningRate * nn.scaling_learningRate;
end
Can anyone let me know, is the training for the NN like this enable it to do the segmentation work? or how should I modify the code to train the NN so that it can generate the output/result as the image matrix in 150x10201 form?
Thank you so much..
Your inputs are TOO big. You should try to work with smaller patches from 19x19 to maximum 30x30 (which already represent 900 neurons into the input layer).
Then comes your main problem: you only have 150 images! And when you train a NN, you need at least three times more training instances than weights into your NN. So be very careful to the architecture you choose.
A CNN may be more adapted to your problem.

How to use Neural network for non binary input and output

I tried to use the modified version of NN back propagation code by Phil Brierley
(www.philbrierley.com). When i try to solve the XOR problem it works perfectly. but when i try to solve a problem of the form output = x1^2 + x2^2 (ouput = sum of squares of input), the results are not accurate. i have scaled the input and ouput between -1 and 1. I get different results every time i run the same program (i understand its due to random wts initialization), but results are very different. i tried changing learning rate but still results converge.
have given the code below
%---------------------------------------------------------
% MATLAB neural network backprop code
% by Phil Brierley
%--------------------------------------------------------
clear; clc; close all;
%user specified values
hidden_neurons = 4;
epochs = 20000;
input = [];
for i =-10:2.5:10
for j = -10:2.5:10
input = [input;i j];
end
end
output = (input(:,1).^2 + input(:,2).^2);
output1 = output;
% Maximum input and output limit and scaling factors
m1 = -10; m2 = 10;
m3 = 0; m4 = 250;
c = -1; d = 1;
%Scale input and output
for i =1:size(input,2)
I = input(:,i);
scaledI = ((d-c)*(I-m1) ./ (m2-m1)) + c;
input(:,i) = scaledI;
end
for i =1:size(output,2)
I = output(:,i);
scaledI = ((d-c)*(I-m3) ./ (m4-m3)) + c;
output(:,i) = scaledI;
end
train_inp = input;
train_out = output;
%read how many patterns and add bias
patterns = size(train_inp,1);
train_inp = [train_inp ones(patterns,1)];
%read how many inputs and initialize learning rate
inputs = size(train_inp,2);
hlr = 0.1;
%set initial random weights
weight_input_hidden = (randn(inputs,hidden_neurons) - 0.5)/10;
weight_hidden_output = (randn(1,hidden_neurons) - 0.5)/10;
%Training
err = zeros(1,epochs);
for iter = 1:epochs
alr = hlr;
blr = alr / 10;
%loop through the patterns, selecting randomly
for j = 1:patterns
%select a random pattern
patnum = round((rand * patterns) + 0.5);
if patnum > patterns
patnum = patterns;
elseif patnum < 1
patnum = 1;
end
%set the current pattern
this_pat = train_inp(patnum,:);
act = train_out(patnum,1);
%calculate the current error for this pattern
hval = (tanh(this_pat*weight_input_hidden))';
pred = hval'*weight_hidden_output';
error = pred - act;
% adjust weight hidden - output
delta_HO = error.*blr .*hval;
weight_hidden_output = weight_hidden_output - delta_HO';
% adjust the weights input - hidden
delta_IH= alr.*error.*weight_hidden_output'.*(1-(hval.^2))*this_pat;
weight_input_hidden = weight_input_hidden - delta_IH';
end
% -- another epoch finished
%compute overall network error at end of each epoch
pred = weight_hidden_output*tanh(train_inp*weight_input_hidden)';
error = pred' - train_out;
err(iter) = ((sum(error.^2))^0.5);
%stop if error is small
if err(iter) < 0.001
fprintf('converged at epoch: %d\n',iter);
break
end
end
%Output after training
pred = weight_hidden_output*tanh(train_inp*weight_input_hidden)';
Y = m3 + (m4-m3)*(pred-c)./(d-c);
% Testing for a new set of input
input_test = [6 -3.1; 0.5 1; -2 3; 3 -2; -4 5; 0.5 4; 6 1.5];
output_test = (input_test(:,1).^2 + input_test(:,2).^2);
input1 = input_test;
%Scale input
for i =1:size(input1,2)
I = input1(:,i);
scaledI = ((d-c)*(I-m1) ./ (m2-m1)) + c;
input1(:,i) = scaledI;
end
%Predict output
train_inp1 = input1;
patterns = size(train_inp1,1);
bias = ones(patterns,1);
train_inp1 = [train_inp1 bias];
pred1 = weight_hidden_output*tanh(train_inp1*weight_input_hidden)';
%Rescale
Y1 = m3 + (m4-m3)*(pred1-c)./(d-c);
analy_numer = [output_test Y1']
plot(err)
This is the sample output i get for problem
state after 20000 epochs
analy_numer =
45.6100 46.3174
1.2500 -2.9457
13.0000 11.9958
13.0000 9.7097
41.0000 44.9447
16.2500 17.1100
38.2500 43.9815
if i run once more i get different results. as can be observed for small values of input i get totally wrong ans (negative ans not possible). for other values accuracy is still poor.
can someone tell what i am doing wrong and how to correct.
thanks
raman