I create NARX network for 16 input and 1 output like this
in=[u1(1) u1(2) ... u1(t)
u2(1) u2(2) ... u2(t)
. . .
u16(1) u16(2) ... u16(t)];
target=[1 2 ... t];
and i want to train with 5 dataset of input and output, but i don't know how to create the one input and target matrix with 5 dataset to train NARX.
You can combine datasets with
catsamples()
For example:
X = catsamples(x1, x2,..., xn)
T = catsamples(t1, t2,..., tn)
The optional parameter 'pad' allows concatenating datasets with varying sizes.
For further informations take a look at catsamples in the MathWorks documentation.
There is also a small example available at MathWorks:
Multiple Sequences with Dynamic Neural Networks
Related
I'm trying to train a hybrid model with GP on top of pre-trained CNN (Densenet, VGG and Resnet) with CIFAR10 data, mimic the ex2 function in the gpflow document. But the testing result is always between 0.1~0.2, which generally means random guess (Wilson+2016 paper shows hybrid model for CIFAR10 data should get accuracy of 0.7). Could anyone give me a hint of what could be wrong?
I've tried same code with simpler cnn models (2 conv layer or 4 conv layer) and both have reasonable results. I've tried to use different Keras applications: Densenet121, VGG16, ResNet50, neither works. I've tried to freeze the weights in the pre-trained models still not working.
def cnn_dn(output_dim):
base_model = DenseNet121(weights='imagenet', include_top=False, input_shape=(32,32,3))
bout = base_model.output
fcl = GlobalAveragePooling2D()(bout)
#for layer in base_model.layers:
# layer.trainable = False
output=Dense(output_dim, activation='relu')(fcl)
md=Model(inputs=base_model.input, outputs=output)
return md
#add gp on top, reference:ex2() function in
#https://nbviewer.jupyter.org/github/GPflow/GPflow/blob/develop/doc/source/notebooks/tailor/gp_nn.ipynb
#needs to slightly change build graph part because keras variable #sharing is not the same as tensorflow
#......
## build graph
with tf.variable_scope('cnn'):
md=cnn_dn(gp_dim)
f_X = tf.cast(md(X), dtype=float_type)
f_Xtest = tf.cast(md(Xtest), dtype=float_type)
#......
## predict
res=np.argmax(sess.run(my, feed_dict={Xtest:xts}),1).reshape(yts.shape)
correct = res == yts.astype(int)
print(np.average(correct.astype(float)))
I finally figure out that the solution is training larger iterations. In the original code, I just use 50 iterations as used in the ex2() function for MNIST data and it is not enough for more complicated network and CIFAR10 data. Adjusting some hyper-parameter (e.g. learning rate and activation function) also helps.
Assuming I have a very simple neural network, like multilayer perceptron. For each layer the activation function is sigmoid and the network are fully connected.
In TensorFlow this might be defined like this:
sess = tf.InteractiveSession()
# Training Tensor
x = tf.placeholder(tf.float32, shape = [None, n_fft])
# Label Tensor
y_ = tf.placeholder(tf.float32, shape = [None, n_fft])
# Declaring variable buffer for weights W and bias b
# Layer structure [n_fft, n_fft, n_fft, n_fft]
# Input -> Layer 1
struct_w = [n_fft, n_fft]
struct_b = [n_fft]
W1 = weight_variable(struct_w, 'W1')
b1 = bias_variable(struct_b, 'b1')
h1 = tf.nn.sigmoid(tf.matmul(x, W1) + b1)
# Layer1 -> Layer 2
W2 = weight_variable(struct_w, 'W2')
b2 = bias_variable(struct_b, 'b2')
h2 = tf.nn.sigmoid(tf.matmul(h1, W2) + b2)
# Layer2 -> output
W3 = weight_variable(struct_w, 'W3')
b3 = bias_variable(struct_b, 'b3')
y = tf.nn.sigmoid(tf.matmul(h2, W3) + b3)
# Calculating difference between label and output using mean square error
mse = tf.reduce_mean(tf.square(y - y_))
# Train the Model
# Gradient Descent
train_step = tf.train.GradientDescentOptimizer(0.3).minimize(mse)
The design target for this model is to map a n_fft points fft spectrogram to another n_fft target spectrogram. Let's assume both the training data and target data are of size [3000, n_fft]. They are stored in variables spec_train and spec_target.
Now here comes the question. For TensorFlow is there any difference between these two trainings?
Training 1:
for i in xrange(200):
train_step.run(feed_dict = {x: spec_train, y_: spec_target})
Training 2:
for i in xrange(200):
for j in xrange(3000):
train = spec_train[j, :].reshape(1, n_fft)
label = spec_target[j, :].reshape(1, n_fft)
train_step.run(feed_dict = {x: train, y_: label})
Thank you very much!
In the first training version, you are training the entire batch of training data at once, which means that the first and the 3000th element of spec_train will be processed using the same model parameters in a single step. This is known as (Batch) Gradient Descent.
In the second training version, you are training a single example from the training data at once, which means that the 3000th element of spec_train will be processed using model parameters that have been updated 2999 times since the first element was most recently processed. This is known as Stochastic Gradient Descent (or it would be if the element was selected at random).
In general, TensorFlow is used with datasets that are too large to process in one batch, so mini-batch SGD (where a subset of the examples are processed in one step) is favored. Processing a single element at a time is theoretically desirable, but is inherently sequential and has high fixed costs because the matrix multiplications and other operations are not as computationally dense. Therefore, processing a small batch (e.g. 32 or 128) of examples at once is the usual approach, with multiple replicas training on different batches in parallel.
See this Stats StackExchange question for a more theoretical discussion of when you should use one approach versus the other.
Yes there's a difference. I think the second way loss function can be bit messy. It's more like online training. For each data point in the whole batch you update all of your parameters. But in the first way it's called the batch gradient where you take one batch at a time and take the average loss then update the parameters.
Please refer this link
https://stats.stackexchange.com/questions/49528/batch-gradient-descent-versus-stochastic-gradient-descent
First answer is really good in this link
I have been using matlab's neural network toolbox lately for my research. I created some neural networks using fitting tool of this toolbox. Now I encountered a problem while testing the network with new data. Basically, I want to test the network with some new data, which do not have the same sample number as the network. For example, I have created the network with the input of 12x36 matrix (12 variables, 36 samples) and output of 1x36 vector (1 variable, 36 samples). Now, I want to get the results for a new data (12x11520 (12 variables and 11520 samples)). However, when I put this data into the network and get the results, I only get an output vector of 1x36 (1 variable 36 samples) while I was expecting to get the results as 1x11520 (1 variable and 11520 samples). I am using the below line to get the results from the neural network named as network.
output_estimate = network(input');
I also tried the line below to get the results but the result did not change.
output_estimate = sim(network,input');
Could you please help me understand this and get the output as the same sample length so that I can compare the expected and actual results for the new data.
Thank you very much in advance.
Irem
I am working on a project thumb recognition. following is code I am reading the 118 images of order 42 X 25 and storing them in training matrix.
training=zeros(118, 1050);
imagefiles = dir('*.png');
nfiles = length(imagefiles);
for ii=1:nfiles
currentfilename = imagefiles(ii).name;
I = imread(currentfilename);
BW=im2bw(I,graythresh(I));
temp = reshape(BW,1,1050);
training(ii,:)=temp;
end
Now I am creating a matrix of labelData to assign labels to images.
labelData = zeros(118,1);
labelData(1:50,:) = 0;
labelData(51:83,:) = 1;
labelData(84:118,:) = 2;
Here i am training my system by giving training data and label data.
options=optimset('MaxIter',5000);
SVMStruct = svmtrain(training,labelData,'Kernel_Function','linear','QuadProg_Opts',options);
BUT when I run this code it is giving me an error like
Error 1 : SVMTRAIN only supports classification into two groups. GROUP contains 3 groups.
Error 2 : SVMStruct = svmtrain(training,labelData,'Kernel_Function','linear','QuadProg_Opts',options);
Kindly help me what is the problem I used it before it was working fine but now I dont know what is going on. Thanks in advance.
Error 1 tells you what the problem is - the MATLAB built-in SVM only supports binary classification. You are assigning 3 classes.
Your options are:
Construct three classifiers: 0 vs. 1,2 then 1 vs. 0,2 then 2 vs. 0,1 and look at the output of each.
Construct 0 vs. not 0 and then 1 vs. 2
Use a multi-class SVM trainer from LIBSVM or svmlight or other such packages.
The error message is pretty clear. MATLAB's svmtrain does not support multiclass classification, that is only two classes are allowed.
So, you have two options: 1) write your own multiclass classifier as a wrapper around svmtrain. You can implement one-vs-all or one-vs-one strategies. 2) use a svm implementation that already supports multiclass classification such as libsvm.
Your problem is in the labelData vector ceck it and find the eror, yoy shoild OAA architector if hthe number of classes is more then .
I'm doing research in speech recognition and I'm trying to design a algorithm for speech recognition using neural network in Matlab. How can i define my structure for the network??
Matlab has a Neural Network Toolbox. Maybe you can take a look at that?
Your question is very general. You should us give more specific details of what you want to do.
However, I give you some tips on using Matlab neural network toolbox out of my experience with License Plate Recognition using this toolbox. hope it helps.
1-You will need to know how many hidden layers you'll be using as well as number of your outputs. In my case, I specified 20 hidden layers, and 9 outputs.
2-You need to have a dataset first. You will be using this dataset for training your neural network. In case of LPR, I used a 90x50 matrix as my dataset. every row represented a digit, holding 50 numbers extracted from a digit's image.
3- You will need a targets matrix to map your dataset to known outputs(hence called training).
The following syntax defines the network mentioned:
net = newff(minmax(datasetNormalized'),[20 9],{'logsig' 'logsig'},'traingdx');
For training the network we write:
[net,tr]=train(net,datasetNormalized',T);
We give a new input to the network using the following code:
[dummy,b]=max(sim(net,m_normalized'));
b is the answer we are looking for here(output with most probability), Therefore we display it to user:
msgbox(['digit is: ' num2str(b)],'Digit recognized','help');
This is the full source code if you want to know exactly what my code is doing:
clc
clear
close all
numOfPhotos=90;
imgRows=100;
imgCols=50;
X=zeros(numOfPhotos,(imgRows*imgCols)/100);
%%Resize Images
% myresize(imgRows,imgCols);
%read train images
datasetIndex=0;
for i=1:numOfPhotos/10
for j=1:numOfPhotos/9
datasetIndex=datasetIndex+1;
im=imread(['resized_train_numbers\' num2str(i) ' (' num2str(j) ').jpg']);
im=im2bw(im,graythresh(im));
c=1;
for g=1:imgRows/10
for e=1:imgCols/10
s=sum(sum(im((g*10-9:g*10),(e*10-9:e*10))));
X(datasetIndex,c)=s;
c=c+1;
end
end
end
end
datasetNormalized=zeros(numOfPhotos,imgRows*imgCols/100);
%%Normalize dataset contents
minDataset=min(min(X));
maxDataset=max(max(X));
for i = 1:numOfPhotos
for j=1:imgRows*imgCols/100
datasetNormalized(i, j) = (X(i,j)-minDataset)/(maxDataset-minDataset);
end
end
%
%%Neural network part
% T=zeros(1,90);
% for i=1:90
% T(i)=ceil(i/10);
% end
T=zeros(9,90);
for j=1:90
i=ceil(j/10);
T(i,j)=1;
end
% net=newff(datasetNormalized',T,20);
net = newff(minmax(datasetNormalized'),[20 9],{'logsig' 'logsig'},'traingdx');
net.performFcn='sse';
net.trainParam.goal=0.01;
net.trainParam.show=20;
net.trainParam.epochs=100;
net.trainParam.mc=0.95;
% net.trainFcn='trainlm';
net.trainParam.min_grad=1e-12;
[net,tr]=train(net,datasetNormalized',T);
%Read input image for recognition
[name file]=uigetfile('*.jpg','Choose Plate Digit Image');
newImg=imread([file name]);
newImg=imresize(newImg,[imgRows imgCols]);
newImg=im2bw(newImg,graythresh(newImg));
figure,imshow(newImg);
m=zeros(1,imgRows*imgCols/100);
c=1;
for g=1:imgRows/10
for e=1:imgCols/10
s=sum(sum(newImg((g*10-9:g*10),(e*10-9:e*10))));
m(c)=s;
c=c+1;
end
end
%Normalize m contents
m_normalized=zeros(1,imgRows*imgCols/100);
for i=1:imgRows*imgCols/100
m_normalized(i)=(m(i)-min(m))/(max(m)-min(m));
end
[dummy,b]=max(sim(net,m_normalized'));
% b=round(sim(net,m_normalized'));
msgbox(['digit is: ' num2str(b)],'Digit recognized','help');