I have a structure for my CNN that will be used in image enhancement. and I want to know how to use Caffe package in Matlab to design and train the network. I don't need to import pretrained network from Caffe as I have a specific structure for the CNN. Does any one have a link or example that guide me on how to do that? Any help will be appreciated.
The Caffe official documentation has a simple example showcasing basic interfaces for Matlab including adding conv layers to the network:
http://caffe.berkeleyvision.org/tutorial/interfaces.html#matlab
Once you get the basics for matlab, the interfaces are mostly similar to python.
Hope that helps
Is it possible to convert a trained convolutional neural network saved as .mat into a .pb format?
I was thinking to import it in opencv using cv::dnn::readNetFromTensorflow()
Are there any alteratives, o I just should retrain the network directly in tensorflow or keras?
I just found about Open Neural Network Exchange.
You can simply save the network from matlab using exportONNXNetwork(net,filename) and load the network inside opencv using cv2.dnn.readNetFromONNX()
I'm going through a few research papers based on neural network, where I came across the word Fine Tuning on pre-trained CNN network. What does it actually do?
Pre-trained:
Firstly we have to understand pre-trained model. Pre-trained models are the models which weights are already trained by someone on a data-set. e.g VGG16 is trained on image-net. Now we want to classify imagenet images. than we can say that If we use pre-trained VGG16 we can classify them easily. Because VGG16 is already trained to classify imagenet objects we don't need to train that again.
Fine-Tuning:
Now I want to classify Cifar-10(classes-10) with VGG16 (classes-1000) and I want to use pre-trained models for this work. Now I have a model which is trained on Image-net which have 1000 classes. So Now I will change the last layer with 10 neurons with softmax activation because Now I want to classify 10 classes not 1000. Now I will fine-tune(change according to my need) my model. I will add a dense layer at the last of the model which have 10 neurons. Now I can use VGG16 (pre-trained for image-net). changing pre-trained model according to our need is known as fine-tuning.
Transfer Learning:
Now the whole concept using pre-trained model and use it to classify our data-set by fine-tuning model is known as transfer-learning
Transfer-learning Example(Using Pre-trained model and Fine-tune it for using it on my data-set)
Here I am using Dense-net pre-trained on image-net and fine-tune my model because I want to use VGG16 net model to classify images in my data-set. and my data set have 5 classes So I am adding last dense-layer having 5 neurons
model=Sequential()
dense_model=keras.applications.densenet.DenseNet121(include_top=False, weights='imagenet', input_tensor=None, input_shape=(224,224,3), pooling=None, classes=1000)
dense_model.trainable = False
dense_model.summary()
# Add the vgg convolutional base model
model.add(dense_model)
# Add new layers
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(128, activation='relu'))
model.add(keras.layers.Dense(5, activation='softmax'))
model.summary()
Pre-trained model link:
https://www.kaggle.com/sohaibanwaar1203/pretrained-densenet
Now what if I want to change the hyper-parameters of the pre-trained model. I want to check which (optimizer,loss-function,number of layers, number of neurons) is working well on my data-set if I use VGG16 (on my data-set). For this reason I will optimize my parameter known as hyper-parameter Optimization
Hyper-parameter Optimization:
if you have knowledge about neural networks you will know that we give random numbers to our neural network. e.g No of dense layers, Number of dense units, Activation's, Dropout percentage. We don't know that neural network with 3 layers will perform well on our data or neural network with 6 layers will perform well on our data. We do experimentation to get the best number for our model. Now experimentation in which you are finding best number for your model is known as fine tuning. Now we have some techniques to Optimize our model like
Grid Search, Random Search. I am sharing notebook by which you will be able to Optimize your model parameters with the help of code.
import math
from keras.wrappers.scikit_learn import KerasRegressor
import keras
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import RandomizedSearchCV, KFold
from sklearn.metrics import make_scorer
from keras.models import Sequential,Model
from keras.layers import Dense,Dropout,Activation,BatchNormalization
from keras import losses
from keras import optimizers
from keras.callbacks import EarlyStopping
from keras import regularizers
def Randomized_Model(lr=0.0001,dropout=0.5,optimizer='Adam',loss='mean_squared_error',
activation="relu",clipnorm=0.1,
decay=1e-2,momentum=0.5,l1=0.01,l2=0.001,
):
#Setting Numbers of units in Every dense layer according to the number of dense layers
no_of_units_in_dense_layer=[]
#backwards loop
#setting up loss functions
loss=losses.mean_squared_error
if(loss=='mean_squared_error'):
loss=losses.mean_squared_error
if(loss=="poisson"):
loss=keras.losses.poisson
if(loss=="mean_absolute_error"):
loss=keras.losses.mean_absolute_percentage_error
if(loss=="mean_squared_logarithmic_error"):
loss=keras.losses.mean_squared_logarithmic_error
if(loss=="binary_crossentropy"):
loss=keras.losses.binary_crossentropy
if(loss=="hinge"):
loss=keras.losses.hinge
#setting up Optimizers
opt=keras.optimizers.Adam(lr=lr, decay=decay, beta_1=0.9, beta_2=0.999)
if optimizer=="Adam":
opt=keras.optimizers.Adam(lr=lr, decay=decay, beta_1=0.9, beta_2=0.999)
if optimizer=="Adagrad":
opt=keras.optimizers.Adagrad(lr=lr, epsilon=None, decay=decay)
if optimizer=="sgd":
opt=keras.optimizers.SGD(lr=lr, momentum=momentum, decay=decay, nesterov=False)
if optimizer=="RMSprop":
opt=keras.optimizers.RMSprop(lr=lr, rho=0.9, epsilon=None, decay=0.0)
if optimizer=="Adamax":
opt=keras.optimizers.Adamax(lr=lr, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0)
#model sequential
model=Sequential()
model.add(Dense(units=64,input_dim=30,activation=activation))
model.add(Dense(units=32,activation=activation))
model.add(Dense(units=8,activation=activation))
model.add(Dense(units=1))
model.compile(loss=loss ,optimizer=opt)
return model
params = {'lr': (0.0001, 0.01,0.0009,0.001,0.002 ),
'epochs': [50,100,25],
'dropout': (0, 0.2,0.4, 0.8),
'optimizer': ['Adam','Adagrad','sgd','RMSprop','Adamax'],
'loss': ["mean_squared_error","hinge","mean_absolute_error","mean_squared_logarithmic_error","poisson"],
'activation' :["relu","selu","linear","sigmoid"],
'clipnorm':(0.0,0.5,1),
'decay':(1e-6,1e-4,1e-8),
'momentum':(0.9,0.5,0.2),
'l1': (0.01,0.001,0.0001),
'l2': (0.01,0.001,0.0001),
}
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import RandomizedSearchCV, KFold
from sklearn.metrics import make_scorer
# model class to use in the scikit random search CV
model = KerasRegressor(build_fn=Randomized_Model, epochs=30, batch_size=3, verbose=1)
RandomizedSearchfit = RandomizedSearchCV(estimator=model, cv=KFold(3), param_distributions=params, verbose=1, n_iter=10, n_jobs=1)
#having some problem in this line
RandomizedSearch_result = RandomizedSearchfit.fit(X, Y )
Now give your X and Y to this model it will find the best parameter selected by you in the param_dict variable. You can also check fine-tuning of CNN in this notebook (Click Here) In this Notebook I am using Talos Library to fine tune my model.
This is another notebook in which I am using SKLearn (Randomised and grid search )to fine tune my model (Click Here)
Fine-tuning is usually called the last step of more complex NN training when you only slightly modify a pre-trained network, usually to improve performance on a specific domain or re-use good input representation in a different task.
Often, it is mentioned in context of transfer learning. E.g., for image recognition, it may mean that you take a network that was trained to recognize 1k classes from ImageNet. You take the pre-trained network and only "fine-tune" the last layer on your task-specific (smaller and presumably simpler dataset).
I would train an alexnet DNN (given by MATLAB function alexnet) from scratch (i.e. without pretraining on ImageNet given by alexnet function). I could to manually set weights but I don't know the from what distribution I can sample my initial weights. Is there a built-in MATLAB option that make it for me?
For example, I've read that Python's library has the option pre-training=off but I don't find a similar option in MATLAB.
I'm using libsvm in matlab, and it seems there no existing method to save the model formed from svmtrain. Instead, the functions provided forces me to retrain everytime. Just saving the svmtrain model variable in a .mat does not work. What should I be doing?
The nice thing about SVM, unlike NN is that training is fast and in some cases can be even done online. I have a multiclass SVM and I save the training vectors, classes and the kernel parameters into a txt file.