I am trying to use the NETLAB toolbox to train a 3-layer (input,hidden,output) feed-forward backpropagation Neural Network. Unfortunately I do not have too much freedom in terms of network architecture I can work with.
I notice NETLAB has the following functions that I need: mlp,mlpbkp,mlpfwd,mlpgrad. I am not sure in what order I need to call the above functions to train the network. The help manual is not of too much help either.
If any of you have used the NETLAB toolbox, kindly let me know.
Also, if you know of other free toolboxes I can use in lieu of NETLAB, kindly, let me know.
Thanks!
You can find some basic examples on usage of NETLAB online here, the following is just the header:
A Simple Program The "Hello world" equivalent in Netlab is a programme
that generates some data, trains an MLP, and plots its predictions.
The online demo is a brief version of a longer demo available with the program, and uses functions mlp and mlpfwd.
In the downloads page you'll find that you can download help files, too.
If you get stuck you may (perhaps as a last resort) want to contact the authors.
edit
I understand that pointing to help files might not be what you were looking for. As you rightly point out, there is little documentation (perhaps more importantly no demos that I could find) on performing backpropagation, and definitely not with 3 layers. The available function mlpbkp backpropagates for a 2-layer network.
Related
I have finished two neural network courses and done loads of reading on the subject. I am comfortable with Tensorflow and Keras and building advanced neural networks (multiple inputs, large data, special layers...). I also have a fairly deep understanding of the underlying mathematics.
My problem is that I know how to build neural networks but don't know the process by which an "expert" would create one for a specific application.
I can:
Collect loads of data and clean it up.
Train the neural network.
Fine tune hyper parameters.
Export it for actual applications.
What I am missing is how to come up with the layers in the neural network (how wide, what kind...). I know it is somewhat trial and error and looking at what has worked for others. But there must be a process that people can use to come up with architectures* that actually work very well. For example state of the art neural networks.
I am looking for a free resource that would help me understand this process of creating a very good architecture*.
*by architecture I mean the different layers that make up the network and their properties
I wrote my masters thesis about the topic:
Thoma, Martin. "Analysis and Optimization of Convolutional Neural Network Architectures." arXiv preprint arXiv:1707.09725 (2017).
Long story short: There are a couple of techniques for analysis (chapter 2.5) and algorithms that learn topoligies (chapter 3), but in practice it is mostly trial and error / gut feeling.
I'll start this post by saying that I acknowledge this may not be the appropriate venue for this question, but wasn't sure where else to start. If there is a more appropriate SE channel, please feel free to suggest.
I've been using Keras for learning how to apply neural networks to different prediction problems. I'm interested in learning TensorFlow as a way to gain a deeper understanding of the inner working of these networks. Obviously, it's possible to switch the backend of Keras to TensorFlow and to use Keras as a high-level API to TensorFlow. However, is there a way to "recover" the TensorFlow code from a compiled Keras model? I'm thinking it would be extremely useful to be able to write a model that I'm familiar with in Keras, and automatically see it's "translation" to TensorFlow as a way to learn this library more quickly.
Any thoughts or suggestions would be helpful. Thanks for reading.
All that Keras is doing is to abstract both Theano and TensorFlow into one unified backend module. Then it uses the functions in the backend to implement the layers and methods you are able to use in Keras.
This in turn means that there is no compilation step involved in generating code for one particular backend. Both Theano and TensorFlow are python libraries, there is no reason for a translation step, Keras just uses the library you specify.
The best way to find out how a model in Keras is written in TensorFlow is probably to search for a simple network with the same dataset and compare examples in TensorFlow and Keras. Another way would be to read the Keras code and lookup the K.<function> in the TensorFlow backend module.
If you are interested in the platform specific code that the individual backends produce, e.g. the CUDA code, then the answer is: it depends. Both Theano and TensorFlow use temporary directories to store the code and the sources. For theano this is ~/.theano by default. But looking at this code will probably not make you any wiser in understanding neural networks and their mechanics.
I'm being required to train a perceptron in Matlab to learn a classification data set (any, really). The only restriction is that the data set must come from the UCI Machine Learning Repository. The problem is that I have really no idea where to begin as my teacher is extremely bad at what he does and never explained it well. I've tried asking other class-mates for help but none of them seem to have the answers. I hope I can get help from this community as it's my last chance. Thank you guys.
Well, we're really not your last chance. There are plenty of tutorials, examples, and resources findable easily from Google that would help (for example, search on "MATLAB perceptron iris" - the iris dataset is a famous example dataset, included in the UCI repository).
But here's a start. I'm assuming that if you've been set the task of training a perceptron in MATLAB, then you have access to Neural Network Toolbox (if you're asking how to implement a perceptron algorithm from scratch in MATLAB, look in a textbook).
Type doc nnet. That will bring up the documentation for Neural Network Toolbox. Then click through to the section labelled "Examples". Scrolling down to the bottom, there are several demos in a section called "Perceptrons". Try looking at the demos "Classification with a 2-Input Perceptron" or "Linearly Non-separable Vectors". Those demos use toy datasets, but should give you an idea of how to train a perceptron.
Then scroll up to the section "Pattern Recognition and Classification", and take a look at the demo "Wine Classification". The Wine dataset this demo uses is part of the UCI repository. Adapt and combine the demos you've now learnt from, to create an example your prof will like.
Neural Network Toolbox also comes with the Iris dataset that is part of the UCI repository. You may also find a demo somewhere that uses this as an example.
Hope that helps!
I am new in neural networks and I need to determine the pattern among a given set of inputs and outputs. So how do I decide which neural network to use for training or even which learning method to use? I have little idea about the pattern or relation between the given input and outputs.
Any sort of help will be appreciated. If you want me to read some stuff then it would be great if links are provided.
If any more info is needed plz say so.
Thanks.
Choosing the right neural networks is something of an art form. It's a bit difficult to give generic suggestions as the best NN for a situation will depend on the problem at hand. As with many of these problems neural netowrks may or may not be the best solution. I'd highly recommned trying out different networks and testing their performance vs a testing data set. When I did this I usually used the ANN tools though the R software package.
Also keep your mind open to other statistical learning techniques as well, things like decision trees and Support Vector Machines may be a better choice for some problems.
I'd suggest the following books:
http://www.amazon.com/Neural-Networks-Pattern-Recognition-Christopher/dp/0198538642
http://www.stats.ox.ac.uk/~ripley/PRbook/#Contents
I'm quite new with this topic so any help would be great. What I need is to optimize a neural network in MATLAB by using GA. My network has [2x98] input and [1x98] target, I've tried consulting MATLAB help but I'm still kind of clueless about what to do :( so, any help would be appreciated. Thanks in advance.
Edit: I guess I didn't say what is there to be optimized as Dan said in the 1st answer. I guess most important thing is number of hidden neurons. And maybe number of hidden layers and training parameters like number of epochs or so. Sorry for not providing enough info, I'm still learning about this.
If this is a homework assignment, do whatever you were taught in class.
Otherwise, ditch the MLP entirely. Support vector regression ( http://www.csie.ntu.edu.tw/~cjlin/libsvm/ ) is much more reliably trainable across a broad swath of problems, and pretty much never runs into the stuck-in-a-local-minima problem often hit with back-propagation trained MLP which forces you to solve a network topography optimization problem just to find a network which will actually train.
well, you need to be more specific about what you are trying to optimize. Is it the size of the hidden layer? Do you have a hidden layer? Is it parameter optimization (learning rate, kernel parameters)?
I assume you have a set of parameters (# of hidden layers, # of neurons per layer...) that needs to be tuned, instead of brute-force searching all combinations to pick a good one, GA can help you "jump" from this combination to another one. So, you can "explore" the search space for potential candidates.
GA can help in selecting "helpful" features. Some features might appear redundant and you want to prune them. However, say, data has too many features to search for the best set of features by some approaches such as forward selection. Again, GA can "jump" from this set candidate to another one.
You will need to find away to encode the data (input parameters, features...) fed to GA. For finding a set of input paras or a good set of features, I think binary encoding should work. In addition, choosing operators for GA to reproduce offsprings is also important. Yet GA needs to be tuned, too (early stopping which can also be applied to ANN).
Here are just some ideas. You might want to search for more info about GA, feature selection, ANN pruning...
Since you're using MATLAB already I suggest you look into the Genetic Algorithms solver (known as GATool, part of the Global Optimization Toolbox) and the Neural Network Toolbox. Between those two you should be able to save quite a bit of figuring out.
You'll basically have to do 2 main tasks:
Come up with a representation (or encoding) for your candidate solutions
Code your fitness function (which basically tests candidate solutions) and pass it as a parameter to the GA solver.
If you need help in terms of coming up with a fitness function, or encoding of candidate solutions then you'll have to be more specific.
Hope it helps.
Matlab has a simple but great explanation for this problem here. It explains both the ANN and GA part.
For more info on using ANN in command line see this.
There is also plenty of litterature on the subject if you google it. It is however not related to MATLAB, but simply the results and the method.
Look up Matthew Settles on Google Scholar. He did some work in this area at the University of Idaho in the last 5-6 years. He should have citations relevant to your work.