I was trying to apply a tansig or tanh function on my fixpointed data which I am using for my neural nework in MatLab, but when I use these functions on embedded.fi files, MatLab says that tanh or tansig function will not work on embedded.fi.
I am trying to set my neural network using fixpointed weights. I will really appreciate it if anyone has solution for that
As the error message states, there is no fixed-point version of tanh and tansig functions in Matlab. The complete list of functions that have fixed-point versions and are supported can be found here:
http://www.mathworks.com/help/hdlcoder/ug/fixed-point-run-time-library-support.html
Related
This question is quite general and mathematical.
I'm using a neural network with a single hidden layer (10 neurons) to approximate a function with 2 input and 2 output variables.
The sigmoid function is used as activation function.
Later, I'm using the derivative for something else. I approximate the derivative with a numerical method. This method uses outputs of the neural network.
I was wondering if the derivative of the approximated function can be obtained easily as the learning algorithm uses derivatives for every neuron.
I could get rid of the numerical method for the derivation if there was a simple way to get the derivative out of the neural net.
I'm thinking of the chain rule, but I'm not sure if this is the right way to go and how to use it correctly.
I have been experimenting with neural networks these days. I have come across a general question regarding the activation function to use. This might be a well known fact to but I couldn't understand properly. A lot of the examples and papers I have seen are working on classification problems and they either use sigmoid (in binary case) or softmax (in multi-class case) as the activation function in the out put layer and it makes sense. But I haven't seen any activation function used in the output layer of a regression model.
So my question is that is it by choice we don't use any activation function in the output layer of a regression model as we don't want the activation function to limit or put restrictions on the value. The output value can be any number and as big as thousands so the activation function like sigmoid to tanh won't make sense. Or is there any other reason? Or we actually can use some activation function which are made for these kind of problems?
for linear regression type of problem, you can simply create the Output layer without any activation function as we are interested in numerical values without any transformation.
more info :
https://machinelearningmastery.com/regression-tutorial-keras-deep-learning-library-python/
for classification :
You can use sigmoid, tanh, Softmax etc.
If you have, say, a Sigmoid as an activation function in output layer of your NN you will never get any value less than 0 and greater than 1.
Basically if the data your're trying to predict are distributed within that range you might approach with a Sigmoid function and test if your prediction performs well on your training set.
Even more general, when predict a data you should come up with the function that represents your data in the most effective way.
Hence if your real data does not fit Sigmoid function well you have to think of any other function (e.g. some polynomial function, or periodic function or any other or a combination of them) but you also should always care of how easily you will build your cost function and evaluate derivatives.
Just use a linear activation function without limiting the output value range unless you have some reasonable assumption about it.
I don't quite understand why a sigmoid function is seen as more useful (for neural networks) than a step function... hoping someone can explain this for me. Thanks in advance.
The (Heaviside) step function is typically only useful within single-layer perceptrons, an early type of neural networks that can be used for classification in cases where the input data is linearly separable.
However, multi-layer neural networks or multi-layer perceptrons are of more interest because they are general function approximators and they are able to distinguish data that is not linearly separable.
Multi-layer perceptrons are trained using backpropapagation. A requirement for backpropagation is a differentiable activation function. That's because backpropagation uses gradient descent on this function to update the network weights.
The Heaviside step function is non-differentiable at x = 0 and its derivative is 0 elsewhere. This means gradient descent won't be able to make progress in updating the weights and backpropagation will fail.
The sigmoid or logistic function does not have this shortcoming and this explains its usefulness as an activation function within the field of neural networks.
It depends on the problem you are dealing with. In case of simple binary classification, a step function is appropriate. Sigmoids can be useful when building more biologically realistic networks by introducing noise or uncertainty. Another but compeletely different use of sigmoids is for numerical continuation, i.e. when doing bifurcation analysis with respect to some parameter in the model. Numerical continuation is easier with smooth systems (and very tricky with non-smooth ones).
I am working on an artificial neural network. I want to implement it in Matlab, but I am unable to find a proper activation function. I need a step function because my output is either 0 or 1. is there any function in Matlab that can be used for this kind of output. Also, I want the reverse function of the same activation function. logsig and tansig are not working for me.
Both tansig and logsig are part of the Neural Network Toolbox as the online documentation makes clear. So, if which tansig returns nothing, then you don't have that toolbox (or at least don't have a version current enough to contain that function). However, both of these functions are extremely simple, and the documentation even gives you the formulae under the "Algorithms" section: tansig, logsig. Both can be implemented as a one line anonymous function if you wanted.
If your question is actually about how to produce a Heaviside step function, Matlab has heaviside (it's part of the Symbolic Math toolbox but a pure numeric version is included – type edit heaviside to see the simple code). However, note that using such a non-differentiable function is problematic for some types of neural networks as this StackOverflow question and answer addresses.
Heaviside did not work for me.. i finally normalized my data between 1 and -1 and then applied tansig.
Thanks
In the lectures we only mention how to train the RBF network with Gausian function and how to use the "newrb" tool box in Matlab. But in the assignemnet I need to create my own RBF network which using the NN toolbox is forbidden. Basically I not even know how to start it and our professor not willing to provide any information.
With some tips I have write my own program but the performance is very bad, I am wonder if any one can give me some helpful tutorial or guides that how to create the RBF network with Gaussian function without using NN toolbox.
I have used k-means to obtain the centers and gaussian function to caculuate the weights, the main probrlem is that I have no idea how to design the method that transform the Input matrix to the RBF matrix. Hope you can help.
This is clearly homework, and it's not clear what your question is. But I think you are wondering how to create the Gram matrix. If so, see:
http://en.wikipedia.org/wiki/Gramian_matrix
You should have the math for how to do each step in your textbook and/or notes.