I have implemented neural networks with back-propagation for learning and it works just fine for XOR but when i tried it for AND and OR it behaves erratic.. during debugging i found out that after certain while in training the output turns 1.0 and is consistent. and when i put the test inputs it returns me 1 only for every case. does this has to do something with bias because i have forced bias to be 1.0 and also do i need to update bias too, though it seems to be a illogical question.. thanks
You don't need to update bias neuron output value (it is ok to set it to 1), but you do need to update bias weights (weights of connections that come from bias neuron to the layer above).
Related
Having neural network with alot of inputs causes my network problems like
Neural network gets stuck and feed forward calculation always gives output as
1.0 because of the output sum being too big and while doing backpropagation, sum of gradients will be too high what causes the
learning speed to be too dramatic.
Neural network is using tanh as an active function in all layers.
Giving alot of thought, I came up with following solutions:
Initalizing smaller random weight values ( WeightRandom / PreviousLayerNeuronCount )
or
After calculation the sum of either outputs or gradients, dividing the sum with the number of 'neurons in previus layer for output sum' and number of 'neurons in next layer for gradient sum' and then passing sum into activation/derivative function.
I don't feel comfortable with solutions I came up with.
Solution 1. does not solve problem entirely. Possibility of gradient or output sum getting to high is still there. Solution 2. seems to solve the problem but I fear that it completely changes network behavior in a way that it might not solve some problems anymore.
What would you suggest me in this situation, keeping in mind that reducing neuron count in layers is not an option?
Thanks in advance!
General things that affect the output backpropagation include weights and biases of early elections, the number of hidden units, the amount of exercise patterns, and long iterations. As an alternative way, the selection of initial weights and biases there are several algorithms that can be used, one of which is an algorithm Nguyen widrow. You can use it to initialize the weights and biases early, I've tried it and gives good results.
If the output is a tanh function, then I get a number between -1 and 1.
How do I go about converting the output to the scale of my y values (which happens to be around 15 right now, but will vary depending on the data)?
Or am I restricted to functions which vary within some kind of known range...?
Just remove the tanh, and your output will be an unrestricted number. Your error function should probably be squared error.
You might have to change the gradient calculation for your back-prop, if this isn't done automatically by your framework.
Edit to add: You almost certainly want to keep the tanh (or some other non-linearity) between the recurrent connections, so remove it only for the output connection.
In most RNNs for classification, most people use a softmax layer on top of their LSTM or tanh layers so I think you can replace the softmax with just a linear output layer. This is what some people do for regular neural networks as well as convolutional neural networks. You will still have the nonlinearity from the hidden layers, but your outputs will not be restricted within a certain range such as -1 and 1. The cost function would probably be the squared error like larspars mentioned.
I am trying to code my own neural networks, but there is something I don't understand about bias terms. I know each link of a neuron to another neuron has a weight, but does a link between a bias and the neuron its connected to also have a weight? Or can I think of the weight always 1 and never gets changed?
Thanks
The bias terms do have weights, and typically, you add bias to every neuron in the hidden layers as well as the neurons in the output layer (prior to squashing).
Have a look at the basic structure of Artificial Neurons, you see the bias is added as wk0 = bk. For more thorough examples, see e.g. this link, containing formulas as well as visualisation of multi-layered NN.
For the discussion of choice of weights, refer to the following stats.stackexchange thread:
https://stats.stackexchange.com/questions/47590/what-are-good-initial-weights-in-a-neural-network
I have a neural network with 2 entry variables, 1 hidden layer with 2 neurons and the output layer with one output neuron. When I start with some randomly (from 0 to 1) generated weights, the network learns the XOR function very fast and good, but in other cases, the network NEVER learns the XOR function! Do you know why this happens and how can I overcome this problem? Could some chaotic behaviour be involved? Thanks!
This is quite normal situation, because error function for multilayer NN is not convex, and optimization converges to local minimum.
You can just keep initial weights that resulted in successful optimization, or run optimizer multiple times starting from different weights, and keep the best solution. Optimization algorithm and learning rate also plays certain role, for example backpropagation with momentum and/or stochastic gradient descent sometimes work better. Also, if you add more neurons, beyond the minimum needed to learn XOR, this also helps.
There exist methodologies designed to find global minimum, such as simulated annealing, but, in practice they are not commonly used for NN optimization, except for some specific cases
I have ran some classification tests in Matlab with feed forward network. Using the standard tansig function the results were better when using more neurons on the hidden layer.
But, when I switched to pure lin I was surprised to see that the results were better when I set a smaller number of neurons on the hidden layer.
Can you help me with an argument for these situation?
The tansig activation function essentially makes it possible than a neuron becomes inactive due to saturation. A linear neuron is always active. Therefore if one linear neuron has bad parameters, it will always affect the outcome of the classification. A higher number of neurons yield a higher probability of bad behavior in this scenario.