I have a simple competitive neural network algorithm which I am trying to enhance by penalizing the 'rival', ie: The output neuron with the second-largest output. My code to do so is this:
% Penalizing rival
network_output(I) = [];
[~, I] = max(network_output);
delta = (1/current_run^3)*(sample'-connections(:,I));
connections(:,I) = normc(connections(:,I) - delta);
Essentially, I removed the winner (neuron at index I) from the outputs, I then find the new winner (overall second), and I updated the connections by reducing the strength of those leading to such neuron using the standard rule. I also apply a decay in terms of current_run^3 where current_run is the iteratio numebr. I also normalize all weights before re-inserting them in the connection matrix.
However, by comparing the number of dead units to the network running without this refinement, I find that the latter performs better.
Any advice?
http://en.m.wikipedia.org/wiki/Competitive_learning
I just read that article and there it says
3. The winner updates each of its weights, moving weight from the connections that gave it weaker signals to the connections that gave it stronger signals.
Also there I didn't find that you need to remove winner.
So, I not sure that you need to remove winner. Maybe that's why network with winner works better?
Sometime it is good idea to remove nodes from neural network during optimization, but after each removal network need too be retrained.
Related
After reading a few papers on Neuro Evolution, more specifically NEAT, I realised that there was very little information regarding how you should weight each synapse at the start of the Neural Network. I understand that at the start, using NEAT, all the input neurons are connected to the output neuron, and then evolution takes place from there. However, should you weight each synapse randomly at the start, or simply set each one to 1?
It doesn't really matter a lot - it matters most how you mutate the weights of the connections in a genome.
However, setting the weights of each genome's connections to a random value is best: it acts like a small random search in the 'right' direction. If you'd set all the weights the same in for each genome, then weights in genomes will be extremely similar: keep in mind that a genome has a lot of connections, and with a mutation rate of 0.3 and two mutation options for example, only 15% of the population will have at least óne different weight after just 1 generation.
So make it something random, like random() * .2 - .1 (distribute between [-0.1, 0.1]). Just figure out what values work best for you.
I'm trying to navigate an agent in a n*n gridworld domain by using Q-Learning + a feedforward neural network as a q-function approximator. Basically the agent should find the best/shortest way to reach a certain terminal goal position (+10 reward). Every step the agent takes it gets -1 reward. In the gridworld there are also some positions the agent should avoid (-10 reward, terminal states,too).
So far I implemented a Q-learning algorithm, that saves all Q-values in a Q-table and the agent performs well.
In the next step, I want to replace the Q-table by a neural network, trained online after every step of the agent. I tried a feedforward NN with one hidden layer and four outputs, representing the Q-values for the possible actions in the gridworld (north,south,east, west).
As input I used a nxn zero-matrix, that has a "1" at the current positions of the agent.
To reach my goal I tried to solve the problem from the ground up:
Explore the gridworld with standard Q-Learning and use the Q-map as training data for the Network once Q-Learning is finished
--> worked fine
Use Q-Learning and provide the updates of the Q-map as trainingdata
for NN (batchSize = 1)
--> worked good
Replacy the Q-Map completely by the NN. (This is the point, when it gets interesting!)
-> FIRST MAP: 4 x 4
As described above, I have 16 "discrete" Inputs, 4 Output and it works fine with 8 neurons(relu) in the hidden layer (learning rate: 0.05). I used a greedy policy with an epsilon, that reduces from 1 to 0.1 within 60 episodes.
The test scenario is shown here. Performance is compared beetween standard qlearning with q-map and "neural" qlearning (in this case i used 8 neurons and differnt dropOut rates).
To sum it up: Neural Q-learning works good for small grids, also the performance is okay and reliable.
-> Bigger MAP: 10 x 10
Now I tried to use the neural network for bigger maps.
At first I tried this simple case.
In my case the neural net looks as following: 100 input; 4 Outputs; about 30 neurons(relu) in one hidden layer; again I used a decreasing exploring factor for greedy policy; over 200 episodes the learning rate decreases from 0.1 to 0.015 to increase stability.
At frist I had problems with convergence and interpolation between single positions caused by the discrete input vector.
To solve this I added some neighbour positions to the vector with values depending on thier distance to the current position. This improved the learning a lot and the policy got better. Performance with 24 neurons is seen in the picture above.
Summary: the simple case is solved by the network, but only with a lot of parameter tuning (number of neurons, exploration factor, learning rate) and special input transformation.
Now here are my questions/problems I still haven't solved:
(1) My network is able to solve really simple cases and examples in a 10 x 10 map, but it fails as the problem gets a bit more complex. In cases where failing is very likely, the network has no change to find a correct policy.
I'm open minded for any idea that could improve performace in this cases.
(2) Is there a smarter way to transform the input vector for the network? I'm sure that adding the neighboring positons to the input vector on the one hand improve the interpolation of the q-values over the map, but on the other hand makes it harder to train special/important postions to the network. I already tried standard cartesian two-dimensional input (x/y) on an early stage, but failed.
(3) Is there another network type than feedforward network with backpropagation, that generally produces better results with q-function approximation? Have you seen projects, where a FF-nn performs well with bigger maps?
It's known that Q-Learning + a feedforward neural network as a q-function approximator can fail even in simple problems [Boyan & Moore, 1995].
Rich Sutton has a question in the FAQ of his web site related with this.
A possible explanation is the phenomenok known as interference described in [Barreto & Anderson, 2008]:
Interference happens when the update of one state–action pair changes the Q-values of other pairs, possibly in the wrong direction.
Interference is naturally associated with generalization, and also happens in conventional supervised learning. Nevertheless, in the reinforcement learning paradigm its effects tend to be much more harmful. The reason for this is twofold. First, the combination of interference and bootstrapping can easily become unstable, since the updates are no longer strictly local. The convergence proofs for the algorithms derived from (4) and (5) are based on the fact that these operators are contraction mappings, that is, their successive application results in a sequence converging to a fixed point which is the solution for the Bellman equation [14,36]. When using approximators, however, this asymptotic convergence is lost, [...]
Another source of instability is a consequence of the fact that in on-line reinforcement learning the distribution of the incoming data depends on the current policy. Depending on the dynamics of the system, the agent can remain for some time in a region of the state space which is not representative of the entire domain. In this situation, the learning algorithm may allocate excessive resources of the function approximator to represent that region, possibly “forgetting” the previous stored information.
One way to alleviate the interference problem is to use a local function approximator. The more independent each basis function is from each other, the less severe this problem is (in the limit, one has one basis function for each state, which corresponds to the lookup-table case) [86]. A class of local functions that have been widely used for approximation is the radial basis functions (RBFs) [52].
So, in your kind of problem (n*n gridworld), an RBF neural network should produce better results.
References
Boyan, J. A. & Moore, A. W. (1995) Generalization in reinforcement learning: Safely approximating the value function. NIPS-7. San Mateo, CA: Morgan Kaufmann.
André da Motta Salles Barreto & Charles W. Anderson (2008) Restricted gradient-descent algorithm for value-function approximation in reinforcement learning, Artificial Intelligence 172 (2008) 454–482
Having neural network with alot of inputs causes my network problems like
Neural network gets stuck and feed forward calculation always gives output as
1.0 because of the output sum being too big and while doing backpropagation, sum of gradients will be too high what causes the
learning speed to be too dramatic.
Neural network is using tanh as an active function in all layers.
Giving alot of thought, I came up with following solutions:
Initalizing smaller random weight values ( WeightRandom / PreviousLayerNeuronCount )
or
After calculation the sum of either outputs or gradients, dividing the sum with the number of 'neurons in previus layer for output sum' and number of 'neurons in next layer for gradient sum' and then passing sum into activation/derivative function.
I don't feel comfortable with solutions I came up with.
Solution 1. does not solve problem entirely. Possibility of gradient or output sum getting to high is still there. Solution 2. seems to solve the problem but I fear that it completely changes network behavior in a way that it might not solve some problems anymore.
What would you suggest me in this situation, keeping in mind that reducing neuron count in layers is not an option?
Thanks in advance!
General things that affect the output backpropagation include weights and biases of early elections, the number of hidden units, the amount of exercise patterns, and long iterations. As an alternative way, the selection of initial weights and biases there are several algorithms that can be used, one of which is an algorithm Nguyen widrow. You can use it to initialize the weights and biases early, I've tried it and gives good results.
I have a neural network with 2 entry variables, 1 hidden layer with 2 neurons and the output layer with one output neuron. When I start with some randomly (from 0 to 1) generated weights, the network learns the XOR function very fast and good, but in other cases, the network NEVER learns the XOR function! Do you know why this happens and how can I overcome this problem? Could some chaotic behaviour be involved? Thanks!
This is quite normal situation, because error function for multilayer NN is not convex, and optimization converges to local minimum.
You can just keep initial weights that resulted in successful optimization, or run optimizer multiple times starting from different weights, and keep the best solution. Optimization algorithm and learning rate also plays certain role, for example backpropagation with momentum and/or stochastic gradient descent sometimes work better. Also, if you add more neurons, beyond the minimum needed to learn XOR, this also helps.
There exist methodologies designed to find global minimum, such as simulated annealing, but, in practice they are not commonly used for NN optimization, except for some specific cases
I have already extracted the features of a fingerprint database then a Neural Network should be applied to classify the images by gender. I haven't worked with NN yet and I know a bit.
What type of NN should be used? Is it Artificial Neural Network or Multi-layer perceptron?
If the image size is not the same among all, does it matter?
Maybe some code sample in this area could help.
A neural network is a function approximator. You can think of it as a high-tech cousin to piecewise linear fitting. If you want to fit the most complex phenomena ever with a single parameter - you are going to get the mean and should not be surprised if it isn't infinitely useful. To get a useful fit, you must couple the nature of the phenomena being modeled with the NN. If you are modeling a planar surface, then you are going to need more than one coefficient (typically 3 or 4 depending on your formulation).
One of the questions behind this question is "what is the basis of fingerprints". By basis I mean the heavily baggaged word from Linear Algebra and calculus that talks about vector spaces, span, and eigens. Once you know what the "basis" is then you can build a neural network to approximate the basis, and this neural network will give reasonable results.
So while I was looking for a paper on the basis, I found this:
http://phys.org/news/2012-02-experts-human-error-fingerprint-analysis.html
http://phys.org/news/2013-07-fingerprint-grading.html
http://phys.org/news/2013-04-forensic-scientists-recover-fingerprints-foods.html
http://phys.org/news/2012-11-method-artificial-fingerprints.html
http://phys.org/news/2011-08-chemist-contributes-method-recovering-fingerprints.html
And here you go, a good document of the basis of fingerprints:
http://math.arizona.edu/~anewell/publications/Fingerprint_Formation.pdf
Taking a very crude stab, you might try growing some variation on an narxnet (nonlinear autogregressive network with external inputs) link. I would grow it until it characterizes your set using some sort of doubling the capacity. I would look at convergence rates as a function of "size" so that the smaller networks inform how long convergence takes for the larger ones. That means it might take a very large network to make this work, but large networks are like the 787 - they cost a lot, take forever to build, and sometimes do not fly well.
If I were being clever, I would pay attention to the article by Kucken and formulate the inputs as some sort of a inverse modeling of a stress field.
Best of luck.
You can try a SOM/LVQ network for classification in MATLAB, and image sizes does matter you should try to normalize the images down to a standard size before doing the feature extraction. This will ensure that each feature vector gets assigned to an input neuron.
function scan(img)
files = dir('*.jpg');
hist = [];
for n = 1 : length(files)
filename = files(n).name;
file = imread(filename);
hist = [hist, imhist(rgb2gray(imresize(file,[ 50 50])))]; %#ok
end
som = selforgmap([10 10]);
som = train(som, hist);
t = som(hist); %extract class data
net = lvqnet(10);
net = train(net, hist, t);
like(img, hist, files, net)
end
Doesn't have code examples but this paper may be helpful: An Effective Fingerprint Verification Technique, Gogoi & Bhattacharyya
This paper presents an effective method for fingerprint verification based on a data mining technique called minutiae clustering and a graph-theoretic approach to analyze the process of fingerprint comparison to give a feature space representation of minutiae and to produce a lower bound on the number of detectably distinct fingerprints. The method also proving the invariance of each individual fingerprint by using both the topological behavior of the minutiae graph and also using a distance measure called Hausdorff distance.The method provides a graph based index generation mechanism of fingerprint biometric data. The self-organizing map neural network is also used for classifying the fingerprints.