I have build a neural network model, with 3 classes. I understand that the best output for a classification process is the boolean 1 for a class and boolean zeros for the other classes , for example the best classification result for a certain class, where the output of a classifire that lead on how much this data are belong to this class is the first element in a vector is [1 , 0 , 0]. But the output of the testing data will not be like that,instead it will be a rational numbers like [2.4 ,-1 , .6] ,So how to interpret this result? How to decide to which class the testing data belong?
I have tried to take the absolute value and turn the maximum element to 1 and the other to zeros, so is this correct?
Learner.
It appears your neural network is bad designed.
Regardless your structure is -number of input-hidden-output- layers, when you are doing a multiple classification problem, you must ensure each of your output neurones are evaluating an individual class, that is, each them has a bounded output, in this case, between 0 and 1. Use almost any of the defined function on the output layer for performing this.
Nevertheles, for the Neural Network to work properly, you must strongly remember, that every single neuron loop -from input to output- operates as a classificator, this is, they define a region on your input space which is going to be classified.
Under this framework, every single neuron has a direct interpretable sense on the non-linear expansion the NN is defining, particularly when there are few hidden layers. This is ensured by the general expression of Neural Networks:
Y_out=F_n(Y_n-1*w_n-t_n)
...
Y_1=F_0(Y_in-1*w_0-t_0)
For example, with radial basis neurons -i.e. F_n=sqrt(sum(Yni-Rni)^2) and w_n=1 (identity):
Yn+1=sqrt(sum(Yni-Rni)^2)
a dn-dim spherical -being dn the dimension of the n-1 layer- clusters classification is induced from the first layer. Similarly, elliptical clusters are induced. When two radial basis neuron layers are added under that structure of spherical/elliptical clusters, unions and intersections of spherical/elliptical clusters are induced, three layers are unions and intersections of the previous, and so on.
When using linear neurons -i.e. F_n=(.) (identity), linear classificators are induced, that is, the input space is divided by dn-dim hyperplanes, and when adding two layers, union and intersections of hyperplanes are induced, three layers are unions and intersections of the previous, and so on.
Hence, you can realize the number of neurons per layer is the number of classificators per each class. So if the geometry of the space is -lets put this really graphically- two clusters for the class A, one cluster for the class B and three clusters for the class C, you will need at least six neurons per layer. Thus, assuming you could expect anything, you can consider as a very rough approximate, about n neurons per class per layer, that is, n neurons to n^2 minumum neurons per class per layer. This number can be increased or decreased according the topology of the classification.
Finally, the best advice here is for n outputs (classes), r inputs:
Have r good classificator neurons on the first layers, radial or linear, for segmenting the space according your expectations,
Have n to n^2 neurons per layer, or as per the dificulty of your problem,
Have 2-3 layers, only increase this number after getting clear results,
Have n thresholding networks on the last layer, only one layer, as a continuous function from 0 to 1 (make the crisp on the code)
Cheers...
Related
I am building a bidirectional LSTM to do multi-class sentence classification.
I have in total 13 classes to choose from and I am multiplying the output of my LSTM network to a matrix whose dimensionality is [2*num_hidden_unit,num_classes] and then apply softmax to get the probability of the sentence to fall into 1 of the 13 classes.
So if we consider output[-1] as the network output:
W_output = tf.Variable(tf.truncated_normal([2*num_hidden_unit,num_classes]))
result = tf.matmul(output[-1],W_output) + bias
and I get my [1, 13] matrix (assuming I am not working with batches for the moment).
Now, I also have information that a given sentence does not fall into a given class for sure and I want to restrict the number of classes considered for a given sentence. So let's say for instance that for a given sentence, I know it can fall only in 6 classes so the output should really be a matrix of dimensionality [1,6].
One option I was thinking of is to put a mask over the result matrix where I multiply the rows corresponding to the classes that I want to keep by 1 and the ones I want to discard by 0, by in this way I will just lose some of the information instead of redirecting it.
Anyone has a clue on what to do in this case?
I think your best bet is, as you seem to have described, using a weighted cross entropy loss function where the weights for your "impossible class" are 0 and 1 for the other possible classes. Tensorflow has a weighted cross entropy loss function.
Another interesting but probably less effective method is to feed whatever information you now have about what classes your sentence can/cannot fall into the network at some point (probably towards the end).
Neural Networks are mostly used to classify. So, the activation of a neuron in the output layer indicates the class of whatever you are classifying.
Is it possible (and correct) to design a NN to get 3D coordinates? This is, three output neurons with values in ranges, for example [-1000.0, 1000.0], each one.
Yes. You can use a neural network to perform linear regression, and more complicated types of regression, where the output layer has multiple nodes that can be interpreted as a 3-D coordinate (or a much higher-dimensional tuple).
To achieve this in TensorFlow, you would create a final layer with three output neurons, each corresponding to a different dimension of your target coordinates, then minimize the root mean squared error between the current output and the known value for each example.
I have come up with a solution for a classification problem using neural networks. I have got the weight vectors for the same too. The data is 5 dimensional and there are 5 neurons in the hidden layer.
Suppose neuron 1 has input weights w11, w12, ...w15
I have to explain the physical interpretation of these weights...like a combination of these weights, what does it represent in the problem.Does any such interpretation exist or is that the neuron has no specific interpretation as such?
A single neuron will not give you any interpretation, but looking at a combination of couple of neuron can tell you which pattern in your data is captured by that set of neurons (assuming your data is complicated enough to have multiple patterns and yet not too complicated that there is too many connections in the network).
The weights corresponding to neuron 1, in your case w11...w15, are the weights that map the 5 input features to that neuron. The weights quantify the extent to which each feature will effect its respective neuron (which is representing some higher dimensional feature, in turn). Each neuron is a matrix representation of these weights, usually after having an activation function applied.
The mathematical formula that determines the value of the neuron matrix is matrix multiplication of the feature matrix and the weight matrix, and using the loss function, which is most basically the sum of the square of the difference between the output from the matrix multiplication and the actual label.Stochastic Gradient Descent is then used to adjust the weight matrix's values to minimize the loss function.
I have big data set (time-series, about 50 parameters/values). I want to use Kohonen network to group similar data rows. I've read some about Kohonen neural networks, i understand idea of Kohonen network, but:
I don't know how to implement Kohonen with so many dimensions. I found example on CodeProject, but only with 2 or 3 dimensional input vector. When i have 50 parameters - shall i create 50 weights in my neurons?
I don't know how to update weights of winning neuron (how to calculate new weights?).
My english is not perfect and I don't understand everything I read about Kohonen network, especially descriptions of variables in formulas, thats why im asking.
One should distinguish the dimensionality of the map, which is usually low (e.g. 2 in the common case of a rectangular grid) and the dimensionality of the reference vectors which can be arbitrarily high without problems.
Look at http://www.psychology.mcmaster.ca/4i03/demos/competitive-demo.html for a nice example with 49-dimensional input vectors (7x7 pixel images). The Kohonen map in this case has the form of a one-dimensional ring of 8 units.
See also http://www.demogng.de for a java simulator for various Kohonen-like networks including ring-shaped ones like the one at McMasters. The reference vectors, however, are all 2-dimensional, but only for easier display. They could have arbitrary high dimensions without any change in the algorithms.
Yes, you would need 50 neurons. However, these types of networks are usually low dimensional as described in this self-organizing map article. I have never seen them use more than a few inputs.
You have to use an update formula. From the same article: Wv(s + 1) = Wv(s) + Θ(u, v, s) α(s)(D(t) - Wv(s))
yes, you'll need 50 inputs for each neuron
you basically do a linear interpolation between the neurons and the target (input) neuron, and use W(s + 1) = W(s) + Θ() * α(s) * (Input(t) - W(s)) with Θ being your neighbourhood function.
and you should update all your neurons, not only the winner
which function you use as a neighbourhood function depends on your actual problem.
a common property of such a function is that it has a value 1 when i=k and falls off with the distance euclidian distance. additionally it shrinks with time (in order to localize clusters).
simple neighbourhood functions include linear interpolation (up to a "maximum distance") or a gaussian function
I have feed-forward neural network with six inputs, 1 hidden layer and two output nodes (1; 0). This NN is learned by 0;1 values.
When applying model, there are created variables confidence(0) and confidence(1), where sum of this two numbers for each row is 1.
My question is: what do these two numbers (confidence(0) and confidence(1)) exactly mean? Are these two numbers probabilities?
Thanks for answers
In general
The confidence values (or scores, as they are called in other programs) represent a measure how, well, confident the model is that the presented example belongs to a certain class. They are highly dependent on the general strategy and the properties of the algorithm.
Examples
The easiest example to illustrate is the majority classifier, who just assigns the same score for all observations based on the proportions in the original testset
Another is example the k-nearest-neighbor-classifier, where the score for a class i is calculated by averaging the distance to those examples which both belong to the k-nearest-neighbors and have class i. Then the score is sum-normalized across all classes.
In the specific example of NN, I do not know how they are calculated without checking the code. I guess it is just the value of output node, sum-normalized across both classes.
Do the confidences represent probabilities ?
In general no. To illustrate what probabilities in this context mean: If an example has probability 0.3 for class "1", then 30% of all examples with similar feature/variable values should belong to class "1" and 70% should not.
As far as I know, his task is called "calibration". For this purpose some general methods exist (e.g. binning the scores and mapping them to the class-fraction of the corresponding bin) and some classifier-dependent (like e.g. Platt Scaling which has been invented for SVMs). A good point to start is:
Bianca Zadrozny, Charles Elkan: Transforming Classifier Scores into Accurate Multiclass Probability Estimates
The confidence measures correspond to the proportion of outputs 0 and 1 that are activated in the initial training dataset.
E.g. if 30% of your training set has outputs (1;0) and the remaining 70% has outputs (0; 1), then confidence(0) = 30% and confidence(1) = 70%