Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I need help about the weight option of libSVM. I'm confused at some point; should we also use the -wi option while doing cross-validation? If so, should we use the calculated weights of the whole data or the calculated weights according to the v-1 subsets (for v-fold cross-validation)? And my second question is should we use -wi option during predict? If so, should we use the calculated weights during training or should we calculate the weights according to the distribution of negative and positive instances in the test data?
For example; we have 50 + data and 200 - data. So after calculating the best c and gamma parameter values we will use -w1 4 -w-1 1 options while training. But what about training during grid search and cross validation? Let's say we are performing 5-fold cross-validation. While training on each remaining 4 subsets, the distribution of negative and possitive instances will probably change. So should we recalculate the weights during this 5-fold cross validation?
Besides shoud we use -w1 4 -w-1 1 options while testing?
Thanks
To answer your first question, if you are applying non-trivial weights to a subset of classes during model training, then you should do the same throughout your training/tuning which includes cross-validation based tuning of C and gamma (otherwise you would be tuning the model based on the cost-sensitive objective/risk/loss function which is different from the one you are actualy specifying by enabling non-trivial class weights)
The class weights are external to libSVM in the sense that they are not calculated by libSVM - that command-line option allows a user to set his/her own class weights to emphasize/reduce importance of a subset of classes. Some people tune the class weights as well but that is a different story.
As for the prediction, the class weights are not used there explicitly (since they come in as a "tweak" to the objective/risk/loss function during the model training/tuning stage so the resulting model is already "aware" of the weights)
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I want to build a Time series prediction model using LSTM,
Which activation function should be used at intermediate layers?
Is Linear activation function is good for Final or Output Layer?
I am normalising my input data in range (0, 1) and inverse normalise after prediction.
Here is My Model:
model = Sequential()
model.add(LSTM(32, input_shape=(input_n, n_features),return_sequences=True,activation='relu'))
model.add(LSTM(32, input_shape=(n_features, input_n), return_sequences=True,activation='relu'))
model.add(Dense(output_n))
model.add(Activation("linear"))
model.compile(loss = 'mean_squared_error', optimizer = 'adam')
model.summary()
Here I have used 'relu' in intermediate layers and Linear activation at my output layer.
Is this approach correct, or in the intermediate layer I should also try with tanh and sigmoid.
What will happen if I will not use any activation function in the intermediate layer, will LSTM take care of this.
Actually LSTM already having tanh and sigmoid activation function for its internal gate calculation.
Word of warning: this is my subjective impression which is mostly (but not completely backed) by scientific research.
I can verify that ReLU and its derivates (PReLU, Leaky ReLU, etc.) have produced the best results for me in the past.
Which of those implementations will produce the best results for you is probably best determined by trying them out, if you can afford to do so.
ReLU is prettymuch better for deep learning models as an activation function.This normalizes your input and output in the range of [0,1] and adds non linearity
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
I collected data from different sources FB, Twitter, Linkedin, then made them in a structured format. As a result now: I'm having a csv file with 10000 rows (10000 person) and the data associated is about their names, age,their interests and buying habits.
I'm really stuck on this step: CLASSIFICATION or CLUSTERING. For the classification I don't really have predefined classes or a model for my users to classify them.
For clustering: I started calculating similarities and KMeans, but still can't get the result I wanted. How can I decide what to choose before moving on to the next step of Collaborative filtering?
Foremost, you have to understand that clustering is a pre-processing activity/task. The idea in clustering is to identify objects with similar properties and group them. The clustering process can be understood in terms of cattle-herding. Wherein the jockey herds loose cattle (read data points) into groups.
Note: If you are looking at the partitioning clustering algorithm family includes K-means, k-modes, k-prototype etc. The algorithm k-means will work only for numerical data. K-modes will work only for categorical data and k-prototype will work for both numerical and categorical data.
Question: Is the data preprocessed? If the answer is no, then you may try the following steps;
Is the data (column values) all categorical (=text) format or numerical or mixed?
a. If all categorical then discretize or bin or interval scale them.
b. if mixed, then discretize or bin or interval scale the categorical values only
c. Perform missing value and outlier treatment for both numerical and categorical data. This will help in retaining maximum variance as well as reduce dimensionality.
d. Normalize the numerical values to a median of zero.
Now apply a suitable clustering algorithm (based on your problem) to determine patterns. Once you have found the patterns, then you may label them. Once the identified patterns are labelled, thereafter or subsequently a classification algorithm can be used to classify any new incoming data points into an appropriate class.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I have the following theoretical questions regarding the conv layer in a CNN. Imagine a conv layer with 6 filters (conv1 layer and its 6 filters in the figure).
1) what guarantees the diversity of learned filters within a conv layer? (I mean, how the learning (optimization process) makes sure that it does not learned the same (similar) filters?
2) diversity of filters within a conv layer is a good thing or not? Is there any research on this?
3) during the learning (optimization process), is there any interaction between the filters of the same layer? if yes, how?
1.
Assuming you are training your net with SGD (or a similar backprop variant) the fact that the weights are initialized at random encourage them to be diverse, since the gradient w.r.t loss for each different random filter is usually different the gradient will "pull" the weights in different directions resulting with diverse filters.
However, there is nothing that guarantees diversity. In fact, sometimes filters become tied to each other (see GrOWL and references therein) or drop to zero.
2.
Of course you want your filters to be as diverse as possible to capture all sorts of different aspects of your data. Suppose your first layer will only have filters responding to vertical edges, how is your net going to cope with classes containing horizontal edges (or other types of textures)?
Moreover, if you have several filters that are the same, why computing the same responses twice? This is highly inefficient.
3.
Using "out-of-the-box" optimizers, the learned filters of each layer are independent of each other (linearity of gradient). However, one can use more sophisticated loss functions/regularization methods to make them dependent.
For instance, using group Lasso regularization, can force some of the filters to zero while keeping the others informative.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I am trying to understand word2vec(word embedding) architecture, and I have few questions about it:
first, why is word2vec model considered a log-linear model? Is it because it uses a soft max at output layer?
second, why does word2vec remove hidden layer? Is it just because of computational complexity?
third, why does word2vec not use activation function? (as compared to NNLM(Neural Network Language Model).
first, why word2vec model is log-linear model? because it uses a soft max at output layer?
Exactly, softmax is a log-linear classification model. The intent is to obtain values at the output that can be considered a posterior probability distribution
second, why word2vec removes hidden layer? it just because of
computational complexity?
third, why word2ved don't use activation function? compare for
NNLM(Neural Network Language Model).
I think your second and third question are linked in the sense that an extra hidden layer and an activation function would make the model more complex than necessary. Note that while no activation is explicitly formulated, we could consider it to be a linear classification function. It appears that the dependencies that the word2vec models try to model can be achieved with a linear relation between the input words.
Adding a non-linear activation function allows the neural network to map more complex functions, which could in turn lead to fit the input onto something more complex that doesn't retain the dependencies word2vec seeks.
Also note that linear outputs don't saturate which facilitates gradient-based learning.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Can anybody think of a real(ish) world example of a problem that can be solved by a single neuron neural network? I'm trying to think of a trivial example to help introduce the concepts.
Using a single neuron to classification is basically logistic regression, as Gordon pointed out.
Logistic regression is the appropriate regression analysis to conduct when the dependent variable is dichotomous (binary). Like all regression analyses, the logistic regression is a predictive analysis. Logistic regression is used to describe data and to explain the relationship between one dependent binary variable and one or more metric (interval or ratio scale) independent variables. (statisticssolutions)
This is a good case to apply logistic regression:
Suppose that we are interested in the factors that influence whether a political candidate wins an election. The outcome (response) variable is binary (0/1); win or lose. The predictor variables of interest are the amount of money spent on the campaign, the amount of time spent campaigning negatively and whether or not the candidate is an incumbent. (ats)
For a single neuron network, I find solving logic functions a good example. Assuming say a sigmoid neuron, you can demonstrate how the network solves AND and OR functions, which are linearly sepparable and how it fails to solve the XOR function which is not.