I am using a GloVe based pre-trained embedded vectors for words in my I/P sentences to a NMT-like model. The model then generates a series of word embeddings as its output for each sentence.
How can I convert these output word embeddings to respective words? One way I tried is using cosine similarity between each output embedding vector and all the i/p embedding vectors. Is there a better way than this?
Also, is there a better way to approach this than using embedding vectors?
First of all the question is lacking a lot of details like the library used for word embedding, the nature of the model, and the training data, etc ...
But I will try to give you an idea what you can do in those situations, assuming you are using some word embedding library like Gensim.
How to get the word from the vector:
We are dealing with predicted word vectors here, so our word vector may not be the exact vector of the original word, we have to use similarity, in Gensim you can use similar_by_vector, something like
target_word_candidates = similar_by_vector(target_word_vector,top=3)
That's would solve the reverse lookup problem, as highlighted here, given all the word vectors how to get the most similar word, but we need to find the best single word according to the context.
You can use some sort of post-processing on the output target word vectors, this would be beneficial for trying to solve some problems like:
1.How to guide the translation of out-of-vocabulary
terms?
2.How to enforce the presence of a
given translation recommendation in the decoder’s
output?
3.How to place these word(s) in the right
position?
One of the ideas is to use an external resource for the target language, i.e. language model, to predict which combination of words are gonna be used. Some other techniques incorporate the external knowledge inside the translation network itself
I recently started learning neural networks, and I thought that creating a sudoku solver would be a nice application for NN. I started learning them with backward propagation neural network, but later I figured that there are tens of neural networks. At this point, I find it hard to learn all of them and then pick an appropriate one for my purpose. Hence, I am asking what would be a good choice for creating this solver. Can back propagation NN work here? If not, can you explain why and tell me which one can work.
Thanks!
Neural networks don't really seem to be the best way to solve sudoku, as others have already pointed out. I think a better (but also not really good/efficient) way would be to use an genetic algorithm. Genetic algorithms don't directly relate to NNs but its very useful to know how they work.
Better (with better i mean more likely to be sussessful and probably better for you to learn something new) ideas would include:
If you use a library:
Play around with the networks, try to train them to different datasets, maybe random numbers and see what you get and how you have to tune the parameters to get better results.
Try to write an image generator. I wrote a few of them and they are stil my favourite projects, with one of them i used backprop to teach a NN what x/y coordinate of the image has which color, and the other aproach combines random generated images with ine another (GAN/NEAT).
Try to use create a movie (series of images) of the network learning to create a picture. It will show you very well how backprop works and what parameter tuning does to the results and how it changes how the network gets to the result.
If you are not using a library:
Try to solve easy problems, one after the other. Use backprop or a genetic algorithm for training (whatever you have implemented).
Try to improove your implementation and change some things that nobody else cares about and see how it changes the results.
List of 'tasks' for your Network:
XOR (basically the hello world of NN)
Pole balancing problem
Simple games like pong
More complex games like flappy bird, agar.io etc.
Choose more problems that you find interesting, maybe you are into image recognition, maybe text, audio, who knows. Think of something you can/would like to be able to do and find a way to make you computer do it for you.
It's not advisable to only use your own NN implemetation, since it will probably not work properly the first few times and you'll get frustratet. Experiment with librarys and your own implementation.
Good way to find almost endless resources:
Use google search and add 'filetype:pdf' in the end in order to only show pdf files. Search for neural network, genetic algorithm, evolutional neural network.
Neither neural nets not GAs are close to ideal solutions for Sudoku. I would advise to look into Constraint Programming (eg. the Choco or Gecode solver). See https://gist.github.com/marioosh/9188179 for example. Should solve any 9x9 sudoku in a matter of milliseconds (the daily Sudokus of "Le monde" journal are created using this type of technology BTW).
There is also a famous "Dancing links" algorithm for this problem by Knuth that works very well https://en.wikipedia.org/wiki/Dancing_Links
Just like was mentioned in the comments, you probably want to take a look at convolutional networks. You basically input the sudoku bord as an two dimensional 'image'. I think using a receptive field of 3x3 would be quite interesting, and I don't really think you need more than one filter.
The harder thing is normalization: the numbers 1-9 don't have an underlying relation in sudoku, you could easily replace them by A-I for example. So they are categories, not numbers. However, one-hot encoding every output would mean a lot of inputs, so i'd stick to numerical normalization (1=0.1, 2 = 0.2, etc.)
The output of your network should be a softmax with of some kind: if you don't use softmax, and instead outupt just an x and y coordinate, then you can't assure that the outputedd square has not been filled in yet.
A numerical value should be passed along with the output, to show what number the network wants to fill in.
As PLEXATIC mentionned, neural-nets aren't really well suited for these kind of task. Genetic algorithm sounds good indeed.
However, if you still want to stick with neural-nets you could have a look at https://github.com/Kyubyong/sudoku. As answered Thomas W, 3x3 looks nice.
If you don't want to deal with CNN, you could find some answers here as well. https://www.kaggle.com/dithyrambe/neural-nets-as-sudoku-solvers
I am doing my project based on health care.I am going to train my autoencoders with the symptoms and the diseases i.e my input is in textual form. Will that work? (I am using Rstudio).Please anyone help me with this
You have to convert the text to vectors/numbers. To do this traditional approaches like Bag of words, Tf-Idf will help but the latest Neural Word Embedding like Word2Vec, RNN Language model etc are the best techniques to obtain numeric representation of text.
Please use any Neural Word Embedding technique and convert the text(word level[word2vec], document level[doc2vec]) into numbers/vectors.
Now these vectors come with some dimension and to compress this representation to even smaller dimension u can use AutoEncoder.
Feel Free to ask any other information required.
Try using Python for these tasks as it has the latest packages.
You can use Autoencoder on Textual data as explained here.
Autoencoder usually worked better on image data but recent approaches changed the autoencoder in a way it is also good on the text data.
have a look at this.
the code is also available in GitHub.
I've recently started using word2vec and doc2vec methods. They are amazing!
But I want to play around with them a bit. As I compared the two methods I saw that the difference is that in doc2vec method, there is one extra input to the neural net, docMatrix. I want to add one more input to the neuralNet (which is a trained vector from somewhere else) and get the output vector for the document. Is it easy to do? Can someone help me to understand what exactly going on in word2vec code?
Thanks :)
I am a newbie. I think idea of NuPIC is really cool and therefore wanted to apply KNN Classifier on
NuPIC's output. I saw there is a KNNClassifier object already in python. I am confused about the input
patter that I should use. In case of MNIST dataset I will be having images where each image is a 2D
array of numbers and will be sparse. I can understand the format of output can be encoded using
categorical encoder in NuPIC but there is no such example of encoding an input that comes in the
form of arrays.
Any help will be highly appreciated.
This might help: http://numenta.org/search.html?q=mnist. There are some good discussions on our mailing lists about MNIST.