Autocomplete with Natural language - autocomplete

My project needs some natural language processing. I'm completely new to the field.
what I'm trying to achieve is that when the User enter the description of the product I look for in my database which description is nearest and suggest that the category, product group and sub-group (the tree of the product).
For this titles 250 extracts products for each subgroup.
What is the specific term in NLP for doing this? I tried googling for a while, but had no luck since I don't know the term. Any good tutorials to start with? Are there any good libraries in doing this specific task?
Thank you.

From what I can tell autocomplete or text prediction/predictive search isn't really a big research area in NLP. It wasn't even covered in any of my graduate level classes and I do research in this area. I think the reason is that there are solutions that exist which are good enough for the vast majority of real world problems.
I'm not sure which language you work in, but the library you want to work with is probably Lucene if you are dealing with java, perhaps setting up a Solr instance if this is a general problem for you and you are dealing with a large number of ontologies.
You can find some reason tutorials/examples here on stack overflow, such as:
How to implements auto suggest using Lucene's new AnalyzingInfixSuggester API?

Related

Lighting diagrams language and notation

I know, the use case might be specific but more and more stuff in all industry sectors is digitalized—and so is the communication between different departments which sometimes talk in very different languages. I searched the internet, but I wasn't able to find a clear answer (either I didn't find the right search phrases or the internet itself just doesn't know).
Here's my scenario: I'm working with several departments which work with diagrams (for example a lighting setup). This diagram solves different purposes:
which devices are used?
where are they placed?
where are they pointing?
how are they configured (e.g. exposure)?
They tend to export their finalized diagram as either an image or a PDF— which is fine if you want to print it out but considerably less helpful if another department (mine) has to work with the raw information. That's where I wondered if there's some kind of industry standard (SVG, XML, JSON, etc.) which is both supported by the programs these departments used and can be interpreted by some sort of programming language. Do you know anything like that?
Thanks in advance!

How to model multilingual database with Zend, l18n mysql?

I know this topic was discusses a couple of times, but none of them represents the ultimate solution for me.
Situation
I'm designing a relational mysql database which later should hold multilingual content. You know this from the Wikipedia or Microsoft Tech Support Pages. The contents should be the same for every language. e.g If translations are missing the site offers you the same content automatically translated or in the languages which the information is available in. If some values are not set, it should fallback to the second or default browser language or translate it e.g. through google. Development environment is Zend.
My ideas so far are for Solving the Problem:
Two Primary Keys: (ID, Language)
Advantage: Easy Database Access through database abstraction layers.
Problem: Foreign Keys, Relations ships, Fallbacks
Columns with language suffix:
Advantage: DB Performance, No relational Problems.
Problem: Database abstraction layers cannot handle this?
Has any concept proven itself or is preferable over the other? Has anyone already created something like this and can share his experience with me? Does a modified Zend DB Controller exist for this situation? How do you link this information to a form?
Thank you for your help, hints and suggestions!
Kind regards,
Manuel
The second option would be not maintainable (this should be added on the minuses side). To actually add another language you'll need to modify table and abstraction layers. Sounds like a nightmare.
The first option seems much more promising but unfortunately there is a lot to do to make it work. However, from my experience this is rather typical solution, so I would not reinvent the wheel.
What I have to add is, language fallback should be done on the Zend side, database would miss some information. You may think of some kind of index table to hold information such as unique id of the contents and available languages. If you need to serve something, you would read such record, compare it against of Accept Languages and ask database again for valid contents (using the most suitable language). The only problem is, you would need to create such an index table somehow (the best way I see would be trigger on inserting contents to your content table).
A lot of work but the problem is not too easy.
I am working on the exact same problem right now.
Somehow it does not make sense to me to add everything into the same database. Lets say I want to go to the extreme and support some 50 languages this would just bloat my DB. So, I tend to keep my main DB in my main language and then introduce some Zend_Translate concept into it. Zend_Translate should give you the fallback solution you are looking for. While the main navigation and core design is not much of a problem for my web site my biggest concern right now is how to store all the main content and how to translate because these elements contain HTML among other things. For the main content I will probably use some alternate approach and use a separate DB with tables for each language.
My plattform will be a community driven database. So I actually gonna rely on humans translating it. You have to store the information anyways, so my first concern is not the database size or performance, but easy usability. So far my idea is to implement some structure as described above, not yet sure if i'll do it in doctrine or not.
Language decision:
Start, application gets users preset language, secondary language, english mother-tong of the article. Fetching the article from the database I will check the following for every column: 1. is the primary language available? 2. Is the secondary language available? 3. If neither of them, display article in mother-tong or english and offer the user to translate it with suggestions from the google translate api. I guess it's gonna be quite a bit of coating and manipulating controllers or building a business model doing this.
#tawfekov is something like this or similar easily realizable with doctrine?

Requirements for a game

I'm writing an iPhone game and I am trying to write some requirements documents. I have never written requirements before so I got the book Software Requirements. I have not finished it yet, but I forsee some issues, as this book is targeted towards a business. My main question is I am the only person involved with this game and I feel the main purpose of the requirements document should be to nail out as many conceptual ideas of how the game works as I can before I am deep into design or construction. Does anyone have suggestions on how I should lay this out, should I still try to mimic the template provided in the book where it makes sense, or since I am both the sole developer and product owner, should I just stick to game concepts?
You're right that traditional SRS documents don't really fit games documentation all that well. Games instead have a general Game Design Document. It's usually created before any work on the game begins, and it's often edited as the development process goes to keep straight the intended end-result and specifics of the game.
While business software requirements documents are like contracts between a client and developer on what to produce, game design docs are more often specifications from the designer to the artists and programmers on what exactly they need to develop.
There is no specific layout to use. But you should consider who you're writing the document for. Is it for a class, for yourself, for peers after the project is done? The level of detail and the kind of things you include will be different depending on your audience. The format itself is very flexible, as long as it's coherent.
Brenda Brathwaite has a good blog entry on this subject which you might find helpful.
There is a semi-recent article from gamedev.net on the subject as well.
[Poor Jacob, you just read a book on the topic, and, collectively, the SO community writes another one for you, along with extra links, and probably with diverging views ;-) ]
Although I'm not familiar with the book you mention in the question, I think that the following suggestion may help you both take seriously, but also relax a bit, about the all too important question of requirements.
Being a "team of one", it is particularly important, and somewhat paradoxical, that you go through the effort of formalizing the requirements. However, rather than putting too much emphasis on the form, you may find an Agile approach to developement (and hence to requirements gathering) more appropriate. With regards to requirements, one of the main advantages of this approach, is flexibility, i.e. the understanding that while they should be formalized (with limited time/effort), requirements should be allowed to change (within limits) as part of an iterative process towards production of the target product.
In very broad terms, this generally go as follows:
write "user stories", these are individual "cards" (yes, physical cards, say 4 inches by 5 inches, are good, for you can then move then around, sort them etc.)
each story tells a particular feature of the application, here the game, from the end-user's perspective. You can/should start all cards with "As a user, I need the game to..." then follow with a particular feature, for example "... show my high score on the same page as the global high-scores are kept [because ... here optional reasons for why user may want this feature].
review each story and assign a rough estimation of the time involved in implementing it
review each story and assign a priority level (scale may vary, but something simple like "Must have from Version 1.0", "Should eventually be in there, for sure", "Would be nice to have" and "Maybe nice to have...")
organize releases, on the basis of what you can do within say 2 or 3 weeks, maximum. If a particular feature were to take too long, schedule it for a later release.
implement the features assigned to the current release
iterate through this release cycle, reviewing the requirements as you go, for the relative importance of features, and also the need of new features may become evident as with the insight provided by using the [incomplete/imperfect] intermediate releases.
Books like the one you describe are focused at a different audience, but there is value in the general concepts presented. Fully developed requirements documents are not as common as you might think. Don't let anyone think that you are a 'bad developer' for not having the most detailed requirements.
Requirements docs might be more important if you need to communicate the requirements with a co-developer.
If you are the sole developer I would strongly recommend that you spend your efforts on the design and implementation of the game, over requirements. If you have a good idea of what you build then let this flow as you build it.
Documentation can help you. The question is what is going to be most beneficial. Maybe design decisions are more critical than requirements for you but not for others. You'll maybe want to have a list of things that people have requested or ideas that you think of but cannot implement straight away. Sometimes a whiteboard can be handy for sketching out things, it's not just a tool for collaboration with other people.
Here's just a general approach...
Solidify the concept...write it in plain English first (ex: The game is a first person shooter. You kill zombies and hunt for treasure.)
Get a paper pad and pencil and draw out the general flow of the game and the main screens the users will encounter...main menu, options screen, help, etc. Make sure it makes sense.
Go to a site like mockingbird and create the detail wireframes for your screens...
Print these out and do some paper prototyping...i.e. put the printout in front of you and 'click' on a button...then bring up the appropriate screen...then click on another button, etc.
Once that makes sense, you can try to start coding your game.
Personally I believe you should use your own way to do this. The most commonly available one's will not match with your requirement. They might be suitable for a common commercial server application but not for a game. And since iPhone gaming is a new trend you may have to look in a different perspective.. You may not be able to fill a document with standard requirements and you may have different set of New type of requirements.
Just a suggestion... Sign up with Google Sites, and create a private site with documentation of the game, requirements, technical aspects, work log, etc... You can share it with select people, and it always keeps edit history.
I like it better than a Wiki because it is more structured, and just plain simple to use.

For a large project, what planning should be done before coding and how should it be approached? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
What is your method of "mapping out" an idea before creating it?
Say I wanted to take on a big project, for example at the scale of a site like Facebook or MySpace. What planning/design steps should I take before I start the actual work?
For example, should I map everything out page by page (their functionalities, data, etc.)?
For a large project first think of a one-liner to description of your site (try to not use any buzzwords here). Next think of three design maxims (rules your design Should never conflict with). Then draw a few views and think up a few user cases (1 day) then work in code for 2 weeks (this will be a throw away prototype so just work as fast as you can forget about bugs and details, don't worry about code smells or design patterns, just make as much as you can), then revaluate all the steps above and throw away your two week prototype, and begin your project in a serious manner applying solid engineering and design. After a month has gone by evaluate your(team) moral and get feedback. If it all seems to be going ok, continue, you got a long ride ahead, otherwise just give up, do a postmortem, and start over with new goals.
I always start with the user interface design. I figure out what the user should be able to do and what controls I will give them to do it. Once I get that laid out in a way I like it, then I start with the code "wiring".
Make a list of all the features that site have.
Make a list of nice to have features.
Make a list of the weakness of the site.
Order that list and prioritize the items that will be built first.
Identify what will be possible to do and what is not.
Meet with your customer and present these results.
Usually I do a mindmap of
problem I am trying to solve,
translated into exact requirements,
then mapping that to user workflows.
The cross linking features of mindmapping softwares make it lot easy. Since mindmapping is 'kind of freeform', I end up concentrating on the 'task' rather than 'representation' (e.g which type of UML diagram should I use to represent this) ?
Once initial ideas are clear then I can work on project plan, spec/design documents using UML for more low level details. This approach usually works well for me.
To see if it works for you or not, you can use FreeMind (opensource mindmaping software, good but currently limited functionality). Then You can try Mindmanager or iMindmap for mindmaping. Both integrate well with other Office products.
Usually I start out by grabbing my scratchbook and just start writing down what I want as in terms of features, this should be quite detailed. And can be quite messy with every thing scrambled together, if so, when you're done make an 'official version' of you're ideas on paper (REAL pen and paper works best for this in my opinion).
Then I start making some scetches of how the pages would look like, what information it must contain and translate that to a global database design. Then work that global design to a more advanced level where all pages come together, with relations between tables and stuff.
After that I build up the most important pages on a code framework (I always make use of a framework, if you don't then forget the framework part), and by 'most important pages' I mean in for example a blog that would be the posts. After that build the not-so-important pages, in case of a blog that could be an archive of posts.
If you have that done, put the code together with a design, or do that while coding if you do not seperate code from HTML/CSS/JS.
Oh and yes, do NOT expand your first idea along the way. Just write that down and implement that afterwards. So if, in case of the blog again, you think half way you want Youtube tags in you're BB-code, write it down. Add that later, offcourse before you're initial site releases.
That's my workflow, at least a basic basic, basic description of it.
Start with "paper prototypes", i. e. take a pencil and sketch each page very roughly. This lets you start from the user perspective, which I think is a good idea.
You can then use the sketches for a first hallway usability test and later as the basis for "wireframes" you would give a web designer to work from.
If you've gone through the complete site once, you probably have a good idea of what the backend should be able to do. You can now use your page sketches and compile a list of the actions a user can trigger by clicking on things. This is the raw material for designing the server-side API that the frontend can call.
Using the calls that need to be served, you can design the backend: What functionalities group nicely, what data needs to be fetched, what do you need to store between page calls (== Session variables) etc.
In this process, I have fared quite well by postponing technology decisions (frameworks, protocols etc.) and even class structure etc., until I've gone through the whole thing once in terms of "what things should do what to what other things" (I guess there's a better term).
I think I would start with an open-source SNS solution that comes close to what you need and then figure out how to add use-specific plug-ins, modules, and themes that achieve your purposes. There are a lot of em out there. Building from scratch is going to take a lot more effort and planning. Most SNS functionality is not worth re-inventing. Focus on what will make your site unique and build upward toward that.
I'm a fairly visual person when it comes to designing software so I sketch out dataflows, class hierarchies, UI and flow charts on whiteboards and paper first.
Butcher paper and colored pens can be particularly fun to use as it's 3 feet wide and comes in 100 foot rolls. When you've got a design that's satisfying or sufficiently complete, tear it off the roll and pin to the wall. Update as necessary.
That technique has worked for some large refactors as well as new projects.
You could start with something very simple and then add features a little at a time. You may reach a point where you want to start over, but the groundwork you did will be beneficial. Or you can try to do the whole thing at once, in which case you'll need the advice already given in the other replies.
One more idea: Specify those features you are not going to include, and other restrictions. These are called constraints, and are as important as the rest of the plan, as it gives you boundaries so you know when you're done planning!
If you work for the same company as this person, start by getting everything in writing so you aren't the one to take the fall when the inevitable happens...

Open Source collaborative filtering frameworks

I was wondering if there exists any open source frameworks that will help me include the following type of functionality to my website:
1) If I am viewing a particular product, I would like to see what other products may be interesting to me. This information may be deduced by calculating for example what other people in my region (or any other characteristic of my profile) bought in addition to the product that I am viewing. Kind of like what Amazon.com does.
2) Deduce relationships between people based on their profile, interaction with one another on the website (via commenting on one another´s posts for example), use of the website in terms of areas most navigated, products bought in common etc.
I am not looking for a open source website with this functionality, but something like an object model into which I can feed information about users and their use of the site including rules about relationships and then at a later point ask it questions described in (1) and (2) above.
Any pointers to white papers / general information about best approaches to do this, or any related links will really help too.
(I am the developer of Taste, which is now part of Apache Mahout)
1) You're really asking for two things here:
a) Recommend items I might like
b) Favor items that are similar to the thing I am currently looking at.
Indeed, Mahout Taste is all about answering a). Everything it does supports systems like this. Take a look at the documentation to get started, and ask any questions to mahout-user#apache.org.
For 1b) in particular, Mahout has two answers:
If you are only interested in what items are similar to the current item, you would be interested in the ItemSimilarity abstraction in Mahout (org.apache.mahout.cf.taste.similarity.ItemSimilarity) and its implementations, like PearsonCorrelationSimilarity. Based on a set of user-item ratings, this could tell you an estimated similarity between any two items. You'd then just pick the most similar items. In fact, look at the TopItems class in Mahout which can just figure this for you quickly.
But also, you can combine a) and b) by computing recommendations, then applying a Rescorer implementation which then favors items that are similar to the currently-viewed item.
2) Yes likewise, you would be interesting the UserSimilarity abstraction, implementations, etc. This would deduce similarities based on item ratings. Mahout however does not help you deduce these ratings by, say, looking at user behavior. This is domain-specific and up to you.
Sound confusing -- read the docs and feel free to follow up on mahout-user#apache.org where I can tell you more.
I am researching the same topic, as I'm working on a project to help people decide how to vote on California's complicated ballot measures. Here are some open-source collaborative filtering engines that I've found:
Vogoo (PHP)
acts_as_recommendable (Ruby on Rails)
Mahout (formerly Taste) (Java)
There's also a good overview of these engines here.
There are also the Duine framework and OpenSlopeOne.
But in my opinion, Mahout is still the best.
You can find a survey about Open Source Recommender Systems here:
http://girlincomputerscience.blogspot.com.br/2012/11/open-source-recommendation-systems.html
Hope it helps!
You can find a List of Recommender Systems here