As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I want to learn more about AI and neural networks. I have some basic idea what it is and how it works, but I want to find a good book or tutorial with good explanations.
Anyone know of some good material ( that is not too technically written ) to read on the subject?
C# implementations are also helpful.
Good question... There is a impressive collection of books and online texts on the subject of neural networks (NN), but many of them are either very technical or focus on somewhat specific domains of application of the NN, artificial vision, pattern recognition and signal processing being the "natural" leaders in this genre.
I'd like to second Chaos' recommendation of
Fundamentals Of Neural Networks
Architectures, Algorithms And Applications
by Laurene Fausett
For an introduction to the field, this book strikes a good balance in terms of accessibility, detail and coverage. The alternative are older books circa 1995-2000 which tend to be a be more of the vulgarization type.
You can also wet your appetite with online resources such as:
A tutorial from learnartificialneuralnetworks.com (the site itself also has several good references)
A class at Williamette University
When all fails, a quick scan of NN FAQs
Finally, at the risk of confusing you, at a time when you need a broad view on the subject, I'd like to plug a specific type of neural network: Numenta's Hierarchical Temporal Memory concept.
Checkout http://ml-class.org an online distant education by Stanford, Prof. Andrew Ng is taking the classes, i think it is the best way to get into Neural Networks and machine learning. Also a similar class on Artificial Intelligence: http://ai-class.com taken by Prof. Peter Norvig, Sebastian Thrun. I cannot say if the videos and course materials will be online even when the course ends, but if it stays then it would be a very good resource.
Also have a look at this book: Neural Networks - A Systematic Introduction by Raul Rojas. This will give details about ANN.
EDIT
Also checkout the upcoming Coursera Neural Network class: https://www.coursera.org/course/neuralnets
I found Fausett's Fundamentals of Neural Networks very accessible.
I have been working with Neural Networks for about 10 years, and still have Neural Networks: A Comprehensive Foundation on my desk.
There are some good articles on code project, here is just a few:
http://www.codeproject.com/KB/recipes/NeuralNetwork_1.aspx
http://www.codeproject.com/KB/recipes/Backprop_ANN.aspx
http://www.codeproject.com/KB/recipes/GA_ANN_XOR.aspx
http://www.codeproject.com/KB/recipes/encogintroxorcs.aspx
You may like to try using FANN, it has bindings for C# (I use the C library) and comes with examples to help get you started.
Try Simon Haykin's book at:
http://www.amazon.com/Neural-Networks-Learning-Machines-Edition/dp/0131471392/ref=sr_1_1?ie=UTF8&qid=1331331157&sr=8-1
However, the best way do it the http://ml-class.org. It is a step-by-step tutorial on solving a few interesting ML problems.
Related
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Is LISP or something like Jess the best choice? I'm interested in writing a program that makes a suggestion based on users' answers. Computational considerations are not really a factor this is pretty much a pattern matching engine. Also I would like to make an app for this and put it up on the web.
UPDATE: I would like to put this up on a blog or website and let people use it from there. I guess my question then is there a particular inference engine that works with the .NET family, or PHP, or something to that effect? What are some of the pros and cons of each options etc.
Step 1. Pick an inference engine. There are many choices. Here's a list: http://en.wikipedia.org/wiki/Expert_system#Shells_or_Inference_Engine
Step 2. Use the language that interfaces with the inference engine.
You'll be much happier leveraging an inference engine for expert systems work.
I would like to put this up on a blog or website and let people use it from there
Trivial.
is there a particular inference engine that works with the .NET family, or PHP, or something to that effect?
Doesn't matter.
Here's the confusion. Your "web site" and your "inference application" have NOTHING to do with each other. Nothing.
Your web site can be done in any tool set you can find. It doesn't matter.
Your inference application can be done in any tool set you can find. It doesn't matter.
Your web site will invoke the inference application through any API that makes sense. The lowest common denominator in API's (the reason that none of these choices matter) is to do this.
Write your inference application as a stand-alone command line tool.
Write your web application to run the stand-alone tool, collect the output and turn the output into an HTML page.
Note that this multi-porocess implementation may be faster and make better use of multi-core processors. It forces the OS to manage the web server (Apache HTTPD, for example), the web application and the expert system as potentially three, separate, parallel processes.
You can also take a look at Prolog. SWI-Prolog (http://www.swi-prolog.org) is very complete and has an HTTP support library included (http://www.swi-prolog.org/pldoc/package/http.html). This paper might be helpful in using SWI-Prolog on the web ("SWI-Prolog and the web" http://dare.uva.nl/record/285350)
And, you can find a tutorial on building expert systems with prolog at: http://www.amzi.com/ExpertSystemsInProlog/
You will hear a lot of subjective opinions here, since few people have experience in more than one language writing expert systems.
I can recommend Common Lisp, as there is quite some literature and existing code available in this language, and it is a very powerful language and not too difficult to learn (read "Practical Common Lisp" by Peter Seibel). Of course, any new high level language requires some effort to learn. For the web application, you can use, e.g., Hunchentoot and CL-WHO, and there are a lot of database bindings (I like Postmodern and CL-SQLite).
I would suggest CLIPS and its .net port clipsnet
http://sourceforge.net/projects/clipsnet/
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have to submit a paper about the usage of source code metrics in software engineering processes. I thought it would be nice to start by introducing some categorization of source code metrics (for example using plane LOC count Vs doing some static analysis of the code) , but I can't find a book or article that will present the different types of metrics and possibly some historical background information (books and article are favored by my instructor over web sites).
any one knows of such a source ?
Patrick Smacchia's NDepend site has a good set of definitions: Metrics Definitions. His NDepend tool (for .NET) provides many different ways of quantifying a codebase.
How useful metrics are, well that is a bit more contentious. They are good for getting a sky level view of a codebase, but are they useful day-to-day? How many shops out there use (or care about) metrics on a weekly basis? I suspect most are more concerned with the bug count: number of bugs fixed and number of bugs reported.
Certainly measures like cohesion and coupling give indicators of possible refactoring points.
Another important metric: How many tests are there, and roughly how much code coverage do those tests represent. Refactoring without tests and good test coverage is a risky business.
Software metric
Code Metrics Overview
Pressman's Software Engineering: A Practitioner's Approach has a lot of good information on software metrics (among other things).
Probably the key piece of advice on metrics is to be careful what you measure. If you measure LOC, that's what people will concentrate on.
"Not everything that can be counted counts, and not everything that counts can be counted."
--Albert Einstein
"Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs."
--Bill Gates
Metrics and Models in Software Quality Engineering
Stephen H. Kan
Not always very fun to read, but it's an excellent book that covers software quality and metrics in depth.
Halstead's "Elements of Software Science" was the seminal work in this area. As I recall the tale, he defined some 15 different metrics, all purporting to measure different qualities of a piece of code.
Unfortunately, van der Meulen & Revilla showed that a couple of key metrics are very strongly correlated with SLOC (source lines of code). (This was not the first time this had been demonstrated, but it was the first paper I could find quickly.) Since SLOC is far easier to measure than the other things, SLOC is what should be measured - and managed.
PSP: A Self-Improvement Process For Software Engineers by Watts S. Humphrey
Also check SLOCCount. This is not a book but a set of tools for counting physical Source Lines of Code. Going through its user manual and the papers in the site about the usage of SLOCCount for estimating Linux kernel size will give you insight about effectiveness of SLOC metrics,issues in using it as metric, usage of SLOC metric in the COCOMO model for effort estimate.
It's not a book, but the metrics used by Sonar offer a good list of practically useful code metrics. Here's the metric definitions page.
As I just mentioned in my question on visualizing changes in metrics, there are the books Object-Oriented Metrics:Measures of Complexity by Henderson-Sellers and the earlier Object-Oriented Software Metrics which I've owned for years.
There's Metrics and Models in Software Quality Engineering, Second Edition, the only one of the books available on Safari which has a chapter discussing the above metrics and some industry results.
I also just found a 2006 book Object-Oriented Metrics in Practice: Using Software Metrics to Characterize, Evaluate, and Improve the Design of Object-Oriented Systems which has no reviews there (not a good sign) but sounds interesting.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Never been a lisp user, so don't take me as too dense while reading this. However;
What is lisp used for today?
I know there are several variants of the language in existence, at least one which will keep it alive commercially for a while longer (AutoLisp, VisualLisp - pretty big support from Autodesk)... But I don't meet everyday people using it. So if you could shed some light on the matter:
What is its primary target market nowadays?
And what do you believe its future will be?.. Will it become just another support language in few apps, or is it going somewhere?
Also, apart from "an editor whose name shall not be spoken";
What other apps keep it as a support language ?
The Lisp dialect Clojure seems to be growing in popularity - you might ask out at http://clojure.org/ in one of the forums to see what real-world apps people are building with it.
One example from my lisp projects:
ShareBot: It downloads stock market data, analyses it and trades automatically. Credits money into my bank account every month!
Aircraft Design : http://www.piano.aero/
LispWorks lists several applications : http://www.lispworks.com/success-stories/index.html
Franz technologies are widely applicable : http://www.franz.com/
I wanted to typeset some music last week, and the program with the best reputation (free or otherwise) seemed to be Lilypond. I was pleasantly surprised to see it's largely written in, and customizable with, Scheme.
Mostly for configuring and extending Emacs!
*ducks*
CoCreate Modeling, now CREO Elements/Direct Modeling, an extensive 3D CAD application uses Common Lisp as its extension language. AFAIK there are now 7M+ LOC in Common Lisp for that application. Actually Common Lisp is not only the extension language, but large parts of the application are written in Common Lisp (plus some C++).
Other than that Lisp is a family of diverse dialects with diverse implementations (Scheme, Common Lisp, Emacs Lisp, Visual Lisp, Clojure, Logo, ...) and several others.
Strengths are for example:
symbolic computing (Maxima, Reduce, Axiom, ACL2, ...)
AI, Semantic Web, ... (see the customer stories of Franz for some examples)
CAD (AutoCAD, CoCreate, and several others)
Music (OpenMusic, Common Music, PWGL, ...)
graphical applications (see the LispWorks customer stories for some examples)
development environments (Emacs and others)
Education (DrScheme, ...)
Also see the topics of the International Lisp conference 2009. This should give you an impression what people are using Lisp for and what new ideas they are thinking about.
Seems to be existent in the job market
24 jobs on dice.
Common Lisp isn't widely used in the field at all, but here is one of the most succesful applications I know of.
ITA Software: Airfare Shopping Engine and Franz lisp has a list of others.
Today lisp is used AI System where the sympolic Data explanation is used.Mainly Lisp is devloped by show the functioning of List. but it use as a symbolic representative language
It is used for anything ant everything that all other programming languages are used for, including web, games, internal applications, ...
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I know that this is subjective and all, but still, can you provide some list of serious applications that were written in Lisp (perhaps along with what Lisp it is)?
Emacs.
At least parts of it are written in Emacs Lisp.
ITA Software's airline-fare search service. See here to understand why this is impressive.
LISP itself! :)
DART. A logistics app written in Common Lisp for the first gulf war, to free Kuwait from Saddam Hussein. One could say that the war started when this app was ready. The developers worked under intense time pressure to make it possible.
This single application is said to have paid back for all (!) US government investment in AI research.
I'm particularly fond of Maxima. Another nice project I've discovered relatively recently is Nyquist.
One of the most impressive Lisp applications is surely Cyc. Cyc is a project started by Doug Lenat that tries to put common-sense knowledge into a knowledge-based system. Not just a bit, but really really lots of knowledge. Humans have learned a lot of widely different things and it was seen a limitation to artificial intelligence software to not know anything of that. What is the typical color of honey? Do we sit on a chair or under a chair? A cup of fresh coffee is hot. When it rains outside, it does not rain inside. Sea water contains salt, but water in a pond does not. Most birds can fly. Some birds can't fly. It is surprising how many of these facts, rules and concepts people know.
Cyc is under continuous development since 1984 and the knowledge base contains (numbers are from 2004) more than 2.5 million facts and rules. It also contains more than 150 thousand concepts (similar to classes in OOP). Cyc is developed in SubL (a dialect of Common Lisp).
Knowledge is entered in CycL (a declarative logic based language). Specially trained people are entering knowledge. Cyc checks that this new knowledge is not in conflict with existing knowledge. It can also derive new conclusions and one can query it. For its answers it can also generate explanations. Cyc has also many domain-specific heuristics implemented.
In the early days the project started developing on Lisp Machines, but today the software is portable and is also available to interested students and researches (OpenCyc and ResearchCyc).
Lisp Machines were used by NASA to check Space Shuttle starts. From a Symbolics press release from 1990:
"Recently the National Aeronautical and Space Administration (NASA) used Symbolics'
high-definition technology to analyze HDTV video images of the
Discovery launch in real-time. This high-definition system enabled
NASA engineers to get an instant replay of critical launch systems.
The engineers were able to enhance and enlarge high-resolution images
of the lift-off in order to analyze the condition of and spot
potential problems with space shuttle tiles."
They used a bunch of Lisp Machines with special image processing boards.
Sawfish was GNOME's default window manager long time ago. Now they default to Metacity, but you may still use Sawfish (also standalone) and it is still actively developed.
Sawfish is very feature-rich and customizable. Just what you would expect from a LISP window manager.
What eventually became Yahoo! shopping was done with Lisp. See the original author's description
AutoCAD
G2
Yahoo Store
Obidos, The Original Amazon Web Server
This according to Steve Yegge.
http://steve.yegge.googlepages.com/tour-de-babel
Whitehouse Publication Server
During the Clinton administration a Lisp application was used to inform the US citizens about government news. The application was written in Common Lisp and ran on top of the CL-HTTP web server on two Symbolics Lisp Machines (later on the Open Genera virtual Lisp Machine). It also used the Statice object-oriented database. The application provided a taxonomy of government areas which could be queried with a web or an email interface for publications. For example citizens could find out what the vice president Al Gore said about environmental issues in a certain week. This was application was shut down by George W. Bush when he came into office.
Orbitz.com?
Another application I really like is PWGL.
PWGL is a visual programming language for computer aided composition and sound synthesis. It is a LispWorks application, which is available for Windows and Mac OS X (free download). It uses a lot of sophisticated graphics (done in OpenGL) for example in its advanced note editor. It can process and create sounds. It also can use Midi input and output. It comes with a really large amount of examples in its tutorial. If you have LispWorks, you can also load the binaries and write your own code using it.
The capabilities of the application are really amazing and if you are willing to learn its usage, it is hours of fun for the hobby composer and hobby sound designer.
Probably, not the most impressive, but really worth mentioning, considering, that it's mostly a one-man effort: http://piano.aero/
It's hard to say which Lisp software is exactly most impressive, but I would add Symbolics Genera and related software to the list of achievements worth mentioning. Also don't forget SLIME.
See this question for a list of many commercial uses of Lisp.
IRCAM's OpenMusic (computer-aided composition environment).
The matchmaking server for Zone: The Battleground is written in Common Lisp. Their website.
http://www.thanandar.de/
LISP, or at least the LISP evaluator.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I'm really interested in Artificial Neural Networks, but I'm looking for a place to start.
What resources are out there and what is a good starting project?
First of all, give up any notions that artificial neural networks have anything to do with the brain but for a passing similarity to networks of biological neurons. Learning biology won't help you effectively apply neural networks; learning linear algebra, calculus, and probability theory will. You should at the very least make yourself familiar with the idea of basic differentiation of functions, the chain rule, partial derivatives (the gradient, the Jacobian and the Hessian), and understanding matrix multiplication and diagonalization.
Really what you are doing when you train a network is optimizing a large, multidimensional function (minimizing your error measure with respect to each of the weights in the network), and so an investigation of techniques for nonlinear numerical optimization may prove instructive. This is a widely studied problem with a large base of literature outside of neural networks, and there are plenty of lecture notes in numerical optimization available on the web. To start, most people use simple gradient descent, but this can be much slower and less effective than more nuanced methods like
Once you've got the basic ideas down you can start to experiment with different "squashing" functions in your hidden layer, adding various kinds of regularization, and various tweaks to make learning go faster. See this paper for a comprehensive list of "best practices".
One of the best books on the subject is Chris Bishop's Neural Networks for Pattern Recognition. It's fairly old by this stage but is still an excellent resource, and you can often find used copies online for about $30. The neural network chapter in his newer book, Pattern Recognition and Machine Learning, is also quite comprehensive. For a particularly good implementation-centric tutorial, see this one on CodeProject.com which implements a clever sort of network called a convolutional network, which constrains connectivity in such a way as to make it very good at learning to classify visual patterns.
Support vector machines and other kernel methods have become quite popular because you can apply them without knowing what the hell you're doing and often get acceptable results. Neural networks, on the other hand, are huge optimization problems which require careful tuning, although they're still preferable for lots of problems, particularly large scale problems in domains like computer vision.
I'd highly recommend this excellent series by Anoop Madhusudanan on Code Project.
He takes you through the fundamentals to understanding how they work in an easy to understand way and shows you how to use his brainnet library to create your own.
Here are some example of Neural Net programming.
http://www.codeproject.com/KB/recipes/neural_dot_net.aspx
you can start reading here:
http://web.archive.org/web/20071025010456/http://www.geocities.com/CapeCanaveral/Lab/3765/neural.html
I for my part have visited a course about it and worked through some literature.
Neural Networks are kind of declasse these days. Support vector machines and kernel methods are better for more classes of problems then backpropagation. Neural networks and genetic algorithms capture the imagination of people who don't know much about modern machine learning but they are not state of the art.
If you want to learn more about AI and machine learning, I recommend reading Peter Norvig's Artificial Intelligence: A Modern Approach. It's a broad survey of AI and lots of modern technology. It goes over the history and older techniques too, and will give you a more complete grounding in the basics of AI and machine Learning.
Neural networks are pretty easy, though. Especially if you use a genetic algorithm to determine the weights, rather then proper backpropagation.
I second dwf's recommendation of Neural Networks for Pattern Recognition by Chris Bishop. Although, it's perhaps not a starter text. Norvig or an online tutorial (with code in Matlab!) would probably be a gentler introduction.
A good starter project would be OCR (Optical Character Recognition). You can scan in pages of text and feed each character through the network in order to perform classification. (You would have to train the network first of course!).
Raul Rojas' book is a a very good start (it's also free). Also, Haykin's book 3rd edition, although of large volume, is very well explained.
I can recommend where not to start. I bought An Introduction to Neural Networks by Kevin Gurney which has good reviews on Amazon and claims to be a "highly accessible introduction to one of the most important topics in cognitive and computer science". Personally, I would not recommend this book as a start. I can comprehend only about 10% of it, but maybe it's just me (English is not my native language). I'm going to look into other options from this thread.
http://www.ai-junkie.com/ann/evolved/nnt1.html is a clear introduction to multi-layers perceptron, although it does not describe the backpropagation algorithm
you can also have a look at generation5.org which provides a lot of articles about AI in general and has some great texts about neural network
If you don't mind spending money, The Handbook of Brain Theory and Neural Networks is very good. It contains 287 articles covering research in many disciplines. It starts with an introduction and theory and then highlights paths through the articles to best cover your interests.
As for a first project, Kohonen maps are interesting for categorization: find hidden relationships in your music collection, build a smart robot, or solve the Netflix prize.
I think a good starting point would always be Wikipedia. There you'll find some usefull links to documentations and projects which use neural nets, too.
Two books that where used during my study:
Introductional course: An introduction to Neural Computing by Igor Aleksander and Helen Morton.
Advanced course: Neurocomputing by Robert Hecht-Nielsen
I found Fausett's Fundamentals of Neural Networks a straightforward and easy-to-get-into introductory textbook.
I found the textbook "Computational Intelligence" to be incredibly helpful.
Programming Collective Intelligence discusses this in the context of Search and Ranking algorithms. Also, in the code available here (in ch.4), the concepts discussed in the book are illustrated in a Python example.
I agree with the other people who said that studying biology is not a good starting point... because theres a lot of irrelevant info in biology. You do not need to understand how a neuron works to recreate its functionality - you only need to simulate its actions. I recomend "How To Create A Mind" by Ray Kurzweil - it goes into the aspect of biology that is relevant for computational models, (creating a simualted neuron by combining several inputs and firing once a threshhold is reached) but ignores the irrelvant stuff like how the neuron actually adds thouse inputs togeather. (You will just use + and an inequality to compare to a threshold, for example)
I should also point out that the book isn't really about 'creating a mind' - it only focuses on heirarchical pattern recognition / the neocortex. The general theme has been talked about since the 1980s I beleive, so there are plenty of older books that probably contain slightly dated forms of the same information. I have read older documents stating that the vision system, for example, is a multi layered pattern recognizer. He contends that this applies to the entire neocortex. Also, take his 'predictions' with a grain of salt - his hardware estimates are probably pretty accurate, but i think he underestimates how complicated simple tasks can be (ex: driving a car). Granted, he has seen a lot of progress (and been part of some of it) but i still think he is over optimistic. There is a big difference between an AI car being able to drive a mile successfully 90% of the time, when compared to the 99.9+% that a human can do. I don't expect any AI to be truly out driving me for atleast 20 years... (I don't count BMWs track cars that need to be 'trained' on the actual course, as they aren't really playing the same game)
If you already have a basic idea of what AI is and how it can be modeled, you may be better off skipping to something more technical.
If you want to do quickly learn about applications of some neural network concepts on a real simulator, there is a great online book (now wiki) called 'Computational Cognitive Neuroscience' at http://grey.colorado.edu/CompCogNeuro/index.php/CCNBook/Main
The book is used at schools as a textbook, and takes you through lots of different brain areas, from individual neurons all the way to higher-order executive functioning.
In addition, each section is augmented with homework 'projects' that are already down for you. Just download, follow the steps, and simulate everything that the chapter talked about. The software they use, Emergent, is a little finnicky but incredibly robust: its the product of more than 10 years of work I believe.
I went through it in an undergrad class this past semester, and it was great. Walks you through everything step by step