Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 8 years ago.
Improve this question
I'm really interested in Artificial Neural Networks, but I'm looking for a place to start.
What resources are out there and what is a good starting project?
First of all, give up any notions that artificial neural networks have anything to do with the brain but for a passing similarity to networks of biological neurons. Learning biology won't help you effectively apply neural networks; learning linear algebra, calculus, and probability theory will. You should at the very least make yourself familiar with the idea of basic differentiation of functions, the chain rule, partial derivatives (the gradient, the Jacobian and the Hessian), and understanding matrix multiplication and diagonalization.
Really what you are doing when you train a network is optimizing a large, multidimensional function (minimizing your error measure with respect to each of the weights in the network), and so an investigation of techniques for nonlinear numerical optimization may prove instructive. This is a widely studied problem with a large base of literature outside of neural networks, and there are plenty of lecture notes in numerical optimization available on the web. To start, most people use simple gradient descent, but this can be much slower and less effective than more nuanced methods like
Once you've got the basic ideas down you can start to experiment with different "squashing" functions in your hidden layer, adding various kinds of regularization, and various tweaks to make learning go faster. See this paper for a comprehensive list of "best practices".
One of the best books on the subject is Chris Bishop's Neural Networks for Pattern Recognition. It's fairly old by this stage but is still an excellent resource, and you can often find used copies online for about $30. The neural network chapter in his newer book, Pattern Recognition and Machine Learning, is also quite comprehensive. For a particularly good implementation-centric tutorial, see this one on CodeProject.com which implements a clever sort of network called a convolutional network, which constrains connectivity in such a way as to make it very good at learning to classify visual patterns.
Support vector machines and other kernel methods have become quite popular because you can apply them without knowing what the hell you're doing and often get acceptable results. Neural networks, on the other hand, are huge optimization problems which require careful tuning, although they're still preferable for lots of problems, particularly large scale problems in domains like computer vision.
I'd highly recommend this excellent series by Anoop Madhusudanan on Code Project.
He takes you through the fundamentals to understanding how they work in an easy to understand way and shows you how to use his brainnet library to create your own.
Here are some example of Neural Net programming.
http://www.codeproject.com/KB/recipes/neural_dot_net.aspx
you can start reading here:
http://web.archive.org/web/20071025010456/http://www.geocities.com/CapeCanaveral/Lab/3765/neural.html
I for my part have visited a course about it and worked through some literature.
Neural Networks are kind of declasse these days. Support vector machines and kernel methods are better for more classes of problems then backpropagation. Neural networks and genetic algorithms capture the imagination of people who don't know much about modern machine learning but they are not state of the art.
If you want to learn more about AI and machine learning, I recommend reading Peter Norvig's Artificial Intelligence: A Modern Approach. It's a broad survey of AI and lots of modern technology. It goes over the history and older techniques too, and will give you a more complete grounding in the basics of AI and machine Learning.
Neural networks are pretty easy, though. Especially if you use a genetic algorithm to determine the weights, rather then proper backpropagation.
I second dwf's recommendation of Neural Networks for Pattern Recognition by Chris Bishop. Although, it's perhaps not a starter text. Norvig or an online tutorial (with code in Matlab!) would probably be a gentler introduction.
A good starter project would be OCR (Optical Character Recognition). You can scan in pages of text and feed each character through the network in order to perform classification. (You would have to train the network first of course!).
Raul Rojas' book is a a very good start (it's also free). Also, Haykin's book 3rd edition, although of large volume, is very well explained.
I can recommend where not to start. I bought An Introduction to Neural Networks by Kevin Gurney which has good reviews on Amazon and claims to be a "highly accessible introduction to one of the most important topics in cognitive and computer science". Personally, I would not recommend this book as a start. I can comprehend only about 10% of it, but maybe it's just me (English is not my native language). I'm going to look into other options from this thread.
http://www.ai-junkie.com/ann/evolved/nnt1.html is a clear introduction to multi-layers perceptron, although it does not describe the backpropagation algorithm
you can also have a look at generation5.org which provides a lot of articles about AI in general and has some great texts about neural network
If you don't mind spending money, The Handbook of Brain Theory and Neural Networks is very good. It contains 287 articles covering research in many disciplines. It starts with an introduction and theory and then highlights paths through the articles to best cover your interests.
As for a first project, Kohonen maps are interesting for categorization: find hidden relationships in your music collection, build a smart robot, or solve the Netflix prize.
I think a good starting point would always be Wikipedia. There you'll find some usefull links to documentations and projects which use neural nets, too.
Two books that where used during my study:
Introductional course: An introduction to Neural Computing by Igor Aleksander and Helen Morton.
Advanced course: Neurocomputing by Robert Hecht-Nielsen
I found Fausett's Fundamentals of Neural Networks a straightforward and easy-to-get-into introductory textbook.
I found the textbook "Computational Intelligence" to be incredibly helpful.
Programming Collective Intelligence discusses this in the context of Search and Ranking algorithms. Also, in the code available here (in ch.4), the concepts discussed in the book are illustrated in a Python example.
I agree with the other people who said that studying biology is not a good starting point... because theres a lot of irrelevant info in biology. You do not need to understand how a neuron works to recreate its functionality - you only need to simulate its actions. I recomend "How To Create A Mind" by Ray Kurzweil - it goes into the aspect of biology that is relevant for computational models, (creating a simualted neuron by combining several inputs and firing once a threshhold is reached) but ignores the irrelvant stuff like how the neuron actually adds thouse inputs togeather. (You will just use + and an inequality to compare to a threshold, for example)
I should also point out that the book isn't really about 'creating a mind' - it only focuses on heirarchical pattern recognition / the neocortex. The general theme has been talked about since the 1980s I beleive, so there are plenty of older books that probably contain slightly dated forms of the same information. I have read older documents stating that the vision system, for example, is a multi layered pattern recognizer. He contends that this applies to the entire neocortex. Also, take his 'predictions' with a grain of salt - his hardware estimates are probably pretty accurate, but i think he underestimates how complicated simple tasks can be (ex: driving a car). Granted, he has seen a lot of progress (and been part of some of it) but i still think he is over optimistic. There is a big difference between an AI car being able to drive a mile successfully 90% of the time, when compared to the 99.9+% that a human can do. I don't expect any AI to be truly out driving me for atleast 20 years... (I don't count BMWs track cars that need to be 'trained' on the actual course, as they aren't really playing the same game)
If you already have a basic idea of what AI is and how it can be modeled, you may be better off skipping to something more technical.
If you want to do quickly learn about applications of some neural network concepts on a real simulator, there is a great online book (now wiki) called 'Computational Cognitive Neuroscience' at http://grey.colorado.edu/CompCogNeuro/index.php/CCNBook/Main
The book is used at schools as a textbook, and takes you through lots of different brain areas, from individual neurons all the way to higher-order executive functioning.
In addition, each section is augmented with homework 'projects' that are already down for you. Just download, follow the steps, and simulate everything that the chapter talked about. The software they use, Emergent, is a little finnicky but incredibly robust: its the product of more than 10 years of work I believe.
I went through it in an undergrad class this past semester, and it was great. Walks you through everything step by step
Related
I will take my graduation project next semester , I decide to complete my high degrees studying ,because I'm from low-income people , I want to dive on anything that helps me to do a paper or research or something supports my situation to gain scholarship.
my supervisor suggests that intrusion detection system using neural network is suitable for me , and he will help me , but I need to know fundamentals on this field .
there is limited resources on this topic , just thesis , papers and researches talk about only overview on IDs using neural network .
can anyone provides me some resources and references introduce me to
intrusion detection system using neural network to learn the fundamentals and basic ?
First, some background; Neural nets are by design black box. It is less important to understand the problem you are solving when designing a neural network than it is when writing a deterministic algorithm to solve it directly. With that in mind, you probably don't need to learn about "intrusion detection systems using neural networks", but would probably benefit more from learning about neural networks and intrusion systems separately.
I will leave it to you to find texts on intrusion detection systems, but would recommend reading the following to get started on what neural networks are, and how they work:
Neural Networks - A Systematic Introduction
If you think you have understood the basis of neural networks conceptually, you will want to learn a programming language. Your options diverge somewhat at this point, but I would suggest that if you want to learn neural nets from an academic perspective and want to have more control over the design and guts of the program, you would probably benefit most from learning C++. There is a wealth of knowledge on the topic of learning C++ online. In fact, probably the most popular page on this website is dedicated to that topic:
The Definitive C++ Book Guide and List
Once you understand neural network fundamentals and C++, the world is your oyster! If you're feeling adventurous, have a look at Kenneth Stanley's NEAT algorithm. The source code will teach you a lot about neural net algorithms.
From here to creating a learning machine that understands intrusion attempts is almost trivial from a programming perspective. You really just need to get the data, which may be really easy or really hard, but your supervisor should be able to help you find data sources on which to train the network once you reach this point.
Good luck!
Can anybody suggest a good tutorial or book for neural networks in Lisp, or a blog, or share some code sample?
I have experience with neural netowrks in the imperative languages C++, Java, C#, but I want to try it in Lisp.
The seminal book AI: a modern approach includes LISP source code on the website: link
Specifically, check out the Learning chapter (perceptron etc)
In the same vein you have Paradigms of AI in Lisp, but it doesn't really touch neural networks if I remember correctly.
While the question is old and my answer is late, I still think it's valuable.
Recently I was looking for some resources on Machine Learning in Common Lisp(hence why I found this question). After doing some more research, I've found this codebase. It contains many interesting things, such as Boltzmann Machines, feed-forward and recurrent backprop neural networks. The author also has other libraries, such as evolutionary algorithms. This code is sure a good way to start.
Yann LeCun, my advisor at NYU, wrote an object-oriented dialect of lisp called Lush while he worked at Bell Labs. It feels like a lispy MATLAB, and is geared towards quick prototyping of numerical experiments and machine learning research. It installs easily if you're using Linux or Mac OS. During the late 90's a good fraction of all checks in the US were being read by the LeNet-5 net that he wrote in Lush.
We use it for most of our research, since it has so much support for convolutional neural networks, linear algebra, and has an easy C/C++ FFI for everything else. It also comes with demo code for implementing neural nets and convolutional networks for image and character classification, which is probably where you'd want to start.
It's in the Ubuntu repositories, but you probably want the latest version from here:
http://lush.sourceforge.net/
Searching on google I found these
book: "Common LISP Modules Artificial Intelligence" (at amazon)
Same at Google Books
library for Fast Artificial Neural Network
And this blog have some posts about ANN
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 2 years ago.
Improve this question
To learn FPGA programming, I plan to code up a simple Neural Network in FPGA (since it's massively parallel; it's one of the few things where an FPGA implementation might have a chance of being faster than a CPU implementation).
Though I'm familiar with C programming (10+ years). I'm not so sure with FPGA development stuff. Can you provide a guided list of what I should do / learn / buy?
Thanks!
Necroposting, but for others like me that come across this question there is an in-depth, though old, treatment of implementing neural networks using FPGAs
It's been three years since I posted this, but it is still being viewed so I thought I'd add another two papers from last year I recently found.
The first talks about FPGA Acceleration of Convolutional Neural Networks. Nallatech performed the work. It's more marketing that an academic paper, but still an interesting read, and might be a jumping off point for someone interesting in experimenting. I am not connected to Nallatech in any way.
The second paper came out of the University of Birmingham, UK, written by Yufeng Hao. It presents A General Neural Network Hardware Architecture on FPGA.
Most attempts at building a 'literal' neural network on an FPGA hit the routing limits very quickly, you might get a few hundred cells before P&R pulls takes longer to finish than your problem is worth waiting for. Most of the research into NN & FPGA takes this approach, concentrating on a minimal 'node' implementation and suggesting scaling is now trivial.
The way to make a reasonably sized neural network actually work is to use the FPGA to build a dedicated neural-network number crunching machine. Get your initial node values in a memory chip, have a second memory chip for your next timestamp results, and a third area to store your connectivity weights. Pump the node values and connection data through using techniques to keep the memory buses saturated (order node loads by CAS line, read-ahead using pipelines). It will take a large number of passes over the previous dataset as you pair off weights with previous values, run them through DSP MAC units to evaluate the new node weights, then push out to the result memory area once all connections evaluated. Once you have a whole timestep finished, reverse the direction of flow so the next timestep writes back to the original storage area.
I want to point out a potential issue with implementing a Neural Network in FPGA. FPGAs have limited amount of routing resources. Unlike logic resources (flops, look-up tables, memories), routing resources are difficult to quantify. Maybe a simple Neural Network will work, but a "massively parallel" one with mesh interconnects might not.
I'd suggest starting with a simple core from OpenCores.org just to get familiar with FPGA flow, and then move on to prototyping a Neural Network. Downloading free Xilinx WebPack, which includes ISIM simulator, is a good start. Later on you can purchase a cheap dev. board with a small FPGA (e.g. Xilinx Spartan 3) to run your designs on.
A neural network may not be the best starting point for learning how to program an FPGA. I would initially try something simpler like a counter driving LEDs or a numeric display and build up from there. Sites that may be of use include:
http://www.fpga4fun.com/ - Excellent examples of simple projects and some boards.
http://opencores.org/ - Very useful reference code for many interfaces, etc...
You may also like to consider using a soft processor in the FPGA to help your transition from C to VHDL or Verilog. That would allow you to move small code modules from one to the other to see the differences in hardware. The choice of language is somewhat arbitrary - I code in VHDL (syntactically similar to ADA) most of the time, but some of my colleagues prefer Verilog (syntactically similar to C). We debate it once in a while but really it's personal choice.
As for the buyers / learners guide, you need:
Patience :) - The design cycle for FPGAs is significantly longer than for software due to the number of extra 'free parameters' in the build, so don't be surprised if it takes a while to get designs working exactly the way you want.
A development board - For learning, I would buy one from one of the three bigger FPGA vendors: Xilinx, Altera or Lattice. My preference is Xilinx at the moment but all three are good. For learning, don't buy one based on the higher-end parts - you don't need to when starting using FPGAs. For Xilinx, get one based on the Spartan series such as the SP601 (I have one myself). For Altera, buy a Cyclone one. The development boards will be significantly cheaper than those for the higher-end parts.
A programming cable - Most companies produce a USB programming cable with a special connector to program the devices on the board (often using JTAG). Some boards have the programming interface built in (such as the SP601 from Xilinx) so you don't need to spend extra money on it.
Build tools - There are many varieties of these but most of the big FPGA vendors provide a solution of their own. Bear in mind that the tools are only free for the smaller lower-performance FPGAs, for example the Xilinx ISE Webpack.
The software comprises stages with which you may not be familiar having come from the software world. The specifics of the tool flow are always changing, but any tool you use should be able to get from your code to your specific device. The last part of this design flow is normally provided by the FPGA vendor because it's hardware-specific and proprietary.
To give you a brief example, the software you need should take your VHDL and Verilog code and (this is the Xilinx version):
'Synthesise' it into constructs that match the building blocks available inside your particular FPGA.
'Translate & map' the design into the part.
'Place & route' the logic in the specific device so it meets your timing requirements (e.g. the clock speed you want the design to run at).
Regardless of what Charles Stewart says, Verilog is a fine place to start. It reminds me of C, just as VHDL reminds me of ADA. No one uses Occam in industry and it isn't common in universities.
For a Verilog book, I recommend these especially Verilog HDL. Verilog does parallel work trivially, unlike C.
To buy, get a relatively cheap Cyclone III eval board from [Altera] or Altera's 3 (e.g. this Cyclone III one with NIOS for $449 or this for $199) or Xilinx.
I'll give you yet a third recommendation: Use VHDL. Yes, on the surface it looks like ADA. While Verilog bears a passing resemblance to C. However, with Verilog you only get the types that come with it out of the box. With VHDL you can define your own new types which lets you program at a higher level (still RTL, of course). I'm pretty sure the Xilinx and Altera free tools support both VHDL and Verilog. "A Designers Guide to VHDL" by Ashenden is a good VHDL book.
VHDL has a standard fixed-point math package which can make NN implementation easier.
It's old, because I haven't thought much about FPGAs in nearly 20 years, and it uses a concurrent programming language that is rather obscure, but Page & Luk, 1991, Compiling Occam into FPGAs covers some crucial topics in a nice way, enough, I think, for your purposes. Two links for trying stuff out:
KRoC is an actively maintained, linux-based Occam compiler, which I know has an active user base.
Roger Peel has a logic synthesis page that has some documentation of his linux-based workflow from Occam code synthesis through to FPGA I/O.
Occam->FPGA isn't where the action is, but it may be a much better place to start than, say, Verilog.
I would recommend looking into xilinx high-level synthesis, especially if you are coming from a C background. It abstracts away the technical details in using a hdl so the designer can focus on the algorithmic implementation.
The are restriction in the type of C code you can write. For example, you can't use dynamically sized data structures, as that would infer dynamically sized hardware.
I'm considering using a neural network to power my enemies in a space shooter game i'm building and i'm wondering; how do you train neural networks when there is no one definitive good set of outputs for the network?
I'm studying neural networks at the moment, and they seem quite useless without well defined input and output encodings, and they don't scale at all to complexity (see http://en.wikipedia.org/wiki/VC_dimension). that's why neural network research has had so little application since the initial hype more than 20-30 years ago while semantic/state based AI took over everyone's interests because of it's success in real world applications.
A so a good place to start might be to figure out how to numerically represent the state of the game as inputs for the neural net.
The next thing would be to figure out what kind of output would correspond to actions in the game.
think about the structure of neural network to use. To get interesting complex behavior from neural networks, the network almost has to be recurrent. You'll need a recurrent network because they have 'memory', but beyond that you don't have much else to go on. However, recurrent networks with any complex structure is really hard to train to behave.
The areas where neural networks have been successful tend to be classification (image, audio, grammar, etc) and limited success in statistical prediction (what word would we expect to come after this word, what will the stock price be tomorrow?)
In short, it's probably better for you to use Neural nets for a small portion of the game rather as the core enemy AI.
You can check out AI Dynamic game difficulty balancing for various AI techniques and references.
(IMO, you can implement enemy behaviors, like "surround the enemy", which will be really cool, without delving into advanced AI concepts)
Edit: since you're making a space shooter game and you want some kind of AI for your enemies, I believe you'll find interesting this link: Steering Behaviors For Autonomous Characters
Have you considered that it's easily possible to modify an FSM in response to stimulus? It is just a table of numbers after all, you can hold it in memory somewhere and change the numbers as you go. I wrote about it a bit in one of my blog fuelled deleriums, and it oddly got picked up by some Game AI news site. Then the guy who built a Ms. Pacman AI that could beat humans and got on the real news left a comment on my blog with a link to even more useful information
here's my blog post with my incoherant ramblings about some idea I had about using markov chains to continually adapt to a game environment, and perhaps overlay and combine something that the computer has learned about how the player reacts to game situations.
http://bustingseams.blogspot.com/2008/03/funny-obsessive-ideas.html
and here's the link to the awesome resource about reinforcement learning that mr. smarty mcpacman posted for me.
http://www.cs.ualberta.ca/%7Esutton/book/ebook/the-book.html
here's another cool link
http://aigamedev.com/open/architecture/online-adaptation-game-opponent/
These are not neural net approaches, but they do adapt and continually learn, and are probably better suited to games than neural networks.
I'll refer you to two of Matthew Buckland's books.
Programming Game AI by example
AI Techniques for Game Programming
The second book goes into back-propagation ANN, which is what most people mean when they
talk about NN anyway.
That said, I think the first book is more useful if you want to create meaningful game AI. There's a nice, meaty section on using FSM successfully (and yes, it's easy to trip yourself up with a FSM).
I will be entering my third year of university in my next academic year, once I've finished my placement year as a web developer, and I would like to hear some opinions on the two modules in the Title.
I'm interested in both, however I want to pick one that will be relevant to my career and that I can apply to systems I develop.
I'm doing an Internet Computing degree, it covers web development, networking, database work and programming. Though I have had myself set on becoming a web developer I'm not so sure about that any more so am trying not to limit myself to that area of development.
I know HCI would help me as a web developer, but do you think it's worth it? Do you think Neural Network knowledge could help me realistically in a system I write in the future?
Thanks.
EDIT:
I thought it would be useful to follow-up with what I decided to do and how it's worked out.
I picked Artificial Neural Networks over HCI, and I've really enjoyed it. Having a peek into cognitive science and machine learning has ignited my interest for the subject area, and I will be hoping to take on a postgraduate project a few years from now when I can afford it.
I have got a job which I am starting after my final exams (which are in a few days) and I was indeed asked if I had done a module in HCI or similar. It didn't seem to matter, as it isn't a front-end developer position!
I would recommend taking the module if you have it as an option, as well as any module consisting of biological computation, it will open up more doors should you want to go onto postgraduate research in the future.
The worthiness depends on three factors:
How familiar are you with the topic already?
How good is the course/class you want to take?
What are your interested in more?
Especially for HCI, there is a broad range of "common sense" information you would also easily obtain from reading a good book or a wider range of articles about it also published on the internet. On the other hand, there indeed exist many deeper insights mostly obtained by Psychology studies. If the course is done right, you can indeed learn a lot about the topic and the real considerations to use for developing an interface.
For Neural Networks, one has to say that this is a typical hype topic. It would be mainly interesting in what application domain the course wants to deal with neural networks. You can be quite sure that you won't program or use any neural networks for web development. On the other hand, if the course is done right, this could be a good opportunity for you to broaden your knowledge. Especially, deepening your understanding about the theory of computer science. This highly depends on how the course is laid out, though.
HCI is a topic which helps your career as a web developer, but only if you feel incompetent in that topic (then it is a must) or it is done very well. Neural Networks is a topic which has more potential of being really interesting hardcore computer science stuff, where you indeed learn a better understanding about something. If you are interested in NN, you should not pass the opportunity to get an education which is not narrowly concentrated on the domain of web development -- and, after all, perhaps find more interest in other stuff (it is always good to know other directions you would perhaps like to go into for the future).
Neural networks sound cool until you read the fine print:
In modern software implementations of
artificial neural networks the
approach inspired by biology has more
or less been abandoned for a more
practical approach based on statistics
and signal processing.
This is something that has mystified me for years. Here you have an amazingly complex and powerful control system (real-world biological neural networks), and an academic discipline that appears to be about modeling these systems in software but that has in reality abandoned that activity.
If you're doing web development, your time is probably better spent in the HCI course.
Go with what interests you the most. The HCI stuff will be much easier to pick up later as needed, you'll likely never get another chance to learn about neural networks!
For prospective employers (at least the good ones!) you need to show a passion and excitement about what you do. I'd sooner hire someone who can enthusiastically talk about neural networks than someone who has an extra credit in HCI.
Unless you want to do the research end of the world, ie, get a Masters/PhD, go HCI.
I studied Neural Computation at University when I studied AI. I now run my own company. The number of times since I studied that I have used my NN skills equals zero. I'm glad I did it, as it was quite fascinating, but I would have found HCI much more useful from the position I'm at now. I think that you'd pick up a lot more insight from an HCI course relevant to the software industry, but if you think you experience should be more on the esoteric/almost arty side of development, go for NN.
Which sounds like more fun? Or, equivalently, which will you work harder at? Pick that one.
Did two courses in NN and some other AI-courses - its fun to poke round with that stuff and I actually managed to implement the stuff in some of the things I've done like face-recognition, and it's useful in some other areas to if you wanna plot your lab data etc. I have never used the NN:s in my web development career though I am sure it could be used for something however what it all really boils down to is to find a client or employee willing pay for it when you can just take the straight path. So I would rather read book about it if I wasn't that hardcore about it.
Fundamental Neural Networks doesn't take to much knowledge in math, and was what I used in my first course.
as a programmer to be you need the knowledge of neural network. if parallel processing is the way to go in hardware then future programmers must be knowledgable in neural network. don't forget that NN works better with noise or imprecise data but other systems may not. Note that most data we use for analysis are sample data which is a fraction of the whole and you could imagine if some in the sample are way off. so you need knowledge of NN if you want to last in computer programming field.