What are the advantages of one tool over the other has ? What are the differences between them. Where which tool is more preferred
Related
I see that you can use datastore to hold key value pairs, process data in chunks, and pass it to mapreduce. Does this mean that the datastore object in Matlab is like a NoSQL database? If not, how does it differ?
In case of any ambiguity about what characterises a NoSQL database, I am considering as a starting point these characteristics obtained from dba.stackexchange: https://dba.stackexchange.com/a/25/35729
You'll find that NoSQL database have few common characteristics. They
can be roughly divided into a few categories:
key/value stores
Bigtable inspired databases (based on the Google Bigtable paper)
Dynamo inspired databases
distributed databases
document databases
In Matlab you can always import java Classes and use any java library, (with the one difference that there is no multithreading). So typically you won’t find many libraries written in matlab that do the same thing as a java library for this reason. In general I would also say it’s harder to write a library in matlab which may be a factor for the lack of libraries as well.
I think your only option is to use a java library, IMHO is a much better choice anyway because java is so much more popular to programmers working with databases, it will always have better libraries which are maintained. The one drawback is that you can’t implement java interfaces in matlab (correct me if I’m wrong). This can become a massive pain.
So not really, here is a Mongo examples on github https://github.com/HanOostdijk/matlab_mongodb
I am extensively using lists in our UVC’s monitor, due to our protocol specifications a lot is modeled using FIFO operation of list.push() and list.pop0(), since pop0() is a very expansive operation in ‘e’ on large list,
Does e language provides a more efficient solution or data structure for dealing with FIFO management?
You can use eTL.
these are templates that implements common cases such as FIFO and more.
the way they are implemented is more efficient and focus on performance.
Yes. Use etl. For example, in order to overcome the poor performance of the list.pop0() function, you can use a deque of uint (instead of list of uint) for better performance.
Yes e support solution for dealing with those kinds of problems.
One of the solutions is using eTL or redefine existing methods.
I'm looking for a little assistance in Scala similar to that provided by pyTables. PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data.
Any suggestions?
I had a quick look at pyTables, and I don't think there's anything remotely like it in Scalaland (or indeed Javaland), but we have a few of the ingredients necessary to make it a possibility if you want to invest the time:
scala.Dynamic to do idiomatic selection on data-driven structures
A bunch of graph databases to provide the underlying navigational persistence substrate (I've had acceptable results from OrientDB, which has a better license than most)
PyTables is a python implementation of HDF5 with some added niceties to let you work on it in a pythonic way, and get good indexing support. I'm not sure if there's a package implemented in a similar way in Scala, but you can use the same HFD5 based hierarchical data storageusing the HDF5 implementation in Java: HDF Java
Are there any special concurrency operators, or is functional style programming good for concurrency? And why?
At the moment, Scala already supports two major strategies for concurrency - thread-based concurrency (derived from Java) and type-safe Actor-based concurrency (inspired by Erlang). In the nearest future (Scala 2.9), there will be two big additions to that:
Software Transactional Memory (the basis for concurrency in Clojure, and probably the second most popular concurrency-style in Haskell)
Parallel Collections (without going into detail, they allow for parallelizing basic collection transformers, like foreach or map across multiple threads).
Actor syntax (concurrency operators) is heavily influenced by Erlang (with some important additions) - with regards to the library you use (standard actors, Akka, Lift, scalaz), there will be different combinations of question and exclamation marks: ! (in the most cases for sending a message one-way), !!, !?, etc.
Besides that, first-class functions make your life way easier even when you work with old Java concurrency frameworks: ExecutorService, Fork-Join Framework, etc.
Above all of that stands immutability that simplifies concurrency a lot, making code more predictable and reliable.
There are a number of language features that make Scala good for concurrency. For example:
Most of the data structures are immutable and don't require anything special for concurrent access.
The functional style is a really good way to do highly concurrent operations.
Scala includes a really handy "actor" framework that helps with concurrent asynchronous operations.
Further reading:
http://www.ibm.com/developerworks/java/library/j-scala02049.html
http://blog.objectmentor.com/articles/2008/08/14/the-seductions-of-scala-part-iii-concurrent-programming
http://akka.io
Well, there is the hype and there is the reality. Scala got a fame for being good for concurrency because it is a functional language and because of its actors library. Functional languages are good for concurrency because they focus on immutability, which helps concurrent algorithms. Actors got their reputation because they are the base to Erlang's track record of massively concurrent systems.
So, in a sense, Scala's reputation is due to being a "me too" of successful techniques. Yet, there is something that Scala does bring to the table, which is its ability to support such additions to the language through libraries, which makes it able to adapt and adopt new techniques as they are devised.
Actors are not native to Scala, and yet there are already there different libraries in wide use that all seem to be. Neither is transactional memory, but, again, there are already libraries that look like they are.
They, these libraries, are even available for java, but there they are clunky to use.
So the secret is not so much what it can do, but that it makes it look easy.
The big keyword here is immutability. See this Wiki page. Since any variable can be defined as mutable or immutable, this is a big win for concurrency, since if an object cannot be changed, it is thread safe and therefore writing concurrent programs is easier.
Just to rain on everyone's parade a bit, see this question.
I've dual-coded a fair few multithreaded things in Scala (mainly using Futures, a bit with Actors too) and C++ (using TBB) since then (mostly Project Euler problems). General picture seems to be that Scala needs ~1/3 the number of lines of code of the C++ solution (and is quicker to write), but the C++ solution will be ~x10 faster runtime (unless you go to some efforts to avoid any "object churn", as shown in the fast version in the answer referenced above, but at that point you're losing much of the elegance of Scala). I'm still on 2.7.7 mind you; haven't tried Scala 2.8 yet.
Using Scala was my first real encounter with a language which strongly emphasised immutability and I'm impressed by its power to hugely simplify the mental model you have to maintain of object "state machines" while programming (and this in turn makes coding for concurrency easier). It's certainly influenced how I approach C++ coding.
There appears to be a myriad of NoSQL databases available these days:
CouchDB
MongoDB
Cassandra
Hadoop
There's also a boundary between these tools and tools such as Redis that work as a memcached replacement.
Without hand waving and throwing too many buzz words - my question is the following:
How does one intelligently decide which tool here makes the most sense for their project? Are the projects similar enough to where the answer to this is subjective, eg: Ruby is better than Python or Python is better than Ruby? Or are we talking Apples and oranges here in that they each of them solve different problems?
What's the best way to educate myself on this new trend?
Perhaps one way to think of it is, programming has recently evolved from using one general-purpose language for everything to using the general-purpose language for most things, plus domain-specific languages for the more appropriate parts. For example, you might use Lua to script artificial intelligence of a character in a game.
NoSQL databases might be similar. SQL is the general purpose database with the longest and broadest adoption. While it could be shoehorned to serve many tasks, programmers are beginning to use NoSQL as a domain-specific database when it is more appropriate.
I would argue, that the 4 major players you named do have quite different featuresets and try to solve different problems with different priority.
For instance, as far as i know Cassandra (and i assume Hadoop) central focus is on large scale installations.
MongoDb tries to be a better scaling alternative to classic SQL servers in providing comparably powerful query functions.
CouchDB's focus is comparably small scale (will not shard at all, "only" replicate), high durability and easy synchronization of data.
You might want to check out http://nosql-database.org/ for some more information.
I am facing pretty much the same problem as you, and i would say there is no real alternative to look at all solutions in detail.
Check out this site: http://cattell.net/datastores/ and in particular the PDF linked at the bottom (CACM Paper). The latter contains an excellent discussion of the relative merits of various data store solutions.
It's easy. NoSQL databases are ACID compliant databases minus some guarantees. So just decide which guarantees you can do without and find the database that fits. If you don't need durability for example, maybe redis is best. Or if you don't need multi-record transactions, then perhaps look into mongodb.