store/query semi-structure data [closed] - perl

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
We had application written in perl that create complex data structure for our subscriber (we have move than 4m subscribers). each subscriber have some conmen fields that are are present in all of them and some other subscriber has missing some.
The data looks like this:
%subscribers = {
"user_001" = {
"name" => "sam",
"age" => "13",
"color" =>['red','blue']
"item"=>{
"old" =>['PC','pen'],
"new" =>['tap','car']
},
"user_002" = {
"name" => "ali",
"age" => "54",
"color" =>['red','null','green']
"item"=>{
"old" =>['phone','TV']
},
"user_003" = {
"name" => "foo",
"age" => "02",
"item"=>{
"old" =>['']
},
....
}
}
our data are more nasty and complex
Now we try to store these data in DB then do some query in them like get user that have new 'TAPs' in item or there age is larger than 30 years.
what we need to know is:
What is the best method to store the data (as MySQL or Oracle db not option), we need something for semi-structure data. How to do these queries taken in mind the preformence.
We jast need headline to start our search (and yes we did our homework using Google ^_^).
BR,
Hosen

Answer 1

It sounds like your dataset is still small and manageable, so you need to be very careful about dismissing traditional database solutions at this early point. You haven't really offered any hard reasons as to why SQL solutions have been dismissed (new features in recent years are targeted squarely at NoSQL use-cases), so as someone that's trawled through this issue myself in the past (in a large perl project) I will offer some questions you should ask yourself:
Will the new technology choice become the authoritative data store, or just something you want to bolt-on with minimum changes to help you service queries?
If you just want to quickly bolt-on a new API to service queries, NoSQL technologies such as MongoDB (with excellent perl driver) become a viable option (and you can slurp in a perl hash as you've described with very little code). If you only use it as a (possibly read-only) cache, you mitigate all the durability concerns and avoid a lot of expensive data cleaning/validation/normalization effort to get you to an 80% solution very quickly.
If you want something durable to replace your current data storage, it's true that there are options other than SQL RDBMS. XML stores like eXistDB are very powerful if you work with XML ecosystems already and your data fits the document-object paradigm where XQuery/XPath makes sense (there's even a perl RPC thing for it). It's worth taking a look at commercial vendors like MarkLogic or EnterpriseDB if you have time pressures and a decent budget. If your data is truly messy and can be efficiently modeled as a graph of entities and relationships, it's tempting to consider things like SparkleDB, Neo4j or Virtuoso however in my limited exposure to these things whilst they have a lot of potential for servicing otherwise impossible or difficult queries/analsyses, they make a terrible place to curate and manage your core business data.
What kinds of queries, reports/analyses do you hope to do? This will determine how much data cleaning and normalization effort will be required. Answering this question will help you focus your choice:
If you think you'll end up doing data cleaning/validation/transformation in order to implement your final choice and make the data queryable, you might as well use a traditional SQL database but explore using it in a "NoSQL" way (there's lots of advice/comparison out there).
If you are hoping to avoid doing a lot of data cleaning/validation/normalization due to lack of time or budget, I'm afraid that the more mature XML/RDF/SPARQL solutions will require 10x more engineering effort to design and establish a working system built around the messy data than simply cleaning it properly in the first place.
If you have truly messy, heterogeneous data (especially when you need to continuously import from 3rd-parties over which you have no control and you want to avoid constant data cleaning effort), then leaving your messy data "as-is" lands you in a spectrum of hurt. At one extreme (in terms of cost but also query power/expressiveness and accuracy) you have the XML/RDF/SPARQL solutions mentioned before. At the cheaper/quicker/simpler (perhaps too simple in many cases) you have contenders such as MongoDB, Cassandra and CouchDB (this is by no means an exhaustive list, and they have differing levels of perl support or quality of perl clients).