We are trying to represent this data in a web application.What will be the appropriate way to represent this data? We thought of using relational structure but data are hierarchical in nature.Is it better to use MongoDB in this scenario ?
As per comment mongo dynamic schemas is a perfect solution.
let assume that we have our document structured like this:
report {
_id...
documentBycode {
_id : "G8",
dataSource,
remarks,
fields : [{
indicator : "sucide",
baseline {
data,
year,
source
},
milestone[{
year : 2017,
value : 15
}, {}
]
...
...
fields : [{
name : "nfhs1996",
value : "n/a",
order : 1 /* this is not mandatory*/
}, {
name : "ndhs2011",
value : "n/a",
order : 2
}
]
]
}
}
then you can add/modify elements as needed inside [arrays] and always get report data by retrieving only one document from datastore.
What's also could be interesting you could have mulitple diffrent reports structures stored in same collection - so you can literally store full ViewModel data AS IS
Any comment welcome!
Related
I'm working on a project where users can post things. But, I'm wondering if my firebase database structure is efficient. Below is how my database looks like so far. I have posts child contains all the post that users will post. and each user will be able to track their own posts by having posts child in uid. Is there a better way of structuring my data? or am I good to go? Any advice would be appreciated!
{
"posts" : {
"-KVRT-4z1AUoztWnF-pe" : {
"caption" : "",
"likes" : 0,
"pictureUrl" : "https://firebasestorage.googleapis.com/v0/b/cloub-4fdbd.appspot.com/o/users%2FufTgaqudXeUciW5bGgCSfoTRUw92%2F208222E1-8E20-42A0-9EEF-8AF34F523878.png?alt=media&token=9ec5301e-d913-44ee-81d0-e0ec117017de",
"timestamp" : 1477946376629,
"writer" : "ufTgaqudXeUciW5bGgCSfoTRUw92"
}
},
"users" : {
"ufTgaqudXeUciW5bGgCSfoTRUw92" : {
"email" : "Test1#gmail.com",
"posts" : {
"-KVRT-4z1AUoztWnF-pe" : {
"timestamp" : 1477946376677
}
},
"profileImageUrl" : "https://firebasestorage.googleapis.com/v0/b/cloub-4fdbd.appspot.com/o/profile_images%2F364DDC66-BDDB-41A4-969E-397A79ECEA3D.png?alt=media&token=c135d337-a139-475c-b7a4-d289555b94ca",
"username" : "Test1"
}
}
}
Working with NoSql Data , your need to take care of few things:-
Avoid Nesting The Data Too Deep
Flatten your dataStructure as possible
Prefer Dictionaries
Security Rules [Firebase]
Try this structure:-
users:{
userID1: {..//Users info..
posts:{
postID1: true,
postID2: true,
postID3: true,
}
},
userID2: {..//Users info..},
userID3: {..//Users info..},
userID4: {..//Users info..},
},
posts: {
userID1 :{
postID1: {..//POST CONTENT },
postID2: {..//POST CONTENT },
postID3: {..//POST CONTENT },
}
}
Keep the data flat and shallow. Store data elsewhere in the tree rather than nest a branch of data under a node that is simply related, duplicating the data if that helps to keep the tree shallow.
The goal is to have fast requests that return only data you need. Consider that every time the tree changes and the client-side listener fires the node and all its children are communicated to the client. Duplication of data across the tree facilitates quick requests with minimal data.
This process of flattening the data is known as "denormalization" and this section of the Firebase Doc does a nice job of providing guidance:
https://firebase.google.com/docs/database/android/structure-data
In your example above I see posts metadata nested under "users", a nested list that grows. Every time something changes under "users" the listener will fire to update the client and all of this data will be transmitted in each response. You could instead consider to fetch the posts data from the "posts" node based on the writer's uuid.
I am very new to noSQL databases. I have started my journey with mongodb Database in noSQL. Upto today I used mySQL and MS SQL Server. I know how to create tables and relationships between them. But this noSQL concept is totally different than that. So, I am asking this question. I hope I will learn sooner to work with noSQL databases.
I have used the following queries in mongodb to create collections and insert data in it:
db.createCollection('Parties');
db.parties.insert({partyName:"Best Sellers", mobileNo:"9876543214"});
db.parties.insert({partyName:"National Traders", mobileNo:"9876543215"});
db.createCollection('items');
db.items.insert({itemName:"Item-A", size:"12"});
db.items.insert({itemName:"Item-B", size:"8"});
db.createCollection('orders');
Visually my order will have these fields:
Order No : AB/123
Date of Order: 17-06-2016
Party Name : National Traders // It will be a select
Item Name Quantity Rate Amount
--------------------------------------------------------
Item-A //Select 200 20 4000
Item-B //Select 100 30 3000
--------------------------------------------------------
300 7000
My question is:
How can I make a query for mongodb to insert in order collection? I have never used something like nested collections. Can anybosy give me an example??
Update:
Here is the diagram of my database in relational database :
How can I convert it to mongodb style??
Update2:
I have tried to add this code for model order in Node.js:
var mongoose = require('mongoose');
var orderSchema = new mongoose.Schema({
salesOrderId: String,
orderDate: Date,
party: {type: Schema.Types.ObjectId, ref:'Party'},
items:[{{type: Schema.Types.ObjectId, ref:'Item'}, quantity: Number, rate: Number}],
dispatches:{
invoiceId: Number,
dispatchDate: Date,
items: [{{type: Schema.Types.ObjectId, ref:'Item'}, quantity: Number, rate: Number}]
}
});
mongoose.model('Order', orderSchema);
MongoDB doesn't follow the relational database data structure.
Copying the SQL data structure to MongoDB is not the ideal way of using MongoDB. Your application should be modified to work with MongoDB style of data structure.
For storing order data, I would suggest using just one collection which has the entire order data.
{
orderNumber : 7283,
party : {partyName:"Best Sellers", mobileNo:"9876543214"},
items : [
{itemName:"Item-A", size:"12"},
{itemName:"Item-B", size:"8"},
.....
]
orderDate : <>,
total : <>,
status : <>,
.....
}
this problem need three collections:
Item - which will hold item data
Party - which will hold party data
Order which will hold all related data
When order is placed by party, in order collection we are creating document and placing in it some subdocuments:
Order data (id, date, etc)
PartyDetails - as a whole document from party collection
Items - contains an array of all items selected by party with details (Quantity, Rate,etc)
Dispatch and DispatchDetails - those depend of how items are shipped
this structure utilities efficiently document oriented storage, and every time you need to check order status, you need just to ask for one document and you will get subdocuments at one go.
To insert data in such collection will depend of application language you will use, but I think this answer will provide you with power of document database solution.
Any comments welcome!
* EDIT *
var itemToInsertA = db.items.find({"itemName" : "Item-A"}).toArray()
var itemToInsertB = db.items.find({"itemName" : "Item-B"}).toArray()
var party = db.parties.find({"partyName" : "National Traders"}).toArray()
var order = {
//_id - will be assigned by db - but we can do it mannually
orderDate : new Date(), // now
salesOrderId : "Need to provide this from sale now or later",
PartyDetails : party[0],
Items : [{
item : itemToInsertA[0],
Quantity : 200,
Rate : 20,
ValueAtTransactionDate : 4000
}, {
item : itemToInsertB[0],
Quantity : 100,
Rate : 30,
ValueAtTransactionDate : 3000
}
]
}
db.orders.insertOne(order)
so finnaly our order looks like this:
{
"_id" : ObjectId("5767d030ecf8248c30af068c"),
"orderDate" : ISODate("2016-06-20T11:14:56.372Z"),
"salesOrderId" : "Need to provide this from sale now or later",
"PartyDetails" : {
"_id" : ObjectId("5767ce26ecf8248c30af0686"),
"partyName" : "National Traders",
"mobileNo" : "9876543215"
},
"Items" : [
{
"item" : {
"_id" : ObjectId("5767ce26ecf8248c30af0687"),
"itemName" : "Item-A",
"size" : "12"
},
"Quantity" : 200.0,
"Rate" : 20.0,
"ValueAtTransactionDate" : 4000.0
},
{
"item" : {
"_id" : ObjectId("5767ce26ecf8248c30af0688"),
"itemName" : "Item-B",
"size" : "8"
},
"Quantity" : 100.0,
"Rate" : 30.0,
"ValueAtTransactionDate" : 3000.0
}
]
}
to create a diagram I used plantUML with QEditor.
plantumpl dump
#startuml
package "Party"{
[PartyId]
[PartyCode]
[PartyName]
[MobileNumber]
}
package "Item"{
[ItemId]
[ItemCode]
[ItemName]
}
package "Order"{
[OrderId]
[SalesOrderId]
[OrderDate]
node "Dispatch"{
[InvoiceId]
[SalesInvoiceId]
[DateOfDispatch]
}
node "DispatchItemTransaction"{
[DispatchItemTId]
[Quantity.]
[Rate.]
[ItemId.]
[InvoiceId.]
}
component PartyDetails
node "Items"{
component ItemDetails
[Quantity]
[Rate]
[Value]
}
}
Item -> ItemDetails
ItemDetails-->ItemId.
Party -down-> PartyDetails
#enduml
First let me explain schema of my collections.
I have 3 collections
company,deal,price
I want to use information from all three collection and make a single reactive,responsive table. Here is the image
Now the schema for price collection is like this
{
"_id" : "kSqH7QydFnPFHQmQH",
"timestamp" : ISODate("2015-10-11T11:49:50.241Z"),
"dealId" : "X5zTJ2y675PjmaLMx",
"deal" : "Games",
"price" : [{
"type" : "worth",
"value" : "Bat"
}, {
"type" : "Persons",
"value" : 4
}, {
"type" : "Cost",
"value" : 5
}],
"company" : "Company1"
}
Schema for company collection is
{
"_id" : "da2da"
"name" : "Company1"
}
Schema for deal collection is
{
"_id" : "X5zTJ2y675PjmaLMx",
"name" : "Games"
}
For each company there will be 3 columns added in table(worth,persons,cost)
For each deal there will be a new row in table.
As the information is coming from 3 collections into a single table. First I want to ask is it wise to make a table from 3 different collections? If yes how could I do that in blaze?
If no. Then I will have to make table from price collection only . What should be schema of this collection in best way.
P.S in both cases I want to make table reactive.
Firstly, I recommend reywood:publish-composite for publishing related collections.
Secondly there is no intrinsic problem in setting up a table like this, you'll first figure out which collection to loop over with your {{#each}} in spacebars and then you'll define helpers that return the values from the related collections to your templates.
As far as your schema design, the choice as to whether to use nesting within a collection vs. using an entirely separate collection is typically driven by size. If the related object is overall "small" then nesting can work well. You automatically get the nested object when you publish and query that collection. If otoh it's going to be "large" and/or you want to avoid having to update every document when something in the related object changes then a separate collection can be better.
If you do separate your collections then you'll want to refer to objects from the other collection by _id and not by name since names can easily change. For example in your price collection you'd want to use companyId: "da2da" instead of company: "Company1"
Am looking to build a blogging system and came across the following blog.
http://blog.mongolab.com/2012/08/why-is-mongodb-wildly-popular/
While it's nice to see how we can store everything in one Mongo document as a json type object (example json from the blog pasted below) rather than distributing data across multiple tables, I'm having trouble understanding how this can accommodate an hypothetically super long comment thread.
{
_id: 1234,
author: { name: "Bob Davis", email : "bob#bob.com" },
post: "In these troubled times I like to …",
date: { $date: "2010-07-12 13:23UTC" },
location: [ -121.2322, 42.1223222 ],
rating: 2.2,
comments: [
{ user: "jgs32#hotmail.com",
upVotes: 22,
downVotes: 14,
text: "Great point! I agree" },
{ user: "holly.davidson#gmail.com",
upVotes: 421,
downVotes: 22,
text: "You are a moron" }
],
tags: [ "Politics", "Virginia" ]
}
Aside from the comments key which is represented as an array of comment objects, allowing us to store an endless number of comments within this document rather than on a separate comments table requiring a join operation to relate if we are to do this with a relational database, the rest of the fields (ie author, post, date, location, rating, tags) can all be done as columns on a relational database table as well.
Since there is a limit of 16MB per document, what happens when this blog attracts a lot of comments?
Also, why can't I store a json object on a relational database column? Afterall it's a text isn't it?
First, a clarification: MongoDB actually stores BSON, which is a essentially superset of JSON that supports more data types.
Since there is a limit of 16MB per document, what happens when this blog attracts a lot of comments?
You won't be able to increase the size past 16MB, so you'll lose the ability to add more comments. But you don't need to store all the comments on the blog post document. You could store the first N, then retire old comments to a comments collection as new ones are added. You could store comments in another collection with a parent reference. The way comments are stored should jive with how you expect them to be used. 16MB of comments would really be a lot - you might even have a special solution to handle the occasional post that gets that kind of activity, an approach that's totally different from the normal way of handling comments.
We can store json in a relational database. So what is the value of Mongo I'm getting?
Here's two ways of storing JSON (in MongoDB).
> db.test.drop()
> db.test.insert({ "name" : { "first" : "Yogi", "last" : "Bear" }, "location" : "Yellowstone", "likes" : ["picnic baskets", "PBJ", "the great outdoors"] })
> db.test.findOne()
{
"_id" : ObjectId("54f9f41f245e945635f2137b"),
"name" : {
"first" : "Yogi",
"last" : "Bear"
},
"location" : "Yellowstone",
"likes" : [
"picnic baskets",
"PBJ",
"the great outdoors"
]
}
var jsonstring = '{ "name" : { "first" : "Yogi", "last" : "Bear" }, "location" : "Yellowstone", "likes" : ["picnic baskets", "PBJ", "the great outdoors"] }'
> db.test.drop
> db.test2.insert({ "myjson" : jsonstring })
> db.test2.findOne()
{
"_id" : ObjectId("54f9f535245e945635f2137d"),
"myjson" : "{ \"name\" : { \"first\" : \"Yogi\", \"last\" : \"Bear\" }, \"location\" : \"Yellowstone\", \"likes\" : [\"picnic baskets\", \"PBJ\", \"the great outdoors\"] }"
}
Can you store and use JSON the first way using a relational database? How useful is JSON stored in the second way compared to the first?
There's lots of other differences between MongoDB and relational databases that make one better than the other for various use cases - but going further into that is too broad for an SO answer.
Can you store and use JSON the first way using a relational database?
How useful is JSON stored in the second way compared to the first?
Sorry are you suggesting that with Mongo json documents can be stored without using escape characters, whereas with a RDBMS I must use escape characters to escape the double quotes? I wasn't aware of that's the case.
I know this has been covered quite a lot on here, however, i'm very new to MongoDB and am struggling with applying answers i've found to my situation.
In short, I have two collections 'total_by_country_and_isrc' which is the output from a MapReduce function and 'asset_report' which contains an asset_id not present in the 'total_by_country_and_isrc' collection or the original raw data collection this was MapReduced from.
An example of the data in 'total_by_country_and_isrc' is:
{ "_id" : { "custom_id" : 4748532, "isrc" : "GBCEJ0100080",
"country" : "AE" }, "value" : 0 }
And an example of the data in the 'asset_report' is:
{ "_id" : ObjectId("51824ef016f3edbb14ef5eae"), "Asset ID" :
"A836656134476364", "Asset Type" : "Web", "Metadata Origination" :
"Unknown", "Custom ID" : "4748532", "ISRC" : "", }
I'd like to end up with the following ('total_by_country_and_isrc_with_asset_id'):
{ "_id" : { "Asset ID" : "A836656134476364", "custom_id" : 4748532,
"isrc" : "GBCEJ0100080", "country" : "AE" }, "value" : 0 }
I know how I would approach with in a relational database but I really want to try and get this working in Mongo as i'm dealing with some pretty large collections and feel Mongo is the right tool for the job.
Can anyone offer some guidance here?
I think you want to use the "reduce" output action: Output to a Collection with an Action. You'll need to regenerate total_by_country_and_isrc, because it doesn't look like asset_report has the fields it needs to generate the keys you already have in total_by_country_and_isrc – so "joining" the data is impossible.
First, write a map method that is capable of generating the same keys from the original collection (used to generate total_by_country_and_isrc) and also from the asset_report collection. Think of these keys as the "join" fields.
Next, map and reduce your original collection to create total_by_country_and_isrc with the correct keys.
Finally, map asset_report with the same method you used to generate total_by_country_and_isrc, but use a reduce function that can be used to reduce the intersection (by key) of this mapped data from asset_report and the data in total_by_country_and_isrc.