Mongo data modeling for monitoring application - mongodb

Question

I am trying to understand the best modeling for monitoring application.
I have a monitoring application which will be running every 30 mins to get stats from the target system and stores the details in MongoDB.
Use case:
Products, Companies
There will be around 2000 products. Products will be added/removed but the growth will not be more than 10% every month. So, I don't expect more than 3000 in the next 1 year.
Companies are the consumers for each products. There will be 1 to 10 companies for each product who are using the product. Consumers count also will go up and down.
So, on each run, we will get list of products along with the corresponding companies. Product details will be like,
Product:
Product name
Total number (this will give the current number available and will change on every poll)
Product weight
Durability days (might change once in a while)
Companies List - Who are using this product
Sample data for product:
{
"productName" : "Small Box",
"total" : NumberLong(1000),
"weight" : "1.5",
"durability" : "20",
"companies" : [
{
"name" : "Nike",
"taken" : NumberLong(10)
},
{
"name" : "Reebok",
"taken" : NumberLong(20)
}
]
}
Here, taken count will keep changing on each poll.
Web application:
There will be 3 screens to show the details.
Dashboard - Which will show high level stats like ( No of products, No of companies, Total size, ....)
Products - List view( To view the complete list )- Will show the details of a product on selecting any product
Here, I will have to show the product details and will have to list the companies who are all consuming.
Companies - List view( To view the complete list )- Will show the details company each selecting any company
Here, I will have to show Company details and all the products it is consuming.
The way, I am storing currently.
Dashboard collection - To show the stats details like, Total products, Total companies, ...
{
"time" :
"totalProducts" : NumberLong(1000),
"totalCompanies" : "1.5",
}
Products collection - Will have the following details.
{
"productName" : "Small Box",
"total" : NumberLong(1000),
"weight" : "1.5",
"durability" : "20",
"companies" : [
{
"name" : "Nike",
"taken" : NumberLong(10)
},
{
"name" : "Reebok",
"taken" : NumberLong(20)
}
]
}
Companies collection - will have the following details
{
"companyName" : "Nike",
"products" : [
{
"name" : "Small Box",
"taken" : NumberLong(10)
},
{
"name" : "Medium Box",
"taken" : NumberLong(20)
}
]
}
So, on each run, I am generating unique Id and adding this id to all the data being stored. I am keeping only last 2 weeks of data in these collections. Data older than 2 weeks will be cleaned every day.
So, when user comes to Dashboard, doing sort by to get the latest record and showing the details. There will be only one record for each run in Dashboard collection and there will be last 2 weeks of records.
When user comes to Products screen, Still will have to get the latest record from Dashboard to get the UniqueId and going to Products collection to get all the records for that UniqueId as there will be around 2000 records for each run. Same for companies collection.
Here, I will have to always show the latest data. I am going to 2 different collection when user goes to Products or Companies screen.
Is there any better approach?

Answer 1

Please check Its help you how to prepare schema
Note : Mongo Version : 3.6.5
Products Coolection
/* 1 */
{
"_id" : ObjectId("5bb1e270269004e06093e178"),
"productName" : "Small Box",
"total" : NumberLong(1000),
"weight" : "1.5",
"durability" : "20",
"companies" : [
ObjectId("5bb1e2d2269004e06093e17b"),
ObjectId("5bb1e2d8269004e06093e17c")
],
"date" : ISODate("2018-10-01T09:28:40.502Z")
}
/* 2 */
{
"_id" : ObjectId("5bb1e293269004e06093e179"),
"productName" : "Large Box",
"total" : 1000.0,
"weight" : "1.2",
"durability" : "20",
"companies" : [
ObjectId("5bb1e2d8269004e06093e17c"),
ObjectId("5bb1e2de269004e06093e17d")
],
"date" : ISODate("2018-10-01T09:28:40.502Z")
}
/* 3 */
{
"_id" : ObjectId("5bb1e29d269004e06093e17a"),
"productName" : "Medium Box",
"total" : 1000.0,
"weight" : "1.2",
"durability" : "20",
"companies" : [
ObjectId("5bb1e2d2269004e06093e17b"),
ObjectId("5bb1e2d8269004e06093e17c"),
ObjectId("5bb1e2de269004e06093e17d")
],
"date" : ISODate("2018-07-01T09:28:40.502Z")
}
Company collection
/* 1 */
{
"_id" : ObjectId("5bb1e2d2269004e06093e17b"),
"companyName" : "Nike"
}
/* 2 */
{
"_id" : ObjectId("5bb1e2d8269004e06093e17c"),
"companyName" : "Reebok"
}
/* 3 */
{
"_id" : ObjectId("5bb1e2de269004e06093e17d"),
"companyName" : "PUMA"
}
Get Single products with comapny
db.getCollection('products').aggregate([{
$match : { "_id" : ObjectId("5bb1e270269004e06093e178") } },
{ $lookup : {
from : 'company',
foreignField : '_id',
localField : 'companies',
as : 'companies'
}
}
])
All Producsts with Company List
db.getCollection('products').aggregate([
{ $lookup : {
from : 'company',
foreignField : '_id',
localField : 'companies',
as : 'companies'
}
}
])
Company by id and its used products
db.getCollection('company').aggregate([{
$match : { "_id" : ObjectId("5bb1e2d2269004e06093e17b") } },
{ $lookup : {
from : 'products',
foreignField : 'companies',
localField : '_id',
as : 'products'
}
}
])
Also by add date fields in each products you get last week data by
db.getCollection('products').aggregate([{
$match : {
date: {
$gte: new Date(new Date() - 7 * 60 * 60 * 24 * 1000)
} } },
{ $lookup : {
from : 'products',
foreignField : 'companies',
localField : '_id',
as : 'products'
}
}
])
Get latest product with company
db.getCollection('products').aggregate([
{ $sort : { date : -1} },
{ $limit : 1},
{ $lookup : {
from : 'company',
foreignField : '_id',
localField : 'companies',
as : 'companies'
}
}
])

Answer 2

So, On each run, I am generating unique Id and adding this id to all the data being stored. I am keeping only last 2 weeks of data in these collections. Data older than 2 weeks will be cleaned every day.
i wouldnt do it like this. i would use the _id, mongodb automaticly give you. and make a runobject that collects all those objects id's. because its better to have 1 key per object than 336 keys ( 48(30 minutes a day) x 14(days in 2 weeks))
i would make a run object that contains a array of company_id's , product_id's and timestamp. and i would make it a possibility that if nothing has changed in 30 minutes that you only use a runobject _id(of the first one that was the same) and a timestamp. so you saved a lot of space.
hopefully i understand you correctly, because it was a tough read for me.