How to avoid huge json documents in mongoDB - mongodb

I am new to mongoDB modelling, I have been working in a small app that just used to have one collection with all my data like this:
{
"name": "Thanos",
"age": 999,
"lastName": "whatever",
"subjects": [
{
"name": "algebra",
"mark": 999
},
{
"name": "quemistry",
"mark": 999
},
{
"name": "whatever",
"mark": 999
}
]
}
I know this is standard in mongoDB since we don't have to map relotionships to other collections like in a relational database. My problem is that my app is growing and my json, even tho it works perfectly fine, it is starting to be huge since it has a few more (and quite big) nested fields:
{
"name": "Thanos",
"age": 999,
"lastName": "whatever",
"subjects": [
{
"name": "algebra",
"mark": 999
},
{
"name": "quemistry",
"mark": 999
},
{
"name": "whatever",
"mark": 999
}
],
"tutors": [
{
"name": "John",
"phone": 2000,
"status": "father"
},
{
"name": "Anne",
"phone": 200000,
"status": "mother"
}
],
"exams": [
{
"id": "exam1",
"file": "file"
},
{
"id": "exam2",
"file": "file"
},
{
"id": "exam3",
"file": "file"
}
]
}
notice that I have simplified the json a lot, the nested fields have way more fields. I have two questions:
Is this a proper way to model Mongodb one to many relationships and how do I avoid such long json documents without splitting into more documents?
Isn't it a performance issue that I have to go through all the students just to get subjects for example?

Related

Aggregate and match multiple subdocuments in mongoDB

So I have the following document structure:
{
"id": 0,
"users": [
{
"name": "bob",
"id": 1000
},
{
"name": "sally",
"id": 2000
}
],
"comments": [
{
"commentor_index": 0,
"comment": "foo",
"emoji": "smile"
},
{
"commentor_index": 0,
"comment": "bar",
"emoji": "heart"
},
{
"commentor_index": 1,
"comment": "zee",
"emoji": "heart"
}
]
}
The comments have the users index in the array.
I would like to be able to get how many comments each user id left across all documents.
I have gotten so far to realize I need to unwind both users and comments with includeArrayIndex for users, but I have trouble rejoining them.

MongoDB lookup with multiple nested levels

In my application, I have a section of comments and replies under some documents.
Here's how my database schema looks like
db.updates.insertOne({
"_id": "62347813d28412ffd82b551d",
"documentID": "17987e64-f848-40f3-817e-98adfd9f4ecd",
"stream": [
{
"id": "623478134c449b218b68f636",
"type": "comment",
"text": "Hey #john, we got a problem",
"authorID": "843df3dbbdfc62ba2d902326",
"taggedUsers": [
"623209d2ab26cfdbbd3fd348"
],
"replies": [
{
"id": "623478284c449b218b68f637",
"type": "reply",
"text": "Not sure, let's involve #jim here",
"authorID": "623209d2ab26cfdbbd3fd348",
"taggedUsers": [
"26cfdbbd3fd349623209d2ab"
]
}
]
}
]
})
db.users.insertMany([
{
"_id": "843df3dbbdfc62ba2d902326",
"name": "Manager"
},
{
"_id": "623209d2ab26cfdbbd3fd348",
"name": "John"
},
{
"_id": "26cfdbbd3fd349623209d2ab",
"name": "Jim"
},
])
I want to join those two collections, and replace user ids with complete user information on all levels. So the final JSON should look like this
{
"_id": "62347813d28412ffd82b551d",
"documentID": "17987e64-f848-40f3-817e-98adfd9f4ecd",
"stream": [
{
"id": "623478134c449b218b68f636",
"type": "comment",
"text": "Hey #john, we got a problem",
"author": {
"_id": "843df3dbbdfc62ba2d902326",
"name": "Manager"
},
"taggedUsers": [
{
"_id": "623209d2ab26cfdbbd3fd348",
"name": "John"
}
],
"replies": [
{
"id": "623478284c449b218b68f637",
"type": "reply",
"text": "Not sure, let's involve #jim here",
"author": {
"_id": "623209d2ab26cfdbbd3fd348",
"name": "John"
},
"taggedUsers": [
{
"_id": "26cfdbbd3fd349623209d2ab",
"name": "Jim"
}
]
}
]
}
]
}
I know how to do the $lookup on the top-level fields, including pipelines, but how can I do with the nested ones?

In Netsuite REST, how do I assign an item location to an item line in a sales order?

I'm using Netsuite's Postman collection (which takes care of the Oauth1 stuff), and am POSTing to this endpoint:
{{proto}}://{{host}}/rest/platform/{{version}}/record/salesorder
... and the body is something like this:
{
"customForm": "999",
"entity": {
"id": "1111"
},
"department": {
"id": "2222"
},
"subsidiary": {
"id": "33"
},
"otherRefNum": "TEST-PO",
"location": {
"id": "444"
},
"item": {
"items": [
{
"item": { "id": "555555" },
"inventorylocation": { "id": "444" },
"price": { "id": "-1" },
"grossAmt": 999,
"quantity": 1
}
]
}
}
I'm trying to assign a location on the item level. The above request creates a sales order ok (without the line-level location assignment) if I remove the inventorylocation line, but with that in there, I get this error: Unknown reference or subrecord field inventorylocation in request body.
Netsuite's REST API documentation is here:
https://system.netsuite.com/help/helpcenter/en_US/APIs/REST_API_Browser/record/v1/2019.2/index.html#tag-salesorder
I have also tried substituting location and moving the fields around a bit, without success. (either the salesorder is created without a line-level location assignment, or I get an error similar to the above error.
Any ideas?
From the documentation you linked, it appears that the field id you need is inventorylocation rather than itemlocation.
salesorder-itemElement
...
giftCertRecipientName Recipient Name: string
id [Missing Label:id]: string
inventorydetail: salesorder-item-inventorydetail
inventorylocation: location
inventorysubsidiary: subsidiary
isClosed Closed: boolean
...
Based on the documentation for a salesOrder-itemElement, it looks like that key is correct.
Have you tried the "location": "{ID}" variation?
In LedgerSync it looks like the request for creating an invoice results in this body:
{
"entity": "309",
"location": "1",
"sublist": {
"items": [
{
"amount": 12345,
"description": "Test Line Item FLURYAOLJLFADYGR-1"
},
{
"amount": 12345,
"description": "Test Line Item FLURYAOLUFUTBYJD-2"
}
]
}
}
There also is a salesOrder-item-inventorydetail object that also contains a location. Perhaps you could use that one like so:
{
"customForm": "999",
"entity": {
"id": "1111"
},
"department": {
"id": "2222"
},
"subsidiary": {
"id": "33"
},
"otherRefNum": "TEST-PO",
"location": {
"id": "444"
},
"item": {
"items": [
{
"item": { "id": "555555" },
"inventorydetail": {
"location": "444"
},
"price": { "id": "-1" },
"grossAmt": 999,
"quantity": 1
}
]
}
}

Calculate aggregates in a bucket in Upsert MongoDB update statement

My application gets measurements from a device that should be stored in a MongoDB database. Each measurement contains values for several probes of the device.
The measurements should displayed in an aggregation for a certain amount of time. I'm using the Bucket pattern in order to prepare the aggregates and simplify indexing and querying.
The following sample shows a document:
{
"DeviceId": "Device1",
"StartTime": 100, "EndTime": 199,
"Measurements": [
{ "timestamp": 100, "probeValues": [ { "id": "1", "t": 30 }, { "id": "2", "t": 67 } ] },
{ "timestamp": 101, "probeValues": [ { "id": "1", "t": 32 }, { "id": "2", "t": 67 } ] },
{ "timestamp": 102, "probeValues": [ { "id": "1", "t": 34 }, { "id": "2", "t": 55 } ] },
{ "timestamp": 103, "probeValues": [ { "id": "1", "t": 27 }, { "id": "2", "t": 30 } ] }
],
"probeAggregates": [
{ "id": "1", "cnt": 4, "total": 123 },
{ "id": "2", "cnt": 4, "total": 219 }
]
}
Updating the values and calculating the aggregates in a single request works well if the document already exists (1st block: query, 2nd: update, 3rd: options):
{
"DeviceId": "Device1",
"StartTime": 100,
"EndTime": 199
},
{
$push: {
"Measurements": {
"timestamp": 103,
"probeValues": [ { "id": "1", "t": 27 }, { "id": "2", "t": 30 } ]
}
},
$inc: {
"probeAggregates.$[probeAggr1].cnt": 1,
"probeAggregates.$[probeAggr1].total": 27,
"probeAggregates.$[probeAggr2].cnt": 1,
"probeAggregates.$[probeAggr2].total": 30
}
},
{
arrayFilters: [
{ "probeAggr1.id": "1" },
{ "probeAggr2.id": "2" }
]
}
Now I want to extend the statement to do a upsert if the document does not exist yet. However, if I do not change the update statement at all, there is the following error:
The path 'probeAggregates' must exist in the document in order to apply array updates.
If I try to prepare the probeAggregates array in case of an insert (e.g. by using $setOnInsert or $addToSet), this leads to another error:
Updating the path 'probeAggregates.$[probeAggr1].cnt' would create a conflict at 'probeAggregates'
Both errors can be explained and seem legit. One way to solve this would be to change the document structure and create one document per device, timeframe and probe and by that simplify the required update statement. In order to keep the number of documents low, I'd rather solve this by changing the update statement. Is there a way to create a valid document in an upsert?
(as I'm just learning to use a document db, feel free to share your experience in the comments on whether it is a good goal to keep the number of documents low in real world scenarios)

Using GraphQL,Springboot,MongoDB.The json is 1000+ lines deeply nested.Instead of updating whole doc,need to update specific key-value at any position

Requirement for mutation is to behave like upsert. For example in below json mutation is required to change "status" under Rooms->Availability section. Which I do not want to hard code like
db.collection.update({
'Rooms.Availability.status':true
},{
$set : {
'Rooms.Availability.status':False
}
})
for specific Array Index because there are possibilities of not having "status" or "Availability" key in some other document.
Below is similar JSON structure. Keys can be different in other JSON documents within same collection.
#GraphQLMutation(name = "updateHotelDetails")
public Hotel updateHotelDetails(Hotel h){ // Do I need to pass whole object as argument or only specific key-value?
mongoTemplate.updateFirst(....); // How can I write update code without hard coding?
}
Document 1:
{
"_id" : ObjectId("71testsrtdtsea6995432"),
"HotelName": "Test71testsrtdtsea699fff",
"Description": ".....",
"Address": {
"Street": "....",
"City": "....",
"State": "...."
},
"Rooms": [
{
"Description": "......",
"Type": ".....",
"Price": "....."
"Availability": [
"status": false,
"readOnly": false
]
},
{
"Description": "......",
"Type": "....",
"Price": "..."
"Availability": [
"status": true,
"readOnly": false
]
"newDynamickey": [
{}
]
},
]
"AdditionalData": [
{
"key1": "Vlaue1",
"key2":"Value2"
},
{...}
]
}