Consider a document with two arrays, lines and nodes each containing many sub-documents and add'l nested documents not shown for clarity
{
{"lines": [
{
"guid": 1
"def": {
"n1": 49,
"n2": 50
}
},
...]
},
{"nodes": [
{
"guid": 49,
"def": {
"x": -43.9709,
"y": 14.8688,
"z": -121.988
}
},
{
"guid": 50,
"def": {
"x": -30.5955,
"y": 29.5512,
"z": -115.024
}
},
...]
}
}
How could one query Mongo to return a restructured document of lines, pulling only what we need from nodes?
{
{"lines": [
{
"guid": 1
"def": {
"n1": {
"guid": 49,
"def": {
"x": -43.9709,
"y": 14.8688,
"z": -121.988
}
}
"n2": {
"guid": 50,
"def": {
"x": -30.5955,
"y": 29.5512,
"z": -115.024
}
}
}
},
...]
}
}
Coming from a healthy understanding of RDBMS, and new to MongoDB. I've been spending a lot of time hacking away with the aggregation framework to try and achieve this, but having no luck. Is there something simple I am missing?
I expect an alternate data model might be more appropriate. The above will no doubt start to crash up against the 16 MB BSON document limit for the concerning use case.
Related
My code is here:
https://github.com/Dyfused/ExplodeX/blob/master/labyrinth-mongodb/src/main/kotlin/explode2/labyrinth/mongo/LabyrinthMongo.kt#L87,L259
document structures
Set example:
{
"musicName": "Ignotus Afterburn",
"musicComposer": "Arcaea Sound Team",
"introduction": "yay",
"coinPrice": 0,
"noterName": "official",
"noterUserId": "f6fe9c4d-98e6-450a-937c-d64848eacc40",
"chartIds": [
"l54uw6y79g1pspqcsvok31ga"
],
"publishTime": {
"$date": {
"$numberLong": "1640966400000"
}
},
"category": 0,
"hidden": false,
"reviewing": false
}
GameRecord example:
{
"playerId": "92623248-b291-430a-81c5-d1175308f902",
"playedChartId": "qsx8ky1c94f9ez1b8rssbsr2",
"score": 994486,
"detail": {
"perfect": 1308,
"good": 7,
"miss": 1
},
"uploadTime": {
"$date": {
"$numberLong": "1"
}
},
"r": 0
}
Program logic
When the sort type is 'DESCENDING_BY_PLAY_COUNT', it just stuck and return something seemed to be wrong.
Line 101 to 234 is just filtering, it have nothing to do with sorting.
I want to sort the sets by the related records count. So I firstly 'lookup' the related records then get the size of the related records. But it seemed has a heavy efficient problem, which I cannot figure out how to resolve.
Relevant pipeline additions:
SearchSort.DESCENDING_BY_PLAY_COUNT -> {
pipeline += lookup("GameRecords", "chartIds", "playedChartId", "playRecords")
pipeline += addFields(Field(
"playCount",
MongoOperator.size.from("\$playRecords")
))
pipeline += sort(descending(SongSetWithPlayCount::playCount))
}
I'm very new to mongodb. I'm trying to do an aggregation pipeline for lookup kinda like SQL left join. Given the following document's schema:
Mongo playground: https://mongoplayground.net/p/yAIwH5V2yv8
Characters:
{
"_id": 1,
"account_id": 1,
"world_id": 0,
"name": "hello"
}
Inventories:
{
"_id": 7,
"character_id": 2,
"type": "EQUIPPED"
}
Items:
{
"_id": 1,
"inventory_id": 7
}
I want to query for characters and look up inventories as well as items in inventories. I was able to achieve this however I would like to separate the inventories field in characters result document.
Current result:
{
"_id": 2,
"account_id": 1,
"world_id": 0,
"name": "hello",
"inventories: [
{
"_id": 1,
"character_id": "2",
"type": "EQUIPPED",
"items: [...]
}
]
}
What I want is based on the type of inventory I want it to be a separate field of the resulted character document something like this:
{
"_id": 2,
"account_id": 1,
"world_id": 0,
"name": "hello",
"equippedInventory: {
"_id": 1,
"character_id": "2",
"type": "EQUIPPED",
"items: [...]
},
"equipInventory: {
"_id": 2,
"character_id": "2",
"type": "EQUIP",
"items: [...]
},
}
Also, is my pipeline the best way to achieve this?
I have some write performance struggle with MongoDB 5.0.8 in an PSA (Primary-Secondary-Arbiter) deployment when one data bearing member goes down.
I am aware of the "Mitigate Performance Issues with PSA Replica Set" page and the procedure to temporarily work around this issue.
However, in my opinion, the manual intervention described here should not be necessary during operation. So what can I do to ensure that the system continues to run efficiently even if a node fails? In other words, as in MongoDB 4.x with the option "enableMajorityReadConcern=false".
As I understand the problem has something to do with the defaultRWConcern. When configuring a PSA Replica Set in MongoDB you are forced to set the DefaultRWConcern. Otherwise the following message will appear when rs.addArb is called:
MongoServerError: Reconfig attempted to install a config that would
change the implicit default write concern. Use the setDefaultRWConcern
command to set a cluster-wide write concern and try the reconfig
again.
So I did
db.adminCommand({
"setDefaultRWConcern": 1,
"defaultWriteConcern": {
"w": 1
},
"defaultReadConcern": {
"level": "local"
}
})
I would expect that this configuration causes no lag when reading/writing to a PSA System with only one data bearing node available.
But I observe "slow query" messages in the mongod log like this one:
{
"t": {
"$date": "2022-05-13T10:21:41.297+02:00"
},
"s": "I",
"c": "COMMAND",
"id": 51803,
"ctx": "conn149",
"msg": "Slow query",
"attr": {
"type": "command",
"ns": "<db>.<col>",
"command": {
"insert": "<col>",
"ordered": true,
"txnNumber": 4889253,
"$db": "<db>",
"$clusterTime": {
"clusterTime": {
"$timestamp": {
"t": 1652430100,
"i": 86
}
},
"signature": {
"hash": {
"$binary": {
"base64": "bEs41U6TJk/EDoSQwfzzerjx2E0=",
"subType": "0"
}
},
"keyId": 7096095617276968965
}
},
"lsid": {
"id": {
"$uuid": "25659dc5-a50a-4f9d-a197-73b3c9e6e556"
}
}
},
"ninserted": 1,
"keysInserted": 3,
"numYields": 0,
"reslen": 230,
"locks": {
"ParallelBatchWriterMode": {
"acquireCount": {
"r": 2
}
},
"ReplicationStateTransition": {
"acquireCount": {
"w": 3
}
},
"Global": {
"acquireCount": {
"w": 2
}
},
"Database": {
"acquireCount": {
"w": 2
}
},
"Collection": {
"acquireCount": {
"w": 2
}
},
"Mutex": {
"acquireCount": {
"r": 2
}
}
},
"flowControl": {
"acquireCount": 1,
"acquireWaitCount": 1,
"timeAcquiringMicros": 982988
},
"readConcern": {
"level": "local",
"provenance": "implicitDefault"
},
"writeConcern": {
"w": 1,
"wtimeout": 0,
"provenance": "customDefault"
},
"storage": {},
"remote": "10.10.7.12:34258",
"protocol": "op_msg",
"durationMillis": 983
}
The collection involved here is under proper load with about 1000 reads and 1000 writes per second from different (concurrent) clients.
MongoDB 4.x with "enableMajorityReadConcern=false" performed "normal" here and I have not noticed any loss of performance in my application. MongoDB 5.x doesn't manage that and in my application data is piling up that I can't get written away in a performant way.
So my question is, if I can get the MongoDB 4.x behaviour back. A write guarantee from the single data bearing node which is available in the failure scenario would be OK for me. But in a failure scenario, having to manually reconfigure the faulty node should actually be avoided.
Thanks for any advice!
At the end we changed the setup to a PSS layout.
This was also recommended in the MongoDB Community Forum.
I have a map, with multiple layers, some of the layers have an events field which may contain 0 or more event listings inside of it.
Some of events are nested deeper into the map, while others are closer to the top layer.
Here is what the graphql query result looks like:
{
"data": {
"users": [
{
"id": 16,
"friends": [
{
"senderId": 16,
"receiverId": 17,
"userByReceiverid": {
"id": 17,
"events": [],
"friends": [
{
"receiverId": 14,
"userByReceiverid": {
"id": 14,
"events": [
{
"id": 3,
"photoUrl": "none",
"name": "hello",
"date": "1982-06-27",
"startTime": "01:00:00+00",
"endTime": "02:00:00+00",
"fee": "$2.00",
"maxNumber": 10,
"ageRestriction": "none",
"about": "amazing",
"allowShare": false,
"private": false,
"timestamp": "2021-06-26T17:57:13.224383+00:00",
"senderId": 14
}
]
}
},
{
"receiverId": 20,
"userByReceiverid": {
"id": 20,
"events": []
}
}
]
}
}
],
"friendsByReceiverid": [
{
"senderId": 20,
"receiverId": 16,
"user": {
"id": 20,
"events": [],
"friendsByReceiverid": [
{
"receiverId": 20,
"user": {
"id": 14,
"events": [
{
"id": 3,
"photoUrl": "none",
"name": "hello",
"date": "1982-06-27",
"startTime": "01:00:00+00",
"endTime": "02:00:00+00",
"fee": "$2.00",
"maxNumber": 10,
"ageRestriction": "none",
"about": "amazing",
"allowShare": false,
"private": false,
"timestamp": "2021-06-26T17:57:13.224383+00:00",
"senderId": 14
}
]
}
},
{
"receiverId": 20,
"user": {
"id": 17,
"events": []
}
}
]
}
}
]
}
]
}
}
I want to search the Map and make a list that contains all the events that are unique, so no duplicates if possible.
How would I go about pulling out only the events field from a map at any given point?
Essentially what you need is a function (visitObject) that goes through each key of the a Map and recursively calls itself till all properties have been checked.
The function should also check for the possible types it can operate on and handle the data so you can call the visitObject function with the right arguments.
I believe the right data structure here would be a Set. In dart as Set is defined as A collection of objects in which each object can occur only once.
However Set a set falls back to the dart method of equality which is the == operator, so you would either need to use a calls for Events or use a package like BuiltValue which will generate the == operator for your value types.
Here is an example that can help you get started, I have left equality checking for you to determine what best and how you want to apply it.
final Set<Map<String, dynamic>> items = {};
void main() {
visitObject(data);
}
void visitObject(object) {
if (object is List) {
for(dynamic item in object) {
visitObject(item);
}
} else if (object is Map) {
if (object.containsKey('events')) {
var _events = object['events'];
if (_events is List && _events.isNotEmpty) {
_events.forEach((e) {
if (!items.contains(e)) {
items.add(e);
}
});
}
}
for (dynamic item in object.values) {
visitObject(item);
}
}
}
Additional Reading for equality:
==Operator
hashCode
BuiltValue
I'm using loopback 3 to build a backend with mongoDB.
So i have 2 models: Object and Attachment. Object have a relation Embeds2Many to Attachment.
Objects look like that in mongoDB
[
{
"fieldA": "valueA1",
"attachments": [
{
"id": 1,
"url": "abc.com/image1"
},
{
"id": 2,
"url": "abc.com/image2"
}
]
},
{
"fieldA": "valueA2",
"attachments": [
{
"id": 4,
"url": "abc.com/image4"
},
{
"id": 5,
"url": "abc.com/image5"
}
]
}
]
The question is: how can i get Objects with attachments.id=4 over the RestAPI?
I have try with the where and include filter. But it didn't work. It look like, that this function is not implemented in loopback3, right?
I have found the solution. It only works on Mongodb, Cloudant and Memory database.
{
"filter": {
"where": {
"attachments.id": 4
}
}
}