I am trying to aggregate multiple nested properties using MongoDB aggregation.
Example collection:
[{
"_id" : ObjectId("59c0b704863b6c23edb9671e"),
"client_id": "c1",
"errors": {
"500": 1,
"404": 3
}
},
{
"_id" : ObjectId("68c0b704863b6c23edb9671e"),
"client_id": "c1",
"errors": {
"404": 1,
"403": 5,
"401": 10
}
}]
Desired aggregation query result:
{
"client_id": "c1",
"errors": {
"500": 1,
"404": 4,
"403": 5,
"401": 10
}
}
1M records (or more) need to be aggregated this way.
The errors keys are not definite or final and can be added on demand, the aggregation should not fail in that case.
I have also considered using arrays with $unwind but that is costly.
I can do that quite easily using map-reduce but the dataset is large and performance is not acceptable.
Is there a way to use aggregation efficiently for this kind of task?
Related
DB collection seatsObj:
{
"product_id": 46539040,
"freeSeating": false,
"tempTransId": "1ecae165f2d86315fea19963d0ded41a",
"seatLayout": {
"colAreas": {
"Count": 2,
"intMaxSeatId": 43,
"intMinSeatId": 2,
"objArea": [
{
"AreaDesc": "EXECUTIVE",
"AreaCode": "0000000003",
"AreaNum": "1",
"HasCurrentOrder": true,
"objRow": [
{
"GridRowId": 1,
"PhyRowId": "A",
"objSeat": [
{
"GridSeatNum": 1,
"SeatStatus": "1",
"seatNumber": 1,
"seatPrice": 400,
"ID": 111
},
{
"GridSeatNum": 2,
"SeatStatus": "0",
"seatNumber": 2,
"seatPrice": 450,
"ID": 112
},
I was able to find ways to locate and update specific fields using:
seatsObj.updateOne(
{"seatLayout.colAreas.objArea.0.objRow.0.objSeat.seatPrice": 470},
{$set: {"seatLayout.colAreas.objArea.0.objRow.0.objSeat.$.ID": 888}});
but i cannot find simple way to return a specific field value from objSeat array element based on search criteria (for example: get 400 as a result of querying seatPrice for the seat with ID = 111). Could anyone give me a direction? From my initial research I have to go into crazy nested $unwind -s and $objectToArray -s, etc... Isn't there a simpler way? Thank you!!
I'm trying to search any value that match with a "name" param, inside any object with any level in a MongoDB collection.
My BSON looks like this:
{
"name": "a",
"sub": {
"name": "b",
"sub": {
"name": "c",
"sub": [{
"name": "d"
},{
"name": "e",
"sub": {
"name": "f"
}
}]
}
}
}
I've created an index with db.collection.createIndex({"name": "text"}); and it seems to work, because it has created more than one.
{
"numIndexesBefore" : 1,
"numIndexesAfter" : 6,
"note" : "all indexes already exist",
"ok" : 1
}
But, when I use this db.collection.find({$text: {$search : "b"}}); to search, it does not work. It just searches at the first level.
I cannot do a search with precision, because the dimensions of the objects/arrays is dynamic and can grow or shrink at any time.
I appreciate your answers.
MongoDB cannot build an index on arbitrarily-nested objects. The index only occurs for the depth specified. In your case, the $text search will only check the top-level name field, but not the name field for any of the nested sub-documents. This is an inherent limitation for indexing.
To my knowledge, MongoDB has no support for handling these kinds of deeply-nested data structures. You really need to break your data out into separate documents in order to handle it correctly. For example, you could break it out into the following:
[
{
"_id": 0,
"name": "a",
"root_id": null,
"parent_id": null
},
{
"_id": 1,
"name": "b",
"root_id": 0,
"parent_id": 0
},
{
"_id": 2,
"name": "c",
"root_id": 0,
"parent_id": 1
},
{
"_id": 3,
"name": "d",
"root_id": 0,
"parent_id": 2
},
{
"_id": 4,
"name": "e",
"root_id": 0,
"parent_id": 2
},
{
"_id": 5,
"name": "f",
"root_id": 0,
"parent_id": 4
}
]
In the above structure, our original query db.collection.find({$text: {$search : "b"}}); will now return the following document:
{
"_id": 1,
"name": "b",
"root_id": 0,
"parent_id": 0
}
From here we can retrieve all related documents by retrieving the root_id value and finding all documents with an _id or root_id matching this value:
db.collection.find({
$or: [
{_id: 0},
{root_id: 0}
]
});
Finding all root-level documents is a simple matter of matching on root_id: null.
The drawback, of course, is that now you need to assemble these documents manually after retrieval by matching a document's parent_id with another document's _id because the hierarchical information has been abstracted away. Using a $graphLookup could help alleviate this somewhat by matching each subdocument with a list of ancestors, but you would still need to determine the nesting order manually.
Regardless of how you choose to structure your documents moving forward, this sort of restructure is going to be needed if you're going to query on arbitrarily-nested content. I would encourage you to consider different possibilities and determine which is most suited for your specific application needs.
Hello Good Developers,
I am facing a situation in MongoDB where I've JSON Data like this
[{
"id": "GLOBAL_EDUCATION",
"general_name": "GLOBAL_EDUCATION",
"display_name": "GLOBAL_EDUCATION",
"profile_section_id": 0,
"translated": [
{
"con_lang": "US-EN",
"country_code": "US",
"language_code": "EN",
"text": "What is the highest level of education you have completed?",
"hint": null
},
{
"con_lang": "US-ES",
"country_code": "US",
"language_code": "ES",
"text": "\u00bfCu\u00e1l es su nivel de educaci\u00f3n?",
"hint": null
}...
{
....
}
]
I am projecting result using the following query :
db.collection.find({
},
{
_id: 0,
id: 1,
general_name: 1,
translated: {
$elemMatch: {
con_lang: "US-EN"
}
}
})
here's a fiddle for the same: https://mongoplayground.net/p/I99ZXBfXIut
I want those records who don't match $elemMatch don't get returned at all.
In the fiddle output, you can see that the second item doesn't have translated attribute, In this case, I don't want the second Item at all to be returned.
I am using Laravel as Backend Tech, I can filter out those records using PHP, but there are lots of records returned, and I think filtering using PHP is not the best option.
You need to use $elemMatch in the first parameter
db.collection.find({
translated: {
$elemMatch: {
con_lang: "IT-EN"
}
}
})
MongoPlayground
I am struggling with some query in MongoDB. Let's say I have standings collection which looks like
{
"competitions: {id: "1", name:"someLeague"},
"standings": [
{
"type": "TOTAL",
"table": [
{
"position": "1",
"team": {
"id": "123",
"name": "XYZ"
},
won: "1",
draw: "2",
lost: "3",
points: "4",
},
{
"position": "2",
"team": {
"id": "321",
"name": "ABC"
}
...
And the fixtures collection which looks like
{
matchDay: "YYYY-MM-DD",
homeTeam: {id: "123", name:"ABC"},
awayTeam: {id: "321", name:"XYZ"},
}
Is it possible to connect this two collection this way that field "homeTeam" in fixtures collection will contain all information including points, won games etc. from standings where type would be total? And same thing with the field awayTeam, with the proviso that information of team would be from array where standings type is away.
There is no means in MongoDB to reference a document of collection A in collection B so that find queries on collection B automatically provide attributes of the referenced document. However, as of MongoDB 3.2 it is possible to use $lookup command as part of an aggregation (see https://stackoverflow.com/a/33511166/3976662) to JOIN (similar to standard SQL) over multiple collections during the query. In your case, you can consider using $lookup in conjunction with $unwind - similar to the example in the MongoDB docs. Spring Data Mongo supports $lookup since 1.10.
The sample structure is as follows:
{
"_id": 1,
"College_name" : "abcdef"
"Students": [
{
"name": "a",
"Grade": "First"
},
{
"name": "b",
"Grade": "Second"
},
{
"name": "c",
"Grade": "First"
},
{
"name": "d",
"Grade": "First"
}
]
}
What I am not getting is - I want to get all ("Grade" : "First") objects in the array and insert only ("Grade" : "First") objects to other collection where ("College_name" : "abcdef"). What is the optimal solution for this?
Is this possible through aggregation??
Please help me out to solve this.
You can fetch the data with {"Grade": "First"} either using aggregation or mongo query -
Aggregation query -
db.coll.aggregate([{'$match':{"Students.Grade":"First"}}, {$project: {"_id":1, "College_name":1}}])
Mongo Query -
db.coll.find({"Students.Grade":"First"}, {"College_name":1})
After that you can insert the data to other collection.
Answer to this question is in the link:
query to retrieve multiple objects in an array in mongodb
i.e.,
Aggregation query will have 4 pipeline operators:
[match, unwind, match, group]
and then can be inserted to the other collection.