I am finding the MongoDB aggregation framework to be extremely powerful - it seems like a good option to flatten out an object. My schema uses a an array of sub objects in an array called materials. The number of materials is variable, but a specific field category will be unique across objects in the array. I would like to use the aggregation framework to flatten the structure and dynamically rename the fields based on the value of the category field. I could not find an easy way to accomplish this using a $project along with $cond. Is there a way?
The reason for the array of material objects is to allow simple searching:
e.g. { 'materials.name' : 'XYZ' } pulls back any document where "XYZ" is found.
E.g. of before and after document
{
"_id" : ObjectId("123456"),
"materials" : [
{
"name" : "XYZ",
"type" : "Red",
...
"category" : "A"
},
{
"name" : "ZYX",
"type" : "Blue",
...
"category" : "B"
}]
}
to
{
"material_A_name" : "XYZ",
"material_A_type" : "Red",
...
"material_B_name" : "ZYX",
"material_B_type" : "Blue",
...
}
There is a request for something like this in jira https://jira.mongodb.org/browse/SERVER-5947 - vote it up if you would like to have this feature.
Meanwhile, there is a work-around if you know up front what the possible values of the keys will be (i.e. all unique values of "category") and I have some sample code on it on my blog.
This would be useful from MongoDB v4.4,
$map to iterate loop of materials array
$map to iterate loop of name and type fields after converting to array using $objectToArray, concat your key fields requirement as per fields and value using $concat,
back to first $map convert returned result from second $map from array to object using $arrayToObject
$unwind deconstruct materials array
$group by null and merge materials object to one object
$replaceRoot to replace object in root
db.collection.aggregate([
{
$project: {
materials: {
$map: {
input: "$materials",
as: "m",
in: {
$arrayToObject: [
{
$map: {
input: {
$objectToArray: {
name: "$$m.name",
type: "$$m.type"
}
},
in: {
k: { $concat: ["material", "_", "$$m.category", "_", "$$this.k"] },
v: "$$this.v"
}
}
}
]
}
}
}
}
},
{ $unwind: "$materials" },
{
$group: {
_id: null,
materials: { $mergeObjects: "$materials" }
}
},
{ $replaceRoot: { newRoot: "$materials" } }
])
Playground
Related
I am trying to update the property name of the json in mongodb document.
{
"_id" : ObjectId("1234556789"),
"apps" : [
{
"_id" : 101,
"regions" : [
"WANAE",
"WANAF"
]
},
{
"_id" : 102,
"regions" : [
"WANAE",
"WANAF"
]
}
]
}
in the above josn, I want to change apps regions to codes. Treid below queries but did not work
db.packs.updateMany( {}, { $rename: { 'apps.$.regions': 'apps.$.codes' } } );
db.packs.updateMany( {}, { $rename: { 'apps.$[].regions': 'apps.$[].codes' } } );
any help
Update: As Joe suggested, I have a aggregation that changes the document with the changes needed and I tried updating the entire collection like below with the aggregated result
db.packs.aggregate([
{
$addFields: {
apps: {
$map: {
input: "$apps",
as: "app",
in: {
_id: "$$app._id",
did: "$$app.did",
name: "$$app.name",
codes: "$$app.regions"
}
}
}
}
},
{
$project:{
"apps.regions":0
}
},
{
$out:"packs"
}
])
As per the documentation, $out should replace the existing collection if it is exists but I received an error that says I have to supply a new collection name Please supply a collection that does not already exist to the $out stage.. Isn't $Out replace the exiting packs with new aggregated results
When you reference a field in an array of objects, like "$apps.regions", the value is an array containing all of the values of that field from all of the elements.
If you set the value of regions directly, each sub document will contain an array of arrays, probably not what you want.
renaming the field in the entire array of objects will require iterating the array, perhaps with $map or $reduce.
If you are using MongoDB 4.2, you can do that with a pipeline in an update:
db.packs.updateMany( {}, [
{$set: {
"apps": {
$map: {
input: "$apps",
in: {
$mergeObjects: [
"$$this",
{
codes: "$$this.regions"
}
]
}
}
}
}},
{$unset: "apps.regions"}
]}
If you are using an earlier version, you'll need to do that with aggregation, perhaps with $out, and then replace the original collection with the updated one.
Below is the mongodb collection sample data. I'm trying to fire a query which will return and array of a property in my collection.
Sample Docs :
{
"_id" : ObjectId("5e940d6c2f804ab99b24a633"),
"accountId" : ObjectId("7e1c1180d59de1704ce43557"),
"description":"some desc",
"configs": {},
"dependencies" : [
{
"commute" : {},
"distance" : {},
"support":{}
}
]
}
Expected output :
{
[0]:commute
[1]:distance
[2]:support
}
I've tried to get the response and iterate the dependencies object also I tried using es6 to convert it to an array object. But it is an expensive operation as this list would be really large. If at all there exists and such approach in Mongo to convert the response into array
You can try below aggregation query :
db.collection.aggregate([
{
$project: {
_id: 0,
dependenciesKeys: {
$reduce: {
input: "$dependencies", // Iterate over 'dependencies' array
initialValue: [],
in: {
$concatArrays: [ /** concat each array returned by map with holding 'value */'
"$$value",
{
$map: {
input: {
$objectToArray: "$$this" /** Convert each object in 'dependencies' to array [{k:...,v:...},{k:...,v:...}] & iterate on each object */
},
in: "$$this.k" // Just return keys from each object
}
}
]
}
}
}
}
}
])
Test : MongoDB-Playground
Ref : $reduce , $map , $concatArrays , $objectToArray & $project
I think you should try Object to array, which is an aggregation pipeline operator.
Also the Aggregation Pipeline is pretty useful concept, it makes you create a pipeline of operations that process your data sequentially. You can imagine it as following; you create a steps 'Pipeline' every step make a certain operation 'Aggregate operation' in your data, which results to some certain data shape,like an array as you want.
This question already has answers here:
Retrieve only the queried element in an object array in MongoDB collection
(18 answers)
Closed 3 years ago.
I have the following structure (this can't be changed, that is I have to work with):
{
"_id" : ObjectId("abc123"),
"notreallyusedfields" : "dontcare",
"data" : [
{
"value" : "value1",
"otherSomtimesInterestingFields": 1
"type" : ObjectId("asd123=type1"),
},
{
"value" : "value2",
"otherSometimesInterestingFields": 1
"type" : ObjectId("asd1234=type2"),
},
many others
]
}
So basically the fields for a schema are inside an array and they can be identified based on the type field inside 1 array element (1 schema field and it's value is 1 element in the array). For me this is pretty strange, but I'm new to NoSQL so this may be ok. (also for different data some fields may be missing and the element order in the data array is not guaranteed)
Maybe it's easier to understand like this:
Table a: type1 column | type2 column | and so on (and these are stored in the array like the above)
My question is: how can you select multiple fields with conditions? What I mean is (in SQL): SELECT * FROM table WHERE type1=value1 AND type2=value2
I can select 1 like this:
db.a.find( {"data.type":ObjectId("asd1234=type2"), "data.value":value2}).pretty()
But I don't know how could I include that type1=value1 or even more conditions. And I think this is not even good because it can match any data.value field inside the array so it doesn't mean that where the type is type2 the value has to be value2.
How would you solve this?
I was thinking of doing 1 query for 1 condition and then do another based on the result. So something like the pipeline for aggregation but as I see $match can't be used more times in an aggregation. I guess there is a way to pipeline these commands but this is pretty ugly.
What am I missing? Or because of the structure of the data I have to do these strange multiple queries?
I've also looked at $filter but the condition there also applies to any element of the array. (Or I'm doing it wrong)
Thanks!
Sorry if I'm not clear enough! Please ask and I can elaborate.
(Basically what I'm trying to do based on this structure ( Retrieve only the queried element in an object array in MongoDB collection ) is this: if shape is square then filter for blue colour, if shape is round then filter for red colour === if type is type1 value has to be value1, if type is type2 value has to be value2)
This can be done like:
db.document.find( { $and: [
{ type:ObjectId('abc') },
{ data: { $elemMatch: { type: a, value: DBRef(...)}}},
{ data: { $elemMatch: { type: b, value: "string"}}}
] } ).pretty()
So you can add any number of "conditions" using $and so you can specify that an element has to have type a and a value b, and another element type b and value c...
If you want to project only the matching elements then use aggregate with filter:
db.document.aggregate([
{$match: { $and: [
{ type:ObjectId('a') },
{ data: { $elemMatch: { Type: ObjectId("b"), value: DBRef(...)}}},
{ data: { $elemMatch: { Type: ObjectId("c"), value: "abc"}}}
] }
},
{$project: {
metadata: {
$filter: {
input: "$data",
as: "data",
cond: { $or: [
{$eq: [ "$$data.Type", ObjectId("a") ] },
{$eq: [ "$$data.Type", ObjectId("b") ] }]}
}
}
}
}
]).pretty()
This is pretty ugly so if there is a better way please post it! Thanks!
If you need to retrieve documents that have array elements matching
multiple conditions, you have to use $elemMatch query operator.
db.collection.find({
data: {
$elemMatch: {
type: "type1",
value: "value1"
}
}
})
This will output whole documents where an element matches.
To output only first matching element in array, you can combine it with $elemMatch projection operator.
db.collection.find({
data: {
$elemMatch: {
type: "type1",
value: "value1"
}
}
},
{
data: {
$elemMatch: {
type: "type1",
value: "value1"
}
}
})
Warning, don't forget to project all other fields you need outside data array.
And if you need to output all matching elements in array, then you have to use $filter in an aggregation $project stage, like this :
db.collection.aggregate([
{
$project: {
data: {
$filter: {
input: "$data",
as: "data",
cond: {
$and: [
{
$eq: [
"$$data.type",
"type1"
]
},
{
$eq: [
"$$data.value",
"value1"
]
}
]
}
}
}
}
}
])
Is it possible to write union queries in Mongo DB using 2 or more collections similar to SQL queries?
I'm using spring mongo template and in my use case, I need to fetch the data from 3-4 collections based on some conditions. Can we achieve this in a single operation?
For example, I have a field named "circuitId" which is present in all 4 collections. And I need to fetch all records from all 4 collections for which that field matches with a given value.
Doing unions in MongoDB in a 'SQL UNION' fashion is possible using aggregations along with lookups, in a single query.
Something like this:
db.getCollection("AnyCollectionThatContainsAtLeastOneDocument").aggregate(
[
{ $limit: 1 }, // Reduce the result set to a single document.
{ $project: { _id: 1 } }, // Strip all fields except the Id.
{ $project: { _id: 0 } }, // Strip the id. The document is now empty.
// Lookup all collections to union together.
{ $lookup: { from: 'collectionToUnion1', pipeline: [...], as: 'Collection1' } },
{ $lookup: { from: 'collectionToUnion2', pipeline: [...], as: 'Collection2' } },
{ $lookup: { from: 'collectionToUnion3', pipeline: [...], as: 'Collection3' } },
// Merge the collections together.
{
$project:
{
Union: { $concatArrays: ["$Collection1", "$Collection2", "$Collection3"] }
}
},
{ $unwind: "$Union" }, // Unwind the union collection into a result set.
{ $replaceRoot: { newRoot: "$Union" } } // Replace the root to cleanup the resulting documents.
]);
Here is the explanation of how it works:
Instantiate an aggregate out of any collection of your database that has at least one document in it. If you can't guarantee any collection of your database will not be empty, you can workaround this issue by creating in your database some sort of 'dummy' collection containing a single empty document in it that will be there specifically for doing union queries.
Make the first stage of your pipeline to be { $limit: 1 }. This will strip all the documents of the collection except the first one.
Strip all the fields of the remaining document by using $project stages:
{ $project: { _id: 1 } },
{ $project: { _id: 0 } }
Your aggregate now contains a single, empty document. It's time to add lookups for each collection you want to union together. You may use the pipeline field to do some specific filtering, or leave localField and foreignField as null to match the whole collection.
{ $lookup: { from: 'collectionToUnion1', pipeline: [...], as: 'Collection1' } },
{ $lookup: { from: 'collectionToUnion2', pipeline: [...], as: 'Collection2' } },
{ $lookup: { from: 'collectionToUnion3', pipeline: [...], as: 'Collection3' } }
You now have an aggregate containing a single document that contains 3 arrays like this:
{
Collection1: [...],
Collection2: [...],
Collection3: [...]
}
You can then merge them together into a single array using a $project stage along with the $concatArrays aggregation operator:
{
"$project" :
{
"Union" : { $concatArrays: ["$Collection1", "$Collection2", "$Collection3"] }
}
}
You now have an aggregate containing a single document, into which is located an array that contains your union of collections. What remains to be done is to add an $unwind and a $replaceRoot stage to split your array into separate documents:
{ $unwind: "$Union" },
{ $replaceRoot: { newRoot: "$Union" } }
VoilĂ . You know have a result set containing the collections you wanted to union together. You can then add more stages to filter it further, sort it, apply skip() and limit(). Pretty much anything you want.
Starting Mongo 4.4, the aggregation framework provides a new $unionWith stage, performing the union of two collections (the combined pipeline results from two collections into a single result set).
Thus, in order to combine documents from 3 collections:
// > db.collection1.find()
// { "circuitId" : 12, "a" : "1" }
// { "circuitId" : 17, "a" : "2" }
// { "circuitId" : 12, "a" : "5" }
// > db.collection2.find()
// { "circuitId" : 12, "b" : "x" }
// { "circuitId" : 12, "b" : "y" }
// > db.collection3.find()
// { "circuitId" : 12, "c" : "i" }
// { "circuitId" : 32, "c" : "j" }
db.collection1.aggregate([
{ $match: { circuitId: 12 } },
{ $unionWith: { coll: "collection2", pipeline: [{ $match: { circuitId: 12 } }] } },
{ $unionWith: { coll: "collection3", pipeline: [{ $match: { circuitId: 12 } }] } }
])
// { "circuitId" : 12, "a" : "1" }
// { "circuitId" : 12, "a" : "5" }
// { "circuitId" : 12, "b" : "x" }
// { "circuitId" : 12, "b" : "y" }
// { "circuitId" : 12, "c" : "i" }
This:
First filters documents from collection1
Then includes documents from collection2 into the pipeline with the new $unionWith stage. The pipeline parameter is an optional aggregation pipeline applied on documents from the collection being merged before the merge happens.
And also includes documents from collection3 into the pipeline with the same $unionWith stage.
Unfortunately document based MongoDB doesn't support JOINS/Unions as in Relational DB engines.
One of the key design principles on MongoDB is to prevent joins using embedded documents as per your application's data fetch patterns.
Having said that, you will need to manage the logic in your application end if you really need to use the 4 collections or you may redesign your DB design as per MongoDB best practices.
For more info : https://docs.mongodb.com/master/core/data-model-design/
I have a mongoDB orders collection, the documents of which look as follows:
[{
"_id" : ObjectId("59537df80ab10c0001ba8767"),
"shipments" : {
"products" : [
{
"orderDetails" : {
"id" : ObjectId("59537df80ab10c0001ba8767")
}
},
{
"orderDetails" : {
"id" : ObjectId("59537df80ab10c0001ba8767")
}
}
]
},
}
{
"_id" : ObjectId("5953831367ae0c0001bc87e1"),
"shipments" : {
"products" : [
{
"orderDetails" : {
"id" : ObjectId("5953831367ae0c0001bc87e1")
}
}
]
},
}]
Now, from this collection, I want to filter out the elements in which, any of the values at shipments.products.orderDetails.id path is same as value at _id path.
I tried:
db.orders.aggregate([{
"$addFields": {
"same": {
"$eq": ["$shipments.products.orderDetails.id", "$_id"]
}
}
}])
to add a field same as a flag to decide whether the values are equal, but the value of same comes as false for all documents.
EDIT
What I want to do is compare the _id field the the documents with all shipments.products.orderDetails.id values in the array.
If even 1 of the shipments.products.orderDetails.ids match the value of the _id field, I want that document to be present in the final result.
PS I am using MongoDB 3.4, and have to use the aggregation pipeline.
Your current attempt fails because the notation returns an "array" in comparison with a "single value".
So instead either use $in where available, which can compare to see if one value is "in" an array:
db.orders.aggregate([
{ "$addFields": {
"same": {
"$in": [ "$_id", "$shipments.products.orderDetails.id" ]
}
}}
])
Or notate both as arrays using $setIsSubset
db.orders.aggregate([
{ "$addFields": {
"same": {
"$setIsSubset": [ "$shipments.products.orderDetails.id", ["$_id"] ]
}
}}
])
Where in that case it's doing a comparison to see if the "sets" have an "intersection" that makes _id the "subset" of the array of values.
Either case will return true when "any" of the id properties within the array entries at the specified path are a match for the _id property of the document.