This question already has answers here:
Retrieve only the queried element in an object array in MongoDB collection
(18 answers)
Closed 3 years ago.
I have the following structure (this can't be changed, that is I have to work with):
{
"_id" : ObjectId("abc123"),
"notreallyusedfields" : "dontcare",
"data" : [
{
"value" : "value1",
"otherSomtimesInterestingFields": 1
"type" : ObjectId("asd123=type1"),
},
{
"value" : "value2",
"otherSometimesInterestingFields": 1
"type" : ObjectId("asd1234=type2"),
},
many others
]
}
So basically the fields for a schema are inside an array and they can be identified based on the type field inside 1 array element (1 schema field and it's value is 1 element in the array). For me this is pretty strange, but I'm new to NoSQL so this may be ok. (also for different data some fields may be missing and the element order in the data array is not guaranteed)
Maybe it's easier to understand like this:
Table a: type1 column | type2 column | and so on (and these are stored in the array like the above)
My question is: how can you select multiple fields with conditions? What I mean is (in SQL): SELECT * FROM table WHERE type1=value1 AND type2=value2
I can select 1 like this:
db.a.find( {"data.type":ObjectId("asd1234=type2"), "data.value":value2}).pretty()
But I don't know how could I include that type1=value1 or even more conditions. And I think this is not even good because it can match any data.value field inside the array so it doesn't mean that where the type is type2 the value has to be value2.
How would you solve this?
I was thinking of doing 1 query for 1 condition and then do another based on the result. So something like the pipeline for aggregation but as I see $match can't be used more times in an aggregation. I guess there is a way to pipeline these commands but this is pretty ugly.
What am I missing? Or because of the structure of the data I have to do these strange multiple queries?
I've also looked at $filter but the condition there also applies to any element of the array. (Or I'm doing it wrong)
Thanks!
Sorry if I'm not clear enough! Please ask and I can elaborate.
(Basically what I'm trying to do based on this structure ( Retrieve only the queried element in an object array in MongoDB collection ) is this: if shape is square then filter for blue colour, if shape is round then filter for red colour === if type is type1 value has to be value1, if type is type2 value has to be value2)
This can be done like:
db.document.find( { $and: [
{ type:ObjectId('abc') },
{ data: { $elemMatch: { type: a, value: DBRef(...)}}},
{ data: { $elemMatch: { type: b, value: "string"}}}
] } ).pretty()
So you can add any number of "conditions" using $and so you can specify that an element has to have type a and a value b, and another element type b and value c...
If you want to project only the matching elements then use aggregate with filter:
db.document.aggregate([
{$match: { $and: [
{ type:ObjectId('a') },
{ data: { $elemMatch: { Type: ObjectId("b"), value: DBRef(...)}}},
{ data: { $elemMatch: { Type: ObjectId("c"), value: "abc"}}}
] }
},
{$project: {
metadata: {
$filter: {
input: "$data",
as: "data",
cond: { $or: [
{$eq: [ "$$data.Type", ObjectId("a") ] },
{$eq: [ "$$data.Type", ObjectId("b") ] }]}
}
}
}
}
]).pretty()
This is pretty ugly so if there is a better way please post it! Thanks!
If you need to retrieve documents that have array elements matching
multiple conditions, you have to use $elemMatch query operator.
db.collection.find({
data: {
$elemMatch: {
type: "type1",
value: "value1"
}
}
})
This will output whole documents where an element matches.
To output only first matching element in array, you can combine it with $elemMatch projection operator.
db.collection.find({
data: {
$elemMatch: {
type: "type1",
value: "value1"
}
}
},
{
data: {
$elemMatch: {
type: "type1",
value: "value1"
}
}
})
Warning, don't forget to project all other fields you need outside data array.
And if you need to output all matching elements in array, then you have to use $filter in an aggregation $project stage, like this :
db.collection.aggregate([
{
$project: {
data: {
$filter: {
input: "$data",
as: "data",
cond: {
$and: [
{
$eq: [
"$$data.type",
"type1"
]
},
{
$eq: [
"$$data.value",
"value1"
]
}
]
}
}
}
}
}
])
Related
I want to determine the data type of a field that lives multiple arrays deep, at specific indexes.
For example, say I have a structure like so:
{ series: [
{
metaData: [
{ reference: "1" },
{ reference: 2 }
]
}, {
metaData: [
{ reference: 3.0 },
{ reference: [4] }
]
}
]}
I want to project the $type of reference at a known pair of indexes, for example series index 1 and metaData index 0, which in the above case be a double. Essentially, I want this as my projection:
{ 'type': { '$type' : '$series.1.metaData.0.reference' }}
And returning something like this:
{ '_id': '12345', 'type': 'double' }
However, I can see from this section of the documentation that this doesn't work, and that I need to use some combination of $elemMatch, $slice, and $.
I just can't wrap my head around how to implement these in a nested manner.
There is no straight way to access element from nested array, you can try below approach if you really want to,
$arrayElemAt to get specific element from provided array and index number, so here we have used nested $arrayElemAt operator to reach reference field's value
$type to get datatype of the value
db.collection.aggregate([
{
$project: {
type: {
$type: {
$arrayElemAt: [
{
$arrayElemAt: [
"$series.metaData.reference",
1 // <= index for "series" field
]
},
0 // <= index for "metaData" field
]
}
}
}
}
])
Playground
I have many documents, but I want to figure out how to get only documents that have ALL FIELDS non null.
Suppose I have these documents:
[
{
'a': 1,
'b': 2,
'c': 3
},
{
'a': 9,
'b': 12
},
{
'a': 5
}
]
So filtering the documents, only the first have ALL FIELDS not null. So filtering out these documents, I would get only the first. How can I do this?
So when you wanted to get only the documents which have ALL FIELDS, without specifying all of them in filter query like this : { a: {$exists : true}, b : {$exists : true}, c : {$exists : true}} then it might not be a good idea, in other way technically if you've 10s of fields in the document then it wouldn't either be a good idea to mention all of them in the query. Anyhow as you don't want to list them all - We can try this hack if it performs well, Let's say if you've a fixed schema & say that all of your documents may contain only fields a, b & c (_id is default & exceptional) but nothing apart from those try this :
If you can get count of total fields, We can check for field count which says all fields do exists, Something like below :
db.collection.aggregate([
/** add a new field which counts no.of fields in the document */
{
$addFields: { count: { $size: { $objectToArray: "$$ROOT" } } }
},
{
$match: { count: { $eq: 4 } } // we've 4 as 3 fields + _id
},
{
$project: { count: 0 }
}
])
Test : mongoplayground
Note : We're only checking for field existence but not checking for false values like null or [] or '' on fields. Also this might not work for nested fields.
Just in case if you wanted to check all fields exist in the document with their names, So if you can pass all fields names as input, then try below query :
db.collection.aggregate([
/** create a field with all keys/field names in the document */
{
$addFields: {
data: {
$let: {
vars: { data: { $objectToArray: "$$ROOT" } },
in: "$$data.k"
}
}
}
},
{
$match: { data: { $all: [ "b", "c", "a" ] } } /** List down all the field names from schema */
},
{
$project: { data: 0 }
}
])
Test : mongoplayground
Ref : aggregation-pipeline
You can try to use explain to check your queries performance.
So I have a questionnaire model:
const schema = new mongoose.Schema({
title: String,
category: String,
description: String,
requirementOption: String,
creationDate: String,
questions: [],
answers: []
})
As you can see the answers is an array. This array contains object that have this structure
{
"participantEmail": "someEmail#email.email"
"currentIndex": 14,
...
}
Now I want to get a specific questionnaire by id, but in answers array I only want specific participant email. So the answers array should have either one element or no element. But I don't want to get null result if there is no such email in the answers array.
I figure it out how to get that specific element from array with this query:
dbModel.findOne({_id: id, 'answers': {$elemMatch: {participantEmail: "someEmail#email.com"}}}, {'answers.$': 1}).exec();
And if that email exists in the answer array I will get this:
"data": {
"questionnaireForParticipant": {
"id": "5d9ca298cba039001b916c55",
"title": null,
"category": null,
"creationDate": null,
"description": null,
"questions": null,
"answers": [
{
"participantEmail": "someEmail#email.com",
....
}
}
}
But if that email is not in the answers array I will get only null. Also I would like to get the title and category and all of the other fields. But I can't seem to find a way to do this.
Since you've this condition 'answers': {$elemMatch: {participantEmail: "someEmail#email.com"}} in filter part of .findOne() - If for given _id document there are no elements in answers. participantEmail array match with input value "someEmail#email.com" then .findOne() will return null as output. So if you wanted to return document irrespective of a matching element exists in answers array or not then try below query :
db.collection.aggregate([
{
$match: { "_id": ObjectId("5a934e000102030405000000") }
},
/** addFields will re-create an existing field or will create new field if there is no field with same name */
{
$addFields: {
answers: {
$filter: { // filter will result in either [] or array with matching elements
input: "$answers",
cond: { $eq: [ "$$this.participantEmail", "someEmail#email.com" ] }
}
}
}
}
])
Test : mongoplayground
Ref : aggregation-pipeline
Note : We've used aggregation as you wanted to return either answers array with matched element or an empty array. Also you can use $project instead of $addFields to transform the output as you wanted to.
The accepted answer is correct, but if you are using mongoose like I do this is how you have to write the accepted answer query:
dbModel.aggregate([
{
$match: { "_id": mongoose.Types.ObjectId("5a934e000102030405000000") }
}]).addFields({
answers: {
$filter: {
input: "$answers",
cond: { $eq: [ "$$this.participantEmail", "someEmail#email.com" ] }
}
}
}).exec();
With this sample input document:
{
_id: 1,
title: "t-1",
category: "cat-abc",
creationDate: ISODate("2020-05-05T07:01:09.853Z"),
questions: [ ],
answers: [
{ participantEmail: "someEmail#email.email", currentIndex: 14 }
]
}
And, with this query:
EMAIL_TO_MATCH = "someEmail#email.email"
db.questionnaire.findOne(
{ _id: 1 },
{ title: 1, category: 1, answers: { $elemMatch: { participantEmail: EMAIL_TO_MATCH } } }
)
The query returns (when the answers.participantEmail matches):
{
"_id" : 1,
"title" : "t-1",
"category" : "cat-abc",
"answers" : [
{
"participantEmail" : "someEmail#email.email",
"currentIndex" : 12
}
]
}
And, when the answers.participantEmail doesn't match or if the amswers array is empty, the result is:
{ "_id" : 1, "title" : "t-1", "category" : "cat-abc" }
NOTE: The $elemMatch used in the above query is a projection operator.
I would like to use mongo projections in order to return less data to my application. I would like to know if it's possible.
Example:
user: {
id: 123,
some_list: [{x:1, y:2}, {x:3, y:4}],
other_list: [{x:5, y:2}, {x:3, y:4}]
}
Given a query for user_id = 123 and some 'projection filter' like user.some_list.x = 1 and user.other_list.x = 1 is it possible to achieve the given result?
user: {
id: 123,
some_list: [{x:1, y:2}],
other_list: []
}
The ideia is to make mongo work a little more and retrieve less data to the application. In some cases, we are discarding 80% of the elements of the collections at the application's side. So, it would be better not returning then at all.
Questions:
Is it possible?
How can I achieve this. $elemMatch doesn't seem to help me. I'm trying something with unwind, but not getting there
If it's possible, can this projection filtering benefit from a index on user.some_list.x for example? Or not at all once the user was already found by its id?
Thank you.
What you can do in MongoDB v3.0 is this:
db.collection.aggregate({
$match: {
"user.id": 123
}
}, {
$redact: {
$cond: {
if: {
$or: [ // those are the conditions for when to include a (sub-)document
"$user", // if it contains a "user" field (as is the case when we're on the top level
"$some_list", // if it contains a "some_list" field (would be the case for the "user" sub-document)
"$other_list", // the same here for the "other_list" field
{ $eq: [ "$x", 1 ] } // and lastly, when we're looking at the innermost sub-documents, we only want to include items where "x" is equal to 1
]
},
then: "$$DESCEND", // descend into sub-document
else: "$$PRUNE" // drop sub-document
}
}
})
Depending on your data setup what you could also do to simplify this query a little is to say: Include everything that does not have a "x" field or if it is present that it needs to be equal to 1 like so:
$redact: {
$cond: {
if: {
$eq: [ { "$ifNull": [ "$x", 1 ] }, 1 ] // we only want to include items where "x" is equal to 1 or where "x" does not exist
},
then: "$$DESCEND", // descend into sub-document
else: "$$PRUNE" // drop sub-document
}
}
The index you suggested won't do anything for the $redact stage. You can benefit from it, however, if you change the $match stage at the start to get rid of all documents which don't match anyway like so:
$match: {
"user.id": 123,
"user.some_list.x": 1 // this will use your index
}
Very possible.
With findOne, the query is the first argument and the projection is the second. In Node/Javascript (similar to bash):
db.collections('users').findOne( {
id = 123
}, {
other_list: 0
} )
Will return the who'll object without the other_list field. OR you could specify { some_list: 1 } as the projection and returned will be ONLY the _id and some_list
$filter is your friend here. Below produces the output you seek. Experiment with changing the $eq fields and target values to see more or less items in the array get picked up. Note how we $project the new fields (some_list and other_list) "on top of" the old ones, essentially replacing them with the filtered versions.
db.foo.aggregate([
{$match: {"user.id": 123}}
,{$project: { "user.some_list": { $filter: {
input: "$user.some_list",
as: "z",
cond: {$eq: [ "$$z.x", 1 ]}
}},
"user.other_list": { $filter: {
input: "$user.other_list",
as: "z",
cond: {$eq: [ "$$z.x", 1 ]}
}}
}}
]);
I am finding the MongoDB aggregation framework to be extremely powerful - it seems like a good option to flatten out an object. My schema uses a an array of sub objects in an array called materials. The number of materials is variable, but a specific field category will be unique across objects in the array. I would like to use the aggregation framework to flatten the structure and dynamically rename the fields based on the value of the category field. I could not find an easy way to accomplish this using a $project along with $cond. Is there a way?
The reason for the array of material objects is to allow simple searching:
e.g. { 'materials.name' : 'XYZ' } pulls back any document where "XYZ" is found.
E.g. of before and after document
{
"_id" : ObjectId("123456"),
"materials" : [
{
"name" : "XYZ",
"type" : "Red",
...
"category" : "A"
},
{
"name" : "ZYX",
"type" : "Blue",
...
"category" : "B"
}]
}
to
{
"material_A_name" : "XYZ",
"material_A_type" : "Red",
...
"material_B_name" : "ZYX",
"material_B_type" : "Blue",
...
}
There is a request for something like this in jira https://jira.mongodb.org/browse/SERVER-5947 - vote it up if you would like to have this feature.
Meanwhile, there is a work-around if you know up front what the possible values of the keys will be (i.e. all unique values of "category") and I have some sample code on it on my blog.
This would be useful from MongoDB v4.4,
$map to iterate loop of materials array
$map to iterate loop of name and type fields after converting to array using $objectToArray, concat your key fields requirement as per fields and value using $concat,
back to first $map convert returned result from second $map from array to object using $arrayToObject
$unwind deconstruct materials array
$group by null and merge materials object to one object
$replaceRoot to replace object in root
db.collection.aggregate([
{
$project: {
materials: {
$map: {
input: "$materials",
as: "m",
in: {
$arrayToObject: [
{
$map: {
input: {
$objectToArray: {
name: "$$m.name",
type: "$$m.type"
}
},
in: {
k: { $concat: ["material", "_", "$$m.category", "_", "$$this.k"] },
v: "$$this.v"
}
}
}
]
}
}
}
}
},
{ $unwind: "$materials" },
{
$group: {
_id: null,
materials: { $mergeObjects: "$materials" }
}
},
{ $replaceRoot: { newRoot: "$materials" } }
])
Playground