MongoDB indexing multi-input search - mongodb

I have a search function that looks like this:
The criteria from this search is passed through to MongoDB's find() method as a criteria object, e.g:
{
designer: "Designer1".
store: "Store1",
category: "Category1",
name: "Keyword",
gender: "Mens",
price: {$gte: 50}
}
I'm only just learning about indexes in MongoDB so please bear with me. I know I can create an index on each individual field, and I can also create a multi-index on several fields. For instance, for one index I could do:
db.products.ensureIndex({designer: 1, store: 1, category: 1, name: 1, gender: 1, price: 1})
The obvious issue arises if someone searches for, say, a category, but not a designer or store it won't be indexed.
I'm currently looking up these terms using an $and operator, so my question is:
How can I create an index that allows for this type of searching with flexibility? Do I have to create an index for each possible combination of these 6 terms? Or if I use $and in my search will it be enough to index each individual term and I'll get the best performance?

$and won't work as MongoDB can only use one index per query at the moment. So if you create an index on each field that you search on, MongoDB will select the best fitting index for that query pattern. You can try with explain() to see which one is selected.
Creating an index for each possible combination is probably not a good idea, as you'd need 6 * 5 * 4 * 3 * 2 * 1 indexes, which is 720 indexes... and you can only have 63 indexes. You could pick the most likely ones perhaps but that won't help a lot.
One solution could be to store your data differently, like:
{
properties: [
{ key: 'designer', value: "Designer1" },
{ key: 'store', value: "Store1" },
{ key: 'category', value: "Category1" },
{ key: 'name', value: "Keyword" },
{ key: 'gender', value: "Mens" },
{ key: 'price', value: 70 },
]
}
Then you can create one index on:
db.so.ensureIndex( { 'properties.key': 1, 'properties.value': 1 } );
And do searches like:
db.so.find( { $and: [
{ properties: { $elemMatch: { key: 'designer', value: 'Designer1' } } },
{ properties: { $elemMatch: { key: 'price', value: { $gte: 30 } } } }
] } )
db.so.find( { $and: [
{ properties: { $elemMatch: { key: 'price', value: { $gte: 45 } } } }
] } )
In both cases, the index is used, but only for the first part of the $and element right now. So do check which key type has the most values, and order your $and elements accordingly in the query.

Related

Find a value in multiple nested fields with the same name in MongoDB

In some documents I have a property with a complex structure like:
{
content: {
foo: {
id: 1,
name: 'First',
active: true
},
bar: {
id: 2,
name: 'Second',
active: false
},
baz: {
id: 3,
name: 'Third',
active: true
},
}
I'm trying to make a query that can find all documents with a given value in the field name across the different second level objects foo, bar, baz
I guess that a solution could be:
db.getCollection('mycollection').find({ $or: [
{'content.foo.name': 'First'},
{'content.bar.name': 'First'},
{'content.baz.name': 'First'}
]})
But a I want to do it dynamic, with no need to specify key names of nested fields, nether repeat the value to find in every line.
If some Regexp on field name were available , a solution could be:
db.getCollection('mycollection').find({'content.*.name': 'First'}) // Match
db.getCollection('mycollection').find({'content.*.name': 'Third'}) // Match
db.getCollection('mycollection').find({'content.*.name': 'Fourth'}) // Doesn't match
Is there any way to do it?
I would say this is a bad schema if you don't know your keys in advance. Personally I'd recommend to change this to an array structure.
Regardless what you can do is use the aggregation $objectToArray operator, then query that newly created object. Mind you this approach requires a collection scan each time you execute a query.
db.collection.aggregate([
{
$addFields: {
arr: {
"$objectToArray": "$content"
}
}
},
{
$match: {
"arr.v.name": "First"
}
},
{
$project: {
arr: 0
}
}
])
Mongo Playground
Another hacky approach you can take is potentially creating a wildcard text index and then you could execute a $text query to search for the name, obviously this comes with the text index/query limitations and might not be right for your usecase.

Mongodb querying - How to specify index traversal order?

Context
I have a users collection with an array of key-value pairs like so:
{
name: 'David',
customFields: [
{ key: 'position', value: 'manager' },
{ key: 'department', value: 'HR' }
]
},
{
name: 'Allison',
customFields: [
{ key: 'position', value: 'employee' },
{ key: 'department', value: 'IT' }
]
}
The field names in customFields are configurable by the application users, so in order for it to be indexable I store them as an array of key-value pairs. Index { 'customFields.key': 1, 'customFields.value': 1} works quite well. Problem is when I want to retrieve the results in either order (ascending or descending). Say I want to sort all users having the custom field position in either order. The below query gives me ascending order:
db.users.find({ customFields: { $elemMatch: { key: 'position' } } })
However, I couldn't figure out how to get the opposite order. I'm pretty sure with the way the indexes are structured, if I could tell mongodb to traverse the index backwards I would get what I want. The other options is to have an other 2 indexes to cover both directions, however it's more costly.
Is there a way to specify the traversal direction? If not, what are some good alternatives? Much appreciated.
EDIT
I'd like to further clarify my situation. I have tried:
db.users.find({ customFields: { $elemMatch: { key: 'position' } } })
.sort({ 'customFields.key': 1, 'customFields.value': 1 })
db.users.find({ customFields: { $elemMatch: { key: 'position' } } })
.sort({'customFields.value': 1 })
These two queries only sort the documents after they have been filtered, meaning the sorting is applied on all the custom fields they have, not on the field matching the query (position in this case). So it seems using the sort method won't be of any help.
Not using any sorting conveniently returns the documents in the correct order in the ascending case. As can be seen in the explain result:
"direction" : "forward",
"indexBounds" : {
"customFields.key" : [
"[\"position\", \"position\"]"
],
"customFields.value" : [
"[MinKey, MaxKey]"
]
}
I'm using exact match for customFields.key so I don't care about it's order. Within each value of customFields.key, the values of customFields.value are arranged in the order specified in the index, so I just take them out as it is and all is good. Now if I could do something like:
db.users.find({ customFields: { $elemMatch: { key: 'position' } } })
.direction('backwards')
It would be the equivalent of using the index { 'customFields.key': -1, 'customFields.value': -1 }. Like I said I don't care about customFields.key order, and 'customFields.value': -1 gives me exactly what I want. I've searched mongodb documentation but I couldn't find anything similar to that.
I could retrieve all the documents matching customFields.key and do the sorting myself, but it's expensive as the number of documents could get very big. Ideally all the filtering and sorting is solved by the index.
Here is the compound index you created:
db.users.createIndex( { "customFields.key": 1, "customFields.value": 1 } )
This index will allow traversal either using both fields in ascending order:
db.users.sort( { "customFields.key": 1, "customFields.value": 1 } )
Or, it can also support traversal using both keys in descending order:
db.users.sort( { "customFields.key": -1, "customFields.value": -1 } )
If you need mixed behavior, i.e. ascending on one field but descending on the other, then you would need to add a second index:
db.users.createIndex( { "customFields.key": 1, "customFields.value": -1 } )
Note that after adding this second index, all four possible combinations are now supported.

MongoDB not using wildcard nested array index

I have the following collection:
{
_id: 12345,
quizzes: [
{
_id: 111111,
questions: []
}
]
},
{
_id: 78910,
quizzes: [
{
_id: 22222
}
]
}
I want to select the documents of a certain quiz from the quizzes that do not have the questions array and want to make sure that it uses the appropriate questions index. So I use the following query:
Answer.find({ 'quizzes.0.questions': { $exists: false } }).explain('queryPlanner');
Which returns:
{
queryPlanner: {
plannerVersion: 1,
namespace: 'iquiz.answers',
indexFilterSet: false,
parsedQuery: { 'quizzes.0.questions': [Object] },
winningPlan: { stage: 'COLLSCAN', filter: [Object], direction: 'forward' },
rejectedPlans: []
}
}
The query is not using any index as seen from the output. I have tried the following indexes and none get used:
{ quizzes.$**: 1 }
{ quizzes.questions: 1 }
{ quizzes.[$**].questions: 1 }
{ quizzes: 1 }
The only 1 that actually gets used:
{ quizzes.0.questions: 1 }
However this is not really practical as I may target any quiz from the quizzes array not just the first one. Is there a certain syntax for the index in my case or this is a current limitation of mongodb? Thanks!
Indexes generally do not help to answer queries in the form of "X does not exist". See also mongodb indexes covering missing values.
To verify whether the index is used, look up some data using a positive condition.

Mongo filter documents by array of objects

I have to filter candidate documents by an array of objects.
In the documents I have the following fields:
skills = [
{ _id: 'blablabla', skill: 'Angular', level: 3 },
{ _id: 'blablabla', skill: 'React', level: 2 },
{ _id: 'blablabla', skill: 'Vue', level: 4 },
];
When I make the request I get other array of skills, for example:
skills = [
{ skill: 'React', level: 2 },
];
So I need to build a query to get the documents that contains this skill and a greater or equal level.
I try doing the following:
const conditions = {
$elemMatch: {
skill: { $in: skills.map(item => item.skill) },
level: { $gte: { $in: skills.map(item => item.level) } }
}
};
Candidate.find(conditions)...
The first one seems like works but the second one doesn't work.
Any idea?
Thank you in advance!
There are so many problems with this query...
First of all item.tech - it had to be item.skill.
Next, $gte ... $in makes very little sense. $gte means >=, greater or equal than something. If you compare numbers, the "something" must be a number. Like 3 >= 5 resolves to false, and 3 >= 1 resolves to true. 3 >= [1,2,3,4,5] makes no sense since it resolves to true to the first 3 elements, and to false to the last 2.
Finally, $elemMatch doesn't work this way. It tests each element of the array for all conditions to match. What you was trying to write was like : find a document where skills array has a subdocument with skill matching at least one of [array of skills] and level is greater than ... something. Even if the $gte condition was correct, the combination of $elementMatch and $in inside doesen't do any better than regular $in:
{
skill: { $in: skills.map(item => item.tech) },
level: { $gte: ??? }
}
If you want to find candidates with tech skills of particular level or higher, it should be $or condition for each skill-level pair:
const conditions = {$or:
skills.map(s=>(
{skill: { $elemMatch: {
skill:s.skill,
level:{ $gte:s.level }
} } }
))
};

How to $set sub-sub-array items in MongoDB

I'm developing a webapp which has a portal-ish component to it (think like multiple panels that can be drug around from column to column and added or removed). I'm using MongoDB to store this info with a format like so...
{
_id: ObjectId(...),
title: 'My Layout',
columns: [
{
order: 1,
width: 30,
panels: [
{ title: 'Panel Title', top: 100, content: '...' },
{ title: 'Panel Title', top: 250, content: '...' },
]
},
{
... multiple columns ...
}
]
}
I'm attempting to use atomic/modifier operations with update() and this is getting confusing. If I wanted to just update one property of a specific panel, how do I reference that?
update(
{ _id: ObjectId(...) },
{ $set: { columns.[???].panels.[???].top: 500 }
)
If you know the index in the array you can access the array element directly using dot notation.
update(
{ _id: ObjectId(xxxx) },
{ $set: { 'columns.0.panels.0.top' : 125}}
)
Make sure you encase the dot notated path in quotes as a string.
Edit:
To give more detail on how this could work dynamically, I'll give an example in PHP:
$action = array("columns.$colNum.panels.$panelNum" => $newValue);
Yes there is the positional operator, but it does not appear to be advanced enough to change arrays within arrays, this may change in MongoDB 1.7.0
There is an alternative you can do instead of trying to stuff this information into a nested document. Try to flatten it out. You can create a collection that has panel & column objects:
column object:
{
_id: // MongoId
type: 'column',
user: 'username',
order: 1,
width: 30,
}
panel object:
{
_id: //MongoId
type: 'panel',
user: 'username',
parentColumn: //the columns _id string
top: 125,
left: 100
}
Then you can find all columns that belong to a user by doing:
find({ type: 'column', user:'username'});
You can find all panels for a specific column by doing:
find({type: 'panel', columnOwner:'ownerID'});
Since each column and panel will have a unique ID given by MongoDB to it, you can easily query and atomically set options.
update({'_id': ObjectId('idstring')}, {$set : { 'top' : 125}});
I too had a situation like this, i followed $addToSet after $pull to replace i.e update what i needed.
update(
{ _id: ObjectId(...) },
{ $addToSet: { columns.$.panels : "new sub-Doc" }
)
And then remove the old value
update({_id:ObjectId(....)},{$pull : { columns.$.panels : {"top" : 100} }})