What's the best Mongo index strategy that includes a date range - mongodb

I have the following schema:
{
a: string;
b: date;
c: number;
}
My query is
find({
a: 'some value',
b: {
$gte: new Date('some date')
}
})
.sort({
c: -1
});
I have an index that is:
{ a: 1, b: 1, c: 1 }
But it's not using this index.
I have several other indexes, and when analyzing my explain(), it shows it's employing multiple other indexes to accomplish my query.
I believe since my "b" query is a date range, that's not considered an equality condition, so maybe that index won't work?
Should I have two indexes:
{ a: 1, c: 1} and separately { b: 1 }

Dates tend to be much more selective than other fields, so when you have an index that looks like {dateField: 1, otherField: 1}, the selectivity of the dateField means that otherField will be useless unless you have multiple items that share the same date.
Depending on what your data distribution actually looks like, you might consider {otherField: 1, dateField: 1} (which means that mongo can go through in sorted order to check whether the docs match your date query). In general, putting your sort field before any fields used in a range query is a good idea.
Mlab's indexing docs are the best resource I've seen on index usage, and they recommend:
A good rule of thumb for queries with sort is to order the indexed fields in this order:
First, the field(s) on which you will query for exact values
Second, one small $in array
Third, the field(s) on which you will sort in the same order and specification as the sort itself (sorting on multiple fields)
Finally, the field(s) on which you will query for a range of values in the order of most selective to least selective (see range operators below)

Related

Mongo Db ESR rule for multiple index keys in a compound index

Collection: appointments
Schema:
{
_id: ObjectId();
userId: string;
calType: string;
status: string;
appointment_start_date_time: string; //UTC ISO string
appointment_end_date_time: string; //UTC ISO string
}
Example:
{
_id: ObjectId('6332b21960f8083d24f3140b')
userId: "6272ccb3-4050-429c-b427-eb104f340962"
calType: "MY Personal Cal"
status: "CONFIRMED"
appointment_start_date_time: "2022-07-08T03:30:00.000Z"
appointment_end_date_time: "2022-07-08T04:00:00.000Z"
}
I want to create a compound index on userId, calType, status, appointment_start_date_time
Based on Mongo Db's ESR rule I would like to determine the arrangement of my keys.
The documentation conveniently gives an example of 3 keys in compound index where the first key is for equality, second for sort and third for range. But in my case I have more than 3 keys.
I would like to know how would the index keys be arranged for a more efficient compound index. In my case userId, calType, status will be used for equality based match whereas appointment_start_date_time will be used for sorting.
Potential queries which I will be making on this collection will be:
All appointments where userId = x, calType = y, status = z sort by appointment_start_date_time ASC
All appointments where userId = x, status = z
All appointments where calType = y, status = z
All appointments where userId = x sort by appointment_start_date_time ASC or DSC
What is the standard when we have multiple keys for equality and one for sorting/range?
None of your sample queries use a ranged filter. Assuming none of these fields contain arrays, applying the ESR rule:
Queries 1 and 2 could be optimally served by an index on
{userId:1, status:1, calType:1, appointment_start_date_time:1}
Query 3 would be best server by this index:
{calType:1, status:1}
Query 4 would be best served by:
{userId:1, appointment_start_date_time:1}
In these optimal cases, the MongoDB server could seek to the first matching index key, scan to the last key in a single pass, and encounter the documents in already sorted order.
It may also be possible to get acceptable performance for queries 1,2, and 4 using the index:
{userId:1, appointment_start_date_time:1, status:1, calType:1}
Using this index, query 4 would still be optimal, but query 1 and 2 would require and additional index seek for each status/calType pair. This would be somewhat less performant than the optimal case, but would still be better than an in-memory sort.

Mongo - How can a narrower query be slower than a generic one?

This is executed immediately:
db.mycollection.find({ strField: 'AAA'}).count()
And this takes a lot to finish:
db.mycollection.find({ strField: 'AAA', dateTimeField: { $exists: true }}).count()
This is how I created my index:
db.mycollection.createIndex({strField: 1, dateTimeField: 1}, { sparse: true })
But it doesn't work even using hint(indexName)
Why this happens and how to fix it?
The { $exists: true } query predicate is problematic, especially if there are documents in the collection for which that field does not exist.
When MongoDB creates an index entry for a document, it collects all of the field values according to the index spec, and concatenates them.
If a field is not present in the document, the index stores null in that field's position.
If the field is explicitly set to null, it also stores null in that field's position.
This means that these 2 documents will have identical entries in the index:
{ strField: 'AAA', dateTimeField: null}
{ strField: 'AAA'}
Note that even with the index being sparse, both documents will be indexed since at least one of the indexes fields exists in each document.
When testing {dateTimeFied:{$exists:true}}, the first document will match, while the second will not.
When processing a count query using an index, if the query can be satisfied by scanning a single range of the index, the query executor can use a count_scan stage, and get the correct result without loading a single document from disk.
Because the executor cannot definitively tell from the index whether or not the field exists, it cannot use a count_scan, and must instead use an ordinary ixscan followed by a fetch stage, and load all of the matching documents from disk in order to arrive at the correct count.
In the case of the first query, the executor would have been able to use a count_scan, while the second would have had to examine all of the documents. You should be able to see this by running explain with the executionStats option on each query.
One way to avoid this pitfall is to take advantage of the fact that MongoDB query operators are type-sensitive. This means that this query will match any document where dateTimeField is greater than epoch 0, and a timestamp:
db.mycollection.find({ strField: 'AAA', dateTimeField: { $gte: new ISODate("1970-01-01T00:00:00Z") }}).count()
This will allow the query executor to count all of the documents that have the matching string and contain a date, but will exclude documents that contain a dateTimeField with a numeric or string value.

Compound Indexes Order in Mongo

Let's say I have the following document schema:
{
_id: ObjectId(...),
name: "Kevin",
weight: 500,
hobby: "scala",
favoriteFood : "chicken",
pet: "parrot",
favoriteMovie : "Diehard"
}
If I create a compound index on name-weight, I will be able to specify a strict parameter (name == "Kevin"), and a range on weight (between 50 and 200). However, I would not be able to do the reverse: specify a weight and give a "range" of names.
Of course compound index order matters where a range query is involved.
If only exact queries will be used (example: name == "Kevin", weight == 100, hobby == "C++"), then does the order actually matter for compound indexes?
When you have an exact query, the order should not matter. But when you want to be sure, the .explain() method on database cursors is your friend. It can tell you which indexes are used and how they are used when you perform a query in the mongo shell.
Important fields of the document returned by explain are:
indexOnly: when it's true, the query was completely covered by the index
n and nScanned: The first one tells you the number of found documents, the second how many documents had to be examined because the indexes couldn't sort them out. The latter shouldn't be notably higher than the first.
millis: number of milliseconds the query took to perform

MongoDB Covered Query For Two Fields Without Compound Index

Can you perform a MongoDB covered query for two fields, for example
db.collection.find( { _id: 1, a: 2 } )
without having a compound index such as
db.collection.ensureIndex( { _id: 1, a: 1 } )
but instead having only one index for _id (you get that by default) and another index for field "a", as in
db.collection.ensureIndex( { a: 1 } )
In other words, I'd like to know if in order to perform a covered query for two fields I need a compound index vs. needing only two single (i.e., not compound) indexes, one for each field.
Queries only use one index.
Your example shows _id as one of the elements of your index? _id Needs to be unique in a collection, so it wouldn't make sense to make a compound index of _id and something else.
If you instead had:
db.collection.ensureIndex( { a: 1, b: 1 })
You could then use the a index as needed, independently, or as a compound index with b.

Does order of indexes matter in MongoDB?

Is tip #25 in Tips and Tricks for MongoDB Developers correct?
It says that this query:
collection.find({"x" : criteria, "y" : criteria, "z" : criteria})
can be optimized with
collection.ensureIndex({"y" : 1, "z" : 1, "x" : 1})
I think it's false because for this to work, x should be in front. I thought the order of indexes matter.
So where did I go wrong?
The order of the fields in the index only matters if the query doesn't include all of the fields in the index. This query is referencing all three fields so the order of the fields in the index doesn't matter.
See more details in the docs on compound indexes.
The order of the fields in the find query object is not relevant.
For beginners who wants to understand it better
Mongodb says The index contains references to documents sorted first by the values of the item field and, within each value of the item field, sorted by values of the stock field." What does this mean ????
let's create a compound index on fields a, b, c, and d in ascescending order(1)
Model.createIndex({ a: 1, b: 1, c: 1, d: 1 });
I visualize it as:
at level-1, list of references sorted in a specified order(1) based on the value of the first index field(a)
at level-2, each reference at level-1, holds another set of references from thier location in a specified order(1) based on the value of the second field in the chain(b).
at level-3, each reference at level-2, holds another set of references from thier location in a specified order(1) based on the value of the second field in the chain(c).
at level-4, each reference at level-3, holds another set of references from thier location in a specified order(1) based on the value of the second field in the chain(d).
This chain forms a tree structure thus is chosen to store in B-TREE data structure.
I would love to call this storage system a compound-index-chain in this context.
Normally we build indices to perform two types of operations 1. Query Operation like find() and 2. Non-query operation like Sort()
Now you created compound index on { a: 1, b: 1, c: 1, d: 1 }. But only index creation is not enought. It becomes inefficient and sometimes useless if you don't structure your database operatons(find and sort) in a way that use those indexes.
Let's dig deeper into what kinds of query supports what kind of index ?
find():
The following prefixes of the compound index supports also indexed find() query operation on fields
{a:1},
{a:1, b:1},
{a:1, b:1, c:1}
// Index prefixes are the beginning subsets of indexed fields
#JohnnyHK already said "The order of the fields in the find query object is not relevant."
The fields could be in ANY ORDER like {b:1, a1} instead of {a1:, b:1}. Index will still be utilized as long as it is find() operation being operated on compound index or the prefix of the compound index.
However the performance of the query will not be same(may degrade) even though the find() query is using the same index and index is being utilized if the order of the the fields in find() is not highly selective than other subsequent fields.
Meaning, if the first field in a query say find({a: 'red', b: tshrt}), has HIGH SELECTIVITY, the query will be less efficient than find({a: 'tshirt', b: 'red'}) as this query hs LOWER SELECTIVITY even though both queries are using one index {a:1, b:1}.
However the HIGHLY SELECTIVE query will perform better than not having any index at all.
I think #Sushil tried to touch this topic.
In case if you are still wondering, Query selectivity refers to how well the query predicate excludes or filters out documents in a collection. Query selectivity can determine whether or not queries can use indexes effectively or even use indexes at all.
Now Let's come to the prefixes of compound indexs
Note:find() behaves differently on this {a:1, c:1} prefix of the compound index {a:1, b:1, c:1, d1} than rest of its prefixes?
In this case, The find() operation will not be able to utilize our compound index efficiently.
What happens is a:1 field index will only be able to support the find query. index on c:1 field will not be used at all because compound-index-chain has been broken in between due to the absence of b:1 index field in the prefix.
So if find() query operates on a and c field together, for field a:1 IXSCAN( i.e use of index on a) and field c COLLSCAN(i.e no use of index) will be used. Meaining the query will be slower than having separate compound index on {a:1,c:1} but faster than not having any index at all.
Conclusion is Index fields are parsed in order; if a query omits a particular index prefix, it is unable to make use of any index fields that follow that prefix.
2. Sort():
For non-query-operation(i.e Sort), the subsets of the compount index must in the same order of the index as well as must also be in the either same or oposite direction of the direction specified for each field while creating the compound index.
Let's see how the our compound index { a: 1, b: 1, c: 1, d: 1 } with ascescending direction behave with sort() operation:
Let's look at the direction of the indexed fields in sorting.
As we know on single field index on {a:1} can support sort on {a:1} same-direction and {a:-1} reverse-direction,
Compound indexes follow the same rules while sorting.
{a:1, b:1, c:1, d:1} // in same-direction as of our compound index
{a:-1, b:-1, c:-1, d:-1 } // in reverse-direction of our compound index
// But these field have neither same-direction nor reverse-direction but is ARBITARY/MIXED. Thus
// Index will be discarded while performing sorting with these fields and directions
{a:-1:, b:1, c:1, d:1}
Another example would be compound index on {a:1, b:-1} can support indexed sorting on {a:1,b:-1} (same-direction) and on {a:-1,b:1}(reverse-direction) BUT NOT support {a:-1, b:-1}.
Now let's look at the order of the indexed fields in sorting
OPTIMUM SORTING:
When a Sort operation using the compound index or using the prefix of the compound index, examining the result set in the memory(RAM) is not needed. Such sorting operation is solely sattisfied by the fields available in the index, gives optimum performance in sorting operation.
For instance:
// compound index
{a:1, b:1, c:1, d:1}
// prefix of the compound index
{a:1},
{a:1, b:1},
{a:1, b:1, c:1}
Compound-Index-Chain-Break:
When a sort operation is partially covered by the compound index, may require to examine the non-indexed matched result set in the memory.
Model.find({ a: 2 }).sort({ c: 1 }); // will not use index for sorting using field c. But will be used for finding
Model.find({ a: { $gt: 2 } }).sort({ c: 1 }); // will not use index for sorting But will be used for finding
// because compound-index-chain-break due to absence of field b of the prefix {a:1, b:1, c:1} of our compound index {a:1, b:1, c:1, d:1}
Sort on Non-prefix Subset:
When prefix keys of the index appear in both the query predicate(i.e find()) and the sort(), that index fields which precedes(or overlap) the sort subset MUST have the equality conditions($eq,$gte,$lte) in the query. So
A compound index can support indexed query on the its index prefixes as well.
Model.find({ c: 5 }).sort({ c: 1 }); // will not use index at all because it does not belongs to any of the prefix of our compound index
Model.find({ b: 3, a: 4 }).sort({ c: 1 }); // will use the index for both finding and sorting as it belongs to one our index prexfix ie. {a:1, b:1, c:1}
Model.find({ a: 4 }).sort({ a: 1, b: 1 }); // will use index for finding but not use index for sorting because a field is overlapped.
Model.find({ a: { $gt: 4 } }).sort({ a: 1, b: 1 }); // will use index for both finding and sorting because overlapped field (a) in the predicate uses equality operator and it belongs to the prefix {a:1, b:1}
Model.find({ a: 5, b: 3 }).sort({ b: 1 }); // will not use index for sorting
Model.find({ a: 5, b: { $lt: 3 } }).sort({ b: 1 }); // will use index for both finding and sorting
Hope this helps somebody
The books states the below scenario
You have 3 queries to run:
Collection.find({x:criteria, y:criteria,z:criteria})
Collection.find({z:criteria, y:criteria,w:criteria})
Collection.find({y:criteria, w:criteria })
To use
collection.ensureIndex({y:1,z:1,x:1})
it is considering the occurrence and as occurrence of y is more, it want all the queries to hit y followed by z and lastly as you will be running the 1st query a thousand times more than the other two hence including x, if this was not the case and you run all 3 queries equally then the suggestion is
Collection.ensureIndex({y:1,w:1,z:1}) .
Moreover as per the MongoDB documentation “The order of fields in a compound index is very important.” But in the above case scenario the use case is different. It is trying to optimize all the use case queries with one index.