I'm using a distict query with filter defined and query is quire slow on database with 73K items. The query looks like this:
db.runCommand({ "distinct": "companies", "key": "approver",
"query": { "doc_readers": { "$in": [ "ROLE_USER", "ROLE_MODUL_CUST", "ROLE_MODUL_PROJECTS" ] } } })
Query stats is here:
"stats" : {
"n" : 73394,
"nscanned" : 146788,
"nscannedObjects" : 73394,
"timems" : 292,
"cursor" : "BtreeCursor doc_readers_1"
},
It shows that it checks every item to get distinct list of approvers here . Is there a way how to create a better index to speed up things? I have a 3 similar queries on one web page so together they take 1 sec to get data.
Update 1: I have the following indexes
Only the first one is beeing used as stats shows ...
{
"v" : 1,
"key" : {
"doc_readers" : 1
},
"name" : "doc_readers_1",
"ns" : "netnotes.companies",
"sparse" : false
},
{
"v" : 1,
"key" : {
"doc_readers" : 1,
"approver" : 1
},
"name" : "doc_readers_1_schvalovatel_1",
"ns" : "netnotes.companies"
},
{
"v" : 1,
"key" : {
"approver" : 1,
"doc_readers" : 1
},
"name" : "schvalovatel_1_doc_readers_1",
"ns" : "netnotes.companies"
},
Related
I'm using mongodb version 3.6.5. I would like to do a query on a collection, and then sorting it based on date. I work on a (what I think) is a pretty large dataset, currently 195064301 data in this collection, and it's growing.
Doing the filter or the sort in separated query work perfectly
db.getCollection('logs').find({session: ObjectId("5af3baa173faa851f8b0090c")})
db.getCollection('logs').find({}).sort({date: 1})
The result is returned is less than 1 sec, but if I try to do it in a single query like so
db.getCollection('logs').find({session: ObjectId("5af3baa173faa851f8b0090c")}).sort({date: 1})
Now it take about 5minutes to return the data. I was thinking it was a index problem, but as far as I can tell, the index seems fine
> db.logs.getIndexes();
[
{
"v" : 2,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "client1.logs"
},
{
"v" : 2,
"key" : {
"session" : 1
},
"name" : "session_1",
"ns" : "client1.logs",
"background" : true
},
{
"v" : 2,
"key" : {
"date" : 1
},
"name" : "date_1",
"ns" : "client1.logs",
"background" : true
},
{
"v" : 2,
"key" : {
"user" : 1
},
"name" : "user_1",
"ns" : "client1.logs",
"background" : true
}
]
I'm still new to mongo, I try thoses request directly in the console, I also tried to use the reIndex() method, but nothing really help.
So I'm hoping there is a solution on this.
Thanks.
After some activities with my database, I lost my index. I had these indexes:
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "collection.statement"
},
{
"v" : 1,
"unique" : true,
"key" : {
"name" : 1
},
"name" : "name_1",
"ns" : "collection.statement"
}
and now I have only first one.
I've entered this command
db.collection.createIndex({
"v" : 1,
"unique" : true,
"key" :{ "name" : 1 },
"name" : "name_1",
"ns" : "collection.statement"
})
and I only get and error message that i have a bad index key pattern.
Please, help me, how to return this index? What I do wrong?
Use this:
db.collection.createIndex( { "name": 1 }, { unique: true } )
You attempt includes internal aspects of the index ("v" : 1), you just need to supply the field(s) and an order for each and the unique instruction.
More details in the docs.
One mongodb collection
{
"_id" : ObjectId("574bbae4d009b5364abaebe5"),
"cityid" : 406,
"location" : {
"type" : "Point",
"coordinates" : [
118.602355,
24.89083
]
},
"shopid" : "a"
}
with about 50, 000 rows;
and indexes:
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"name" : "_id_",
"ns" : "pingan-test.shop_actinfo_collection_0530"
},
{
"v" : 1,
"key" : {
"location" : "2dsphere"
},
"name" : "location_2dsphere",
"ns" : "pingan-test.shop_actinfo_collection_0530",
"2dsphereIndexVersion" : 3
},
{
"v" : 1,
"key" : {
"shopid" : 1,
"cityid" : 1
},
"name" : "shopid_1_cityid_1",
"ns" : "pingan-test.shop_actinfo_collection_0530"
}
]
I query this collection like:
body = {'cityid': 2, 'location': {'$near': {'$geometry': {'type': 'Point', 'coordinates': [122.0, 31.0]}}}, 'shopid': {'$in': ['a','b']}}
results = collection.find(body, {'shopid': 1, '_id':0},).batch_size(20).limit(20)
shops = list(results)
The question is that it run about 400ms. But it just take 30ms if we don't care about location.
why and how to fix? please.
You have an index on shopid and cityid, but you search for cityid. Since the index is ordered by shopid first it cannot be used to search by cityid. If you change the index to cityid: 1, shopid: 1, then you will see a performance improvement because your query will be able to search using the index.
after all, i got it.
I just create a index to cityid: 1, shopid: 1, "location" : "2dsphere"
, and then, world peace。
and thanks #tiramisu again.
i have a large collection of "messages" with 'to', 'from', 'type', and 'visible_to' fields that I want to query against with a fairly complex query that pulls only the messages to/from a particular user of a particular set of types that are visible to that user. Here is an actual example:
{
"$and": [
{
"$and": [
{
"$or": [
{
"to": "52f65f592f1d88ebcb00004f"
},
{
"from": "52f65f592f1d88ebcb00004f"
}
]
},
{
"$or": [
{
"type": "command"
},
{
"type": "image"
}
]
}
]
},
{
"$or": [
{
"public": true
},
{
"visible_to": "52f65f592f1d88ebcb00004f"
}
]
}
]
}
With indexes:
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"expires" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "expires_1",
"background" : true,
"safe" : null
},
{
"v" : 1,
"key" : {
"from" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "from_1",
"background" : true,
"safe" : null
},
{
"v" : 1,
"key" : {
"type" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "type_1",
"background" : true,
"safe" : null
},
{
"v" : 1,
"key" : {
"ts" : 1,
"type" : -1
},
"ns" : "n2-mongodb.messages",
"name" : "ts_1_type_-1",
"background" : true,
"safe" : null
},
{
"v" : 1,
"key" : {
"to" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "to_1",
"background" : true,
"safe" : null
},
{
"v" : 1,
"key" : {
"visible_to" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "visible_to_1",
"background" : true,
"safe" : null
},
{
"v" : 1,
"key" : {
"public" : 1,
"visible_to" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "public_1_visible_to_1"
},
{
"v" : 1,
"key" : {
"to" : 1,
"from" : 1
},
"ns" : "n2-mongodb.messages",
"name" : "to_1_from_1"
}
]
And here is the explain(true) output from our MongoDB 2.2.2 instance, which looks like a full scan:
{
"cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 0,
"nscannedObjects" : 35702,
"nscanned" : 35702,
"nscannedObjectsAllPlans" : 35702,
"nscannedAllPlans" : 35702,
"scanAndOrder" : false,
"indexOnly" : false,
"nYields" : 1,
"nChunkSkips" : 0,
"millis" : 85,
"indexBounds" : {
},
"allPlans" : [
{
"cursor" : "BasicCursor",
"n" : 0,
"nscannedObjects" : 35702,
"nscanned" : 35702,
"indexBounds" : {
}
}
],
"server" : "XXXXXXXX"
}
Looking at the explain output, MongoDB is not using any indexes for this - is there a way to get it to use at least the compound index {to: 1, from: 1} to dramatically narrow the search space? Or is there a better way to optimize this query? Or is MongoDB wholly unsuited for a query like this?
To force the MongoDB query optimizer to adopt a specific approach, you can use the $hint operator.
From the docs,
The $hint operator forces the query optimizer to use a specific index to fulfill the query. Specify the index either by the index name or by document.
The query optimizer in MongoDB 2.6 will include support for applying indexes to complex queries.
I am searching
db.runCommand({
geoNear: "users",
near:[56,11],
maxDistance:100/6387,
distanceMultiplier:6387,
query: { "city": /^bos/}
})
So here i am trying to get results with distances(priority) & city name that starts with bos
as we make LIKE query call in mysql so
i am not able to get results.
Please tell me how can i do that?
Adding the "query" parameter to the command should give you the results that you are looking for.
One possible reason that your query might not be working is that regular expressions are case-sensitive unless specified as case-insensitive with the 'i' flag.
For example:
> db.places.save({_id:1, city:"CityA", loc:[56.01,11.01]})
> db.places.save({_id:2, city:"CityB", loc:[56.02,11.02]})
> db.places.ensureIndex({loc:"2d"})
> db.runCommand({ geoNear: "places", near:[56,11], maxDistance:1, distanceMultiplier:1, query: { "city": /^city/} })
{
"ns" : "test.places",
"near" : "1100100000111111111100110000110000111111111100110000",
"results" : [ ],
"stats" : {
"time" : 0,
"btreelocs" : 0,
"nscanned" : 2,
"objectsLoaded" : 2,
"avgDistance" : NaN,
"maxDistance" : 0
},
"ok" : 1
}
Nothing is returned. Now the command is rerun, with the case-insensitive flag in the regex part of the query:
> db.runCommand({ geoNear: "places", near:[56,11], maxDistance:1, distanceMultiplier:1, query: { "city": /^city/i} })
{
"ns" : "test.places",
"near" : "1100100000111111111100110000110000111111111100110000",
"results" : [
{
"dis" : 0.014142135623729393,
"obj" : {
"_id" : 1,
"city" : "CityA",
"loc" : [
56.01,
11.01
]
}
},
{
"dis" : 0.02828427124746381,
"obj" : {
"_id" : 2,
"city" : "CityB",
"loc" : [
56.02,
11.02
]
}
}
],
"stats" : {
"time" : 0,
"btreelocs" : 0,
"nscanned" : 2,
"objectsLoaded" : 2,
"avgDistance" : 0.0212132034355966,
"maxDistance" : 0.028292673810819097
},
"ok" : 1
}
>
Both locations are returned. The documentation on using regular expression with Mongo queries may be found in the "Regular Expressions" section of the "Advanced Queries" page:
http://www.mongodb.org/display/DOCS/Advanced+Queries#AdvancedQueries-RegularExpressions
Hopefully the above will resolve your issue and allow you to perform the correct query.