I have a collection that has different query params to support,as such I create indexes for every keys. I just added a few more and get an error:
pymongo.errors.OperationFailure: add index fails, too many indexes for collection key:{ foo: 1 }
and then I notice that the maximum number of indexes per collection in mongodb is just 64, Can I change this number ?
The max is built into MongoDB:
http://docs.mongodb.org/manual/reference/limits/
Number of Indexes per Collection
A single collection can have no more than 64 indexes.
Related
I have two sharded collections on 12 shards, with the same number of documents. The shard key of Collection1 is compound (two fields are used), and its document consists of 4 fields. The shard key of Collection2 two is single, and its documents consists of 5 fields.
Via db.collection.stats() command, I get the information about the indexes.
What seems strange to me, is that for the Collection1, the total size of _id index is 1342MB.
Instead, the total size of the _id index for Collection2 is 2224MB. Is this difference reasonable? I was awaiting that the total size would be more less the same because of the same number of docucments. Note that the sharding key for both collections, does not integrate the _id field.
MongoDB uses prefix compression for indexes.
This means that if sequential values in the index begin with the same series of bytes, the bytes are stored for the first value, and subsequent values contain a tag indicating the length of the prefix.
Depending on the datatype of the _id value, this could be quite a bit.
There may also be orphaned documents causing one node to have more entries in its _id index.
The relevant fields of the documents in my collection are the following:
{
point: {
type: Point,
coordinates: [15.6446464, 45.231323]
}
score: 24
}
I have a 2dsphere index on point and a "normal", descending index on score.
I want to run the following query:
db.properties.find({point: {$geoWithin: <some polygon> }}).sort({score: -1}).limit(2000)
Is there any way to make mongo use the index on point for the find part, and then the index on score for sorting?
The collection has about 700k documents, the find part can return tens of thousands of documents, each of which has up to a MB.
The current problem is that, when using the point index, the size of the returned collection is too big for sorting in memory. When using the score index, the query is too slow because of a sequential scan on coordinates.
When executing your current query, MongoDB will only use the index on point. After running the find you will have a subset of the data and therefore Mongo can no longer use the index on score. You could instead create a composite index on point and score with score indexed in descending order. Even though the first values are unique, it helps to speed up the sorting as MongoDB can use the index to sort on score rather than having to scan through the entire document (which can be up to a MB in size).
The composite index follows the general rule of thumb when indexing. In general the order of an index should be:
Fields on which you will query for exact values.
Fields on which you will sort.
Fields on which you will query for a range of values.
However, as per your comment this composite index is not very fast and this suggest that MongoDB can't load the entire index into memory. How much RAM have you allocated to MongoDB ? Is there any chance you can increase this ?
sorting 2 millions of records using mongo sort is possible or not?
From the MongoDB Documentation, it is clearly mentioned that "When the sort operation consumes more than 32 megabytes, MongoDB returns an error."
But I have a requirement to sort huge number of records. How to do it?
It's possible. The documentation states that 32MB limit is there only when MongoDB sorts data in-memory i.e. without using an index.
When the sort operation consumes more than 32 megabytes, MongoDB
returns an error. To avoid this error, either create an index to
support the sort operation or use sort() in conjunction with limit().
The specified limit must result in a number of documents that fall
within the 32 megabyte limit.
I suggest that you add an index on the field on which you want to sort with ensureIndex command:
db.coll.ensureIndex({ sortFieldName : 1});
If you're sorting on multiple fields, you will need to add an compound index on the fields your sorting on (order of the fields in index matter):
db.coll.ensureIndex({ sortFieldName1 : 1, sortFieldName2 : 1});
Is it possible to have a second Collection that is exactly the same as the first Collection, but with a limit and sort operator applied to it?
If the first Collection have 1000+ records and have new records being added, the second Collection will have all the new records, but limited to N newest records (sort by timestamp field).
The reason for doing this is to overcome a limitation in my database driver that does not have sort and limit implemented yet.
It sounds like what you want is a capped collection.
A capped collection is automatically limited to a fixed amount of records. When they are full and you add a new document, the oldest document is deleted. All records are guaranteed to be in insertion order when you query from the collection without an explicit sort. The biggest limitation is that documents in a capped collection can not be updated when that would increase their size.
Capped collections need to be created explicitely with the createCollection function. This shell command would create a capped collection limited to 1000 documents:
db.createCollection( "your_collection_name", { capped: true, size: 1000 } );
When you want to convert an already existing collection to a capped collection, you can use the convertToCapped database command:
db.runCommand({"convertToCapped": "your_existing_collection_name", size: 1000});
I'm currently replacing a big MySQL join-table with a MongoDB collection. One of the queries performed on the old MySQL table was limiting the amount of records for a certain key (exclusive join on a LIMIT ORDER BY record-set). But how to do this in MongoDB?
Many thanks in advance!
You can use sort({key:1}) to do order by (use -1 instead of 1 for descending order) and limit(N) to limit the returned result to N documents.
If instead you want to get all except the top N documents you would use:
db.collection.find({user:"X"}).sort({key:-1}).skip(1000)
This will return all the documents except the top 1000 sorted by key.