I have run into a problem where I have exceeded the allowed BSON size of 16MB and I am getting this error now whenever I try to do something on my collection.
Now my question is, how do I repair and solve the problem?
How do I check whether it is an individual document within my collection, or the collection itself exceeding the limit
How do I remove the offending document? I just keep getting this error whenever I try doing something with this collection now.
I have already tried db.repairDatabase(), but just keep getting the same error:
"errmsg" : "exception: BSONObj size: 1718558820 (0x666F2064) is invalid. Size must be between 0 and 16793600(16MB) First element: ...: ?type=32",
"code" : 10334,
"ok" : 0
Look at the size. It's obviously not a size, it's four ASCII characters. Go and find your bug.
Related
I want to get a large number (1 million) of documents by their object id, stored in a list obj_ids. So I use
docs = collection.find({'_id' : {'$in' : obj_ids}})
However, when trying to access the documents (e.g. list(docs)) I get
pymongo.errors.DocumentTooLarge: BSON document too large (19889042 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
which confuses me. As I understand this, the document size is 16 MB for a single document. But I don't think I have any document exceeding this limit:
I did not get this error message when inserting any of the documents in the first place.
This error does not show up if I chunk the ObjectIds into 2 subsets, and recombine the results later.
So if there is not some too big document in my collection, what is the error message about?
Your query {'_id' : {'$in' : obj_ids}} is the issue, that's too large, not the documents themselves.
You'll need to refactor your approach; maybe do it in batches and join the results.
I am getting the 'Sort operation used more than the maximum 33554432 bytes of RAM' error on some query. However what is even more troublesome, is that I can't even run it with .explain():
> db.collection.find({...}, {...}).limit(5000).sort({...})
Error: error: {
"ok" : 0,
"errmsg" : "errmsg: \"Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.\"",
"code" : 96,
"codeName" : "OperationFailed"
}
> db.collection.find({...}, {...}).limit(5000).sort({...}).explain()
2019-07-22T09:15:38.246+0000 E QUERY [thread1] Error: explain failed: {
"ok" : 0,
"errmsg" : "errmsg: \"Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.\"",
"code" : 96,
"codeName" : "OperationFailed"
} :
_getErrorWithCode#src/mongo/shell/utils.js:25:13
throwOrReturn#src/mongo/shell/explainable.js:31:1
constructor/this.finish#src/mongo/shell/explain_query.js:172:24
DBQuery.prototype.explain#src/mongo/shell/query.js:521:12
#(shell):1:1
Tried to recreate the index but nothing changed.
What could be happening here?
Edit: To clarify, the sort field is indexed and in the correct sort order. I was wondering why the explain fails, which seemed odd to me and I thought there might be data corruption going on here. How are we meant to diagnose a problematic query with explain if it fails on what we are trying to diagnose?
Edit 2: Upon further diagnosis I could literally pinpoint it to .limit(4308) works and .limit(4309) barfs. However there is nothing wrong with the 4309th record...
Furthermore this happens in one env and not the other that are identical expect for data.
For any time travelers from the future:
RE .explain(), seems to be just a quirk in Mongo. To see the query plan the limit must be reduced. I guess as silly as this sounds Mongo actually runs the query and then shows the query plan...
Worth noting that this is Mongo 3.4. Might have changed by now...
Our performance problem came down to having a huge subobject property (let's call it .metaData). However since we know it's problematic we didn't include it in the projection. But - it does appear in the find criteria as {metaData: {$exists: true}}. I guess mongo fetches the whole thing and keeps it in memory only to do {$exists: true} on it. That led the query to blow up the 32M memory limit eventhough the actual result requires much less memory and the sort field is indexed.
So we live to write more bugs another day...
I'm creating a index in mongo:
db.table.createIndex({"chr" : 1, "Start" : 1, "End" : 1, "Ref" : 1, "Alt" : 1})
It runs for some time and gives an error msg:
error in monogdb "errmsg" : "WiredTigerIndex::insert: key too large to index, failing
How do I fix this error?
In MongoDB, since 2.6, the total size of an index entry must be less than 1024 bytes. Documentation here
In other terms, at least one of your documents has a large value in one of the field you are trying to index.
It's not a good idea in general to index very large values like that because it creates a big index which is less efficient compared to a smaller one and it takes more space in RAM which could be put to better use on a MongoDB node.
You could use this : mongod --setParameter failIndexKeyTooLong=false.
But it doesn't look like a good idea. If you have a large text to index, you should consider using the Full Text index or you could use a Hashed index.
This can also be caused by having both a text index and a standard index for the same field. By deleting one of them you will be able to resolve this issue.
I have two collections both having more than 15 million entries. I am doing indexing on both the collections and doing find() on both . I am getting the following error
""10334:BSONObj size: 27624158 (0xDE82A501) is invalid. Size must be between 0 and 16793600(16MB) First element: _id: ObjectId('532d4a424a33b081be8a0315')".
How can I resolve this error ? I hv already done database repair but it didnt work.
We're experiencing an issue renaming a collection in MongoDB.
The collection has some fairly long index names, but this is not an issue in the general use of the collection. We can write to it and query it, and the indexes are effective (we can tell by the drop in query performance if the indexes aren't present).
When we rename the collection however, it fails with the message below:
MongoDB.Driver.MongoCommandException: Command 'renameCollection'
failed: exception: collection name length of 43 exceeds maximum length
of 32, allowing for index names (response: { "errmsg" : "exception:
collection name length of 43 exceeds maximum length of 32, allowing
for index names", "code" : 16451, "ok" : 0.0 })
For various reasons I won't go into here, renaming collections is an important part of our workflow.
If anyone knows of any workarounds for this issue, it'd be much appreciated.
Thanks.
Update:
Here's the DB/Collection/Index names:
I don't know if it's acceptable to have less readable index names, but you can specify the names yourself so they don't get so long(leaving more room for your collection names).
db.someReallyLongCollectionName.ensureIndex({
IncludeInLocationBasedSearch: 1,
TotalBranches: 1,
Categories: 1
}, { name: 'short1'});
You could even name indexes by single characters, and have some reference for what they mean somewhere else.
That second argument ensureIndex takes is for options, full docs here: http://docs.mongodb.org/manual/reference/method/db.collection.ensureIndex/