MongoDB Assertion Error - mongodb

Question

I started seeing this in my mongodb production database logs
ns:my_app_production.artists query:{ $query: {}, $orderby: { semester: 1, name: 1 } }
Wed Jul 25 19:20:59 [conn199] Assertion: 10334:Invalid BSONObj spec size: -286331154 (EEEEEEEE) first element:_id: "agelio-batle"
Wed Jul 25 19:20:59 [conn199] assertion 10334 Invalid BSONObj spec size: -286331154 (EEEEEEEE) first element:_id: "agelio-batle"
first i tried running a repair
mongo --repair
...
Wed Jul 25 22:20:39 [initandlisten] Assertion: 10334:Invalid BSONObj spec size: -286331154 (EEEEEEEE) first element:_id: "agelio-batle"
0x467eaa 0x4183ca 0x62dd82 0x643478 0x532b22 0x64d196 0x6578b7 0x65ac31 0x65cd75 0x65d6d9 0x51a419 0x6195f5 0x61b0c5 0x61bd3d 0x4914c8 0x47ad9a 0x5e2e7c 0x5e60b9 0x5e78ad 0x6346a2
[0x467eaa]
# several more what i assume memory addresses omitted
[0x6346a2]
Wed Jul 25 22:20:39 [initandlisten] assertion 10334 Invalid BSONObj spec size: -286331154 (EEEEEEEE) first element:_id: "agelio-batle" ns:my_app_production.artists query:{}
Wed Jul 25 22:20:39 [initandlisten] exception in initAndListen std::exception: nextSafe(): { $err: "Invalid BSONObj spec size: -286331154 (EEEEEEEE) first element:_id: "a...", code: 10334 }, terminating
Wed Jul 25 22:20:39 dbexit:
...
i also tried running db.repairDatabase(); from the mongo shell with the same results.
I have never seen this before. Typically with mongodb a repair fixes most problems so im not sure how to proceed or troubleshoot this. any ideas?

Answer 1

If an overall repair fails you can mongodump out the individual collections with a --repair option and attempt to isolate the issues. You can even pass queries in to filter out the corrupt data from a corrupted collection, essentially working around the bad data, but it is an incremental and often slow process. This is why it is always recommended to take backups and run in replica sets to avoid the scenarion where you are left with a potentially corrupt data set.
That said, if you have no way to restore from a backup or another replica set member, then you can try something like (with the database shut down):
mongodump --dbpath /path/to/source/data/files --repair --db <dbname> --out /path/to/repaired
If that does not work, then to skip the index read (which might be tripping you up):
mongodump --forceTableScan --dbpath /path/to/source/data/files --repair --db <dbname> --out /path/to/repaired
Thanks to Ren in the comments, the other thing you can try is dropping/rebuilding the index. The table scan option (mongodump walks _id index by default) will avoid using the indexes for the dump. So assuming you get the path and the database names right, the second option should work.