MongoRestore Create Index Phase Uses 100% resources and locks up database - mongodb

Question

I'm using MongoDB. I have a table with 7M records and a weighted text search index.
When i do a MongoRestore, the create index phase of the restore uses 100% of my database's resources. MongoDB unresponsive to anything until it is done. My db is locked to any incoming connections. In fact it stops reporting any progress of the index creation to my output at that point, and my mongodb client starts getting request timeout errors. I can still tail into the server side mongodb logs to check the progress of the index creation.
I need the database to be responsive while this process is happening. It works just fine for all my other tables, which are a bit smaller. The next largest table, which works great, and still uses a weighted text search index is around 3M records.
What do i do?! Thanks.

Answer 1

I haven't tried this, but it seems that indexes created with { background: true } are dumped with this property by mongodump. This property will be passed to mongorestore during the index creation phase.
Maybe you could recreate some strategical indexes with the background option, and then dump the database. Then, the restore processes should put less strain on the server, and finish faster. Read and write operations should be allowed while MondoDB rebuilds the backgrounded indexes.
Notice that background index builds take longer to complete and result in a larger index. Also, this will not work with secondary replica set members, since background index creation operations will be foregrounded on them.
http://docs.mongodb.org/manual/tutorial/build-indexes-in-the-background/
http://docs.mongodb.org/manual/tutorial/build-indexes-on-replica-sets/
HTH.

Answer 2

I ran into similar issue(s):
Mongo restore took up so much resources that other database operations would simply time out or take on the order of a minute to complete (the restore is in essence a denial of service attack on the DB).
Mongo restore index phase completely blocked the database.
I found that to limit the bandwidth for the restore issue 1 was solved. I used the linux tc command line tool to achieve this. Tweaking the rate and burst from very low until other database operations started to be affected and then scaling it back a bit. The command looked as follows:
sudo tc qdisc change dev enp3s0 root tbf rate 30000kbit burst 40000kbit latency 5ms
To solve issue 2 I found this link which suggests you either:
update the *.metadata.json files in the dump directory to add background:true if not present.
use mongorestore's --noIndexRestore option to avoid accidentally building any indexes in the foreground, and then create the indexes with background:true after mongorestore finishes restoring the data.
Of course all of this is an issue because MongoDB best practices are not followed, which are to have the operational database always work in some form of replica set. If replication is present then one have many more options available such as (oversimplified) taking one out of the larger replication set, restore to it, and then move it back into the replication set.