Better way to move MongoDB Collection to another Collection - mongodb

Question

In my web scraping project i need to move previous day scraped data from mongo_collection to mongo_his_collection
I am using this query to move data
for record in collection.find():
his_collection.insert(record)
collection.remove()
It works fine but sometimes it break when MongoDB collection contain above 10k rows
Suggest me some optimized query which will take less resources and do the same task

Answer 1

You could use a MapReduce job for this.
MapReduce allows you to specify a out-collection to store the results in.
When you hava a map function which emits each document with its own _id as key and a reduce function which returns the first (and in this case only because _id's are unique) entry of the values array, the MapReduce is essentially a copy operation from the source-collection to the out-collection.
Untested code:
db.runCommand(
{
mapReduce: "mongo_collection",
map: function(document) {
emit(document._id, document);
},
reduce: function(key, values) {
return values[0];
},
out: {
merge:"mongo_his_collection"
}
}
)

Answer 2

If both your collections are in the same database, I believe you're looking for renameCollection.
If not, you unfortunately have to do it manually, using a targeted mongodump / mongorestore command:
mongodump -d your_database -c mongo_collection
mongorestore -d your_database -c mongo_his_collection dump/your_database/mongo_collection.bson
Note that I just typed these two commands from the top of my head without actually testing them, so do make sure you check them before running them in production.
[EDIT]: sorry, I just realised that this was something you needed to do on a regular basis. In that case, mongodump / mongorestore probably isn't the best solution.
I don't see anything wrong with your solution - it would help if you edited your question to explain what you mean by "it breaks".

Answer 3

The query breaks because you are not limiting the find(). When you create a cursor on the server mongod will try to load the entire result set in memory. This will cause problems and/or fail if your collection is too large.
To avoid this use a skip/limit loop. Here is an example in Java:
long count = 0
while (true) {
MongoClient client = new MongoClient();
DBCursor = client.getDB("your_DB_name").getCollection("mongo_collection").find().sort(new BasicDBObject("$natural", 1)).skip(count).limit(100);
while (cursor.hasNext()) {
client.getDB("your_DB_name").getCollection("mongo_his_collection").insert(cursor.next());
count++;
}
}
This will work, but you would get better performance by batching the writes as well. To do that build an array of DBObjects from the cursor and write them all at once with one insert.
Also if the Collection is being altered while you are copying there is no guarantee that you will traverse all documents as some may end up getting moved if they increase in size.

Answer 4

You can try mongodump & mongorestore.

Answer 5

You can use renameCollection to do it directly. Or if on different mongods, use cloneCollection.
References:
MongoDB docs renameCollection: http://docs.mongodb.org/manual/reference/command/renameCollection/#dbcmd.renameCollection
MongoDB docs cloneCollection: http://docs.mongodb.org/manual/reference/command/cloneCollection/
Relevant blog post: http://blog.shlomoid.com/2011/08/how-to-move-mongodb-collection-between.html