Can't figure out why mongo database becomes bigger after migration? - mongodb

Question

I'm new to mongodb. I have a local server and a remote server. After migrate the mongo database from the local server to a remote server using mongodump/mongorestore tools, I found out that the size of database became bigger on remote server.
Here is my sample :
on local server (Ubuntu 14.04.2 LTS, mongo 3.0.5):
> show dbs
Daily_data 7.9501953125GB
Monthly_data 0.453125GB
Weekly_data 1.953125GB
on remote server (CentOS 6.7, mongo 2.4.3):
> show dbs
Daily_data 9.94921875GB
Monthly_data 0.953125GB
Weekly_data 3.9521484375GB
I also checked the status of one collection to compare, the count is the same but the size (like indexSize, totalIndexSize, etc) has changed:
this is the status of collection on the local server:
> db.original_prices.stats()
{
"ns" : "Daily_data.original_prices",
"count" : 9430984,
"size" : 2263436160,
"avgObjSize" : 240,
"numExtents" : 21,
"storageSize" : 2897301504,
"lastExtentSize" : 756662272,
"paddingFactor" : 1,
"paddingFactorNote" : "paddingFactor is unused and unmaintained in 3.0. It remains hard coded to 1.0 for compatibility only.",
"userFlags" : 1,
"capped" : false,
"nindexes" : 2,
"indexDetails" : {
},
"totalIndexSize" : 627777808,
"indexSizes" : {
"_id_" : 275498496,
"symbol_1_dateTime_1" : 352279312
},
"ok" : 1
}
this is the status of collection on the remote server:
> db.original_prices.stats()
{
"ns" : "Daily_data.original_prices",
"count" : 9430984,
"size" : 1810748976,
"avgObjSize" : 192.00000508960676,
"storageSize" : 2370023424,
"numExtents" : 19,
"nindexes" : 2,
"lastExtentSize" : 622702592,
"paddingFactor" : 1,
"systemFlags" : 1,
"userFlags" : 0,
"totalIndexSize" : 639804704,
"indexSizes" : {
"_id_" : 305994976,
"symbol_1_dateTime_1" : 333809728
},
"ok" : 1
}
If mongodump/mongorestore is a good save way to migrate the mongo database?

Answer 1

The problem here as you seem to have already noticed is the index as is clearly shown that it is the indexSize that has grown here, and there is a perfectly logical explanation.
When running the restore the indexes are rebuilt but in a way that avoids blocking the other write operations happening in the restore operation. This is similar to the process employed in Build Indexes in the Background as described in the documentation, not exactly the same but close.
In order to get the most optimal index size it is best to first drop indexes from the target database and use the --noIndexRestore option with the mongorestore command as this will prevent index building during the data load.
Then when complete you can run a regular createIndex exluding any usage of the "background" option so the indexes are created in the foreground. The result will be that the database will be blocked from read and write during index creation, but the resulting indexes will be of a smaller size.
As for the general practice, you will note that other data sizes will in fact come out "smaller" as in the process of "rebuilding" then any slack space present in the source will not be created when the data is restored.
The data from mongodump is in a binary format and should always be used in preference to the textual format of mongoexport and related mongoimport, when of course taking data from one MongoDB instance and to use on another, since that is not the purpose of those tools.
Other alternates ae file system copies such as an LVM snapshot, which will of course restore in exactly the same state as the backup copy was made.

Answer 2

Factors that can affect the disk size of your collection include the underlying hardware, filesystem, and configuration. In your case, the prevailing factor seems to be a difference in the storage engine used on the local and remote servers: your local server is running Mongo 3.0 while the remote is running an older version. This is apparent based on the presence of the paddingFactorNote property, however you can confirm by running db.version() in both environments.
Between Mongo 2.4/2.6 and Mongo 3.0 there were several important changes to how collections are stored, not least the addition of the WiredTiger storage engine as an alternative to the default mmapv1 storage engine. There were also changes to how the mmapv1 engine (which you are using) pads documents during allocation to accommodate growth in document size.
The other major reason for the size differences comes from your use of mongorestore. During normal usage, mongo databases are not stored in a way that minimizes disk usage. However, mongorestore rebuilds the database/collection in a compact way, which is why for the collection you posted, the remote storageSize is smaller.