Shards are not equally balanced in cluster - mongodb

Question

I have 2 shards.
One is over standalone server and another over replicaset:
mongos> db.runCommand({listshards:1})
{
"shards" : [
{
"_id" : "shard0000",
"host" : "mongo3:10001"
},
{
"_id" : "set1",
"host" : "set1/mongo1:10001,mongo2:10001"
}
],
"ok" : 1
}
I've inserted about 30M records.
As far as I understand mongo should balance equally the data between the shards, but it does not happen:
mongos> db.stats()
{
"raw" : {
"set1/mongo1:10001,mongo2:10001" : {
"db" : "my_ginger",
"collections" : 3,
"objects" : 5308714,
"avgObjSize" : 811.9953284354742,
"dataSize" : 4310650968,
"storageSize" : 4707774464,
"numExtents" : 23,
"indexes" : 2,
"indexSize" : 421252048,
"fileSize" : 10666115072,
"nsSizeMB" : 16,
"ok" : 1
},
"mongo3:10001" : {
"db" : "my_ginger",
"collections" : 6,
"objects" : 25162626,
"avgObjSize" : 1081.6777010475776,
"dataSize" : 27217851444,
"storageSize" : 28086624096,
"numExtents" : 38,
"indexes" : 6,
"indexSize" : 1903266512,
"fileSize" : 34276900864,
"nsSizeMB" : 16,
"ok" : 1
}
},
"objects" : 30471340,
"avgObjSize" : 1034.6936633571088,
"dataSize" : 31528502412,
"storageSize" : 32794398560,
"numExtents" : 61,
"indexes" : 8,
"indexSize" : 2324518560,
"fileSize" : 44943015936,
"ok" : 1
}
What am I doing wrong?
Thanks.

Answer 1

According the sh.status() output in the comments, you have 164 chunks on shard0000 (the single host) and 85 on set1 (the replica set). There are a couple of common reasons that this kind of imbalance can happen:
You picked a bad shard key (monotonically increasing or similar)
All your data was initially on a single shard and is being rebalanced
The balancer will be continuously attempting to move chunks from the high shard to the low while at the same time moving the max-chunk around (for people that pick the aforementioned monotonically increasing keys, this helps). However, there can only be one migration at the time, so this will take some time, especially if you continue writing/reading from the shards at the same time. If things are really bad, and you did pick a poor shard key, then this may persist for some time.
If all your data was on one shard first, and then you added another, then you have a similar problem - it will take a while for the chunk count to stabilise because half the data has to be moved from the original shard (in addition to its other activities) to balance things out. The balancer will pick low range chunks to move first in general, so if these are less likely to be in memory (back to the poor shard key again), then they will have to be paged in before they can be migrated.
To check the balancer is running:
http://docs.mongodb.org/manual/reference/method/sh.setBalancerState/#sh.getBalancerState
Then, to see what it has been up to, connect to a mongos (last 10 operations):
use config
db.changelog.find().sort({$natural:-1}).limit(10).pretty()
Similarly you will see messaging in the primary logs of each shard relating to the migrations, how long they are taking etc. if you want to see their performance.