MongoDB balancer is not starting - mongodb

sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5dfa6c3cb121a735f9ad8f6e")
}
shards:
{ "_id" : "s0", "host" : "s0/localhost:37017,localhost:37018,localhost:37019", "state" : 1 }
{ "_id" : "s1", "host" : "s1/localhost:47017,localhost:47018,localhost:47019", "state" : 1 }
{ "_id" : "s2", "host" : "s2/localhost:57017,localhost:57018,localhost:57019", "state" : 1 }
active mongoses:
"4.2.1" : 1
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Balancer active window is set between 00:00 and 23:59 server local time
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "apple", "primary" : "s2", "partitioned" : true, "version" : { "uuid" : UUID("20431f1a-ddb1-4fac-887e-c4c5db01b211"), "lastMod" : 1 } }
apple.user
shard key: { "userId" : 1 }
unique: false
balancing: true
chunks:
undefined undefined
too many chunks to print, use verbose if you want to force print
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
undefined undefined
too many chunks to print, use verbose if you want to force print
After starting balancing using sh.startBalancer() when I see the status the balancer running status is still false.
Is there anything need to configure while creating shard?

After starting balancing using sh.startBalancer() when I see the
status the balancer running status is still false.
Balancer is a process which is responsible for evenly distributing chunks across a sharded cluster. It is an automatic process. By default, the balancer is enabled. It runs on the primary of the config server replica-set (mongos in 3.4 version or earlier).
The balancer runs only when needed. The balancer process checks the chunk distribution across the cluster and looks for certain migration thresholds. It identifies which shard has too many chunks in the cluster. If it detects an imbalance it starts a Balancer Round. It moves the chunks across shards in the cluster in an attempt to achieve an even data distribution.
From the sh.status output in the post, the balancer is enabled and not running.
balancer:
Currently enabled: yes
Currently running: no
NOTE: The balancer will run automatically when even chunk distribution is needed.
You can manually start and stop the balancer any time; the commands sh.startBalancer() enable the balancer and sh.stopBalancer() disables the balancer temporarily when needed.
sh.getBalancerState() tells if the balancer is enabled and or not. sh.enableBalancing() does not start balancing. Rather, it allows balancing of a collection the next time the balancer runs.
Reference: See Sharded Cluster Balancer.

Related

Is there a primary shard in DBs in which sh.enableSharding() has not been yet executed?

MongoDB sharding cluster uses a "primary shard" to hold collection data in DBs in which sharding has been enabled (with sh.enableSharding()) but the collection itself has not been yet enabled (with sh.shardCollection()). The mongos process choses automatically the primary shard, except if the user state it explicitly as parameter of sh.enableSharding()
However, what happens in DBs where sh.enableSharding() has not been executed yet? Is there some "global primary" for these cases? How can I know which one it is? sh.status() doesn't show information about it...
I'm using MongoDB 4.2 version.
Thanks!
The documentation says:
The mongos selects the primary shard when creating a new database by picking the shard in the cluster that has the least amount of data.
If enableSharding is called on a database which already exists, the above quote would define the location of the database prior to sharding being enabled on it.
sh.status() shows where the database is stored:
MongoDB Enterprise mongos> use foo
switched to db foo
MongoDB Enterprise mongos> db.foo.insert({a:1})
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise mongos> sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5eade78756d7ba8d40fc4317")
}
shards:
{ "_id" : "shard01", "host" : "shard01/localhost:14442,localhost:14443", "state" : 1 }
{ "_id" : "shard02", "host" : "shard02/localhost:14444,localhost:14445", "state" : 1 }
active mongoses:
"4.3.6" : 2
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "config", "primary" : "config", "partitioned" : true }
{ "_id" : "foo", "primary" : "shard02", "partitioned" : false, "version" : { "uuid" : UUID("ff618243-f4b9-4607-8f79-3075d14d737d"), "lastMod" : 1 } }
{ "_id" : "test", "primary" : "shard01", "partitioned" : false, "version" : { "uuid" : UUID("4d76cf84-4697-4e8c-82f8-a0cfad87be80"), "lastMod" : 1 } }
foo is not partitioned and stored in shard02.
If enableSharding is called on a database which doesn't yet exist, the database is created and, the primary shard is specified, the specified shard is used as the primary shard. Test code here.

mongos returns empty collection when queried from different applications

We use sharded MongoDB configuration with 2 shards. There are several front applications, which retrieve data via mongos service (installed on each application server).
Problem is that at some point running the same query from Application server #1 will return empty collection, while running the same query from Application server #3 will return correct results.
The problem persists on Application server #1 until flush command is run. As documentation states:
"You should only need to run flushRouterConfig after movePrimary has
been run or after manually clearing the jumbo chunk flag."
But neither Primary was moved nor was flag cleared.
Any ideas why could this happen?
Config file
# mongos.conf
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/
# where to write logging data.
systemLog:
destination: file
logAppend: false
logRotate: rename
path: /var/log/mongos.log
# how the process runs
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongos.pid # location of pidfile
# network interfaces
net:
port: 27017
bindIp: 127.0.0.1,94.130.167.133,192.168.0.1 # Listen to local interface only, comment to listen on all interfaces.
security:
keyFile: /var/lib/database/mongos/.keyfile
sharding:
configDB: myproject-config/192.168.0.15:27027,192.168.0.16:27017,192.168.0.17:27017
printShardingStatus()
MongoDB shell version: 3.2.8
connecting to: cron0.myproject.smapps.org:27017/test
config.locks collection empty or missing. be sure you are connected to a mongos
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("2c5bc3fd9b2385s9613b0la5")
}
shards:
{ "_id" : "myproject0", "host" : "myproject0/192.168.0.11:27017,192.168.0.12:27017", "state" : 1, "tags" : [ ] }
{ "_id" : "myproject1", "host" : "myproject1/192.168.0.13:27017,192.168.0.18:27017", "state" : 1, "tags" : [ "game" ] }
{ "_id" : "myproject2", "host" : "myproject2/192.168.0.26:27017,192.168.0.28:27017", "state" : 1, "tags" : [ "game" ] }
active mongoses:
"4.0.6" : 14
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
8 : Success
databases:
{ "_id" : "myproject", "primary" : "myproject0", "partitioned" : true, "version" : { "uuid" : BinData(4,"+YxjwrKzTuekDlKutwT1IA=="), "lastMod" : 1 } }
...

None of the hosts for the replica set could be contacted

I am having a problem with my sharded cluster.
I setup a new cluster, with 1 router, 2 replica set shards(2 nodes each), and a single 3 cluster config cluster.
I believe I setup everything correctly, created collections, added indexes, but when I go to insert or query data into the collections, I get the error:
Error: error: {
"ok" : 0,
"errmsg" : "None of the hosts for replica set configReplSet could be contacted.",
"code" : 71
}
configReplSet is my config replica set. It is accessible from the box, I was able to use a mongo shell to log into the primary of the RS.
Any help into what would cause this error would be greatly appreciated.
Here is my sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("58a761728dfc0e1090b7c592")
}
shards:
{ "_id" : "rs0", "host" : "rs0/mdbshard-b1:27017,mdbshard-b1rep:27017" }
{ "_id" : "rs1", "host" : "rs1/mdbshard-b2rep:27018,mdbshard-b2:27017" }
active mongoses:
"3.2.12" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
and my shard router config file
replication:
localPingThresholdMs: 15
sharding:
autoSplit: true
configDB: "configReplSet/mdbcfg-b1:27019,mdbcfg-b2:27019,mdbcfg-b3:27019"
chunkSize: 64
processManagement:
fork: true
systemLog:
destination: file
path: "/var/log/mongodb/mongodb.log"
logAppend: true
Please let me know if you need any other information, I would be happy to provide it.
Check if this helps Relevant Error

Check configuration of mongodb setup

I have setup mongodb for cluster environment. My config server and router is running on one machine, whereas sharding is running on three different machine. I want to know if any command available which I can run on terminal(On which configsvr and router is running) and it will display all routername,configserver(associated with it),Other sharded databases(associated with it).
To simplify it more.
Suppose I run mycommand/piece of code, it display.
router1---> configserver1----> Shardeddb1
----> Shardeddb2
-----> shardeddb3
Modifying to make it more clear.
My router1 and configserver1 are running on single machine(say ip 19.0.0.123), Shardeddb1(say ip 19.0.0.124),Shardeddb2(say ip 19.0.0.125),Shardeddb3(say ip 19.0.0.126).
I want to make Shardeddb1 as a primary and (Shardeddb2,Shardeddb3) as secondary. If I run sh.status(); it show me details but not about which database belong to which machine. So is there any script which can show me more details?
sharding version: {
"_id" : 1,
"version" : 4,
"minCompatibleVersion" : 4,
"currentVersion" : 5,
"clusterId" : ObjectId("545b632e9be3f019d6ef788f")
}
shards:
{ "_id" : "ps1", "host" : "ps1/19.0.0.123:27017","draining" : true }
{ "_id" : "ps2", "host" : "ps2/19.0.0.124:27017" }
{ "_id" : "shard0000", "host" : "19.0.0.125:27017" }
{ "_id" : "shard0001", "host" : "19.0.0.126:27017" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "test", "partitioned" : true, "primary" : "shard0000" }
{ "_id" : "demo", "partitioned" : true, "primary" : "shard0000" }
{ "_id" : "db", "partitioned" : false, "primary" : "ps1" }
{ "_id" : "mongotestDB", "partitioned" : true, "primary" : "ps1" }
mongotestDB.logcoll
shard key: { "id" : 1 }
chunks:
shard0000 4
shard0001 9
ps2 7
ps1 5
too many chunks to print, use verbose if you want to force print
Since your diagram shows otherwise:
You can either have exactly 1 or 3 config servers.
You should always have your mongos instances have the exact same string for the configdb parameter. And this string has to hold all config servers in the same order. Otherwise, you risk metadata corruption.
All configservers and mongos need to be able to connect to and be connected by all nodes in the cluster.
The easiest way to have an overview on your cluster is the free MongoDB Management Service monitoring.
Running mongos' and configservers on the same machine is possible - as long as you keep an eye on the load. If things get rough, and the configservers have delays in updating the metadata because the mongos consume all the IO, you might worsen things. If chunk splits are delayed (and they are more likely under high load), this might cause JumboChunks, which have to be split manually and - until this is done - can't be migrated. Therefor, it is a Very Bad Idea™ to have mongos instances run on the configservers, imvho. A far better solution is to have the mongos instances run on the application servers, one on each.

Mongodb sharded replica

I have 3 nodes for sharding and configserver (sharding servers run on standard port 27017 and configserver running on port 27019)
stage-mongo1-tmp, stage-mongo2-tmp, and stage-mongo3-tmp
and a query router
stage-query0-mongo
in my current setup.
Sharding is working perfect as expected.
--- Sharding Status ---
sharding version: {
"_id" : 1,
"version" : 3,
"minCompatibleVersion" : 3,
"currentVersion" : 4,
"clusterId" : ObjectId("5321a5cc8a18e5280f7c9d5a")
}
shards:
{ "_id" : "shard0000", "host" : "stage-mongo1-tmp:27017" }
{ "_id" : "shard0001", "host" : "stage-mongo2-tmp:27017" }
{ "_id" : "shard0002", "host" : "stage-mongo3-tmp:27017" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "testdb", "partitioned" : true, "primary" : "shard0000" }
testdb.testcollection
shard key: { "_id" : "hashed" }
chunks:
shard0001 28
shard0002 31
shard0000 28
too many chunks to print, use verbose if you want to force print
Now, I was enabling replica set on these nodes. I logged in to stage-mongo1-tmp, and ran
rs.initiate()
and added stage-mongo2-tmp and stage-mongo3-tmp as the replica members as
rs.add("stage-mongo2-tmp")
Log files says replication enabled and elected one primary.
rs.conf()
was showing good output
[rsBackgroundSync] replSet syncing to: stage-mongo1-tmp:27017
[rsSync] oplog sync 2 of 3
[rsSyncNotifier] replset setting oplog notifier to stage-mongo1-tmp:27017
[rsSync] replSet initial sync building indexes
[rsSync] replSet initial sync cloning indexes for : ekl_omniscient
[rsSync] build index ekl_omniscient.serviceability { _id: "hashed" }
[rsSync] build index done. scanned 571242 total records. 3.894 secs
replSet RECOVERING
replSet initial sync done
replSet SECONDARY
However, when I test the High-availability by taking one node down, mongos on the query node is returning error saying
mongos> show dbs;
Thu Mar 13 20:17:04.394 listDatabases failed:{
"code" : 11002,
"ok" : 0,
"errmsg" : "exception: socket exception [CONNECT_ERROR] for stage-mongo1-tmp:27017"} at src/mongo/shell/mongo.js:46
When I connect to one of the other node, one has automatically elected as primary. But still, my queries are returning errors.
What am I doing wrong here in replica set? Why is it not high-available? Do I need to add more servers to make it high-available? I am looking for a minimum set of servers to implement this.
Figured out. We add shard for the replica set.
sh.addShard("rs0/:port,..)
Once this is done, we need to enable sharding on db level and collection level. This would enable sharding and replica.