I am having a problem with my sharded cluster.
I setup a new cluster, with 1 router, 2 replica set shards(2 nodes each), and a single 3 cluster config cluster.
I believe I setup everything correctly, created collections, added indexes, but when I go to insert or query data into the collections, I get the error:
Error: error: {
"ok" : 0,
"errmsg" : "None of the hosts for replica set configReplSet could be contacted.",
"code" : 71
}
configReplSet is my config replica set. It is accessible from the box, I was able to use a mongo shell to log into the primary of the RS.
Any help into what would cause this error would be greatly appreciated.
Here is my sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("58a761728dfc0e1090b7c592")
}
shards:
{ "_id" : "rs0", "host" : "rs0/mdbshard-b1:27017,mdbshard-b1rep:27017" }
{ "_id" : "rs1", "host" : "rs1/mdbshard-b2rep:27018,mdbshard-b2:27017" }
active mongoses:
"3.2.12" : 1
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
and my shard router config file
replication:
localPingThresholdMs: 15
sharding:
autoSplit: true
configDB: "configReplSet/mdbcfg-b1:27019,mdbcfg-b2:27019,mdbcfg-b3:27019"
chunkSize: 64
processManagement:
fork: true
systemLog:
destination: file
path: "/var/log/mongodb/mongodb.log"
logAppend: true
Please let me know if you need any other information, I would be happy to provide it.
Check if this helps Relevant Error
Related
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5dfa6c3cb121a735f9ad8f6e")
}
shards:
{ "_id" : "s0", "host" : "s0/localhost:37017,localhost:37018,localhost:37019", "state" : 1 }
{ "_id" : "s1", "host" : "s1/localhost:47017,localhost:47018,localhost:47019", "state" : 1 }
{ "_id" : "s2", "host" : "s2/localhost:57017,localhost:57018,localhost:57019", "state" : 1 }
active mongoses:
"4.2.1" : 1
autosplit:
Currently enabled: yes
balancer:
Currently enabled: yes
Currently running: no
Balancer active window is set between 00:00 and 23:59 server local time
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "apple", "primary" : "s2", "partitioned" : true, "version" : { "uuid" : UUID("20431f1a-ddb1-4fac-887e-c4c5db01b211"), "lastMod" : 1 } }
apple.user
shard key: { "userId" : 1 }
unique: false
balancing: true
chunks:
undefined undefined
too many chunks to print, use verbose if you want to force print
{ "_id" : "config", "primary" : "config", "partitioned" : true }
config.system.sessions
shard key: { "_id" : 1 }
unique: false
balancing: true
chunks:
undefined undefined
too many chunks to print, use verbose if you want to force print
After starting balancing using sh.startBalancer() when I see the status the balancer running status is still false.
Is there anything need to configure while creating shard?
After starting balancing using sh.startBalancer() when I see the
status the balancer running status is still false.
Balancer is a process which is responsible for evenly distributing chunks across a sharded cluster. It is an automatic process. By default, the balancer is enabled. It runs on the primary of the config server replica-set (mongos in 3.4 version or earlier).
The balancer runs only when needed. The balancer process checks the chunk distribution across the cluster and looks for certain migration thresholds. It identifies which shard has too many chunks in the cluster. If it detects an imbalance it starts a Balancer Round. It moves the chunks across shards in the cluster in an attempt to achieve an even data distribution.
From the sh.status output in the post, the balancer is enabled and not running.
balancer:
Currently enabled: yes
Currently running: no
NOTE: The balancer will run automatically when even chunk distribution is needed.
You can manually start and stop the balancer any time; the commands sh.startBalancer() enable the balancer and sh.stopBalancer() disables the balancer temporarily when needed.
sh.getBalancerState() tells if the balancer is enabled and or not. sh.enableBalancing() does not start balancing. Rather, it allows balancing of a collection the next time the balancer runs.
Reference: See Sharded Cluster Balancer.
We use sharded MongoDB configuration with 2 shards. There are several front applications, which retrieve data via mongos service (installed on each application server).
Problem is that at some point running the same query from Application server #1 will return empty collection, while running the same query from Application server #3 will return correct results.
The problem persists on Application server #1 until flush command is run. As documentation states:
"You should only need to run flushRouterConfig after movePrimary has
been run or after manually clearing the jumbo chunk flag."
But neither Primary was moved nor was flag cleared.
Any ideas why could this happen?
Config file
# mongos.conf
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/
# where to write logging data.
systemLog:
destination: file
logAppend: false
logRotate: rename
path: /var/log/mongos.log
# how the process runs
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongos.pid # location of pidfile
# network interfaces
net:
port: 27017
bindIp: 127.0.0.1,94.130.167.133,192.168.0.1 # Listen to local interface only, comment to listen on all interfaces.
security:
keyFile: /var/lib/database/mongos/.keyfile
sharding:
configDB: myproject-config/192.168.0.15:27027,192.168.0.16:27017,192.168.0.17:27017
printShardingStatus()
MongoDB shell version: 3.2.8
connecting to: cron0.myproject.smapps.org:27017/test
config.locks collection empty or missing. be sure you are connected to a mongos
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("2c5bc3fd9b2385s9613b0la5")
}
shards:
{ "_id" : "myproject0", "host" : "myproject0/192.168.0.11:27017,192.168.0.12:27017", "state" : 1, "tags" : [ ] }
{ "_id" : "myproject1", "host" : "myproject1/192.168.0.13:27017,192.168.0.18:27017", "state" : 1, "tags" : [ "game" ] }
{ "_id" : "myproject2", "host" : "myproject2/192.168.0.26:27017,192.168.0.28:27017", "state" : 1, "tags" : [ "game" ] }
active mongoses:
"4.0.6" : 14
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 0
Migration Results for the last 24 hours:
8 : Success
databases:
{ "_id" : "myproject", "primary" : "myproject0", "partitioned" : true, "version" : { "uuid" : BinData(4,"+YxjwrKzTuekDlKutwT1IA=="), "lastMod" : 1 } }
...
In ref to mongo dba course trying to create replica set as asked shown by instructor in El Capitano (Single machine only), I get following error. I have three members:
(mongodb was installed using homebrew)
Step I: Setting up config
cfg ={ _id :"abc", members:[{_id:0, host:"localhost:27001"}, {_id:1, host:"localhost:27002"}, {_id:2, host:"localhost:27003"}] }
{
"_id" : "abc",
"members" : [
{
"_id" : 0,
"host" : "localhost:27001"
},
{
"_id" : 1,
"host" : "localhost:27002"
},
{
"_id" : 2,
"host" : "localhost:27003"
}
]
}
STEP II: Initialize the Config.
rs.reconfig(cfg)
2015-10-05T11:34:27.082-0400 E QUERY Error: Could not retrieve replica set config: { "ok" : 0, "errmsg" : "not running with --replSet", "code" : 76 }
at Function.rs.conf (src/mongo/shell/utils.js:1017:11)
at Function.rs.reconfig (src/mongo/shell/utils.js:969:22)
at (shell):1:4 at src/mongo/shell/utils.js:1017
Make sure you have the replSetName name configured in /etc/mongod.conf
replication:
replSetName: "somename"
Then restart your mongod.
sudo service mongod stop
sudo service mongod start
sudo service mongod restart
You are not running replica set with the repl set name.The solution is to set a replication set name in the mongod config file using the paramater --replSet.
eg) --replSet=test_replica
Once changes are done in config file restart the server.
I'm trying to configure a replica set of 3 members on 3 different Linux machines. Im running the mongod with replset in the config file. I mistakley set 'rs.initialize' in 2 machines, and now I have found the primary and tried to add the other instances, but it says that the config file is not from the same version.
How can I remove the replica set and start everything back from scratch?
If the following is true:
Is this a brand-new deployment.
There is no data you need to keep.
You can do the following:
Shutdown all 3 mongods.
Remove all files and directories from the
"dbpath" partition in all 3 machines
Restart all 3 mongods Connect
to one of the mongodds and submit the following command
config = { "_id": "rs0", "members" : [
{ "_id" : 0, "host" : "##Your DNS NAME:PORTNUMBER#" },
{ "_id" : 1, "host" : "##Your DNS NAME:PORTNUMBER#" },
{ "_id" : 2, "host" : "##Your DNS NAME:PORTNUMBER#" } ]
}
rs.initiate(config)
I have 3 nodes for sharding and configserver (sharding servers run on standard port 27017 and configserver running on port 27019)
stage-mongo1-tmp, stage-mongo2-tmp, and stage-mongo3-tmp
and a query router
stage-query0-mongo
in my current setup.
Sharding is working perfect as expected.
--- Sharding Status ---
sharding version: {
"_id" : 1,
"version" : 3,
"minCompatibleVersion" : 3,
"currentVersion" : 4,
"clusterId" : ObjectId("5321a5cc8a18e5280f7c9d5a")
}
shards:
{ "_id" : "shard0000", "host" : "stage-mongo1-tmp:27017" }
{ "_id" : "shard0001", "host" : "stage-mongo2-tmp:27017" }
{ "_id" : "shard0002", "host" : "stage-mongo3-tmp:27017" }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "testdb", "partitioned" : true, "primary" : "shard0000" }
testdb.testcollection
shard key: { "_id" : "hashed" }
chunks:
shard0001 28
shard0002 31
shard0000 28
too many chunks to print, use verbose if you want to force print
Now, I was enabling replica set on these nodes. I logged in to stage-mongo1-tmp, and ran
rs.initiate()
and added stage-mongo2-tmp and stage-mongo3-tmp as the replica members as
rs.add("stage-mongo2-tmp")
Log files says replication enabled and elected one primary.
rs.conf()
was showing good output
[rsBackgroundSync] replSet syncing to: stage-mongo1-tmp:27017
[rsSync] oplog sync 2 of 3
[rsSyncNotifier] replset setting oplog notifier to stage-mongo1-tmp:27017
[rsSync] replSet initial sync building indexes
[rsSync] replSet initial sync cloning indexes for : ekl_omniscient
[rsSync] build index ekl_omniscient.serviceability { _id: "hashed" }
[rsSync] build index done. scanned 571242 total records. 3.894 secs
replSet RECOVERING
replSet initial sync done
replSet SECONDARY
However, when I test the High-availability by taking one node down, mongos on the query node is returning error saying
mongos> show dbs;
Thu Mar 13 20:17:04.394 listDatabases failed:{
"code" : 11002,
"ok" : 0,
"errmsg" : "exception: socket exception [CONNECT_ERROR] for stage-mongo1-tmp:27017"} at src/mongo/shell/mongo.js:46
When I connect to one of the other node, one has automatically elected as primary. But still, my queries are returning errors.
What am I doing wrong here in replica set? Why is it not high-available? Do I need to add more servers to make it high-available? I am looking for a minimum set of servers to implement this.
Figured out. We add shard for the replica set.
sh.addShard("rs0/:port,..)
Once this is done, we need to enable sharding on db level and collection level. This would enable sharding and replica.