I want to run shard on my local server.
please help me step by step.
I want to create multi instance of mongod on my cpu(16 core).
By the way, I'm Collection that consist of 7 million documents, whether they went missing when I run shard?
I use this script for creating 3 shard on one replicaSet:
# clean everything up
echo "killing mongod and mongos"
killall mongod
killall mongos
echo "removing data files"
rm -rf /media/mongo/data/config
rm -rf /media/mongo/data/shard*
rm -rf /data/config
rm -rf /data/shard*
# For mac make sure rlimits are high enough to open all necessary connections
ulimit -n 2048
# start a replica set and tell it that it will be shard0
mkdir -p /media/mongo/data/shard0/rs0
mongod --replSet s0 --logpath "/var/log/mongodb/s0-r0.log" --dbpath /media/mongo/data/shard0/rs0 --port 37017 --fork --shardsvr --smallfiles
sleep 5
# connect to one server and initiate the set
mongo --port 37017 << 'EOF'
config = { _id: "s0", members:[
{ _id : 0, host : "localhost:37017" },
]};
rs.initiate(config)
EOF
# start a replicate set and tell it that it will be a shard1
mkdir -p /media/mongo/data/shard1/rs0
mongod --replSet s1 --logpath "/var/log/mongodb/s1-r0.log" --dbpath /media/mongo/data/shard1/rs0 --port 47017 --fork --shardsvr --smallfiles
sleep 5
mongo --port 47017 << 'EOF'
config = { _id: "s1", members:[
{ _id : 0, host : "localhost:47017" },
]};
rs.initiate(config)
EOF
# start a replicate set and tell it that it will be a shard2
mkdir -p /media/mongo/data/shard2/rs0
mongod --replSet s2 --logpath "/var/log/mongodb/s2-r0.log" --dbpath /media/mongo/data/shard2/rs0 --port 57017 --fork --shardsvr --smallfiles
sleep 5
mongo --port 57017 << 'EOF'
config = { _id: "s2", members:[
{ _id : 0, host : "localhost:57017" },
]};
rs.initiate(config)
EOF
# now start 3 config servers
rm cfg-a.log cfg-b.log cfg-c.log
mkdir -p /media/mongo/data/config/config-a /media/mongo/data/config/config-b /media/mongo/data/config/config-c
mongod --logpath "/var/log/mongodb/cfg-a.log" --dbpath /media/mongo/data/config/config-a --port 57040 --fork --configsvr --smallfiles
mongod --logpath "/var/log/mongodb/cfg-b.log" --dbpath /media/mongo/data/config/config-b --port 57041 --fork --configsvr --smallfiles
mongod --logpath "/var/log/mongodb/cfg-c.log" --dbpath /media/mongo/data/config/config-c --port 57042 --fork --configsvr --smallfiles
# now start the mongos on port 27018
rm mongos-1.log
sleep 5
mongos --port 27018 --logpath "/var/log/mongodb/mongos-1.log" --configdb localhost:57040,localhost:57041,localhost:57042 --fork
echo "Waiting 60 seconds for the replica sets to fully come online"
sleep 60
echo "Connnecting to mongos and enabling sharding"
# add shards and enable sharding on the test db
mongo --port 27018 << 'EOF'
db.adminCommand( { addshard : "s0/"+"localhost:37017" } );
db.adminCommand( { addshard : "s1/"+"localhost:47017" } );
db.adminCommand( { addshard : "s2/"+"localhost:57017" } );
db.adminCommand({enableSharding: "IBSng"});
EOF
sleep 5
echo "Done setting up sharded environment on localhost"
But I don't know that how to add shard key on my collection.
I speed write/read(I/O) is very important.
Sharding will not help improve query performance if it is done on a single machine which has huge number of documents(7 million is not small).
Reason : MongoDB uses memory mapped files which means copy of your data and indexes is stored in RAM and whenever there is a query it fetches it from the RAM itself. In the current scenario your queries are slower because your data + indexes size is so large that it will not fit in RAM , hence there will be lot of I/O activity to get data from disk which is the bottleneck.
What else can be done to improve query performance (incase Sharding is not an option):
Increase RAM on your machine
Use indexes
Redesign schema
Note - Even if you implement the above points, there is a limit to how much query performance can be improved, with sharding linear scaling of read/write throughput can be expected.
Related
Is something wrong with this?
$ sudo mongos --logpath "mongos-1.log" --configdb localhost:57040,localhost:57041,localhost:57042 --fork
FailedToParse: mirrored config server connections are not supported; for config server replica sets be sure to use the replica set connection string
try 'mongos --help' for more information
I had the same problem as you just have to update your --configdb to "repSetConfigName/localhost:57040,localhost:57041,localhost:57042"
here is an example :
this is my configuration of the config server:
mongod --configsvr --dbpath cfg0 --replSet conf --port 27050 --fork --logpath log.cfg0 --logappend
mongod --configsvr --dbpath cfg1 --replSet conf --port 27051 --fork --logpath log.cfg1 --logappend
mongod --configsvr --dbpath cfg2 --replSet conf --port 27052 --fork --logpath log.cfg2 --logappend
then I used
# connect to any of the mongod process on port 27050, 27051, 27052
mongo --port 27050
rs.initiate(
{
_id: "conf",
configsvr: true,
members: [
{ _id : 0, host : "localhost:27050" },
{ _id : 1, host : "localhost:27051" },
{ _id : 2, host : "localhost:27052" }
]
}
)
fianlly
mongos --configdb "conf/localhost:27050,localhost:27051,localhost:27052" --fork --logpath log.mongos0 --port 27200
Just appendix to #sylia-baraka answer... If you are using MongoDB 3.4 or newer you need to use CSRS instead of SCCC, read more here.
In version 3.4, MongoDB removes support for SCCC config servers. Before you can upgrade your sharded clusters to 3.4, you must convert your config servers from SCCC to CSRS. To convert to CSRS, follow the procedure to change your 3.2.x SCCC to 3.2.x CSRS.
configdb should be like this :
"your_replSetName/host:port,host:port,host:port"
I am trying to create a small replica set on my computer in Windows.
Here is what I have done so far:
I have made three directories in data by
mkdir \data\rs1 \data\rs2 \data\rs3
Now, I am starting three mongo instances:
start mongod --replSet m101 --logpath 1.log --dbpath \data\rs1 --port 27017 --smallfiles --oplogSize 64
start mongod --replSet m101 --logpath 2.log --dbpath \data\rs2 --port 27018 --smallfiles --oplogSize 64
start mongod --replSet m101 --logpath 3.log --dbpath \data\rs3 --port 27019 --smallfiles --oplogSize 64
Now I am starting mongod.exe by command line and connect to a mongo shell by typing
mongo --port 27017
Here is my replica set:
config = { _id: "m101", members:[
{ _id : 0, host : "localhost:27017"},
{ _id : 1, host : "localhost:27018"},
{ _id : 2, host : "localhost:27019"} ]
};
Now when I type rs.initiate(config); , It shows this node was not started with replSet option.
I also tried to run command line options on admin
use admin
db.runCommand("getCmdLineOpts")
and I don't see any --replSet options in there.
I get
{
"argv" : [ "mongod" ],
"parsed" : {},
"ok" : 1
}
I know many questions related to this problem are available and I tried many of them but none of them worked. I am new to this and learning this from one of the courses of MongoDB University. Please can anyone tell me what am I doing wrong?
I have question about use mongos with Shard,I build shard cluster in our dev environment
I had two shards(NOT include replset just for test) one config and one mongos
when I shutdown a shard the mongos can't find data(such as find(),count()...).it throw a socket exception
like this:
socket exception [CONNECT_ERROR] for 127.0.0.1:20000
code:11002
can anyone help me to figure out where is the problem,i saw the docs before it wrote about in the sharding cluster if a shard crashed the mongos can still work.
i appreciate your help,thx i use these shell to build it
shard 1
mongod --shardsvr --port 10000 --dbpath c:\shard\data\shard1\r0 --oplogSize 64 --logpath c:\shard\logs\r0.log --logappend
shard2
mongod --shardsvr --port 20000 --dbpath c:\shard\data\shard2\r0 --oplogSize 64 --logpath c:\shard\logs\r3.log --logappend
config
mongod --configsvr --dbpath c:\shard\data\config --port 40000 --logpath c:\shard\logs\config.log --logappend
mongos
mongos --configdb 127.0.0.1:40000 --port 50000 --chunkSize 1 --logpath c:\shard\logs\mongos.log --logappend
mongo 127.0.0.1:50000/admin
db.runCommand({addshard:"127.0.0.1:10000",name:"ShardSetA"})
db.runCommand({addshard:"127.0.0.1:20000",name:"ShardSetB"})
db.runCommand({enablesharding:"Staff"});
var server=["127.0.0.1:10000","127.0.0.1:20000"]
db.runCommand({shardcollection:"Staff.StaffInfo",key:{StaffName:1}})
for ( var x=65; x<=65+25; x+=10 ){
var index=x+10>65+25?65+25:x+10;
db.runCommand( { split : "Staff.StaffInfo" , middle : { StaffName: String.fromCharCode(index)} } );}
var i=0;
for ( var x=65; x<=97+25; x+=10 ){db.adminCommand( {moveChunk : "Staff.StaffInfo" ,find : {StaffName: x}, to : server[i%2] });i++}
I had solved this problem
there are three situations for the problem
1. query from normal shard ,mongodb can provide right data
2.query from all the shards(include offline shard) mongodb will throw exception
3.query from offline shard mongodb wiil throw exception
so if you want to keep high-performance sharding please use replset!
if you query from other dirvers please set partial flag to provide return data from normal shard
I'm trying the MongoDB's shard tutorial, for some simple testing (proof-of-concept project).
I want to try sharding on a single machine. Is this possible and/or does it make any sense?
When I follow the guide steps, it does not work.
First, I start the mongod configsrv database instances on my machine:
bin/mongod --configsvr --dbpath $BASEDIR/data/configdb --port 27019 &
bin/mongod --configsvr --dbpath $BASEDIR/data/configdb1 --port 27020 &
bin/mongod --configsvr --dbpath $BASEDIR/data/configdb2 --port 27021 &
Then, I start the mongos instances so that they "bind" to the config servers:
HOST=$(uname -n) # my machine's name
# starts on default poort 27017
bin/mongos --configdb $HOST:27019,$HOST:27020,$HOST:27021
Until here, everything looks good.
Now I want to add a Shard to the cluster:
bin/mongo --host $(uname -n)
It enters the MongoDB shell.
connecting to: my.machine.name:27017/test
But when I try to add a new shard, I have the following error:
mongos> sh.addShard( "rs1/my.machine.name:27017" )
{
"ok" : 0,
"errmsg" : "couldn't connect to new shard socket exception [CONNECT_ERROR] for rs1/my.machine.name:27017"
}
I have tried with ip, machine's alias, localhost ... nothing seems to work.
Anyone could help me on this? Maybe I'm missing a point.
Thanks in advance
I had the same problem, you have to run
mongod &
on your each of your shards
and then you can call
sh.addShard( "rs1/my.machine.name:27017" )
Its possible, follow below sequence.
lets say we want to create a cluster with 3 shards each with 1 replica on same machine.
Start first mongod server (shard 1 with replica 1)with a replica set named s1
Start second mongod server (shard 2 with replica 1)with a replica set named s2
Start third mongod server (shard 3 with replica 1)with a replica set named s3
mkdir -p /data/shard0/rs0 /data/shard0/rs1 /data/shard0/rs2
/bin/mongod --replSet Sv1 --logpath "Sv1-r0.log" --dbpath /data/shard0/rs0 --port 37017
/bin/mongod --replSet Sv2 --logpath "Sv2-r0.log" --dbpath /data/shard0/rs1 --port 47017
/bin/mongod --replSet Sv3 --logpath "Sv3-r0.log" --dbpath /data/shard0/rs2 --port 57017
This will start 3 mongod servers/shards with one replica for each shard
/bin/mongo --port 37017 << 'EOF'
config = { _id: "Sv1", members:[
{ _id : 0, host : "<ip>:37017" }]};
rs.initiate(config)
EOF
This will add replicas to replica set Sv1. This should be done for each shard i.e. on each port 47017 and 57017
Now we have cluster ready with shards and replicas but no one knows other shards and replicas, thus we need config server
/bin/mongod --logpath "cfg-a.log" --dbpath /data/config/config-a --port 57040 --fork --configsvr
Then start mongos for admin access,
/bin/mongos --logpath "mongos-1.log" --configdb 10.88.66.218:57040
Now add shards to our cluster,
/bin/mongo <<'EOF'
db.adminCommand( { addShard : "Sv1/"+"<ip>:37017" } );
db.adminCommand( { addShard : "Sv2/"+"<ip>:47017" } );
db.adminCommand( { addShard : "Sv3/"+"<ip>:57017" } );
use db
db.createCollection("col1")
db.adminCommand({enableSharding:"db"}) db.adminCommand({shardCollection: "db.col1", key: {_id:"hashed"}});
EOF
And that's it.... Cluster should be up with 3 shards. You can check with sh.status() command on mongos.
I do a MongoDB homework and follow the steps, but I faced the problem " server is not running with replset"
I do the step is :
`"start mongod --replSet m101 --logpath 1.log --dbpath \data\rs1 --port 27017 --smallfiles --oplogSize 64
start mongod --replSet m101 --logpath 2.log --dbpath \data\rs2 --port 27018 --smallfiles --oplogSize 64
start mongod --replSet m101 --logpath 3.log --dbpath \data\rs3 --port 27019 --smallfiles --oplogSize 64
Now connect to a mongo shell and make sure it comes up.
mongo --port 27017
Now you will create the replica set. Type the following commands into the mongo shell:
config = { _id: "m101", members:[`
{ _id : 0, host : "localhost:27017"},
{ _id : 1, host : "localhost:27018"},
{ _id : 2, host : "localhost:27019"} ]
};
rs.initiate(config);"
I do not know where is the problem, someone said the problem maybe is another mongod is running. But I have not any other mongod running in the same time.
Thank you!
It happened the same to me as well. The problem was that I have runned mongod without any parameters before I started launching the nodes.
First kill all the mongo, mongod and mongos instances to guarantee the environment is clear.
ps -ef | grep -i 'mongo'
sudo service mongod stop
The problem cause is simple. The default port for mongod and mongos is 27017. So, what happened to me was that the first replica node was never created. The following command was connecting me to a default mongod instance:
mongo --port 27017
I hope this solves your problem. ;)
Check what command line options the mongod you are connected to in the shell is running with:
> use admin
switched to db admin
> db.runCommand("getCmdLineOpts")
{
"argv" : [
"mongod",
"--dbpath",
"/data/db",
"--replSet",
"rs0"
],
"parsed" : {
"replication" : {
"replSet" : "rs0"
},
"storage" : {
"dbPath" : "/data/db"
}
},
"ok" : 1
}
Do you see the replSet option?
You can also try a different way of initiating the replica set. Instead of typing in the configuration and passing it to rs.initiate(), just run rs.initiate(). It will set up default config with one member- the server you are connected to in the shell. Then you can add the other two servers. See the tutorial for more information.
For MacOS Homebrew setup & default configuration
Edit default configuration mongodb file nano /opt/homebrew/etc/mongod.conf
Insert the following piece of code
replication:
replSetName: "rs0"
Save and restart the mongodb service brew services restart mongodb-community#6.0
Exec the following command in mongosh shell rs.initiate()
$ mongosh
> use admin
admin> rs.initiate()
success response:
{
info2: 'no configuration specified. Using a default configuration for the set',
me: '127.0.0.1:27017',
ok: 1
}