Here is the topology I have running off self hosted Ubuntu machines on AWS (EC2)
MongoDB version 5.0.3
Mongos - 1 server
Config Servers (3 servers)
Shards (3) each is a ReplicaSet with 3 members each, therefore 9 data nodes
For various reasons, which don't seem to be in the logs, all 3 members of Shard 3 went down. Restarting the processes on these 3 members was the most obvious step, however, this was the output.
ubuntu#XXXXXXXX:~$ mongosh
Current Mongosh Log ID: 619aa17d4fac3c77a109b11b
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017
Eventually, one member came back, as SECONDARY, restarting the mongod processes on the other members seemed to cascade and kill the first member. At least so it appeared.
Following this guide, I entered mongosh on Shard 3, Member 1 (3-1), I then disconnected the other members from the ReplicaSet. After this, Member one refuses to start. The process logs show this
ubuntu#XXXXXXX:~$ sudo systemctl status mongod
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Sun 2021-11-21 20:02:31 UTC; 798ms ago
Docs: https://docs.mongodb.org/manual
Process: 1094 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=killed, signal=ABRT)
Main PID: 1094 (code=killed, signal=ABRT)
Nov 21 20:02:26 ip-172-31-32-42 systemd[1]: Started MongoDB Database Server.
Nov 21 20:02:31 ip-172-31-32-42 systemd[1]: mongod.service: Main process exited, code=killed, sta>
Nov 21 20:02:31 ip-172-31-32-42 systemd[1]: mongod.service: Failed with result 'signal'.
Is it possible to get any of these members in RS3 back up?
Is it possible to restore the data that has been sharded on Shard 3, of course this is only part of it, Shard 1 & Shard 2 (and their ReplicaSets are okay)
Related
i was using mongodb and it was fine.
then i wanted to convert it to replica set and i get into some problems and i uninstalled it.
after reinstalling (10 times and doing everything on internet xD) why i check status with systemctl status it say failed with exit_code ( i know my conf file dont have problem).
what can i do? i even installed the 3.3 version and even it doesnt start anymore.
i used anything that it came to my mind (purging config files & lot more...).
i really dont want to reinstall my os (really cant).
this is my sudo systemctl status mongod
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-02-18 20:05:20 +0330; 8s ago
Docs: https://docs.mongodb.org/manual
Process: 147513 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=1/FAILURE)
Main PID: 147513 (code=exited, status=1/FAILURE)
Feb 18 20:05:20 nima-Lenovo-ideapad-320-15AST systemd[1]: Started MongoDB Database Server.
Feb 18 20:05:20 nima-Lenovo-ideapad-320-15AST mongod[147513]: about to fork child process, waiting until server is ready for connections.
Feb 18 20:05:20 nima-Lenovo-ideapad-320-15AST mongod[147527]: forked process: 147527
Feb 18 20:05:20 nima-Lenovo-ideapad-320-15AST mongod[147513]: ERROR: child process failed, exited with 1
Feb 18 20:05:20 nima-Lenovo-ideapad-320-15AST mongod[147513]: To see additional information in this output, start without the "--fork" option.
Feb 18 20:05:20 nima-Lenovo-ideapad-320-15AST systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE
Feb 18 20:05:20 nima-Lenovo-ideapad-320-15AST systemd[1]: mongod.service: Failed with result 'exit-code'.
I solved the problem by changing the default mongodb port from port 27017 to port 27018 in /etc/mongod.conf.
I'm sure this will come handy to a lot of people.
And for the last part, after uninstalling mongodb I removed mongod.service files (every file) in the system and systemd directories in root and installed mongodb again.
(so I think uninstalling mongodb wasn't complete at first time. And 2 instances interfere with each other. Now everything works fine in mongodb with port 27018).
When I set bindIp of mongod.conf file as my wifi router's default gateway i.e bindIp:127.0.0.1,198.168.3.1. my MongoDB server stops and I get following error when I check its status.
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset:
Active: failed (Result: exit-code) since Thu 2020-01-23 17:42:38 IST; 1s ago
Docs: https://docs.mongodb.org/manual
Process: 3046 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited
Main PID: 3046 (code=exited, status=48)
Jan 23 17:42:38 sarvahapc systemd[1]: Started MongoDB Database Server.
Jan 23 17:42:38 sarvahapc systemd[1]: mongod.service: Main process exited, code=
Jan 23 17:42:38 sarvahapc systemd[1]: mongod.service: Unit entered failed state.
Jan 23 17:42:38 sarvahapc systemd[1]: mongod.service: Failed with result 'exit-c
lines 1-11/11 (END)
But when I set bindIp: 0.0.0.0 in mongod.conf file my MongoDB server runs smoothly and it allows remote access to my MongoDB server for other systems which are in my wifi network.
I want to give remote access to my MongoDB server to other systems which are in my wifi network. Can anyone help me with this?
Here are the logs for my mongodb status which show in it is in running status
ubuntu#ip-172-31-9-130:/etc$ sudo service mongod status
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset:
Active: active (running) since Fri 2018-09-28 09:50:43 UTC; 37min ago
Docs: https://docs.mongodb.org/manual
Main PID: 28315 (mongod)
CGroup: /system.slice/mongod.service
└─28315 /usr/bin/mongod --config /etc/mongod.conf
Sep 28 09:50:43 ip-172-31-9-130 systemd[1]: Started MongoDB Database Server.
Sep 28 09:50:43 ip-172-31-9-130 mongod[28315]: 2018-09-28T09:50:43.848+0000 I CO
lines 1-10/10 (END)
But when I connect it with robomongo it throws error
Failed to connect to the server
Please help!!!
I am getting this error instead of my shell prompt (running Ubuntu 16.04LTS).
I'm pretty new to MongoDB. I had to downgrade from version 3.4 to 3.2:
https://docs.mongodb.com/manual/release-notes/3.4-downgrade-standalone/
I followed those instructions (I thought) pretty faithfully, but now can't access the shell to confirm, etc.
scorpian555#scorpo:~$ mongo
MongoDB shell version: 3.2.15
connecting to: test
2017-07-13T14:41:31.434-0400 W NETWORK [thread1] Failed to connect to
127.0.0.1:27017, in(checking socket for error after poll), reason:
errno:111 Connection refused
2017-07-13T14:41:31.434-0400 E QUERY [thread1] Error: couldn't
connect to server 127.0.0.1:27017, connection attempt failed :
connect#src/mongo/shell/mongo.js:229:14
#(connect):1:6
exception: connect failed
There was a socket file that the error seems to be referencing at /tmp/mongodb-27017.sock
Per these, I ended up removing that socket file, then tried to reboot the mongod daemon:
'Failed to unlink socket file" error in MongoDB 3.0
MongoDB - Permission denied for socket: /tmp/mongodb-27017.sock
service mongod restart
service mongod status
scorpian555#scorpo:~$ service mongod status
● mongod.service - High-performance, schema-free document-oriented
database
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor
preset: enabled)
Active: active (running) since Thu 2017-07-13 14:58:08 EDT; 4s ago
Docs: https://docs.mongodb.org/manual
Main PID: 4763 (mongod)
CGroup: /system.slice/mongod.service
└─4763 /usr/bin/mongod --quiet --config /etc/mongod.conf
Jul 13 14:58:08 scorpo systemd[1]: Started High-performance, schema-
free document-oriented database.
Per those, I removed the socket file and ran:
scorpian555#scorpo:~$ ls -ld /tmp
drwxrwxrwt 17 root root 20480 Jul 13 14:58 /tmp
I think I'm a little lost here, I was under the impression that, upon restart of the daemon, it would recreate this temporary file, at which point I would check the permissions.
I can't find the link at the moment, but had found elsewhere instructions to make sure the owner of the mongod.sock file was mongod (not root) and I would have to chown it if it wasn't mongod.
I'm pretty new to MongoDB, like I said. I found this as another possible issue, but wanted to see if I should dive down this rabbit hole or whether I might be looking in the wrong place?
Connection refused to MongoDB errno 111
Thanks.
My issue was that, while I started the mongod service, I did not actually run the daemon with sudo as needed.
We have a mongodb with 2 shardings, each of the shardings has those servers:
Shard 1: Master, running MongoD and Config server
Shard 1-s1: Slave, running MongoD and MongoS server
Shard 1-s2: Slave, running MongoD and MongoS and Arbiter server
Shard 2: Master, running MongoD and Config Server
Shard 2-s1: Slave, running MongoD and Config and MongoS server
Shard 2-s2: Slave, running MongoD and MongoS and Arbiter server
But the mongodb allways failed in recent days, after days of search, i find out that the MongoD runs at Shard 1(Master) always going down after reviced too many connections, other MongoD don't have this problem.
When the S1Master's MongoD running with too many connections for about 2 hours, the 4 Mongos Server will shut down one by one. Here is the Mongos's error log(10.81.4.72:7100 runs MongoD):
Tue Aug 20 20:01:52 [conn8526] DBClientCursor::init call() failed
Tue Aug 20 20:01:52 [conn3897] ns: user.dev could not initialize cursor across all shards because : stale config detected for ns: user.
dev ParallelCursor::_init # s01/10.36.31.36:7100,10.42.50.24:7100,10.81.4.72:7100 attempt: 0
Tue Aug 20 20:01:52 [conn744] ns: user.dev could not initialize cursor across all shards because : stale config detected for ns: user.d
ev ParallelCursor::_init # s01/10.36.31.36:7100,10.42.50.24:7100,10.81.4.72:7100 attempt: 0
I don't know why this mongod revieved so many connections, the chunks shows the sharding works well.