Why mongod.service is killed by my system? - mongodb

Question

I am using MongoDB 4.4 version on ubuntu 20.4, using 2 mongo instances on the same server. I encountered this problem:
Sep 03 10:45:25 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 10:45:25 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 10:45:25 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 10:45:25 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 10:55:46 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 10:55:46 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 10:55:46 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 10:55:46 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 11:05:05 mlnode systemd[1]: Stopping MongoDB Database Server...
Sep 03 11:05:05 mlnode systemd[1]: mongod.service: Succeeded.
Sep 03 11:05:05 mlnode systemd[1]: Stopped MongoDB Database Server.
Sep 03 11:05:05 mlnode systemd[1]: Started MongoDB Database Server.
Sep 03 11:13:20 mlnode systemd[1]: /lib/systemd/system/mongod.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/mongodb/mongod.pid → /run/mongodb/mongod.pid; please update the unit file accordingly.
Sep 03 11:25:47 mlnode systemd[1]: /lib/systemd/system/mongod.service:11: PIDFile= references a path below legacy directory /var/run/, updating /var/run/mongodb/mongod.pid → /run/mongodb/mongod.pid; please update the unit file accordingly.
Sep 03 11:56:47 mlnode systemd[1]: mongod.service: Main process exited, code=killed, status=6/ABRT
Sep 03 11:56:47 mlnode systemd[1]: mongod.service: Failed with result 'signal'
Wondering why the system is killing mongod.service, I don't have any clue. Then I checked my mongo logs:
{"t":{"$date":"2020-09-03T11:56:46.379+05:30"},"s":"F", "c":"-", "id":23083, "ctx":"conn10155","msg":"Invariant failure","attr":{"expr":"ret","error":"UnknownError: -31803: WT_NOTFOUND: item not found","file":"src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp","line":1598}}
{"t":{"$date":"2020-09-03T11:56:46.379+05:30"},"s":"F", "c":"-", "id":23084, "ctx":"conn10155","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2020-09-03T11:56:46.379+05:30"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"conn10155","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

Answer 1

First, stop mongodb if you restarted:
sudo systemctl stop mongod
Next, I detected the database corruption:
sudo mongod --repair --dbpath /var/lib/mongodb
Next, run commands:
chown -R mongodb:mongodb /var/lib/mongodb
chown mongodb:mongodb /tmp/mongodb-27017.sock
Finally
sudo systemctl start mongod
From: https://sondnpt00343.medium.com/how-to-fix-mongod-service-32dbbe51a4ee

Answer 2

I figured out that my Linux ulimits were not set to the recommended values of mongod instance.
One of the limits ( locked-in-memory ) was 65536 and in /lib/systemd/system/mongod.service LimitMEMLOCK was infinity.
So I've changed that to 65536, restarted the service and it was working again.
This worked for me!

Answer 3

First, I detected the database corruption.
sudo mongod --repair --dbpath /var/lib/mongodb
Then, I restarted the service but I faced another problem (exit-code).
Next, I ran following commands on the terminal :
chown -R mongodb:mongodb /var/lib/mongodb
chown mongodb:mongodb /tmp/mongodb-27017.sock
Finally, I restarted the mongodb service.
sudo systemctl restart mongod
And then this worked for me very well.