Running mongod service failed in Ubuntu 18.04.6 - mongodb

Question

mongod service failed to start. After rebooting the machine, checking if mongod is allowed to read and write in the files/directories, I don't find the root of the issue.
Checking the service, it fails:
(base) avy#machine:~$ sudo service mongod status
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2022-08-07 14:16:22 CEST; 2min 20s ago
Docs: https://docs.mongodb.org/manual
Process: 5230 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=100)
Main PID: 5230 (code=exited, status=100)
août 07 14:16:22 machine systemd[1]: Started MongoDB Database Server.
août 07 14:16:22 machine systemd[1]: mongod.service: Main process exited, code=exited, status=100/n/a
août 07 14:16:22 machine systemd[1]: mongod.service: Failed with result 'exit-code'.
I looked if the database (dbPath storage) was still owned by mongod and it is:
(base) avy#machine:~$ ls -l
drwxrwxrwx 5 avy mongodb 4096 août 6 13:08 mongodatabases
My config file is very simple and I did not write on it since I used MongoDB for the first time:
# mongod.conf
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/
# Where and how to store data.
storage:
dbPath: /home/avy/mongodatabases
journal:
enabled: true
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# network interfaces
net:
port: 27017
bindIp: 127.0.0.1
# how the process runs
processManagement:
timeZoneInfo: /usr/share/zoneinfo
It seemed it was due to 27017.sock:
(base) avy#machine:~$ mongo
MongoDB shell version v5.0.10
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Error: couldn't connect to server 127.0.0.1:27017,
connection attempt failed:
SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect#src/mongo/shell/mongo.js:372:17
#(connect):2:6
exception: connect failed
exiting with code 1
I changed the owning:
(base) avy#machine:/tmp$ sudo chown mongodb:mongodb mongodb-27017.sock
(base) avy#machine:/tmp$ ls -l
srwx------ 1 mongodb mongodb 0 août 7 15:27 mongodb-27017.sock
it is still failling.
UPDATE, answer to rickhg12hs comment What's in /var/log/mongodb/mongod.log?
(base) avy#machine:/var/log/mongodb$ ls -l
total 112156
-rw------- 1 mongodb mongodb 114841063 août 7 17:37 mongod.log
There is only this file.
Does mongod still have an old lock file that needs to be deleted?
There is one in the desired dbPath:
(base) avy#machine:~/mongodatabases$ ls -l
...
-rw------- 1 mongodb mongodb 5 août 5 10:49 mongod.lock
...
I noticed something, let me know what you think about it
# the whole result of command
(base) avy#machine:~$ mongod
{"t":{"$date":"2022-08-07T17:37:43.960+02:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"thread1","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2022-08-07T17:37:43.960+02:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"thread1","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2022-08-07T17:37:43.960+02:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"thread1","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-08-07T17:37:43.960+02:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"thread1","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"thread1","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"thread1","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"thread1","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"CONTROL", "id":5945603, "ctx":"thread1","msg":"Multi threading initialized"}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":4638,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"machine"}}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.10","gitVersion":"bbf5bc7e16d1713c9349a09adf4901ca37472e66","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{}}}
{"t":{"$date":"2022-08-07T17:37:43.963+02:00"},"s":"E", "c":"NETWORK", "id":23024, "ctx":"initandlisten","msg":"Failed to unlink socket file","attr":{"path":"/tmp/mongodb-27017.sock","error":"Operation not permitted"}}
{"t":{"$date":"2022-08-07T17:37:43.963+02:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":40486,"file":"src/mongo/transport/transport_layer_asio.cpp","line":1019}}
{"t":{"$date":"2022-08-07T17:37:43.963+02:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
When I focus on this line I saw it is still trying to run dbPath on /data/db contrary to what the config file tells:
{"t":{"$date":"2022-08-07T17:37:43.962+02:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":4638,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"machine"}}
besides it says the connection to the socket 27017 is not permitted
{"t":{"$date":"2022-08-07T17:37:43.963+02:00"},"s":"E", "c":"NETWORK", "id":23024, "ctx":"initandlisten","msg":"Failed to unlink socket file","attr":{"path":"/tmp/mongodb-27017.sock","error":"Operation not permitted"}}
so I tried this:
(base) avy#machine:~$ sudo mongod --repair --config /etc/mongod.conf
But this is the same problem
UPDATE 2
What's in /lib/systemd/system/mongod.service?
[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target
[Service]
User=mongodb
Group=mongodb
EnvironmentFile=-/etc/default/mongod
ExecStart=/usr/bin/mongod --config /etc/mongod.conf
PIDFile=/var/run/mongodb/mongod.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings
[Install]
WantedBy=multi-user.target
Does the tail of the contents of /var/log/mongodb/mongod.log have some enlightening messages?
I am not expert of database at all so "enlightening message" is difficult to judge for me. But I checked the behavior of a typical "good day" and the day it started to fail. I think this is due to a violent shutdown.
# a good day
{"t":{"$date":"2022-08-04T23:49:03.773+02:00"},"s":"I", "c":"CONTROL", "id":23138, "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCode":0}}
{"t":{"$date":"2022-08-05T10:49:00.638+02:00"},"s":"I", "c":"CONTROL", "id":20698, "ctx":"-","msg":"***** SERVER RESTARTED *****"}
# some lines
{"t":{"$date":"2022-08-05T10:49:00.651+02:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":2027,"port":27017,"dbPath":"/home/avy/mongodatabases","architecture":"64-bit","host":"machine"}}
{"t":{"$date":"2022-08-05T10:49:00.651+02:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.10","gitVersion":"bbf5bc7e16d1713c9349a09adf4901ca37472e66","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2022-08-05T10:49:00.651+02:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2022-08-05T10:49:00.651+02:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/home/avy/mongodatabases","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"path":"/var/log/mongodb/mongod.log"}}}}
{"t":{"$date":"2022-08-05T10:49:00.654+02:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/home/avy/mongodatabases","storageEngine":"wiredTiger"}}
I shutdown without issue but it seems that today the schema is different:
{"a":"7FEA0E67261F","b":"7FEA0E551000","o":"12161F","s":"clone","s+":"3F"}}} # you see no {"exitCode":0}
{"t":{"$date":"2022-08-07T08:56:02.403+02:00"},"s":"I", "c":"CONTROL", "id":20698, "ctx":"-","msg":"***** SERVER RESTARTED *****"}
... # some lines
{"t":{"$date":"2022-08-07T08:56:02.416+02:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1932,"port":27017,"dbPath":"/home/avy/mongodatabases","architecture":"64-bit","host":"machine"}}
{"t":{"$date":"2022-08-07T08:56:02.416+02:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.10","gitVersion":"bbf5bc7e16d1713c9349a09adf4901ca37472e66","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2022-08-07T08:56:02.416+02:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2022-08-07T08:56:02.416+02:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/home/av/mongodatabases","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"path":"/var/log/mongodb/mongod.log"}}}}
and then the initandlisten operation started to fail.
Before mongod starts, there should be no lock file
I removed all .lock files related to mongo in the whole machine.
I tried once again repair command
(base) avy#machine:~$ mongod --repair --dbpath "/home/avy/mongodatabases"
{"t":{"$date":"2022-08-07T19:24:17.116+02:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2022-08-07T19:24:17.116+02:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2022-08-07T19:24:17.117+02:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-08-07T19:24:17.117+02:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2022-08-07T19:24:17.118+02:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-08-07T19:24:17.118+02:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2022-08-07T19:24:17.118+02:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2022-08-07T19:24:17.118+02:00"},"s":"I", "c":"CONTROL", "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2022-08-07T19:24:17.119+02:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":8346,"port":27017,"dbPath":"/home/avy/mongodatabases","architecture":"64-bit","host":"machine"}}
{"t":{"$date":"2022-08-07T19:24:17.119+02:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.10","gitVersion":"bbf5bc7e16d1713c9349a09adf4901ca37472e66","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2022-08-07T19:24:17.119+02:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2022-08-07T19:24:17.119+02:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"repair":true,"storage":{"dbPath":"/home/avy/mongodatabases"}}}}
{"t":{"$date":"2022-08-07T19:24:17.123+02:00"},"s":"F", "c":"STORAGE", "id":28661, "ctx":"initandlisten","msg":"Unable to read the storage engine metadata file","attr":{"error":{"code":38,"codeName":"FileNotOpen","errmsg":"Failed to read metadata from /home/mongodatabases/storage.bson"}}}
{"t":{"$date":"2022-08-07T19:24:17.123+02:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28661,"file":"src/mongo/db/storage/storage_engine_metadata.cpp","line":91}}
{"t":{"$date":"2022-08-07T19:24:17.123+02:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
About the version used:
(base) avy#machine:~$ mongod --version
db version v5.0.10
Build Info: {
"version": "5.0.10",
"gitVersion": "bbf5bc7e16d1713c9349a09adf4901ca37472e66",
"openSSLVersion": "OpenSSL 1.1.1 11 Sep 2018",
"modules": [],
"allocator": "tcmalloc",
"environment": {
"distmod": "ubuntu1804",
"distarch": "x86_64",
"target_arch": "x86_64"
}
}