MongoDB service would not start - mongodb

I installed the MongoDB using the following commands:
sudo apt update
sudo apt install mongodb
and when I checked the status of the service:
mongodb.service - An object/document-oriented database
Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2019-11-06 10:25:11 PST; 54min ago
Docs: man:mongod(1)
Process: 1030 ExecStart=/usr/bin/mongod --unixSocketPrefix=${SOCKETPATH} --config ${CONF} $DAEMON_OPTS (code=exited, status=100)
Main PID: 1030 (code=exited, status=100)
Nov 06 10:25:09 JDL systemd[1]: Started An object/document-oriented database.
Nov 06 10:25:11 JDL systemd[1]: mongodb.service: Main process exited, code=exited, status=100/n/a
Nov 06 10:25:11 JDL systemd[1]: mongodb.service: Failed with result 'exit-code'.
I tried creating /data/db and changed its permission.
I was also able to run the mongod command and even restarted the PC. I also tried deleting lock file and did repair in MongoDB but the result was still the same.
How can I fix this one? I tried to search for solutions but none of it seemed to work for me.
Additional Info
running sudo tail -n 20 /var/log/mongodb/mongod.log showed me this
2019-11-05T22:17:52.153+0800 I CONTROL [initandlisten] build environment:
2019-11-05T22:17:52.153+0800 I CONTROL [initandlisten] distarch: x86_64
2019-11-05T22:17:52.153+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-11-05T22:17:52.153+0800 I CONTROL [initandlisten] options: { config: "./etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2019-11-05T22:17:52.154+0800 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-11-05T22:17:52.154+0800 I STORAGE [initandlisten]
2019-11-05T22:17:52.154+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-11-05T22:17:52.154+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-11-05T22:17:52.154+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=5431M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-11-05T22:17:52.723+0800 E STORAGE [initandlisten] WiredTiger error (-31802) [1572963472:723937][2926:0x7f9a371b00c0], txn-recover: unsupported WiredTiger file version: this build only supports versions up to 2, and the file is version 3: WT_ERROR: non-specific WiredTiger error
2019-11-05T22:17:52.724+0800 E STORAGE [initandlisten] WiredTiger error (0) [1572963472:724065][2926:0x7f9a371b00c0], txn-recover: WiredTiger is unable to read the recovery log.
2019-11-05T22:17:52.724+0800 E STORAGE [initandlisten] WiredTiger error (0) [1572963472:724099][2926:0x7f9a371b00c0], txn-recover: This may be due to the log files being encrypted, being from an older version or due to corruption on disk
2019-11-05T22:17:52.724+0800 E STORAGE [initandlisten] WiredTiger error (0) [1572963472:724124][2926:0x7f9a371b00c0], txn-recover: You should confirm that you have opened the database with the correct options including all encryption and compression options
2019-11-05T22:17:52.724+0800 E STORAGE [initandlisten] WiredTiger error (-31802) [1572963472:724152][2926:0x7f9a371b00c0], txn-recover: Recovery failed: WT_ERROR: non-specific WiredTiger error
2019-11-05T22:17:52.759+0800 E - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 413
2019-11-05T22:17:52.759+0800 I STORAGE [initandlisten] exception in initAndListen: Location28595: -31802: WT_ERROR: non-specific WiredTiger error, terminating
2019-11-05T22:17:52.759+0800 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2019-11-05T22:17:52.759+0800 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2019-11-05T22:17:52.759+0800 I CONTROL [initandlisten] now exiting
2019-11-05T22:17:52.759+0800 I CONTROL [initandlisten] shutting down with code:100

If you don't need database then remove all the files in database folder (in Ubuntu path will be /usr/lib/mongodb) using rm -rf /usr/lib/mongodb and if you require database then follow this steps:
When you upgrade MongoDB Server, let's say from 3.4 to 3.6. You also need to set database feature compatibility version using this command before upgrading.
This command must be run in MongoDB Server v3.4
db.adminCommand( { setFeatureCompatibilityVersion: "3.4" } )
and then upgrade to the next major release.
For eg., you must upgrade your MongoDB server from 3.4 to 3.6 not directly to 4.0
for more details follow MongoDB Docs

Related

MongoDB repair failed due to Invariant failure rs.get() src/mongo/db/catalog/database.cpp

MongoDB Version: 3.4.24
MongoDB hosted on Linux server was abruptly shut down due to the over-utilization of memory.
Initiated the mongodb repair using: sudo mongod -f /etc/mongodrepair.conf --repair
The whole database is of 2.5TB, while repairing/re-indexing the db, close to 2.4TB was successfully repaired but the last 972MB of DB repair was failed due to Invariantfailure error.
Repair Logs
2020-07-04T17:17:07.441+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] build index on: test.summary properties: { v: 1, key: { totalVolume: -1 }, name: "totalV$
lume_-1", ns: "test.summary", background: true }
2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] build index on: test.summary properties: { v: 1, key: { ts: -1 }, name: "ts_-1", ns: "test.summary", background: true }
2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:08.673+0000 I - [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 195
2020-07-04T17:17:08.673+0000 I - [initandlisten]
***aborting after invariant() failure
2020-07-04T17:17:08.717+0000 F - [initandlisten] Got signal: 6 (Aborted).
Restart Logs
2020-07-04T17:39:14.476+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] MongoDB starting : pid=20485 port=27017 dbpath=/home/db324 64-bit host=ip-*-*-*-*
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] db version v3.4.24
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] modules: none
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] build environment:
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] distarch: x86_64
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "-.-.-.-", port: 27017 }, replication: $
oplogSizeMB: 10240, replSetName: "rs1" }, storage: { dbPath: "/home/db324", directoryPerDB: true, engine: "wiredTiger", journal: { enabled$
true }, wiredTiger: { engineConfig: { cacheSizeGB: 108.0 } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.lo$
" } }
2020-07-04T17:39:14.480+0000 W - [initandlisten] Detected unclean shutdown - /home/db324/mongod.lock is not empty.
2020-07-04T17:39:14.499+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten]
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] ** WARNING: The configured WiredTiger cache size is more than 80% of available RAM.
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=110592M,session_max=20000,eviction=(threads_min=4,t$
reads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000)$
checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667272][20485:0x7fdbcb287580], txn-recover: Main recovery loop$
starting at 73368/128
2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667951][20485:0x7fdbcb287580], txn-recover: Recovering log 733$
8 through 73369
2020-07-04T17:39:14.733+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:733044][20485:0x7fdbcb287580], txn-recover: Recovering log 733$
9 through 73369
2020-07-04T17:39:15.164+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1593884355:164908][20485:0x7fdbcb287580], test/collectio$
-56-3854974571131417844.wt, WT_SESSION.open_cursor: /home/db324/test/collection-56-3854974571131417844.wt: handle-read: pread: failed $
to read 4096 bytes at offset 28672: WT_ERROR: non-specific WiredTiger error
2020-07-04T17:39:15.164+0000 I - [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredT$
ger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 113
2020-07-04T17:39:15.165+0000 I - [initandlisten]
***aborting after invariant() failure
Is there a way to repair/recover the last part of the DB? or
Is there a way to ignore the broken DB? or
Is it possible to cut out the whole 2.4TB data sans the last error db and create a new MongoDB instance with 2.4TB?
I would greatly appreciate the help.
Thanks in Advance
The repair log indicates that it failed building an index on ns: "test.summary",
The other log gives you the file name, and the offset of the error:
/home/db324/test/collection-56-3854974571131417844.wt: handle-read: pread: failed $
to read 4096 bytes at offset 28672
The data after that point in the file is likely unsalvageable. You might try:
Make a backup of the existing files
Remove the file /home/db324/test/collection-56-3854974571131417844.wt
Re-run mongod --repair on this dbpath
If all goes well, it will create a new, empty file for that collection.
If you need to try to salvage that data, after the above succeeds so that you know the rest of the data files are consistent, recopy that file from the backup and attempt repair again.

MongoDB 4.2 doesnt start on ubuntu 18.04 after reinstall - process immediately stops

I have problem with mongoDB - I going step by step with official tutorial:
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
Also I installed through this answer( of course I changed some commands to download 4.2 not 4.0 like it is in this answer): https://askubuntu.com/questions/842592/apt-get-fails-on-16-04-or-18-04-installing-mongodb/842599#842599
But afterall it still doesnt work:
adrian#ubuntu:~$ sudo service mongod start
adrian#ubuntu:~$ sudo service mongod status
● mongod.service - MongoDB Database Server
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2019-09-26 09:37:53 CEST; 2s ago
Docs: https://docs.mongodb.org/manual
Process: 5220 ExecStart=/usr/bin/mongod --config /etc/mongod.conf (code=exited, status=62)
Main PID: 5220 (code=exited, status=62)
Sep 26 09:37:51 ubuntu systemd[1]: Started MongoDB Database Server.
Sep 26 09:37:53 ubuntu systemd[1]: mongod.service: Main process exited, code=exited, status=62/n/a
Sep 26 09:37:53 ubuntu systemd[1]: mongod.service: Failed with result 'exit-code'.
Even when I try to investigate mongo logs it still doesnt tell me much:
adrian#ubuntu:~$ sudo tail -n 100 /var/log/mongodb/mongod.log
2019-09-26T09:37:51.893+0200 I CONTROL [main] ***** SERVER RESTARTED *****
2019-09-26T09:37:51.895+0200 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] MongoDB starting : pid=5220 port=27017 dbpath=/var/lib/mongodb 64-bit host=ubuntu
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] db version v4.2.0
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] git version: a4b751dcf51dd249c5865812b390cfd1c0129c30
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1 11 Sep 2018
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] allocator: tcmalloc
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] modules: none
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] build environment:
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] distmod: ubuntu1804
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] distarch: x86_64
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] target_arch: x86_64
2019-09-26T09:37:51.901+0200 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2019-09-26T09:37:51.901+0200 I STORAGE [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-09-26T09:37:51.901+0200 I STORAGE [initandlisten]
2019-09-26T09:37:51.901+0200 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-09-26T09:37:51.901+0200 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-09-26T09:37:51.901+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=11527M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
2019-09-26T09:37:52.400+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:400536][5220:0x7fc6c8d05b00], txn-recover: Recovering log 20 through 21
2019-09-26T09:37:52.460+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:460670][5220:0x7fc6c8d05b00], txn-recover: Recovering log 21 through 21
2019-09-26T09:37:52.544+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:544472][5220:0x7fc6c8d05b00], txn-recover: Main recovery loop: starting at 20/896 to 21/256
2019-09-26T09:37:52.607+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:607556][5220:0x7fc6c8d05b00], txn-recover: Recovering log 20 through 21
2019-09-26T09:37:52.643+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:643077][5220:0x7fc6c8d05b00], txn-recover: Recovering log 21 through 21
2019-09-26T09:37:52.669+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:669301][5220:0x7fc6c8d05b00], txn-recover: Set global recovery timestamp: (0,0)
2019-09-26T09:37:52.677+0200 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2019-09-26T09:37:52.682+0200 I STORAGE [initandlisten] Timestamp monitor starting
2019-09-26T09:37:52.683+0200 I CONTROL [initandlisten]
2019-09-26T09:37:52.683+0200 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-09-26T09:37:52.684+0200 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-09-26T09:37:52.684+0200 I CONTROL [initandlisten]
2019-09-26T09:37:52.698+0200 I SHARDING [initandlisten] Marking collection local.system.replset as collection version: <unsharded>
2019-09-26T09:37:52.698+0200 F CONTROL [initandlisten] ** IMPORTANT: UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: BadValue: Invalid value for version, found 3.6, expected '4.2' or '4.0'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.6" }. See http://dochub.mongodb.org/core/4.0-feature-compatibility.). If the current featureCompatibilityVersion is below 4.0, see the documentation on upgrading at http://dochub.mongodb.org/core/4.0-upgrade-fcv.
2019-09-26T09:37:52.698+0200 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2019-09-26T09:37:52.698+0200 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
2019-09-26T09:37:52.698+0200 I - [initandlisten] Stopping further Flow Control ticket acquisitions.
2019-09-26T09:37:52.698+0200 I STORAGE [initandlisten] Deregistering all the collections
2019-09-26T09:37:52.698+0200 I STORAGE [initandlisten] Timestamp monitor shutting down
2019-09-26T09:37:52.698+0200 I STORAGE [initandlisten] WiredTigerKVEngine shutting down
2019-09-26T09:37:52.699+0200 I STORAGE [initandlisten] Shutting down session sweeper thread
2019-09-26T09:37:52.699+0200 I STORAGE [initandlisten] Finished shutting down session sweeper thread
2019-09-26T09:37:52.699+0200 I STORAGE [initandlisten] Shutting down journal flusher thread
2019-09-26T09:37:52.779+0200 I STORAGE [initandlisten] Finished shutting down journal flusher thread
2019-09-26T09:37:52.779+0200 I STORAGE [initandlisten] Shutting down checkpoint thread
2019-09-26T09:37:52.779+0200 I STORAGE [initandlisten] Finished shutting down checkpoint thread
2019-09-26T09:37:52.779+0200 I STORAGE [initandlisten] Downgrading WiredTiger datafiles.
2019-09-26T09:37:52.808+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:808033][5220:0x7fc6c8d05b00], txn-recover: Recovering log 21 through 22
2019-09-26T09:37:52.848+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:848763][5220:0x7fc6c8d05b00], txn-recover: Recovering log 22 through 22
2019-09-26T09:37:52.891+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:891333][5220:0x7fc6c8d05b00], txn-recover: Main recovery loop: starting at 21/3328 to 22/256
2019-09-26T09:37:52.944+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:944712][5220:0x7fc6c8d05b00], txn-recover: Recovering log 21 through 22
2019-09-26T09:37:52.977+0200 I STORAGE [initandlisten] WiredTiger message [1569483472:977288][5220:0x7fc6c8d05b00], txn-recover: Recovering log 22 through 22
2019-09-26T09:37:53.003+0200 I STORAGE [initandlisten] WiredTiger message [1569483473:3143][5220:0x7fc6c8d05b00], txn-recover: Set global recovery timestamp: (0,0)
2019-09-26T09:37:53.023+0200 I STORAGE [initandlisten] shutdown: removing fs lock...
2019-09-26T09:37:53.023+0200 I CONTROL [initandlisten] now exiting
2019-09-26T09:37:53.023+0200 I CONTROL [initandlisten] shutting down with code:62
I will be very grateful for any advice
Remove complete mongodb from your ubuntu
sudo apt-get purge mongo* // or sudo apt remove mongo*
sudo apt-get purge mongodb-org*
Backup your database and remove log and data using folder
sudo rm -r /var/log/mongodb
sudo rm -r /var/lib/mongodb
Now install again with this
$ wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc \
sudo apt-key add -
Add sources
$ echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.2 multiverse" \
sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
Then install mongodb package again
sudo apt-get install -y mongodb-org=4.2.0 \
mongodb-org-server=4.2.0 \
mongodb-org-shell=4.2.0 \
mongodb-org-mongos=4.2.0 \
mongodb-org-tools=4.2.0
Start service using mongodb
$ systemctl enable mongod.service
$ sudo service mongod start
Or Follow this doc installation process
i also have this issue, i solved by change the bindIp in mongo config
On Linux, a default /etc/mongod.conf configuration file is included
when using a package manager to install MongoDB.
sudo nano /etc/mongod.conf
then edit the bindIP to
bindIp: [127.0.0.1, X.X.X.X]
Then restart mongodb .
sudo service mongod stop
sudo service mongod start

Mongodb ERROR: child process failed, exited with error number 51

mongodb was closed by some reason. system shutdown abnormally.
when i try to restart mongodb use:
mongod -f /usr/local/mongodb/etc/mongo.conf
got
about to fork child process, waiting until server is ready for connections.
forked process: 28402
ERROR: child process failed, exited with error number 51
To see additional information in this output, start without the "--fork" option.
mongodb.log:
2019-02-19T13:22:24.105+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] MongoDB starting : pid=28131 port=27017 dbpath=/usr/local/mongodb/data/mongodata 64-bit host=nginx.xchs
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] db version v3.6.2
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] git version: 489d177dbd0f0420a8ca04d39fd78d0a2c539420
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] modules: none
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] build environment:
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] distmod: rhel62
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] distarch: x86_64
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] options: { config: "/usr/local/mongodb/etc/mongo.conf", net: { bindIp: "0.0.0.0" }, processManagement: { fork: true }, storage: { dbPath: "/usr/local/mongodb/data/mongodata" }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/mongodb/data/log/mongolog/mongodb.log" } }
2019-02-19T13:22:24.112+0800 W - [initandlisten] Detected unclean shutdown - /usr/local/mongodb/data/mongodata/mongod.lock is not empty.
2019-02-19T13:22:24.113+0800 I - [initandlisten] Detected data files in /usr/local/mongodb/data/mongodata created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-02-19T13:22:24.113+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten]
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3424M,session_max=20000,eviction= (threads_min=4,threads_max=4),config_base=false,statistics=(fast),log= (enabled=true,archive=true,path=journal,compressor=snappy),file_manager= (close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (0) [1550553744:602025][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 24576B block at offset 1671168: block header checksum of 3727800471 doesn't match expected checksum of 605632
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (0) [1550553744:602085][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 227)
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1550553744:602097][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-02-19T13:22:24.602+0800 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-02-19T13:22:24.602+0800 F - [initandlisten]
***aborting after fassert() failure
how to resolve this problem?
Open your config file in VIM or in a plain text editor,
got to this line:
storage:
dbPath: THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example:
/var/mongodb/db/mydb
Go to your terminal to confirm if the path under dbPath exists, type:
ls THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example for my case here:
ls /var/mongodb/db/mydb
If the result is directory does not exist, then just recreate it:
mkdir -p THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example:
mkdir -p /var/mongodb/db/mydb
If you found that the directory exists then it was corrupted during shutdown,as highlighted in the answere above.
Go to your terminal and run the following command:
mongod --dbpath /data/db --repair
Please Refer the above from the Mongo DB Offcial Docs
Please Try
mongod --repair --dbpath /path/to/data/db

MongoDB can not start by the command `service mongod start`

when i run the command sudo service mongod start the MongoDB cannot start and when i run the service mongod status it shows it has not been started, and the log shows:
2018-07-05T14:32:03.595+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] MongoDB starting : pid=15656 port=27017 dbpath=/var/lib/mongodb 64-bit host=iZbp1ik57sceo226yjpivxZ
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] db version v4.0.0-rc7
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] git version: 7230641bb09b1ceb04c3135cf83a5044c4838906
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] allocator: tcmalloc
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] modules: none
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] build environment:
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] distmod: ubuntu1604
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] distarch: x86_64
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] target_arch: x86_64
2018-07-05T14:32:03.605+0800 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-07-05T14:32:03.605+0800 I STORAGE [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-07-05T14:32:03.605+0800 I STORAGE [initandlisten]
2018-07-05T14:32:03.605+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-07-05T14:32:03.605+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-07-05T14:32:03.605+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=488M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-07-05T14:32:04.204+0800 E STORAGE [initandlisten] WiredTiger error (13) [1530772324:204155][15656:0x7fb5cb734a00], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1530772324:204155][15656:0x7fb5cb734a00], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied
2018-07-05T14:32:04.204+0800 E STORAGE [initandlisten] WiredTiger error (13) [1530772324:204458][15656:0x7fb5cb734a00], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1530772324:204458][15656:0x7fb5cb734a00], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied
2018-07-05T14:32:04.204+0800 E STORAGE [initandlisten] WiredTiger error (13) [1530772324:204610][15656:0x7fb5cb734a00], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied Raw: [1530772324:204610][15656:0x7fb5cb734a00], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied
2018-07-05T14:32:04.204+0800 F STORAGE [initandlisten] Failed to start up WiredTiger under any compatibility version.
2018-07-05T14:32:04.204+0800 F STORAGE [initandlisten] 13: Permission denied
2018-07-05T14:32:04.204+0800 F - [initandlisten] Fatal Assertion 28595 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 194
2018-07-05T14:32:04.204+0800 F - [initandlisten]
***aborting after fassert() failure
but when i tried mongod --auth -f /etc/mongod.conf the server can start and when visit the log file it shows that:
2018-07-05T14:35:34.228+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2018-07-05T14:35:34.239+0800 I CONTROL [initandlisten] MongoDB starting : pid=15665 port=27017 dbpath=/var/lib/mongodb 64-bit host=iZbp1ik57sceo226yjpivxZ
2018-07-05T14:35:34.239+0800 I CONTROL [initandlisten] db version v4.0.0-rc7
2018-07-05T14:35:34.239+0800 I CONTROL [initandlisten] git version: 7230641bb09b1ceb04c3135cf83a5044c4838906
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] allocator: tcmalloc
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] modules: none
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] build environment:
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] distmod: ubuntu1604
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] distarch: x86_64
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] target_arch: x86_64
2018-07-05T14:35:34.240+0800 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2018-07-05T14:35:34.240+0800 I STORAGE [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-07-05T14:35:34.240+0800 I STORAGE [initandlisten]
2018-07-05T14:35:34.240+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2018-07-05T14:35:34.240+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2018-07-05T14:35:34.240+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=488M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-07-05T14:35:35.005+0800 I STORAGE [initandlisten] WiredTiger message [1530772535:5560][15665:0x7f0ab9512a00], txn-recover: Main recovery loop: starting at 18/8960
2018-07-05T14:35:35.113+0800 I STORAGE [initandlisten] WiredTiger message [1530772535:113878][15665:0x7f0ab9512a00], txn-recover: Recovering log 18 through 19
2018-07-05T14:35:35.179+0800 I STORAGE [initandlisten] WiredTiger message [1530772535:179582][15665:0x7f0ab9512a00], txn-recover: Recovering log 19 through 19
2018-07-05T14:35:35.248+0800 I STORAGE [initandlisten] WiredTiger message [1530772535:248647][15665:0x7f0ab9512a00], txn-recover: Set global recovery timestamp: 0
2018-07-05T14:35:35.283+0800 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2018-07-05T14:35:35.311+0800 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-07-05T14:35:35.311+0800 I CONTROL [initandlisten]
2018-07-05T14:35:35.312+0800 I CONTROL [initandlisten]
2018-07-05T14:35:35.312+0800 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-07-05T14:35:35.312+0800 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-07-05T14:35:35.312+0800 I CONTROL [initandlisten]
2018-07-05T14:35:35.312+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 7856 processes, 65535 files. Number of processes should be at least 32767.5 : 0.5 times number of files.
2018-07-05T14:35:35.312+0800 I CONTROL [initandlisten]
2018-07-05T14:35:35.336+0800 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
2018-07-05T14:35:35.337+0800 I NETWORK [initandlisten] waiting for connections on port 27017
and the whole config file is as below:
# mongod.conf
# Where and how to store data.
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
# engine:
# mmapv1:
# wiredTiger:
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# network interfaces
net:
port: 27017
bindIp: 0.0.0.0
# how the process runs
processManagement:
timeZoneInfo: /usr/share/zoneinfo
#security:
security:
authorization: enabled
#operationProfiling:
#replication:
#sharding:
## Enterprise-Only Options:
#auditLog:
#snmp:
So, what is the matter with it? why the first method can not run it but the second method can start it?
According to #Markus W Mahlberg, and after run these command, the problem has been solved.
[sudo] chown mongodb.mongodb -R /var/lib/mongodb
If you ran the command mongod as root on the server while testing your configuration file, as in:
$ mongod -f /etc/mongod.conf
The mongo server will generate the mongo.lock file and the WiredTiger.lock files and further files in the mongodb folder with root:root ownership as in:
-rw------- 1 mongodb mongodb 36864 Nov 25 12:03 index-95--8062076833940689665.wt
-rw------- 1 mongodb mongodb 45056 Nov 25 12:03 index-96--8062076833940689665.wt
-rw------- 1 mongodb mongodb 36864 Nov 25 12:03 index-99--8062076833940689665.wt
drwx------ 2 mongodb mongodb 4096 Nov 25 15:23 journal/
-rw------- 1 mongodb mongodb 45056 Nov 25 15:32 _mdb_catalog.wt
-rw------- 1 root root 0 Nov 25 15:32 mongod.lock
-rw------- 1 mongodb mongodb 36864 Nov 25 15:32 sizeStorer.wt
-rw------- 1 mongodb mongodb 114 Oct 17 17:41 storage.bson
-rw------- 1 mongodb mongodb 44 Oct 17 17:41 WiredTiger
-rw------- 1 mongodb mongodb 4096 Nov 25 15:32 WiredTigerLAS.wt
-rw------- 1 root root 21 Nov 25 15:22 WiredTiger.lock
-rw------- 1 mongodb mongodb 1201 Nov 25 15:32 WiredTiger.turtle
-rw------- 1 mongodb mongodb 188416 Nov 25 15:32 WiredTiger.wt
The service is most likely configured to run mongod with the mongodb user. which will not have permission to remove, or overwrite the WiredTiger.lock nor the mongod.lock or any other file that were created when mongod was running using the root user. In the case above the /var/lib/mongodb/WiredTiger.turtle
The solution is to reset the ownership of all the files in the folder in the case above the /var/lib/mongodb/ folder to the mongodb user:
$ sudo chown -R mongodb:mongodb /var/lib/mongodb/
That should solve the issue.
On a mac OS
Browse to the data directory you configured when installing mongodb. For ex: data/db where the mongod.lock and WiredTiger.lock are present
sudo chown -R {username}:{groupname} /data/db
Make sure you have stopped any mongo processes
run 'mongod' command
You should now see the prompt which is now ready to accept connections.

Mongod dies upon startup with Fatal Assertion 16252

MongoDB continues to crash upon on a VM I've built. Googled the fatal assertion number to no avail. Any pointers would be appreciated...
Details:
On a VM system running Ubuntu 14.04.
Latest stable mongo packages from 10gen:
# dpkg -l | grep mongo
ii libmongo-client0:amd64 0.1.7-1 amd64 Alternate C driver for the MongoDB document-oriented datastore
ii mongodb-org 2.6.3 amd64 MongoDB open source document-oriented database system (metapackage)
ii mongodb-org-mongos 2.6.3 amd64 MongoDB sharded cluster query router
ii mongodb-org-server 2.6.3 amd64 MongoDB database server
ii mongodb-org-shell 2.6.3 amd64 MongoDB shell client
ii mongodb-org-tools 2.6.3 amd64 MongoDB tools
Plenty of free disk.
Log from /var/log/mongodb/mongod.log:
2014-06-30T16:11:14.474-0600 ***** SERVER RESTARTED *****
2014-06-30T16:11:14.479-0600 [initandlisten] MongoDB starting : pid=27406 port=27017 dbpath=/var/lib/mongodb 64-bit host=foobar
2014-06-30T16:11:14.479-0600 [initandlisten] db version v2.6.3
2014-06-30T16:11:14.479-0600 [initandlisten] git version: 255f67a66f9603c59380b2a389e386910bbb52cb
2014-06-30T16:11:14.479-0600 [initandlisten] build info: Linux build12.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2014-06-30T16:11:14.479-0600 [initandlisten] allocator: tcmalloc
2014-06-30T16:11:14.479-0600 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2014-06-30T16:11:14.481-0600 [initandlisten] journal dir=/var/lib/mongodb/journal
2014-06-30T16:11:14.482-0600 [initandlisten] recover begin
2014-06-30T16:11:14.482-0600 [initandlisten] info no lsn file in journal/ directory
2014-06-30T16:11:14.482-0600 [initandlisten] recover lsn: 0
2014-06-30T16:11:14.482-0600 [initandlisten] recover /var/lib/mongodb/journal/j._0
2014-06-30T16:11:14.498-0600 [initandlisten] recover cleaning up
2014-06-30T16:11:14.498-0600 [initandlisten] removeJournalFiles
2014-06-30T16:11:14.500-0600 [initandlisten] recover done
2014-06-30T16:11:14.720-0600 [initandlisten] waiting for connections on port 27017
2014-06-30T16:11:14.751-0600 [IndexRebuilder] Fatal Assertion 16252
2014-06-30T16:11:14.758-0600 [IndexRebuilder] 0x11da801 0x117c9f9 0x115f43d 0xa1a7cf 0xa1ccab 0xa1ced3 0x8f4ad0 0xb89da3 0xb8ac78 0x11625f2 0x121f219 0x7fad84a56182 0x7fad83d5b30d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11da801]
/usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x117c9f9]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xcd) [0x115f43d]
/usr/bin/mongod(_ZN5mongo4Lock7DBWrite9lockOtherERKNS_10StringDataE+0xa6f) [0xa1a7cf]
/usr/bin/mongod(_ZN5mongo4Lock7DBWrite6lockDBERKSs+0x17b) [0xa1ccab]
/usr/bin/mongod(_ZN5mongo4Lock7DBWriteC1ERKNS_10StringDataE+0x63) [0xa1ced3]
/usr/bin/mongod(_ZN5mongo6Client12WriteContextC1ERKSsS3_+0x40) [0x8f4ad0]
/usr/bin/mongod(_ZN5mongo14IndexRebuilder7checkNSERKSt4listISsSaISsEE+0xd3) [0xb89da3]
/usr/bin/mongod(_ZN5mongo14IndexRebuilder3runEv+0x188) [0xb8ac78]
/usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xd2) [0x11625f2]
/usr/bin/mongod() [0x121f219]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fad84a56182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fad83d5b30d]
2014-06-30T16:11:14.758-0600 [IndexRebuilder]
***aborting after fassert() failure
Looks like a corrupted journal file or entire database (not sure how that happened?).
move the database files, and restored from a backup dump and all appears to be good.
This happened again after a apt-get upgrade. Wonder if the install script doesn't properly shutdown the database before upgrading, thus corrupting the journal?