Mongodb ERROR: child process failed, exited with error number 51 - mongodb

mongodb was closed by some reason. system shutdown abnormally.
when i try to restart mongodb use:
mongod -f /usr/local/mongodb/etc/mongo.conf
got
about to fork child process, waiting until server is ready for connections.
forked process: 28402
ERROR: child process failed, exited with error number 51
To see additional information in this output, start without the "--fork" option.
mongodb.log:
2019-02-19T13:22:24.105+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] MongoDB starting : pid=28131 port=27017 dbpath=/usr/local/mongodb/data/mongodata 64-bit host=nginx.xchs
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] db version v3.6.2
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] git version: 489d177dbd0f0420a8ca04d39fd78d0a2c539420
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] modules: none
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] build environment:
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] distmod: rhel62
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] distarch: x86_64
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] options: { config: "/usr/local/mongodb/etc/mongo.conf", net: { bindIp: "0.0.0.0" }, processManagement: { fork: true }, storage: { dbPath: "/usr/local/mongodb/data/mongodata" }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/mongodb/data/log/mongolog/mongodb.log" } }
2019-02-19T13:22:24.112+0800 W - [initandlisten] Detected unclean shutdown - /usr/local/mongodb/data/mongodata/mongod.lock is not empty.
2019-02-19T13:22:24.113+0800 I - [initandlisten] Detected data files in /usr/local/mongodb/data/mongodata created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-02-19T13:22:24.113+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten]
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3424M,session_max=20000,eviction= (threads_min=4,threads_max=4),config_base=false,statistics=(fast),log= (enabled=true,archive=true,path=journal,compressor=snappy),file_manager= (close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (0) [1550553744:602025][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 24576B block at offset 1671168: block header checksum of 3727800471 doesn't match expected checksum of 605632
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (0) [1550553744:602085][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 227)
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1550553744:602097][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-02-19T13:22:24.602+0800 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-02-19T13:22:24.602+0800 F - [initandlisten]
***aborting after fassert() failure
how to resolve this problem?

Open your config file in VIM or in a plain text editor,
got to this line:
storage:
dbPath: THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example:
/var/mongodb/db/mydb
Go to your terminal to confirm if the path under dbPath exists, type:
ls THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example for my case here:
ls /var/mongodb/db/mydb
If the result is directory does not exist, then just recreate it:
mkdir -p THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example:
mkdir -p /var/mongodb/db/mydb
If you found that the directory exists then it was corrupted during shutdown,as highlighted in the answere above.
Go to your terminal and run the following command:
mongod --dbpath /data/db --repair
Please Refer the above from the Mongo DB Offcial Docs

Please Try
mongod --repair --dbpath /path/to/data/db

Related

MongoDB repair failed due to Invariant failure rs.get() src/mongo/db/catalog/database.cpp

MongoDB Version: 3.4.24
MongoDB hosted on Linux server was abruptly shut down due to the over-utilization of memory.
Initiated the mongodb repair using: sudo mongod -f /etc/mongodrepair.conf --repair
The whole database is of 2.5TB, while repairing/re-indexing the db, close to 2.4TB was successfully repaired but the last 972MB of DB repair was failed due to Invariantfailure error.
Repair Logs
2020-07-04T17:17:07.441+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] build index on: test.summary properties: { v: 1, key: { totalVolume: -1 }, name: "totalV$
lume_-1", ns: "test.summary", background: true }
2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] build index on: test.summary properties: { v: 1, key: { ts: -1 }, name: "ts_-1", ns: "test.summary", background: true }
2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$
2020-07-04T17:17:08.673+0000 I - [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 195
2020-07-04T17:17:08.673+0000 I - [initandlisten]
***aborting after invariant() failure
2020-07-04T17:17:08.717+0000 F - [initandlisten] Got signal: 6 (Aborted).
Restart Logs
2020-07-04T17:39:14.476+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] MongoDB starting : pid=20485 port=27017 dbpath=/home/db324 64-bit host=ip-*-*-*-*
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] db version v3.4.24
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] modules: none
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] build environment:
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] distarch: x86_64
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "-.-.-.-", port: 27017 }, replication: $
oplogSizeMB: 10240, replSetName: "rs1" }, storage: { dbPath: "/home/db324", directoryPerDB: true, engine: "wiredTiger", journal: { enabled$
true }, wiredTiger: { engineConfig: { cacheSizeGB: 108.0 } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.lo$
" } }
2020-07-04T17:39:14.480+0000 W - [initandlisten] Detected unclean shutdown - /home/db324/mongod.lock is not empty.
2020-07-04T17:39:14.499+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten]
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] ** WARNING: The configured WiredTiger cache size is more than 80% of available RAM.
2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=110592M,session_max=20000,eviction=(threads_min=4,t$
reads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000)$
checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress),
2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667272][20485:0x7fdbcb287580], txn-recover: Main recovery loop$
starting at 73368/128
2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667951][20485:0x7fdbcb287580], txn-recover: Recovering log 733$
8 through 73369
2020-07-04T17:39:14.733+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:733044][20485:0x7fdbcb287580], txn-recover: Recovering log 733$
9 through 73369
2020-07-04T17:39:15.164+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1593884355:164908][20485:0x7fdbcb287580], test/collectio$
-56-3854974571131417844.wt, WT_SESSION.open_cursor: /home/db324/test/collection-56-3854974571131417844.wt: handle-read: pread: failed $
to read 4096 bytes at offset 28672: WT_ERROR: non-specific WiredTiger error
2020-07-04T17:39:15.164+0000 I - [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredT$
ger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 113
2020-07-04T17:39:15.165+0000 I - [initandlisten]
***aborting after invariant() failure
Is there a way to repair/recover the last part of the DB? or
Is there a way to ignore the broken DB? or
Is it possible to cut out the whole 2.4TB data sans the last error db and create a new MongoDB instance with 2.4TB?
I would greatly appreciate the help.
Thanks in Advance
The repair log indicates that it failed building an index on ns: "test.summary",
The other log gives you the file name, and the offset of the error:
/home/db324/test/collection-56-3854974571131417844.wt: handle-read: pread: failed $
to read 4096 bytes at offset 28672
The data after that point in the file is likely unsalvageable. You might try:
Make a backup of the existing files
Remove the file /home/db324/test/collection-56-3854974571131417844.wt
Re-run mongod --repair on this dbpath
If all goes well, it will create a new, empty file for that collection.
If you need to try to salvage that data, after the above succeeds so that you know the rest of the data files are consistent, recopy that file from the backup and attempt repair again.

How to fix 'Sessions collection is not set up' error when trying to convert existing mongodb instance to replica set

I have installed the latest version of mongodb (v4.0.5) using the following link
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-red-hat/
Following this now I'm trying to set up replication using the instructions provided in the following tutorial
But as soon as i try the command at point "2.Start your mongod instances in their own shell windows by issuing the following commands"
it gives me the following error
"Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured"
Command I used: mongod --replSet rs0 --port 27019 --bind_ip localhost,[MY_IP] --dbpath /srv/mongodb/rs0-2 --smallfiles --oplogSize 128
A more detailed output for the command is as follows:
2018-12-31T15:37:03.618+0530 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] MongoDB starting : pid=xxx port=27017 dbpath=/srv/mongodb/rs0-0 64-bit host=xxx
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] db version v4.0.5
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] git version: xxx
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] allocator: tcmalloc
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] modules: none
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] build environment:
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] distmod: rhel70
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] distarch: x86_64
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] target_arch: x86_64
2018-12-31T15:37:03.623+0530 I CONTROL [initandlisten] options: { net: { bindIp: "localhost,xxx", port: 27017 }, replication: { replSet: "rs0" }, storage: { dbPath: "/srv/mongodb/rs0-0" } }
2018-12-31T15:37:03.624+0530 I STORAGE [initandlisten] Detected data files in /srv/mongodb/rs0-0 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-12-31T15:37:03.625+0530 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3399M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-12-31T15:37:04.683+0530 I STORAGE [initandlisten] WiredTiger message [1546250824:683476][30064:0x7f4cb133db40], txn-recover: Main recovery loop: starting at 5/6144 to 6/256
2018-12-31T15:37:04.796+0530 I STORAGE [initandlisten] WiredTiger message [1546250824:796244][30064:0x7f4cb133db40], txn-recover: Recovering log 5 through 6
2018-12-31T15:37:04.871+0530 I STORAGE [initandlisten] WiredTiger message [1546250824:871037][30064:0x7f4cb133db40], txn-recover: Recovering log 6 through 6
2018-12-31T15:37:04.945+0530 I STORAGE [initandlisten] WiredTiger message [1546250824:945584][30064:0x7f4cb133db40], txn-recover: Set global recovery timestamp: 0
2018-12-31T15:37:04.964+0530 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2018-12-31T15:37:04.975+0530 I CONTROL [initandlisten]
2018-12-31T15:37:04.975+0530 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-12-31T15:37:04.975+0530 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2018-12-31T15:37:04.975+0530 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-12-31T15:37:04.975+0530 I CONTROL [initandlisten]
2018-12-31T15:37:04.975+0530 I CONTROL [initandlisten]
2018-12-31T15:37:04.976+0530 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-12-31T15:37:04.976+0530 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-12-31T15:37:04.976+0530 I CONTROL [initandlisten]
2018-12-31T15:37:04.976+0530 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2018-12-31T15:37:04.976+0530 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2018-12-31T15:37:04.976+0530 I CONTROL [initandlisten]
2018-12-31T15:37:04.995+0530 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/srv/mongodb/rs0-0/diagnostic.data'
2018-12-31T15:37:04.996+0530 I REPL [initandlisten] Did not find local voted for document at startup.
2018-12-31T15:37:04.997+0530 I REPL [initandlisten] Rollback ID is 1
2018-12-31T15:37:04.997+0530 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2018-12-31T15:37:04.997+0530 I CONTROL [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured
2018-12-31T15:37:04.997+0530 I NETWORK [initandlisten] waiting for connections on port 27017
2018-12-31T15:37:04.998+0530 I CONTROL [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: config.system.sessions does not exist
I tried following other tutorials where the commands were without the --ip_bind option but i still faced the same error.
I tried searching on the net for this specific error but have not come across any solution for my error.
You need to initiate it with rs.initiate().
Once you launched as you did, you can run this from an other terminal
mongo --eval 'rs.initiate();'
If you are familiar with docker, you can reproduce the error and the initialisation:
docker run --rm -it mongo:4.0 /bin/bash -c "\
mongod --replSet rs0 --fork --logpath /var/log/mongodb/mongod.log &&\
sleep 5 &&\
mongo --eval 'rs.initiate();' &&\
tail -n +0 -f /var/log/mongodb/mongod.log"
I used the --fork to run mongod as a deamon, in order to be able to call the rs.initiate() a bit later

mongo: “exception: connect failed”

I am trying to connect to the mongo server, using the "mongo" command in the command prompt and this error is showing: "exception: connect failed"
Here is how I am trying to do it:
enter image description here
This is the mongodb.log
2018-03-27T14:00:19.299+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2018-03-27T14:00:19.647+0800 I CONTROL [main] Trying to start Windows service 'MongoDB'
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] MongoDB starting : pid=10480 port=27017 dbpath=d:\mongodbData\db 64-bit host=DESKTOP-KEA31I3
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] db version v3.6.3
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1u-fips 22 Sep 2016
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] allocator: tcmalloc
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] modules: none
2018-03-27T14:00:19.648+0800 I CONTROL [initandlisten] build environment:
2018-03-27T14:00:19.649+0800 I CONTROL [initandlisten] distmod: 2008plus-ssl
2018-03-27T14:00:19.649+0800 I CONTROL [initandlisten] distarch: x86_64
2018-03-27T14:00:19.649+0800 I CONTROL [initandlisten] target_arch: x86_64
2018-03-27T14:00:19.649+0800 I CONTROL [initandlisten] options: { service: true, storage: { dbPath: "d:\mongodbData\db" }, systemLog: { destination: "file", logAppend: true, path: "d:\mongodbData\log\mongodb.log" } }
2018-03-27T14:00:19.651+0800 I - [initandlisten] Detected data files in d:\mongodbData\db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-03-27T14:00:19.651+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7632M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2018-03-27T14:00:19.834+0800 I STORAGE [initandlisten] WiredTiger message [1522130419:833660][10480:140722874245456], txn-recover: Main recovery loop: starting at 4/6272
2018-03-27T14:00:19.970+0800 I STORAGE [initandlisten] WiredTiger message [1522130419:970024][10480:140722874245456], txn-recover: Recovering log 4 through 5
2018-03-27T14:00:20.046+0800 I STORAGE [initandlisten] WiredTiger message [1522130420:46191][10480:140722874245456], txn-recover: Recovering log 5 through 5
2018-03-27T14:00:20.172+0800 I CONTROL [initandlisten]
2018-03-27T14:00:20.172+0800 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-03-27T14:00:20.172+0800 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2018-03-27T14:00:20.172+0800 I CONTROL [initandlisten]
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten] ** Start the server with --bind_ip <address> to specify which IP
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten]
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten]
2018-03-27T14:00:20.173+0800 I CONTROL [initandlisten] ** WARNING: The file system cache of this machine is configured to be greater than 40% of the total memory. This can lead to increased memory pressure and poor performance.
2018-03-27T14:00:20.174+0800 I CONTROL [initandlisten] See http://dochub.mongodb.org/core/wt-windows-system-file-cache
2018-03-27T14:00:20.174+0800 I CONTROL [initandlisten]
2018-03-27T14:00:20.402+0800 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory 'd:/mongodbData/db/diagnostic.data'
2018-03-27T14:00:20.404+0800 I NETWORK [initandlisten] waiting for connections on port 27017
2018-03-27T14:00:20.404+0800 I STORAGE [initandlisten] Service running
Please help me. How to fix it
Run this following command :
sudo service mongodb stop
sudo rm /var/lib/mongodb/mongod.lock
sudo service mongod restart
mongo

MongoDB server running, I get "mongo: command not found"

I`ve started mongoDB and it seems like its running fine, I think (see message below)? Then I open a new terminal window and I try to use the "mongo" command I get the following message:
-bash: mongo: command not found
When I try commands such as "use newtest", it says
-bash: use: command not found
My Path:
PATH=/Users/myname/.npm-global/bin:$PATH
Here is the message when I start mongoDB:
myname-MacBook-Air:~ myname$ mongod
I CONTROL [initandlisten] MongoDB starting : pid=581 port=27017 dbpath=/data/db 64-bit host=myname-MacBook-Air.local
I CONTROL [initandlisten] db version v3.6.2
I CONTROL [initandlisten] git version: 489d177dbd0f0420a8ca04d39fd78d0a2c539420
I CONTROL [initandlisten] OpenSSL version: OpenSSL 0.9.8zh 14 Jan 2016
I CONTROL [initandlisten] allocator: system
I CONTROL [initandlisten] modules: none
I CONTROL [initandlisten] build environment:
I CONTROL [initandlisten] distarch: x86_64
I CONTROL [initandlisten] target_arch: x86_64
I CONTROL [initandlisten] options: {}
W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3584M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
I STORAGE [initandlisten] WiredTiger message [1519475976:405758][581:0x7fff95b02340], txn-recover: Main recovery loop: starting at 4/3200
I STORAGE [initandlisten] WiredTiger message [1519475976:408878][581:0x7fff95b02340], txn-recover: Recovering log 4 through 5
I STORAGE [initandlisten] WiredTiger message [1519475976:564148][581:0x7fff95b02340], txn-recover: Recovering log 5 through 5
I CONTROL [initandlisten]
I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
I CONTROL [initandlisten]
I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
I CONTROL [initandlisten] ** Start the server with --bind_ip <address> to specify which IP
I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
I CONTROL [initandlisten]
I CONTROL [initandlisten]
I CONTROL [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
I NETWORK [initandlisten] waiting for connections on port 27017
I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
What am I doing wrong here to start mongodB? Or am I misunderstanding it?
Thanks :)

Mongo won't start. Getting cryptic Assertion: 28595:13: Permission denied error

I tried to start up my computer's mongodb service and it wouldn't start. I spent over an hour trying to debug it during which time I tried many solutions. Trying all of them may have messed up my mongo install, but I tried reinstalling mongo to get a fresh start. That didn't work either. I now have the following error and cannot find anything on google about how to fix it.
2017-06-01T16:57:02.339-0400 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] MongoDB starting : pid=15070 port=27017 dbpath=/var/lib/mongodb 64-bit host=moss-greenelab
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] db version v3.2.13
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] git version: 23899209cad60aaafe114f6aea6cb83025ff51bc
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] modules: none
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] build environment:
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] distmod: ubuntu1604
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] distarch: x86_64
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] target_arch: x86_64
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2017-06-01T16:57:02.361-0400 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-01T16:57:02.361-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-01T16:57:02.364-0400 E STORAGE [initandlisten] WiredTiger (13) [1496350622:364565][15070:0x7f80e0292c80], connection: /var/lib/mongodb/journal/WiredTigerPreplog.0000000002: file-remove: unlink: Permission denied
2017-06-01T16:57:02.383-0400 I - [initandlisten] Assertion: 28595:13: Permission denied
2017-06-01T16:57:02.383-0400 I STORAGE [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2017-06-01T16:57:02.383-0400 I CONTROL [initandlisten] dbexit: rc: 100
I'm running Ubuntu 16.04.
Here's the telling part of the log:
/var/lib/mongodb/journal/WiredTigerPreplog.0000000002: file-remove: unlink: Permission denied
So have a look at ls -la /var/lib/mongodb/journal and see what the permissions on those files are. I'm betting that those (an probably others) are not properly owned by the correct user, which is normally mongodb on Ubuntu.
Assuming you've got an issue with ownership there, you can just run chown -R mongodb:mongodb /var/lib/mongodb.
If that doesn't work, as a last resort you can always just wipe out your data directory (assuming you don't need your data!) by running rm -rf /var/lib/mongodb/*.