I recently had some storage issues on my PC so I changed the mongo.conf file to reduce the size of the journal. These are the lines which I put in.
mmapv1:
smallFiles: true
However, every time when I try to run mongo with the new changes I get this error(without the change it works just fine). Can you tell me what's the problem?
2016-02-11T22:32:14.002+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2016-02-11T22:32:14.023+0000 W FTDC [ftdc] Uncaught exception in 'UnknownError Caught std::exception of type boost::filesystem::filesystem_error: boost::filesystem::file_size: No such file or directory: "/data/db/diagnostic.data/metrics.interim.temp"' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
I would try deleting the directory /data/db/diagnostic.data.
You can also disable the diagnostic data collection by starting mongo with the option:
setParameter:
diagnosticDataCollectionEnabled: false
I had the same problem after MongoDb failed when HD was fulled upto 100%. Here is the log:
2016-10-03T10:13:42.017+0000 I CONTROL [initandlisten] MongoDB starting : pid=12630 port=27017 dbpath=/var/lib/mongo 64-bit host=drft006
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] db version v3.2.9
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] modules: none
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] build environment:
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] distmod: rhel62
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] distarch: x86_64
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] target_arch: x86_64
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017, unixDomainSocket: { filePermissions: 511 } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-10-03T10:13:42.045+0000 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-10-03T10:13:42.045+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2016-10-03T10:13:42.045+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2016-10-03T10:13:42.045+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=93G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-10-03T10:13:42.465+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2016-10-03T10:13:42.465+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-10-03T10:13:42.467+0000 I NETWORK [initandlisten] waiting for connections on port 27017
2016-10-03T10:13:43.012+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2016-10-03T10:13:43.045+0000 W FTDC [ftdc] Uncaught exception in 'UnknownError: Caught std::exception of type boost::filesystem::filesystem_error: boost::filesystem::file_size: No such file or directory: "/var/lib/mongo/diagnostic.data/metrics.interim.temp"' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
I kill-ed mongod and cleared the contents of the diagnostic folders:
> rm -f /var/lib/mongo/diagnostic.data/*
After the restart of mongo log showed:
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] MongoDB starting : pid=12803 port=27017 dbpath=/var/lib/mongo 64-bit host=drft006
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] db version v3.2.9
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] modules: none
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] build environment:
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] distmod: rhel62
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] distarch: x86_64
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] target_arch: x86_64
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017, unixDomainSocket: { filePermissions: 511 } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-10-03T10:17:46.772+0000 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-10-03T10:17:46.772+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2016-10-03T10:17:46.772+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2016-10-03T10:17:46.772+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=93G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-10-03T10:17:47.128+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2016-10-03T10:17:47.128+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-10-03T10:17:47.129+0000 I NETWORK [initandlisten] waiting for connections on port 27017
and mongo was ok.
In case you are using nodejs, try to disable/remove debug mode in your server code.
Steps:
rm -rf /{your_db_name}/diagnostic.data/
remove or comment out mongoose.set('debug', true);
restart mongo and node server
In my case I use docker-compose. First I remove the diagnostic.data folder from my mounted volume:
sudo rm -rf /srv/mongo/diagnostic.data
And then when I use docker-compose up the diagnostic.data folder will be re-created and everything was working again.
Related
mongodb was closed by some reason. system shutdown abnormally.
when i try to restart mongodb use:
mongod -f /usr/local/mongodb/etc/mongo.conf
got
about to fork child process, waiting until server is ready for connections.
forked process: 28402
ERROR: child process failed, exited with error number 51
To see additional information in this output, start without the "--fork" option.
mongodb.log:
2019-02-19T13:22:24.105+0800 I CONTROL [main] ***** SERVER RESTARTED *****
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] MongoDB starting : pid=28131 port=27017 dbpath=/usr/local/mongodb/data/mongodata 64-bit host=nginx.xchs
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] db version v3.6.2
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] git version: 489d177dbd0f0420a8ca04d39fd78d0a2c539420
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] allocator: tcmalloc
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] modules: none
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] build environment:
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] distmod: rhel62
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] distarch: x86_64
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] target_arch: x86_64
2019-02-19T13:22:24.112+0800 I CONTROL [initandlisten] options: { config: "/usr/local/mongodb/etc/mongo.conf", net: { bindIp: "0.0.0.0" }, processManagement: { fork: true }, storage: { dbPath: "/usr/local/mongodb/data/mongodata" }, systemLog: { destination: "file", logAppend: true, path: "/usr/local/mongodb/data/log/mongolog/mongodb.log" } }
2019-02-19T13:22:24.112+0800 W - [initandlisten] Detected unclean shutdown - /usr/local/mongodb/data/mongodata/mongod.lock is not empty.
2019-02-19T13:22:24.113+0800 I - [initandlisten] Detected data files in /usr/local/mongodb/data/mongodata created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-02-19T13:22:24.113+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten]
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-02-19T13:22:24.113+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=3424M,session_max=20000,eviction= (threads_min=4,threads_max=4),config_base=false,statistics=(fast),log= (enabled=true,archive=true,path=journal,compressor=snappy),file_manager= (close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (0) [1550553744:602025][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: read checksum error for 24576B block at offset 1671168: block header checksum of 3727800471 doesn't match expected checksum of 605632
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (0) [1550553744:602085][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: WiredTiger.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 227)
2019-02-19T13:22:24.602+0800 E STORAGE [initandlisten] WiredTiger error (-31804) [1550553744:602097][28131:0x7f41a84cea80], file:WiredTiger.wt, WT_CURSOR.next: the process must exit and restart: WT_PANIC: WiredTiger library panic
2019-02-19T13:22:24.602+0800 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2019-02-19T13:22:24.602+0800 F - [initandlisten]
***aborting after fassert() failure
how to resolve this problem?
Open your config file in VIM or in a plain text editor,
got to this line:
storage:
dbPath: THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example:
/var/mongodb/db/mydb
Go to your terminal to confirm if the path under dbPath exists, type:
ls THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example for my case here:
ls /var/mongodb/db/mydb
If the result is directory does not exist, then just recreate it:
mkdir -p THE_PATH_TO_THE_DIRECTORY_YOU_STORE_YOUR_DB
for example:
mkdir -p /var/mongodb/db/mydb
If you found that the directory exists then it was corrupted during shutdown,as highlighted in the answere above.
Go to your terminal and run the following command:
mongod --dbpath /data/db --repair
Please Refer the above from the Mongo DB Offcial Docs
Please Try
mongod --repair --dbpath /path/to/data/db
What is the difference between these two mongo data folders?
I have my mongod dbpath pointed to C:/data/db, however I also noticed I have a data folder in C:\Program Files\MongoDB\Server\4.0\ which also contains indexes and collections.
Only reason I noticed this is because I got an unclean shutdown and lost 177 million docs so was trying to look closer at the issue (side note: was not able to recover even though mongo could rebuild the index for that missing collection)
Edit
Here is a log file from the server folder where I could see a dbpath pointing elsewhere
**aborting after fassert() failure
2018-11-18T11:40:34.164+0100 I CONTROL [main] ***** SERVER RESTARTED *****
2018-11-18T11:40:36.984+0100 I CONTROL [main] Trying to start Windows service 'MongoDB'
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] MongoDB starting : pid=4252 port=27017 dbpath=C:\Program Files\MongoDB\Server\4.0\data 64-bit host=DESKTOP-IO5LL15
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] db version v4.0.4
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] git version: f288a3bdf201007f3693c58e140056adf8b04839
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] allocator: tcmalloc
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] modules: none
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] build environment:
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] distmod: 2008plus-ssl
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] distarch: x86_64
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] target_arch: x86_64
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] options: { config: "C:\Program Files\MongoDB\Server\4.0\bin\mongod.cfg", net: { bindIp: "127.0.0.1", port: 27017 }, service: true, storage: { dbPath: "C:\Program Files\MongoDB\Server\4.0\data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "C:\Program Files\MongoDB\Server\4.0\log\mongod.log" } }
2018-11-18T11:40:37.099+0100 W STORAGE [initandlisten] Detected unclean shutdown - C:\Program Files\MongoDB\Server\4.0\data\mongod.lock is not empty.
2018-11-18T11:40:37.100+0100 I STORAGE [initandlisten] Detected data files in C:\Program Files\MongoDB\Server\4.0\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-11-18T11:40:37.100+0100 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I tried to start up my computer's mongodb service and it wouldn't start. I spent over an hour trying to debug it during which time I tried many solutions. Trying all of them may have messed up my mongo install, but I tried reinstalling mongo to get a fresh start. That didn't work either. I now have the following error and cannot find anything on google about how to fix it.
2017-06-01T16:57:02.339-0400 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] MongoDB starting : pid=15070 port=27017 dbpath=/var/lib/mongodb 64-bit host=moss-greenelab
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] db version v3.2.13
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] git version: 23899209cad60aaafe114f6aea6cb83025ff51bc
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] modules: none
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] build environment:
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] distmod: ubuntu1604
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] distarch: x86_64
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] target_arch: x86_64
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2017-06-01T16:57:02.361-0400 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-01T16:57:02.361-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-01T16:57:02.364-0400 E STORAGE [initandlisten] WiredTiger (13) [1496350622:364565][15070:0x7f80e0292c80], connection: /var/lib/mongodb/journal/WiredTigerPreplog.0000000002: file-remove: unlink: Permission denied
2017-06-01T16:57:02.383-0400 I - [initandlisten] Assertion: 28595:13: Permission denied
2017-06-01T16:57:02.383-0400 I STORAGE [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2017-06-01T16:57:02.383-0400 I CONTROL [initandlisten] dbexit: rc: 100
I'm running Ubuntu 16.04.
Here's the telling part of the log:
/var/lib/mongodb/journal/WiredTigerPreplog.0000000002: file-remove: unlink: Permission denied
So have a look at ls -la /var/lib/mongodb/journal and see what the permissions on those files are. I'm betting that those (an probably others) are not properly owned by the correct user, which is normally mongodb on Ubuntu.
Assuming you've got an issue with ownership there, you can just run chown -R mongodb:mongodb /var/lib/mongodb.
If that doesn't work, as a last resort you can always just wipe out your data directory (assuming you don't need your data!) by running rm -rf /var/lib/mongodb/*.
my instance running in docker. after close it uncleanly, it can't be started. with log below
I CONTROL [initandlisten] MongoDB starting : pid=13 port=27017
dbpath=/data/db 64-bit host=94a9490feef9ce51e9e55af47ab5c357-a0000
I CONTROL [initandlisten] db version v3.2.6
I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
I CONTROL [initandlisten] allocator: tcmalloc
I CONTROL [initandlisten] modules: none
I CONTROL [initandlisten] build environment:
I CONTROL [initandlisten] distmod: debian71
I CONTROL [initandlisten] distarch: x86_64
I CONTROL [initandlisten] target_arch: x86_64
I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, security: { authorization: "disabled" }, storage: { dbPath: "/data/db", directoryPerDB: true, journal: { enabled: true } } }
I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=150G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
I - [initandlisten] Assertion: 28595:2: No such file or directory
I STORAGE [initandlisten] exception in initAndListen: 28595 2: No such file or directory, terminating
I CONTROL [initandlisten] dbexit: rc: 100
which file can't be found is not show, how to deal next?
I have a website ( Analytics Service ) for website and i have about +3K concurrent user .. i'm using php with MongoDB
MongoDB architecture is ( 1 Query , 3 Config server , 2 Shard ) . i'm sure that my mongo configuration is working fine . but shard 1 stopped suddenly with no reason. and when i tried to run sudo services mongod restart mongo doesn't run. when i view the log file i get the last lines
2016-10-15T09:35:52.334-0400 I CONTROL [main] ***** SERVER RESTARTED *****
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] MongoDB starting : pid=2168 port=27017 dbpath=/var/lib/mongodb 64-bit host=Analytics-DB-shard-01
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] db version v3.2.8
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d15957sdaaf0
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] allocator: tcmalloc
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] modules: none
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] build environment:
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] distmod: ubuntu1404
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] distarch: x86_64
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] target_arch: x86_64
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-10-15T09:35:52.373-0400 E NETWORK [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted
2016-10-15T09:35:52.373-0400 I - [initandlisten] Fatal Assertion 28578
2016-10-15T09:35:52.373-0400 I - [initandlisten]
***aborting after fassert() failure
i can't figure out mongo problems but i'm trying to manage it to work correctly as much as possible .. any idea how to make sure that it won't dropped again ?