I have a website ( Analytics Service ) for website and i have about +3K concurrent user .. i'm using php with MongoDB
MongoDB architecture is ( 1 Query , 3 Config server , 2 Shard ) . i'm sure that my mongo configuration is working fine . but shard 1 stopped suddenly with no reason. and when i tried to run sudo services mongod restart mongo doesn't run. when i view the log file i get the last lines
2016-10-15T09:35:52.334-0400 I CONTROL [main] ***** SERVER RESTARTED *****
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] MongoDB starting : pid=2168 port=27017 dbpath=/var/lib/mongodb 64-bit host=Analytics-DB-shard-01
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] db version v3.2.8
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d15957sdaaf0
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] allocator: tcmalloc
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] modules: none
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] build environment:
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] distmod: ubuntu1404
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] distarch: x86_64
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] target_arch: x86_64
2016-10-15T09:35:52.343-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-10-15T09:35:52.373-0400 E NETWORK [initandlisten] Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted
2016-10-15T09:35:52.373-0400 I - [initandlisten] Fatal Assertion 28578
2016-10-15T09:35:52.373-0400 I - [initandlisten]
***aborting after fassert() failure
i can't figure out mongo problems but i'm trying to manage it to work correctly as much as possible .. any idea how to make sure that it won't dropped again ?
Related
What is the difference between these two mongo data folders?
I have my mongod dbpath pointed to C:/data/db, however I also noticed I have a data folder in C:\Program Files\MongoDB\Server\4.0\ which also contains indexes and collections.
Only reason I noticed this is because I got an unclean shutdown and lost 177 million docs so was trying to look closer at the issue (side note: was not able to recover even though mongo could rebuild the index for that missing collection)
Edit
Here is a log file from the server folder where I could see a dbpath pointing elsewhere
**aborting after fassert() failure
2018-11-18T11:40:34.164+0100 I CONTROL [main] ***** SERVER RESTARTED *****
2018-11-18T11:40:36.984+0100 I CONTROL [main] Trying to start Windows service 'MongoDB'
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] MongoDB starting : pid=4252 port=27017 dbpath=C:\Program Files\MongoDB\Server\4.0\data 64-bit host=DESKTOP-IO5LL15
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] db version v4.0.4
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] git version: f288a3bdf201007f3693c58e140056adf8b04839
2018-11-18T11:40:37.092+0100 I CONTROL [initandlisten] allocator: tcmalloc
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] modules: none
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] build environment:
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] distmod: 2008plus-ssl
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] distarch: x86_64
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] target_arch: x86_64
2018-11-18T11:40:37.093+0100 I CONTROL [initandlisten] options: { config: "C:\Program Files\MongoDB\Server\4.0\bin\mongod.cfg", net: { bindIp: "127.0.0.1", port: 27017 }, service: true, storage: { dbPath: "C:\Program Files\MongoDB\Server\4.0\data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "C:\Program Files\MongoDB\Server\4.0\log\mongod.log" } }
2018-11-18T11:40:37.099+0100 W STORAGE [initandlisten] Detected unclean shutdown - C:\Program Files\MongoDB\Server\4.0\data\mongod.lock is not empty.
2018-11-18T11:40:37.100+0100 I STORAGE [initandlisten] Detected data files in C:\Program Files\MongoDB\Server\4.0\data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2018-11-18T11:40:37.100+0100 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I tried to start up my computer's mongodb service and it wouldn't start. I spent over an hour trying to debug it during which time I tried many solutions. Trying all of them may have messed up my mongo install, but I tried reinstalling mongo to get a fresh start. That didn't work either. I now have the following error and cannot find anything on google about how to fix it.
2017-06-01T16:57:02.339-0400 I CONTROL [main] ***** SERVER RESTARTED *****
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] MongoDB starting : pid=15070 port=27017 dbpath=/var/lib/mongodb 64-bit host=moss-greenelab
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] db version v3.2.13
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] git version: 23899209cad60aaafe114f6aea6cb83025ff51bc
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] allocator: tcmalloc
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] modules: none
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] build environment:
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] distmod: ubuntu1604
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] distarch: x86_64
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] target_arch: x86_64
2017-06-01T16:57:02.343-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2017-06-01T16:57:02.361-0400 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-06-01T16:57:02.361-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=18G,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-06-01T16:57:02.364-0400 E STORAGE [initandlisten] WiredTiger (13) [1496350622:364565][15070:0x7f80e0292c80], connection: /var/lib/mongodb/journal/WiredTigerPreplog.0000000002: file-remove: unlink: Permission denied
2017-06-01T16:57:02.383-0400 I - [initandlisten] Assertion: 28595:13: Permission denied
2017-06-01T16:57:02.383-0400 I STORAGE [initandlisten] exception in initAndListen: 28595 13: Permission denied, terminating
2017-06-01T16:57:02.383-0400 I CONTROL [initandlisten] dbexit: rc: 100
I'm running Ubuntu 16.04.
Here's the telling part of the log:
/var/lib/mongodb/journal/WiredTigerPreplog.0000000002: file-remove: unlink: Permission denied
So have a look at ls -la /var/lib/mongodb/journal and see what the permissions on those files are. I'm betting that those (an probably others) are not properly owned by the correct user, which is normally mongodb on Ubuntu.
Assuming you've got an issue with ownership there, you can just run chown -R mongodb:mongodb /var/lib/mongodb.
If that doesn't work, as a last resort you can always just wipe out your data directory (assuming you don't need your data!) by running rm -rf /var/lib/mongodb/*.
When I am trying to follow the tutorial by typing
mongod --port 27017 --dbpath /data/db1
console responds with
C:\WINDOWS\system32>mongod --port 27017 --dbpath /data/db1
2017-03-03T14:43:54.019+0100 I CONTROL [initandlisten] MongoDB starting : pid=1776 port=27017 dbpath=/data/db1 64-bit host=Tim-V5
2017-03-03T14:43:54.020+0100 I CONTROL [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
2017-03-03T14:43:54.021+0100 I CONTROL [initandlisten] db version v3.4.2
2017-03-03T14:43:54.021+0100 I CONTROL [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b
2017-03-03T14:43:54.022+0100 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1u-fips 22 Sep 2016
2017-03-03T14:43:54.023+0100 I CONTROL [initandlisten] allocator: tcmalloc
2017-03-03T14:43:54.023+0100 I CONTROL [initandlisten] modules: none
2017-03-03T14:43:54.023+0100 I CONTROL [initandlisten] build environment:
2017-03-03T14:43:54.023+0100 I CONTROL [initandlisten] distmod: 2008plus-ssl
2017-03-03T14:43:54.024+0100 I CONTROL [initandlisten] distarch: x86_64
2017-03-03T14:43:54.024+0100 I CONTROL [initandlisten] target_arch: x86_64
2017-03-03T14:43:54.025+0100 I CONTROL [initandlisten] options: { net: { port: 27017 }, storage: { dbPath: "/data/db1" } }
2017-03-03T14:43:54.025+0100 I STORAGE [initandlisten] exception in initAndListen: 29 Data directory /data/db1 not found., terminating
2017-03-03T14:43:54.026+0100 I NETWORK [initandlisten] shutdown: going to close listening sockets...
2017-03-03T14:43:54.026+0100 I NETWORK [initandlisten] shutdown: going to flush diaglog...
2017-03-03T14:43:54.027+0100 I CONTROL [initandlisten] now exiting
2017-03-03T14:43:54.027+0100 I CONTROL [initandlisten] shutting down with code:100
Even I already have created folder with
mkdir C:\data\db
and checked that it exists!
Following with
mongo --port 27017
It responds with
MongoDB shell version v3.4.2
connecting to: mongodb://127.0.0.1:27017/
MongoDB server version: 3.4.2
Server has startup warnings:
2017-03-03T10:02:59.647+0100 I CONTROL [initandlisten]
2017-03-03T10:02:59.647+0100 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2017-03-03T10:02:59.647+0100 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2017-03-03T10:02:59.647+0100 I CONTROL [initandlisten]
I am using windows 10, how can I solve this? It's rather frustating
my instance running in docker. after close it uncleanly, it can't be started. with log below
I CONTROL [initandlisten] MongoDB starting : pid=13 port=27017
dbpath=/data/db 64-bit host=94a9490feef9ce51e9e55af47ab5c357-a0000
I CONTROL [initandlisten] db version v3.2.6
I CONTROL [initandlisten] git version: 05552b562c7a0b3143a729aaa0838e558dc49b25
I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
I CONTROL [initandlisten] allocator: tcmalloc
I CONTROL [initandlisten] modules: none
I CONTROL [initandlisten] build environment:
I CONTROL [initandlisten] distmod: debian71
I CONTROL [initandlisten] distarch: x86_64
I CONTROL [initandlisten] target_arch: x86_64
I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, security: { authorization: "disabled" }, storage: { dbPath: "/data/db", directoryPerDB: true, journal: { enabled: true } } }
I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=150G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
I - [initandlisten] Assertion: 28595:2: No such file or directory
I STORAGE [initandlisten] exception in initAndListen: 28595 2: No such file or directory, terminating
I CONTROL [initandlisten] dbexit: rc: 100
which file can't be found is not show, how to deal next?
I recently had some storage issues on my PC so I changed the mongo.conf file to reduce the size of the journal. These are the lines which I put in.
mmapv1:
smallFiles: true
However, every time when I try to run mongo with the new changes I get this error(without the change it works just fine). Can you tell me what's the problem?
2016-02-11T22:32:14.002+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2016-02-11T22:32:14.023+0000 W FTDC [ftdc] Uncaught exception in 'UnknownError Caught std::exception of type boost::filesystem::filesystem_error: boost::filesystem::file_size: No such file or directory: "/data/db/diagnostic.data/metrics.interim.temp"' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
I would try deleting the directory /data/db/diagnostic.data.
You can also disable the diagnostic data collection by starting mongo with the option:
setParameter:
diagnosticDataCollectionEnabled: false
I had the same problem after MongoDb failed when HD was fulled upto 100%. Here is the log:
2016-10-03T10:13:42.017+0000 I CONTROL [initandlisten] MongoDB starting : pid=12630 port=27017 dbpath=/var/lib/mongo 64-bit host=drft006
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] db version v3.2.9
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] modules: none
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] build environment:
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] distmod: rhel62
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] distarch: x86_64
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] target_arch: x86_64
2016-10-03T10:13:42.018+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017, unixDomainSocket: { filePermissions: 511 } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-10-03T10:13:42.045+0000 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-10-03T10:13:42.045+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2016-10-03T10:13:42.045+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2016-10-03T10:13:42.045+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=93G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-10-03T10:13:42.465+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2016-10-03T10:13:42.465+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-10-03T10:13:42.467+0000 I NETWORK [initandlisten] waiting for connections on port 27017
2016-10-03T10:13:43.012+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
2016-10-03T10:13:43.045+0000 W FTDC [ftdc] Uncaught exception in 'UnknownError: Caught std::exception of type boost::filesystem::filesystem_error: boost::filesystem::file_size: No such file or directory: "/var/lib/mongo/diagnostic.data/metrics.interim.temp"' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
I kill-ed mongod and cleared the contents of the diagnostic folders:
> rm -f /var/lib/mongo/diagnostic.data/*
After the restart of mongo log showed:
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] MongoDB starting : pid=12803 port=27017 dbpath=/var/lib/mongo 64-bit host=drft006
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] db version v3.2.9
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] git version: 22ec9e93b40c85fc7cae7d56e7d6a02fd811088c
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] allocator: tcmalloc
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] modules: none
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] build environment:
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] distmod: rhel62
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] distarch: x86_64
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] target_arch: x86_64
2016-10-03T10:17:46.745+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017, unixDomainSocket: { filePermissions: 511 } }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "enabled" }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2016-10-03T10:17:46.772+0000 I - [initandlisten] Detected data files in /var/lib/mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-10-03T10:17:46.772+0000 W - [initandlisten] Detected unclean shutdown - /var/lib/mongo/mongod.lock is not empty.
2016-10-03T10:17:46.772+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2016-10-03T10:17:46.772+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=93G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-10-03T10:17:47.128+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongo/diagnostic.data'
2016-10-03T10:17:47.128+0000 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-10-03T10:17:47.129+0000 I NETWORK [initandlisten] waiting for connections on port 27017
and mongo was ok.
In case you are using nodejs, try to disable/remove debug mode in your server code.
Steps:
rm -rf /{your_db_name}/diagnostic.data/
remove or comment out mongoose.set('debug', true);
restart mongo and node server
In my case I use docker-compose. First I remove the diagnostic.data folder from my mounted volume:
sudo rm -rf /srv/mongo/diagnostic.data
And then when I use docker-compose up the diagnostic.data folder will be re-created and everything was working again.