MongoDB Enterprise failing to start with Wired Tiger as service - mongodb

Question

I made the installation of WiredTiger, but find that I cannot start the DB using the standard:
sudo service mongod start
It seems when I look into the log file (tail /var/log/mongodb/mongod.log):
2016-08-01T14:48:12.029+0200 I CONTROL [main] ***** SERVER RESTARTED *****
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] MongoDB starting : pid=1238 port=27017 dbpath=/var/lib/mongodb 64-bit host=xxx
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] db version v3.2.8
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips 1 Mar 2016
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] allocator: tcmalloc
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] modules: enterprise
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] build environment:
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] distmod: ubuntu1604
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] distarch: x86_64
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] target_arch: x86_64
2016-08-01T14:48:12.032+0200 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, storage: { dbPath: "/var/lib/mongodb", engine: "wiredTiger", journ
al: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } }
2016-08-01T14:48:12.052+0200 I STORAGE [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: /var/lib/mongodb/mongod.lock errno:13 Permission denied Is a mongod instance already
running?, terminating
2016-08-01T14:48:12.052+0200 I CONTROL [initandlisten] dbexit: rc:
100
# mongod.conf
# for documentation of all options, see:
# http://docs.mongodb.org/manual/reference/configuration-options/
# Where and how to store data.
storage:
dbPath: /var/lib/mongodb
journal:
enabled: true
engine: wiredTiger
# mmapv1:
# wiredTiger:
# where to write logging data.
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
# network interfaces
net:
port: 27017
bindIp: 127.0.0.1
I have changed the permissions as follows on:
chmod -R 777 /var/log/mongodb
chown -R /var/log/mongodb
But nothing makes a difference. There is no service running. I can run from the command line:
mongod
which gives the following:
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] MongoDB starting : pid=5776 port=27017 dbpath=/data/db 64-bit host=brett
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] db version v3.2.8
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips 1 Mar 2016
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] allocator: tcmalloc
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] modules: enterprise
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] build environment:
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] distmod: ubuntu1604
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] distarch: x86_64
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] target_arch: x86_64
2016-08-01T15:35:51.291+0200 I CONTROL [initandlisten] options: {}
2016-08-01T15:35:51.311+0200 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-08-01T15:35:51.311+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-08-01T15:35:51.722+0200 I CONTROL [initandlisten]
2016-08-01T15:35:51.722+0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-08-01T15:35:51.722+0200 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-08-01T15:35:51.722+0200 I CONTROL [initandlisten]
2016-08-01T15:35:51.722+0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-08-01T15:35:51.722+0200 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-08-01T15:35:51.722+0200 I CONTROL [initandlisten]
2016-08-01T15:35:51.722+0200 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2016-08-01T15:35:51.722+0200 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-08-01T15:35:51.723+0200 I NETWORK [initandlisten] waiting for connections on port 27017
But I cannot get mongod as a service! Trying permissions:
sudo chown -R mongodb:mongodb /var/lib/mongodb
or
sudo chown -R user:machine /var/lib/mongodb
or
sudo chown -R root:machine /var/lib/mongodb
If I run the following:
mongod --config /etc/mongod.conf
I can then connect to the DB with mongo
as follows:
MongoDB shell version: 3.2.8
connecting to: test
Server has startup warnings:
2016-08-09T19:46:31.583+0200 I CONTROL [initandlisten]
2016-08-09T19:46:31.583+0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-08-09T19:46:31.583+0200 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-08-09T19:46:31.583+0200 I CONTROL [initandlisten]
2016-08-09T19:46:31.583+0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-08-09T19:46:31.583+0200 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-08-09T19:46:31.583+0200 I CONTROL [initandlisten]
Here is the output from ls -la /var/lib/mongodb
total 216
drwxrwxrwx 4 xxx root 4096 Aug 19 12:37 .
drwxr-xr-x 78 root root 4096 Aug 9 15:05 ..
-rwxrwxrwx 1 xxx xxx 36864 Aug 10 11:22 collection-0--5694223804331577236.wt
drwxrwxrwx 2 xxx xxx 4096 Aug 10 11:22 diagnostic.data
-rwxrwxrwx 1 xxx xxx 36864 Aug 10 11:22 index-1--5694223804331577236.wt
drwxrwxrwx 2 xxx xxx 4096 Aug 9 19:46 journal
-rwxrwxrwx 1 xxx xxx 16384 Aug 10 11:22 _mdb_catalog.wt
-rwxrwxrwx 1 xxx xxx 36864 Aug 10 11:22 sizeStorer.wt
-rwxrwxrwx 1 xxx xxx 95 Aug 9 19:30 storage.bson
-rwxrwxrwx 1 xxx xxx 46 Aug 9 19:30 WiredTiger
-rwxrwxrwx 1 xxx xxx 4096 Aug 10 11:22 WiredTigerLAS.wt
-rwxrwxrwx 1 xxx xxx 21 Aug 9 19:30 WiredTiger.lock
-rwxrwxrwx 1 xxx xxx 918 Aug 10 11:22 WiredTiger.turtle
-rwxrwxrwx 1 xxx xxx 45056 Aug 10 11:22 WiredTiger.wt
I also tried removing the huge files as was suggested. But I can never find a way to get the service started.
Here is me running mongod:
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] MongoDB starting : pid=9030 port=27017 dbpath=/data/db 64-bit host=xxx
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] db version v3.2.8
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips 1 Mar 2016
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] allocator: tcmalloc
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] modules: enterprise
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] build environment:
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] distmod: ubuntu1604
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] distarch: x86_64
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] target_arch: x86_64
2016-08-19T12:53:53.713+0200 I CONTROL [initandlisten] options: {}
2016-08-19T12:53:53.732+0200 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-08-19T12:53:53.733+0200 I STORAGE [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: /data/db/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
2016-08-19T12:53:53.733+0200 I CONTROL [initandlisten] dbexit: rc: 100
here is:
ls -ld /data/db/
drwxr-xr-x 4 mongodb nogroup 4096 Aug 19 13:10 /data/db/
I can start mongo using
sudo mongod
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] MongoDB starting : pid=12360 port=27017 dbpath=/data/db 64-bit host=brett
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] db version v3.2.8
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] git version: ed70e33130c977bda0024c125b56d159573dbaf0
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g-fips 1 Mar 2016
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] allocator: tcmalloc
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] modules: enterprise
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] build environment:
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] distmod: ubuntu1604
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] distarch: x86_64
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] target_arch: x86_64
2016-08-19T13:13:47.117+0200 I CONTROL [initandlisten] options: {}
2016-08-19T13:13:47.137+0200 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-08-19T13:13:47.137+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-08-19T13:13:47.548+0200 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2016-08-19T13:13:47.548+0200 I CONTROL [initandlisten]
2016-08-19T13:13:47.549+0200 I CONTROL [initandlisten]
2016-08-19T13:13:47.549+0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-08-19T13:13:47.549+0200 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-08-19T13:13:47.549+0200 I CONTROL [initandlisten]
2016-08-19T13:13:47.549+0200 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-08-19T13:13:47.549+0200 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2016-08-19T13:13:47.549+0200 I CONTROL [initandlisten]
2016-08-19T13:13:47.549+0200 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2016-08-19T13:13:47.549+0200 I NETWORK [HostnameCanonicalizationWorker] Starting hostname canonicalization worker
2016-08-19T13:13:47.550+0200 I NETWORK [initandlisten] waiting for connections on port 27017
But still no joy getting mongo as a service :(
Here is the /etc/init/mongod.conf
# Ubuntu upstart file at /etc/init/mongod.conf
# Recommended ulimit values for mongod or mongos
# See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
#
limit fsize unlimited unlimited
limit cpu unlimited unlimited
limit as unlimited unlimited
limit nofile 64000 64000
limit rss unlimited unlimited
limit nproc 64000 64000
kill timeout 300 # wait 300s between SIGTERM and SIGKILL.
pre-start script
DAEMONUSER=${DAEMONUSER:-mongodb}
if [ ! -d /var/lib/mongodb ]; then
mkdir -p /var/lib/mongodb && chown mongodb:mongodb /var/lib/mongodb
fi
if [ ! -d /var/log/mongodb ]; then
mkdir -p /var/log/mongodb && chown mongodb:mongodb /var/log/mongodb
fi
touch /var/run/mongodb.pid
chown $DAEMONUSER /var/run/mongodb.pid
end script
start on runlevel [2345]
stop on runlevel [06]
script
ENABLE_MONGOD="yes"
CONF=/etc/mongod.conf
DAEMON=/usr/bin/mongod
DAEMONUSER=${DAEMONUSER:-mongodb}
DAEMONGROUP=${DAEMONGROUP:-mongodb}
if [ -f /etc/default/mongod ]; then . /etc/default/mongod; fi
# Handle NUMA access to CPUs (SERVER-3574)
# This verifies the existence of numactl as well as testing that the command works
NUMACTL_ARGS="--interleave=all"
if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
then
NUMACTL="$(which numactl) -- $NUMACTL_ARGS"
DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}
else
NUMACTL=""
DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}
fi
if [ "x$ENABLE_MONGOD" = "xyes" ]
then
exec start-stop-daemon --start \
--chuid $DAEMONUSER:$DAEMONGROUP \
--pidfile /var/run/mongodb.pid \
--make-pidfile \
--exec $NUMACTL $DAEMON $DAEMON_OPTS
fi
end script

Answer 1

As others wrote... Error message points that problem is that process cannot create /var/lib/mongodb/mongod.lock
You need to understand that even you say (as root) 'service mongod start' (when you use sudo -command, command is executed as root), process (mongod) is started as user 'mongod'. So, that directory /var/lib/mongodb must be exist and be writable by userid mongod.
mkdir -p /var/lib/mongodb ; chown -R mongod: /var/lib/mongodb
sudo service mongod restart
(OR 'sudo -u mongod mongod -f /etc/mongod.conf')