I need to install a mongodb instance with a lot of data storage.
We have a Lustre FS with hundreds of terabytes, but when monogdb start show me this error:
Mon Jul 15 12:06:50.898 [initandlisten] exception in initAndListen: 10310 Unable to lock file: /var/lib/mongodb/mongod.lock. Is a mongod instance already running?, terminating
Mon Jul 15 12:06:50.898 dbexit:
But the permissions should be fine:
# ls -lart /project/mongodb/
total 8
drwxr-xr-x 19 root root 4096 Jul 15 11:12 ..
-rwxr-xr-x 1 mongod mongod 0 Jul 15 11:54 mongod.lock
drwxr-xr-x 2 mongod mongod 4096 Jul 15 12:10 .
And no other running process:
# ps -fu mongod
UID PID PPID C STIME TTY TIME CMD
#
Has anyone done this (Lustre+mongodb)?
# rm mongod.lock
rm: remove regular empty file `mongod.lock'? y
# ls -lrt
total 0
# ls -lart
total 8
drwxr-xr-x 19 root root 4096 Jul 15 11:12 ..
drwxr-xr-x 2 mongod mongod 4096 Jul 15 12:10 .
# ps aux | grep mongod
root 25865 0.0 0.0 103296 884 pts/15 S+ 13:04 0:00 grep mongod
# service mongod start
Starting mongod: about to fork child process, waiting until server is ready for connections.
forked process: 25935
all output going to: /var/log/mongo/mongod.log
ERROR: child process failed, exited with error number 100
[FAILED]
I realize that this is an old question, but I feel I should set the record straight.
MongoDB, or any DB or any application can run against a lustre file system without issues. However, by default, lustre clients do not explicitly set user_xattr or flock (enable).
Having set -o flock or even -o localflock while mounting the file system would have resolved the issue.
Related
containerd: file label is container_ro_file_t but container process runs as spc_t. Is process label spc_t correct if selinux enabled for containerd or did i miss some setting with containerd?
K8s version: 1.23.8
Containerd version: 1.6.6
selinux enable by setting [enable_selinux = true] in /etc/containerd/config.toml
// create pod using tomcat official image then check the process and file label
$kubectl exec tomcat -it -- ps -eZ
system_u:system_r:spc_t:s0 1 ? 00:00:26 java
system_u:system_r:spc_t:s0 45 pts/0 00:00:00 ps
$kubectl exec tomcat -it -- ls -FlaZ
drwxr-xr-x. 1 root root system_u:object_r:container_ro_file_t:s0 4096 Jun 28 00:54 ./
drwxr-xr-x. 1 root root system_u:object_r:container_ro_file_t:s0 4096 Jun 28 00:50 ../
drwxr-xr-x. 2 root root system_u:object_r:container_ro_file_t:s0 4096 Jun 28 00:54 bin/
#containerd is running as container_runtime_t:
$ps -eZ | grep containerd
system_u:system_r:container_runtime_t:s0 912 ? 00:00:10 containerd
system_u:system_r:container_runtime_t:s0 1327 ? 00:00:00 containerd-shim
//seems run as spc_t is correct
$sesearch -T -t container_var_lib_t | grep spc_t
type_transition container_runtime_t container_ro_file_t : process spc_t;
Issue resolved after adding version in /etc/containerd/config.toml
version=2
I am stuck with the following error while starting mongod service systemctl start mongod
{"t":{"$date":"2020-08-27T20:48:20.219+00:00"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongo"}}
I have already checked /var/lib/mongo folder permissions and seem to be ok:
[root#**system]# ls -l / | grep var
drwxr-xr-x. 21 root root 4096 Jun 25 07:43 var
[root#**system]# ls -l /var | grep lib
drwxr-xr-x. 6 root root 56 Aug 27 20:38 lib
[root#** system]# ls -l /var/lib | grep mongo
drwxr-xr-x. 4 mongod mongod 4096 Aug 27 20:16 mongo
Any idea on why I am getting the error?
I'm running an executable and trying to save terminal output to a file:
# ll
total 132
-rw-r--r--. 1 root root 496 Jun 14 11:41 mpx-debug.h
-rw-r--r--. 1 root root 12775 Jun 14 11:41 mpx-dig.c
-rw-r--r--. 1 root root 3526 Jun 14 11:41 mpx-hw.h
-rwxr-xr-x. 1 root root 65176 Jun 14 14:28 mpx-mini-test
-rw-r--r--. 1 root root 40480 Jun 14 11:41 mpx-mini-test.c
-rw-r--r--. 1 root root 205 Jun 14 11:41 mpx-mm.h
# ./mpx-mini-test
XSAVE is supported by HW & OS
XSAVE processor supported state mask: 0x1f
XSAVE OS supported state mask: 0x1f
BNDREGS: size: 64 user: 1 supervisor: 0 aligned: 0
BNDCSR: size: 64 user: 1 supervisor: 0 aligned: 0
no MPX support
Aborted (core dumped)
#
# ./mpx-mini-test | tee -a mpx-mini-test.log
#
# cat mpx-mini-test.log
#
As you can see, there're some prints in terminal without "| tee -a **.log".
But with "| tee -a **.log", nothing is saved to file.
I suspect it has something to do with the "Abort (core dumped)", did some googling but can't figure out exactly why. Is there anyone knows why? And how could I save all terminal outputs(including the "Abort (core dumped)") to file? Thanks in advance.
After configuring a standalone Concourse 2.4.0 per the instructions, everything seems to be up and running. However, when trying to run the "hello world" example, I can see the following error in the Concourse UI:
runc create: exit status 1: rootfs ("/volumes/live/a72f9a0d-3506-489b-5b9b-168744b892c1/volume") does not exist
"web" start command:
./concourse web \
--basic-auth-username admin \
--basic-auth-password admin \
--session-signing-key session_signing_key \
--tsa-host-key host_key \
--tsa-authorized-keys authorized_worker_keys \
--external-url http://myconcoursedomain:8080 \
--postgres-data-source postgres://user:pass#mydbserver/concourse
"worker" start command:
./concourse worker \
--work-dir worker \
--tsa-host 127.0.0.1 \
--tsa-public-key host_key.pub \
--tsa-worker-private-key worker_key
I'm wondering if the problem occurs since the "missing" directory is created in the directory specified in the "start worker" command, instead of at the actual root directory:
~/concourse# ls -la worker
total 145740
drwxr-xr-x 5 root root 4096 Nov 15 23:07 .
drwxr-xr-x 3 root root 4096 Nov 15 23:07 ..
drwxr-xr-x 3 root root 4096 Nov 15 23:07 2.4.0
drwxr-xr-x 2 root root 4096 Nov 15 23:09 depot
drwxr-xr-x 1 root root 24 Nov 15 23:07 volumes
-rw-r--r-- 1 root root 42142052352 Nov 15 23:15 volumes.img
Concourse is installed on Ubuntu 14.04:
uname -r
4.4.0-47-generic
uname -a
Linux ubuntu-2gb-nyc3-01 4.4.0-47-generic #68~14.04.1-Ubuntu SMP Wed Oct 26 19:42:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
For reasons that I still do not understand, it appears that if you specify the --work-dir value to be /opt/concourse/worker, then the worker will work with this kernel version without issue.
I was using a relative path to a worker directory within a dir in my user folder as my --work-dir value.
The MongoDB version is v2.6.3 on my server, and mongod is running:
ubuntu#koala:/var/log/mongodb$ ps -ef | grep mongo
root 7434 1 17 Jun16 ? 06:57:26 mongod -f /etc/mongodb-onepiece.conf --fork
I am using logrotate to daily rotate the log file of MongoDB. A strange problem just occurred with logrotate.
I check the log file:
ubuntu#koala:/var/log/mongodb$ ls -lth | grep mongodb
-rw-r--r-- 1 ubuntu ubuntu 1.9G Jun 18 10:23 mongodb-onepiece.log.1
-rw-r--r-- 1 ubuntu ubuntu 0 Jun 17 07:35 mongodb-onepiece.log
-rw-r--r-- 1 ubuntu ubuntu 838M Jun 15 07:35 mongodb-onepiece.log.3.gz
-rw-r--r-- 1 ubuntu ubuntu 22 Jun 14 20:52 mongodb-onepiece.log.2.gz
-rw-r--r-- 1 ubuntu ubuntu 1.1G Jun 4 17:10 mongodb-onepiece.log.4.gz
-rw-r--r-- 1 ubuntu ubuntu 53M May 29 19:14 mongodb-onepiece.log.5.gz
The most up-to-date log file is .log.1 instead of .log. When I use tail -fn to check the log.1 file, I can see that the log is still appending to it, and it's growing:
ubuntu#koala:/var/log/mongodb$ tail -fn 2 mongodb-onepiece.log.1
2015-06-18T10:36:50.163+0800 [initandlisten] connection accepted from 192.168.1.52:50278 #2507 (49 connections now open)
2015-06-18T10:36:50.163+0800 [conn2503] command koala.$cmd command: isMaster { ismaster: 1 } keyUpdates:0 numYields:0 reslen:178 0ms
This means that MongoDB is logging to the file that is't not supposed. As can be seen from the mongod config file, MongoDB should log to the logpath:
ubuntu#koala:/var/log/mongodb$ vim /etc/mongodb-onepiece.conf
dbpath=/var/lib/mongodb-onepiece
logpath=/var/log/mongodb/mongodb-onepiece.log
logappend=true
bind_ip = 192.168.1.*
port = 47017
fork=true
journal=true
master = true
From the above, I assume that the problem was not with the logrotate config, but with MongoDB writing to the wrong file. Everyday when logrotate starts, it only checks .log file and finds out it's empty, then it will stop rotating the log.
If I restart the mongod daemon, the logpath will be correct for a moment (writing to the right log file). For that day, the .log file is not empty, then it will be successfully rotated to .log.1 file. But the same problem will happen again after log rotating ,i.e., MongoDB will be logging to .log.1 file afterwards. The cycle comes here.
The logrotate config file is given here:
ubuntu#koala:/var/log/mongodb$ vim /etc/logrotate.d/mongodb
/var/log/mongodb/*.log {
daily
rotate 52
missingok
copytruncate
notifempty
compress
delaycompress
}
The same logrotate config just works fine with other MongoDB logs on the other server with MongoDB v2.6.5 and I suppose postrotate is not the trick here (I have also tried postrotate but without luck).
How to solve this problem?
I'm not a mongo expert, but:
You should be following the official documentation https://docs.mongodb.org/v2.6/tutorial/rotate-log-files/
If you are going to use a logrotate config file, as you indicated, then you need a postrotate lint to your config (failure to do so is why mongodb continues to log to the log.1 file)
postrotate
kill -SIGUSR1 `cat /var/run/mongodb.pid` >/dev/null 2>&1 || true