How to apply multi VFs with virtio-net vDPA on host? - virtio

Recently, I was developing vDPA drivers for our NIC. When testing virtio-net vDPA with multi VFs, I found that kernel vDPA framework allocated same DMA addresses for multi VFs, then different VF operate the same DMA address, such as updating the used index, cause kernel virtqueues works abnormally.
The steps are as follows:
enable the NIC sriov, create 4 VFs, we can see that 4 vdpa management devices are create successful.
*[root#localhost ~]# echo 4 > /sys/class/net/enp1s0np0/device/sriov_numvfs
[root#localhost ~]# vdpa mgmtdev show
pci/0000:01:08.0:
supported_classes net
pci/0000:01:08.1:
supported_classes net
pci/0000:01:08.2:
supported_classes net
pci/0000:01:08.3:
supported_classes net*
add vdpa device and enable virtio vdpa module. create 4 vdpa device and the driver are binded successful.
*[root#localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.0 name vdpa0
[root#localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.1 name vdpa1
[root#localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.2 name vdpa2
[root#localhost ~]# vdpa dev add mgmtdev pci/0000:01:08.3 name vdpa3
[root#localhost ~]# modprobe virtio_vdpa
[root#localhost ~]# ls -l /sys/bus/vdpa/drivers/virtio_vdpa/
total 0
--w-------. 1 root root 4096 Nov 21 16:55 bind
lrwxrwxrwx. 1 root root 0 Nov 21 16:55 module -> ../../../../module/virtio_vdpa
--w-------. 1 root root 4096 Nov 21 16:55 uevent
--w-------. 1 root root 4096 Nov 21 16:55 unbind
lrwxrwxrwx. 1 root root 0 Nov 21 16:55 vdpa0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.0/vdpa0
lrwxrwxrwx. 1 root root 0 Nov 21 16:55 vdpa1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.1/vdpa1
lrwxrwxrwx. 1 root root 0 Nov 21 16:55 vdpa2 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.2/vdpa2
lrwxrwxrwx. 1 root root 0 Nov 21 16:55 vdpa3 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:08.3/vdpa3*
check the kernel messages output, we can see the virtio-net device and dma addresses.
*[Mon Nov 21 16:55:42 2022] virtio_net virtio0: devname virtio0 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio0: devname virtio0 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio1: devname virtio1 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio1: devname virtio1 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio2: devname virtio2 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio2: devname virtio2 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio0 enp1s0v0: renamed from eth5
[Mon Nov 21 16:55:42 2022] virtio_net virtio3: devname virtio3 name input.0 index 0 dmaaddr ffffc000
[Mon Nov 21 16:55:42 2022] virtio_net virtio3: devname virtio3 name output.0 index 1 dmaaddr ffff8000
[Mon Nov 21 16:55:42 2022] virtio_net virtio1 enp1s0v1: renamed from eth6
[Mon Nov 21 16:55:42 2022] virtio_net virtio2 enp1s0v2: renamed from eth7
[Mon Nov 21 16:55:42 2022] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
[Mon Nov 21 16:55:42 2022] virtio_net virtio3 enp1s0v3: renamed from eth5*
It seems that kernel vDPA framework assigns the same virtqueue dma address to four different vf.
This application scenario refers to the vDPA description of Red Hat: https://www.redhat.com/en/blog/vdpa-kernel-framework-part-3-usage-vms-and-containers
My kernel version is 5.15.15, kernel vDPA options are all enabled when compile the kernel.
[root#localhost linux-5.15.15]# cat .config | grep VDPA
`CONFIG_VIRTIO_VDPA=m
CONFIG_VDPA=y
CONFIG_VDPA_SIM=m
CONFIG_VDPA_SIM_NET=m
CONFIG_VDPA_SIM_BLOCK=m
#CONFIG_VDPA_USER is not set
CONFIG_MLX5_VDPA=y
CONFIG_MLX5_VDPA_NET=m
CONFIG_VP_VDPA=m
CONFIG_VHOST_VDPA=m`
[root#localhost linux-5.15.15]#
So, is there anything i missed or misunderstood about the kernel vDPA framework, if someone can give some advice, that will be great help. thanks a lot.
I've tested with vhost-vdpa, start VMs with QEMU, and multi VFs works well.

I’ve investigated the virtio specification, and found that the virtio 1.1 specification has updated and add a new feature bit VIRTIO_F_SR_IOV. I guess that there will be some adaptation in the kernel vDPA framework to support multi vf. But currently I haven't seen the implementation in the 5.19 kernel yet. However, this confirms that the kernel framework is not yet supported.
The requirement for VIRTIO_F_SR_IOV as follows( see virtio 1.1 specification chapter 6.1/6.2):
If VIRTIO_F_SR_IOV has been negotiated, a driver MAY enable virtual functions through the device's PCI SR-IOV capability structure. A driver MUST NOT negotiate VIRTIO_F_SR_IOV if the device does not have a PCI SR-IOV capability structure or is not a PCI device. A driver MUST negotiate VIRTIO_F_SR_IOV and complete the feature negotiation (including checking the FEATURES_OK device status bit) before enabling virtual functions through the device's PCI SR-IOV capability structure. After once successfully negotiating VIRTIO_F_SR_IOV, the driver MAY enable virtual functions through the device's PCI SR-IOV capability structure even if the device or the system has been fully or partially reset, and even without re-negotiating VIRTIO_F_SR_IOV after the reset.
A device SHOULD offer VIRTIO_F_SR_IOV if it is a PCI device and presents a PCI SR-IOV capability structure, otherwise it MUST NOT offer VIRTIO_F_SR_IOV
see changes:
https://www.oasis-open.org/committees/ballot.php?id=3218
https://github.com/oasis-tcs/virtio-spec/issues/11
we are looking forward the future kernel version will support this feature.

Related

Cant't mount usb via udev rules

After trying for quite a long time to create a udev rule to automatically mount a USB-Stick to a directory, I have to realize that I can't get any further without help.
Here is my udev-rule stored at /etc/udev/rules.d/60-usbspecialmount.rules
ACTION=="add", SUBSYSTEMS=="usb", KERNEL=="sd?", ATTRS{serial}=="3f980000.usb", RUN+="/bin/mount /dev/%k /media/securitycam
ACTION=="remove", SUBSYSTEMS=="usb", KERNEL=="sd?", RUN+="/bin/umount /dev/%k"
after restarting the udev service with:
sudo service udev restart
and plugging in the USB-Device, the SYSLOG-File says following:
Sep 13 05:47:17 raspberrypi systemd[1]: Stopping Rule-based Manager for Device Events and Files...
Sep 13 05:47:17 raspberrypi systemd[1]: systemd-udevd.service: Succeeded.
Sep 13 05:47:17 raspberrypi systemd[1]: Stopped Rule-based Manager for Device Events and Files.
Sep 13 05:47:17 raspberrypi systemd[1]: systemd-udevd.service: Consumed 1.424s CPU time.
Sep 13 05:47:17 raspberrypi systemd[1]: Starting Rule-based Manager for Device Events and Files...
Sep 13 05:47:17 raspberrypi systemd[1]: Started Rule-based Manager for Device Events and Files.
Sep 13 05:48:04 raspberrypi kernel: [ 6571.111154] usb 1-1.4: new high-speed USB device number 25 using dwc_otg
Sep 13 05:48:04 raspberrypi kernel: [ 6571.242509] usb 1-1.4: New USB device found, idVendor=090c, idProduct=2000, bcdDevice=11.00
Sep 13 05:48:04 raspberrypi kernel: [ 6571.242543] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Sep 13 05:48:04 raspberrypi kernel: [ 6571.242563] usb 1-1.4: Product: Flash Disk
Sep 13 05:48:04 raspberrypi kernel: [ 6571.242581] usb 1-1.4: Manufacturer: USB
Sep 13 05:48:04 raspberrypi kernel: [ 6571.243643] usb-storage 1-1.4:1.0: USB Mass Storage device detected
Sep 13 05:48:04 raspberrypi kernel: [ 6571.244623] usb-storage 1-1.4:1.0: Quirks match for vid 090c pid 2000: 800000
Sep 13 05:48:04 raspberrypi kernel: [ 6571.244816] scsi host0: usb-storage 1-1.4:1.0
Sep 13 05:48:04 raspberrypi mtp-probe: checking bus 1, device 25: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4"
Sep 13 05:48:04 raspberrypi mtp-probe: bus: 1, device: 25 was not an MTP device
Sep 13 05:48:04 raspberrypi mtp-probe: checking bus 1, device 25: "/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4"
Sep 13 05:48:04 raspberrypi mtp-probe: bus: 1, device: 25 was not an MTP device
Sep 13 05:48:05 raspberrypi kernel: [ 6572.491851] scsi 0:0:0:0: Direct-Access USB Flash Disk 1100 PQ: 0 ANSI: 6
Sep 13 05:48:05 raspberrypi kernel: [ 6572.492586] sd 0:0:0:0: Attached scsi generic sg0 type 0
Sep 13 05:48:05 raspberrypi kernel: [ 6572.493253] sd 0:0:0:0: [sda] 61736960 512-byte logical blocks: (31.6 GB/29.4 GiB)
Sep 13 05:48:05 raspberrypi kernel: [ 6572.493639] sd 0:0:0:0: [sda] Write Protect is off
Sep 13 05:48:05 raspberrypi kernel: [ 6572.493663] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
Sep 13 05:48:05 raspberrypi kernel: [ 6572.494095] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 13 05:48:05 raspberrypi kernel: [ 6572.498538] sda: sda1
Sep 13 05:48:05 raspberrypi kernel: [ 6572.500805] sd 0:0:0:0: [sda] Attached SCSI removable disk
Sep 13 05:48:06 raspberrypi kernel: [ 6572.599846] F2FS-fs (sda): Magic Mismatch, valid(0xf2f52010) - read(0x0)
Sep 13 05:48:06 raspberrypi kernel: [ 6572.599878] F2FS-fs (sda): Can't find valid F2FS filesystem in 1th superblock
Sep 13 05:48:06 raspberrypi kernel: [ 6572.600343] F2FS-fs (sda): Magic Mismatch, valid(0xf2f52010) - read(0x0)
Sep 13 05:48:06 raspberrypi kernel: [ 6572.600358] F2FS-fs (sda): Can't find valid F2FS filesystem in 2th superblock
Sep 13 05:48:06 raspberrypi systemd-udevd[3817]: sda: Process '/bin/mount /dev/sda /media/securitycam' failed with exit code 32.
Sep 13 05:48:06 raspberrypi usbmount[3825]: USAGE: /dev/sda:PTUUID="7f81bc38"PTTYPE="dos"
Sep 13 05:48:06 raspberrypi usbmount[3825]: /dev/sda does not contain a filesystem or disklabel
Sep 13 05:48:06 raspberrypi systemd-udevd[3817]: sda: Process '/usr/share/usbmount/usbmount add' failed with exit code 1.
Sep 13 05:48:06 raspberrypi usbmount[3845]: USAGE: filesystem
Sep 13 05:48:06 raspberrypi usbmount[3845]: executing command: mount -tvfat -osync,noexec,nodev,noatime,nodiratime,uid=jonas,gid=users,dmask=0002,fmask=0003 /dev/sda1 /media/usb0
Sep 13 05:48:06 raspberrypi kernel: [ 6573.003468] FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Sep 13 05:48:06 raspberrypi usbmount[3845]: executing command: run-parts /etc/usbmount/mount.d
Here is what fdisk -l says about my USB-Device:
Disk /dev/sda: 29.44 GiB, 31609323520 bytes, 61736960 sectors
Disk model: Flash Disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7f81bc38
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 224 61736959 61736736 29.4G c W95 FAT32 (LBA)
My idea is that it has something to do with the Devicename. In syslog it says:
usbmount[3825]: /dev/sda does not contain a filesystem or disklabel
so maybe usbmount should look for /dev/sda1 which includes a filesystem
as you can see with the command:
sudo blkid -p /dev/sda1
/dev/sda1: LABEL="PHILIPS UFD" UUID="863A-2B54" VERSION="FAT32" BLOCK_SIZE="512" TYPE="vfat" USAGE="filesystem" PART_ENTRY_SCHEME="dos" PART_ENTRY_UUID="7f81bc38-01" PART_ENTRY_TYPE="0xc" PART_ENTRY_FLAGS="0x80" PART_ENTRY_NUMBER="1" PART_ENTRY_OFFSET="224" PART_ENTRY_SIZE="61736736" PART_ENTRY_DISK="8:0"
Regrettably the bash script /usr/share/usbmount/usbmount has a variable called: $DEVNAME when it processes the condition "add" from the udev-rule and this variable includes /dev/sda as you can see in the syslog-file.
Is there a possibility to change the variable name or is my solution approach completely wrong?

failing k8s mongodb pod after some time - DBPathInUse: Unable to create/open the lock file

I'm running a kubernetes cluster (bare metal) with a mongodb (version 4, as my server cannot handle newer versions) replicaset (2 replicas), which is initially working, but from time to time (sometimes 24 hours, somtimes 10 days) one or more mongodb pods are failing.
Warning BackOff 2m9s (x43454 over 6d13h) kubelet Back-off restarting failed container
The relevant part of the logs should be
DBPathInUse: Unable to create/open the lock file: /bitnami/mongodb/data/db/mongod.lock (Read-only file system). Ensure the user executing mongod is the owner of the lock file and has the appropriate permissions. Also make sure that another mongod instance is not already running on the /bitnami/mongodb/data/db directory
But I do not change anything and initially it is working. Also the second pod is currently running (but which will fail the next days).
I'm using longhorn (before I tried nfs) for the storage and I installed mongodb using bitnami helm chart with these values:
image:
registry: docker.io
repository: bitnami/mongodb
digest: "sha256:916202d7af766dd88c2fff63bf711162c9d708ac7a3ffccd2aa812e3f03ae209" # tag: 4.4.15
pullPolicy: IfNotPresent
architecture: replicaset
replicaCount: 2
updateStrategy:
type: RollingUpdate
containerPorts:
mongodb: 27017
auth:
enabled: true
rootUser: root
rootPassword: "password"
usernames: ["user"]
passwords: ["userpass"]
databases: ["db"]
service:
portName: mongodb
ports:
mongodb: 27017
persistence:
enabled: true
accessModes:
- ReadWriteOnce
size: 8Gi
volumePermissions:
enabled: true
livenessProbe:
enabled: false
readinessProbe:
enabled: false
logs
mongodb 21:25:05.55 INFO ==> Advertised Hostname: mongodb-1.mongodb-headless.mongodb.svc.cluster.local
mongodb 21:25:05.55 INFO ==> Advertised Port: 27017
mongodb 21:25:05.56 INFO ==> Pod name doesn't match initial primary pod name, configuring node as a secondary
mongodb 21:25:05.59
mongodb 21:25:05.59 Welcome to the Bitnami mongodb container
mongodb 21:25:05.60 Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 21:25:05.60 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 21:25:05.60
mongodb 21:25:05.60 INFO ==> ** Starting MongoDB setup **
mongodb 21:25:05.64 INFO ==> Validating settings in MONGODB_* env vars...
mongodb 21:25:05.78 INFO ==> Initializing MongoDB...
mongodb 21:25:05.82 INFO ==> Deploying MongoDB with persisted data...
mongodb 21:25:05.83 INFO ==> Writing keyfile for replica set authentication...
mongodb 21:25:05.88 INFO ==> ** MongoDB setup finished! **
mongodb 21:25:05.92 INFO ==> ** Starting MongoDB **
{"t":{"$date":"2022-10-29T21:25:05.961+00:00"},"s":"I", "c":"CONTROL", "id":20698, "ctx":"main","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2022-10-29T21:25:05.963+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2022-10-29T21:25:05.968+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-10-29T21:25:05.968+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2022-10-29T21:25:05.969+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2022-10-29T21:25:06.011+00:00"},"s":"I", "c":"STORAGE", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/bitnami/mongodb/data/db","architecture":"64-bit","host":"mongodb-1"}}
{"t":{"$date":"2022-10-29T21:25:06.011+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.15","gitVersion":"bc17cf2c788c5dda2801a090ea79da5ff7d5fac9","openSSLVersion":"OpenSSL 1.1.1n 15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian10","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2022-10-29T21:25:06.012+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","version":"Kernel 5.15.0-48-generic"}}}
{"t":{"$date":"2022-10-29T21:25:06.012+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/opt/bitnami/mongodb/conf/mongodb.conf","net":{"bindIp":"*","ipv6":false,"port":27017,"unixDomainSocket":{"enabled":true,"pathPrefix":"/opt/bitnami/mongodb/tmp"}},"processManagement":{"fork":false,"pidFilePath":"/opt/bitnami/mongodb/tmp/mongodb.pid"},"replication":{"enableMajorityReadConcern":true,"replSetName":"rs0"},"security":{"authorization":"disabled","keyFile":"/opt/bitnami/mongodb/conf/keyfile"},"setParameter":{"enableLocalhostAuthBypass":"true"},"storage":{"dbPath":"/bitnami/mongodb/data/db","directoryPerDB":false,"journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"logRotate":"reopen","path":"/opt/bitnami/mongodb/logs/mongodb.log","quiet":false,"verbosity":0}}}}
{"t":{"$date":"2022-10-29T21:25:06.013+00:00"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"DBPathInUse: Unable to create/open the lock file: /bitnami/mongodb/data/db/mongod.lock (Read-only file system). Ensure the user executing mongod is the owner of the lock file and has the appropriate permissions. Also make sure that another mongod instance is not already running on the /bitnami/mongodb/data/db directory"}}
{"t":{"$date":"2022-10-29T21:25:06.013+00:00"},"s":"I", "c":"REPL", "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":10000}}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"COMMAND", "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"NETWORK", "id":20562, "ctx":"initandlisten","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"NETWORK", "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"STORAGE", "id":4784906, "ctx":"initandlisten","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"-", "id":20520, "ctx":"initandlisten","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"REPL", "id":4784907, "ctx":"initandlisten","msg":"Shutting down the replica set node executor"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"NETWORK", "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"CONTROL", "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"STORAGE", "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"STORAGE", "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"-", "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2022-10-29T21:25:06.014+00:00"},"s":"I", "c":"FTDC", "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"}
{"t":{"$date":"2022-10-29T21:25:06.015+00:00"},"s":"I", "c":"CONTROL", "id":20565, "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2022-10-29T21:25:06.015+00:00"},"s":"I", "c":"CONTROL", "id":23138, "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}
Update
I checked the syslog and before the the logs Nov 14 23:07:17 k8s-worker2 kubelet[752]: E1114 23:07:17.749057 752 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"mongodb\" with CrashLoopBackOff: \"back-off 10s restarting failed container=mongodb pod=mongodb-2_mongodb(314f2776-ced4-4ba3-b90b-f927dc079770)\"" pod="mongodb/mongodb-2" podUID=314f2776-ced4-4ba3-b90b-f927dc079770
I find these logs:
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341806] sd 2:0:0:1: [sda] tag#42 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=11s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341866] sd 2:0:0:1: [sda] tag#42 Sense Key : Medium Error [current]
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341891] sd 2:0:0:1: [sda] tag#42 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341899] sd 2:0:0:1: [sda] tag#42 CDB: Write(10) 2a 00 00 85 1f b8 00 00 40 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.341912] blk_update_request: critical medium error, dev sda, sector 8724408 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.352012] Aborting journal on device sda-8.
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.354980] EXT4-fs error (device sda) in ext4_reserve_inode_write:5726: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.355103] sd 2:0:0:1: [sda] tag#40 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357056] sd 2:0:0:1: [sda] tag#40 Sense Key : Medium Error [current]
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357061] sd 2:0:0:1: [sda] tag#40 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357066] sd 2:0:0:1: [sda] tag#40 CDB: Write(10) 2a 00 00 44 14 88 00 00 10 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357068] blk_update_request: critical medium error, dev sda, sector 4461704 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.357088] EXT4-fs error (device sda): ext4_dirty_inode:5922: inode #131080: comm mongod: mark_inode_dirty error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.359566] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131081 starting block 557715)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.361432] EXT4-fs error (device sda) in ext4_dirty_inode:5923: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.362792] Buffer I/O error on device sda, logical block 557713
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.364010] Buffer I/O error on device sda, logical block 557714
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365222] sd 2:0:0:1: [sda] tag#43 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=8s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365228] sd 2:0:0:1: [sda] tag#43 Sense Key : Medium Error [current]
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365230] sd 2:0:0:1: [sda] tag#43 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365233] sd 2:0:0:1: [sda] tag#43 CDB: Write(10) 2a 00 00 44 28 38 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.365234] blk_update_request: critical medium error, dev sda, sector 4466744 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.367434] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131083 starting block 558344)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.367442] Buffer I/O error on device sda, logical block 558343
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368593] sd 2:0:0:1: [sda] tag#41 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368597] sd 2:0:0:1: [sda] tag#41 Sense Key : Medium Error [current]
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368599] sd 2:0:0:1: [sda] tag#41 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368602] sd 2:0:0:1: [sda] tag#41 CDB: Write(10) 2a 00 00 44 90 70 00 00 10 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.368604] blk_update_request: critical medium error, dev sda, sector 4493424 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370907] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131081 starting block 561680)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370946] sd 2:0:0:1: [sda] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370949] sd 2:0:0:1: [sda] tag#39 Sense Key : Medium Error [current]
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370952] sd 2:0:0:1: [sda] tag#39 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370949] EXT4-fs error (device sda): ext4_journal_check_start:83: comm kworker/u4:0: Detected aborted journal
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.370954] sd 2:0:0:1: [sda] tag#39 CDB: Write(10) 2a 00 00 10 41 98 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.372081] blk_update_request: critical medium error, dev sda, sector 1065368 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.374353] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131080 starting block 133172)
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.374396] Buffer I/O error on device sda, logical block 133171
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.388492] EXT4-fs error (device sda) in __ext4_new_inode:1136: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.390763] EXT4-fs error (device sda) in ext4_create:2786: Journal has aborted
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.391732] sd 2:0:0:1: [sda] tag#46 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392941] sd 2:0:0:1: [sda] tag#46 Sense Key : Medium Error [current]
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392944] sd 2:0:0:1: [sda] tag#46 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392948] sd 2:0:0:1: [sda] tag#46 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.392950] blk_update_request: critical medium error, dev sda, sector 0 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.395562] Buffer I/O error on dev sda, logical block 0, lost sync page write
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396945] sd 2:0:0:1: [sda] tag#45 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396953] sd 2:0:0:1: [sda] tag#45 Sense Key : Medium Error [current]
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396955] sd 2:0:0:1: [sda] tag#45 Add. Sense: Unrecovered read error
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396958] sd 2:0:0:1: [sda] tag#45 CDB: Write(10) 2a 08 00 84 00 00 00 00 08 00
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396959] blk_update_request: critical medium error, dev sda, sector 8650752 op 0x1:(WRITE) flags 0x20800 phys_seg 1 prio class 0
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.396930] EXT4-fs (sda): I/O error while writing superblock
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.399771] Buffer I/O error on dev sda, logical block 1081344, lost sync page write
Nov 14 23:06:59 k8s-worker2 kernel: [3413829.403897] JBD2: Error -5 detected when updating journal superblock for sda-8.
Nov 14 23:07:01 k8s-worker2 systemd[1]: run-docker-runtime\x2drunc-moby-d1c0f0dc3e024723707edfc12e023b98fb98f1be971177ecca5ac0cfdc91ab87-runc.w3zzIL.mount: Deactivated successfully.
Nov 14 23:07:05 k8s-worker2 kubelet[752]: E1114 23:07:05.415798 752 dns.go:157] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 46.38.252.230 46.38.225.230 2a03:4000:0:1::e1e6"
Nov 14 23:07:06 k8s-worker2 kubelet[752]: E1114 23:07:06.412219 752 dns.go:157] "Nameserver limits exceeded" err="Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 46.38.252.230 46.38.225.230 2a03:4000:0:1::e1e6"
Nov 14 23:07:06 k8s-worker2 systemd[1]: run-docker-runtime\x2drunc-moby-d1c0f0dc3e024723707edfc12e023b98fb98f1be971177ecca5ac0cfdc91ab87-runc.nK23K3.mount: Deactivated successfully.
Nov 14 23:07:11 k8s-worker2 systemd[1]: run-docker-runtime\x2drunc-moby-d1c0f0dc3e024723707edfc12e023b98fb98f1be971177ecca5ac0cfdc91ab87-runc.L5TkRU.mount: Deactivated successfully.
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411831] sd 2:0:0:1: [sda] tag#44 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411888] sd 2:0:0:1: [sda] tag#44 Sense Key : Medium Error [current]
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411898] sd 2:0:0:1: [sda] tag#44 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411952] sd 2:0:0:1: [sda] tag#44 CDB: Write(10) 2a 00 00 44 28 40 00 00 50 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.411965] blk_update_request: critical medium error, dev sda, sector 4466752 op 0x1:(WRITE) flags 0x0 phys_seg 10 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.419273] EXT4-fs warning (device sda): ext4_end_bio:344: I/O error 7 writing to inode 131083 starting block 558354)
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430398] sd 2:0:0:1: [sda] tag#47 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=15s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430407] sd 2:0:0:1: [sda] tag#47 Sense Key : Medium Error [current]
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430409] sd 2:0:0:1: [sda] tag#47 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430412] sd 2:0:0:1: [sda] tag#47 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.430415] blk_update_request: critical medium error, dev sda, sector 0 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.433686] Buffer I/O error on dev sda, logical block 0, lost sync page write
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.436088] EXT4-fs (sda): I/O error while writing superblock
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444291] sd 2:0:0:1: [sda] tag#32 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=14s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444300] sd 2:0:0:1: [sda] tag#32 Sense Key : Medium Error [current]
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444304] sd 2:0:0:1: [sda] tag#32 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444308] sd 2:0:0:1: [sda] tag#32 CDB: Write(10) 2a 00 00 41 01 18 00 00 08 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.444313] blk_update_request: critical medium error, dev sda, sector 4260120 op 0x1:(WRITE) flags 0x3000 phys_seg 1 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.449491] Buffer I/O error on dev sda, logical block 532515, lost async page write
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453591] sd 2:0:0:1: [sda] tag#33 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453600] sd 2:0:0:1: [sda] tag#33 Sense Key : Medium Error [current]
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453603] sd 2:0:0:1: [sda] tag#33 Add. Sense: Unrecovered read error
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453607] sd 2:0:0:1: [sda] tag#33 CDB: Write(10) 2a 08 00 00 00 00 00 00 08 00
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.453610] blk_update_request: critical medium error, dev sda, sector 0 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.459072] Buffer I/O error on dev sda, logical block 0, lost sync page write
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.461189] EXT4-fs (sda): I/O error while writing superblock
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.464347] EXT4-fs (sda): Remounting filesystem read-only
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.466527] EXT4-fs (sda): failed to convert unwritten extents to written extents -- potential data loss! (inode 131081, error -30)
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.470833] Buffer I/O error on device sda, logical block 561678
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.473548] Buffer I/O error on device sda, logical block 561679
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.477384] EXT4-fs (sda): failed to convert unwritten extents to written extents -- potential data loss! (inode 131083, error -30)
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.482014] Buffer I/O error on device sda, logical block 558344
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.484881] Buffer I/O error on device sda, logical block 558345
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.487224] Buffer I/O error on device sda, logical block 558346
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.488837] Buffer I/O error on device sda, logical block 558347
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.490543] Buffer I/O error on device sda, logical block 558348
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.492061] Buffer I/O error on device sda, logical block 558349
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.493494] Buffer I/O error on device sda, logical block 558350
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.494931] Buffer I/O error on device sda, logical block 558351
Not sure, if this is really related to the problem.
Generally when you see this error message:
"error":"DBPathInUse: Unable to create/open the lock file: /bitnami/mongodb/data/db/mongod.lock (Read-only file system)
It most probably mean that your mongodb pod did not shutted down gracefully and had no time to remove the mongod.lock file so when your pod was re-created in another k8s node the "new" mongod process cannot start because it is finding the previous mongod.lock file.
The easiest way to resolve the current availability issue is to scale up and add immediately one more replicaSet member so the new member to init-sync from the available good member:
helm upgrade mongodb bitnami/mongodb \
--set architecture=replicaset \
--set auth.replicaSetKey=myreplicasetkey \
--set auth.rootPassword=myrootpassword \
--set replicaCount=3
and elect again primary.
You can check if mongoDB replicaSet elected PRIMARY from mongo shell inside the pod with the command:
rs.status()
For affected pod with the issue you can do as follow:
You can plan maitenance window and scale down ( scaling down stateFullset do not expect to automatically delete the pvc/pv , but good to make backup just in case.
After you scale down you can start custom helper pod to mount the pv so you can remove the mongod.lock file:
Temporary pod that you will start to mount the affected dbPath and remove the mongodb.lock file:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: mongo-pvc-helper
spec:
securityContext:
runAsUser: 0
containers:
- command:
- sh
- -c
- while true ; do echo alive ; sleep 10 ; done
image: busybox
imagePullPolicy: Always
name: mongo-pvc-helper
resources: {}
securityContext:
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /mongodata
name: mongodata
volumes:
- name: mongodata
persistentVolumeClaim:
claimName: <your_faulty_pod_pvc_name>
EOF
After you start the pod you can do:
kubectl exec mongo-pvc-helper -it sh
$ chown -R 0:0 /mongodata
$ rm /mongodata/mongod.lock
$ exit
Or you can complete wipe up the entire pv(if you prefer safely to init-sync entirely this member):
rm -rf /mongodata/*
And terminate the pod so you can finish the process:
kubectl delete pod mongo-pvc-helper
And again scale-up:
helm upgrade mongodb bitnami/mongodb \
--set architecture=replicaset \
--set auth.replicaSetKey=myreplicasetkey \
--set auth.rootPassword=myrootpassword \
--set replicaCount=2
Btw, good to have at least 3x data members in replicaSet for better redundancy to allow during single member down event election to keep still the PRIMARY up and running...
How to troubleshoot this further:
Ensure your pods have the terminationGracePeriod set (at least 10-20 sec) so it allow some time for the mongod process to flush data to storage and remove the mongod.lock file.
Depending from pod memory limits/requests , you can set some safer value for storage.wiredTiger.engineConfig.cacheSizeGB (if not set it is allocating ~50% from memory ).
Check the kubelet logs from node where pod was killed there maybe more details why pod was killed.
I think #R2D2's extensive answer makes some good points about how to recover from the situation. I very much agree with their recommendation to use 3 data bearing nodes which aligns with fault tolerance considerations. With the additional logs you were able to add, I am arriving at the same conclusion that your storage subsystem is the problem here which is going to be the actual cause of your MongoDB failing.
In your initial query the following log line was specifically highlighted:
DBPathInUse: Unable to create/open the lock file: /bitnami/mongodb/data/db/mongod.lock (Read-only file system). Ensure the user executing mongod is the owner of the lock file and has the appropriate permissions. Also make sure that another mongod instance is not already running on the /bitnami/mongodb/data/db directory
Specifically: (Read-only file system). Now in the new logs you have provided the host itself is reporting:
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.459072] Buffer I/O error on dev sda, logical block 0, lost sync page write
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.461189] EXT4-fs (sda): I/O error while writing superblock
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.464347] EXT4-fs (sda): Remounting filesystem read-only
Nov 14 23:07:14 k8s-worker2 kernel: [3413844.466527] EXT4-fs (sda): failed to convert unwritten extents to written extents -- potential data loss! (inode 131081, error -30)
Specifically Remounting filesystem to read-only. If mongod is using any of these mount points for its operation then we would expect the system to no longer be able to function properly if it can no longer write to them. The database process itself may terminate, which is something the storage node watchdog could be configured to do (in subsequent versions).
In any case, the issues with the storage look quite serious, they include text like this: failed to convert unwritten extents to written extents -- potential data loss! It seems imperative that you look into this further and resolve any issues as soon as possible.
Relatedly, you mentioned:
I'm using longhorn (before I tried nfs) for the storage
The logs also suggest EXT4-fs is at play here. I think all of these have been known to have issues or otherwise be suboptimal for usage with MongoDB. From their documentation:
With the WiredTiger storage engine, using XFS is strongly recommended for data bearing nodes to avoid performance issues that may occur when using EXT4 with WiredTiger.
From elsewhere on the same page (emphasis added):
With the WiredTiger storage engine, WiredTiger objects may be stored on remote file systems if the remote file system conforms to ISO/IEC 9945-1:1996 (POSIX.1). Because remote file systems are often slower than local file systems, using a remote file system for storage may degrade performance.
I don't have any personal experience with Longhorn, but you can see an example here where instability with that storage system caused the same DBPathInUse error that you observed. There are other reports of people having nothing but problems with storage constantly detaching itself.
In short - instability with the storage subsystem is what is both causing the mongod process/pod to fail as well as preventing it from recovering. The problem is compounded by the fact that you only have 2 members in the replica set which provides no fault tolerance. Once you lose one member the other one will not be able to operate as a PRIMARY since there is no majority. Increasing the replica set to 3 members will at least provide fault tolerance of 1 node. The storage issues are a separate problem that should be pursued further via another question focused more on how that component is configured in your environment.
Some time ago I had something like that. That is always sad experience.
According to answer done by #R2D2. When you see (Read-only file system) in your logs - it can mean many things all not good. For instance when Linux starts file system is read-only, when everything is OK it is switched to read-write. That is not your case - so - just an example.
Please see that file system was marked as read-only due to io-errors. Looks like hard drive is corrupted. Check system, on which Kubernetes is running - fsck for Linux - like described here.
When drive is fixed restart Kubernetes - some data is lost, count on mongo complaining about data integrity... Nothing more than mongod --repair comes to my mind. Aaaand it can be that lock file should also be deleted before repair, but it should complain about it - like - "there is another instance", or "I can't set lock - file exists".
Besides that - use SMART monitoring, also mentioned later at the page.
Newer, faster, bigger drives are also more fragile. That is the price.
If you have backup... Yes I know - I've mentioned about my case - since then I have backup... Good luck!

Modem on Raspbian not connecting with `nmcli`

I need to configure the [TRM240 modem][1] to work with a Raspbian system. I followed the procedure reported in the answer [here][2], but the connection isn't working.
I attach some screenshots documenting the procedure and the commands that I issued.
First, look for connected modems:
pi#raspberrypi:~ $ mmcli -L
Found 1 modems:
/org/freedesktop/ModemManager1/Modem/0 [QUALCOMM INCORPORATED] QUECTEL Mobile Broadband Module
Check the unique modem found:
pi#raspberrypi:~ $ mmcli -m 0
/org/freedesktop/ModemManager1/Modem/0 (device id 'e308f44de689f834e8cbc5b2122a4ffca8ddd3f5')
-------------------------
Hardware | manufacturer: 'QUALCOMM INCORPORATED'
| model: 'QUECTEL Mobile Broadband Module'
| revision: 'EC21ECGAR06A04M1G'
| supported: 'gsm-umts
| lte
| gsm-umts, lte'
| current: 'gsm-umts, lte'
| equipment id: '864394040056931'
-------------------------
System | device: '/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2'
| drivers: 'option1, qmi_wwan'
| plugin: 'Generic'
| primary port: 'cdc-wdm0'
| ports: 'ttyUSB0 (qcdm), ttyUSB2 (at), cdc-wdm0 (qmi), wwan0 (net), ttyUSB3 (at)'
-------------------------
Numbers | own : 'unknown'
-------------------------
Status | lock: 'unknown'
| unlock retries: 'unknown'
| state: 'failed'
| failed reason: 'sim-missing'
| power state: 'unknown'
| access tech: 'unknown'
| signal quality: '0' (cached)
-------------------------
Modes | supported: 'allowed: 2g, 3g, 4g; preferred: none'
| current: 'allowed: any; preferred: none'
-------------------------
Bands | supported: 'unknown'
| current: 'unknown'
-------------------------
IP | supported: 'ipv4, ipv6, ipv4v6'
-------------------------
SIM | path: 'none'
-------------------------
Bearers | paths: 'none'
Here, the Status section leaves me a little bit puzzled, but I try to go on.
I tried the following command using different options for ifname: cdc-wdm0, ttyUSB2 and ttyUSB3. They all end up to the same result.
pi#raspberrypi:~ $ sudo nmcli c add type gsm ifname cdc-wdm0 con-name modem apn mobile.vodafone.it
Connection 'modem' (9c34027f-5df5-4fd6-9792-223e225dfd41) successfully added.
Check the connections and try to take up modem:
pi#raspberrypi:~ $ nmcli con
NAME UUID TYPE DEVICE
br-12ff13c2c4e0 26450e11-6671-47c2-91af-6bf96405921b bridge br-12ff13c2c4e0
br-acc4bdcbe6e7 440cfcc0-5318-4a9e-9259-3fb8700cf240 bridge br-acc4bdcbe6e7
docker0 8af67071-3cb2-4235-882f-1cc1f50fa7ca bridge docker0
modem 9c34027f-5df5-4fd6-9792-223e225dfd41 gsm --
Try to take up:
pi#raspberrypi:~ $ nmcli con up modem
Error: Connection activation failed: No suitable device found for this connection.
Status of service NetworkManager:
pi#raspberrypi:~ $ systemctl status NetworkManager
● NetworkManager.service - Network Manager
Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-08-21 07:46:57 UTC; 2h 35min ago
Docs: man:NetworkManager(8)
Main PID: 710 (NetworkManager)
Tasks: 3 (limit: 4915)
Memory: 13.8M
CPU: 3.862s
CGroup: /system.slice/NetworkManager.service
└─710 /usr/sbin/NetworkManager --no-daemon
Aug 21 07:48:53 raspberrypi NetworkManager[710]: <info> [1597996133.0883] device (veth6ffb100): link connected
Aug 21 08:14:11 raspberrypi NetworkManager[710]: <error> [1597997651.4341] audit: failed to open auditd socket: Protocol not supported
Aug 21 08:14:11 raspberrypi NetworkManager[710]: <info> [1597997651.4343] audit: op="device-managed" arg="managed:1" pid=11841 uid=0 result="success"
Aug 21 08:16:48 raspberrypi NetworkManager[710]: <info> [1597997808.9388] audit: op="device-autoconnect" arg="autoconnect:1" pid=12910 uid=1000 result="fail" reason="Not authorized to perform this operat
Aug 21 08:17:41 raspberrypi NetworkManager[710]: <info> [1597997861.0575] audit: op="connection-add-activate" pid=13286 uid=1000 result="fail" reason="gsm: GSM mobile broadband connection requires a 'gsm
Aug 21 08:25:25 raspberrypi NetworkManager[710]: <info> [1597998325.2594] audit: op="connection-add-activate" pid=16224 uid=1000 result="fail" reason="gsm: GSM mobile broadband connection requires a 'gsm
Aug 21 10:06:31 raspberrypi NetworkManager[710]: <info> [1598004391.9223] audit: op="connection-activate" uuid="06583333-e03e-431f-9376-94c38b6b607d" name="modem" result="fail" reason="No suitable device
Aug 21 10:19:24 raspberrypi NetworkManager[710]: <info> [1598005164.3349] keyfile: add connection /etc/NetworkManager/system-connections/modem-9c34027f-5df5-4fd6-9792-223e225dfd41 (9c34027f-5df5-4fd6-979
Aug 21 10:19:24 raspberrypi NetworkManager[710]: <info> [1598005164.3386] audit: op="connection-add" uuid="9c34027f-5df5-4fd6-9792-223e225dfd41" name="modem" pid=27261 uid=0 result="success"
Aug 21 10:20:29 raspberrypi NetworkManager[710]: <info> [1598005229.6409] audit: op="connection-activate" uuid="06583333-e03e-431f-9376-94c38b6b607d" name="modem" result="fail" reason="No suitable device
lines 1-21/21 (END)
Here I see the error <error> [1597997651.4341] audit: failed to open auditd socket: Protocol not supported...
Status of service ModemManager (there are warnings, but not errors):
pi#raspberrypi:~ $ systemctl status ModemManager
● ModemManager.service - Modem Manager
Loaded: loaded (/lib/systemd/system/ModemManager.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-08-21 07:46:57 UTC; 2h 35min ago
Main PID: 712 (ModemManager)
Tasks: 5 (limit: 4915)
Memory: 8.7M
CPU: 4.123s
CGroup: /system.slice/ModemManager.service
├─712 /usr/sbin/ModemManager
└─884 /usr/lib/libqmi/qmi-proxy
Aug 21 07:47:19 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Registered 'nas' (version 1.25) client with ID '2'
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Allocating new client ID...
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Registered 'wms' (version 1.10) client with ID '1'
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Allocating new client ID...
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Registered 'uim' (version 1.46) client with ID '1'
Aug 21 07:47:20 raspberrypi ModemManager[712]: <warn> (ttyUSB2): port attributes not fully set
Aug 21 07:47:20 raspberrypi ModemManager[712]: <warn> couldn't load Supported Bands: 'QMI operation failed: Cannot send message: QMI service 'dms' version '1.1' required, got version '1.0''
Aug 21 07:47:20 raspberrypi ModemManager[712]: <warn> couldn't load Power State: 'QMI operation failed: Cannot send message: QMI service 'dms' version '1.1' required, got version '1.0''
Aug 21 07:47:55 raspberrypi ModemManager[712]: <warn> Modem couldn't be initialized: Couldn't check unlock status: Couldn't get SIM lock status after 6 retries
Aug 21 07:47:55 raspberrypi ModemManager[712]: <info> Modem: state changed (unknown -> failed)
System info:
pi#raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux
pi#raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 9.13 (stretch)
Release: 9.13
Codename: stretch
It may be that the SIM is not working properly or the modem is not recognizing it.
This is what the output
-------------------------
Status | lock: 'unknown'
| unlock retries: 'unknown'
| state: 'failed'
| failed reason: 'sim-missing'
| power state: 'unknown'
| access tech: 'unknown'
| signal quality: '0' (cached)
-------------------------
points out.
Check this answer.
(I didn't find it before posting my question.)

varnish 4.1 default.vcl permissions denied

When I'm trying to add magento 2 varnish.vcl file by creating a symbolic link, varnish service stop working with error permission denied, while if I use default varnish configuration file varnish works smooth.
My Stack is ubuntu 16.04, varnish 4.1
ls -al
drwxr-xr-x 2 root root 4096 Mar 21 13:14 .
drwxr-xr-x 96 root root 4096 Mar 21 12:56 ..
lrwxrwxrwx 1 root root 44 Mar 21 13:14 default.vcl -> /var/www/bazaar/varnish.vcl
-rw-r--r-- 1 root root 1225 Aug 22 2017 default.vcl_bak
-rw-r--r-- 1 root root 37 Mar 21 12:56 secret
here is the status for varnish service
● varnish.service - Varnish HTTP accelerator
Loaded: loaded (/lib/systemd/system/varnish.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/varnish.service.d
└─customexec.conf
Active: failed (Result: exit-code) since Wed 2018-03-21 13:59:08 UTC; 2s ago
Docs: https://www.varnish-cache.org/docs/4.1/
man:varnishd
Process: 3093 ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m (code=exited, status=2)
Main PID: 3093 (code=exited, status=2)
Mar 21 13:59:08 bazaar systemd[1]: Stopped Varnish HTTP accelerator.
Mar 21 13:59:08 bazaar systemd[1]: Started Varnish HTTP accelerator.
Mar 21 13:59:08 bazaar varnishd[3093]: Error: Cannot read -f file (/etc/varnish/default.vcl): Permission denied
Mar 21 13:59:08 bazaar systemd[1]: varnish.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 21 13:59:08 bazaar systemd[1]: varnish.service: Unit entered failed state.
Mar 21 13:59:08 bazaar systemd[1]: varnish.service: Failed with result 'exit-code'.
my current user for nginx is bazaar
and permissions for varnish.vcl is as follow
-rw-r--r-- 1 bazaar bazaar 7226 Mar 21 13:24 varnish.vcl
Any hint or help will be highly appreciated.
Thanks.
It is likely that the user (vcache) does not have access to read in the parent directory(s) /var/www/bazaar.

Finding device base address to communicate via inb() and outb()

I am trying to communicate with a disk drive using inb(), inw(), outb() and outw() commands so I can find specific information about the drive. However, to use these commands, I need the correct I/O ports for the device. When I have the correct I/O ports, I can find the information I am looking for very easily, however, I do not know a way to find the base address of a device's I/O ports in Linux.
In DOS, I am able to use Hdat2 to find the device's base address, however, I am trying to find the address in Linux. Is there a way to find which device maps to which I/O port in Linux?
There is a file in /proc called ioports that contains some information but I don't how to associate this information with specific devices.
Any help would be greatly appreciated. Thanks!
So I did find something, although it isn't the most elegant solution and it definitely doesn't work everywhere, it has worked on my hardware so I figured I would share.
First, you have to get the address of the SATA Controller from the lspci command like Nikolai showed (the -D just shows the full domain numbers):
# lspci -D
...
0000:00:1f.2 SATA controller: Intel Corporation 82801IR 6 port SATA AHCI Controller
...
Now with this address (0000:00:1f.2) you can go into /sys.
In /sys/bus/pci/devices, your device should be listed:
# ls -l /sys/bus/pci/devices
...
lrwxrwxrwx 1 root root 0 Jan 14 12:35 0000:00:1f.2 -> ../../../devices/pci0000:00/0000:00:1f.2
Now in this directory there will be several hostX directories.
# ls -l /sys/bus/pci/devices/0000\:00\:1f.2/
...
drwxr-xr-x 4 root root 0 Jan 13 12:40 host0
drwxr-xr-x 4 root root 0 Jan 13 12:40 host1
drwxr-xr-x 3 root root 0 Jan 13 12:40 host2
drwxr-xr-x 3 root root 0 Jan 13 12:40 host3
drwxr-xr-x 3 root root 0 Jan 13 12:40 host4
drwxr-xr-x 4 root root 0 Jan 14 08:21 host5
...
In one of these hostX directories, there will be a targetX:X:X directory. This targetX:X:X directory will then have a directory called X:X:X:X (the X's are numbers that can vary).
# ls -R /sys/bus/pci/devices/0000\:00\:1f.2/host0
/sys/bus/pci/devices/0000:00:1f.2/host0:
power scsi_host:host0 target0:0:0 uevent
/sys/bus/pci/devices/0000:00:1f.2/host0/target0:0:0:
0:0:0:0 power uevent
...
In the X:X:X:X directory, there is a link named "block:sdX" (where X is a letter). This sdX is the name of the drive that this directory corresponds to.
# ls -l /sys/bus/pci/devices/0000\:00\:1f.2/host0/target0\:0\:0/0\:0\:0\:0/
lrwxrwxrwx 1 root root 0 Jan 14 15:01 block:sda -> ../../../../../../block/sda
So /dev/sda corresponds to host 0 on the SATA Controller at 0000:00:1f.2. Now to find the address that we can use to talk to /dev/sda through inb() and outb() commands, we look in the file named "resource" in /sys/bus/pci/devices/0000:00:1f.2/.
# cat /sys/bus/pci/devices/0000\:00\:1f.2/resource
0x000000000000fe00 0x000000000000fe07 0x0000000000000101
0x000000000000fe10 0x000000000000fe13 0x0000000000000101
0x000000000000fe20 0x000000000000fe27 0x0000000000000101
0x000000000000fe30 0x000000000000fe33 0x0000000000000101
0x000000000000fec0 0x000000000000fedf 0x0000000000000101
0x00000000ff970000 0x00000000ff9707ff 0x0000000000000200
0x0000000000000000 0x0000000000000000 0x0000000000000000
The address we are looking for is fe00, which is on the first line. We want the first line because it is host 0, if it were host 1, we would look on the second line, and host 2 the third line, and so on. The host number was given by the hostX directory that we found earlier. Each line in the resource file is separated into 3 columns:
Column 1 = beginning address
Column 2 = end address
Column 3 = flags
So this is how I get from /dev/sda to 0xfe00 in order to send commands to the device.
If anybody know any better way to do this, I am all ears...
The device is most probably hanging off of the PCI bus, so lspci(8) is the first to look at. Then figure out where under /sys the controller is described. Here, for example, I have:
~$ lspci
...
03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 1078 (rev 04)
...
~$ ll /sys/bus/pci/devices/0000\:03\:00.0/
total 0
drwxr-xr-x 4 root root 0 Dec 16 11:57 ./
drwxr-xr-x 6 root root 0 Dec 16 11:57 ../
-rw-r--r-- 1 root root 4096 Dec 16 11:57 broken_parity_status
lrwxrwxrwx 1 root root 0 Dec 16 11:57 bus -> ../../../../bus/pci/
-r--r--r-- 1 root root 4096 Dec 16 11:57 class
-rw-r--r-- 1 root root 4096 Dec 16 11:57 config
-r--r--r-- 1 root root 4096 Dec 16 11:57 device
lrwxrwxrwx 1 root root 0 Dec 16 11:57 driver -> ../../../../bus/pci/drivers/megaraid_sas/
-rw------- 1 root root 4096 Dec 16 11:57 enable
drwxr-xr-x 5 root root 0 Dec 16 11:57 host0/
-r--r--r-- 1 root root 4096 Dec 16 11:57 irq
-r--r--r-- 1 root root 4096 Dec 16 11:57 local_cpus
-r--r--r-- 1 root root 4096 Dec 16 11:57 modalias
-r--r--r-- 1 root root 4096 Dec 16 11:57 pools
drwxr-xr-x 2 root root 0 Dec 16 11:57 power/
-r--r--r-- 1 root root 4096 Dec 16 11:57 resource
-rw------- 1 root root 262144 Dec 16 11:57 resource0
-rw------- 1 root root 256 Dec 16 11:57 resource2
-rw------- 1 root root 262144 Dec 16 11:57 resource3
-r-------- 1 root root 32768 Dec 16 11:57 rom
lrwxrwxrwx 1 root root 0 Dec 16 11:57 subsystem -> ../../../../bus/pci/
-r--r--r-- 1 root root 4096 Dec 16 11:57 subsystem_device
-r--r--r-- 1 root root 4096 Dec 16 11:57 subsystem_vendor
--w------- 1 root root 4096 Dec 16 11:57 uevent
-r--r--r-- 1 root root 4096 Dec 16 11:57 vendor
This shows controller's PCI configuration space. See the details in Linux Device Drivers, Third Edition. Chapter 12: PCI Drivers.
Edit:
Take a look into this partition and mass-storage naming howto for help on Linux drive naming.
Are you accessing hardware from a userspace program or from a kernel module?
If you're doing it from userspace, the reason it's hard to find physical address information is that nobody accesses hardware that way; anything that needs to touch raw hardware lives in the kernel.
If you're writing a kernel module, you get address information from in-kernel structures, not by accessing /sys/...
The path seems have chanced in kernel 3.10, this is how I find the corresponding device node:
$ ls -l /sys/bus/pci/devices/0000\:00\:1f.2/ata1/host0/target0\:0\:0/0\:0\:0\:0/block/
total 0
drwxr-xr-x 10 root root 0 Oct 17 08:35 sda
$ ls -l /sys/bus/pci/devices/0000\:00\:1f.2/ata2/host1/target1\:0\:0/1\:0\:0\:0/block/
total 0
drwxr-xr-x 9 root root 0 Oct 17 08:35 sdb