I am trying to throttle the disk I/O of a Docker container using the blkio controller (without destroying the container), but I am unsure how to find out which device to run the throttling on.
The Docker container is running Mongo. Running a df -h inside the bash of the container gives the following:
root#82e7bdc56db0:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-202:1-524400-916a3c171357c4f0349e0145e34e7faf60720c66f9a68badcc09d05397190c64 10G 379M 9.7G 4% /
tmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/xvda1 32G 3.2G 27G 11% /data/db
shm 64M 0 64M 0% /dev/shm
Is there a way to find out which device to limit on the host machine? Thanks!
$docker info
Containers: 9
Running: 9
Paused: 0
Stopped: 0
Images: 6
Server Version: 1.12.1
Storage Driver: devicemapper
Pool Name: docker-202:1-524400-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.694 GB
Data Space Total: 107.4 GB
Data Space Available: 30.31 GB
Metadata Space Used: 3.994 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.143 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-38-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.675 GiB
Name: ip-172-31-6-72
ID: 4RCS:IMKM:A5ZT:H5IA:6B4B:M3IG:XGWK:2223:UAZX:GHNA:FUST:E5XC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
df -h on the host machine:
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 377M 6.1M 371M 2% /run
/dev/xvda1 32G 3.2G 27G 11% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 377M 0 377M 0% /run/user/1001
Related
Friends,
I'm trying to deploy a mysql cluster on minikube using the helm chart of Bitnami. It's not working apparently because of lack of space since I'm getting the following error: mkdir: cannot create directory '/bitnami/mysql/data': No space left on device.
I am running minikube (version: v1.15.0) on a macOS with 500GB RAM, more than the half of it is still free. Any ideas about how could I solve this problem?
I ssh into the minikube environment and run df -h. This the result:
$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 3.4G 487M 3.0G 15% /
devtmpfs 1.8G 0 1.8G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 18M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 1.9G 176K 1.9G 1% /tmp
**/dev/vda1 17G 16G 0 100% /mnt/vda1**
It seems it minikube is really out of space. What can be done in this case?
Here the complete logs of my pod:
mysql 17:08:49.22 Welcome to the Bitnami mysql container
mysql 17:08:49.22 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mysql
mysql 17:08:49.22 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mysql/issues
mysql 17:08:49.23
mysql 17:08:49.23 INFO ==> ** Starting MySQL setup **
mysql 17:08:49.24 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 17:08:49.24 INFO ==> Initializing mysql database
mkdir: cannot create directory '/bitnami/mysql/data': No space left on device
minikube stop && minikube delete then
minikube start --disk-size 50000mb
Since you are using Minikube on a MacOS, Minikube is virtualized. Thus even if you have space on your entire device, you need to allocate more space in the VM, as Abhijit previously mentioned.
Some of my friends told me that larger swap partition is very bad like thousands of web hits/min should be possible on my server,
The swap space is 16 GB, and i installed centos7 with "CWP" -> Control web panel
with enable csf, should i consider decrease swap partition space if it possible without formatting the server and how? or there is a solution to maintain this space not to harm the server,,
[root#server ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 201M 16G 2% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda2 1.8T 256G 1.5T 15% /
/dev/sdb1 1.8T 275G 1.5T 16% /backup
/dev/sda5 16G 83M 15G 1% /tmp
/dev/sda1 969M 187M 716M 21% /boot
tmpfs 3.2G 0 3.2G 0% /run/user/0
tmpfs 3.2G 0 3.2G 0% /run/user/1075
[root#server ~]#
I found that swap file i more effective and easier to do in my case, so i disabled the swap partition and created a swapfile instead..
Here how you do it exactly.
I am trying to debug the storage usage in my kubernetes pod. I have seen the pod is evicted because of Disk Pressure. When i login to running pod, the see the following
Filesystem Size Used Avail Use% Mounted on
overlay 30G 21G 8.8G 70% /
tmpfs 64M 0 64M 0% /dev
tmpfs 14G 0 14G 0% /sys/fs/cgroup
/dev/sda1 30G 21G 8.8G 70% /etc/hosts
shm 64M 0 64M 0% /dev/shm
tmpfs 14G 12K 14G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 14G 0 14G 0% /proc/acpi
tmpfs 14G 0 14G 0% /proc/scsi
tmpfs 14G 0 14G 0% /sys/firmware
root#deploy-9f45856c7-wx9hj:/# du -sh /
du: cannot access '/proc/1142/task/1142/fd/3': No such file or directory
du: cannot access '/proc/1142/task/1142/fdinfo/3': No such file or directory
du: cannot access '/proc/1142/fd/4': No such file or directory
du: cannot access '/proc/1142/fdinfo/4': No such file or directory
227M /
root#deploy-9f45856c7-wx9hj:/# du -sh /tmp
11M /tmp
root#deploy-9f45856c7-wx9hj:/# du -sh /dev
0 /dev
root#deploy-9f45856c7-wx9hj:/# du -sh /sys
0 /sys
root#deploy-9f45856c7-wx9hj:/# du -sh /etc
1.5M /etc
root#deploy-9f45856c7-wx9hj:/#
As we can see 21G is consumed, but when i try to run du -sh it just returns 227M. I would like to find out who(which directory) is consuming the space
According to the docs Node Conditions, DiskPressure has to do with conditions on the node causing kubelet to evict the pod. It doesn't necessarily mean it's the pod that caused the conditions.
DiskPressure
Available disk space and inodes on either the node’s root filesystem
or image filesystem has satisfied an eviction threshold
You may want to investigate what's happening on the node instead.
Looks like the process 1142 is still running and holding file descriptors and/or perhaps some space (You may have other processes and other file descriptors too not being released) Is it the kubelet?. To alleviate the problem you can verify that it's running and then kill it:
$ ps -Af | grep 1142
$ kill -9 1142
P.D. You need to provide more information about the processes and what's running on that node.
Can anyone tell me how do I check the current available volume, consumed space, on my Mongodb pod on the new Openshift Online Platform? After allocating around 4GB of space, I am unclear on what's the volume that has been consumed until now.
Any light thrown on this will help. Thank you.
You can run oc exec <mongodb_pod> -- df -H .
For example, I get the following output when I run it for the sample python app in my cluster:
$ oc exec os-sample-python-1-hrbq7 -- df -H
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-253:1-94278-dbe79cf53785ab8a1b083f53c88088ab667a01f45e8a8725b26fbff82eef2a33 11G 689M 11G 7% /
tmpfs 4.2G 0 4.2G 0% /dev
tmpfs 4.2G 0 4.2G 0% /sys/fs/cgroup
/dev/vda1 11G 3.0G 7.8G 28% /etc/hosts
shm 68M 0 68M 0% /dev/shm
tmpfs 4.2G 17k 4.2G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 4.2G 0 4.2G 0% /proc/scsi
I have a problem which I can't find the answer to anywhere online - so I thought I would ask it here. Below is a code snippet of the directories in my Raspberry Pi. I have a 16GB SD Card installed in my Pi, and the total size of all those directories is only about 5GB. When I try and install anything it says "Out of Memory", and I have already cleared all log files, etc to try and free up space.
But the real question is, why does it say full when it has a 16GB card and I've only installed apache2 and php5 on it?
edward#raspberrypi:/ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 3.6G 3.5G 0 100% /
devtmpfs 214M 0 214M 0% /dev
tmpfs 218M 0 218M 0% /dev/shm
tmpfs 218M 4.6M 213M 3% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 218M 0 218M 0% /sys/fs/cgroup
/dev/mmcblk0p1 60M 20M 41M 34% /boot
tmpfs 44M 0 44M 0% /run/user/1000
tmpfs 44M 0 44M 0% /run/user/1001
Thanks for any help.
Try to type in: raspi-config and after that a menu pops up, with the option to expand your file system.