I am newbie to docker and I have started my docker containers and I am not able to see the cpu/memory stats being written to the files in cgroup system slice files. Please have a look at the snapshot below
[root#docker docker-5ae16d70ee83102b0a8a65adc4468a9251e22ed8b8916e786564869da2b09524.scope]# ls -rlt
total 0
-r--r--r--. 1 root root 0 Aug 6 03:19 cpu.stat
Am I missing something here? Do I need to change anything to get the performance stats of containers
Related
I've changed log_driver to "local" in daemon.json docker configuration file, because an high activity level on rados gateway logs had satured disk space. My will was to change to journald to have logrotate. Unfortunately, after restart the docker daemon, many Ceph services did disappeared even as containers images. So now that node had caused an HEALTH_ERR because it lost 1 mgr, 1 mon and 3 osd services at the same time.
I've tried to use some ceph commands inside cephadm shell (on another node), but it freezes and nothing happened. What can I try to do to restore the node's services and cluster health?
I created a Postgres server locally using virualbox using Ubuntu 16.04. Using the import tool to move it to Google cloud seemed to work fine, but the root drive shows 100% full. None of the disk expansion instructions (including creating a snapshot and recreating the boot drive) seem to make any space available.
There seems to be a boot drive and a root drive. But the root drive shows it is all used. The boot drive shows space available, but it should be 15G in size not 720M.
Filesystem Size Used Avail Use% Mounted on
udev 1.8G 0 1.8G 0% /dev
tmpfs 370M 5.3M 365M 2% /run
/dev/mapper/techredo--vg-root 2.5G 2.5G 0 100% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sdb1 720M 121M 563M 18% /boot
tmpfs 370M 0 370M 0% /run/user/406485188
I checked if is possible to use LVM in GCP instances, and I find out that you're free to use it but is not supported by Google Cloud since instances doesn't use LVM by default.
On the other hand, you need to make sure that the Linux Guest Environment is installed in your instance, so you can get the automatic resizing feature. Please follow this guide to learn how to validate: https://cloud.google.com/compute/docs/images/install-guest-environment#wgei
Since your root partition is full and you're not able to install more programs I suggest you 2 workarounds:
Workaround 1: Create a new VirtualBox VM with and import it again, please note that your root partition is pretty small (2.5G) so I suggest you next time create a partition with at least 10GB, and avoid use LVM during the installation.
After your instance is ready in GCP, please check if the Linux Guest Environment is installed in your instance, if not install it: https://cloud.google.com/compute/docs/images/install-guest-environment
Workaround 2: Check which directory is causing problems and then which files are consuming your disk space, delete them to gain space, install the Guest enviroment and try to resize your instance.
a) To check the directories and files sizes, follow these steps:
There are several tools that can display your disk usage graphically but since your root partition is full you'll have to get the information by running commands (old school style).
Please follow these steps:
Please go to the root directory:
cd /
Please run this command to get the size of the main subdirectories under the root partition:
sudo du -aBM -d 1 . -R | sort -nr | head -20
NOTE: Identify which directory is eating your root partition.
Please run this command to get a full list of the files and its sizes:
du -k *| sort -nr | cut -f2 | xargs -d '\n' du -sh
NOTE: The above command will display all files and directories too fast, so in order to scroll down slowly, please run the same command adding the "less" instruction:
du -k *| sort -nr | cut -f2 | xargs -d '\n' du -sh |less
Press the spacebar to scroll down.
Please keep in mind that you have to go to the directory you want to analyze before running the commands in step 3 or 4 (just in case you want to analyze another directory).
Additional to this you can run the command "apt-get clean" to clear the downloaded packages (.deb files) that usually consumes a high part of your disk.
b) To resize your instance, you have 2 options:
Resize your VM instance "primary-server" by following this guide[1].
NOTE: The steps included in this option are pretty easy to follow, if this doesn't work try the second option which requires advanced Linux abilities.
Create a snapshot from the VM instance "primary-server".
2.1 Create a new instance based on a Linux distribution.
2.2 Once it's been created, stop the instance.
2.3 Follow this guide to add an additional disk[2].
NOTE: Basically you have to edit the instance "primary-server" and add an additional disk, don't forget to select the snapshot option from the "Source type" list and click on the snapshot you just created.
2.4 Start the instance.
2.5 Mount the disk by following this guide[3].
NOTE: Please skip step 4. The additional disk it's actually a boot disk so it's been already formatted. So, don't apply the format to it; just mount it, .
2.6 Check the permissions of the file "/etc/fstab".
NOTE: The permissions should be "-rw-r--r--" and the owner "root"
2.6.1 Delete files to reduce the disk size.
2.7 Unmount the disk at OS level.
2.8 Stop the instance.
2.9 Detach the additional disk from the new instance in GCP.
NOTE: Please follow this guide[4] and instead of clicking on X next to the boot disk, please click on X next to the additional disk.
2.10 Create a new instance and instead of using an image in the "boot disk" section, please use the disk you just restored.
NOTE: For this please go to the "Boot disk" section and click on the "Change" button, then go to the "Existing" TAB and select the disk you just restored.
REFERENCES:
[1] https://cloud.google.com/compute/docs/disks/add-persistent-disk#inaccessible_instance
[2] https://cloud.google.com/compute/docs/disks/add-persistent-disk#create_disk
[3] https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting
[4] https://cloud.google.com/compute/docs/disks/detach-reattach-boot-disk#detach_disk
Please let me know the results.
CephFS is evaluated as a distributed filesystem in my lab with bonnie++ in the
following setup:
Virtualbox VMs are used with the following settings:
RAM: 1024 MiB
One SATA hard drive for OS: 8 GiB
One dedicated SATA hard drive for distributed file system for each OSD: 8 GiB
OS: CentOS 7
Ceph version: 10.2
Ceph is configured for three (3) replicas, i.e.
three VMs are operated as OSDs:
node1 hosts ceph-mon, ceph-osd and ceph-mds
node2 and node3 host ceph-mon and ceph-osd.
admin node is idle
The client runs bonnie++ on a CephFS mounts.
Somehow, bonnie++ fails to run through. Instead, it exits with the following
error message:
Writing intelligently...Can't write block.: No space left on device
See attached screenshot:
Funnily enough, df -h reports that 8.7 GiB available disk space.
See next screenshot:
Is this normal ceph behaviour? What do you recommend to fix it?
I'm running the kubelet 1.5.4 and finding that even when I run low on disk space it will not delete old containers.
I have may exited containers, over an hour hold:
$ sudo docker ps -a | grep 'Exited (0) About an hour' | wc -l
78
But the kubelet does not seem to notice them (using a log level of 10 I see no references in kubelet logs to the container IDs).
Is this something that has changed recently? My previous 1.2 setup seemed to GC old containers.
Kubelet has a feature to gc images but not sure what you mean by gc'ing containers?
See the proposal spec here (it does seem as if it's due to be deprecated as well)
I'm experiencing some weird behavior on the google compute engine. I made a new instance with ubuntu on it. I installed a node app I'm working on, pulled code from github etc...
then I installed mongodb and nginx. The weird thing is, every time I leave the session, and reconnect, my mongodb and nginx installation files disappear.
for example, when I install nginx I find the nginx installation on /etc/nginx where I can find like nginx.conf. but when I left the compute engine console session, and reconnected later, that directory was gone. same thing is happening with mongodb.
my node installation under /home/abdul/mystuff doesn't disappear though.
is this normal? is it a setting?
details:
this is an ubuntu image (idk which version, and not sure how to check)
using the following to install nginx:
sudo apt-get update
sudo apt-get install nginx
result of command
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part /var/lib/docker/aufs
sdb 8:16 0 5G 0 disk
└─sdb1 8:17 0 5G 0 part /home
Looks like you're running a Docker container on your instance (/var/lib/docker/aufs) and installing the software inside the container.
If you want to save changes back to the image, it is possible to use the docker commit command, but this is almost definitely not what you want.
Instead, use a Dockerfile to build images and update it whenever you want to make a change. This way you can easily recreate the image and make changes without starting from scratch. For persistence (.e.g. config files and databases) use volumes, which are just directories stored outside of the Union File System as normal directories on the host.