I am following this guide to set up my cluster. It all works fine.
However, when I install fabric8 in this cluster I run out of disk on the minions. The image, kube.vmdk, is only about 6GB. It is the /var/lib/docker which gets filled up. How do I solve this?
Using the GUI for vmware the option to resize the disk is 'greyed out'.
Should I attach a second disk to the minions and then mount this disk? Where should I mount it? /var/lib/docker?
I would appreciate any input.
Docker's image is store in /var/lib/docker(more precisely, it store in storage driver's directory, /var/lib/docker/aufs when using aufs storage driver) , so when Kubernetes report disk gets filled up, it check that directory.
So you can
Remove all the images in docker(not necessary, you can copy everything to new dir).
stop docker daemon.
mount your new disk to /var/lib/docker/ or /var/lib/docker
start docker daemon.
If you are not sure what storage driver your docker is using, type docker info in your node, will get something contain this:
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 139
Dirperm1 Supported: true
It seems that you run out of the space of the disk. You can remove all the files in /var/lib/docker, and mount the second disk. Finally you need restart your dockerd.
Related
I created a postgres VM in GCP using this instructions https://joncloudgeek.com/blog/deploy-postgres-container-to-compute-engine/#create-a-compute-instance-running-a-postgres-container with a 10GB disk, everything has worked fine for the last couple of months but I seem to have run out of space on /dev/sda1. So i increased the disk size to 400GB but I can't seem to be able to resize /dev/sda1 using the standard command "sudo growpart /dev/sda 1" I keep getting command not found.
Solution for me:
Create a machine image of the container.
Spin up a new VM based on the machine image created.
Delete old VM.
This created a new Postgres VM with 400GB of disk.
I created a virtual server but found that the space was too small, so I wanted to add additional disk space to it. After attaching it, I'm not able to see the newly attached disk using df -h in the virtual server. How can I see the newly attached disk?
There are several commands that you can use: fdisk -l and lsblk.
The first disk drive should appear as xvda. Usually, there is a second disk drive xvdb that is used for swap. Your new disk will appear as xvdc. Note that naming is OS-specific.
You will need to partition the new disk, format a file system and then mount the file system to a directory. The exact steps are OS-dependent.
Note: Following steps may change wrt OS. Below example specfic to IBM VSI with ubuntu 18.04 LTS
First confirm the disk is attached to VSI(virtual server instance)and check the number of disk at the node by fdisk command.
fdisk -l | grep xvdc
Create a directory for new partition to be mounted
mkdir /data1
Make the partition using fdisk command
fdisk /dev/xvdc
After the above command it will ask some info
n # And then hit enter until it creates the partition.
Format the partition using mkfs command mkfs.ext4
mkfs.ext4 /dev/xvdc
Mount the partitoined on the new directory
mount /dev/xvdc /data1
Check the disk and its mounted points
df -h
I am trying to read a file from server in spark cluster mode using kubernetes, so i put my file on all workers and i mount driver volume using
val conf = new SparkConf().setAppName("sparksetuptest")
.set("spark.kubernetes.driver.volumes.hostPath.host.mount.path", "/file-directory")
Everything works fine here but when i execute it shows that file not found at specific location.
So i mount directory to executor with .set("spark.kubernetes.executor.volumes.hostPath.host.mount.path", "/file-directory")
But now i am not able to execute program it stuck in a never ending process while fetching data.
Please suggest something, so that i can mount my directory with executor and read that file.
this is an example from nfs-example
spark.kubernetes.driver.volumes.nfs.images.options.server=example.com
spark.kubernetes.driver.volumes.nfs.images.options.path=/data
I think you need to declare the path that you want to mount under options.path and the spark.kubernetes.driver.volumes.[VolumeType].[VolumeName].mount.path is the mount path in your container
For example:
If I want to mount /home/lemon/data on the node of k8s to the path /data the docker container with VolumeName exepv, then
conf.set("spark.kubernetes.executor.volumes.hostPath.exepv.mount.path","/data")
conf.set("spark.kubernetes.executor.volumes.hostPath.exepv.options.path", "/home/lemon/data")
after this, you can access the path /data in your executor container
I am using the official mongo Docker image to start a MongoDB container where my boot disk is limited (e.g. 10G) I configured the docker to run with Google Cloud Logging driver and was hoping Google to store all the logs and save my local disk space. However, I notice the disk continues to grow:
$ df -h
/dev/sda1 9.9G 4.5G 4.9G 49%
As I digged deeper I realized the size of docker containers seems to be growing over time.
$ sudo du -sh /var/lib/docker/
3.6G /var/lib/docker/
However, I can't go further as somehow I can't access the directories within.
If I go inside the docker and du -sh the root, I don't find any suspicious directories occupying space.
So my problem is how do I find out where the disk space is used and how do I eliminate it.
My docker startup command (shown without project options)
docker run -d --log-driver=gcplogs mongo mongod
EDIT: I noticed the size growing has stopped at 4.5GB from ~3GB for a while. So I supposed it has reached some equilibrium now.
I am running Docker on windows and I have a database with some entries on disk at C:\data\db.
I want to add this database to my container. I have tried numerous ways to do this but failed.
I tried: docker run -p 27017:27017 -v //c/data/db:/data/db --name mongodb devops-mongodb
In my dockerfile I have:
RUN mkdir -p /data/db
VOLUME /data/db
But this doesn't add my current database on disk to the container. It creates a fresh /data/db directory and persists the data I add to it.
The docs here https://docs.docker.com/userguide/dockervolumes/ under 'Mount a host directory as a data volume' specifically told me to execute the -v //c/data/db:/data/db but this isn't working.
Any ideas?
You're using Boot2Docker (which runs inside a Virtual Machine). Boot2Docker uses VirtualBox guest additions to make directories on your Windows machine available to Docker running inside the Virtual Machine.
By default, only the C:\Users directory (on Windows), or /Users/ directory (on OS X) is shared with the virtual machine. Anything outside those directories is not shared with the Virtual Machine, which results in Docker creating an empty directory at the specified location for the volume.
To share directories outside C:\Users\ with the Virtual Machine, you have to manually configure Boot2Docker to share those. You can find the steps needed in the VirtualBox guest addition section of the README;
If some other path or share is desired, it can be mounted at run time by doing something like:
$ mount -t vboxsf -o uid=1000,gid=50 your-other-share-name /some/mount/location
It is also important to note that in the future, the plan is to have any share which is created in VirtualBox with the "automount" flag turned on be mounted during boot at the directory of the share name (ie, a share named home/jsmith would be automounted at /home/jsmith).
Please be aware that using VirtualBox guest additions have a really bad impact on performance (reading/writing to the volume will be really slow). Which could be fine for development, but should be used with caution.