We want to run a data science model using docker images in Azure batch service.
Our docker image size is 20 GB to 25 GB. Can it be run on Azure batch service?
It depends. Currently, Azure Batch, by default using standard image configurations:
On Linux, maps container images on disk to the local temporary disk (or resource disk) with the azure-batch published container images. The selected VM Size and its local disk size and free space, would determine if your image would fit. Note that all task data would reside on this disk as well.
On Windows, maps container images (unmodified) to the OS disk. The size and free space of the OS disk would determine if your image would fit or not.
Generally speaking, a container image of that size is unwieldy and you would incur very large costs in not only distribution time for non-trivial pool sizes, but also unpack time.
Related
context
Our current context is the following: researchers are running HPC calculations on our Kubernetes cluster. Unfortunately, some pods cannot get scheduled because the container engine (here Docker) is not able to pull the images because the node is running out of disk space.
hypotheses
images too big
The first hypothesis is that the images are too big. This probably the case because we know that some images are bigger than 7 GB.
datasets being decompressed locally
Our second hypothesis is that some people are downloading their datasets locally (e.g. curl ...) and inflate them locally. This would generate the behavior we are observing.
Envisioned solution
I believe that this problem is a good case for a daemon set that would have access to the node's file system. Typically, this pod would calculate the total disk space used by all the pods on the node and would expose them as a Prometheus metric. From there is would be easy to set alert rules in place to check which pods have grown a lot over a short period of time.
How to calculate the total disk space used by a pod?
The question then becomes: is there a way to calculate the total disk space used by a pod?
Does anyone have any experience with this?
Kubernetes does not track overall storage available. It only knows things about emptyDir volumes and the filesystem backing those.
For calculating total disk space you can use below command
kubectl describe nodes
From the above output of the command you can grep ephemeral-storage which is the virtual disk size; this partition is also shared and consumed by Pods via emptyDir volumes, image layers,container logs and container writable layers.
Check where the process is still running and holding file descriptors and/or perhaps some space (You may have other processes and other file descriptors too not being released). Check Is that kubelet.
You can verify by running $ ps -Af | grep xxxx
With Prometheus you can calculate with the below formula
sum(node_filesystem_size_bytes)
Please go through Get total and free disk space using Prometheus for more information.
We're planning to migrate our software to run in kubernetes with auto scalling, this is our current infrastructure:
PHP and apache are running in Google Compute Engine n1-standard-4 (4 vCPUs, 15 GB memory)
MySql is running in Google Cloud SQL
Data files (csv, pdf) and the code are storing in a single SSD Persistent Disk
I found many posts that recomments to store the data file in the Google Cloud Storage and use the API to fetch the file and uploading to the bucket. We have very limited time so I decide to use NFS to share the data files over the pods, the problem is nfs speed is slow, it's around 100mb/s when I copying the file with pv, the result from iperf is 1.96 Gbits/sec.Do you know how to achieve the same result without implement the cloud storage? or increase the NFS speed?
Data files (csv, pdf) and the code are storing in a single SSD Persistent Disk
There's nothing stopping you from volume mounting an SSD into the Pod so you can continue to use an SSD. I can only speak to AWS terminology, but some EC2 instances come with "local" SSD hardware, and thus you would only need to use a nodeSelector to ensure your Pods were scheduled onto machines that had said local storage available.
Where you're going to run into problems is if you are currently just using one php+apache and thus just one SSD, but now you want to scale the application up and it requires that all php+apache have access to the same SSD. That's a classic distributed application architecture problem, and something kubernetes itself can't fix for you.
If you're willing to expend the effort, you can also try any one of the other distributed filesystems (Ceph, GlusterFS, etc) and see if they perform better for your situation. Then again, "We have very limited time" I guess pretty much means that's off the table.
I set up standalone ironic with mitaka version. I created a whole disk image (ubuntu14.04) by virt-install and used Coreos pxe image (Here) as deploy kernel and ramdisk. (ubuntu 14.04) The disk image size is 10G, and it could be deployed successfully on my node. When I logged in the node and checked the disk info, it only used 10G disk size for /dev/sda. The physical disk size of node is 500G.
How do I make my image to use whole disk size of node after deploying? Thanks.
You have a swap partition after the root partition and that will block the online growing of the root FS.
You need to either move the swap to the front of the disk or remove it entirely.
Is it worth using SSD as boot disk? I'm not planning to access local disks within pods.
Also, GCP by default creates 100GB disk. If I use 20GB disk, will it cripple the cluster or it's OK to use smaller sized disks?
Why one or the other?. Kubernetes (Google Conainer Engine) is mainly Memory and CPU intensive unless your applications need a huge throughput on the hard drives. If you want to save money you can create tags on the nodes with HDD and use the node-affinity to tweak which pods goes where so you can have few nodes with SSD and target them with the affinity tags.
I would always recommend SSD considering the small difference in price and large difference in performance. Even if it just speeds up the deployment/upgrade of containers.
Reducing the disk size to what is required for running your PODs should save you more. I cannot give a general recommendation for disk size since it depends on the OS you are using and how many PODs you will end up on each node as well as how big each POD is going to be. To give an example: When I run coreOS based images with staging deployments for nginx, php and some application servers I can reduce the disk size to 10gb with ample free room (both for master and worker nodes). On the extreme side - If I run self-contained golang application containers without storage need, each POD will only require a few MB space.
I have cluster running in Azure.
I have multiple gigabytes of log data under D:\SvcFab\Log\Traces. Is there way to control amount of trace data that is collected/stored? Will the logs grow indefinitely?
Also the D:\SvcFab\ReplicatorLog has 8GB of preallocated data as specified by SharedLogSizeInMB parameter (https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-configuration). How can I change this setting in Azure cluster or should it always be kept default?
For Azure clusters the SvcFab\Log folder will grow up to 5GB. It will also shrink if detects your disk is running out of space (<1GB). There are no controls for this in Azure.
This may be old but if you still have this issue, the solution for this is to add parameter in arm template for service fabric cluster .. there are some other ways to do this but this one is the most guaranteed one
https://techcommunity.microsoft.com/t5/azure-paas-developer-blog/reduce-log-size-on-service-fabric-node/ba-p/1017493