As the title says (and also here https://docs.kasten.io/latest/restrictions.html).
My company is using K10 latest (v5.0.2) as a backup tool for our Openshift cluster.
We are required to use a S3 compatible storage provisioner.
We moved from MinIO to Ceph because of some issues with MinIO (excessive memory usage, MinIO Pods handling, ...) yet we found out that Ceph RGW is not supported from k10 and it seems this makes our backup fail: from the Kasten console it appears that only the ObjectBucketClaim manifest is backed up but not the data contained within the bucket.
Also, when restoring, the ObjectBucketClaims remain in "pending" status.
I am stuck and I don't know what to suggest to my storage department: I told them to giveup using MinIO and start using Ceph but its RGW is not supported by K10.
Any suggestions on how I can handle this situation?
Thanks in advance.
One of our containers is using ephemeral storage but we don't know why. The app running in the container shouldn't be writing anything to the disk.
We set the storage limit to 20MB but it's still being evicted. We could increase the limit but this seems like a bandaid fix.
We're not sure what or where this container is writing to, and I'm not sure how to check that. When a container is evicted, the only information I can see is that the container exceeded its storage limit.
Is there an efficient way to know what's being written, or is our only option to comb through the code?
Adding details to the topic.
Pods use ephemeral local storage for scratch space, caching, and logs.
Pods can be evicted due to other pods filling the local storage, after which new pods are not admitted until sufficient storage has been reclaimed.
The kubelet can provide scratch space to Pods using local ephemeral storage to mount emptyDir volumes into containers.
For container-level isolation, if a container's writable layer and log usage exceeds its storage limit, the kubelet marks the Pod for eviction.
For pod-level isolation the kubelet works out an overall Pod storage limit by summing the limits for the containers in that Pod. In this case, if the sum of the local ephemeral storage usage from all containers and also the Pod's emptyDir volumes exceeds the overall Pod storage limit, then the kubelet also marks the Pod for eviction.
To see what files have been written since the pod started, you can run:
find / -mount -newer /proc -print
This will output a list of files modified more recently than '/proc'.
/etc/nginx/conf.d
/etc/nginx/conf.d/default.conf
/run/secrets
/run/secrets/kubernetes.io
/run/secrets/kubernetes.io/serviceaccount
/run/nginx.pid
/var/cache/nginx
/var/cache/nginx/fastcgi_temp
/var/cache/nginx/client_temp
/var/cache/nginx/uwsgi_temp
/var/cache/nginx/proxy_temp
/var/cache/nginx/scgi_temp
/dev
Also, try without the '-mount' option.
To see if any new files are being modified, you can run some variations of the following command in a Pod:
while true; do rm -f a; touch a; sleep 30; echo "monitoring..."; find / -mount -newer a -print; done
and check the file size using the du -h someDir command.
Also, as #gohm'c pointed out in his answer, you can use sidecar/ephemeral debug containers.
Read more about Local ephemeral storage here.
We're not sure what or where this container is writing to, and I'm not sure how to check that.
Try look into the container volumeMounts section that is mounted with emptyDir, then add a sidecar container (eg. busybox) to start a shell session where you can check the path. If your cluster support ephemeral debug container you don't need the sidecar container.
I have a small kubernetes cluster with AWX running.
I would like to make some changes, the PV is a filesystem on one of the nodes.
Is it possible to migrate it to a different PV, like NFS?
Also, I would like to change the CPU and memory limits. But I guess I will have to redeploy it.
Should I try to migrate the PV or delete everything and recreate it?
Thanks
Assuming that you have dynamic provisioning enabled I advice you to use pv-migrate.
This is a cli tool/kubectl plugin to easily migrate the contents of one Kubernetes PersistentVolume to another.
Common use cases:
You have a database with a bound 30 GB PersistentVolumeClaim. It occurred 30 GB was not enough and you filled all the disk space rather quickly. And sadly your StorageClass/provisioner doesn't support volume expansion. Now you need to create a new PVC of 100 GB and somehow copy all the data to the new volume, as-is, with its permissions and so on.
You need to move a PersistentVolumeClaim from one namespace to another.
To migrate contents of PersistentVolumeClaim pvc-a in namespace name-space-a to the PersistentVolumeClaim pvc-b in namespace name-space-b, use the following command:
$ kubectl pv-migrate \
--source-namespace name-space-a \
--source pvc-a \
--dest-namespace name-space-b \
--dest pvc-b
Take also a look at: change-pv-reclaim-policy, resizing-persistent-volumes-using-kubernetes.
When using jiva datadirhost on a ZFS mountpoint(with xattr enabled) I got this
time="2018-12-02T20:39:48Z" level=fatal msg="Error running start replica command: failed to find extents, error: invalid argument".
If we create an ext4 zvol based storagepool it works. Is this expected behaviour? I am using Kubernetes 1.9.7 on ubuntu 16.04 with ZFS.
OpenEBS has two storage engines currently supported,
cStor (Recommended)
Jiva
Jiva volumes are created from local filesystem or mounted file-system, and can't consume the block device directly. This makes Jiva will work only with those file systems which provides extent mapping. ZFS does not support filefrag as of now that's why it gives above error.
On Other hand, cStor storage engine, volumes are created on a pool created on block devices. Storage pool create by cStor is native zpool(ZFS). You can get more details Concepts-> CASEngines from openEBS doc site.
I'm running a MySQL deployment on Kubernetes however seems like my allocated space was not enough, initially I added a persistent volume of 50GB and now I'd like to expand that to 100GB.
I already saw the a persistent volume claim is immutable after creation, but can I somehow just resize the persistent volume and then recreate my claim?
Yes, as of 1.11, persistent volumes can be resized on certain cloud providers. To increase volume size:
Edit the PVC (kubectl edit pvc $your_pvc) to specify the new size. The key to edit is spec.resources.requests.storage:
Terminate the pod using the volume.
Once the pod using the volume is terminated, the filesystem is expanded and the size of the PV is increased. See the above link for details.
It is possible in Kubernetes 1.9 (alpha in 1.8) for some volume types: gcePersistentDisk, awsElasticBlockStore, Cinder, glusterfs, rbd
It requires enabling the PersistentVolumeClaimResize admission plug-in and storage classes whose allowVolumeExpansion field is set to true.
See official docs at https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims
Update: volume expansion is available as a beta feature starting Kubernetes v1.11 for in-tree volume plugins. It is also available as a beta feature for volumes backed by CSI drivers as of Kubernetes v1.16.
If the volume plugin or CSI driver for your volume support volume expansion, you can resize a volume via the Kubernetes API:
Ensure volume expansion is enabled for the StorageClass (allowVolumeExpansion: true is set on the StorageClass) associated with your PVC.
Request a change in volume capacity by editing your PVC (spec.resources.requests).
For more information, see:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#expanding-persistent-volumes-claims
https://kubernetes-csi.github.io/docs/volume-expansion.html
No, Kubernetes does not support automatic volume resizing yet.
Disk resizing is an entirely manual process at the moment.
Assuming that you created a Kubernetes PV object with a given capacity and the PV is bound to a PVC, and then attached/mounted to a node for use by a pod. If you increase the volume size, pods would continue to be able to use the disk without issue, however they would not have access to the additional space.
To enable the additional space on the volume, you must manually resize the partitions. You can do that by following the instructions here. You'd have to delete the pods referencing the volume first, wait for it to detach, than manually attach/mount the volume to some VM instance you have access to, and run through the required steps to resize it.
Opened issue #35941 to track the feature request.
There is some support for this in 1.8 and above, for some volume types, including gcePersistentDisk and awsBlockStore, if certain experimental features are enabled on the cluster.
For other volume types, it must be done manually for now. In addition, support for doing this automatically while pods are online (nice!) is coming in a future version (currently slated for 1.11):
For now, these are the steps I followed to do this manually with an AzureDisk volume type (for managed disks) which currently does not support persistent disk resize (but support is coming for this too):
Ensure PVs have reclaim policy "Retain" set.
Delete the stateful set and related pods. Kubernetes should release the PVs, even though the PV and PVC statuses will remain Bound. Take special care for stateful sets that are managed by an operator, such as Prometheus -- the operator may need to be disabled temporarily. It may also be possible to use Scale to do one pod at a time. This may take a few minutes, be patient.
Resize the underlying storage for the PV(s) using the Azure API or portal.
Mount the underlying storage on a VM (such as the Kubernetes master) by adding them as a "Disk" in the VM settings. In the VM, use e2fsck and resize2fs to resize the filesystem on the PV (assuming an ext3/4 FS). Unmount the disks.
Save the JSON/YAML configuration of the associated PVC.
Delete the associated PVC. The PV should change to status Released.
Edit the YAML config of the PV, after which the PV status should be Available:
specify the new volume size in spec.capacity.storage,
remove the spec.claimref uid and resourceVersion fields, and
remove status.phase.
Edit the saved PVC configuration:
remove the metadata.resourceVersion field,
remove the metadata pv.kubernetes.io/bind-completed and pv.kubernetes.io/bound-by-controller annotations, and
change the spec.resources.requests.storage field to the updated PV size, and
remove all fields inside status.
Create a new resource using the edited PVC configuration. The PVC should start in Pending state, but both the PV and PVC should transition relatively quickly to Bound.
Recreate the StatefulSet and/or change the stateful set configuration to restart pods.
In terms of PVC/PV 'resizing', that's still not supported in k8s, though I believe it could potentially arrive in 1.9
It's possible to achieve the same end result by dealing with PVC/PV and (e.g.) GCE PD though..
For example, I had a gitlab deployment, with a PVC and a dynamically provisioned PV via a StorageClass resource. Here are the steps I ran through:
Take a snapshot of the PD (provided you care about the data)
Ensure the ReclaimPolicy of the PV is "Retain", patch if necessary as detailed here: https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/
kubectl describe pv <name-of-pv> (useful when creating the PV manifest later)
Delete the deployment/pod (probably not essential, but seems cleaner)
Delete PVC and PV
Ensure PD is recognised as being not in use by anything (e.g. google console, compute/disks page)
Resize PD with cloud provider (with GCE, for example, this can actually be done at an earlier stage, even if the disk is in use)
Create k8s PersistentVolume manifest (this had previously been done dynamically via the use of the StorageClass resource). In the PersistentVolume yaml spec, I had "gcePersistentDisk: pdName: <name-of-pd>" defined, along with other details that I'd grabbed at step 3. make sure you update the spec.capacity.storage to the new capacity you want the PV to have (although not essential, and has no effect here, you may want to update the storage capacity/value in your PVC manifest, for posterity)
kubectl apply (or equivalent) to recreate your deployment/pod, PVC and PV
note: some steps may not be essential, such as deleting some of the existing deployment/pod.. resources, though I personally prefer to remove them, seeing as I know the ReclaimPolicy is Retain, and I have a snapshot.
The first thing you can do is, check for the storage class that you are using, see if allowVolumeExpansion is set to `true. If yes then simply update PVC with requested volume and check for status in PVCs.
If this doesn't work for you then try this (for AWS users).
Check for the attached volume id in the PV (under awsElasticBlockStore -> `volume).
Go to Volumes in AWS, and modify volume to whatever is required
SSH into the node to which is volume is currently attached (to find node name describe pod and check for node key)
use lsblk to list the volume attached
Run resize2fs or xfs_growfs based on what type of volume you have.
exec into the pod run df -h and check the volume.
Note: You can only modify a volume once in 6 hours.
Edit the PVC (kubectl edit pvc $your_pvc) to specify the new size. The key to edit is spec.resources.requests.storage:
Even though this answer worked quite well for one pvc of my statefulset, the others didn't managed to resize. I guess it's because the pods restarted too quick, leaving no time for the resizing process to start due to the backoff. In fact, the pods started fast but took some time to be considered as ready (increasing backoff).
Here's my workaround:
Update the pvc
Backup the sts spec
k get sts <sts-name> -o yaml > sts.yaml
Then delete the sts with cascade=orphan. Thus, the pods will still run
kubectl delete sts --cascade=orphan <sts-name>
Then delete one of the pod whose pvc wouldn't resize
kubectl delete pod <pod-name>
Wait for the pvc to resize
kubectl get pvc -w
Reapply the sts so the pod comes back
kubectl apply -f sts.yaml
Wait for the pod to come back
Repeat until all pvc are resized!
Below is how we can expand the volume size of azure disks mounted on statefulset(STS) pod when storage class is used.(AWS EBS and GCP Persistent volumes should be similar).
Summary:
Delete the statefulset.
Update the volume size on the PVC. Wait till the condition message prompts to start up the pods.
Apply new statefulset with updated volume size and you should see the volume getting resized when the pod starts up.
Complete Steps:
Check if volume resize is enabled in the storage class.
kubectl get storageclass
First, delete the statefulset. This is required because
The volumes should be unmounted and detached from the node before it can be resized.
The volume size on the STS YAML is immutable (cant be updated).
We will have to create a new STS with higher volume size later on. Don't forget to backup the STS YAML if you don't have it in your repo's.
After deleting the STS, wait for some time so that k8s can detach the volume from the node.
Next, modify the PVC with higher value for volume size.
At this point, if the volume is still attached, you will see below warning message in the PVC events.
Either the volume is still mounted to the pod or you just have to wait and give some time to k8s.
Next, run the describe command on the PVC, you should now see a message(in conditions) prompting you to start up the pod.
kubectl describe pvc app-master-volume-app-master-0
In the earlier step, we had deleted the statefulset. Now we need to create and apply a new STS with higher volume size. This should match the value earlier modified in the PVC spec.
When the new pod gets created, you will see pod event like shown below which indicates that the volume resize is successful.
Yes, it can be, after version 1.8, have a look at volume expansion here
Volume expansion was introduced in v1.8 as an Alpha feature
I have persistent volume with self created StorageClass (allowVolumeExpansion: true).
PV spec: accessMode: readWriteOnce
PVC spec: same
When I upgrade PV, changes are not reflected in PVC.