Having difficulty to find a proper tool or combination of tools in order to safely create selective backups of resources in EKS.
Valero seems to be a good option. It is not clear how the PVC snapshots are performed and if they can be performed directly by Valero or it requires a special drive to be attached as available storageclass. In this case if there is already production in place it requires surgical integrations and potential data loss to just replace storage class of all already deployed services with PVC usage.
Volume Snapshot Custom Resource Definitions (CRDs) and Volume Snapshot controller seem to work manually without being able to use the solution out of the box. This requires probably some POD cronjob that has access to all storageclasses available through serviceaccount injection. This seems that is a best fit if there are engineers that will perform snaps before upgrades or migrations.
Does anybody else have experience with other opensource tools that provide all above functionality and offer a simple UI to preview all backups and select storage platform for snaps?
Thanks in advance
Related
I have a job, which I want to run regularly in Kubernetes 1.19.3 (DigitalOcean).
For this job, I need to take a snapshot of a PVC and do stuff to it. I know how can I run a job and mount a volume to the pod it runs, but I have a hard time finding out how to take that snapshot at the beginning of this job.
Is there any way to do it?
The tool of choice to take PV snapshots in K8s is VolumeSnapshots.
The trouble with them is that they don't come yet) with functionality for periodic triggering. So, you would have to create them from a K8s CronJob. However, doing so is not terribly straight forward, since your CronJob Pod would need to have a K8s client installed and require access to the K8s API Server with RBAC.
There are a couple of options to get there, reaching from writing your own image from scratch to using open-source solutions based on the clients from this project k8s client libraries.
Seeing that dynamic K8s manifest applying is somewhat badly supported by K8s, I actually started an open source project myself, that you could use for this purpose: K8sCrud.
We have started setting up a Kubernetes cluster. On Production, we have 4 Mysql Nodes(2 Active Master, 2 Active slaves). Complete servers are on-premise, There is NO cloud providers usage.
Now how do I configure storage? I mean should I use PV / PVC? How will it work. Should I use local PV? Can someone explain to me this?
You need to use PersistentVolumes and PersistentVolumeClaims in order to achieve that.
A PersistentVolume (PV) is a piece of storage in the cluster that has
been provisioned by an administrator or dynamically provisioned using
Storage Classes.
A PersistentVolumeClaim (PVC) is a request for storage by a user.
Claims can request specific size and access modes (e.g., they can be
mounted once read/write or many times read-only).
Containers are ephemeral. When the container is restarted all the changes made prior to it are lost. Databases, however expect the data is persistent, therefore you need persistent volumes. You have to create a storage claim and the pod must be configured to mount the claimed storage.
Here you will find a simple guide showing how to deploy MySQL with a PersistentVolume. However, I strongly recommend getting familiar with the official docs that I have linked in order to fully understand the concept and adjust the access mode, class, size, etc according to your needs.
Please let me know if that helped.
From the Kubernetes' validation source code, at least those resources are immutable after creation:
Persistent Volumes
Storage Classes
Why is that ?
This is a core concept on Kubernetes. A few specs are immutable because their change has impact in the basic structure of the resource it's connected.
For example, changing the Persistent Volumes may impact pods that are using this PV. Let's suppose you have a mysql pod running on a PV and you change it in a way that all the data is gone.
On Kubernetes 1.18 Secrets and ConfigMaps also became immutable as an Alpha feature, meaning that this will be the new default soon. Check the GitHub Issue here.
What is it good for?
The most popular and the most convenient way of consuming Secrets and
ConfigMaps by Pods is consuming it as a file. However, any update to a
Secret or ConfigMap object is quickly (roughly within a minute)
reflected in updates of the file mounted for all Pods consuming them.
That means that a bad update (push) of Secret and/or ConfigMap can
very quickly break the entire application.
Here you can read more about the motivation behind this decision.
In this KEP, we are proposing to introduce an ability to specify that
contents of a particular Secret/ConfigMap should be immutable for its
whole lifetime. For those Secrets/ConfigMap, Kubelets will not be
trying to watch/poll for changes to updated mounts for their Pods.
Given there are a lot of users not really taking advantage of
automatic updates of Secrets/ConfigMaps due to consequences described
above, this will allow them to:
protect themselves better for accidental bad updates that could cause outages of their applications
achieve better performance of their cluster thanks to significant reduction of load on apiserver
I developed a web application for our students and i would like to run this now in a kubernetes container environment. Every user (could be seen as tenant) gets its own application environment (1:1 relation).
the application environment consists of 2 pods (1x webserver, 1x database), defined by a deployment and a service.
I am using kubernetes v1.17.2 and i would like to use the feature of dynamic PersistentVolumeClaims together with the possibility to keep data of a specific user (tenant) between the deletion and re-creation of a new pod (e.g. case of updating to a new application version or after a hardware reboot).
I thought about using a environment variable at pod-creation (e.g. user-1, user-2, user-x,...) and using this information to allow a reusing of a dynamic created PersistentVolume.
is there any best-practise or concept how this can be achieved?
best regards
shane
The outcome that you wish to achieve will be strongly connected to the solution that you are currently using.
It will differ between Kubernetes instances that are provisioned in cloud (for example GKE) and Kubernetes instances on premises (for example: kubeadm, kubespray).
Talking about the possibility to retain user data please refer to official documentation: Kubernetes.io: Persistent volumes reclaiming. It shows a way to retain data inside a pvc.
Be aware of that local static provisioner does not support dynamic provisioning.
The local volume static provisioner manages the PersistentVolume lifecycle for pre-allocated disks by detecting and creating PVs for each local disk on the host, and cleaning up the disks when released. It does not support dynamic provisioning.
Github.com: Storage local static provisioner
Contrary to that VMware Vsphere supports dynamic provisioning. If you are using this solution please refer to this documentation
In your question there is a lack of specific explanation of users in your environment. Are they inside your application or are they outside? Is the application authenticating users? One of solution will be to create users inside of Kubernetes by service accounts and limit their view to namespace specifically created for them.
For service account creation please refer to: Kubernetes.io: Configure service account.
Additionally you could also look on Statefulsets.
I have a service that generates a picture. Once it's ready, the user will be able to download it.
What is the recommended way to share a storage volume between a worker pod and a backend service?
In general the recommended way is "don't". While a few volume providers support multi-mounting, it's very hard to do that in a way that isn't sadmaking. Preferably use an external services like AWS S3 for hosting the actual file content and store references in your existing database(s). If you need a local equivalent, check out Minio for simple cases.
Personally i will not recommended it to do. better then that you two container side one pod if having dependency on each other. so if one pod goes fail that file manager also delete and create at particular time if needed