We need to made volume to be managed easily. We ned to use PV volume, but we want to be able to start volume on any node, and data not stored on node (if node
crash no problems in this way) so we think about flocker with Ceph backend. What's the best solution for production ?
Flocker is not required. The functionality you are seeking is what Kubernetes Volume Plugins provide.
The way to think about a Kubernetes Persistent Volume (PV) is that it is a configuration object that stores information about a specific network storage asset. When a user submits a claim, assuming it finds a match, it will bind to one of the Persistent Volumes in the pool of available Persistent Volumes. This means your claim is bound to an object that contains information about a specific network storage asset.
When a claim is specified in a Pod or RC, the runtime is able to ascertain the PV bound to the claim and then ascertain which Kubernetes Volume Plugin to use and what parameters to pass itm based on the properties of the PV.
As such, wherever your Pods run in the cluster, they will be able to perform a network mount of the storage asset described in the PV. None of this data will be local. The pod can die and be restarted on any node in the cluster and it will reconnect to the same network storage asset specified in the PV.
Any Kubernetes Volume Plugin, with the exception of EmptyDir and HostPath, can be specified in a Persistent Volume Definition. So you could create a PV that uses the Ceph RBD volume plugin and you would have the functionality that you seek.
Typically Ceph will be run in its own cluster. There are some examples of ceph running in a container (with more work to be done), and in that case, Ceph and your app could share kubernetes nodes.
Related
To use storage inside Kubernetes PODs I can use volumes and persistent volumes. While the volumes like emptyDir are ephemeral, I could use hostPath and many other cloud based volume plugins which would provide a persistent solution in volumes itself.
In that case why should I be using Persistent Volume then?
It is very important to understand the main differences between Volumes and PersistentVolumes. Both Volumes and PersistentVolumes are Kubernetes resources which provides an abstraction of a data storage facility.
Volumes: let your pod write to a filesystem that exists as long as the pod exists. They also let you share data between containers in the same pod but data in that volume will be destroyed when the pod is restarted. Volume decouples the storage from the Container. Its lifecycle is coupled to a pod.
PersistentVolumes: serves as a long-term storage in your Kubernetes cluster. They exist beyond containers, pods, and nodes. A pod uses a persistent volume claim to to get read and write access to the persistent volume. PersistentVolume decouples the storage from the Pod. Its lifecycle is independent. It enables safe pod restarts and sharing data between pods.
When it comes to hostPath:
A hostPath volume mounts a file or directory from the host node's
filesystem into your Pod.
hostPath has its usage scenarios but in general it might not recommended due to several reasons:
Pods with identical configuration (such as created from a PodTemplate) may behave differently on different nodes due to different files on the nodes
The files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged Container or modify the file permissions on the host to be able to write to a hostPath volume
You don't always directly control which node your pods will run on, so you're not guaranteed that the pod will actually be scheduled on the node that has the data volume.
If a node goes down you need the pod to be scheduled on other node where your locally provisioned volume will not be available.
The hostPath would be good if for example you would like to use it for log collector running in a DaemonSet.
I recommend the Kubernetes Volumes Guide as a nice supplement to this topic.
PersistentVoluemes is cluster-wide storage and allows you to manage the storage more centrally.
When you configure a volume (either using hostPath or any of the cloud-based volume plugins) then you need to do this configuration within the POD definition file. Every configuration information, required to configure storage for the volume, goes within the POD definition file.
When you have a large environment with a lot of users and a large number of PODs then users will have to configure storage every time for each POD they deploy. Whatever storage solution is used, the user who deploys the POD will have to configure that storage on all of his/her POD definition files. If a change needs to be made then the user will have to make this change on all of his/her PODs. After a certain scale, this is not the most optimal way to manage storage.
Instead, you would like to manage this centrally. You would like to manage the storage in such a way that an Administrator can create a large pool of storage and users can carve out a part of this storage as required, and this is exactly what you can do using PersistentVolumes and PersistentVolumeClaims.
Use PersistentVolumes when you need to set up a database like MongoDB, Redis, Postgres & MySQL. Because it's long-term storage and not deeply coupled with your pods! Perfect for database applications. Because they will not die with the pods.
Avoid Volumes when you need long-term storage. Because they will die with the pods!
In my case, when I have to store something, I will always go for persistent volumes!
According to the documentation:
A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned ... It is a resource in the cluster just like a node is a cluster resource...
So I was reading about all currently available plugins for PVs and I understand that for 3rd-party / out-of-cluster storage this doesn't matter (e.g. storing data in EBS, Azure or GCE disks) because there are no or very little implications when adding or removing nodes from a cluster. However, there are different ones such as (ignoring hostPath as that works only for single-node clusters):
csi
local
which (at least from what I've read in the docs) don't require 3rd-party vendors/software.
But also:
... local volumes are subject to the availability of the underlying node and are not suitable for all applications. If a node becomes unhealthy, then the local volume becomes inaccessible by the pod. The pod using this volume is unable to run. Applications using local volumes must be able to tolerate this reduced availability, as well as potential data loss, depending on the durability characteristics of the underlying disk.
The local PersistentVolume requires manual cleanup and deletion by the user if the external static provisioner is not used to manage the volume lifecycle.
Use-case
Let's say I have a single-node cluster with a single local PV and I want to add a new node to the cluster, so I have 2-node cluster (small numbers for simplicity).
Will the data from an already existing local PV be 1:1 replicated into the new node as in having one PV with 2 nodes of redundancy or is it strictly bound to the existing node only?
If the already existing PV can't be adjusted from 1 to 2 nodes, can a new PV (created from scratch) be created so it's 1:1 replicated between 2+ nodes on the cluster?
Alternatively if not, what would be the correct approach without using a 3rd-party out-of-cluster solution? Will using csi cause any change to the overall approach or is it the same with redundancy, just different "engine" under the hood?
Can a new PV be created so it's 1:1 replicated between 2+ nodes on the cluster?
None of the standard volume types are replicated at all. If you can use a volume type that supports ReadWriteMany access (most readily NFS) then multiple pods can use it simultaneously, but you would have to run the matching NFS server.
Of the volume types you reference:
hostPath is a directory on the node the pod happens to be running on. It's not a directory on any specific node, so if the pod gets recreated on a different node, it will refer to the same directory but on the new node, presumably with different content. Aside from basic test scenarios I'm not sure when a hostPath PersistentVolume would be useful.
local is a directory on a specific node, or at least following a node-affinity constraint. Kubernetes knows that not all storage can be mounted on every node, so this automatically constrains the pod to run on the node that has the directory (assuming the node still exists).
csi is an extremely generic extension mechanism, so that you can run storage drivers that aren't on the list you link to. There are some features that might be better supported by the CSI version of a storage backend than the in-tree version. (I'm familiar with AWS: the EBS CSI driver supports snapshots and resizing; the EFS CSI driver can dynamically provision NFS directories.)
In the specific case of a local test cluster (say, using kind) using a local volume will constrain pods to run on the node that has the data, which is more robust than using a hostPath volume. It won't replicate the data, though, so if the node with the data is deleted, the data goes away with it.
I'm teaching myself Kubernetes with a 5 Rpi cluster, and I'm a bit confused by the way Kubernetes treats Persistent Volumes with respect to Pod Scheduling.
I have 4 worker nodes using ext4 formatted 64GB micro SD cards. It's not going to give GCP or AWS a run for their money, but it's a side project.
Let's say I create a Persistent volume Claim requesting 10GB of storage on worker1, and I deploy a service which relies on this PVC, is that service then forced to be scheduled on worker1?
Should I be looking into distributed file systems like Ceph or Hdfs so that Pods aren't restricted to being scheduled on a particular node?
Sorry if this seems like a stupid question, I'm self taught and still trying to figure this stuff out! (Feel free to improve my tl;dr doc for kubernetes with a pull req)
just some examples, as already mentioned it depends on your storage system, as i see you use the local storage option
Local Storage:
yes the pod needs to be run on the same machine where the pv is located (your case)
ISCSI/Trident San:
no, the node will mount the iscsi block device where the pod will be scheduled
(as mentioned already volume binding mode is an important keyword, its possible you need to set this to 'WaitForFirstConsumer')
NFS/Trident Nas:
no, its nfs, mountable from everywhere if you can access and auth against it
VMWare VMDK's:
no, same as iscsi, the node which gets the pod scheduled mounts the vmdk from the datastore
ceph/rook.io:
no, you get 3 options for storage, file, block an object storage, every type is distributed so you can schedule a pod on every node.
also ceph is the ideal system for carrying a distributed software defined storage on commodity hardware, what i can recommend is https://rook.io/ basically an opensource ceph on 'container-steroids'
Let's say I create a Persistent volume Claim requesting 10GB of storage on worker1, and I deploy a service which relies on this PVC, is that service then forced to be scheduled on worker1?
This is a good question. How this works depends on your storage system. The StorageClass defined for your Persistent Volume Claim contains information about Volume Binding Mode. It is common to use dynamic provisioning volumes, so that the volume is first allocated when a user/consumer/Pod is scheduled. And typically this volume does not exist on the local Node but remote in the same data center. Kubernetes also has support for Local Persistent Volumes that are physical volumes located on the same Node, but they are typically more expensive and used when you need high disk performance and volume.
I am new at Kubernetes and am trying to understand the most efficient and secure way to handle sensitive persisted data that interacts with a k8 pod. I have the following requirements when I start a pod in a k8s cluster:
The pod should have persisted storage.
Data inside the pod should be persistent even if the pod crashes or restarts.
I should be able to easily add or remove data from hostPath into the pod. (Not sure if this is feasible since I do not know how the data will behave if the pod starts on a new node in a multi node environment. Do all nodes have access to the data on the same hostPath?)
Currently, I have been using StatefulSets with a persistent volume claim on GKE. The image that I am using has a couple of constraints as follows:
I have to mount a configuration file into the pod before it starts. (I am currently using configmaps to pass the configuration file)
The pod that comes up, creates its own TLS certificates which I need to pass to other pods. (Currently I do not have a process in place to do this and thus have been manually copy pasting these certificates into other pods)
So, how do I maintain a common persisted storage that handles sensitive data between multiple pods and how do I add pre-configured data to this storage? Any guidance or suggestions are appreciated.
I believe this documentation on creating a persistent disk with multiple readers [1] is what you are looking for. you will however only be able to have the pods read from the disk since GCP does not support "WRITEMANY".
Regarding hostpaths, the mount point is on the pod the volume is a directory on the node. I believe the hostpath is confined to individual nodes.
[1] https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/readonlymany-disks
[2] https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
What is the difference between persistent volume (PV) and persistent volume claim (PVC) in Kubernetes/ Openshift by referring to documentation?
What is the difference between both in simple terms?
From the docs
PVs are resources in the cluster. PVCs are requests for those resources and also act as claim checks to the resource.
So a persistent volume (PV) is the "physical" volume on the host machine that stores your persistent data. A persistent volume claim (PVC) is a request for the platform to create a PV for you, and you attach PVs to your pods via a PVC.
Something akin to
Pod -> PVC -> PV -> Host machine
PVC is a declaration of need for storage that can at some point become available / satisfied - as in bound to some actual PV.
It is a bit like the asynchronous programming concept of a promise. PVC promises that it will at some point "translate" into storage volume that your application will be able to use, and one of defined characteristics like class, size, and access mode (ROX, RWO, and RWX).
This is a way to abstract thinking about a particular storage implementation away from your pods/deployments. Your application in most cases does not need to declare "give me NFS storage from server X of size Y"; it is more like "I need persistent storage of default class and size Y".
With this, deployments on different clusters can choose to differently satisfy that need. One can link an EBS device, another can provision a GlusterFS, and your core manifests are still the same in both cases.
Furthermore, you can have Volume Claim Templates defined in your deployment, so that each pod gets a reflecting PVC created automatically (i.e., supporting infrastructure-agnostic storage definition for a group of scalable pods where each needs its own dedicated storage).
Short:
- Here you have the storage! PersistentVolume (PV)
- You get the storage if you really need it! PersistentVolumeClaim (PVC)
A PersistentVolume (PV) is a piece of storage in the cluster or central storage let's say 100GB.
A PersistentVolumeClaim (PVC) is a request for storage by a user for the application to use 10GB.
In real life scenario, PV is whole cake and PVC is piece of cake (But you can have a whole cake if there are no other people to eat (just like if there are no other application to use you can use whole PV )).
Short and Simple
Persistent Volume - Available storage let's say you have 100Gi
Persistent Volume Claim - You request from Persistent Volume, let's say you request 10Gi you'll get it but if you request 110Gi you won't get it.
A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by server/storage/cluster administrator or dynamically provisioned using Storage Classes. It is a resource in the cluster just like node.
A PersistentVolumeClaim (PVC) is a request for storage by a user which can be attained from PV. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., they can be mounted ReadWriteOnce, ReadOnlyMany or ReadWriteMany.
A Persistent Volume Claim is telling you what options you have access to in a particular cluster and they got this circular at this store called Smart Tech with some ads about your configuration options, those ads are the Persistent Volume Claim.
Inside your config file you write out the different Persistent Volume Claims that you are going to have inside your cluster, kind of like your wish list to Santa, but of course you are going to go take that to the sales guy at Smart Tech when you are done.
So you write a config file that says there should a 600gb hard drive option available to all your clusters and a 1TB hard drive option as well.
When you choose one of these options of the Persistent Volume Claim you go and request that Kubernetes (the sales guy) goes and gets that option for you, the option you have chosen, Kubernetes has to look through these instances of storage options in the stock room that are readily available. These instances of hard drives can be used right away and they are considered statically provisioned because they are created ahead of time.
On the other hand, there is dynamically provisioned options that were created on the fly, when you asked Kubernetes the sales guy, so kind of like just-in-time production, it got created when you immediately asked for it.
So the Persistent Volume Claim is the stores advertisement of options and whichever one you choose Kubernetes will go get it, either one in storage or create one on the fly.
The Persistent Volume is the actual product or options that you get back from Kubernetes that you asked for. If Kubernetes does not have what you asked for it will try to create it on the fly for you.
So the PVC is what Smart Tech is advertising they have to offer to your cluster which Kubernetes the sales guy will get for you and the PV is the actual finished product delivered to you.
PersistentVolume(PV) and PersistentVolumeClaim(PVC) are the resources APIs provided by the Kubernetes.
PV is a piece of storage which supposed to preallocated by an admin. And PVC is a request for a piece of storage by a user.
Persistent Volume — low level representation of a storage volume.
Persistent Volume Claim — binding between a Pod and Persistent Volume.
Storage Class — allows for dynamic provisioning of Persistent Volumes.
You can find some common when comparing PV and PVC with node and pods.
PV like a node, which defines the storage.
PVC like pods that requires the resources (Mem, CPU) and get them in case the node has the resources to allocate, which in this case it's a storage.