We are trying to create a solution where we want to replicate changes made to a folder inside any of the pods of the Statefulset. Any file changes inside that folder on any POD should also reflect in other pods. Is there a sidecar solution for this requirement? Because we know that Statefulset will create separate PVs for each POD and there won't be any common mount across the pods of the Statefulset.
You can try using the NFS or file system like EFS using that you will be able to implement the ReadWritemany.
For ref Azure File.
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: statefulset-azurefile
labels:
k8s-app: nginx
version: v1
spec:
serviceName: statefulset-azurefile
replicas: 1
template:
metadata:
labels:
k8s-app: nginx
version: v1
spec:
containers:
- name: statefulset-azurefile
image: nginx
volumeMounts:
- name: persistent-storage
mountPath: /mnt/azurefile
volumeClaimTemplates:
- metadata:
name: persistent-storage
annotations:
volume.beta.kubernetes.io/storage-class: azurefile
spec:
accessModes: [ "ReadWriteMany" ]
resources:
requests:
storage: 5Gi
Demo : https://github.com/andyzhangx/demo/tree/master/linux/statefulset
If volumeClaimTemplates not work as expected use the persistentVolumeClaim
Article to read about ReadWriteMany access Mode :https://learn.microsoft.com/en-us/azure/aks/azure-files-volume
If you are on other cloud providers like GCP, AWS, Oracle(OCI) provides different file services.
GCP- Filestore
AWS- EFS
OCI- Filestorage
OCI article if you want to explore : https://enabling-cloud.github.io/oci-learning/manual/StaticPersistentVolumeOnOCI.html
Related
I am able to deploy the database service itself, but when I try to deploy with a persistent volume claim as well, the deployment silently fails. Below is the deployment.yaml file I am using. The service deploys fine if I remove the first 14 lines that define the persistent volume claim.
apiVersion: apps/v1
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: timescale-pvc-1
namespace: my-namespace
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: standard
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: timescale
spec:
selector:
matchLabels:
app: timescale
replicas: 1
template:
metadata:
labels:
app: timescale
spec:
containers:
- name: timescale
image: timescale/timescaledb:2.3.0-pg11
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5432
env:
- name: POSTGRES_PASSWORD
value: "password"
- name: POSTGRES_DB
value: "metrics"
volumes:
- name: timescaledb-pv
persistentVolumeClaim:
claimName: timescale-pvc-1
Consider StatefulSet for running stateful apps like databases. Deployment is preferred for stateless services.
You are using the below storage class in the PVC.
storageClassName: standard
Ensure the storage class supports dynamic storage provisioning.
Are you creating a PV along with PVC and Deployment? A Deployment, Stateful set or a Pod can only use PVC if there is a PV available.
If you're creating the PV as well then there's a possibility of a different issue. Please share the logs of your Deployment and PVC
I have a use case where there will be telecom application running within the several pods(every pod will host some configured service for billing for specific client) and this expects the service to store the state so obvious choice is statefulset .
Now the problem is I need to use iscsi as storage in the backend for these pods, Can you please point me to some reference to such use case - I am looking for Yaml for PV PVC and statefulset and how to link them . These PV PVC shall use iscsi as storage option.
Yes you are right statefulset is option however you might can also use the deployment.
You have not mentioned which cloud provider you will be using but still sharing one note : iscsi storage is not optimized with GKE cotnainer OS running K8s nodes so if you are no GCP GKE change OS or would suggest using the Ubuntu image first.
You can start the iscsi service on the Ubuntu first.
You can use the Minio or OpenEBS for the storage option also.
Sharing the details if for OpenEBS
Create GCP disks for attaching nodes as a mount or you can dynamically provision it using the YAML as per need.
apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
name: disk-pool
annotations:
cas.openebs.io/config: |
- name: PoolResourceRequests
value: |-
memory: 2Gi
- name: PoolResourceLimits
value: |-
memory: 4Gi
spec:
name: disk-pool
type: disk
poolSpec:
poolType: striped
blockDevices:
blockDeviceList:
- blockdevice-<Number>
- blockdevice-<Number>
- blockdevice-<Number>
Stoage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-sc-rep1
annotations:
openebs.io/cas-type: cstor
cas.openebs.io/config: |
- name: StoragePoolClaim
value: "disk-pool"
- name: ReplicaCount
value: "1"
provisioner: openebs.io/provisioner-iscsi
Application workload
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
storageClassName: openebs-sc-rep1
if you are on AWS EBS using the iscsi.
For testing you can also checkout
https://cloud.ibm.com/catalog/content/minio
Few links :
https://support.zadarastorage.com/hc/en-us/articles/360039289891-Kubernetes-CSI-Getting-started-with-Zadara-CSI-on-GKE-Google-Kubernetes-Engine-
I am trying to create a PV and PVC which is accessible by a pod and a cronjob too on the same node.
Currently I am using the following yaml file
apiVersion: v1
kind: PersistentVolume
metadata:
name: wwwroot
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 100Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "wwwroot"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
app: wwwroot
name: wwwroot
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
status: {}
in the statefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: web
name: web
spec:
selector:
matchLabels:
app: web
serviceName: web
replicas: 1
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: xy:latest
ports:
- containerPort: 80
- containerPort: 443
volumeMounts:
- mountPath: /app/wwwroot
name: wwwroot
subPath: tempwwwroot
imagePullSecrets:
- name: xy
restartPolicy: Always
serviceAccountName: ""
volumeClaimTemplates:
- metadata:
name: wwwroot
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi
This works fine with the pods, but I have a cronjob that needs to access the same wwwroot PV.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
labels:
app: import
name: import
spec:
schedule: "09 16 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: import
image: xy:latest
imagePullPolicy: Always
volumeMounts:
- mountPath: /app/wwwroot
name: wwwroot
subPath: tempwwwroot
imagePullSecrets:
- name: xy
restartPolicy: OnFailure
volumes:
- name: wwwroot
persistentVolumeClaim:
claimName: wwwroot
To the best of my knowledge, you cannot bind these PVs to a folder on the host machine like in Docker (which would be nice, but I can live with that). When I run a kubectl get pvc with the following setup, it seem to use the same PVC name wwwroot, but clearly it is not. I have /bin/bash-d into the pod created by the cronjob and the wwwroot folder does not contain the files from the web pod wwwroot folder.
Any idea what am I doing wrong? Also what is the best option the create PV/PVCs on a local 1 node Kubernetes setup? I was also thinking about to create a NFS share on the host for easier access to the files.
Also when first created the PV/PVC and deployed the web statefulSet, the web pod should populate the wwwroot PV with CSS and JS files, but seems like when it bounds to the PV, it deletes everything from that folder and I have to manually copy those files again. Any workaround on that?
You are mounting pv on two pod created and using pv by setting accessModes: ReadWriteOnce
You can check them below.
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
Also instant of local you can use storage class NFS volume which can give you read-write-many. I am not sure Storage class local provide it as it is not in official documents. Chk below
https://kubernetes.io/docs/concepts/storage/persistent-volumes/
P.S. Never use storage class in actual deployment as node in k8s come and go.
I am using kubeadm localy at two physical machines. I don't have any cloud resources, and i want to build a mongodb auto scaling (localy for start, maybe later at cloud). So i have to use the local storage of my two physical machines. I suppose i have to create a local storage class and volumes. I am very new to kubernetes so dont judge me hard. As i read here https://kubernetes.io/blog/2019/04/04/kubernetes-1.14-local-persistent-volumes-ga/ local persisent volumes are only for one node? Is there any way to take advance of my both physical machines storages and build a simple mongo db scaling, using kubernetes mongo operator and ops manager? I made a few tests here, but i could achieve my goal. pod has unbound immediate PersistentVolumeClaims ops manager
What i was thinking in first place, was to "break" my two hard drives into many piecies, and use sharding for mongo dv scaling
thanks in advace.
Well, you can use a NFS Server with the same volume mounted in both nodes sharing the same mount point.
Please be aware this approach is not recommended for production.
There are tons of howtos of how configure nfs server, example:
https://www.tecmint.com/install-nfs-server-on-ubuntu/
https://www.tecmint.com/how-to-setup-nfs-server-in-linux/
With NFS working you can use the hostPath to mount the nfs in you pods:
Create the PV and the PVC:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-volume
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/nfs/data"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
And use the volume in your deployment file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-pv
spec:
replicas: 1
selector:
matchLabels:
app: test-pv
template:
metadata:
labels:
app: test-pv
spec:
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: /data
name: pv-storage
volumes:
- name: pv-storage
persistentVolumeClaim:
claimName: pv-claim
I have a DaemonSet that creates flink task manager pods, one per each node.
Nodes
Say I have two nodes
node-A
node-B
Pods
the daemonSet would create
pod-A on node-A
pod-B on node-B
Persistent Volume Claim
I am on AKS and want to use azure-disk for Persistent Storage
According to the docs : [https://learn.microsoft.com/en-us/azure/aks/azure-disks-dynamic-pv ]
an azure disk can be associated on to single node
say I create
pvc-A for pv-A attached to node-A
pvc-B for pv-B attached to node-B
Question
How can I associate pod-A on node-A to use pcv-A ?
UPDATE:
After much googling, i stumbled upon that it might be better/cleaner to use a StatefulSet instead. This does mean that you won't get the features available to you via DaemonSet like one pod per node.
https://medium.com/#zhimin.wen/persistent-volume-claim-for-statefulset-8050e396cc51
If you use a persistentVolumeClaim in your daemonset definition, and the persistentVolumeClaim is satisfied with PV with the type of hostPath, your daemon pods will read and write to the local path defined by hostPath. This behavior will help you separate the storage using one PVC.
This might not directly apply to your situation but I hope this helps whoever searching for something like a "volumeClaimTemplate for DaemonSet" in the future.
Using the same example as cookiedough (thank you!)
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: x
namespace: x
labels:
k8s-app: x
spec:
selector:
matchLabels:
name: x
template:
metadata:
labels:
name: x
spec:
...
containers:
- name: x
...
volumeMounts:
- name: volume
mountPath: /var/log
volumes:
- name: volume
persistentVolumeClaim:
claimName: my-pvc
And that PVC is bound to a PV (Note that there is only one PVC and one PV!)
apiVersion: v1
kind: PersistentVolume
metadata:
creationTimestamp: null
labels:
type: local
name: mem
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
hostPath:
path: /tmp/mem
type: Directory
storageClassName: standard
status: {}
Your daemon pods will actually use /tmp/mem on each node. (There's at most 1 daemon pod on each node so that's fine.)
The way to attach a PVC to your DaemonSet pod is not any different than how you do it with other types of pods. Create your PVC and mount it as a volume onto the pod.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-pvc
namespace: x
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
This is what the DaemonSet manifest would look like:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: x
namespace: x
labels:
k8s-app: x
spec:
selector:
matchLabels:
name: x
template:
metadata:
labels:
name: x
spec:
...
containers:
- name: x
...
volumeMounts:
- name: volume
mountPath: /var/log
volumes:
- name: volume
persistentVolumeClaim:
claimName: my-pvc