I create a Deployment with a volumeMount that references a PersistentVolumeClaim along with a memory request on a cluster with nodes in 3 difference AZs us-west-2a, us-west-2b, and us-west-2c.
The Deployment takes a while to start while the PersistentVolume is being dynamically created but they both eventually start up.
The problem I am running into is that the PersistentVolume is made in us-west-2c and the only node the pod can run on is already over allocated.
Is there a way for me to create the Deployment and claim such that the claim is not made in a region where no pod can start up?
I believe you're looking for Topology Awareness feature.
Topology Awareness
In Multi-Zone clusters, Pods can be spread across
Zones in a Region. Single-Zone storage backends should be provisioned
in the Zones where Pods are scheduled. This can be accomplished by
setting the Volume Binding Mode.
Kubernetes released topology-aware dynamic provisioning feature with kubernetes version 1.12, and I believe this will solve your issue.
Related
i am running an AKS cluster and i have deployed Prometheus and Alertmanager via deployment resources in k8s and they also are controlled by replicaset.The issue is that sometimes the restart of Alertmanger get stuck.It is related to accessMode of PVC.During restart,k8s will start the new pod in a different node from the currently node where the running pod is assigned,depending on resource utilization on the node.In simple words it means,same PVC is accessed from 2 different pods assigned to different nodes.This is not allowed because in the config of PVC i am using accessMode ReadWriteOnce.Looking this comment in github for prometheus operator seems to be by design that option accessMode ReadWriteMany is not allowed.
So my questions, why such design and what could happen if i change accessMode to ReadWriteMany?Any practical experience?
I am in the process of deploying ActiveMQ 5.15 in HA on Kubernetes. Previously I was using a deployment and a clusterIP Service. And it was working fine. The master will boot up and the slave will wait for the lock to be acquired. If I delete the pod which is the master one, the slave picks up and becomes the master.
Now I want to try with statefulset basing myself on this thread.
Deployment is done successfully and two pods were created with id0 and id1. But what I noticed is that both pods were master. They were both started. I noticed also that two PVC were created id0 and id1 in the case of Statefulset compared to deployment which had only 1 PVC. Could that be the issue since it is no more a shared storage? Can we still achieve a master/slave setup with Statefulset?
I noticed also that two PVC were created id0 and id1 in the case of statefulset compared to deployment which had only 1 PVC. Could that be the issue since it is no more a shared storage?
You are right. When using k8s StatefulSets each Pod gets its own persistent storage (dedicated PVC and PV), and this persistent storage is not shared.
When a Pod gets terminated and is rescheduled on a different Node, the Kubernetes controller will ensure that the Pod is associated with the same PVC which will guarantee that the state is intact.
In your case, to achieve a master/slave setup, consider using a shared network location / filesystem for persistent storage like:
NFS storage for on-premise k8s cluster.
AWS EFS for EKS.
or Azure Files for AKS.
Check the complete list of PersistentVolume types currently supported by Kubernetes (implemented as plugins).
Let's say I have a daemonset running in my k8s cluster, and each pod created by the daemonset creates and writes to a directory on the node where it's running. Is there a way to automatically sync the folders with one in the masters? Given I have a multi-master cluster.
You can have a persistent volume that has ReadWriteMany access mode so that all daemonsets can share the same set of data between them.
A simple example is here: http://snippi.com/s/qpge73r
Edit: as #matt commented few drivers support that as here https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
I want to apply VPA vertical pod auto scaling for database pods. Can we use VPA for database auto scaling (Vertical) as VPA requires at least 2 replicas (ref : https://github.com/kubernetes/autoscaler/issues/1665#issuecomment-464679271) as it delete pods when set criteria is reached. So pods are deleted hence also data.
What is good practice for using VPA with database pods?
As I understand it, the real question is how to run a stateful workload with multiple replicas.
Use StatefulSets to configure n replicas for a database. StatefulSet pods have stable names which are preserved across pod restarts (and reincarnations). Combined with PersistentVolumeClaim templates (accepted with StatefulSet spec) and headless services, it is capable of retaining same volumes and network FQDN across reincarnations.
Take a look at Helm charts for various databases, e.g. MySQL chart, for useful insights.
On a side note, it might be worthwhile to consider using an operator for the database application you're using. Operators for most applications can be found on https://operatorhub.io.
VPA - Vertical pod autoscaler can work in 2 ways:
Recommendation mode - it will recommend the requests and limits for pods based on resources used
Auto mode - it will automatically analyze the usage and set the request and limits on pods. This will result in pod termination to recreate it with new specification as stated here:
Due to Kubernetes limitations, the only way to modify the resource requests of a running Pod is to recreate the Pod. If you create a VerticalPodAutoscaler with an updateMode of "Auto", the VerticalPodAutoscaler evicts a Pod if it needs to change the Pod's resource requests.
Cloud.google.com: Kubernetes Engine: Docs: Concepts: Vertical pod autoscaler
Please refer to above link for more information regarding the concepts of VPA.
The fact that it needs at least 2 replicas is most probably connected with the fact of high availability. As the pods are getting evicted to support new limits they are unable to process the request. If it came to situation where there is only 1 replica at the time, this replica wouldn't be able to respond to requests when in terminating/recreating state.
There is an official guide to run VPA on GKE:
Cloud.google.com: Kubernetes Engine: How to: Vertical pod autoscaling
VPA supports: Deployments as well as StatefulSets.
StatefulSet
Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.
If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution.
Kubernetes.io: StatefulSet
Configuring StatefulSet with PersistentVolumes will ensure that the data stored on PV will not be deleted in case of pod termination.
To be able to use your database with replicas > 1 you will need to have replication implemented within your database environment.
There are guides/resources/solutions on running databases within Kubernetes environment. Please choose the solution most appropriate to your use case. Some of them are:
Kubernetes.io: Run replicated stateful application
Github.com: Zalando: Postgres operator
Github.com: Oracle: Mysql operator
After deploying your database you will be able to run below command to extract the name of the StatefulSet:
$ kubectl get sts
You can then apply the name of the StatefulSet to the VPA like below:
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: DB-VPA
spec:
targetRef:
apiVersion: "apps/v1"
kind: StatefulSet
name: <INSERT_DB_STS_HERE>
updatePolicy:
updateMode: "Auto"
I encourage you also to read this article:
Cloud.google.com: Blog: To run or not to run a database on Kubernetes, what to consider
I have several operational deployments on minikube locally and am trying to deploy them on GCP with kubernetes.
When I describe a pod created by a deployment (which created a replication set that spawned the pod):
kubectl get po redis-sentinel-2953931510-0ngjx -o yaml
It indicates it landed on one of the kubernetes vms.
I'm having trouble with deployments that work separately failing due to lack of resources e.g. cpu even though I provisioned a VM above the requirements. I suspect the cluster is placing the pods on it's own nodes and running out of resources.
How should I proceed?
Do I introduce a vm to be orchestrated by kubernetes?
Do I enlarge the kubernetes nodes?
Or something else all together?
It was a resource problem and node pool size was inhibiting the deployments.I was mistaken in trying to provide google compute instances and disks.
I ended up provisioning Kubernetes node pools with more cpu and disk space and solved it. I also added elasticity by provisioning autoscaling.
here is a node pool documentation
here is a terraform Kubernetes deployment
here is the machine type documentation