I am running Kubernetes in a GKE cluster using version 1.6.6 and another cluster with 1.6.4. Both are experiencing issues with failing over GCE compute disks.
I have been simulating failures using kill 1 inside the container or killing the GCE node directly. Sometimes I get lucky and the pod will get created on the same node again. But most of the time this isn't the case.
Looking at the event log it shows the error trying to mount 3 times and it fails to do anything more. Without human intervention it never corrects it self. I am forced to kill the pod multiple times until it works. During maintenances this is a giant pain.
How do I get Kubernetes to fail over with volumes properly ? Is there a way to tell the deployment to try a new node on failure ? Is there a way to remove the 3 retry limit ?
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: dev-postgres
namespace: jolene
spec:
revisionHistoryLimit: 0
template:
metadata:
labels:
app: dev-postgres
namespace: jolene
spec:
containers:
- image: postgres:9.6-alpine
imagePullPolicy: IfNotPresent
name: dev-postgres
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgres-data
env:
[ ** Removed, irrelevant environment variables ** ]
ports:
- containerPort: 5432
livenessProbe:
exec:
command:
- sh
- -c
- exec pg_isready
initialDelaySeconds: 30
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
exec:
command:
- sh
- -c
- exec pg_isready --host $POD_IP
initialDelaySeconds: 5
timeoutSeconds: 3
periodSeconds: 5
volumes:
- name: postgres-data
persistentVolumeClaim:
claimName: dev-jolene-postgres
I have tried this with and without PersistentVolume / PersistentVolumeClaim.
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
name: dev-jolene-postgres
spec:
capacity:
storage: "1Gi"
accessModes:
- "ReadWriteOnce"
claimRef:
namespace: jolene
name: dev-jolene-postgres
gcePersistentDisk:
fsType: "ext4"
pdName: "dev-jolene-postgres"
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dev-jolene-postgres
namespace: jolene
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
By default, every node is schedulable, so there is no need to explicitly mention it in deployment. and feature which can mention retry limits is still in progress, which can be tracked here, https://github.com/kubernetes/kubernetes/issues/16652
Related
I am using payara/server-full in Kubernetes. I want to add a persistent volume so that all configuration made to the Payara server via the Admin UI is perstisted after the pod is recreated, including uploaded .war files.
Right now my deployment looks like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name:
spec:
selector:
matchLabels:
app: myapp
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: payara/server-full
imagePullPolicy: "Always"
ports:
- name: myapp-default
containerPort: 8080
- name: myapp-admin
containerPort: 4848
How to augment that yaml file to make use of a persistent volume?
Which path(s) within payara should be synced with the persistent volume so that Payara's configuration isn't lost after redeployment ?
Which additional yaml files do I need?
So after a longer conideration of the problem I realised I need to persist everything under /opt/payara/appserver/glassfish/domains for all configuration made via the Admin UI to be persisted. However if I simply start the pod with a volumeMount pointing to that path, i.e.
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
selector:
matchLabels:
app: myapp
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: myapp
spec:
volumes:
- name: myapp-vol
persistentVolumeClaim:
claimName: myapp-rwo-pvc
containers:
- name: myapp
image: payara/server-full
imagePullPolicy: "Always"
ports:
- name: myapp-default
containerPort: 8080
- name: myapp-admin
containerPort: 4848
volumeMounts:
- mountPath: "/opt/payara/appserver/glassfish/domains"
and
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myapp-rwo-pvc
labels:
app: dont-delete-autom
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
then the Payara server won't be able to start successfully, because Kubernetes will mount an empty persistent volume into that location. Payara needs however config files which are originally located within /opt/payara/appserver/glassfish/domains.
What I needed to do is to provision the volume with the data by default located in that folder. But how to do that when the only way to access the PV is to mount it into a pod?
Fist I scaled the above deployment to 0 with:
kubectl scale --replicas=0 deployment/myapp
This deletes all pods accessing the persistent volume.
Then I created a "provisioning" pod which mounts the previously created persistent volume into /tmp.
apiVersion: v1
kind: Pod
metadata:
labels:
app: myapp
name: pv-provisioner
namespace: default
spec:
containers:
- image: payara/server-full
imagePullPolicy: Always
name: pv-provisioner
ports:
- containerPort: 8080
name: myapp-default
protocol: TCP
- containerPort: 4848
name: myapp-admin
protocol: TCP
volumeMounts:
- mountPath: "/tmp"
name: myapp-vol
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: 500m
memory: 128Mi
volumes:
- name: myapp-vol
persistentVolumeClaim:
claimName: myapp-rwo-pvc
Then I used the following commands to copy the necessary data first from the "provisioning" pod to a local folder /tmp and then back from /tmp to the persistent volume (previously mounted into pv-provisioner:/tmp). There is no option to copy directly from pod:/a to pod:/b
kubectl cp pv-provisioner:/opt/payara/appserver/glassfish/domains/. tmp
kubectl cp tmp/. pv-provisioner:/tmp
As a result everything stored under /opt/payara/appserver/glassfish/domains/ in the original payara container was now copied into the persistent volume identified by the persistence volume claim "myapp-rwo-pvc".
To finish it up I deleted the provisioning pod and scaled the deployment back up:
kubectl delete pod pv-provisioner
kubectl scale --replicas=3 deployment/myapp
The payara server is now starting successfully and any configuration made via the Admin UI, including .war deployments is persisted, such that the payara pods can be killed any time and after the restart everything is as before.
Thanks for reading.
I am trying to setup a Postgresql Stateful-Set on a k3s single node (Raspberry Pi 4 8Gb) cluster which will be common between any services that end up needing postgresql. Currently, since this is just a single node, I am using the Rancher local-path storage provisioner (this will probably change as I add nodes) pointing to an external hdd set to mount, though this will probably change when I get around to adding nodes.
My pod spins up (after figuring out a small headache with mounting the data directory which was as per this issue), and I can access the postgresql instance with kubectl port-forward -n common pod/postgresql-stateful-set-0 5432:5432, and making whatever changes I need.
At this point, my problem manifests, whereby I notice that no data is persisted in my data directory. I have found this issue which is the exact issue I've encountered, but following all the examples of "fixes", from playing with the directory locations (which gave me chmod issues like encountered with the mounting data directory issues), to changing to a pv I defined.
Find below my yml file which is use kubectl apply -f postgres.yml and is based on the bitnami helm template:
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: postgresql-storage
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
---
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgresql-pv
namespace: common
spec:
storageClassName: postgresql-storage
persistentVolumeReclaimPolicy: Retain
capacity:
storage: 20Gi
accessModes:
- ReadWriteMany
hostPath:
path: "/mnt/storage/k3s/common/postgresql"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
namespace: common
name: pvc-postgresql
spec:
storageClassName: postgresql-storage
accessModes:
- ReadWriteMany
resources:
requests:
storage: 20Gi
---
apiVersion: v1
kind: Secret
metadata:
name: postgresql-password
namespace: common
data:
POSTGRES_PASSWORD: <PWD>
---
apiVersion: v1
kind: ConfigMap
metadata:
name: postgresql-configmap
namespace: common
data:
POSTGRESQL_PORT_NUMBER: "5432"
PGDATA: /var/lib/postgresql/data/pgdata
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgresql-stateful-set
namespace: common
labels:
name: postgres-stateful-set
spec:
replicas: 1
serviceName: postgresql-stateful-set
updateStrategy:
rollingUpdate: {}
type: RollingUpdate
selector:
matchLabels:
app: postgresql
template:
metadata:
labels:
app: postgresql
spec:
containers:
- name: postgresql
image: docker.io/postgres:14.2-alpine
envFrom:
- configMapRef:
name: postgresql-configmap
- secretRef:
name: postgresql-password
ports:
- name: tcp-postgresql
containerPort: 5432
livenessProbe:
failureThreshold: 6
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
exec:
command:
- /bin/sh
- -c
- exec pg_isready -U "rootAdmin" -h 127.0.0.1 -p 5432 -d rootDefault
readinessProbe:
failureThreshold: 6
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
exec:
command:
- /bin/sh
- -c
- -e
- exec pg_isready -U "rootAdmin" -h 127.0.0.1 -p 5432 -d rootDefault
resources:
limits:
memory: "300Mi"
cpu: "300m"
requests:
cpu: 250m
memory: 256Mi
volumeMounts:
- name: dshm
mountPath: /dev/shm
- name: postgresql-data
mountPath: /var/lib/postgresql
volumes:
- name: dshm
emptyDir:
medium: Memory
volumeClaimTemplates:
- metadata:
name: postgresql-data
spec:
storageClassName: postgresql-storage
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "20Gi"
---
kind: Service
apiVersion: v1
metadata:
name: postgresql-svc
namespace: common
spec:
selector:
app: postgresql
type: ClusterIP
clusterIP: None
ports:
- name: tcp-postgresql
port: 5432
protocol: TCP
Couple of things I should probably mention:
I added my own Storage class for the purposes of retaining the record rather then delete on the record being removed.
using the alpine version just for the sake of smaller image size.
The base OS that k3s is hosted on is Ubuntu 20.04 LTS
K3s was setup using this ansible playbook here
I have exec-ed into the container to see what is in the directory defined by PGDATA and confirmed that data was written.
Using kubeadm to create a cluster, I have a master and work node.
Now I want to share a persistentVolume in the work node, which will be bound with Postgres pod.
Expecting the code will create persistentVolume in the path /postgres of work node, but it seems the hostPath will not work in a cluster, how should I assign this property to the specific node?
kind: PersistentVolume
apiVersion: v1
metadata:
name: pv-postgres
labels:
type: local
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/postgres"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-postgres
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
spec:
selector:
matchLabels:
app: postgres
replicas: 1
strategy: {}
template:
metadata:
labels:
app: postgres
spec:
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
volumes:
- name: vol-postgres
persistentVolumeClaim:
claimName: pvc-postgres
containers:
- name: postgres
image: postgres:12
imagePullPolicy: Always
env:
- name: DB_USER
value: postgres
- name: DB_PASS
value: postgres
- name: DB_NAME
value: postgres
ports:
- name: postgres
containerPort: 5432
volumeMounts:
- mountPath: "/postgres"
name: vol-postgres
livenessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
- postgres
initialDelaySeconds: 30
timeoutSeconds: 5
readinessProbe:
exec:
command:
- pg_isready
- -h
- localhost
- -U
- postgres
initialDelaySeconds: 5
timeoutSeconds: 1
---
apiVersion: v1
kind: Service
metadata:
name: postgres
spec:
ports:
- name: postgres
port: 5432
targetPort: postgres
selector:
app: postgres
As per docs.
A hostPath volume mounts a file or directory from the host node’s filesystem into your Pod. This is not something that most Pods will need, but it offers a powerful escape hatch for some applications.
In short, hostPath type refers to node (machine or VM) resource, where you will schedule pod. It mean that you already need to have this folder on this node.
To assign resources to specify node you have to use nodeSelector in your Deployment, PV.
Depends on the scenario, using hostPath is not the best idea, however I will provide below example YAMLs which might show you concept. Based on your YAMLs but with nginx image.
kind: PersistentVolume
apiVersion: v1
metadata:
name: pv-postgres
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp/postgres" ## this folder need exist on your node. Keep in minds also who have permissions to folder. Used tmp as it have 3x rwx
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ubuntu18-kubeadm-worker1
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-postgres
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
spec:
selector:
matchLabels:
app: postgres
replicas: 1
strategy: {}
template:
metadata:
labels:
app: postgres
spec:
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: /home ## path to folder inside container
name: vol-postgres
affinity: ## specified affinity to schedule all pods on this specific node with name ubuntu18-kubeadm-worker1
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ubuntu18-kubeadm-worker1
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
volumes:
- name: vol-postgres
persistentVolumeClaim:
claimName: pvc-postgres
persistentvolume/pv-postgres created
persistentvolumeclaim/pvc-postgres created
deployment.apps/postgres created
Unfortunately PV is bounded to PVC in 1:1 relationship, so for each time, you would need to create PV and PVC.
However if you are using hostPath it's enough to specify nodeAffinity, volumeMounts and volumes in Deployment YAML without PV and PVC.
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
spec:
selector:
matchLabels:
app: postgres
replicas: 1
strategy: {}
template:
metadata:
labels:
app: postgres
spec:
containers:
- image: nginx:latest
name: nginx
volumeMounts:
- mountPath: /home
name: vol-postgres
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ubuntu18-kubeadm-worker1
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
volumes:
- name: vol-postgres
hostPath:
path: /tmp/postgres
deployment.apps/postgres created
user#ubuntu18-kubeadm-master:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
postgres-77bc9c4566-jgxqq 1/1 Running 0 9s
user#ubuntu18-kubeadm-master:~$ kk exec -ti postgres-77bc9c4566-jgxqq /bin/bash
root#ubuntu18-kubeadm-worker1:/# cd home
root#ubuntu18-kubeadm-worker1:/home# ls
test.txt txt.txt
There are ways to achieve it. You can mount your volume into a NAS or create a storage cluster using disks and create a persistent volume and persistent volume claim for that. If your use-case is to have persistence in local storage then you can create a local-storage storageclass in one of your cluster nodes and that volume space can be used by any pod in your cluster. To create a local-storage storageclass, refer this (https://kubernetes.io/blog/2019/04/04/kubernetes-1.14-local-persistent-volumes-ga/)
I have a pod running a cassandra container. I want to create a keyspace once the container starts. I tried using the postStart hook. For some reason it does not fail but the keyspace does not get created. But I tried the same command in the readinessProbe as a hack and it worked fine. Can someone help me understand what's wrong with my configuration. Thanks in advance
apiVersion: v1
kind: PersistentVolume
metadata:
name: local-cass-volume-1
labels:
type: local
app: test
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /tmp/data/cass-volume-1
---
apiVersion: v1
kind: Service
metadata:
name: test-cassandra
labels:
app: test
spec:
ports:
- port: 9042
selector:
app: test
tier: cass
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cass-pv-claim
labels:
app: test
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: test-cassandra
labels:
app: test
spec:
strategy:
type: Recreate
template:
metadata:
labels:
app: test
tier: cass
spec:
containers:
- image: cassandra:latest
name: cassandra
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9042
name: cass-port
volumeMounts:
- name: cass-persistent-storage
mountPath: /var/lib/cassandra
readinessProbe:
exec:
command: ["cqlsh", "-e", "CREATE KEYSPACE IF NOT EXISTS test1234 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;"]
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 10
failureThreshold: 5
lifecycle:
postStart:
exec:
command: ["/bin/bash", "-c", "until echo $'CREATE KEYSPACE IF NOT EXISTS test908 WITH replication = {\'class\': \'SimpleStrategy\', \'replication_factor\': \'1\'} AND durable_writes = true;' | cqlsh ; do echo boo; sleep 2; done"]
volumes:
- name: cass-persistent-storage
persistentVolumeClaim:
claimName: cass-pv-claim
I had the same issue of postStart hook commands not executing. After went through this tried readinessProbe trick and it worked for me as well. The issue was, I am executing curl commands to which the same service should reply back. even though its postStart hook application needed a few more seconds to reply back. adding sleep command before executing curl commands solved my issue.
I would like to run specific command after initialization of deployment is successful.
This is my yaml file:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: auth
spec:
replicas: 1
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: {{my-service-image}}
env:
- name: NODE_ENV
value: "docker-dev"
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 3000
However, I would like to run command for db migration after (not before) deployment is successfully initialized and pods are running.
I can do it manually for every pod (with kubectl exec), but this is not very scalable.
I resolved it using lifecycles:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: auth
spec:
replicas: 1
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: {{my-service-image}}
env:
- name: NODE_ENV
value: "docker-dev"
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 3000
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", {{cmd}}]
You can use Helm to deploy a set of Kubernetes resources. And then, use a Helm hook, e.g. post-install or post-upgrade, to run a Job in a separate docker container. Set your Job to invoke db migration. A Job will run >=1 Pods to completion, so it fits here quite well.
I chose to use a readinessProbe
My application requires configuration after the process has completely started.
The postStart command was running before the app was ready.
readinessProbe:
exec:
command: [healthcheck]
initialDelaySeconds: 30
periodSeconds: 2
timeoutSeconds: 1
successThreshold: 3
failureThreshold: 10