Rabbitmq pod stuck in pending state - kubernetes

I tries running rabbitmq following the book Kubernetes for developers (page 180):
rabbitmq.yml
---
# EXPORT SERVICE INTERFACE
kind: Service
apiVersion: v1
metadata:
name: message-queue
labels:
app: rabbitmq
role: master
tier: queue
spec:
ports:
- port: 5672
targetPort: 5672
selector:
app: rabbitmq
role: master
tier: queue
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rabbitmq-pv-claim
labels:
app: rabbitmq
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rabbitmq
spec:
replicas: 1
selector:
matchLabels:
app: rabbitmq
role: master
tier: queue
template:
metadata:
labels:
app: rabbitmq
role: master
tier: queue
spec:
containers:
- name: rabbitmq
image: bitnami/rabbitmq:3.7
envFrom:
- configMapRef:
name: bitnami-rabbitmq-config
ports:
- name: queue
containerPort: 5672
- name: queue-mgmt
containerPort: 15672
livenessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 120
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 5
volumeMounts:
- name: rabbitmq-storage
mountPath: /bitnami
volumes:
- name: rabbitmq-storage
persistentVolumeClaim:
claimName: rabbitmq-pv-claim
# kubectl describe pod rabbitmq-5499d4b67d-cdlb8
Name: rabbitmq-5499d4b67d-cdlb8
Namespace: default
Priority: 0
Node: <none>
Labels: app=rabbitmq
pod-template-hash=5499d4b67d
role=master
tier=queue
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/rabbitmq-5499d4b67d
Containers:
rabbitmq:
Image: bitnami/rabbitmq:3.7
Ports: 5672/TCP, 15672/TCP
Host Ports: 0/TCP, 0/TCP
Liveness: exec [rabbitmqctl status] delay=120s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [rabbitmqctl status] delay=10s timeout=3s period=5s #success=1 #failure=3
Environment Variables from:
bitnami-rabbitmq-config ConfigMap Optional: false
Environment: <none>
Mounts:
/bitnami from rabbitmq-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xh899 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
rabbitmq-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rabbitmq-pv-claim
ReadOnly: false
default-token-xh899:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xh899
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 28s (x13 over 12m) default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.

0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
The pod cannot be scheduled because the PVC that it is using is not "bound". You need to investigate why the PVC is not bound, something related to the storage system that you use.

Related

Kubernetes Pod is unable to mount volumes to GCP Filestore

I am new to Kubernetes, and as a part of tutorial I have spun up a GKE cluster and a GCP Filestore instance.
Now I am trying to mount Grafana's volume to this Filestore instance. However, it is getting timed out. I am unable to decipher where the mistake lies. I need your help in addressing the issue.
PFB the output.
C:\Users\ak>kubectl describe pod/grafana-7c666cff94-vkgh4
Name: grafana-7c666cff94-vkgh4
Namespace: bc
Priority: 0
Node: gke-bc-gke-cluster-bc-nodepool-9496e187-zsnw/10.51.0.5
Start Time: Fri, 02 Sep 2022 16:21:28 +0530
Labels: app=grafana
pod-template-hash=7c666cff94
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/grafana-7c666cff94
Containers:
grafana:
Container ID:
Image: grafana/grafana:8.4.4
Image ID:
Port: 3000/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 250m
memory: 750Mi
Liveness: tcp-socket :3000 delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:3000/robots.txt delay=10s timeout=2s period=30s #success=1 #failure=3
Environment: <none>
Mounts:
/var/lib/grafana from fileserver (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v7qjd (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
fileserver:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: fileserver-claim
ReadOnly: false
kube-api-access-v7qjd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 43m default-scheduler Successfully assigned bluecopa/grafana-7c666cff94-vkgh4 to gke-bc-gke-cluster-bc-nodepool-9496e187-zsnw
Warning FailedMount 4m15s (x11 over 40m) kubelet MountVolume.SetUp failed for volume "fileserver" : mount failed: exit status 1
Mounting command: /home/kubernetes/containerized_mounter/mounter
Mounting arguments: mount -t nfs 10.168.189.130:/bc_fs /var/lib/kubelet/pods/cf44b980-7461-4c0e-a32f-673588160692/volumes/kubernetes.io~nfs/fileserver
Output: Mount failed: mount failed: exit status 32
Mounting command: chroot
Mounting arguments: [/home/kubernetes/containerized_mounter/rootfs mount -t nfs xx.xx.xx.xx:/bc_fs /var/lib/kubelet/pods/cf44b980-7461-4c0e-a32f-673588160692/volumes/kubernetes.io~nfs/fileserver]
Output: mount.nfs: Connection timed out
Warning FailedMount 3m16s (x12 over 37m) kubelet Unable to attach or mount volumes: unmounted volumes=[fileserver], unattached volumes=[fileserver kube-api-access-v7qjd]: timed out waiting for the condition
Warning FailedMount 59s (x7 over 41m) kubelet Unable to attach or mount volumes: unmounted volumes=[fileserver], unattached volumes=[kube-api-access-v7qjd fileserver]: timed out waiting for the condition
PV.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: fileserver
namespace: bluecopa
spec:
capacity:
storage: 200Gi
accessModes:
- ReadWriteMany
nfs:
path: /bc_fs
server: xx.xx.xx.xx
PVC.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fileserver-claim
namespace: bluecopa
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
volumeName: fileserver
resources:
requests:
storage: 100Gi
Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: grafana
name: grafana
namespace: bluecopa
spec:
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
securityContext:
fsGroup: 472
supplementalGroups:
- 0
containers:
- name: grafana
image: grafana/grafana:8.4.4
imagePullPolicy: IfNotPresent
ports:
- containerPort: 3000
name: http-grafana
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /robots.txt
port: 3000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 2
livenessProbe:
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 3000
timeoutSeconds: 1
resources:
requests:
cpu: 250m
memory: 750Mi
volumeMounts:
- mountPath: /var/lib/grafana
name: fileserver
volumes:
- name: fileserver
persistentVolumeClaim:
claimName: fileserver-claim
While using the volume mounts in pods we need to watch out for security context
Use the securitycontext as follows in deployment file
securityContext:
runAsUser: 0
Use the following security context in the deployment file
This will help you out to mount the volume without any issues.
For more information check out this documents
Doc1 &
Doc2
Here is the output of deployment pod

Getting error mkdir: cannot create directory ‘/bitnami/rabbitmq’: Permission denied when creating Kubernetes pod of Rabbitmq

While learning Kubernetes going by the book Kubernetes for developer, I am stuck at this point now.
I am trying to start Rabbitmq pod but but after lot of troubleshooting I have managed to get to this point but do not get clue where do I fix to get rid of the permission denied error.
# kubectl get pods
NAME READY STATUS RESTARTS AGE
rabbitmq-56c67d8d7d-s8vp5 0/1 CrashLoopBackOff 5 5m40s
if I look at the logs of this contianer thats where I found:
# kubectl logs rabbitmq-56c67d8d7d-s8vp5
21:22:58.49
21:22:58.50 Welcome to the Bitnami rabbitmq container
21:22:58.51 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
21:22:58.51 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
21:22:58.52 Send us your feedback at containers#bitnami.com
21:22:58.52
21:22:58.52 INFO ==> ** Starting RabbitMQ setup **
21:22:58.54 INFO ==> Validating settings in RABBITMQ_* env vars..
21:22:58.56 INFO ==> Initializing RabbitMQ...
21:22:58.57 INFO ==> Generating random cookie
mkdir: cannot create directory ‘/bitnami/rabbitmq’: Permission denied
Here is my rabbitmq-deployment.yml
---
# EXPORT SERVICE INTERFACE
kind: Service
apiVersion: v1
metadata:
name: message-queue
labels:
app: rabbitmq
role: master
tier: queue
spec:
ports:
- port: 5672
targetPort: 5672
selector:
app: rabbitmq
role: master
tier: queue
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rabbitmq-pv-claim
labels:
app: rabbitmq
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rabbitmq
spec:
replicas: 1
selector:
matchLabels:
app: rabbitmq
role: master
tier: queue
template:
metadata:
labels:
app: rabbitmq
role: master
tier: queue
spec:
nodeSelector:
boardType: x86vm
containers:
- name: rabbitmq
image: bitnami/rabbitmq:3.7
envFrom:
- configMapRef:
name: bitnami-rabbitmq-config
ports:
- name: queue
containerPort: 5672
- name: queue-mgmt
containerPort: 15672
livenessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 120
timeoutSeconds: 5
failureThreshold: 6
readinessProbe:
exec:
command:
- rabbitmqctl
- status
initialDelaySeconds: 10
timeoutSeconds: 3
periodSeconds: 5
volumeMounts:
- name: rabbitmq-storage
mountPath: /bitnami
volumes:
- name: rabbitmq-storage
persistentVolumeClaim:
claimName: rabbitmq-pv-claim
This is the rabbitmq-storage-class.yml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: rabbitmq-storage-class
labels:
app: rabbitmq
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
and persistant-volume.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: rabbitmq-pv-claim
labels:
app: rabbitmq
spec:
storageClassName: manual
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /bitnami
Logs:
# kubectl describe pods rabbitmq-5f7f787479-fpg6g
Name: rabbitmq-5f7f787479-fpg6g
Namespace: default
Priority: 0
Node: kube-worker-vm2/192.168.1.36
Start Time: Mon, 03 May 2021 12:29:17 +0100
Labels: app=rabbitmq
pod-template-hash=5f7f787479
role=master
tier=queue
Annotations: cni.projectcalico.org/podIP: 192.168.222.4/32
cni.projectcalico.org/podIPs: 192.168.222.4/32
Status: Running
IP: 192.168.222.4
IPs:
IP: 192.168.222.4
Controlled By: ReplicaSet/rabbitmq-5f7f787479
Containers:
rabbitmq:
Container ID: docker://bbdbb9c5d4b6737519d3dcf4bdda242a7fe904f2336334afe686e9b204fd6d5c
Image: bitnami/rabbitmq:3.7
Image ID: docker-pullable://bitnami/rabbitmq#sha256:8b6057997b74ebc81e934dd6c94e9da745635faa2d79b382cfda27b9176e0e6d
Ports: 5672/TCP, 15672/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 03 May 2021 12:30:48 +0100
Finished: Mon, 03 May 2021 12:30:48 +0100
Ready: False
Restart Count: 4
Liveness: exec [rabbitmqctl status] delay=120s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [rabbitmqctl status] delay=10s timeout=3s period=5s #success=1 #failure=3
Environment Variables from:
bitnami-rabbitmq-config ConfigMap Optional: false
Environment: <none>
Mounts:
/bitnami from rabbitmq-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-4qmxr (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
rabbitmq-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rabbitmq-pv-claim
ReadOnly: false
default-token-4qmxr:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-4qmxr
Optional: false
QoS Class: BestEffort
Node-Selectors: boardType=x86vm
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m20s default-scheduler Successfully assigned default/rabbitmq-5f7f787479-fpg6g to kube-worker-vm2
Normal Created 96s (x4 over 2m18s) kubelet Created container rabbitmq
Normal Started 95s (x4 over 2m17s) kubelet Started container rabbitmq
Warning
BackOff 65s (x12 over 2m16s) kubelet Back-off restarting failed container
Normal Pulled 50s (x5 over 2m18s) kubelet Container image "bitnami/rabbitmq:3.7" already present on machine
When creating an image, the image creator often chooses to use a user other than root to run the process. This is the case for your image, and the user does not have write permissions on the /bitnami directory. You can verify this by commenting out the volume.
To fix the issue, you need to set a security contect for your pod: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod
Not sure about the exact syntax, but something like this should do the trick:
spec:
securityContext:
fsGroup: 1001 # the userid that is used in the image
nodeSelector:
boardType: x86vm
containers:
- name: rabbitmq
image: bitnami/rabbitmq:3.7
envFrom:
- configMapRef:
name: bitnami-rabbitmq-config
This makes the directory writeable by the user in the image.
Another thing: A deployment is for stateless services by design. If you have state to keep, always use a statefulset. It's very similiar to a deployment from a configuration point of view, but Kubernetes treats it very differently. See https://www.youtube.com/watch?v=Vrxr-7rjkvM for good explanation.
As per bitnami documentation, it depends on the kubernetes distribution
Quote from documentation
Adjust permissions of persistent volume mountpoint
As the image run as non-root by default, it is necessary to adjust the ownership of the persistent volume so that the container can write data into it.
By default, the chart is configured to use Kubernetes Security Context to automatically change the ownership of the volume. However, this feature does not work in all Kubernetes distributions. As an alternative, this chart supports using an initContainer to change the ownership of the volume before mounting it in the final destination.
You can enable this initContainer by setting volumePermissions.enabled to true.

Pods not getting scheduled to node with matching labels

I'm getting this error when exec'ing into my pod. Error from server (BadRequest): pod es-master-5cb49c68cc-w6dxv does not have a host assigned
It seemed to be related to my nodeAffinity but I don't see anything immediately wrong with it. I can't seem to get my deployment to attach its pod to any of my nodes. I don't have any taints or tolerations setup on the node or pod. I've tried switching to labels that are automatically generated that are on every node, but nothing seems to work. I've even tried removing my affinity section entirely, and also tried adding nodeSelector to spec by itself.
Here is my deployment config and output from kubectl describe pod -n elasticsearch
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: elasticsearch
role: master
name: es-master
namespace: elasticsearch
spec:
replicas: 3
selector:
matchLabels:
component: elasticsearch
role: master
template:
metadata:
labels:
component: elasticsearch
role: master
annotations:
iam.amazonaws.com/role: {REDACTED}
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
component: elasticsearch
role: master
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/region
operator: In
values:
- us-east-2
Name: es-master-866f7fb558-298ht
Namespace: elasticsearch
Priority: 0
Node: <none>
Labels: component=elasticsearch
pod-template-hash=866f7fb558
role=master
Annotations: iam.amazonaws.com/role: {REDACTED}
kubernetes.io/psp: eks.privileged
Status: Pending
IP:
Controlled By: ReplicaSet/es-master-866f7fb558
Init Containers:
init-sysctl:
Image: busybox:1.27.2
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xflv6 (ro)
Containers:
elasticsearch:
Image: amazon/opendistro-for-elasticsearch:0.9.0
Ports: 9300/TCP, 9200/TCP, 9600/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Limits:
cpu: 2
memory: 12Gi
Requests:
cpu: 2
memory: 12Gi
Liveness: tcp-socket :transport delay=60s timeout=1s period=10s #success=1 #failure=3
Environment:
CLUSTER_NAME: logs
NUMBER_OF_MASTERS: 3
NODE_MASTER: true
NODE_INGEST: false
NODE_DATA: false
NETWORK_HOST: 0.0.0.0
TRANSPORT_TLS_PEM_PASS:
HTTP_TLS_PEM_PASS:
NODE_NAME: es-master-866f7fb558-298ht (v1:metadata.name)
DISCOVERY_SERVICE: elasticsearch-discovery
KUBERNETES_NAMESPACE: elasticsearch (v1:metadata.namespace)
PROCESSORS: 2 (limits.cpu)
ES_JAVA_OPTS: -Xms6g -Xmx6g
Mounts:
/usr/share/elasticsearch/config/admin-crt.pem from certs (ro,path="admin-crt.pem")
/usr/share/elasticsearch/config/admin-key.pem from certs (ro,path="admin-key.pem")
/usr/share/elasticsearch/config/admin-root-ca.pem from certs (ro,path="admin-root-ca.pem")
/usr/share/elasticsearch/config/elasticsearch.yml from config (rw,path="elasticsearch.yml")
/usr/share/elasticsearch/config/elk-crt.pem from certs (ro,path="elk-crt.pem")
/usr/share/elasticsearch/config/elk-key.pem from certs (ro,path="elk-key.pem")
/usr/share/elasticsearch/config/elk-root-ca.pem from certs (ro,path="elk-root-ca.pem")
/usr/share/elasticsearch/config/logging.yml from config (rw,path="logging.yml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xflv6 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch
Optional: false
certs:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-tls-data
Optional: false
default-token-xflv6:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xflv6
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 59s (x3 over 3m44s) default-scheduler 0/8 nodes are available: 8 Insufficient cpu.
All nodes are m5a.large ec2 instances.
The error is pretty clear 0/8 nodes are available: 8 Insufficient cpu which means nodes don't have 2 cpu cores free as specified in requests. Solution is to either provision nodes with more cpu or reduce the cpu requests in pod spec.

pod has unbound immediate PersistentVolumeClaims (repeated 3 times)

What is wrong with below.
# config for es data node
apiVersion: v1
kind: ConfigMap
metadata:
namespace: infra
name: elasticsearch-data-config
labels:
app: elasticsearch
role: data
data:
elasticsearch.yml: |-
cluster.name: ${CLUSTER_NAME}
node.name: ${NODE_NAME}
discovery.seed_hosts: ${NODE_LIST}
cluster.initial_master_nodes: ${MASTER_NODES}
network.host: 0.0.0.0
node:
master: false
data: true
ingest: false
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
---
# service for es data node
apiVersion: v1
kind: Service
metadata:
namespace: infra
name: elasticsearch-data
labels:
app: elasticsearch
role: data
spec:
ports:
- port: 9300
name: transport
selector:
app: elasticsearch
role: data
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
namespace: infra
name: elasticsearch-data
labels:
app: elasticsearch
role: data
spec:
serviceName: "elasticsearch-data"
replicas: 1
template:
metadata:
labels:
app: elasticsearch-data
role: data
spec:
containers:
- name: elasticsearch-data
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.0
env:
- name: CLUSTER_NAME
value: elasticsearch
- name: NODE_NAME
value: elasticsearch-data
- name: NODE_LIST
value: elasticsearch-master,elasticsearch-data,elasticsearch-client
- name: MASTER_NODES
value: elasticsearch-master
- name: "ES_JAVA_OPTS"
value: "-Xms300m -Xmx300m"
ports:
- containerPort: 9300
name: transport
volumeMounts:
- name: config
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
readOnly: true
subPath: elasticsearch.yml
- name: elasticsearch-data-persistent-storage
mountPath: /data/db
volumes:
- name: config
configMap:
name: elasticsearch-data-config
initContainers:
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data-persistent-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
statefull output:
Name: elasticsearch-data-0
Namespace: infra
Priority: 0
Node: <none>
Labels: app=elasticsearch-data
controller-revision-hash=elasticsearch-data-76bdf989b6
role=data
statefulset.kubernetes.io/pod-name=elasticsearch-data-0
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/elasticsearch-data
Init Containers:
increase-vm-max-map:
Image: busybox
Port: <none>
Host Port: <none>
Command:
sysctl
-w
vm.max_map_count=262144
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-9nhmg (ro)
Containers:
elasticsearch-data:
Image: docker.elastic.co/elasticsearch/elasticsearch:7.3.0
Port: 9300/TCP
Host Port: 0/TCP
Environment:
CLUSTER_NAME: elasticsearch
NODE_NAME: elasticsearch-data
NODE_LIST: elasticsearch-master,elasticsearch-data,elasticsearch-client
MASTER_NODES: elasticsearch-master
ES_JAVA_OPTS: -Xms300m -Xmx300m
Mounts:
/data/db from elasticsearch-data-persistent-storage (rw)
/usr/share/elasticsearch/config/elasticsearch.yml from config (ro,path="elasticsearch.yml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-9nhmg (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
elasticsearch-data-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: elasticsearch-data-persistent-storage-elasticsearch-data-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch-data-config
Optional: false
default-token-9nhmg:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-9nhmg
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 46s (x4 over 3m31s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
kubectl get sc
NAME PROVISIONER AGE
standard (default) kubernetes.io/gce-pd 5d19h
kubectl get pv
No resources found in infra namespace.
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch-data-persistent-storage-elasticsearch-data-0 Pending gp2 8h
It looks like there is some issue with your PVC.
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch-data-persistent-storage-elasticsearch-data-0 Pending gp2 8h
As you can see your PV is also not created.I think there is an issue with your storage class.Looks like gp2 storage class is not available in your cluster.
Either run this yaml file if you are in AWS EKS
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
fsType: ext4
or simply change the storage class to standard in GCP GKE
From the docs here
The storage for a given Pod must either be provisioned by a
PersistentVolume Provisioner based on the requested storage class, or
pre-provisioned by an admin.
There should be a StorageClass which can dynamically provision the PV and mention that storageClassName in the volumeClaimTemplates or there needs to be a PV which can satisfy the PVC.
volumeClaimTemplates:
- metadata:
name: elasticsearch-data-persistent-storage
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "standard"
resources:
requests:
storage: 10Gi
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
namespace: default
name: elasticsearch-data
labels:
app: elasticsearch
role: data
spec:
serviceName: "elasticsearch-data"
replicas: 1
template:
metadata:
labels:
app: elasticsearch-data
role: data
spec:
containers:
- name: elasticsearch-data
image: docker.elastic.co/elasticsearch/elasticsearch:7.3.0
env:
- name: CLUSTER_NAME
value: elasticsearch
- name: NODE_NAME
value: elasticsearch-data
- name: NODE_LIST
value: elasticsearch-master,elasticsearch-data,elasticsearch-client
- name: MASTER_NODES
value: elasticsearch-master
- name: "ES_JAVA_OPTS"
value: "-Xms300m -Xmx300m"
ports:
- containerPort: 9300
name: transport
volumeMounts:
- name: config
mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
readOnly: true
subPath: elasticsearch.yml
- name: elasticsearch-data-persistent-storage
mountPath: /data/db
volumes:
- name: config
configMap:
name: elasticsearch-data-config
initContainers:
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: elasticsearch-data-persistent-storage
annotations:
volume.beta.kubernetes.io/storage-class: "standard"
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
---
This worked for me. Like Avinash said I simply changed the storage class to standard in GCP GKE

kubernetes Pod's readinessProbe errored but endpoint not removed from Service

I'm running Spinnaker on Kubernetes 1.10.111. One of the Spinnaker services is a Pod running a service called Clouddriver. This Pod was running fine, but then the readinessProbe started erroring continuously. Kubernetes docs say
readinessProbe: Indicates whether the Container is ready to service requests. If the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod.
— https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes
But this Pod's IP is still in the Service's endpoints. Why?
Clouddriver Pod YAML
kubectl -n spinnaker-test get pods spin-clouddriver-5559d44484-mp8q9 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: spotify.backend-service
creationTimestamp: 2019-02-15T20:46:38Z
generateName: spin-clouddriver-5559d44484-
labels:
app: spin
app.kubernetes.io/managed-by: halyard
app.kubernetes.io/name: clouddriver
app.kubernetes.io/part-of: spinnaker
app.kubernetes.io/version: 1.12.1
cluster: spin-clouddriver
pod-template-hash: "1115800040"
name: spin-clouddriver-5559d44484-mp8q9
namespace: spinnaker-test
ownerReferences:
- apiVersion: extensions/v1beta1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: spin-clouddriver-5559d44484
uid: ce79561c-3161-11e9-acdf-42010a800082
resourceVersion: "53541277"
selfLink: /api/v1/namespaces/spinnaker-test/pods/spin-clouddriver-5559d44484-mp8q9
uid: caa66d7c-3162-11e9-acdf-42010a800082
spec:
containers:
- env:
- name: JAVA_OPTS
value: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
- name: SPRING_PROFILES_ACTIVE
value: local
image: gcr.io/spinnaker-marketplace/clouddriver:4.3.1-20190130095322
imagePullPolicy: IfNotPresent
lifecycle: {}
name: clouddriver
ports:
- containerPort: 7002
protocol: TCP
readinessProbe:
exec:
command:
- wget
- --no-check-certificate
- --spider
- -q
- http://localhost:7002/health
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "20"
memory: 5000Mi
requests:
cpu: "20"
memory: 5000Mi
securityContext:
allowPrivilegeEscalation: false
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/spinnaker/config
name: spin-clouddriver-files-1952526246
- mountPath: /home/halyard/.hal/k8s-spinnaker/staging/dependencies
name: spin-clouddriver-files-1757773194
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-w2lt5
readOnly: true
dnsPolicy: ClusterFirst
nodeName: gke-production-us-ce-terraform-201812-d63606d6-9vq9
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 720
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: spin-clouddriver-files-1952526246
secret:
defaultMode: 420
secretName: spin-clouddriver-files-1952526246
- name: spin-clouddriver-files-1757773194
secret:
defaultMode: 420
secretName: spin-clouddriver-files-1757773194
- name: default-token-w2lt5
secret:
defaultMode: 420
secretName: default-token-w2lt5
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2019-02-15T20:46:38Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2019-02-15T20:53:40Z
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2019-02-15T20:46:38Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://3509b48511b1ea7bc97812cb82831c559d9410cb9eaaa26b4f492d881603fb31
image: gcr.io/spinnaker-marketplace/clouddriver:4.3.1-20190130095322
imageID: docker-pullable://gcr.io/spinnaker-marketplace/clouddriver#sha256:466228b97b8c4a61a0270c53ae4c397eb04bc3661bc4f1ee9ef4d5fce70d187d
lastState: {}
name: clouddriver
ready: true
restartCount: 0
state:
running:
startedAt: 2019-02-15T20:47:26Z
hostIP: 10.178.32.98
phase: Running
podIP: 10.179.34.24
qosClass: Guaranteed
startTime: 2019-02-15T20:46:38Z
Describing the Pod shows the readinessProbe has been continuously erroring for over a day.
kubectl -n spinnaker-test describe pods spin-clouddriver-5559d44484-mp8q9
Name: spin-clouddriver-5559d44484-mp8q9
Namespace: spinnaker-test
Node: gke-production-us-ce-terraform-201812-d63606d6-9vq9/10.178.32.98
Start Time: Fri, 15 Feb 2019 15:46:38 -0500
Labels: app=spin
app.kubernetes.io/managed-by=halyard
app.kubernetes.io/name=clouddriver
app.kubernetes.io/part-of=spinnaker
app.kubernetes.io/version=1.12.1
cluster=spin-clouddriver
pod-template-hash=1115800040
Annotations: kubernetes.io/psp=spotify.backend-service
Status: Running
IP: 10.179.34.24
Controlled By: ReplicaSet/spin-clouddriver-5559d44484
Containers:
clouddriver:
Container ID: docker://3509b48511b1ea7bc97812cb82831c559d9410cb9eaaa26b4f492d881603fb31
Image: gcr.io/spinnaker-marketplace/clouddriver:4.3.1-20190130095322
Image ID: docker-pullable://gcr.io/spinnaker-marketplace/clouddriver#sha256:466228b97b8c4a61a0270c53ae4c397eb04bc3661bc4f1ee9ef4d5fce70d187d
Port: 7002/TCP
Host Port: 0/TCP
State: Running
Started: Fri, 15 Feb 2019 15:47:26 -0500
Ready: True
Restart Count: 0
Limits:
cpu: 20
memory: 5000Mi
Requests:
cpu: 20
memory: 5000Mi
Readiness: exec [wget --no-check-certificate --spider -q http://localhost:7002/health] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
JAVA_OPTS: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
SPRING_PROFILES_ACTIVE: local
Mounts:
/home/halyard/.hal/k8s-spinnaker/staging/dependencies from spin-clouddriver-files-1757773194 (rw)
/opt/spinnaker/config from spin-clouddriver-files-1952526246 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-w2lt5 (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
spin-clouddriver-files-1952526246:
Type: Secret (a volume populated by a Secret)
SecretName: spin-clouddriver-files-1952526246
Optional: false
spin-clouddriver-files-1757773194:
Type: Secret (a volume populated by a Secret)
SecretName: spin-clouddriver-files-1757773194
Optional: false
default-token-w2lt5:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-w2lt5
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 3m (x321 over 1d) kubelet, gke-production-us-ce-terraform-201812-d63606d6-9vq9 Readiness probe errored: rpc error: code = DeadlineExceeded desc = context deadline exceeded
But Service still has the Pod's IP of 10.179.34.24 in its Endpoints.
kubectl -n spinnaker-test describe services spin-clouddriver
Name: spin-clouddriver
Namespace: spinnaker-test
Labels: app=spin
cluster=spin-clouddriver
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"spin","cluster":"spin-clouddriver"},"name":"spin-clouddriver","namesp...
Selector: app=spin,cluster=spin-clouddriver
Type: ClusterIP
IP: 10.178.65.100
Port: <unset> 7002/TCP
TargetPort: 7002/TCP
Endpoints: 10.179.34.24:7002
Session Affinity: None
Events: <none>
kubectl -n spinnaker-test describe endpoints spin-clouddriver
Name: spin-clouddriver
Namespace: spinnaker-test
Labels: app=spin
cluster=spin-clouddriver
Annotations: <none>
Subsets:
Addresses: 10.179.34.24
NotReadyAddresses: <none>
Ports:
Name Port Protocol
---- ---- --------
<unset> 7002 TCP
Events: <none>
footnotes
GKE 1.10.11-gke.1 to be exact, but the fact that it's GKE shouldn't matter.
A probe by the kubelet can end in one of three states:
successful
failed (command returned a non-0 exit code)
errored (command did not return before the timeout elapsed, the command does not exist inside the container, etc)
Here is the code (in 1.10.11) where the event probe errored is recorded. Note that err != nil.
Here is the code that calls the above function - when err != nil (the probe returned an error), the result is discarded.
Only probes that fail will actually cause the pod's ready state to be changed.