eureka pod turn to pending state when running a period of time in kubernetes cluster - kubernetes

Question

I am deploy a eureka pod in kubernetes cluster(v1.15.2),today the pod turn to be pending state and the actual state is running.Other service could not access to eureka, the eureka icon to indicate pod status shows:this pod is in a pending state.This is my stateful deploy yaml:
{
"kind": "StatefulSet",
"apiVersion": "apps/v1beta2",
"metadata": {
"name": "eureka",
"namespace": "dabai-fat",
"selfLink": "/apis/apps/v1beta2/namespaces/dabai-fat/statefulsets/eureka",
"uid": "92eefc3d-4601-4ebc-9414-8437f9934461",
"resourceVersion": "20195760",
"generation": 21,
"creationTimestamp": "2020-02-01T16:55:54Z",
"labels": {
"app": "eureka"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "eureka"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "eureka"
}
},
"spec": {
"containers": [
{
"name": "eureka",
"image": "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0",
"ports": [
{
"name": "server",
"containerPort": 8761,
"protocol": "TCP"
},
{
"name": "management",
"containerPort": 8081,
"protocol": "TCP"
}
],
"env": [
{
"name": "APP_NAME",
"value": "eureka"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "APP_OPTS",
"value": " --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/"
},
{
"name": "APOLLO_META",
"valueFrom": {
"configMapKeyRef": {
"name": "fat-config",
"key": "apollo.meta"
}
}
},
{
"name": "ENV",
"valueFrom": {
"configMapKeyRef": {
"name": "fat-config",
"key": "env"
}
}
}
],
"resources": {
"limits": {
"cpu": "2",
"memory": "1Gi"
},
"requests": {
"cpu": "2",
"memory": "1Gi"
}
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 10,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"imagePullSecrets": [
{
"name": "regcred"
}
],
"schedulerName": "default-scheduler"
}
},
"serviceName": "eureka-service",
"podManagementPolicy": "Parallel",
"updateStrategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"partition": 0
}
},
"revisionHistoryLimit": 10
},
"status": {
"observedGeneration": 21,
"replicas": 1,
"readyReplicas": 1,
"currentReplicas": 1,
"updatedReplicas": 1,
"currentRevision": "eureka-5976977b7d",
"updateRevision": "eureka-5976977b7d",
"collisionCount": 0
}
}
this is the describe output of the pending state pod:
$ kubectl describe pod eureka-0
Name: eureka-0
Namespace: dabai-fat
Priority: 0
Node: uat-k8s-01/172.19.104.233
Start Time: Mon, 23 Mar 2020 18:40:11 +0800
Labels: app=eureka
controller-revision-hash=eureka-5976977b7d
statefulset.kubernetes.io/pod-name=eureka-0
Annotations: <none>
Status: Running
IP: 172.30.248.8
IPs: <none>
Controlled By: StatefulSet/eureka
Containers:
eureka:
Container ID: docker://5e5eea624e1facc9437fef739669ffeaaa5a7ab655a1297c4acb1e4fd00701ea
Image: registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0
Image ID: docker-pullable://registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka#sha256:7cd4878ae8efec32984a2b9eec623484c66ae11b9449f8306017cadefbf626ca
Ports: 8761/TCP, 8081/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Mon, 23 Mar 2020 18:40:18 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 2
memory: 1Gi
Environment:
APP_NAME: eureka
POD_NAME: eureka-0 (v1:metadata.name)
APP_OPTS: --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/
APOLLO_META: <set to the key 'apollo.meta' of config map 'fat-config'> Optional: false
ENV: <set to the key 'env' of config map 'fat-config'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xnrwt (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
default-token-xnrwt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xnrwt
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 360s
node.kubernetes.io/unreachable:NoExecute for 360s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16h default-scheduler Successfully assigned dabai-fat/eureka-0 to uat-k8s-01
Normal Pulling 16h kubelet, uat-k8s-01 Pulling image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
Normal Pulled 16h kubelet, uat-k8s-01 Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
Normal Created 16h kubelet, uat-k8s-01 Created container eureka
Normal Started 16h kubelet, uat-k8s-01 Started container eureka
how could this happen? what should I do to avoid this situation? After I restart the eureka pod,this problem disappeared,but I still want to know the reason cause this problem.

Answer 1

Sounds like a Kubernetes bug? Try to reproduce it on the current version of Kubernetes. You can also dive into the kubelet logs to see if there is anything useful on those.