Pod restart very often after adding resources - kubernetes

Question

Pod restart very often after adding resources.
Before that the resources were not added.
The pod doesn't restart at all, or maybe it only happens once or twice a day.
I'm not sure if resources affect health-check or not, so pod restarts very often.
apiVersion: apps/v1
kind: Deployment
metadata:
name: testservice-dpm
labels:
app: testservice-api
spec:
replicas: 1
selector:
matchLabels:
app: testservice-api
template:
metadata:
labels:
app: testservice-api
spec:
containers:
- name: testservice
image: testservice:v6.0.0
env:
- name: MSSQL_PORT
value: "1433"
resources:
limits:
cpu: 500m
memory: 1000Mi
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 80
name: test-p
volumeMounts:
- name: test-v
mountPath: /app/appsettings.json
subPath: appsettings.json
livenessProbe:
httpGet:
path: /api/ServiceHealth/CheckLiveness
port: 80
scheme: HTTP
initialDelaySeconds: 3
timeoutSeconds: 1
periodSeconds: 3
successThreshold: 1
failureThreshold: 1
readinessProbe:
httpGet:
path: /api/ServiceHealth/CheckReadiness
port: 80
scheme: HTTP
initialDelaySeconds: 3
timeoutSeconds: 1
periodSeconds: 3
successThreshold: 1
failureThreshold: 1
volumes:
- name: test-v
configMap:
name: testservice-config
Below are the results describe all of the testservice pods.
testservice-dpm-d7979cc69-rwxr4
(restart 7 times in 10 minutes and still Back-off restarting failed container now)
Name: testservice-dpm-d7979cc69-rwxr4
Namespace: testapi
Priority: 0
Node: workernode3/yyy.yyy.yy.yy
Start Time: Thu, 30 Dec 2021 12:48:50 +0700
Labels: app=testservice-api
pod-template-hash=d7979cc69
Annotations: kubectl.kubernetes.io/restartedAt: 2021-12-29T20:02:45Z
Status: Running
IP: xx.xxx.x.xxx
IPs:
IP: xx.xxx.x.xxx
Controlled By: ReplicaSet/testservice-dpm-d7979cc69
Containers:
testservice:
Container ID: docker://86a50f98b48bcf8bfa209a478c1127e998e36c1c7bcece71599f50feabb89834
Image: testservice:v6.0.0
Image ID: docker-pullable://testservice#sha256:57a3955d07febf4636eeda1bc6a18468aacf66e883d7f6d8d3fdcb5163724a84
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 30 Dec 2021 12:55:13 +0700
Finished: Thu, 30 Dec 2021 12:55:19 +0700
Ready: False
Restart Count: 7
Limits:
cpu: 500m
memory: 1000Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://:80/api/ServiceHealth/CheckLiveness delay=3s timeout=1s period=3s #success=1 #failure=1
Readiness: http-get http://:80/api/ServiceHealth/CheckReadiness delay=3s timeout=1s period=3s #success=1 #failure=1
Environment:
MSSQL_PORT: 1433
Mounts:
/app/appsettings.json from authen-v (rw,path="appsettings.json")
/etc/localtime from tz-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fd9bt (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
authen-v:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: testservice-config
Optional: false
tz-config:
Type: HostPath (bare host directory volume)
Path: /usr/share/zoneinfo/Asia/Bangkok
HostPathType: File
kube-api-access-fd9bt:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned testapi/testservice-dpm-d7979cc69-rwxr4 to workernode3
Warning Unhealthy 11m (x2 over 11m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": dial tcp xx.xxx.x.xxx:80: connect: connection refused
Warning Unhealthy 11m (x3 over 11m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 11m (x3 over 11m) kubelet Liveness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckLiveness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal Killing 11m (x3 over 11m) kubelet Container testservice failed liveness probe, will be restarted
Normal Created 10m (x4 over 11m) kubelet Created container testservice
Normal Started 10m (x4 over 11m) kubelet Started container testservice
Normal Pulled 10m (x4 over 11m) kubelet Container image "testservice:v6.0.0" already present on machine
Warning BackOff 80s (x51 over 11m) kubelet Back-off restarting failed container
testservice-dpm-d7979cc69-7nq28
(restart 4 times in 10 minutes and running now)
Name: testservice-dpm-d7979cc69-7nq28
Namespace: testapi
Priority: 0
Node: workernode3/yyy.yyy.yy.yy
Start Time: Thu, 30 Dec 2021 12:47:37 +0700
Labels: app=testservice-api
pod-template-hash=d7979cc69
Annotations: kubectl.kubernetes.io/restartedAt: 2021-12-29T20:02:45Z
Status: Running
IP: xx.xxx.x.xxx
IPs:
IP: xx.xxx.x.xxx
Controlled By: ReplicaSet/testservice-dpm-d7979cc69
Containers:
testservice:
Container ID: docker://03739fc1694370abda202ba56928b46fb5f3ef7545f527c2dd73764e55f725cd
Image: testservice:v6.0.0
Image ID: docker-pullable://testservice#sha256:57a3955d07febf4636eeda1bc6a18468aacf66e883d7f6d8d3fdcb5163724a84
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 30 Dec 2021 12:48:44 +0700
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 30 Dec 2021 12:48:10 +0700
Finished: Thu, 30 Dec 2021 12:48:14 +0700
Ready: True
Restart Count: 4
Limits:
cpu: 500m
memory: 1000Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://:80/api/ServiceHealth/CheckLiveness delay=3s timeout=1s period=3s #success=1 #failure=1
Readiness: http-get http://:80/api/ServiceHealth/CheckReadiness delay=3s timeout=1s period=3s #success=1 #failure=1
Environment:
MSSQL_PORT: 1433
Mounts:
/app/appsettings.json from authen-v (rw,path="appsettings.json")
/etc/localtime from tz-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-slz4b (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
authen-v:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: testservice-config
Optional: false
tz-config:
Type: HostPath (bare host directory volume)
Path: /usr/share/zoneinfo/Asia/Bangkok
HostPathType: File
kube-api-access-slz4b:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned testapi/testservice-dpm-d7979cc69-7nq28 to workernode3
Warning Unhealthy 14m (x2 over 14m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": dial tcp xx.xxx.x.xxx:80: connect: connection refused
Warning Unhealthy 14m (x3 over 14m) kubelet Readiness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckReadiness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 14m (x3 over 14m) kubelet Liveness probe failed: Get "http://xx.xxx.x.xxx:80/api/ServiceHealth/CheckLiveness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal Killing 14m (x3 over 14m) kubelet Container testservice failed liveness probe, will be restarted
Warning BackOff 14m (x2 over 14m) kubelet Back-off restarting failed container
Normal Started 14m (x4 over 14m) kubelet Started container testservice
Normal Pulled 14m (x4 over 14m) kubelet Container image "testservice:v6.0.0" already present on machine
Normal Created 14m (x4 over 14m) kubelet Created container testservice
testservice-dpm-d7979cc69-z566c
(no restart in 10 minutes and running now)
Name: testservice-dpm-d7979cc69-z566c
Namespace: testapi
Priority: 0
Node: workernode3/yyy.yyy.yy.yy
Start Time: Thu, 30 Dec 2021 12:47:30 +0700
Labels: app=testservice-api
pod-template-hash=d7979cc69
Annotations: kubectl.kubernetes.io/restartedAt: 2021-12-29T20:02:45Z
Status: Running
IP: xx.xxx.x.xxx
IPs:
IP: xx.xxx.x.xxx
Controlled By: ReplicaSet/testservice-dpm-d7979cc69
Containers:
testservice:
Container ID: docker://19c3a672cd8453e1c5526454ffb0fbdec67fa5b17d6d8166fae38930319ed247
Image: testservice:v6.0.0
Image ID: docker-pullable://testservice#sha256:57a3955d07febf4636eeda1bc6a18468aacf66e883d7f6d8d3fdcb5163724a84
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 30 Dec 2021 12:47:31 +0700
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 1000Mi
Requests:
cpu: 100m
memory: 100Mi
Liveness: http-get http://:80/api/ServiceHealth/CheckLiveness delay=3s timeout=1s period=3s #success=1 #failure=1
Readiness: http-get http://:80/api/ServiceHealth/CheckReadiness delay=3s timeout=1s period=3s #success=1 #failure=1
Environment:
MSSQL_PORT: 1433
Mounts:
/app/appsettings.json from authen-v (rw,path="appsettings.json")
/etc/localtime from tz-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-cpdnc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
authen-v:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: testservice-config
Optional: false
tz-config:
Type: HostPath (bare host directory volume)
Path: /usr/share/zoneinfo/Asia/Bangkok
HostPathType: File
kube-api-access-cpdnc:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16m default-scheduler Successfully assigned testapi/testservice-dpm-d7979cc69-z566c to workernode3
Normal Pulled 16m kubelet Container image "testservice:v6.0.0" already present on machine
Normal Created 16m kubelet Created container testservice
Normal Started 16m kubelet Started container testservice