I am having trouble configuring the Crunchy Data PostgreSQL operator in my Digital Ocean managed kubernetes cluster. Per their official installation/troubleshooting guide, I changed the default storage classes in the provided manifest file to do-block-storage and I've tried toggling the disable_fsgroup value, all to no avail. I'm getting the following output from running kubectl describe... on the operator pod:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> Successfully assigned pgo/postgres-operator-697fd6dbb6-n764r to test-dev-pool-35jcv
Normal Started 69s kubelet, test-dev-pool-35jcv Started container event
Normal Created 69s kubelet, test-dev-pool-35jcv Created container event
Normal Pulled 69s kubelet, test-dev-pool-35jcv Container image "registry.developers.crunchydata.com/crunchydata/pgo-event:centos7-4.5.0" already present on machine
Normal Started 68s (x2 over 69s) kubelet, test-dev-pool-35jcv Started container scheduler
Normal Created 68s (x2 over 69s) kubelet, test-dev-pool-35jcv Created container scheduler
Normal Pulled 68s (x2 over 69s) kubelet, test-dev-pool-35jcv Container image "registry.developers.crunchydata.com/crunchydata/pgo-scheduler:centos7-4.5.0" already present on machine
Normal Started 64s (x2 over 69s) kubelet, test-dev-pool-35jcv Started container operator
Normal Created 64s (x2 over 70s) kubelet, test-dev-pool-35jcv Created container operator
Normal Pulled 64s (x2 over 70s) kubelet, test-dev-pool-35jcv Container image "registry.developers.crunchydata.com/crunchydata/postgres-operator:centos7-4.5.0" already present on machine
Normal Started 64s (x2 over 70s) kubelet, test-dev-pool-35jcv Started container apiserver
Normal Created 64s (x2 over 70s) kubelet, test-dev-pool-35jcv Created container apiserver
Normal Pulled 64s (x2 over 70s) kubelet, test-dev-pool-35jcv Container image "registry.developers.crunchydata.com/crunchydata/pgo-apiserver:centos7-4.5.0" already present on machine
Warning BackOff 63s (x4 over 67s) kubelet, test-dev-pool-35jcv Back-off restarting failed container
Any ideas?
Edit: Solved! I was specifying the default storage incorrectly. The proper edits are
- name: BACKREST_STORAGE
value: "digitalocean"
- name: BACKUP_STORAGE
value: "digitalocean"
- name: PRIMARY_STORAGE
value: "digitalocean"
- name: REPLICA_STORAGE
value: "digitalocean"
- name: STORAGE5_NAME
value: "digitalocean"
- name: STORAGE5_ACCESS_MODE
value: "ReadWriteOnce"
- name: STORAGE5_SIZE
value: "1Gi"
- name: STORAGE5_TYPE
value: "dynamic"
- name: STORAGE5_CLASS
value: "do-block-storage"
See this GitHub issue for how to correctly format the file for DO.
Related
I am trying to deploy on a minimal cluster and failing
How can I tweak the configuration to make the availability green?
My Input:
My application is a spring- angular (please suggest an easy way where I can deploy both)
My docker-compose created 2 containers. I pushed them to registry (tagged)
When deploying in Workload, I added 1 container after another, and clicked Deploy. The result error is above
Is there a file I need to create - a kind of yml or yaml etc?
kubectl get pods
> NAME READY STATUS RESTARTS AGE
> nginx-1-d...7-2s6hb 0/2 CrashLoopBackOff 18 25m
> nginx-1-6..d7-7645w 0/2 CrashLoopBackOff 18 25m
> nginx-1-6...7-9qgjx 0/2 CrashLoopBackOff 18 25m
Events from describe
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m default-scheduler Successfully assigned default/nginx-1-5d...56xp4 to gke-cluster-huge-default-pool-b6..60-4rj5
Normal Pulling 17m kubelet Pulling image "eu.gcr.io/p..my/py...my_appserver#sha256:479bf3e12ee2b410d730...579b940adc8845be74956f5"
Normal Pulled 17m kubelet Successfully pulled image "eu.gcr.io/py..my/py...emy_appserver#sha256:479bf3e12ee2b4..8b99a178ee05e8579b940adc8845be74956f5" in 11.742649177s
Normal Created 15m (x5 over 17m) kubelet Created container p..my-appserver-sha256-1
Normal Started 15m (x5 over 17m) kubelet Started container p..emy-appserver-sha256-1
Normal Pulled 15m (x4 over 17m) kubelet Container image "eu.gcr.io/py...my/pya...my_appserver#sha256:479bf3e12ee2b41..e05e8579b940adc8845be74956f5" already present on machine
Warning BackOff 2m42s (x64 over 17m) kubelet Back-off restarting failed container
I am using kubernetes to deploy ozone (a sub for hdfs), and basically followed instructions from here and here (just a few steps).
First I created few pvs with hostpath to my local dir, then I slightly edited the yamls from ozone/kubernetes/example/ozone by changing nfs claim to host path claim:
volumeClaimTemplates:
- metadata:
name: data
spec:
storageClassName: manual
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 5Gi
selector:
matchLabels:
type: local
and I commented out the nodeAffinity settings in datanode-stateful.yaml since my kubernetes only had master node.
The deployment was succesful.
Then I applied the csi and pv-test as the instructions in csi protocol said, the pv (bucket in s3v) was automatically established, and pvc did bound the pv, but the test pod stopped at containerCreating.
Attaching the pv-test pod desc:
Name: ozone-csi-test-webserver-778c8c87b7-rngfk
Namespace: default
Priority: 0
Node: k8s-master/192.168.100.202
Start Time: Fri, 18 Jun 2021 14:23:54 +0800
Labels: app=ozone-csi-test-webserver
pod-template-hash=778c8c87b7
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/ozone-csi-test-webserver-778c8c87b7
Containers:
web:
Container ID:
Image: python:3.7.3-alpine3.8
Image ID:
Port: <none>
Host Port: <none>
Args:
python
-m
http.server
--directory
/www
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-gqknv (ro)
/www from webroot (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
webroot:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: ozone-csi-test-webserver
ReadOnly: false
default-token-gqknv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gqknv
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 7m7s (x58 over 122m) kubelet, k8s-master MountVolume.SetUp failed for volume "pvc-1913bd70-09fd-4eba-a459-73fe3bd397b8" : rpc error: code = Unknown desc =
Warning FailedMount 31s (x54 over 120m) kubelet, k8s-master Unable to mount volumes for pod "ozone-csi-test-webserver-778c8c87b7-rngfk_default(b1a59143-00b9-47f6-94fe-1845c29aee93)": timeout expired waiting for volumes to attach or mount for pod "default"/"ozone-csi-test-webserver-778c8c87b7-rngfk". list of unmounted volumes=[webroot]. list of unattached volumes=[webroot default-token-gqknv]
Attach events for the whole process:
7m51s Normal SuccessfulCreate statefulset/s3g create Claim data-s3g-0 Pod s3g-0 in StatefulSet s3g success
7m51s Warning FailedScheduling pod/s3g-0 pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
7m51s Normal SuccessfulCreate statefulset/scm create Pod scm-0 in StatefulSet scm successful
7m51s Normal SuccessfulCreate statefulset/om create Pod om-0 in StatefulSet om successful
7m51s Warning FailedScheduling pod/om-0 pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
7m51s Warning FailedScheduling pod/datanode-0 pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
7m51s Normal SuccessfulCreate statefulset/datanode create Pod datanode-0 in StatefulSet datanode successful
7m51s Normal SuccessfulCreate statefulset/datanode create Claim data-datanode-0 Pod datanode-0 in StatefulSet datanode success
7m51s Normal SuccessfulCreate statefulset/scm create Claim data-scm-0 Pod scm-0 in StatefulSet scm success
7m51s Normal SuccessfulCreate statefulset/s3g create Pod s3g-0 in StatefulSet s3g successful
7m51s Normal SuccessfulCreate statefulset/om create Claim data-om-0 Pod om-0 in StatefulSet om success
7m51s Warning FailedScheduling pod/scm-0 pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
7m50s Normal Scheduled pod/s3g-0 Successfully assigned default/s3g-0 to hadoop104
7m50s Normal Scheduled pod/datanode-0 Successfully assigned default/datanode-0 to hadoop103
7m50s Normal Scheduled pod/scm-0 Successfully assigned default/scm-0 to hadoop104
7m50s Normal Scheduled pod/om-0 Successfully assigned default/om-0 to hadoop103
7m49s Normal Created pod/datanode-0 Created container datanode
7m49s Normal Started pod/datanode-0 Started container datanode
7m49s Normal Pulled pod/datanode-0 Container image "apache/ozone:1.1.0" already present on machine
7m48s Normal SuccessfulCreate statefulset/datanode create Claim data-datanode-1 Pod datanode-1 in StatefulSet datanode success
7m48s Warning FailedScheduling pod/datanode-1 pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
7m48s Normal Pulled pod/scm-0 Container image "apache/ozone:1.1.0" already present on machine
7m48s Normal Created pod/scm-0 Created container init
7m48s Normal Started pod/scm-0 Started container init
7m48s Normal Pulled pod/s3g-0 Container image "apache/ozone:1.1.0" already present on machine
7m48s Normal Created pod/s3g-0 Created container s3g
7m48s Normal Started pod/s3g-0 Started container s3g
7m48s Normal SuccessfulCreate statefulset/datanode create Pod datanode-1 in StatefulSet datanode successful
7m46s Normal Scheduled pod/datanode-1 Successfully assigned default/datanode-1 to hadoop104
7m45s Normal Created pod/datanode-1 Created container datanode
7m45s Normal Pulled pod/datanode-1 Container image "apache/ozone:1.1.0" already present on machine
7m44s Normal Created pod/scm-0 Created container scm
7m44s Normal Started pod/scm-0 Started container scm
7m44s Normal Started pod/datanode-1 Started container datanode
7m44s Normal Pulled pod/scm-0 Container image "apache/ozone:1.1.0" already present on machine
7m43s Warning FailedScheduling pod/datanode-2 pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
7m43s Normal SuccessfulCreate statefulset/datanode create Pod datanode-2 in StatefulSet datanode successful
7m43s Normal SuccessfulCreate statefulset/datanode create Claim data-datanode-2 Pod datanode-2 in StatefulSet datanode success
7m42s Normal Scheduled pod/datanode-2 Successfully assigned default/datanode-2 to hadoop103
7m38s Normal Pulled pod/datanode-2 Container image "apache/ozone:1.1.0" already present on machine
7m38s Normal Created pod/datanode-2 Created container datanode
7m38s Normal Started pod/datanode-2 Started container datanode
7m23s Normal ScalingReplicaSet deployment/csi-provisioner Scaled up replica set csi-provisioner-5649bc9474 to 1
7m23s Warning FailedCreate daemonset/csi-node Error creating: pods "csi-node-" is forbidden: error looking up service account default/csi-ozone: serviceaccount "csi-ozone" not found
7m22s Normal Scheduled pod/csi-node-nbfnw Successfully assigned default/csi-node-nbfnw to hadoop104
7m22s Normal Scheduled pod/csi-provisioner-5649bc9474-n5jf2 Successfully assigned default/csi-provisioner-5649bc9474-n5jf2 to hadoop103
7m22s Normal SuccessfulCreate replicaset/csi-provisioner-5649bc9474 Created pod: csi-provisioner-5649bc9474-n5jf2
7m22s Normal Scheduled pod/csi-node-c97fz Successfully assigned default/csi-node-c97fz to hadoop103
7m22s Normal SuccessfulCreate daemonset/csi-node Created pod: csi-node-c97fz
7m22s Normal SuccessfulCreate daemonset/csi-node Created pod: csi-node-nbfnw
7m14s Normal Pulling pod/csi-node-c97fz Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
7m14s Normal Pulling pod/csi-provisioner-5649bc9474-n5jf2 Pulling image "quay.io/k8scsi/csi-provisioner:v1.0.1"
7m13s Normal Pulling pod/csi-node-nbfnw Pulling image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
6m56s Warning Unhealthy pod/om-0 Liveness probe failed: dial tcp 10.244.1.7:9862: connect: connection refused
6m56s Normal Killing pod/om-0 Container om failed liveness probe, will be restarted
6m55s Normal Created pod/om-0 Created container om
6m55s Normal Started pod/om-0 Started container om
6m55s Normal Pulled pod/om-0 Container image "apache/ozone:1.1.0" already present on machine
6m48s Normal Pulled pod/csi-provisioner-5649bc9474-n5jf2 Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.0.1"
6m48s Normal Started pod/csi-provisioner-5649bc9474-n5jf2 Started container ozone-csi
6m48s Normal Created pod/csi-provisioner-5649bc9474-n5jf2 Created container ozone-csi
6m48s Normal Pulled pod/csi-provisioner-5649bc9474-n5jf2 Container image "apache/ozone:1.1.0" already present on machine
6m48s Normal Started pod/csi-provisioner-5649bc9474-n5jf2 Started container csi-provisioner
6m48s Normal Created pod/csi-provisioner-5649bc9474-n5jf2 Created container csi-provisioner
6m45s Normal Pulled pod/csi-node-nbfnw Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
6m44s Normal Started pod/csi-node-nbfnw Started container driver-registrar
6m44s Normal Started pod/csi-node-nbfnw Started container csi-node
6m44s Normal Created pod/csi-node-nbfnw Created container csi-node
6m44s Normal Created pod/csi-node-nbfnw Created container driver-registrar
6m44s Normal Pulled pod/csi-node-nbfnw Container image "apache/ozone:1.1.0" already present on machine
6m25s Normal Pulled pod/csi-node-c97fz Successfully pulled image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2"
6m25s Normal Pulled pod/csi-node-c97fz Container image "apache/ozone:1.1.0" already present on machine
6m25s Normal Started pod/csi-node-c97fz Started container csi-node
6m25s Normal Created pod/csi-node-c97fz Created container csi-node
6m17s Normal Created pod/csi-node-c97fz Created container driver-registrar
6m17s Normal Pulled pod/csi-node-c97fz Container image "quay.io/k8scsi/csi-node-driver-registrar:v1.0.2" already present on machine
6m17s Normal Started pod/csi-node-c97fz Started container driver-registrar
6m3s Normal Provisioning persistentvolumeclaim/ozone-csi-test-webserver External provisioner is provisioning volume for claim "default/ozone-csi-test-webserver"
6m3s Normal ScalingReplicaSet deployment/ozone-csi-test-webserver Scaled up replica set ozone-csi-test-webserver-7cbdc5d65c to 1
6m3s Normal SuccessfulCreate replicaset/ozone-csi-test-webserver-7cbdc5d65c Created pod: ozone-csi-test-webserver-7cbdc5d65c-dpzhc
6m3s Normal ExternalProvisioning persistentvolumeclaim/ozone-csi-test-webserver waiting for a volume to be created, either by external provisioner "org.apache.hadoop.ozone" or manually created by system administrator
6m2s Warning FailedScheduling pod/ozone-csi-test-webserver-7cbdc5d65c-dpzhc pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
6m1s Normal ProvisioningSucceeded persistentvolumeclaim/ozone-csi-test-webserver Successfully provisioned volume pvc-cd01c58d-793f-41ce-9e12-057ade02e07c
5m59s Normal Scheduled pod/ozone-csi-test-webserver-7cbdc5d65c-dpzhc Successfully assigned default/ozone-csi-test-webserver-7cbdc5d65c-dpzhc to hadoop104
97s Warning FailedMount pod/ozone-csi-test-webserver-7cbdc5d65c-dpzhc Unable to attach or mount volumes: unmounted volumes=[webroot], unattached volumes=[webroot default-token-l9lng]: timed out waiting for the condition
94s Warning FailedMount pod/ozone-csi-test-webserver-7cbdc5d65c-dpzhc MountVolume.SetUp failed for volume "pvc-cd01c58d-793f-41ce-9e12-057ade02e07c" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = Unknown desc =
This question already has answers here:
Why is a kubernetes pod with a simple hello-world image getting a CrashLoopBackOff message
(2 answers)
Closed 1 year ago.
I keep seeing Back-off restarting failed container when try to use docker official image https://hub.docker.com/_/hello-world to create any pod\deployment. When I switch to other images like nginx everything works well.
How can I debug and fix this issue?
Below are the event logs after creating the pod with kubectl.
root#ip-10-229-68-221:~# kubectl get event --watch
LAST SEEN TYPE REASON OBJECT MESSAGE
24s Normal Scheduled pod/helloworld-656898b9bb-98vrv Successfully assigned default/helloworld-656898b9bb-98vrv to kind-lab-worker
23s Normal Pulling pod/helloworld-656898b9bb-98vrv Pulling image "hello-world:linux"
16s Normal Pulled pod/helloworld-656898b9bb-98vrv Successfully pulled image "hello-world:linux" in 7.371731633s
1s Normal Created pod/helloworld-656898b9bb-98vrv Created container hello-world
0s Normal Started pod/helloworld-656898b9bb-98vrv Started container hello-world
1s Normal Pulled pod/helloworld-656898b9bb-98vrv Container image "hello-world:linux" already present on machine
13s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
24s Normal Scheduled pod/helloworld-656898b9bb-sg6fs Successfully assigned default/helloworld-656898b9bb-sg6fs to kind-lab-worker
23s Normal Pulling pod/helloworld-656898b9bb-sg6fs Pulling image "hello-world:linux"
13s Normal Pulled pod/helloworld-656898b9bb-sg6fs Successfully pulled image "hello-world:linux" in 9.661065021s
13s Normal Created pod/helloworld-656898b9bb-sg6fs Created container hello-world
13s Normal Started pod/helloworld-656898b9bb-sg6fs Started container hello-world
13s Normal Pulled pod/helloworld-656898b9bb-sg6fs Container image "hello-world:linux" already present on machine
11s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
24s Normal Scheduled pod/helloworld-656898b9bb-vhhfm Successfully assigned default/helloworld-656898b9bb-vhhfm to kind-lab-worker
23s Normal Pulling pod/helloworld-656898b9bb-vhhfm Pulling image "hello-world:linux"
18s Normal Pulled pod/helloworld-656898b9bb-vhhfm Successfully pulled image "hello-world:linux" in 5.17232683s
3s Normal Created pod/helloworld-656898b9bb-vhhfm Created container hello-world
2s Normal Started pod/helloworld-656898b9bb-vhhfm Started container hello-world
3s Normal Pulled pod/helloworld-656898b9bb-vhhfm Container image "hello-world:linux" already present on machine
2s Warning BackOff pod/helloworld-656898b9bb-vhhfm Back-off restarting failed container
24s Normal SuccessfulCreate replicaset/helloworld-656898b9bb Created pod: helloworld-656898b9bb-vhhfm
24s Normal SuccessfulCreate replicaset/helloworld-656898b9bb Created pod: helloworld-656898b9bb-sg6fs
24s Normal SuccessfulCreate replicaset/helloworld-656898b9bb Created pod: helloworld-656898b9bb-98vrv
24s Normal ScalingReplicaSet deployment/helloworld Scaled up replica set helloworld-656898b9bb to 3
79s Normal Killing pod/nginx Stopping container nginx
0s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
0s Normal Pulled pod/helloworld-656898b9bb-sg6fs Container image "hello-world:linux" already present on machine
0s Normal Created pod/helloworld-656898b9bb-sg6fs Created container hello-world
0s Normal Started pod/helloworld-656898b9bb-sg6fs Started container hello-world
0s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-vhhfm Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
0s Normal Pulled pod/helloworld-656898b9bb-vhhfm Container image "hello-world:linux" already present on machine
0s Normal Pulled pod/helloworld-656898b9bb-98vrv Container image "hello-world:linux" already present on machine
0s Normal Created pod/helloworld-656898b9bb-vhhfm Created container hello-world
1s Normal Created pod/helloworld-656898b9bb-98vrv Created container hello-world
0s Normal Started pod/helloworld-656898b9bb-vhhfm Started container hello-world
0s Normal Started pod/helloworld-656898b9bb-98vrv Started container hello-world
0s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-vhhfm Back-off restarting failed container
0s Normal Pulled pod/helloworld-656898b9bb-sg6fs Container image "hello-world:linux" already present on machine
0s Normal Created pod/helloworld-656898b9bb-sg6fs Created container hello-world
0s Normal Started pod/helloworld-656898b9bb-sg6fs Started container hello-world
0s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-vhhfm Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-vhhfm Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
0s Normal Pulled pod/helloworld-656898b9bb-vhhfm Container image "hello-world:linux" already present on machine
0s Normal Created pod/helloworld-656898b9bb-vhhfm Created container hello-world
0s Normal Started pod/helloworld-656898b9bb-vhhfm Started container hello-world
0s Warning BackOff pod/helloworld-656898b9bb-vhhfm Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
0s Normal Pulled pod/helloworld-656898b9bb-98vrv Container image "hello-world:linux" already present on machine
0s Normal Created pod/helloworld-656898b9bb-98vrv Created container hello-world
0s Normal Started pod/helloworld-656898b9bb-98vrv Started container hello-world
0s Warning BackOff pod/helloworld-656898b9bb-98vrv Back-off restarting failed container
0s Warning BackOff pod/helloworld-656898b9bb-vhhfm Back-off restarting failed container
0s Normal Pulled pod/helloworld-656898b9bb-sg6fs Container image "hello-world:linux" already present on machine
0s Normal Created pod/helloworld-656898b9bb-sg6fs Created container hello-world
0s Normal Started pod/helloworld-656898b9bb-sg6fs Started container hello-world
0s Warning BackOff pod/helloworld-656898b9bb-sg6fs Back-off restarting failed container
hello-world image is not a long-running process: it just outputs text and stops.
Kubernetes Pod is by default expecting long-running processes, and if it stops it automatically restart the container.
This behavior is defined by the restartPolicy parameter of a pod and can have different values:
Always: always restart stopped container (the default)
OnFailure: restart containers that exited with !=0
Never: don't restart containers
So in your case you should use one of the 2 last as the container is expected to stop normally.
I want to check inside the pod the flaw I'm having inside the image but the errors below appear. As I verified in describe, the name of the container is correct. What else can I do to get the connection in the cluster?
Command: kubectl exec -it -c airflow-console -n airflow airflow-console-xxxxxxx-xxxxx bash
error: unable to upgrade connection: container not found ("airflow-console")
Command: kubectl describe pod/airflow-console-xxxxxxx-xxxxx -n airflow
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 37m default-scheduler Successfully assigned airflow/airflow-console-xxxxxxx-xxxxx to aks-test
Normal Pulling 37m kubelet, aks-test Pulling image "test.azurecr.io/airflow:2"
Normal Pulled 37m kubelet, aks-test Successfully pulled image "test.azurecr.io/airflow:2"
Warning BackOff 36m kubelet, aks-test Back-off restarting failed container
Normal Pulled 36m (x3 over 37m) kubelet, aks-test Container image "k8s.gcr.io/git-sync:v3.1.2" already present on machine
Normal Created 36m (x3 over 37m) kubelet, aks-test Created container git-sync
Normal Started 36m (x3 over 37m) kubelet, aks-test Started container git-sync
Normal Created 36m (x3 over 36m) kubelet, aks-test Created container airflow-console
Normal Pulled 36m (x2 over 36m) kubelet, aks-test Container image "test.azurecr.io/airflow:2" already present on machine
Normal Started 36m (x3 over 36m) kubelet, aks-test Started container airflow-console
Warning BackOff 2m15s (x178 over 36m) kubelet, aks-test Back-off restarting failed container
this line
Warning BackOff 2m15s (x178 over 36m) kubelet, aks-test Back-off restarting failed container
shows that your pod/container is in a failed state. This will prevent you from execution commands in the container due to it not being alive.
To learn why your pod/container is in a bad state, you should look at the logs of the failed container
kubectl logs -n airflow airflow-console-xxxxxxx-xxxxx -c airflow-console
or the logs off the previous container that failed. (sometimes it helps)
kubectl logs -n airflow airflow-console-xxxxxxx-xxxxx -c airflow-console -p
This explain the main reason why a user cannot exec into a container.
I am using Kubernetes version 1.10.
I am trying to pull an image from a local docker repo. I already have the correct secret created.
[root#node1 ~]# kubectl get secret
NAME TYPE DATA AGE
arm-docker kubernetes.io/dockerconfigjson 1 10m
Checking the secret in detail gives me the correct auth token
[root#node1 ~]# kubectl get secret arm-docker --output="jsonpath={.data.\.dockerconfigjson}" | base64 -d
{"auths":{"armdocker.rnd.se":{"username":"<MY-USERNAME>","password":"<MY-PASSWORD>","email":"<MY-EMAIL>","auth":"<CORRECT_AUTH_TOKEN>"}}}
But when I create a Pod, Im getting the following ERROR :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13s default-scheduler Successfully assigned authorization-backend-deployment-8fd5fc8d4-msxvd to node6
Normal SuccessfulMountVolume 13s kubelet, node6 MountVolume.SetUp succeeded for volume "default-token-w7vlf"
Normal BackOff 4s (x4 over 10s) kubelet, node6 Back-off pulling image "armdocker.rnd.se/proj/authorization_backend:3.6.15"
Warning Failed 4s (x4 over 10s) kubelet, node6 Error: ImagePullBackOff
Normal Pulling 1s (x2 over 12s) kubelet, node6 pulling image "armdocker.rnd.se/proj/authorization_backend:3.6.15"
Warning Failed 1s (x2 over 12s) kubelet, node6 Failed to pull image "armdocker.rnd.se/proj/authorization_backend:3.6.15": rpc error: code = Unknown desc = Error response from daemon: Get https://armdocker.rnd.se/v1/_ping: Not Found
Warning Failed 1s (x2 over 12s) kubelet, node6 Error: ErrImagePull
Why is it looking for /v1/_ping ? Can I disable this somehow ?
Im unable to understand what is the problem here.
Once defined your secret, you need to use it inside your pod (you didn't whether you used it).
kind: Pod
...
spec:
imagePullSecrets:
- name: arm-docker