I am following the Fedora getting started guide (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/fedora/fedora_ansible_config.md) and trying to run the pod fedoraapache. But kubectl always shows fedoraapache as pending:
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS
fedoraapache fedoraapache fedora/apache 192.168.226.144/192.168.226.144 name=fedoraapache Pending
Since it is pending, I cannot run kubectl log pod fedoraapache. So,
I instead run kubectl describe pod fedoraapache, which shows the following errors:
Fri, 20 Mar 2015 22:00:05 +0800 Fri, 20 Mar 2015 22:00:05 +0800 1 {kubelet 192.168.226.144} implicitly required container POD created Created with docker id d4877bdffd4f2a13a17d4cc93c27c1c93d5494807b39ee8a823f5d9350e404d4
Fri, 20 Mar 2015 22:00:05 +0800 Fri, 20 Mar 2015 22:00:05 +0800 1 {kubelet 192.168.226.144} failedSync Error syncing pod, skipping: API error (500): Cannot start container d4877bdffd4f2a13a17d4cc93c27c1c93d5494807b39ee8a823f5d9350e404d4: (exit status 1)
Fri, 20 Mar 2015 22:00:15 +0800 Fri, 20 Mar 2015 22:00:15 +0800 1 {kubelet 192.168.226.144} implicitly required container POD created Created with docker id 1c32b4c6e1aad0e575f6a155aebefcd5dd96857b12c47a63bfd8562fba961747
Fri, 20 Mar 2015 22:00:15 +0800 Fri, 20 Mar 2015 22:00:15 +0800 1 {kubelet 192.168.226.144} implicitly required container POD failed Failed to start with docker id 1c32b4c6e1aad0e575f6a155aebefcd5dd96857b12c47a63bfd8562fba961747 with error: API error (500): Cannot start container 1c32b4c6e1aad0e575f6a155aebefcd5dd96857b12c47a63bfd8562fba961747: (exit status 1)
Fri, 20 Mar 2015 22:00:15 +0800 Fri, 20 Mar 2015 22:00:15 +0800 1 {kubelet 192.168.226.144} failedSync Error syncing pod, skipping: API error (500): Cannot start container 1c32b4c6e1aad0e575f6a155aebefcd5dd96857b12c47a63bfd8562fba961747: (exit status 1)
Fri, 20 Mar 2015 22:00:25 +0800 Fri, 20 Mar 2015 22:00:25 +0800 1 {kubelet 192.168.226.144} failedSync Error syncing pod, skipping: API error (500): Cannot start container 8b117ee5c6bf13f0e97b895c367ce903e2a9efbd046a663c419c389d9953c55e: (exit status 1)
Fri, 20 Mar 2015 22:00:25 +0800 Fri, 20 Mar 2015 22:00:25 +0800 1 {kubelet 192.168.226.144} implicitly required container POD created Created with docker id 8b117ee5c6bf13f0e97b895c367ce903e2a9efbd046a663c419c389d9953c55e
Fri, 20 Mar 2015 22:00:25 +0800 Fri, 20 Mar 2015 22:00:25 +0800 1 {kubelet 192.168.226.144} implicitly required container POD failed Failed to start with docker id 8b117ee5c6bf13f0e97b895c367ce903e2a9efbd046a663c419c389d9953c55e with error: API error (500): Cannot start container 8b117ee5c6bf13f0e97b895c367ce903e2a9efbd046a663c419c389d9953c55e: (exit status 1)
Fri, 20 Mar 2015 22:00:35 +0800 Fri, 20 Mar 2015 22:00:35 +0800 1 {kubelet 192.168.226.144} implicitly required container POD failed Failed to start with docker id 4b463040842b6a45db2ab154652fd2a27550dbd2e1a897c98473cd0b66d2d614 with error: API error (500): Cannot start container 4b463040842b6a45db2ab154652fd2a27550dbd2e1a897c98473cd0b66d2d614: (exit status 1)
Fri, 20 Mar 2015 22:00:35 +0800 Fri, 20 Mar 2015 22:00:35 +0800 1 {kubelet 192.168.226.144} implicitly required container POD created Created with docker id 4b463040842b6a45db2ab154652fd2a27550dbd2e1a897c98473cd0b66d2d614
Fri, 20 Mar 2015 21:42:35 +0800 Fri, 20 Mar 2015 22:00:35 +0800 109 {kubelet 192.168.226.144} implicitly required container POD pulled Successfully pulled image "kubernetes/pause:latest"
Fri, 20 Mar 2015 22:00:35 +0800 Fri, 20 Mar 2015 22:00:35 +0800 1 {kubelet 192.168.226.144} failedSync Error syncing pod, skipping: API error (500): Cannot start container 4b463040842b6a45db2ab154652fd2a27550dbd2e1a897c98473cd0b66d2d614: (exit status 1)
There are several reasons container can fail to start:
the container command itself fails and exits -> check your docker image and start up script to make sure it works.
Use
sudo docker ps -a to find the offending container and
sudo docker logs <container> to check for failure inside the container
a dependency is not there: that happens for example when one tries to mount a volume that is not present, for example Secrets that are not created yet.
--> make sure the dependent volumes are created.
Kubelet is unable to start the container we use for holding the network namespace. Some things to try are:
Can you manually pull and run gcr.io/google_containers/pause:0.8.0? (This is the image used for the network namespace container at head right now.)
As mentioned already, /var/log/kubelet.log should have more detail; but log location is distro-dependent, so check https://github.com/GoogleCloudPlatform/kubernetes/wiki/Debugging-FAQ#checking-logs.
Step one is to describe the pod and see the problems:
$ kubectl describe pod <pod Name>
or
If you use master node, then you can configure the master node to run pods with:
$ kubectl taint nodes --all node-role.kubernetes.io/master-
Check if is Kubelet is running on your machine. I came across this problem one time and discovered that Kubelet was not running, which explains why the pod status was stuck in "pending". Kubelet runs as a systemd service in my environment, so if that is also the case for you then the following command will help you check its status:
systemctl status kubelet
Related
After creating a simple nginx deployment, my pod status shows as "PENDING". When I run kubectl get pods command, I get the following:
NAME READY STATUS RESTARTS AGE
nginx-deployment-6b474476c4-dq26w 0/1 Pending 0 50m
nginx-deployment-6b474476c4-wjblx 0/1 Pending 0 50m
If I check on my node health, I get:
Taints: node.kubernetes.io/disk-pressure:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: kubernetes-master
AcquireTime: <unset>
RenewTime: Wed, 05 Aug 2020 12:43:57 +0530
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Wed, 05 Aug 2020 09:12:31 +0530 Wed, 05 Aug 2020 09:12:31 +0530 CalicoIsUp Calico is running on this node
MemoryPressure False Wed, 05 Aug 2020 12:43:36 +0530 Tue, 04 Aug 2020 23:01:43 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure True Wed, 05 Aug 2020 12:43:36 +0530 Tue, 04 Aug 2020 23:02:06 +0530 KubeletHasDiskPressure kubelet has disk pressure
PIDPressure False Wed, 05 Aug 2020 12:43:36 +0530 Tue, 04 Aug 2020 23:01:43 +0530 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 05 Aug 2020 12:43:36 +0530 Tue, 04 Aug 2020 23:02:06 +0530 KubeletReady kubelet is posting ready status. AppArmor enabled
You can remove the taint for disk pressure using below command but ideally you need to investigate why kubelet is reporting disk pressure . The node may be out of disk space.
kubectl taint nodes <nodename> node.kubernetes.io/disk-pressure-
This will get you out of pending state of the nginx pods.
#manjeet,
What's the out put of 'df -kh' on the node?
Find the disk/partiion/pv that has pressure. Increase it. Then restart kubelet. Then remove the taint. Things should work.
Question
node-problem-detector is mentioned in Monitor Node Health documentation if K8S. How do we use it if it is not in GCE? Does it feed information to Dashboard or provide API metrics?
"This tool aims to make various node problems visible to the upstream layers in cluster management stack. It is a daemon which runs on each node, detects node problems and reports them to apiserver."
Err Ok but... What does that actually mean? How can I tell if it went to the api server?
What does the before and after look like? Knowing that would help me understand what it's doing.
Before installing Node Problem Detector I see:
Bash# kubectl describe node ip-10-40-22-166.ec2.internal | grep -i condition -A 20 | grep Ready -B 20
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 20 Jun 2019 12:30:05 -0400 Thu, 20 Jun 2019 12:30:05 -0400 WeaveIsUp Weave pod has set this
OutOfDisk False Thu, 20 Jun 2019 18:27:39 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 20 Jun 2019 18:27:39 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 20 Jun 2019 18:27:39 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 20 Jun 2019 18:27:39 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 20 Jun 2019 18:27:39 -0400 Thu, 20 Jun 2019 12:30:14 -0400 KubeletReady kubelet is posting ready status
After installing Node Problem Detector I see:
Bash# helm upgrade --install npd stable/node-problem-detector -f node-problem-detector.values.yaml
Bash# kubectl rollout status daemonset npd-node-problem-detector #(wait for up)
Bash# kubectl describe node ip-10-40-22-166.ec2.internal | grep -i condition -A 20 | grep Ready -B 20
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
DockerDaemon False Thu, 20 Jun 2019 22:06:17 -0400 Thu, 20 Jun 2019 22:04:14 -0400 DockerDaemonHealthy Docker daemon is healthy
EBSHealth False Thu, 20 Jun 2019 22:06:17 -0400 Thu, 20 Jun 2019 22:04:14 -0400 NoVolumeErrors Volumes are attaching successfully
KernelDeadlock False Thu, 20 Jun 2019 22:06:17 -0400 Thu, 20 Jun 2019 22:04:14 -0400 KernelHasNoDeadlock kernel has no deadlock
ReadonlyFilesystem False Thu, 20 Jun 2019 22:06:17 -0400 Thu, 20 Jun 2019 22:04:14 -0400 FilesystemIsNotReadOnly Filesystem is not read-only
NetworkUnavailable False Thu, 20 Jun 2019 12:30:05 -0400 Thu, 20 Jun 2019 12:30:05 -0400 WeaveIsUp Weave pod has set this
OutOfDisk False Thu, 20 Jun 2019 22:07:10 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasSufficientDisk kubelet has sufficient disk space available
MemoryPressure False Thu, 20 Jun 2019 22:07:10 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 20 Jun 2019 22:07:10 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 20 Jun 2019 22:07:10 -0400 Thu, 20 Jun 2019 12:29:44 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 20 Jun 2019 22:07:10 -0400 Thu, 20 Jun 2019 12:30:14 -0400 KubeletReady kubelet is posting ready status
Note I asked for help coming up with a way to see this for all nodes, Kenna Ofoegbu came up with this super useful and readable gem:
zsh# nodes=$(kubectl get nodes | sed '1d' | awk '{print $1}') && for node in $nodes; do; kubectl describe node | sed -n '/Conditions/,/Ready/p' ; done
Bash# (same command, gives errors)
Ok so now I know what Node Problem Detector does but... what good is adding a condition to the node, how do I use the condition to do something useful?
Question: How to use Kubernetes Node Problem Detector?
Use Case #1: Auto heal borked nodes
Step 1.) Install Node Problem Detector, so it can attach new condition metadata to nodes.
Step 2.) Leverage Planetlabs/draino to cordon and drain nodes with bad conditions.
Step 3.) Leverage https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler to auto heal. (When the node is cordon and drained it'll be marked unscheduleable, this will trigger a new node to be provisioned, and then the bad node's resource utilization will be super low which will cause the bad node to get deprovisioned)
Source: https://github.com/kubernetes/node-problem-detector#remedy-systems
Use Case #2: Surface the unhealthy node event so that it can be detected by Kubernetes, and then injested into your monitoring stack so you have an auditable historic record that the event occurred and when.
These unhealthy node events are logged somewhere on the host node, but usually, the host node is generating so much noisy/useless log data that these events aren't usually collected by default.
Node Problem Detector knows where to look for these events on the host node and filters out the noise when it sees the signal of a negative outcome it'll post it to its pod log, which isn't noisy.
The pod log is likely getting ingested into an ELK and Prometheus Operator stack, where it can be detected, alerted on, stored, and graphed.
Also, note that nothing is stopping you from implementing both use cases.
Update, added a snippet of node-problem-detector.helm-values.yaml file per request in comment:
log_monitors:
#https://github.com/kubernetes/node-problem-detector/tree/master/config contains the full list, you can exec into the pod and ls /config/ to see these as well.
- /config/abrt-adaptor.json #Adds ABRT Node Events (ABRT: automatic bug reporting tool), exceptions will show up under "kubectl describe node $NODENAME | grep Events -A 20"
- /config/kernel-monitor.json #Adds 2 new Node Health Condition Checks "KernelDeadlock" and "ReadonlyFilesystem"
- /config/docker-monitor.json #Adds new Node Health Condition Check "DockerDaemon" (Checks if Docker is unhealthy as a result of corrupt image)
# - /config/docker-monitor-filelog.json #Error: "/var/log/docker.log: no such file or directory", doesn't exist on pod, I think you'd have to mount node hostpath to get it to work, gain doesn't sound worth effort.
# - /config/kernel-monitor-filelog.json #Should add to existing Node Health Check "KernelDeadlock", more thorough detection, but silently fails in NPD pod logs for me.
custom_plugin_monitors: #[]
# Someone said all *-counter plugins are custom plugins, if you put them under log_monitors, you'll get #Error: "Failed to unmarshal configuration file "/config/kernel-monitor-counter.json""
- /config/kernel-monitor-counter.json #Adds new Node Health Condition Check "FrequentUnregisteredNetDevice"
- /config/docker-monitor-counter.json #Adds new Node Health Condition Check "CorruptDockerOverlay2"
- /config/systemd-monitor-counter.json #Adds 3 new Node Health Condition Checks "FrequentKubeletRestart", "FrequentDockerRestart", and "FrequentContainerdRestart"
Considering node-problem-detector is a Kubernetes addon, you would need to install that addon on your own Kubernetes server.
A Kubernetes CLuster has an addon-manager that will use it.
Do you mean:how to install it?
kubectl create -f https://github.com/kubernetes/node-problem-detector.yaml
I've created a Kubernetes cluster in Google Compute Engine using cluster/kube-up.sh. How can I find out what Linux image GCE used to create the virtual machines? I've logged into some nodes using SSH and the usual commands (uname -a etc) don't tell me.
The default config file at kubernetes/cluster/gce/config-default.sh doesn't seem to offer any clues.
It uses something called Google Container VM image. Check out the blogpost announcing it here:
https://cloudplatform.googleblog.com/2016/09/introducing-Google-Container-VM-Image.html
There are two simple ways to look at it
In the Kubernetes GUI based dashboard, click on the nodes
From command line of the kubernetes master node use kubectl describe
pods/{pod-name}
(Make sure to select the correct namespace, if you are using any.)
Here is a sample output, please look into the "image" label of the output
kubectl describe pods/fedoraapache
Name: fedoraapache
Namespace: default
Image(s): fedora/apache
Node: 127.0.0.1/127.0.0.1
Labels: name=fedoraapache
Status: Running
Reason:
Message:
IP: 172.17.0.2
Replication Controllers: <none>
Containers:
fedoraapache:
Image: fedora/apache
State: Running
Started: Thu, 06 Aug 2015 03:38:37 -0400
Ready: True
Restart Count: 0
Conditions:
Type Status
Ready True
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Thu, 06 Aug 2015 03:38:35 -0400 Thu, 06 Aug 2015 03:38:35 -0400 1 {scheduler } scheduled Successfully assigned fedoraapache to 127.0.0.1
Thu, 06 Aug 2015 03:38:35 -0400 Thu, 06 Aug 2015 03:38:35 -0400 1 {kubelet 127.0.0.1} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Thu, 06 Aug 2015 03:38:36 -0400 Thu, 06 Aug 2015 03:38:36 -0400 1 {kubelet 127.0.0.1} implicitly required container POD created Created with docker id 98aeb13c657b
Thu, 06 Aug 2015 03:38:36 -0400 Thu, 06 Aug 2015 03:38:36 -0400 1 {kubelet 127.0.0.1} implicitly required container POD started Started with docker id 98aeb13c657b
Thu, 06 Aug 2015 03:38:37 -0400 Thu, 06 Aug 2015 03:38:37 -0400 1 {kubelet 127.0.0.1} spec.containers{fedoraapache} created Created with docker id debe7fe1ff4f
Thu, 06 Aug 2015 03:38:37 -0400 Thu, 06 Aug 2015 03:38:37 -0400 1 {kubelet 127.0.0.1} spec.containers{fedoraapache} started Started with docker id debe7fe1ff4f
I have some trouble mounting an nfs volume inside a docker container.
I have one minion(labcr3) and one pod(httpd-php).
Below you can find all relevant details.
Can you help me ?
Thanks and Regards
Phisical host(minion) is able to mount the nfs volume.
[root#LABCR3 ~]# df -h /NFS
Filesystem Size Used Avail Use% Mounted on
192.168.240.1:/mnt/VOL1/nfs-ds 500G 0 500G 0% /NFS
Here the priviliged mode enable on kubernetes on minion
root#LABCR3 ~]# cat /etc/kubernetes/config | grep PRIV
KUBE_ALLOW_PRIV="--allow_privileged=true"
Describe of pod
[root#labcr1 pods]# kubectl describe pod httpd-php
Name: httpd-php
Namespace: default
Image(s): centos7-httpd-php-alive-new
Node: labcr3/10.99.1.203
Labels: cont=httpd-php-mxs,name=httpd-php
Status: Pending
Reason:
Message:
IP:
Replication Controllers:
Containers:
httpd-php-mxs:
Image: centos7-httpd-php-alive-new
State: Waiting
Reason: Image: centos7-httpd-php-alive-new is ready, container is creating
Ready: False
Restart Count: 0
Conditions:
Type Status
Ready False
Events:
FirstSeen LastSeen CountFrom SubobjectPath Reason Message
Fri, 04 Dec 2015 15:29:48 +0100 Fri, 04 Dec 2015 15:29:48 +0100 1 {scheduler } scheduled Successfully assigned httpd-php to labcr3
Fri, 04 Dec 2015 15:31:53 +0100 Fri, 04 Dec 2015 15:38:09 +0100 4 {kubelet labcr3} failedMount Unable to mount volumes for pod "httpd-php_default": exit status 32
Fri, 04 Dec 2015 15:31:53 +0100 Fri, 04 Dec 2015 15:38:09 +0100 4 {kubelet labcr3} failedSync Error syncing pod, skipping: exit status 32
Kubelet logs on minons
-- Logs begin at ven 2015-12-04 14:25:39 CET, end at ven 2015-12-04 15:34:39 CET. --
dic 04 15:33:58 LABCR3 kubelet[1423]: E1204 15:33:58.986220 1423 pod_workers.go:111] Error syncing pod 7915461d-9a93-11e5-b8eb-d4bed9b48f94, skipping:
dic 04 15:33:58 LABCR3 kubelet[1423]: E1204 15:33:58.973581 1423 kubelet.go:1190] Unable to mount volumes for pod "httpd-php_default": exit status 32;
dic 04 15:33:58 LABCR3 kubelet[1423]: Output: mount.nfs: Connection timed out
dic 04 15:33:58 LABCR3 kubelet[1423]: Mounting arguments: labsn1:/mnt/VOL1/nfs-ds /var/lib/kubelet/pods/7915461d-9a93-11e5-b8eb-d4bed9b48f94/volumes/kube
dic 04 15:33:58 LABCR3 kubelet[1423]: E1204 15:33:58.973484 1423 mount_linux.go:103] Mount failed: exit status 32
dic 04 15:31:53 LABCR3 kubelet[1423]: E1204 15:31:53.939521 1423 pod_workers.go:111] Error syncing pod 7915461d-9a93-11e5-b8eb-d4bed9b48f94, skipping:
dic 04 15:31:53 LABCR3 kubelet[1423]: E1204 15:31:53.927865 1423 kubelet.go:1190] Unable to mount volumes for pod "httpd-php_default": exit status 32;
dic 04 15:31:53 LABCR3 kubelet[1423]: Output: mount.nfs: Connection timed out
dic 04 15:31:53 LABCR3 kubelet[1423]: Mounting arguments: labsn1:/mnt/VOL1/nfs-ds /var/lib/kubelet/pods/7915461d-9a93-11e5-b8eb-d4bed9b48f94/volumes/kube
dic 04 15:31:53 LABCR3 kubelet[1423]: E1204 15:31:53.927760 1423 mount_linux.go:103] Mount failed: exit status 32
dic 04 15:21:31 LABCR3 kubelet[1423]: E1204 15:21:31.119117 1423 reflector.go:183] watch of *api.Service ended with: 401: The event in requested in
UPDATE 12/18/2015
Yes, is the first pod with nfs mount.
nfs is mounted on the minion, so is working fine and there isn't ACL applyed to the nfs
this morning I've changed the config like this:
apiVersion: v1
kind: Pod
metadata:
name: httpd-php
labels:
name: httpd-php
cont: httpd-php-mxs
spec:
containers:
- name: httpd-php-mxs
image: "centos7-httpd-php-alive-new"
command: ["/usr/sbin/httpd","-DFOREGROUND"]
imagePullPolicy: "Never"
ports:
- containerPort: 80
volumeMounts:
- mountPath: "/NFS/MAXISPORT/DOCROOT"
name: mynfs
volumes:
- name: mynfs
persistentVolumeClaim:
claimName: nfs-claim
[root#labcr1 KUBE]# cat pv/nfspv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: nfspv
spec:
capacity:
storage: 500Gi
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Recycle
nfs:
path: /mnt/VOL1/nfs-ds
server: 10.99.1.202
[root#labcr1 KUBE]# cat pv/nfspvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs-claim
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 500Gi
[root#labcr1 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
httpd-php 1/1 Running 1 29m
[root#labcr1 ~]# kubectl get pv
NAME LABELS CAPACITY ACCESSMODES STATUS CLAIM REASON
nfspv <none> 536870912000 RWX Bound default/nfs-claim
[root#labcr1 ~]# kubectl get pvc
NAME LABELS STATUS VOLUME
nfs-claim map[] Bound nfspv
[root#labcr1 ~]# kubectl describe pod httpd-php
Name: httpd-php
Namespace: default
Image(s): centos7-httpd-php-alive-new
Node: labcr3/10.99.1.203
Labels: cont=httpd-php-mxs,name=httpd-php
Status: Running
Reason:
Message:
IP: 172.17.76.2
Replication Controllers: <none>
Containers:
httpd-php-mxs:
Image: centos7-httpd-php-alive-new
State: Running
Started: Fri, 18 Dec 2015 15:12:39 +0100
Ready: True
Restart Count: 1
Conditions:
Type Status
Ready True
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Fri, 18 Dec 2015 14:44:20 +0100 Fri, 18 Dec 2015 14:44:20 +0100 1 {kubelet labcr3} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Fri, 18 Dec 2015 14:44:20 +0100 Fri, 18 Dec 2015 14:44:20 +0100 1 {scheduler } scheduled Successfully assigned httpd-php to labcr3
Fri, 18 Dec 2015 14:44:20 +0100 Fri, 18 Dec 2015 14:44:20 +0100 1 {kubelet labcr3} implicitly required container POD created Created with docker id 3b6d520e66b0
Fri, 18 Dec 2015 14:44:20 +0100 Fri, 18 Dec 2015 14:44:20 +0100 1 {kubelet labcr3} implicitly required container POD started Started with docker id 3b6d520e66b0
Fri, 18 Dec 2015 14:44:21 +0100 Fri, 18 Dec 2015 14:44:21 +0100 1 {kubelet labcr3} spec.containers{httpd-php-mxs} started Started with docker id 859ea73a6cdd
Fri, 18 Dec 2015 14:44:21 +0100 Fri, 18 Dec 2015 14:44:21 +0100 1 {kubelet labcr3} spec.containers{httpd-php-mxs} created Created with docker id 859ea73a6cdd
Fri, 18 Dec 2015 15:12:38 +0100 Fri, 18 Dec 2015 15:12:38 +0100 1 {kubelet labcr3} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Fri, 18 Dec 2015 15:12:38 +0100 Fri, 18 Dec 2015 15:12:38 +0100 1 {kubelet labcr3} implicitly required container POD created Created with docker id bdfed6fd4c97
Fri, 18 Dec 2015 15:12:38 +0100 Fri, 18 Dec 2015 15:12:38 +0100 1 {kubelet labcr3} implicitly required container POD started Started with docker id bdfed6fd4c97
Fri, 18 Dec 2015 15:12:39 +0100 Fri, 18 Dec 2015 15:12:39 +0100 1 {kubelet labcr3} spec.containers{httpd-php-mxs} created Created with docker id ab3a39784b4e
Fri, 18 Dec 2015 15:12:39 +0100 Fri, 18 Dec 2015 15:12:39 +0100 1 {kubelet labcr3} spec.containers{httpd-php-mxs} started Started with docker id ab3a39784b4e
Now the minion is up and running but inside the pods I can't see the nfs mounted....
[root#httpd-php /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-253:2-3413- f90c89a604c59b6aacbf95af649ef48eb4df829fdfa4b51eee4bfe75a6a156c3 99G 444M 93G 1% /
tmpfs 5.9G 0 5.9G 0% /dev
shm 64M 0 64M 0% /dev/shm
tmpfs 5.9G 0 5.9G 0% /sys/fs/cgroup
tmpfs 5.9G 0 5.9G 0% /run/secrets
/dev/mapper/centos_labcr2-var 248G 3.8G 244G 2% /etc/hosts
I'm trying to get a ghost blog deployed on GKE, working off of the persistent disks with WordPress tutorial. I have a working container that runs fine manually on a GKE node:
docker run -d --name my-ghost-blog -p 2368:2368 -d us.gcr.io/my_project_id/my-ghost-blog
I can also correctly create a pod using the following method from another tutorial:
kubectl run ghost --image=us.gcr.io/my_project_id/my-ghost-blog --port=2368
When I do that I can curl the blog on the internal IP from within the cluster, and get the following output from kubectl get pod:
Name: ghosty-nqgt0
Namespace: default
Image(s): us.gcr.io/my_project_id/my-ghost-blog
Node: very-long-node-name/10.240.51.18
Labels: run=ghost
Status: Running
Reason:
Message:
IP: 10.216.0.9
Replication Controllers: ghost (1/1 replicas created)
Containers:
ghosty:
Image: us.gcr.io/my_project_id/my-ghost-blog
Limits:
cpu: 100m
State: Running
Started: Fri, 04 Sep 2015 12:18:44 -0400
Ready: True
Restart Count: 0
Conditions:
Type Status
Ready True
Events:
...
The problem arises when I instead try to create the pod from a yaml file, per the Wordpress tutorial. Here's the yaml:
metadata:
name: ghost
labels:
name: ghost
spec:
containers:
- image: us.gcr.io/my_project_id/my-ghost-blog
name: ghost
env:
- name: NODE_ENV
value: production
- name: VIRTUAL_HOST
value: myghostblog.com
ports:
- containerPort: 2368
When I run kubectl create -f ghost.yaml, the pod is created, but is never ready:
> kubectl get pod ghost
NAME READY STATUS RESTARTS AGE
ghost 0/1 Running 11 3m
The pod continuously restarts, as confirmed by the output of kubectl describe pod ghost:
Name: ghost
Namespace: default
Image(s): us.gcr.io/my_project_id/my-ghost-blog
Node: very-long-node-name/10.240.51.18
Labels: name=ghost
Status: Running
Reason:
Message:
IP: 10.216.0.12
Replication Controllers: <none>
Containers:
ghost:
Image: us.gcr.io/my_project_id/my-ghost-blog
Limits:
cpu: 100m
State: Running
Started: Fri, 04 Sep 2015 14:08:20 -0400
Ready: False
Restart Count: 10
Conditions:
Type Status
Ready False
Events:
FirstSeen LastSeen Count From SubobjectPath Reason Message
Fri, 04 Sep 2015 14:03:20 -0400 Fri, 04 Sep 2015 14:03:20 -0400 1 {scheduler } scheduled Successfully assigned ghost to very-long-node-name
Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} implicitly required container POD created Created with docker id dbbc27b4d280
Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} implicitly required container POD started Started with docker id dbbc27b4d280
Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} spec.containers{ghost} created Created with docker id ceb14ba72929
Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} spec.containers{ghost} started Started with docker id ceb14ba72929
Fri, 04 Sep 2015 14:03:27 -0400 Fri, 04 Sep 2015 14:03:27 -0400 1 {kubelet very-long-node-name} implicitly required container POD pulled Pod container image "gcr.io/google_containers/pause:0.8.0" already present on machine
Fri, 04 Sep 2015 14:03:30 -0400 Fri, 04 Sep 2015 14:03:30 -0400 1 {kubelet very-long-node-name} spec.containers{ghost} started Started with docker id 0b8957fe9b61
Fri, 04 Sep 2015 14:03:30 -0400 Fri, 04 Sep 2015 14:03:30 -0400 1 {kubelet very-long-node-name} spec.containers{ghost} created Created with docker id 0b8957fe9b61
Fri, 04 Sep 2015 14:03:40 -0400 Fri, 04 Sep 2015 14:03:40 -0400 1 {kubelet very-long-node-name} spec.containers{ghost} created Created with docker id edaf0df38c01
Fri, 04 Sep 2015 14:03:40 -0400 Fri, 04 Sep 2015 14:03:40 -0400 1 {kubelet very-long-node-name} spec.containers{ghost} started Started with docker id edaf0df38c01
Fri, 04 Sep 2015 14:03:50 -0400 Fri, 04 Sep 2015 14:03:50 -0400 1 {kubelet very-long-node-name} spec.containers{ghost} started Started with docker id d33f5e5a9637
...
This cycle of created/started goes on forever, if I don't kill the pod. The only difference from the successful pod is the lack of a replication controller. I don't expect this is the problem because the tutorial mentions nothing about rc.
Why is this happening? How can I create a successful pod from config file? And where would I find more verbose logs about what is going on?
If the same docker image is working via kubectl run but not working in a pod, then something is wrong with the pod spec. Compare the full output of the pod as created from spec and as created by rc to see what differs by running kubectl get pods <name> -o yaml for both. Shot in the dark: is it possible the env vars specified in the pod spec are causing it to crash on startup?
Maybe you could use different restart Policy in the yaml file?
What you have I believe is equivalent to
- restartPolicy: Never
no replication controller. You may try to add this line to yaml and set it to Always (and this will provide you with RC), or to OnFailure.
https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/pod-states.md#restartpolicy
Container logs may be useful, with kubectl logs
Usage:
kubectl logs [-p] POD [-c CONTAINER]
http://kubernetes.io/v1.0/docs/user-guide/kubectl/kubectl_logs.html