If I run this:
kubectl get pods -n kube-system
I get this output:
NAME READY STATUS RESTARTS AGE
coredns-6fdd4f6856-6bl64 0/1 Pending 0 1h
coredns-6fdd4f6856-xgrbm 0/1 Pending 0 1h
kubernetes-dashboard-65c76f6c97-c69jg 0/1 Pending 0 13m
supposedly I need a kubernetes scheduler in order to actually launch containers? Does anyone know how to initiate a kube-scheduler?
More than a Kubernetes scheduler issue, it looks like it's more about not having enough resources on your nodes (or no nodes at all) in your cluster to schedule any workloads. You can check your nodes with:
$ kubectl get nodes
Also, you are not likely able to see any control plane resource on the kube-system namespace because you may be using managed services like EKS or GKE.
Related
I have 2 pods running on default namespace as shown below
NAMESPACE NAME READY STATUS RESTARTS AGE
default alpaca-prod 1/1 Running 0 36m
default alpaca-test 1/1 Running 0 4m26s
kube-system coredns-78fcd69978-xd7jw 1/1 Running 0 23h
But when I try to get deployments I do not see any
kubectl get deployments
No resources found in default namespace.
Can someone explain this behavior ?
I am running k8 on Minikube.
I think these are pods which were spawned without Deployment, StatefulSet or DaemonSet.
You can run pod like this using the command, e.g.:
kubectl run nginx-test --image=nginx -n default
pods created via DaemonSet usually end with -xxxxx
pods created via Deployment usually end with -xxxxxxxxxx-xxxxx
pods created via StatefulSet usually end with -0, -1 etc.
pods created without upper resource, usually have exact name as you specified e.g. nginx-test, nginx, etc.
So my guess that is a standalone Pod resource (last option)
I would like to perform rolling back a deployment in my environment.
Command:
kubectl rollout undo deployment/foo
Steps which are perform:
create pods with old configurations
delete old pods
Is there a way to not perform last step - for example - developer would like to check why init command fail and debug.
I didn't find information about that in documentation.
Yes it is possible, before doing rollout, first you need to remove labels (corresponding to replica-set controlling that pod) from unhealthy pod. This way pod won't belong anymore to the deployment and even if you do rollout, it will still be there. Example:
$kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
sleeper 1/1 1 1 47h
$kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
sleeper-d75b55fc9-87k5k 1/1 Running 0 5m46s pod-template-hash=d75b55fc9,run=sleeper
$kubectl label pod sleeper-d75b55fc9-87k5k pod-template-hash- run-
pod/sleeper-d75b55fc9-87k5k labeled
$kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
sleeper-d75b55fc9-87k5k 1/1 Running 0 6m34s <none>
sleeper-d75b55fc9-swkj9 1/1 Running 0 3s pod-template-hash=d75b55fc9,run=sleeper
So what happens here, we have a pod sleeper-d75b55fc9-87k5k which belongs to sleeper deployment, we remove all labels from it, deployment detects that pod "has gone" so it creates a new one sleeper-d75b55fc9-swkj9, but the old one is still there and ready for debugging. Only pod sleeper-d75b55fc9-swkj9 will be affected by rollout.
Is there a way to check whether a pod status is in the completed state? I have a Pod that I only wanted to use once (where init containers didn't quite serve my purpose) and want to write a check to wait for Completed status.
I am able to get it for Running, Pending, but not for Completed.
Running:
[user#sandbox gcp_kubernetes_installation]$ kubectl get pods --field-selector=status.phase=Running -n mynamespace
NAME READY STATUS RESTARTS AGE
mssql-deployment-795dfcf9f7-l2b44 1/1 Running 0 6m
data-load-pod 1/1 Running 0 5m
Pending:
[user#sandbox gcp_kubernetes_installation]$ kubectl get pods --field-selector=status.phase=Pending -n mynamespace
NAME READY STATUS RESTARTS AGE
app-deployment-0 0/1 Pending 0 5m
Completed:
[user#sandbox gcp_kubernetes_installation]$ kubectl get pod -n namespace
NAME READY STATUS RESTARTS AGE
mssql-deployment-795dfcf9f7-l2b44 1/1 Running 0 11m
data-load-data-load-pod 0/1 Completed 0 10m
app-deployment-0 0/1 Pending 0 10m
[user#sandbox gcp_kubernetes_installation]$ kubectl get pods --field-selector=status.phase=Completed -n namespace
No resources found.
I believe there may be a bug in the field-selector, but just wondering if there are any fixes or details on a workaround.
The correct status.phase for completed pods is Succeeded.
So, to filter only completed pods, you should use this:
kubectl get pod --field-selector=status.phase=Succeeded
Although, the use of bare pods is not recommended. Consider using a Job Controller:
A Job creates one or more Pods and ensures that a specified number of
them successfully terminate. As pods successfully complete, the Job
tracks the successful completions.
You can check job conditions and wait for them with this:
kubectl wait --for=condition=complete job/myjob
I am running a cluster on GKE with a single node pool. It has 3 nodes and can scale from 1 to 99 nodes. The cluster uses the nginx-ingress controller
On this cluster, I want to deploy apps. An app is scoped by a namespace and consists of 3 deployments and one ingress (defining paths to access the application from the internet). Each deployment runs a single replica of a container.
Deploying a couple of apps works fine, but deploying many apps (requiring the node pool to scale up) breaks everything:
All pods start having warnings (including those successfully deployed earlier)
kubectl get pods --namespace bcd
NAME READY STATUS RESTARTS AGE
actions-664b7d79f5-7qdkw 1/1 Unknown 1 35m
actions-664b7d79f5-v8s2m 1/1 Running 1 18m
core-85cb74f89b-ns49z 1/1 Unknown 1 35m
core-85cb74f89b-qqzfp 1/1 Running 1 18m
nlu-77899ddbf-8pd7k 1/1 Running 1 27m
All nodes becomes unready:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-clients-projects-default-pool-f9af73d4-gzwr NotReady <none> 42m v1.9.7-gke.6
gke-clients-projects-default-pool-f9af73d4-p5l2 NotReady <none> 21m v1.9.7-gke.6
gke-clients-projects-default-pool-f9af73d4-wnxc NotReady <none> 37m v1.9.7-gke.6
Deleting the namespace to remove all resources from the cluster also seems to fail as after a long while the pods remain active but still in an unknown state.
How can I safely add more apps and let the cluster autoscale?
The reason seems to be that not knowing the resources needed for each pod, the scheduler schedules them on any available node, potentially exhausting available resources and putting the Docker daemon in an inconsistent state.
The solution is to specify resources requests and limits: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container
Trying to set up the jupyterhub server on Google Kubernetes following this tutorial. Everything went through fine. But when I install jupyterhub/jupyterhub image with helm, it's always showing the pods are pending:
kubectl --namespace=jupyter-server get pod
NAME READY STATUS RESTARTS AGE
hub-6dbd4df8b8-nqvnf 0/1 Pending 0 17h
proxy-7bb666576c-fx726 0/2 Pending 0 17h
Even after 17 hours.
The helm version is 2.6.2 as suggested in the tutorial. And I am using 3 f1-micro instances in the kubernetes cluster. Are these instances too small? Thanks for any advice.
Try describing the pod, and the describing the nodes in the cluster, to get more info about why exactly they're still pending:
kubectl describe po/hub-6dbd4df8b8-nqvnf -n jupyter-server
kubectl describe po/proxy-7bb666576c-fx726 -n jupyter-server
kubectl describe nodes