I see a lot of heavy documentation online related to Kubernetes deployment but still can't find the definition of 0/0.
> $ k get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
async-handler-redis-master 1/1 1 1 211d
bbox-inference-4k-pilot-2d-boxes 0/0 0 0 148d
What exactly does it mean to be 0/0? It's deployed but not ready? Why is it not ready? How do I make this deployment READY?
It means replica of your deployment is 0. In other words you don't have any pods under this deployment so 0/0 means 0 out of 0 pod is ready.
You can;
kubectl scale deployment <deployment-name> --replicas=1
Related
I have 2 pods running on default namespace as shown below
NAMESPACE NAME READY STATUS RESTARTS AGE
default alpaca-prod 1/1 Running 0 36m
default alpaca-test 1/1 Running 0 4m26s
kube-system coredns-78fcd69978-xd7jw 1/1 Running 0 23h
But when I try to get deployments I do not see any
kubectl get deployments
No resources found in default namespace.
Can someone explain this behavior ?
I am running k8 on Minikube.
I think these are pods which were spawned without Deployment, StatefulSet or DaemonSet.
You can run pod like this using the command, e.g.:
kubectl run nginx-test --image=nginx -n default
pods created via DaemonSet usually end with -xxxxx
pods created via Deployment usually end with -xxxxxxxxxx-xxxxx
pods created via StatefulSet usually end with -0, -1 etc.
pods created without upper resource, usually have exact name as you specified e.g. nginx-test, nginx, etc.
So my guess that is a standalone Pod resource (last option)
I do not want to decrease the number of pods controlled by StatefulSet, and i think that decreasing pods is a dangerous operation in production env.
so... , is there some way ? thx ~
I'm not sure if this is what you are looking for but you can scale a StatefulSet
Use kubectl to scale StatefulSets
First, find the StatefulSet you want to scale.
kubectl get statefulsets <stateful-set-name>
Change the number of replicas of your StatefulSet:
kubectl scale statefulsets <stateful-set-name> --replicas=<new-replicas>
To show you an example, I've deployed a 2 pod StatefulSet called web:
$ kubectl get statefulsets.apps web
NAME READY AGE
web 2/2 60s
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 63s
web-1 1/1 Running 0 44s
$ kubectl describe statefulsets.apps web
Name: web
Namespace: default
CreationTimestamp: Wed, 23 Oct 2019 13:46:33 +0200
Selector: app=nginx
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"name":"web","namespace":"default"},"spec":{"replicas":2,"select...
Replicas: 824643442664 desired | 2 total
Update Strategy: RollingUpdate
Partition: 824643442984
Pods Status: 2 Running / 0 Waiting / 0 Succeeded / 0 Failed
...
Now if we do scale this StatefulSet up to 5 replicas:
$ kubectl scale statefulset web --replicas=5
statefulset.apps/web scaled
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 3m41s
web-1 1/1 Running 0 3m22s
web-2 1/1 Running 0 59s
web-3 1/1 Running 0 40s
web-4 1/1 Running 0 27s
$ kubectl get statefulsets.apps web
NAME READY AGE
web 5/5 3m56s
You do not have any downtime in already working pods.
i think that decreasing pods is a dangerous operation in production env.
I agree with you.
As Crou wrote, it is possible to do this operation with kubectl scale statefulsets <stateful-set-name> but this is an imperative operation and it is not recommended to do imperative operations in a production environment.
In a production environment it is better to use a declarative operation, e.g. have the number of replicas in a text file (e.g. stateful-set-name.yaml) and deploy them with kubectl apply -f <stateful-set-name>.yaml with this way of working, it is easy to store the yaml-files in Git so you have full control of all changes and can revert/rollback to a previous configuration. When you store the declarative files in a Git repository you can use a CICD solution e.g. Jenkins or ArgoCD to 1) validate the operation (e.g. not allow decrease) and 2) first deploy to a test-environment and see that it works, before applying the changes to the production environment.
I recommend the book (new edition) Kubernetes Up&Running 2nd ed that describes this procedure in Chapter 18 (new chapter).
Being new to the K8s, I am trying to clean up the whole namespace after running some tests on a Windows 10 machine. In short, I thought it would be as easy as running kubectl.exe delete deployment but the deployments are created back after a second and I don't know how to get rid of them. See the followings for the details of what I did:
1.kubectl get deployments,rs (to see what we already have)
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.extensions/postgresql 1 1 1 1 18m
deployment.extensions/redis 1 1 1 1 16m
NAME DESIRED CURRENT READY AGE
replicaset.extensions/postgresql-c8cb9fff6 1 1 1 18m
replicaset.extensions/redis-5678477b7c 1 1 1 16m
2. kubectl scale deployment redis --replicas=0 (Scale down the deployment)
deployment.extensions "redis" scaled
3. kubectl get deployments,rs (Check again how it looks)
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.extensions/postgresql 1 1 1 1 21m
deployment.extensions/redis 0 0 0 0 19m
NAME DESIRED CURRENT READY AGE
replicaset.extensions/postgresql-c8cb9fff6 1 1 1 21m
replicaset.extensions/redis-5678477b7c 0 0 0 19m
4. kubectl delete deployment.extensions/redis (Delete the deployment)
deployment.extensions "redis" deleted
5. kubectl get deployments,rs (Check again and see that it is back!)
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.extensions/postgresql 1 1 1 1 23m
deployment.extensions/redis 1 1 1 1 27s
NAME DESIRED CURRENT READY AGE
replicaset.extensions/postgresql-c8cb9fff6 1 1 1 23m
replicaset.extensions/redis-5678477b7c 1 1 1 27s
6. kubectl.exe get events (Looking into the events):
Among other things I can see "Scaled down replica set redis-5678477b7c to 0" and then "Scaled up replica set redis-5678477b7c to 1" which looks like it was never actually deleted but immediately scaled up again after the delete command was executed.
Not sure what I am missing but have already checked a couple of other posts like Kubernetes pod gets recreated when deleted and How to delete all resources from Kubernetes one time? but neither one worked for me.
Forgot to say that the K8s cluster is managed by the Docker Desktop.
Use kubectl delete deployment <the name of deployment >
If you need to clean up whole namespace , use kubectl delete namespace <namespace-name>
Then re-create the same namespace by kubectl create ns command , if you need the same namespace.
You can also clean up the namespace by using --all options with objects:
e.g
kubecetl delete deployment --all
kubecetl delete statefulset --all
kubectl delete pvc --all
kubectl delete secrets --all
kubectl delete service --all
and so on.
As pointed out by #David Maze, you're deleting the ReplicaSet instead of the Deployment that's managing it.
From the documentation:
You can define Deployments to create new ReplicaSets
The Deployment will automatically create and manage a ReplicaSet to control the pods. You need to delete the Deployment to erase it's managed resources.
I would like to perform rolling back a deployment in my environment.
Command:
kubectl rollout undo deployment/foo
Steps which are perform:
create pods with old configurations
delete old pods
Is there a way to not perform last step - for example - developer would like to check why init command fail and debug.
I didn't find information about that in documentation.
Yes it is possible, before doing rollout, first you need to remove labels (corresponding to replica-set controlling that pod) from unhealthy pod. This way pod won't belong anymore to the deployment and even if you do rollout, it will still be there. Example:
$kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
sleeper 1/1 1 1 47h
$kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
sleeper-d75b55fc9-87k5k 1/1 Running 0 5m46s pod-template-hash=d75b55fc9,run=sleeper
$kubectl label pod sleeper-d75b55fc9-87k5k pod-template-hash- run-
pod/sleeper-d75b55fc9-87k5k labeled
$kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
sleeper-d75b55fc9-87k5k 1/1 Running 0 6m34s <none>
sleeper-d75b55fc9-swkj9 1/1 Running 0 3s pod-template-hash=d75b55fc9,run=sleeper
So what happens here, we have a pod sleeper-d75b55fc9-87k5k which belongs to sleeper deployment, we remove all labels from it, deployment detects that pod "has gone" so it creates a new one sleeper-d75b55fc9-swkj9, but the old one is still there and ready for debugging. Only pod sleeper-d75b55fc9-swkj9 will be affected by rollout.
I have kuberentes cluster. One master and one worker.
I install metric-server for auto scaling and then i run stress test
$ kubectl run autoscale-test --image=ubuntu:16.04 --requests=cpu=1000m --command sleep 1800
deployment "autoscale-test" created
$ kubectl autoscale deployment autoscale-test --cpu-percent=25 --min=1 --max=5
deployment "autoscale-test" autoscaled
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
autoscale-test Deployment/autoscale-test 0% / 25% 1 5 1 1m
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
autoscale-test-59d66dcbf7-9fqr8 1/1 Running 0 9m
kubectl exec autoscale-test-59d66dcbf7-9fqr8 -- apt-get update
kubectl exec autoscale-test-59d66dcbf7-9fqr8 -- apt-get install stress
$ kubectl exec autoscale-test-59d66dcbf7-9fqr8 -- stress --cpu 2 --timeout 600s &
stress: info: [227] dispatching hogs: 2 cpu, 0 io, 0 vm, 0 hdd
everything works fine and the pod was auto scaled but after that the pod that was created by autoscale is still running and they do not terminate after the stress test
the hpa shows that the 0% of cpu is in use but the 5 autoscaled pod still running
#kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
autoscale-test Deployment/autoscale-test 0%/25% 1 5 5 74m
#kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default autoscale-test-8f4d84bbf-7ddjw 1/1 Running 0 61m
default autoscale-test-8f4d84bbf-bmr59 1/1 Running 0 61m
default autoscale-test-8f4d84bbf-cxt26 1/1 Running 0 61m
default autoscale-test-8f4d84bbf-x9jws 1/1 Running 0 61m
default autoscale-test-8f4d84bbf-zbhvk 1/1 Running 0 71m
I wait for an hour but nothing happen
From the documentation:
--horizontal-pod-autoscaler-downscale-delay: The value for this option is a duration that specifies how long the autoscaler has to wait before another downscale operation can be performed after the current one has completed. The default value is 5 minutes (5m0s).
Note: When tuning these parameter values, a cluster operator should be
aware of the possible consequences. If the delay (cooldown) value is
set too long, there could be complaints that the Horizontal Pod
Autoscaler is not responsive to workload changes. However, if the
delay value is set too short, the scale of the replicas set may keep
thrashing as usual.
Finally, just before HPA scales the target, the scale recommendation
is recorded. The controller considers all recommendations within a
configurable window choosing the highest recommendation from within
that window. This value can be configured using the
--horizontal-pod-autoscaler-downscale-stabilization-window flag, which defaults to 5 minutes. This means that scaledowns will occur
gradually, smoothing out the impact of rapidly fluctuating metric
values.