Kubernetes pod fail/restart simulation - kubernetes

Question

We have a data visualization server hosted in Kubernetes pods. The dashboards in that data viz are displayed in the browser of different monitors/terminals for near-real time operational reporting. Sometimes the pods fail, and when they come alive again, the browser redirects to Single Sign-On page instead of going to the dashboard the URL is originally configured to.
The server are hosted in I would presume a replica set. There are two pods that exist as far as I can tell.
I was granted privilege on using kubectl to solve this problem, but still quite new with the whole Kubernetes thing. Using kubectl, how do I simulate pod failure/restart for testing purposes? Since the pods are in duplicate, shutting one of them will only redirect the traffic to the other pod. How to make both pods fail/restart at the same time? (I guess doing kubectl delete pod on both pods will do, but I want to make sure k8s will respawn the pods automatically, and not delete them forever).

Answer 1

If I understand the use case correctly, you might want to use kubectl scale command. This will give you the flexibility to make the replica count to zero to N by running a simple kubectl scale command. See examples. Also, if you are using deployment, you can just do the kubectl delete pod, the deployment controller will spawn a new one to satisfy the replica count.
kubectl scale deployment/<DEPLOYMENT-NAME> --replicas=<DESIRED-NUMBER-OF-REPLICA>
short example:
kubectl scale deployment/deployment-web --replicas=0
deployment.apps/deployment-web scaled
Long Example:
// create a deployment called, deployment-web with two replicas.
kubectl create deployment deployment-web --image=nginx --replicas 2
deployment.apps/deployment-web created
// verify that both replicas are up
kubectl get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
deployment-web 2/2 2 2 13s
// expose the deployment with a service [OPTIONAL-STEP, ONLY FOR EXPLANATION]
kubectl expose deployment deployment-web --port 80
service/deployment-web exposed
//verify that the service is created
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
deployment-web ClusterIP 10.233.24.174 <none> 80/TCP 5s
// dump the list of end-points for that service, there would be one for each replica. Notice the two IPs in the 2nd column.
kubectl get ep
NAME ENDPOINTS AGE
deployment-web 10.233.111.6:80,10.233.115.9:80 12s
//scale down to 1 replica for the deployment
kubectl scale --current-replicas=2 --replicas=1 deployment/deployment-web
deployment.apps/deployment-web scaled
// Notice the endpoint is reduced from 2 to 1.
kubectl get ep
NAME ENDPOINTS AGE
deployment-web 10.233.115.9:80 43s
// also note that there is only one pod remaining
kubectl get pod
NAME READY STATUS RESTARTS AGE
deployment-web-64c769b44-qh2qf 1/1 Running 0 105s
// scale down to zero replica
kubectl scale --current-replicas=1 --replicas=0 deployment/deployment-web
deployment.apps/deployment-web scaled
// The endpoint list is empty
kubectl get ep
NAME ENDPOINTS AGE
deployment-web <none> 9m4s
//Also, both pods are gone
kubectl get pod
No resources found in default namespace.
// When you are done with testing. restore the replicas
kubectl scale --current-replicas=0 --replicas=2 deployment/deployment-web
deployment.apps/deployment-web scaled
//endpoints and pods are restored back
kubectl get ep
NAME ENDPOINTS AGE
deployment-web 10.233.111.8:80,10.233.115.11:80 10m
foo-svc 10.233.115.6:80 50m
kubernetes 192.168.22.9:6443 6d23h
kubectl get pod -l app=deployment-web
NAME READY STATUS RESTARTS AGE
deployment-web-64c769b44-b72k5 1/1 Running 0 8s
deployment-web-64c769b44-mt2dd 1/1 Running 0 8s