I have a Cassandra/Kubernetes cluster on GCP
manuchadha25#cloudshell:~ (copper-frame-262317)$ kubectl get statefulsets --all-namespaces
NAMESPACE NAME READY AGE
cass-operator cluster1-dc1-default-sts 3/3 2d9h
manuchadha25#cloudshell:~ (copper-frame-262317)$ kubectl get all -n cass-operator
NAME READY STATUS RESTARTS AGE
pod/cass-operator-5f8cdf99fc-9c5g4 1/1 Running 0 2d9h
pod/cluster1-dc1-default-sts-0 2/2 Running 0 2d9h
pod/cluster1-dc1-default-sts-1 2/2 Running 0 2d9h
pod/cluster1-dc1-default-sts-2 2/2 Running 0 2d9h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cass-operator-metrics ClusterIP 10.51.243.147 <none> 8383/TCP,8686/TCP 2d9h
service/cassandra-loadbalancer LoadBalancer 10.51.240.24 34.91.214.233 9042:30870/TCP 37h
service/cassandradatacenter-webhook-service ClusterIP 10.51.243.86 <none> 443/TCP 2d9h
service/cluster1-dc1-all-pods-service ClusterIP None <none> <none> 2d9h
service/cluster1-dc1-service ClusterIP None <none> 9042/TCP,8080/TCP 2d9h
service/cluster1-seed-service ClusterIP None <none> <none> 2d9h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cass-operator 1/1 1 1 2d9h
NAME DESIRED CURRENT READY AGE
replicaset.apps/cass-operator-5f8cdf99fc 1 1 1 2d9h
NAME READY AGE
statefulset.apps/cluster1-dc1-default-sts 3/3 2d9h
manuchadha25#cloudshell:~ (copper-frame-262317)$
I want to scale it down from 3 nodes to 2 nodes. I am tried running the following commands but both failed.
manuchadha25#cloudshell:~ (copper-frame-262317)$ kubectl scale statefulsets cluster1-dc1-default-sts --replicas=2
Error from server (NotFound): statefulsets.apps "cluster1-dc1-default-sts" not found
What is the right command to scale down cluster?
Use -n parameter to specify correct namespace where the statfulset is deployed. Without the namespace it's trying to delete from default namespace where the statfulset cluster1-dc1-default-sts does not exist.
kubectl scale statefulsets cluster1-dc1-default-sts --replicas=2 -n cass-operator
Execute command in correct namespace using -n parameter (-n cass-operator in your case)
kubectl scale statefulsets cluster1-dc1-default-sts --replicas=2 -n cass-operator
You can also change namespace for all subsequent commands using
kubectl config set-context --current --namespace=cass-operator
Related
I want to configure Kubernetes Dashboard on a remote server using this guide: https://k21academy.com/docker-kubernetes/kubernetes-dashboard/
I installed it using:
kubernetes#kubernetes1:~$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.1/aio/deploy/recommended.yaml
List service:
kubernetes#kubernetes1:~$ kubectl get all -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
pod/dashboard-metrics-scraper-64bcc67c9c-q8f7j 1/1 Running 0 71m
pod/kubernetes-dashboard-66c887f759-pq58q 1/1 Running 0 71m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/dashboard-metrics-scraper ClusterIP 10.105.143.75 <none> 8000/TCP 71m
service/kubernetes-dashboard ClusterIP 10.102.209.213 <none> 443/TCP 71m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dashboard-metrics-scraper 1/1 1 1 71m
deployment.apps/kubernetes-dashboard 1/1 1 1 71m
NAME DESIRED CURRENT READY AGE
replicaset.apps/dashboard-metrics-scraper-64bcc67c9c 1 1 1 71m
replicaset.apps/kubernetes-dashboard-66c887f759 1 1 1 71m
kubernetes#kubernetes1:~$
But when I try to edit the port according to the guide I get:
kubernetes#kubernetes1:~$ kubectl edit service/kubernetes-dashboard
Error from server (NotFound): services "kubernetes-dashboard" not found
kubernetes#kubernetes1:~$
Do you know how I can change the port?
Seems like you are looking into default or some other namespace.
you can try
kubectl edit service/kubernetes-dashboard -n kubernetes-dashboard
A nice tool for namespace switching
curl -LO https://github.com/kvaps/kubectl-use/raw/master/kubectl-use
chmod +x ./kubectl-use
sudo mv ./kubectl-use /usr/local/bin/kubectl-use
then
kubectl use kubernetes-dashboard
After this, you do not need to specify namespace -n kubernetes-dashboard in the edit command, or kubectl get pods, it will use kubernetes-dashboard as a default context.
kubectl-use
I installed kind to play around with Kubernetes.
If I use top and sort by CPU usage (key C), then I see that kube-apiserver is constantly consuming 5 to 10% CPU.
Why?
I don't have installed something up to now:
guettli#p15:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-558bd4d5db-ntg7c 1/1 Running 0 40h
kube-system coredns-558bd4d5db-sx8w9 1/1 Running 0 40h
kube-system etcd-kind-control-plane 1/1 Running 0 40h
kube-system kindnet-9zkkg 1/1 Running 0 40h
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 40h
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 40h
kube-system kube-proxy-dthwl 1/1 Running 0 40h
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 40h
local-path-storage local-path-provisioner-547f784dff-xntql 1/1 Running 0 40h
guettli#p15:~$ kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 40h
kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 40h
guettli#p15:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane,master 40h v1.21.1
guettli#p15:~$ kubectl get nodes --all-namespaces
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane,master 40h v1.21.1
I am curious. Where does the CPU usage come from? How can I investigate this?
Even in an empty cluster with just one master node, there are at least 5 components that reach out to the API server on a regular basis:
kubelet for the master node
Controller manager
Scheduler
CoreDNS
Kube proxy
This is because API Server acts as the only entry point for all components in Kubernetes to know what the cluster state should be and take action if needed.
If you are interested in the details, you could enable audit logs in the API server and get a very verbose file with all the requests being made.
How to do so is not the goal of this answer, but you can start from the apiserver documentation.
I tried to deploy nginx server using kubernetes. I was able to create deployment and thn create service. But when i gave the curl command im facing an error. Im not able to curl and open nginx webpage in browser.
Below are the commands i used and error i got.
kubectl get pods
NAME READY STATUS RESTARTS AGE
curl 1/1 Running 8 15d
curl-deployment-646445496f-59fs9 1/1 Running 7 15d
hello-5d448ffc76-cwzcl 1/1 Running 13 23d
hello-node-7567d9fdc9-ffdkx 1/1 Running 8 20d
my-nginx-5b6fb7fb46-bdzdq 0/1 ContainerCreating 0 15d
mytestwebapp 1/1 Running 10 21d
nginx-6799fc88d8-w76cb 1/1 Running 5 13d
nginx-deployment-66b6c48dd5-9mkh8 1/1 Running 12 23d
nginx-test-795d659f45-d9shx 1/1 Running 4 13d
rss-site-7b6794856f-9586w 2/2 Running 40 15d
rss-site-7b6794856f-z59vn 2/2 Running 78 21d
jit#jit-Vostro-15-3568:~$ kubectl logs webserver
Error from server (NotFound): pods "webserver" not found
jit#jit-Vostro-15-3568:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-node LoadBalancer 10.104.134.171 <pending> 8080:31733/TCP 13d
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23d
my-nginx NodePort 10.103.114.92 <none> 8080:32563/TCP,443:32397/TCP 15d
nginx NodePort 10.110.113.60 <none> 80:30985/TCP 13d
nginx-test NodePort 10.109.16.192 <none> 8080:31913/TCP 13d
jit#jit-Vostro-15-3568:~$ curl kube-worker-1:30985
curl: (6) Could not resolve host: kube-worker-1
As you can see you have pod called nginx, that indicates that you have had nginx server already deployed in pod on your cluster. You don't have pod called webserver that's why you're getting
Error from server (NotFound): pods "webserver" not found error.
Also to access nginx service try to pass curl it via ip:port:
$ curl 10.110.113.60:30985
If you point a web browser to http://IP_OF_NODE:ASSIGNED_PORT (where IP_OF_NODE is an IP address of one of your nodes and ASSIGNED_PORT is the port assigned during the create service command), you should see the NGINX Welcome page!
Take a look: nginx-app-kubernetes.
I tried the above scenario locally.
do a kubectl describe svc <svc-name>
check whether it have any end-points.
probably it doesn't have any endpoints
I have local multi machine vagrant kubernetes cluster which is created using code here.
I have created kubernetes replication controller created using kubia-rc.yaml.
vagrant#k8s-head:~$ kubectl get rc
NAME DESIRED CURRENT READY AGE
kubia 3 3 3 26h
vagrant#k8s-head:~$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubia-l28dv 1/1 Running 1 26h
kubia-vd7jf 1/1 Running 1 26h
kubia-wsv42 1/1 Running 1 26h
Then I have created the service of type LoadBalancer using this yaml here.
The output of the command is success and it displays successfully created service
vagrant#k8s-head:~$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubia ClusterIP 10.103.199.175 <none> 80/TCP 26h
kubia-loadbalancer LoadBalancer 10.107.166.22 <pending> 80:30865/TCP 25h
The output of kubia-loadbalancer is always <pending> and don't know what could the issue.
What is wrong with my setup?
I am probably missing some of the basic. kubectl logs command usage is the following:
"kubectl logs [-f] [-p] POD [-c CONTAINER] [options]"
list of my pods is the following:
ubuntu#master:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-master 1/1 Running 0 24m
kube-system kube-apiserver-master 1/1 Running 0 24m
kube-system kube-controller-manager-master 1/1 Running 0 24m
kube-system kube-discovery-982812725-3kt85 1/1 Running 0 24m
kube-system kube-dns-2247936740-kimly 3/3 Running 0 24m
kube-system kube-proxy-amd64-gwv99 1/1 Running 0 20m
kube-system kube-proxy-amd64-r08h9 1/1 Running 0 24m
kube-system kube-proxy-amd64-szl6w 1/1 Running 0 14m
kube-system kube-scheduler-master 1/1 Running 0 24m
kube-system kubernetes-dashboard-1655269645-x3uyt 1/1 Running 0 24m
kube-system weave-net-4g1g8 1/2 CrashLoopBackOff 7 14m
kube-system weave-net-8zdm3 1/2 CrashLoopBackOff 8 20m
kube-system weave-net-qm3q5 2/2 Running 0 24m
I assume POD for logs command is anything from the second "name" column above. So, I try the following commands.
ubuntu#master:~$ kubectl logs etcd-master
Error from server: pods "etcd-master" not found
ubuntu#master:~$ kubectl logs weave-net-4g1g8
Error from server: pods "weave-net-4g1g8" not found
ubuntu#master:~$ kubectl logs weave-net
Error from server: pods "weave-net" not found
ubuntu#master:~$ kubectl logs weave
Error from server: pods "weave" not found
So, what is the POD in the logs command?
I have got the same question about services as well. How to identify a SERVICE to supply into a command, for example for 'describe' command?
ubuntu#master:~$ kubectl get services --all-namespaces
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes 100.64.0.1 <none> 443/TCP 40m
kube-system kube-dns 100.64.0.10 <none> 53/UDP,53/TCP 39m
kube-system kubernetes-dashboard 100.70.83.136 <nodes> 80/TCP 39m
ubuntu#master:~$ kubectl describe service kubernetes-dashboard
Error from server: services "kubernetes-dashboard" not found
ubuntu#master:~$ kubectl describe services kubernetes-dashboard
Error from server: services "kubernetes-dashboard" not found
Also, is it normal that weave-net-8zdm3 is in CrashLoopBackOff state? It seems I have got one for each connected worker. If it is not normal, how can I fix it? I have found similar question here: kube-dns and weave-net not starting but it does not give any practical answer.
Thanks for your help!
It seems you are running your pods in a different namespace than default.
ubuntu#master:~$ kubectl get pods --all-namespaces returns your pods but ubuntu#master:~$ kubectl logs etcd-masterreturns not found. Try running kubectl logs etcd-master --all-namespaces or if you know your namespace kubectl logs etcd-mastern --namespace=mynamespace.
The same thing goes for your services.