Kubernetes pod not starting - kubernetes

I have a kubernetes cluster with 5 nodes. When I add a simple nginx pod it will be scheduled to one of the nodes but it will not start up. It will not even pull the image.
This is the nginx.yaml file:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
when I describe the pod there is one event: Successfully assigned busybox to up02 When I log in to the up02 and check to see if there are any images pulled I see it didn't get pulled so I pulled it manually (I thought maybe it needs some kick start ;) )
The pod will allways stay in the Container creating state. It's not only with this pod, the problem is with any pod I try to add.
There are some pods running on the machine which is necessary for Kubernetes to operate:
up#up01:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default busybox 0/1 ContainerCreating 0 11m
default nginx 0/1 ContainerCreating 0 22m
kube-system dummy-2088944543-n1cd5 1/1 Running 0 5d
kube-system etcd-up01 1/1 Running 0 5d
kube-system kube-apiserver-up01 1/1 Running 0 5d
kube-system kube-controller-manager-up01 1/1 Running 0 5d
kube-system kube-discovery-1769846148-xfpls 1/1 Running 0 5d
kube-system kube-dns-2924299975-5rzz8 4/4 Running 0 5d
kube-system kube-proxy-17bpl 1/1 Running 2 3d
kube-system kube-proxy-3pk63 1/1 Running 0 3d
kube-system kube-proxy-h3wrj 1/1 Running 0 5d
kube-system kube-proxy-wzqv4 1/1 Running 0 3d
kube-system kube-proxy-z3xxx 1/1 Running 0 3d
kube-system kube-scheduler-up01 1/1 Running 0 5d
kube-system kubernetes-dashboard-3203831700-3xfbd 1/1 Running 0 5d
kube-system weave-net-6c0nr 2/2 Running 0 3d
kube-system weave-net-dchhf 2/2 Running 0 5d
kube-system weave-net-hshvg 2/2 Running 4 3d
kube-system weave-net-n684c 2/2 Running 1 3d
kube-system weave-net-r5319 2/2 Running 0 3d

You can do
kubectl describe pods <pod>
to get more info on what's happening.

Can you recreate the nginx pod again in namespace kube-system?
kubectl create --namespace kube-system -f nginx.yaml
this should fix your problem.
Second, do you have proxy in your environment, take a look as well.

make sure that your namespace and service account information is correct. if you've configured your services or deployments to use a namespace or service account, that namespace needs to exist.
If you configured it to use a non - default service account then that has to exist as well, and the service account should be created after the namespace.
you shouldn't be necessarily using the kube system namespace. namespaces exist so there can be more than one of them and control the flow of traffic inside of a cluster.
you should also set potentially set permissions for your namespace. read this here.
https://kubernetes.io/docs/reference/access-authn-authz/rbac/#service-account-permissions

Related

Kiali Dashboard Not able to fetch the k8 namespaces application

I have successfully installed istio and deployed some sample app and application is up and running.
root#master:~# kubectl get pod
NAME READY STATUS RESTARTS AGE
mydata-v1-847cd777c4-kc495 2/2 Running 0 39m
mydata-v2-65bbf55977-j67xp 2/2 Running 0 39m
myweb-66dc56ccd6-5g64b 2/2 Running 0 40m
NAME READY STATUS RESTARTS AGE
grafana-784c89f4cf-cxpcz 1/1 Running 0 15d
istio-egressgateway-bd477794-qv7n8 1/1 Running 0 15d
istio-ingressgateway-79df7c789f-qlqcf 1/1 Running 0 15d
istiod-6dc55bbdd-t5klg 1/1 Running 0 15d
jaeger-7f78b6fb65-xhz8j 1/1 Running 0 15d
kiali-dc84967d9-99lwv 1/1 Running 1 13d
prometheus-7bfddb8dbf-nd4gn 2/2 Running 35 15d
Next i changed kiali dashboard cluster IP to Nodeport to access the dash brad from the browser
kubectl patch svc kiali -n istio-system --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":30010}]'
Finally i can able to access the dashboard using node port with my host Ip http://machineip_port/ and could see my k8 namespaces without any apps please find the attached screen shot
could you please help me someone last one week i am running into this issue.
The problem is that
"Namespaces that do not exist at the time of install but are created
later in the future will not be accessible by Kiali". Resource.
So, first, keep in mind you should not edit kiali's ConfigMap, but only Kiali's Custom Resource Definition(CRD), which is used by Kiali Operator.
Run kubectl edit kiali kiali in the namespace you have the CRD available.
Then add the following under spec:
spec:
deployment:
accessible_namespaces:
- ["**"]
This will give Kiali access to all current namespaces and to any you'll create in the future.

Kubernetes coredns pods stuck in Pending status. Cannot start the dashboard

I am building a Kubernetes cluster following this tutorial, and I have troubles to access the Kubernetes dashboard. I already created another question about it that you can see here, but while digging up into my cluster, I think that the problem might be somewhere else and that's why I create a new question.
I start my master, by running the following commands:
> kubeadm reset
> kubeadm init --apiserver-advertise-address=[MASTER_IP] > file.txt
> tail -2 file.txt > join.sh # I keep this file for later
> kubectl apply -f https://git.io/weave-kube/
> kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-kb2zq 0/1 Pending 0 2m46s
coredns-fb8b8dccf-nnc5n 0/1 Pending 0 2m46s
etcd-kubemaster 1/1 Running 0 93s
kube-apiserver-kubemaster 1/1 Running 0 93s
kube-controller-manager-kubemaster 1/1 Running 0 113s
kube-proxy-lxhvs 1/1 Running 0 2m46s
kube-scheduler-kubemaster 1/1 Running 0 93s
Here we can see that I have two coredns pods stuck in Pending state forever, and when I run the command :
> kubectl -n kube-system describe pod coredns-fb8b8dccf-kb2zq
I can see in the Events part the following Warning :
Failed Scheduling : 0/1 nodes are available 1 node(s) had taints that the pod didn't tolerate.
Since it is a Warning and not and Error, and that as a Kubernetes newbie, taints does not mean much to me, I tried to connect a node to the master (using the previously saved command) :
> cat join.sh
kubeadm join [MASTER_IP]:6443 --token [TOKEN] \
--discovery-token-ca-cert-hash sha256:[ANOTHER_TOKEN]
> ssh [USER]#[WORKER_IP] 'bash' < join.sh
This node has joined the cluster.
On the master, I check that the node is connected:
> kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubemaster NotReady master 13m v1.14.1
kubeslave1 NotReady <none> 31s v1.14.1
And I check my pods :
> kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-kb2zq 0/1 Pending 0 14m
coredns-fb8b8dccf-nnc5n 0/1 Pending 0 14m
etcd-kubemaster 1/1 Running 0 13m
kube-apiserver-kubemaster 1/1 Running 0 13m
kube-controller-manager-kubemaster 1/1 Running 0 13m
kube-proxy-lxhvs 1/1 Running 0 14m
kube-proxy-xllx4 0/1 ContainerCreating 0 2m16s
kube-scheduler-kubemaster 1/1 Running 0 13m
We can see that another kube-proxy pod have been created and is stuck in ContainerCreating status.
And when I am doing a describe again :
kubectl -n kube-system describe pod kube-proxy-xllx4
I can see in the Events part multiple identical Warnings :
Failed create pod sandbox : rpx error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Get https://k8s.gcr.io/v1/_ping: dial tcp: lookup k8s.gcr.io on [::1]:53 read up [::1]43133->[::1]:53: read: connection refused
Here are my repositories :
docker image ls
REPOSITORY TAG
k8s.gcr.io/kube-proxy v1.14.1
k8s.gcr.io/kube-apiserver v1.14.1
k8s.gcr.io/kube-controller-manager v1.14.1
k8s.gcr.io/kube-scheduler v1.14.1
k8s.gcr.io/coredns 1.3.1
k8s.gcr.io/etcd 3.3.10
k8s.gcr.io/pause 3.1
And so, for the dashboard part, I tried to start it with the command
> kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard.yaml
But the dashboard pod is stuck in Pending state.
kubectl -n kube-system get pod
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-kb2zq 0/1 Pending 0 40m
coredns-fb8b8dccf-nnc5n 0/1 Pending 0 40m
etcd-kubemaster 1/1 Running 0 38m
kube-apiserver-kubemaster 1/1 Running 0 38m
kube-controller-manager-kubemaster 1/1 Running 0 39m
kube-proxy-lxhvs 1/1 Running 0 40m
kube-proxy-xllx4 0/1 ContainerCreating 0 27m
kube-scheduler-kubemaster 1/1 Running 0 38m
kubernetes-dashboard-5f7b999d65-qn8qn 1/1 Pending 0 8s
So, event though my problem originaly was that I cannot access to my dashboard, I guess that the real problem is deeper thant that.
I know that I just put a lot of information here, but I am a k8s beginner and I am completely lost on this.
There is an issue I experienced with coredns pods stuck in a pending mode when setting up your own cluster; which I resolve by adding pod network.
Looks like because there is no Network Addon installed, the nodes are taint as not-ready. Installing the Addon would remove the taints and the Pods will be able to schedule. In my case adding flannel fixed the issue.
EDIT: There is a note about this in the official k8s documentation - Create cluster with kubeadm:
The network must be deployed before any applications. Also, CoreDNS
will not start up before a network is installed. kubeadm only
supports Container Network Interface (CNI) based networks (and does
not support kubenet).
Actually it is the opposite of a deep or serious issue. This is a trivial issue. Always you see a pod stuck on Pending state, it means the scheduler is having a hard time to schedule the pod; mostly because there are no enough resources on the node.
In your case it is a taint that has the node, and your pod doesn't have the toleration. What you have to do is to describe the node and get the taint:
kubectl describe node | grep -i taints
Note: you might have more then one taint. So you might want to do kubectl describe no NODE since with grep you will only see one taint.
Once you get the taint, that will be something like hello=world:NoSchedule; which means key=value:effect, you will have to add a toleration section in your Deployment. This is an example Deployment so you can see how it should look like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 10
strategy:
type: Recreate
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: nginx
name: nginx
ports:
- containerPort: 80
name: http
tolerations:
- effect: NoExecute #NoSchedule, PreferNoSchedule
key: node
operator: Equal
value: not-ready
tolerationSeconds: 3600
As you can see there is the toleration section in the yaml. So, if I would have a node with node=not-ready:NoExecute taint, no pod would be able to be scheduled on that node, unless would have this toleration.
Also you can remove the taint, if you don need it. To remove a taint you would describe the node, get the key of the taint and do:
kubectl taint node NODE key-
Hope it makes sense. Just add this section to your deployment, and it will work.
Set up the flannel network tool.
Running commands:
$ sysctl net.bridge.bridge-nf-call-iptables=1
$ kubectl apply -f
https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml

Deploy GitLab with Helm. Nginx-ingress pods can't start

Call install:
helm install --name gitlab1 -f values.yaml gitlab/gitlab-omnibus
I see Pods can't start.
And I see error:no service with name nginx-ingress/default-http-backend found: services "default-http-backend" is forbidden: User "system:serviceaccount:nginx-ingress:default" cannot get services in the namespace "nginx-ingress"
I think about ABAC/RBAC... But what doing with this...
Logs from nginx pod:
# kubectl logs nginx-ndxhn --namespace nginx-ingress
[dumb-init] Unable to detach from controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] Child spawned with PID 7.
[dumb-init] Unable to attach to controlling tty (errno=25 Inappropriate ioctl for device).
[dumb-init] setsid complete.
I0530 21:30:23.232676 7 launch.go:105] &{NGINX 0.9.0-beta.11 git-a3131c5 https://github.com/kubernetes/ingress}
I0530 21:30:23.232749 7 launch.go:108] Watching for ingress class: nginx
I0530 21:30:23.233708 7 launch.go:262] Creating API server client for https://10.233.0.1:443
I0530 21:30:23.234080 7 nginx.go:182] starting NGINX process...
F0530 21:30:23.251587 7 launch.go:122] no service with name nginx-ingress/default-http-backend found: services "default-http-backend" is forbidden: User "system:serviceaccount:nginx-ingress:default" cannot get services in the namespace "nginx-ingress"
[dumb-init] Received signal 17.
[dumb-init] A child with PID 7 exited with exit status 255.
[dumb-init] Forwarded signal 15 to children.
[dumb-init] Child exited with status 255. Goodbye.
# kubectl get svc -w --namespace nginx-ingress nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 10.233.25.0 <pending> 80:32048/TCP,443:31430/TCP,22:31636/TCP 9m
# kubectl describe svc --namespace nginx-ingress nginx
Name: nginx
Namespace: nginx-ingress
Labels: <none>
Annotations: service.beta.kubernetes.io/external-traffic=OnlyLocal
Selector: app=nginx
Type: LoadBalancer
IP: 10.233.25.0
IP: 1.1.1.1
Port: http 80/TCP
TargetPort: 80/TCP
NodePort: http 32048/TCP
Endpoints:
Port: https 443/TCP
TargetPort: 443/TCP
NodePort: https 31430/TCP
Endpoints:
Port: git 22/TCP
TargetPort: 22/TCP
NodePort: git 31636/TCP
Endpoints:
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default gitlab1-gitlab-75576c4589-lnf56 0/1 Running 2 11m
default gitlab1-gitlab-postgresql-f66555d65-nqvqx 1/1 Running 0 11m
default gitlab1-gitlab-redis-58cf598657-ksptm 1/1 Running 0 11m
default gitlab1-gitlab-runner-55d458ccb7-g442z 0/1 CrashLoopBackOff 6 11m
default glusterfs-9cfcr 1/1 Running 0 1d
default glusterfs-k422g 1/1 Running 0 1d
default glusterfs-tjtvq 1/1 Running 0 1d
default heketi-75dcfb7d44-thxpm 1/1 Running 0 1d
default nginx-nginx-ingress-controller-775b5b9c6d-hhvlr 1/1 Running 0 2h
default nginx-nginx-ingress-default-backend-7bb66746b9-mzgcb 1/1 Running 0 2h
default nginx-pod1 1/1 Running 0 1d
kube-lego kube-lego-58c9f5788d-pdfb5 1/1 Running 0 11m
kube-system calico-node-hq2v7 1/1 Running 3 2d
kube-system calico-node-z4nts 1/1 Running 3 2d
kube-system calico-node-z9r9v 1/1 Running 4 2d
kube-system kube-apiserver-k8s-m1.me 1/1 Running 4 2d
kube-system kube-apiserver-k8s-m2.me 1/1 Running 5 1d
kube-system kube-apiserver-k8s-m3.me 1/1 Running 3 2d
kube-system kube-controller-manager-k8s-m1.me 1/1 Running 4 2d
kube-system kube-controller-manager-k8s-m2.me 1/1 Running 4 1d
kube-system kube-controller-manager-k8s-m3.me 1/1 Running 3 2d
kube-system kube-dns-7bd4d5fbb6-r2rnf 3/3 Running 9 2d
kube-system kube-dns-7bd4d5fbb6-zffvn 3/3 Running 9 2d
kube-system kube-proxy-k8s-m1.me 1/1 Running 3 2d
kube-system kube-proxy-k8s-m2.me 1/1 Running 3 1d
kube-system kube-proxy-k8s-m3.me 1/1 Running 3 2d
kube-system kube-scheduler-k8s-m1.me 1/1 Running 4 2d
kube-system kube-scheduler-k8s-m2.me 1/1 Running 4 1d
kube-system kube-scheduler-k8s-m3.me 1/1 Running 4 2d
kube-system kubedns-autoscaler-679b8b455-pp7jd 1/1 Running 3 2d
kube-system kubernetes-dashboard-55fdfd74b4-6z8qp 1/1 Running 0 1d
kube-system tiller-deploy-75b7d95f5c-8cmxh 1/1 Running 0 1d
nginx-ingress default-http-backend-6679b97b47-w6cx7 1/1 Running 0 11m
nginx-ingress nginx-ndxhn 0/1 CrashLoopBackOff 6 11m
nginx-ingress nginx-nk2jg 0/1 CrashLoopBackOff 6 11m
nginx-ingress nginx-rz7xj 0/1 CrashLoopBackOff 6 11m
Logs on runner:
# kubectl logs gitlab1-gitlab-runner-55d458ccb7-g442z
+ cp /scripts/config.toml /etc/gitlab-runner/
+ /entrypoint register --non-interactive --executor kubernetes
Running in system-mode.
ERROR: Registering runner... failed runner=tQtCbx5U status=couldn't execute POST against http://gitlab1-gitlab.default:8005/api/v4/runners: Post http://gitlab1-gitlab.default:8005/api/v4/runners: dial tcp 10.233.7.205:8005: i/o timeout
PANIC: Failed to register this runner. Perhaps you are having network problems
PVC is fine
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gitlab1-gitlab-config-storage Bound pvc-c957bd23-644f-11e8-8f10-4ccc6a60fcbe 1Gi RWO gluster-heketi 13m
gitlab1-gitlab-postgresql-storage Bound pvc-c964e7d0-644f-11e8-8f10-4ccc6a60fcbe 30Gi RWO gluster-heketi 13m
gitlab1-gitlab-redis-storage Bound pvc-c96f9146-644f-11e8-8f10-4ccc6a60fcbe 5Gi RWO gluster-heketi 13m
gitlab1-gitlab-registry-storage Bound pvc-c959d377-644f-11e8-8f10-4ccc6a60fcbe 30Gi RWO gluster-heketi 13m
gitlab1-gitlab-storage Bound pvc-c9611ab1-644f-11e8-8f10-4ccc6a60fcbe 30Gi RWO gluster-heketi 13m
gluster1 Bound pvc-922b5dc0-6372-11e8-8f10-4ccc6a60fcbe 5Gi RWO gluster-heketi 1d
# kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
I think about ABAC/RBAC... But what doing with this...
You are correct, and the error message explains exactly what is wrong. There are two paths forward: you can fix the Role and RoleBinding for the default ServiceAccount in the nginx-ingress namespace, or you can switch the Deployment to use a ServiceAccount other than default in order to assign that Deployment the specific permissions required. I recommend the latter, but the former may be less typing.
The rough version of the Role and RoleBinding lives in the nginx-ingress repo but may need to be adapted for your needs, including updating the apiVersion away from v1beta1
After that change has taken place, you'll need to delete the nginx-ingress Pods in order for them to pick up their new Role and conduct whatever initialization tasks nginx does during startup.
Separately, you will for sure want to fix this business:
Post http://gitlab1-gitlab.default:8005/api/v4/runners: dial tcp 10.233.7.205:8005: i/o timeout
I can't offer more concrete actions without knowing more about your CNI setup and the state of affairs of the actual GitLab Pod, but an I/O timeout is certainly a very weird error to get for in cluster communication.

kubectl logs not working after creating cluster with kubeadm

I followed the guide on "Using kubeadm to Create a Cluster" but I am not able to view logs using kubectl:
root#o1:~# kubectl logs -n kube-system etcd-o1
Error from server: Get https://149.156.11.4:10250/containerLogs/kube-system/etcd-o1/etcd: tls: first record does not look like a TLS handshake
The above IP address is the cloud frontend address not the address of the VM which probably causes the problem. Some other kubectl cmds seem to work:
root#o1:~# kubectl cluster-info
Kubernetes master is running at https://10.6.16.88:6443
KubeDNS is running at https://10.6.16.88:6443/api/v1/namespaces/kube-system/services/kube-dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root#o1:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-o1 1/1 Running 0 3h
kube-system kube-apiserver-o1 1/1 Running 0 3h
kube-system kube-controller-manager-o1 1/1 Running 0 3h
kube-system kube-dns-545bc4bfd4-mhbfb 3/3 Running 0 3h
kube-system kube-flannel-ds-lw87h 2/2 Running 0 1h
kube-system kube-flannel-ds-rkqxg 2/2 Running 2 1h
kube-system kube-proxy-hnhfs 1/1 Running 0 3h
kube-system kube-proxy-qql4r 1/1 Running 0 1h
kube-system kube-scheduler-o1 1/1 Running 0 3h
Please help.
Maybe change the address in the $HOME/admin.conf.

Kubernetes Autoscaling not recognizing Heapster

I have a kubernetes cluster that I'm trying to build from scratch without using their build scripts. Pretty much everything is working except for autoscaling. For some reason the control-manager can't find or doesn't know heapster is running.
I have a ticket open but it seems no responses
https://github.com/kubernetes/kubernetes/issues/18652
UPDATE: the issue has been answered on Github
Things I have setup.
Here is a list of all the pods currently
[root#kube-master test] [dev] # kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default my-nginx-8kmlz 1/1 Running 0 11h
default my-nginx-z8cxb 1/1 Running 0 11h
kube-system heapster-v10-vdc1v 3/3 Running 0 11h
kube-system kube-apiserver-10.122.0.20 1/1 Running 0 4d
kube-system kube-controller-manager-10.122.0.20 1/1 Running 1 9h
kube-system kube-dns-6iw3a 4/4 Running 0 4d
kube-system kube-proxy-10.122.0.20 1/1 Running 0 3d
kube-system kube-proxy-10.122.42.163 1/1 Running 0 4d
kube-system kube-proxy-10.122.43.138 1/1 Running 1 4d
kube-system kube-scheduler-10.122.0.20 1/1 Running 1 4d
So heapster is running against my proxy and I can access
http://10.122.0.20:8080/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/default/pods/my-nginx-8kmlz/stats
It returns stats about the pod.
I'm really not sure what i am missing.
Here is what the output of a autoscale looks like
[root#kube-master test] [dev] # kubectl get hpa
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
my-nginx ReplicationController/my-nginx/scale 80% <waiting> 1 5 22h
In my controller logs the only thing I really see is
W1224 18:27:43.425126 1 horizontal.go:185] Failed to reconcile my-nginx: failed to compute desired number of replicas based on CPU utilization for ReplicationController/default/my-nginx: failed to get cpu utilization: failed to get CPU consumption and request: some pods do not have request for cpu
you need to assign cpu request / limit for pods in deployment file example
resources:
requests:
cpu: "100m"
limits:
cpu: "250m"
Some time this happen because resource metrics not enabled.
you can verify with blow command:
kubectl top pod -n <namespace>
if you are getting pods then metrics is enabled:
https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/#resource-metrics-pipeline