Minio Kubernetes Intstallation no memory error - kubernetes

I am trying to install the minio storage using kubernetes on my local .
Following the link , However i am facing error with no memory in all types of install ..
I am not sure how to set up the presistantVolume in my case.
https://github.com/minio/operator/blob/master/README.md
I am trying to create persistent volume so that enough memory will be available in the path i am selecting
cat pv.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
kubectl create -f pv.yaml
kubectl get sc
kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
hostpath (default) docker.io/hostpath Delete Immediate false 131m
local-storage kubernetes.io/no-provisioner Delete WaitForFirstConsumer false 56m
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-node
spec:
capacity:
storage: 10Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storage-class: local-storage
local:
path: /mnt/d/minio
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- docker-desktop
kubectl create -f pvc.yaml
error: error parsing pvc.yaml: error converting YAML to JSON: yaml: line 8: mapping values are not allowed in this context
:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
docker-desktop Ready control-plane,master 126m v1.21.2
See 'kubectl get --help' for usage.
:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-558bd4d5db-j72z4 1/1 Running 1 128m
kube-system coredns-558bd4d5db-vw98z 1/1 Running 1 128m
kube-system etcd-docker-desktop 1/1 Running 1 128m
kube-system kube-apiserver-docker-desktop 1/1 Running 1 128m
kube-system kube-controller-manager-docker-desktop 1/1 Running 1 128m
kube-system kube-proxy-tqfnr 1/1 Running 1 128m
kube-system kube-scheduler-docker-desktop 1/1 Running 1 128m
kube-system storage-provisioner 1/1 Running 2 127m
kube-system vpnkit-controller 1/1 Running 12 127m
minio-operator console-6b6cf8946c-vxcqh 1/1 Running 0 76m
minio-operator minio-operator-69fd675557-s62nl 1/1 Running 0 76m
:/$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 251G 1.9G 237G 1% /
tmpfs 6.2G 401M 5.8G 7% /mnt/wsl
tools 477G 69G 409G 15% /init
none 6.1G 0 6.1G 0% /dev
none 6.2G 12K 6.2G 1% /run
none 6.2G 0 6.2G 0% /run/lock
none 6.2G 0 6.2G 0% /run/shm
none 6.2G 0 6.2G 0% /run/user
tmpfs 6.2G 0 6.2G 0% /sys/fs/cgroup
C:\ 477G 69G 409G 15% /mnt/c
D:\ 932G 132M 932G 1% /mnt/d
/dev/sdd 251G 2.7G 236G 2% /mnt/wsl/docker-desktop-data/isocache
none 6.2G 12K 6.2G 1% /mnt/wsl/docker-desktop/shared-sockets/host-services
/dev/sdc 251G 132M 239G 1% /mnt/wsl/docker-desktop/docker-desktop-proxy
/dev/loop0 396M 396M 0 100% /mnt/wsl/docker-desktop/cli-tools
I beeilve creating a persistent volume and using that in a namesapce and using that namespace whil creating a tenant should solve this issue. But i am stuck with the error of no memory available

As per the code:
if (memReqSize < minMemReq) {
return {
error: "The requested memory size must be greater than 2Gi",
request: 0,
limit: 0,
};
}
You need 2GB of RAM per node. Since you have 4 nodes, you need 8 GB of RAM for Minio alone. It's likely that you don't have the enough RAM to run this.

Related

Getting The node was low on resource: ephemeral-storage?

I am trying to understand master/node concept depoyment in labs.playwithk8s.com https://labs.play-with-k8s.com/
I have two nodes and one master.
It has the following config memory.
node1 ~]$ kubectl describe pod myapp-7f4dffc449-qh7pk
Name: myapp-7f4dffc449-qh7pk
Namespace: default
Priority: 0
Node: node3/192.168.0.16
Start Time: Tue, 07 Feb 2023 12:31:23 +0000
Labels: app=myapp
pod-template-hash=7f4dffc449
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/myapp-7f4dffc449
Containers:
myapp:
Container ID:
Image: changan1111/newdocker:latest
Image ID:
Port: 3000/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Limits:
cpu: 500m
ephemeral-storage: 1Gi
memory: 1Gi
Requests:
cpu: 500m
ephemeral-storage: 1Gi
memory: 1Gi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-t4nf7 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-t4nf7:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-t4nf7
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 34s default-scheduler Successfully assigned default/myapp-7f4dffc449-qh7pk to node3
Normal Pulling 31s kubelet Pulling image "changan1111/newdocker:latest"
Warning Evicted 25s kubelet The node was low on resource: ephemeral-storage.
Warning ExceededGracePeriod 15s kubelet Container runtime did not kill the pod within specified grace period.
My yaml file is here: https://raw.githubusercontent.com/changan1111/UserManagement/main/kube/kube.yaml
Looks like that i am not seeing anything wrong.. but still i am seeing The node was low on resource: ephemeral-storage
How to resolve this?
Disk Usage:
overlay 10G 130M 9.9G 2% /
tmpfs 64M 0 64M 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sdb 64G 29G 36G 44% /etc/hosts
shm 64M 0 64M 0% /dev/shm
shm 64M 0 64M 0% /var/lib/docker/containers/403c120b0dd0909bd34e66d86c58fba18cd71468269e1aaa66e3244d331c3a1e/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/56dd63dad42dd26baba8610f70f1a0bd22fdaea36742c32deca3c196ce181851/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/50c4585ae8cc63de9077c1a58da67cc348c86a6643ca21a06b8998f94a2a2daf/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/6e9529ad6e6a836e77b17c713679abddf861fdc0e86946484dc2ec68a00ca2ff/mounts/shm
tmpfs 16G 12K 16G 1% /var/lib/kubelet/pods/8e56095e-b0ec-4f13-a022-d29d04897410/volumes/kubernetes.io~secret/kube-proxy-token-j7sl8
shm 64M 0 64M 0% /var/lib/docker/containers/2b84d6dfebd4ea0c379588985cd43b623004632e71d63d07a39d521ddf694e8e/mounts/shm
tmpfs 16G 12K 16G 1% /var/lib/kubelet/pods/1271ca18-97d0-48d2-9280-68eb8c57795f/volumes/kubernetes.io~secret/kube-router-token-rmpqv
shm 64M 0 64M 0% /var/lib/docker/containers/c4506095bf36356790795353862fc13b759d72af8edc0e4233341f2d3234fa02/mounts/shm
tmpfs 16G 12K 16G 1% /var/lib/kubelet/pods/39885a73-d724-4be8-a9cf-3de8756c5b0c/volumes/kubernetes.io~secret/coredns-token-ckxbw
tmpfs 16G 12K 16G 1% /var/lib/kubelet/pods/8f137411-3af6-4e44-8be4-3e4f79570531/volumes/kubernetes.io~secret/coredns-token-ckxbw
shm 64M 0 64M 0% /var/lib/docker/containers/c32431f8e77652686f58e91aff01d211a5e0fb798f664ba675715005ee2cd5b0/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/3e284dd5f9b321301647eeb42f9dd82e81eb78aadcf9db7b5a6a3419504aa0e9/mount
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m16s default-scheduler Successfully assigned default/myapp-b5856bb-4znkj to node4
Normal Pulling 3m15s kubelet Pulling image "changan1111/newdocker:latest"
Normal Pulled 83s kubelet Successfully pulled image "changan1111/newdocker:latest" in 1m51.97169753s
Normal Created 28s kubelet Created container myapp
Normal Started 27s kubelet Started container myapp
Warning Evicted 1s kubelet Pod ephemeral local storage usage exceeds the total limit of containers 500Mi.
Normal Killing 1s kubelet Stopping container myapp
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 2
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
imagePullSecrets:
- name: dockercreds
containers:
- name: myapp
image: changan1111/newdocker:latest
resources:
limits:
memory: "2Gi"
cpu: "500m"
ephemeral-storage: "2Gi"
requests:
ephemeral-storage: "1Gi"
cpu: "500m"
memory: "1Gi"
ports:
- containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
name: myapp
spec:
selector:
app: myapp
ports:
- protocol: TCP
port: 80
targetPort: 3000
nodePort: 31110
type: LoadBalancer
Worker nodes may be running out of disk space in which case you should see something like no space left on device or The node was low on resource: ephemeral-storage.
Mitigation is to specify larger disk size for node VMs during Composer environment creation.
Pod eviction and scheduling problems are side effects of Kubernetes limits and requests, usually caused by a lack of planning. See Understanding Kubernetes pod evicted and scheduling problems for more information.
Refer to the similar SO how to set a quota limits.ephemeral-storage, requests.ephemeral-storage to limit this, as otherwise any container can write any amount of storage to its node filesystem.
Warning : Pod ephemeral local storage usage exceeds the total limit of containers 500Mi.
It may be because you're putting an upper limit of ephemeral-storage usage by setting resources.limits.ephemeral-storage to 500Mi. Try removing the limits.ephemeral-storage if safe or change the value depending upon your requirement.
Also see How to determine kubernetes pod ephemeral storage request and limit and how to Avoid running out of ephemeral storage space on your Kubernetes worker Nodes for more information.

Cannot delete pod in status ContainerCreating

I have a failed pod which is not properly created. I used these steps:
kubernetes#kubernetes1:~$ cd /opt/registry
kubernetes#kubernetes1:/opt/registry$ kubectl create -f private-registry1.yaml
persistentvolume/pv1 created
kubernetes#kubernetes1:/opt/registry$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default private-repository-k8s-6ddbcd9c45-s6dfq 0/1 ContainerCreating 0 2d1h
kube-system calico-kube-controllers-58dbc876ff-dgs77 1/1 Running 4 (125m ago) 2d13h
kube-system calico-node-czmzc 1/1 Running 4 (125m ago) 2d13h
kube-system calico-node-q4lxz 1/1 Running 4 (125m ago) 2d13h
kube-system coredns-565d847f94-k94z2 1/1 Running 4 (125m ago) 2d13h
kube-system coredns-565d847f94-nt27m 1/1 Running 4 (125m ago) 2d13h
kube-system etcd-kubernetes1 1/1 Running 5 (125m ago) 2d13h
kube-system kube-apiserver-kubernetes1 1/1 Running 5 (125m ago) 2d13h
kube-system kube-controller-manager-kubernetes1 1/1 Running 5 (125m ago) 2d13h
kube-system kube-proxy-97djs 1/1 Running 5 (125m ago) 2d13h
kube-system kube-proxy-d8bzs 1/1 Running 4 (125m ago) 2d13h
kube-system kube-scheduler-kubernetes1 1/1 Running 5 (125m ago) 2d13h
As you can see the pod is stucked in status ContainerCreating. I tried to delete it:
kubernetes#kubernetes1:/opt/registry$ kubectl get deployments --all-namespaces
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default private-repository-k8s 0/1 1 0 2d2h
kube-system calico-kube-controllers 1/1 1 1 2d14h
kube-system coredns 2/2 2 2 2d14h
Delete command:
kubernetes#kubernetes1:/opt/registry$ kubectl delete -n default deployment private-repository-k8s
deployment.apps "private-repository-k8s" deleted
kubernetes#kubernetes1:/opt/registry$ kubectl get deployments --all-namespaces
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system calico-kube-controllers 1/1 1 1 2d14h
kube-system coredns 2/2 2 2 2d14h
kubernetes#kubernetes1:/opt/registry$ kubectl create -f private-registry1.yaml
Error from server (AlreadyExists): error when creating "private-registry1.yaml": persistentvolumes "pv1" already exists
private-registry1.yaml configuration:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv1
spec:
capacity:
storage: 256Mi # specify your own size
volumeMode: Filesystem
persistentVolumeReclaimPolicy: Retain
local:
path: /opt/registry # can be any path
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions: # specify the node label which maps to your control-plane node.
- key: kubernetes1
operator: In
values:
- controlplane-1
accessModes:
- ReadWriteOnce # only 1 node will read/write on the path.
# - ReadWriteMany # multiple nodes will read/write on the path
Do you know how I can delete pv1?
You can delete the PV using the following two commands:
kubectl delete pv <pv_name> --grace-period=0 --force
And then deleting the finalizer using:
kubectl patch pv <pv_name> -p '{"metadata": {"finalizers": null}}'
As you have created using a file, you can also use the following command to delete the pv:
kubectl delete -f private-registry1.yaml

K8s Volume doesn't detach from host

We're using Kubernetes on-premise and it's currently running on VMWare. So far, we have been successfull in being able to provision volumes for the apps that we deploy. The problem comes if the pods - for whatever reason - switch to a different worker node. When that happens, the disk fails to mount to the second worker as it's already present on the first worker where the pod was originally running. See below:
As it stands, we have no app on either worker1 or worker2:
[root#worker01 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 200G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 199.5G 0 part
├─vg_root-lv_root 253:0 0 20G 0 lvm /
├─vg_root-lv_swap 253:1 0 2G 0 lvm
├─vg_root-lv_var 253:2 0 50G 0 lvm /var
└─vg_root-lv_k8s 253:3 0 20G 0 lvm /mnt/disks
sr0 11:0 1 1024M 0 rom
[root#worker02 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 200G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 199.5G 0 part
├─vg_root-lv_root 253:0 0 20G 0 lvm /
├─vg_root-lv_swap 253:1 0 2G 0 lvm
├─vg_root-lv_var 253:2 0 50G 0 lvm /var
└─vg_root-lv_k8s 253:3 0 20G 0 lvm /mnt/disks
sr0 11:0 1 4.5G 0 rom
Next we create our PVC with the following:
[root#master01 ~]$ cat app-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: app-pvc
annotations:
volume.beta.kubernetes.io/storage-class: thin-disk
namespace: tools
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
[root#master01 ~]$ kubectl create -f app-pvc.yaml
persistentvolumeclaim "app-pvc" created
This works fine as the disk is created and bound:
[root#master01 ~]$ kubectl get pvc -n tools
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
app-pvc Bound pvc-d4bf77cc-294e-11e9-9106-005056a4b1c7 10Gi RWO thin-disk 12s
[root#master01 ~]$ kubectl get pv -n tools
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-d4bf77cc-294e-11e9-9106-005056a4b1c7 10Gi RWO Delete Bound tools/app-pvc thin-disk 12s
Now we can deploy our application which creates the pod and sorts storage etc:
[centos#master01 ~]$ cat app.yaml
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: app
namespace: tools
spec:
replicas: 1
template:
metadata:
labels:
app: app
spec:
containers:
- image: sonatype/app3:latest
imagePullPolicy: IfNotPresent
name: app
ports:
- containerPort: 8081
- containerPort: 5000
volumeMounts:
- mountPath: /app-data
name: app-data-volume
securityContext:
fsGroup: 2000
volumes:
- name: app-data-volume
persistentVolumeClaim:
claimName: app-pvc
---
apiVersion: v1
kind: Service
metadata:
name: app-service
namespace: tools
spec:
type: NodePort
ports:
- port: 80
targetPort: 8081
protocol: TCP
name: http
- port: 5000
targetPort: 5000
protocol: TCP
name: docker
selector:
app: app
[centos#master01 ~]$ kubectl create -f app.yaml
deployment.extensions "app" created
service "app-service" created
This deploys fine:
[centos#master01 ~]$ kubectl get pods -n tools
NAME READY STATUS RESTARTS AGE
app-6588cf4b87-wvwg2 0/1 ContainerCreating 0 6s
[centos#neb-k8s02-master01 ~]$ kubectl describe pod app-6588cf4b87-wvwg2 -n tools
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18s default-scheduler Successfully assigned nexus-6588cf4b87-wvwg2 to neb-k8s02-worker01
Normal SuccessfulMountVolume 18s kubelet, worker01 MountVolume.SetUp succeeded for volume "default-token-7cv62"
Normal SuccessfulAttachVolume 15s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-d4bf77cc-294e-11e9-9106-005056a4b1c7"
Normal SuccessfulMountVolume 7s kubelet, worker01 MountVolume.SetUp succeeded for volume "pvc-d4bf77cc-294e-11e9-9106-005056a4b1c7"
Normal Pulled 7s kubelet, worker01 Container image "acme/app:latest" already present on machine
Normal Created 7s kubelet, worker01 Created container
Normal Started 6s kubelet, worker01 Started container
We can also see the disk has been created and mounted in VMWare for Worker01 and not for Worker02:
[root#worker01 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 200G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 199.5G 0 part
├─vg_root-lv_root 253:0 0 20G 0 lvm /
├─vg_root-lv_swap 253:1 0 2G 0 lvm
├─vg_root-lv_var 253:2 0 50G 0 lvm /var
└─vg_root-lv_k8s 253:3 0 20G 0 lvm /mnt/disks
sdb 8:16 0 10G 0 disk /var/lib/kubelet/pods/1e55ad6a-294f-11e9-9175-005056a47f18/volumes/kubernetes.io~vsphere-volume/pvc-d4bf77cc-294e-11e9-9106-005056a4b1c7
sr0 11:0 1 1024M 0 rom
[root#worker02 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 200G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 199.5G 0 part
├─vg_root-lv_root 253:0 0 20G 0 lvm /
├─vg_root-lv_swap 253:1 0 2G 0 lvm
├─vg_root-lv_var 253:2 0 50G 0 lvm /var
└─vg_root-lv_k8s 253:3 0 20G 0 lvm /mnt/disks
sr0 11:0 1 4.5G 0 rom
If Worker01 falls over then Worker02 kicks in and we can see the disk being attached to the other node:
[root#worker02 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 200G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 199.5G 0 part
├─vg_root-lv_root 253:0 0 20G 0 lvm /
├─vg_root-lv_swap 253:1 0 2G 0 lvm
├─vg_root-lv_var 253:2 0 50G 0 lvm /var
└─vg_root-lv_k8s 253:3 0 20G 0 lvm /mnt/disks
sdb 8:16 0 10G 0 disk /var/lib/kubelet/pods/a0695030-2950-11e9-9175-005056a47f18/volumes/kubernetes.io~vsphere-volume/pvc-d4bf77cc-294e-11e9-9106-005056a4b1c7
sr0 11:0 1 4.5G 0 rom
However, seeing as though the disk is now attached to Worker01 and Worker02, Worker01 will no longer start citing the following error in vCenter:
Cannot open the disk '/vmfs/volumes/5ba35d3b-21568577-efd4-469e3c301eaa/kubevols/kubernetes-dynamic-pvc-e55ad6a-294f-11e9-9175-005056a47f18.vmdk' or one of the snapshot disks it depends on.
This error occurs because (I assume) Worker02 has access to the disk and is reading/writing from/to it. Shouldn't Kubernetes detach the disk from nodes that do not need it if it's been attached to another node. How can we go about fixing this issue? If a pods moves to another host due to node failure then we have to manually detach the disk and then start the other worker manually.
Any and all help appreciated.
First, I'll assume your running in tree vsphere disks.
Second, in this case (and more so, with CSI) kubernetes doesn't have control over all volume operations. The VMWare functionality for managing attachment and detachment of a disk is implemented in the volume plugin which you are using. Kubernetes doesn't strictly control all volume attachment/detachment semantics as a generic function.
To see the in-tree implementation details, check out:
https://kubernetes.io/docs/concepts/storage/volumes/#vspherevolume
Overall i think the way you are doing failover is going to mean that when your worker1 pod dies, worker2 can schedule. At that point, worker1 should not be able to grab the same PVC, and it should not schedule until the worker2 pod dies.
However if worker1 is scheduling, it means that Vsphere is trying to (erroneously) let worker1 start, and the kubelet is failing.
There is a chance that this is a bug in the VMWare driver in that it will bind a persistent volume even though it is not ready to.
To further elaborate, details about how worker2 is being launched may be helped. Is it a separate replication controller ? or is it running outside of kubernetes? If the latter, then the volumes wont be managed the same way, and you cant use a the same PVC as the locking mechanism.

kubernetes metrics server don't start

I try to connect in the dashboard of kubernetes.
I have the latest version of kubernetes v1.12 with kubeadm , in a server.
I download from github the metrics-server and run:
Kubctl create -f deploy/1.8+
but i get this error
kube-system metrics-server-5cbbc84f8c-tjfxd 0/1 Pending 0 12m
with out log to debug
error: the server doesn't have a resource type "logs"
I don't want to install heapster because is DEPRECATED.
UPDATE
Hello, and thanks.
i run the taint command i get:
error: at least one taint update is required
and the command
kubectl describe deployment metrics-server -n kube-system
i get this output:
Name: metrics-server
Namespace: kube-system
CreationTimestamp: Thu, 18 Oct 2018 14:34:42 +0000
Labels: k8s-app=metrics-server
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata": {"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"metrics-...
Selector: k8s-app=metrics-server
Replicas: 1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: k8s-app=metrics-server
Service Account: metrics-server
Containers:
metrics-server:
Image: k8s.gcr.io/metrics-server-amd64:v0.3.1
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/tmp from tmp-dir (rw)
Volumes:
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: metrics-server-5cbbc84f8c (1/1 replicas created)
Events: <none>
Command:
kubectl get nodes
The output for this is just the IP of the node, and nothing special.
Any ideas, or what to do to work the dashboard for kubernetes.
I suppose you are trying setup metrics-server on your master node.
If you issue kubectl describe deployment metrics-server -n kube-system I believe you will see something like this:
Name: metrics-server Namespace:
kube-system CreationTimestamp: Thu, 18 Oct 2018 15:57:34 +0000
Labels: k8s-app=metrics-server Annotations:
deployment.kubernetes.io/revision: 1 Selector:
k8s-app=metrics-server Replicas: 1 desired | 1 updated |
1 total | 0 available | 1 unavailable
But if you will describe your node you will see taint that prevent you from scheduling new pods on master node:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-master-1 Ready master 17m v1.12.1
kubectl describe node kube-master-1
Name: kube-master-1
...
Taints: node-role.kubernetes.io/master:NoSchedule
You have to remove this taint:
kubectl taint node kube-master-1 node-role.kubernetes.io/master:NoSchedule-
node/kube-master-1 untainted
Result:
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-xvc77 2/2 Running 0 20m
kube-system coredns-576cbf47c7-rj4wh 1/1 Running 0 21m
kube-system coredns-576cbf47c7-vsjsf 1/1 Running 0 21m
kube-system etcd-kube-master-1 1/1 Running 0 20m
kube-system kube-apiserver-kube-master-1 1/1 Running 0 20m
kube-system kube-controller-manager-kube-master-1 1/1 Running 0 20m
kube-system kube-proxy-xp5zh 1/1 Running 0 21m
kube-system kube-scheduler-kube-master-1 1/1 Running 0 20m
kube-system metrics-server-5cbbc84f8c-l2t76 1/1 Running 0 18m
But this is not the best approach. Good approach is to join worker and set up metrics-server there. There won't be any issues and there is no need to touch taint on master node.
Hope it will help you.
The above answer by "Vit" is correct, either remove taint from existing node group or create new node group without any taint.

Minikube NodeUnderDiskPressure issue

I'm constantly running into NodeUnderDiskPressure in my pods that are running in Minikube. Using minikube ssh to see df -h, I'm using 50% max on all of my mounts. In fact, one is 50% and the other 5 are <10%.
$ df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 7.3G 503M 6.8G 7% /
devtmpfs 7.3G 0 7.3G 0% /dev
tmpfs 7.4G 0 7.4G 0% /dev/shm
tmpfs 7.4G 9.2M 7.4G 1% /run
tmpfs 7.4G 0 7.4G 0% /sys/fs/cgroup
/dev/sda1 17G 7.5G 7.8G 50% /mnt/sda1
$ df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
rootfs 1.9M 4.1K 1.9M 1% /
devtmpfs 1.9M 324 1.9M 1% /dev
tmpfs 1.9M 1 1.9M 1% /dev/shm
tmpfs 1.9M 657 1.9M 1% /run
tmpfs 1.9M 14 1.9M 1% /sys/fs/cgroup
/dev/sda1 9.3M 757K 8.6M 8% /mnt/sda1
The probably usually just goes away after 1-5 minutes. Strangely, restarting Minikube doesn't seem to speed up this process. I've tried removing all evicted pods but, again, disk usage doesn't actually look very high.
The docker images I'm using are just under 2GB and I'm trying to spin up just a few of them, so that should still leave me with plenty of headroom.
Here's some kubectl describe output:
$ kubectl describe po/consumer-lag-reporter-3832025036-wlfnt
Name: consumer-lag-reporter-3832025036-wlfnt
Namespace: default
Node: <none>
Labels: app=consumer-lag-reporter
pod-template-hash=3832025036
tier=monitor
type=monitor
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"consumer-lag-reporter-3832025036","uid":"342b0f72-9d12-11e8-a735...
Status: Pending
IP:
Created By: ReplicaSet/consumer-lag-reporter-3832025036
Controlled By: ReplicaSet/consumer-lag-reporter-3832025036
Containers:
consumer-lag-reporter:
Image: avery-image:latest
Port: <none>
Command:
/bin/bash
-c
Args:
newrelic-admin run-program python manage.py lag_reporter_runner --settings-module project.settings
Environment Variables from:
local-config ConfigMap Optional: false
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-sjprm (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
default-token-sjprm:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-sjprm
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 15s (x7 over 46s) default-scheduler No nodes are available that match all of the following predicates:: NodeUnderDiskPressure (1).
Is this a bug? Anything else I can do to debug this?
I tried:
Cleaning up evicted pods (with kubectl get pods -a)
Cleaning up unused images
(with minikube ssh + docker images)
Cleaning up all non-running containers (with
minikube ssh + docker ps -a)
The disk usage remained low as shown in my question. I simply recreated a minikube cluster and used the --disk-size flag and this solved my problem. The key thing to note is that even though df showed that I was barely using any disk, it helped to make the disk even bigger.