I see below error in pod logs:
, the following error information was pulled from the glusterfs log to help diagnose this issue:
[2020-01-10 20:57:47.132637] E [glusterfsd-mgmt.c:1804:mgmt_getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server
[2020-01-10 20:57:47.132690] E [glusterfsd-mgmt.c:1940:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:vol_32dd7b246275)
I have glusterfs installed on three servers - server 1, 2 and 3. I am using heketi to do dynamic provisioning of PVC. PVC creation is successful but pod creation shows below status while I try to mount something on this volume:
kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod 0/2 ContainerCreating 0 4m22s
Not an answer to Heketi's issue, but introducing an alternative solution to provide persistent storage in Kubernetes based on Gluster.
Kadalu Project(https://kadalu.io) is a Gluster based solution which natively integrates with Kubernetes without using Glusterd(Gluster's management layer).
Kadalu provides Kubernetes storage in just two steps.
Install Kadalu Operator using,
$ kubectl create -f https://kadalu.io/operator-latest.yaml
Register your storage device, through which you can provision persistent volumes to the applications running in Kubernetes.(Install Kubectl plugin in master node using pip3 install kubectl-kadalu)
$ kubectl kadalu storage-add storage-pool1 --type Replica3 \
--device node1:/dev/vdc \
--device node2:/dev/vdc \
--device node3:/dev/vdc
Thats it! Now ready for PV claims. Refer this link for Quick start https://kadalu.io/docs/quick-start
Latest blog post(https://kadalu.io/blog/kadalu-kubernetes-storage) explains the different configurations available with Kadalu.
Related
I have installed a Kubernetes cluster using kind k8s as it was easier to setup and run in my local VM. I also installed Docker separately. I then created a docker image for Spring boot application I built for printing messages to the stdout. It was then added to kind k8s local registry. Using this newly created local image, I created a deployment in the kubernetes cluster using kubectl apply -f config.yaml CLI command. Using similar method I've also deployed fluentd hoping to collect logs from /var/log/containers that would be mounted to fluentD container.
I noticed /var/log/containers/ symlink link doesn't exist. However there is /var/lib/docker/containers/ and it has folders for some containers that were created in the past. None of the new container IDs doesn't seem to exist in /var/lib/docker/containers/ either.
I can see logs in the console when I run kubectl logs pod-name even though I'm unable to find the logs in the local storage.
Following the answer in another thread given by stackoverflow member, I was able to get some information but not all.
I have confirmed Docker is configured with json logging driver by running the following command.
docker info | grep -i logging
When I run the following command (found in the thread given above) I can get the image ID.
kubectl get pod pod-name -ojsonpath='{.status.containerStatuses[0].containerID}'
However I cannot use it to inspect the docker image using docker inspect as Docker is not aware of such image which I assume it due to the fact it is managed by kind control plane.
Appreciate if the experts in the forum can assist to identify where the logs are written and recreate the /var/log/containers symbolink link to access the container logs.
It's absolutely normal that your local installed Docker doesn't have containers running in pod created by kind Kubernetes. Let me explain why.
First, we need to figure out, why kind Kubernetes actually needs Docker. It needs it not for running containers inside pods. It needs Docker to create container which will be Kubernetes node - and on this container you will have pods which will have containers that are you looking for.
kind is a tool for running local Kubernetes clusters using Docker container “nodes”.
So basically the layers are : your VM -> container hosted on yours VM's docker which is acting as Kubernetes node -> on this container there are pods -> in those pods are containers.
In kind quickstart section you can find more detailed information about image used by kind:
This will bootstrap a Kubernetes cluster using a pre-built node image. Prebuilt images are hosted atkindest/node, but to find images suitable for a given release currently you should check the release notes for your given kind version (check with kind version) where you'll find a complete listing of images created for a kind release.
Back to your question, let's find missing containers!
On my local VM, I setup kind Kubernetes and I have installed kubectl tool Then, I created an example nginx-deployment. By running kubectl get pods I can confirm pods are working.
Let's find container which is acting as node by running docker ps -a:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1d2892110866 kindest/node:v1.21.1 "/usr/local/bin/entr…" 50 minutes ago Up 49 minutes 127.0.0.1:43207->6443/tcp kind-control-plane
Okay, now we can exec into it and find containers. Note that kindest/node image is not using docker as the container runtime but crictl.
Let's exec into node: docker exec -it 1d2892110866 sh:
# ls
bin boot dev etc home kind lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var
#
Now we are in node - time to check if containers are here:
# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
135c7ad17d096 295c7be079025 47 minutes ago Running nginx 0 4e5092cab08f6
ac3b725061e12 295c7be079025 47 minutes ago Running nginx 0 6ecda41b665da
a416c226aea6b 295c7be079025 47 minutes ago Running nginx 0 17aa5c42f3512
455c69da57446 296a6d5035e2d 57 minutes ago Running coredns 0 4ff408658e04a
d511d62e5294d e422121c9c5f9 57 minutes ago Running local-path-provisioner 0 86b8fcba9a3bf
116b22b4f1dcc 296a6d5035e2d 57 minutes ago Running coredns 0 9da6d9932c9e4
2ebb6d302014c 6de166512aa22 57 minutes ago Running kindnet-cni 0 6ef310d8e199a
2a5e0a2fbf2cc 0e124fb3c695b 57 minutes ago Running kube-proxy 0 54342daebcad8
1b141f55ce4b2 0369cf4303ffd 57 minutes ago Running etcd 0 32a405fa89f61
28c779bb79092 96a295389d472 57 minutes ago Running kube-controller-manager 0 2b1b556aeac42
852feaa08fcc3 94ffe308aeff9 57 minutes ago Running kube-apiserver 0 487e06bb5863a
36771dbacc50f 1248d2d503d37 58 minutes ago Running kube-scheduler 0 85ec6e38087b7
Here they are. You can also notice that there are other container which are acting as Kubernetes Components.
For further debugging containers I would suggest reading documentation about debugging Kubernetes nodes with crictl.
Please also note that on your local VM there is file ~/.kube/config which has information needed for kubectl to communicate between your VM and the Kubernetes cluster (in case of kind Kubernetes - docker container running locally).
Hope It will help you. Feel free to ask any question.
EDIT - ADDED INFO HOW TO SETUP MOUNT POINTS
Answering question from comment about mounting directory from node to local VM. We need to setup "Extra Mounts". Let's create a definition needed for kind Kubernetes:
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
# add a mount from /path/to/my/files on the host to /files on the node
extraMounts:
- hostPath: /tmp/logs/
containerPath: /var/log/pods
# optional: if set, the mount is read-only.
# default false
readOnly: false
# optional: if set, the mount needs SELinux relabeling.
# default false
selinuxRelabel: false
# optional: set propagation mode (None, HostToContainer or Bidirectional)
# see https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
# default None
propagation: Bidirectional
Note that I'm using /var/log/pods instead of /var/log/containers/ - it is because on the cluster created by kind Kubernetes containers directory has only symlinks to logs in pod directory.
Save this yaml, for example as cluster-with-extra-mount.yaml , then create a cluster using this (create a directory /tmp/logs before applying this command!):
kind create cluster --config=/tmp/cluster-with-extra-mount.yaml
Then all containers logs will be in /tmp/logs on your VM.
I have recently started using kubectl krew (v0.3.4), which later was used to install "status" plugin (v0.4.1).
I am managing right now different clusters, and I'm checking the nodes' status. Most of the clusters answer something exactly like:
Node/[NodeName], created 25d ago linux Oracle Linux Server 7.8
(amd64), kernel 4.1.12-124.36.4.el7uek.x86_64, kubelet v1.18.2, kube-proxy v1.18.2
cpu: 0.153/7 (2%)
mem: 4.4GB/7.1GB (63%)
ephemeral-storage: 2.2GB
There is one cluster that answers, for some reason:
Node/[nodeName], created 11d ago
linux Oracle Linux Server 7.8 (amd64), kernel 4.1.12-124.26.5.el7uek.x86_64, kubelet v1.18.2, kube-proxy v1.18.2
cpu: 5, mem: 7.1GB, ephemeral-storage: 2.2GB
(Let me clarify that I'm trying to automate some resources checking and the way resources are differently displayed is quite annoying, plus the used vs total resources is exactly what I need!)
I am absolutely unable to locate the status plugin repo, and I have no idea where to go with this issue. kubectl version says that both clusters have the same server version, I'm executing the kubectl status command from my local in both cases and... I am completely out of ideas.
Does anyone know why this might be happening, or when can I go to look for answers?
To display used and total resources you can use kubectl top
Display Resource (CPU/Memory/Storage) usage.
The top command allows you to see the resource consumption for nodes or pods.
This command requires Metrics Server to be correctly configured and working on the server.
Available Commands:
node Display Resource (CPU/Memory/Storage) usage of nodes
pod Display Resource (CPU/Memory/Storage) usage of pods
Usage:
kubectl top [flags] [options]
You can also have a look at Tools for Monitoring Resources inside Kubernetes docs.
As for doing the same using Kubernetes Python Client you can use:
from kubernetes.config import load_kube_config
from kubernetes.client import CustomObjectsApi
load_kube_config()
cust = CustomObjectsApi()
cust.list_cluster_custom_object('metrics.k8s.io', 'v1beta1', 'nodes') # All node metrics
cust.list_cluster_custom_object('metrics.k8s.io', 'v1beta1', 'pods') # All Pod Metrics
Currently, I obtaine various information from the GoogleCloudPlatform management console screen, but in the future I would like to obtain it using API.
The information obtained is as follows.
Kubernetes Engine>Clusters>Cluster Size
Kubernetes Engine>Workloads>Pods
Please teach the API corresponding to each information acquisition.
GKE UI under the hood calls Kubernetes API to get information and show in UI.
You can use kubectl to query Kubernetes API to get that information.
kubectl get nodes
kubectl get pods
If you turn on the verbose mode in kubectl then it will show what REST API its calling on the kubernetes api server.
kubectl --v=8 get nodes
kubectl --v=8 get pods
The REST API for nodes and pods are
GET https://kubernetes-api-server-endpoint:6443/api/v1/nodes?limit=500
GET https://kubernetes-api-server-endpoint:6443/api/v1/namespaces/default/pods?limit=500
Here is the doc on how to configure Kubectl to connect with GKE.
Here is the doc from kubernetes on different ways to access Kubernetes API.
You can also use kubectl proxy for trying it out.
Remember to call above rest apis you need to authenticate to kubernetes api server either with a certificate or with a bearer token.
You need to:
install your command line
connect to your project
connect to your cluster
retrieve the number of pod inside your cluster
Install your command line
You can use your prefered command line or you can use the active cloud shell of your browser (the online command line interface integrated to Google Cloud Platform).
Option A) Using your own command line program, you need to install Google Cloud command (gcloud) on your machine.
Option B) Otherwise if you use the active cloud shell, just click on the active cloud shell button on the top of the page.
Connect to your project
(only for option A)
Login to your gcloud platform: gcloud auth login
$ gcloud auth login
Your browser has been opened to visit:
https://accounts.google.com/signin/oauth/oauthchooseaccount?client_id=65654645461.apps.googleusercontent.com&as=yJ_pR_9VSHEGFKSDhzpiw&destination=http%3A%2F%2Flocalhost%3A8085&approval_state=!ChRVVHYTE11IxY2FVbTIxb2xhbTk0SBIfczcxb2xyQ3hfSFVXNEJxcmlYbTVkb21pNVlhOF9CWQ%E2%88%99AJDr988AKKKKKky48vyl43SPBJ-gsNQf8w57Djasdasd&oauthgdpr=1&oauthriskyscope=1&xsrfsig=ChkAASDasdmanZsdasdNF9sDcdEftdfECwCAt5Eg5hcHByb3ZhbF9zdGF0ZRILZGVzdGluYXRpb24ASDfsdf1Eg9vYXV0aHJpc2t5c2NvcGU&flowName=GeneralOAuthFlow
Connect to your project: gcloud config set project your_project_id
$ gcloud projects list
PROJECT_ID NAME PROJECT_NUMBER
first-project-265905 My Project 117684542848
second-project-435504 test 895475526863
$ gcloud config set project first-project-265905
Connect to your cluster
Connected to your project, you need to connect to your cluster.
gcloud container clusters get-credentials your_cluster_name
$ gcloud container clusters list
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
test-cluster-1 asia-northeast1-a 1.33.33-gke.24 45.600.23.72 f1-micro 1.13.11-gke.14 3 RUNNING
$ gcloud container clusters get-credentials test-cluster-1
Fetching cluster endpoint and auth data.
kubeconfig entry generated for test-cluster-1.
Retrieve the number of nodes/pods inside your cluster
inside a given name space run the command
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-test-cluster-1-default-pool-d85b49-2545 NotReady 24m v1.13.11-gke.14
gke-test-cluster-1-default-pool-d85b49-2dr0 NotReady 3h v1.13.11-gke.14
gke-test-cluster-1-default-pool-d85b49-2f31 NotReady 1d v1.13.11-gke.14
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 0/1 Pending 0 44s
nginx 0/1 Pending 0 1m
Speaking about Python, Kubernetes Engine API could be used in this case.
Kubernetes Engine > Clusters > Cluster Size
In particular, a method get(projectId=None, zone=None, clusterId=None, name=None, x__xgafv=None)
returns an object that contains "currentNodeCount" value.
Kubernetes Engine > Workloads > Pods
A code example for listing pods could be found here:
Access Clusters Using the Kubernetes API
Is there any way to get hold of the log file of the pod in Kubernetes cluster?
I know I can fetch logs using "kubectl exec log -f $POD_NAME" command but I want to get access to log file directly.
It depends on the logging driver you're using
I'm assuming you're using the default json logging driver here, but you can see the node the pod is scheduled on by using kubectl get po -o wide
Then, logon to that node and you'll see the docker logs of the container under /var/lib/docker/containers/<long_container_id>/<long_container_id>-json.log
You will need to use docker ps and docker inspect to determine the long container id.
Run kubectl get pod <pod_name> -n <namespace> -o jsonpath='{.spec.nodeName}' to get the node this Pod is running on.
ssh into the node and you'll find the logs for the Pod at /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/.
The files within the /var/log/pods/<namespace>_<pod_name>_<pod_id>/<container_name>/ directory are symlinks to where your container runtime writes its container log files. So unlike jaxxstorm's answer, it doesn't matter which container runtime you're running.
I normally retrieve it from /var/log/containers where you will find all the containers' logs deployed on that particular machine
I am trying to setup Kubernetes for the first time. I am following the Fedora Manual installation guide: http://kubernetes.io/v1.0/docs/getting-started-guides/fedora/fedora_manual_config.html
I am trying to get the kubernetes addons running , specifically the kube-ui. I created the service and replication controller like so:
kubectl create -f cluster/addons/kube-ui/kube-ui-rc.yaml --namespace=kube-system
kubectl create -f cluster/addons/kube-ui/kube-ui-svc.yaml --namespace=kube-system
When i run
kubectl get events --namespace=kube-system
I see errors such as this:
Failed to pull image "gcr.io/google_containers/pause:0.8.0": image pull failed for gcr.io/google_containers/pause:0.8.0, this may be because there are no credentials on this request. details: (Authentication is required.)
How am i supposed to tell kubernetes to authenticate? This isnt covered in the documentation. So how do i fix this?
This happened due to a recent outage to gce storage as a result of which all of us went through this error while pulling images from gcr (which uses gce storage on the backend).
Are you still seeing this error ?
as the message says, you need credentials. Are you using Google Container Engine? Then you need to run
gcloud config set project <your-project>
gcloud config set compute/zone <your-zone, like us-central1-f>
gcloud beta container clusters get-credentials --cluster <your-cluster-name>
then your GCE cluster will have the credentials