I installed the latest kubernetes on my cluster, along with containerd and crictl. however, when it seems that I need to use sudo to work with containerd EG: sudo containerd ps. Without sudo, I get a permission denied errors.
based on kubernetes instruction, I need to add this:
sudo vim /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
So when I use crictl it'll point to containerd. however, when I perform crictl pods it gives me a permission denied error. Is it possible to work with crictl / containerd without sudo?
Can't seem to find much information online, a lot of it is related to docker.
I have AKS cluster deployed(version 1.19) on Azure, part of the deployment in kube-system namespace there are 2 azure-cni-networkmonitor pods, when opening a bash in one of the pods using:
kubectl exec -t -i -n kube-system azure-cni-networkmonitor-th6pv -- bash
I've noticed that although I'm running as root in the container:
uid=0(root) gid=0(root) groups=0(root)
There are some files that I can't open for reading, read commands are resulting in permission denied error, for example:
cat: /run/containerd/io.containerd.runtime.v1.linux/k8s.io/c3bd2dfc2ad242e1a706eb3f42be67710630d314cfeb4b96ec35f35869264830/rootfs/sys/module/zswap/uevent: Permission denied
File stat:
Access: (0200/--w-------) Uid: ( 0/ root) Gid: ( 0/ root)
Linux distribution running on container:
Common Base Linux Delridge
Although the file is non-readable, I shouldn't have a problem to read it as root right?
Any idea why would this happen? I don't see there any SELinux enabled.
/proc and /sys are special filesystems created and maintained by the kernel to provide interfaces into settings and events in the system. The uevent files are used to access information about the devices or send events.
If a given subsystem implements functionality to expose information via that interface, you can cat the file:
[root#home sys]# cat /sys/devices/system/cpu/cpu0/uevent
DRIVER=processor
MODALIAS=cpu:type:x86,ven0000fam0006mod003F:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000B,000C,000D,000E,000F,0010,0011,0013,0017,0018,0019,001A,001B,001C,002B,0034,003A,003B,003D,0068,006F,0070,0072,0074,0075,0076,0079,0080,0081,0089,008C,008D,0091,0093,0094,0096,0097,0099,009A,009B,009C,009D,009E,009F,00C0,00C5,00E7,00EB,00EC,00F0,00F1,00F3,00F5,00F6,00F9,00FA,00FB,00FD,00FF,0120,0123,0125,0127,0128,0129,012A,012D,0140,0165,024A,025A,025B,025C,025D,025F
But if that subsystem doesn't expose that interface, you just get permission denied - even root can't call kernel code that's not there.
I'm attempting to write MLflow artifacts to an NFS-mounted PVC. It's a new PVC mounting at /opt/mlflow, but MLflow seems to have permission writing to it. The specific error I'm getting is
PermissionError: [Errno 13] Permission denied: '/opt/mlflow'
I ran the same deployment with an S3-backed artifact store, and that worked just fine. That was on my home computer though, and I don't have the ability to do that at work. The MLflow documentation seems to indicate that I don't need any special syntax for NFS mounts.
Independent of MLflow you can approach this in a standard file permission way.
Exec into your pod and view the permissions at that file path
kubectl exec -it <pod>
ls -l /opt/mlflow
Within your pod/container see what user you are running as
whoami
If your user doesn't have access to that filepath, then you could adjust the file permissions by mounting the pvc into a different pod that runs under a user with permission to adjust them. Or you could try using fsGroup to control the permissions of the files mounted, which you can read more about here.
Using Google Cloud & Kubernetes engine:
Is there a smart way to view or mount a
PersistentVolume(physical Storage, in the case of Google PD) to a local drive/remote computer/macos, or anything able to view data on the volume - to be able to backup or just view files.
Maybe using something like FUSE and in my case osxfuse.
Obviously I can mount a container and exec,
but maybe there are other ways?
Tried to ssh into the node and cd to /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet
But I get cd: pods: Permission denied
Regarding sharing PersistnetDisk between other VM's it was discussed here. If you want to use the same PD on many nodes, it would work only in read-only mode.
Easiest way to check what's inside the PD is to SSH to node (like you mentioned), but it will require superuser privileges (sudo) rights.
- SSH to node
$ sudo su
$ cd /home/kubernetes/containerized_mounter/rootfs/var/lib/kubelet/plugins/kubernetes.io/gce-pd/mounts
$ ls
Now you will get a few records, depends on how many PVC you have. Name of folder is the same as name you get from kubectl get pv.
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-53091548-57af-11ea-a629-42010a840131 1Gi RWO Delete Bound default/pvc-postgres standard 42m
Enter to it using cd
$ cd <pvc_name>
in my case:
$ cd gke-gke-metrics-d24588-pvc-53091548-57af-11ea-a629-42010a840131
now you can list all files inside this PersistentDisk
...gke-gke-metrics-d24588-pvc-53091548-57af-11ea-a629-42010a840131 # ls
lost+found text.txt
$ cat text.txt
This is test
It's not empty
There is tutorial on Github where user used sshfs but on MacOS.
===
Alternative way to mount PD to your local machine is to use NFS. However, you would need to configure it. Later you could specify mount in your Deployment and your local machine.
More details can be found here.
===
To create backup's you can consider Persistent disk snapshots.
I am looking to execute
apt install tcpdump
but facing permission denial, upon looking to set the directory to root, it is asking me for password and I don't know from where to get that password.
I installed nginx helm chart from stable/nginx repository with no RBAC
Please see snapshot for details on error, while I tried installing tcpdump in the pod after doing ssh into it.
In Using GDB with Nginx, you can find troubleshooting section:
Shortly:
find the node where your pod is running (kubectl get pods -o wide)
ssh into the node
find the docker_ID for this image (docker ps | grep pod_name)
run docker exec -it --user=0 --privileged docker_ID bash
Note: Runtime privilege and Linux capabilities
When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host. Additional information about running with --privileged is available on the Docker Blog.
Additional resources:
ROOT IN CONTAINER, ROOT ON HOST
Hope this help.