kubernetes : deployment fails Error response from daemon: OCI runtime create failed: - kubernetes

Question

I have set up a two node cluster on raspberry pi and am trying to set up a ghost blog platform. I created a deployment and I see that the containers are crashing with the status "CrashLoopback" ANy help with the troubleshooting is much appreciated. thanks
Warning Failed 7m13s kubelet, kube-node-2 Error: failed to start container "ghost": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/docker/volumes/994c880776e4f19087b08dbd0b36362f71ed32a412691f0aac28c6f024ea20ed/_data\\\" to rootfs \\\"/var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged\\\" at \\\"/var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged/var/lib/ghost/content\\\" caused \\\"mkdir /var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged/var/lib/ghost/content: permission denied\\\"\"": unknown
Normal Pulling 6m31s (x5 over 10m) kubelet, kube-node-2 pulling image "arm32v7/ghost"
Normal Created 6m5s (x5 over 9m49s) kubelet, kube-node-2 Created container
Normal Pulled 6m5s (x5 over 9m50s) kubelet, kube-node-2 Successfully pulled image "arm32v7/ghost"
Warning BackOff 18s (x29 over 8m5s) kubelet, kube-node-2 Back-off restarting failed container
I see that it has something to down with permissions but unsure what these permissions are. Here are the output from PV and PVC
kubectl describe pv pv-ghost
Name: pv-ghost
Labels: pv=pv-ghost
Annotations: pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pv-protection]
StorageClass:
Status: Bound
Claim: default/pvc-ghost
Reclaim Policy: Retain
Access Modes: RWX
Capacity: 3Gi
Node Affinity: <none>
Message:
Source:
Type: NFS (an NFS mount that lasts the lifetime of a pod)
Server: 192.168.178.50
Path: /mnt/ud-hdd/ghost/data
ReadOnly: false
Events: <none>
kubectl describe pvc pvc-ghost
Name: pvc-ghost
Namespace: default
StorageClass:
Status: Bound
Volume: pv-ghost
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 3Gi
Access Modes: RWX
Events: <none>
Mounted By: ghost-deployment-6cd67cbb7f-5vh5j
ghost-deployment-6cd67cbb7f-67xtr
ghost-deployment-6cd67cbb7f-fvq8w
ghost-deployment-6cd67cbb7f-sgksk
ghost-deployment-6cd67cbb7f-wgtzf
ghost-deployment-6cd67cbb7f-xrrdc
Error after provide nfs server the root access
kubectl logs ghost-deployment-5759f4855c-294cl
tar: /var/lib/ghost/content.orig: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors

Answer 1

The permission denied error relates to the following operation: mkdir /var/lib/docker/overlay2/c83d0b7a7a4e0f576c63c39907aad3d54b4657d6520ca47fbd201e1e3b3a9fe3/merged/var/lib/ghost/content: permission denied
It's not clear from the info you've provided but if I had to guess you're trying to create a directory on the NFS share. If so, you most likely don't have write permission by the NFS server to write to that path.