Kubernetes doesn't allow to mount file to container - kubernetes

Question

I ran into the below error when trying to deploy an application in a kubernetes cluster. It looks like kubernetes doesn't allow to mount a file to containers, do you know the possible reason?
deployment config file
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: model-loader-service
namespace: "{{ .Values.nsPrefix }}-aai"
spec:
selector:
matchLabels:
app: model-loader-service
template:
metadata:
labels:
app: model-loader-service
name: model-loader-service
spec:
containers:
- name: model-loader-service
image: "{{ .Values.image.modelLoaderImage }}:{{ .Values.image.modelLoaderVersion }}"
imagePullPolicy: {{ .Values.pullPolicy }}
env:
- name: CONFIG_HOME
value: /opt/app/model-loader/config/
volumeMounts:
- mountPath: /etc/localtime
name: localtime
readOnly: true
- mountPath: /opt/app/model-loader/config/
name: aai-model-loader-config
- mountPath: /var/log/onap
name: aai-model-loader-logs
- mountPath: /opt/app/model-loader/bundleconfig/etc/logback.xml
name: aai-model-loader-log-conf
subPath: logback.xml
ports:
- containerPort: 8080
- containerPort: 8443
- name: filebeat-onap-aai-model-loader
image: {{ .Values.image.filebeat }}
imagePullPolicy: {{ .Values.pullPolicy }}
volumeMounts:
- mountPath: /usr/share/filebeat/filebeat.yml
name: filebeat-conf
- mountPath: /var/log/onap
name: aai-model-loader-logs
- mountPath: /usr/share/filebeat/data
name: aai-model-loader-filebeat
volumes:
- name: localtime
hostPath:
path: /etc/localtime
- name: aai-model-loader-config
hostPath:
path: "/dockerdata-nfs/{{ .Values.nsPrefix }}/aai/model-loader/appconfig/"
- name: filebeat-conf
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/filebeat.yml
Details information of this issue:
message: 'invalid header field value "oci runtime error: container_linux.go:247:
starting container process caused \"process_linux.go:359: container init
caused \\\"rootfs_linux.go:53: mounting \\\\\\\"/dockerdata-nfs/onap/log/filebeat/logback/filebeat.yml\\\\\\\"
to rootfs \\\\\\\"/var/lib/docker/aufs/mnt/7cd32a29938e9f70a727723f550474cb5b41c0966f45ad0c323360779f08cf5c\\\\\\\"
at \\\\\\\"/var/lib/docker/aufs/mnt/7cd32a29938e9f70a727723f550474cb5b41c0966f45ad0c323360779f08cf5c/usr/share/filebeat/filebeat.yml\\\\\\\"
caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n"'
....
$ docker version
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:38:45 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:38:45 2017
OS/Arch: linux/amd64
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.3-rancher3", GitCommit:"772c4c54e1f4ae7fc6f63a8e1ecd9fe616268e16", GitTreeState:"clean", BuildDate:"2017-11-27T19:51:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Answer 1

caused "not a directory" is kind of self explanatory. What is the exact volume and volumeMount definition you use ? do you use subPath in your declaration ?
EDIT: change
- name: filebeat-conf
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/filebeat.yml
to
- name: filebeat-conf
hostPath:
path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/
and add subPath: filebeat.yml to volumeMount

Answer 2

SELinux may also be the culprit here. Log on to the node and execute sestatus. If the policy is disabled, you will see the output as SELINUX=disabled else it will be something similar to this:
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: mcs
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31
First Option:
You can either disable the selinux by editing the /etc/selinux/config file and update the SELINUX=permissive to SELINUX=disabled. Once done, reboot the machine and deploy to see if fixed. However, this is not the recommonded way and can be seen as a temporary fix.
Second Option:
Log on to the node and execute ps -efZ | grep kubelet which will give something like this.
system_u:system_r:kernel_t:s0 root 1592 1 2 May23 ? 09:58:18 /usr/local/bin/kubelet --anonymous-auth=false
Now, from this output capture the string system_u:system_r:kernel_t:s0 which can be changed to security context as below in your deployment.
securityContext:
seLinuxOptions:
user: system_u
role: system_r
type: spc_t
level: s0
Deploy your application and check the logs if it is fixed. Do let me know if this works for you or need any further help.

Answer 3

Is this a multi-node cluster? If so, the file needs to exist on all Kubernetes nodes since the pod is typically scheduled on a randomly available host machine. In any case, ConfigMaps are a much better way to supply static/read-only files to a container.