I currently have a job that runs a script. This script needs to create a file on the host file system.
To do so I make use of a hostPath volume with:
volumeMounts:
- mountPath: /var/logs/test
name: joblogs
volumes:
- hostPath:
path: /root/test
type: DirectoryOrCreate
name: joblogs
I used
chcon -Rt svirt_sandbox_file_t /root/test
to allow the writing on this directory, but even if files are created in
/var/logs/test
they are not on
/root/test
on the host.
EDIT: The pod itself runs on the same node I am talking about.
Related
I have read some tutorials of how to mount a volume in container and run the script on host/node directly. These are the examples given.
DeamonSet pod spec
hostPID: true
nodeSelector:
cloud.google.com/gke-local-ssd: "true"
volumes:
- name: setup-script
configMap:
name: local-ssds-setup
- name: host-mount
hostPath:
path: /tmp/setup
initContainers:
- name: local-ssds-init
image: marketplace.gcr.io/google/ubuntu1804
securityContext:
privileged: true
volumeMounts:
- name: setup-script
mountPath: /tmp
- name: host-mount
mountPath: /host
command:
- /bin/bash
- -c
- |
set -e
set -x
# Copy setup script to the host
cp /tmp/setup.sh /host
# Copy wait script to the host
cp /tmp/wait.sh /host
# Wait for updates to complete
/usr/bin/nsenter -m/proc/1/ns/mnt -- chmod u+x /tmp/setup/wait.sh
# Give execute priv to script
/usr/bin/nsenter -m/proc/1/ns/mnt -- chmod u+x /tmp/setup/setup.sh
# Wait for Node updates to complete
/usr/bin/nsenter -m/proc/1/ns/mnt /tmp/setup/wait.sh
# If the /tmp folder is mounted on the host then it can run the script
/usr/bin/nsenter -m/proc/1/ns/mnt /tmp/setup/setup.sh
containers:
- image: "gcr.io/google-containers/pause:2.0"
name: pause
(There is a configmap for composing the .sh files. I just skip that)
What does "/usr/bin/nsenter -m/proc/1/ns/mnt" mean? Is this a command to run something on host? what is "/proc/1/ns/mnt" ?
Lets start from the namepaces to understand this in detail :
Namespaces in container helps to isolate resources among the process. Namespaces controls the resources from the kernal and allocate to the process. This provides a great isolation among different containers that may run in a system.
Having said that, it will also make things complicated with these access restrictions to the namespaces. so comes the nsenter command , which will give the conatiners access to the namespaces. something similar to the sudo command.
.This command can give us access to mount, UTS, IPC, Network, PID,user,cgroup, and time namespaces.
the -m in your example is --mount which will access to the mount namespace specified by that file.
I have been using below to mount docker in my deployment file. This way From the inside of my code, I can build custom images using docker command
volumeMounts:
- name: dockersock
mountPath: "/var/run/docker.sock"
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
Now in newer kubernetes version privileged and hostPath is disabled. Due to this on new kubernetes above will not work.
I know i have few alternatives like Kaniko for building images, but I am not sure how to use it in code and what changes I need to make in my deployment file.
In order to mount a directory to a container i used bind mounts https://docs.docker.com/storage/bind-mounts/
Now i'm trying to find a way to replace $docker run -v command.
If you are using kubernetes as that is there in your tag. You can mount a volume as hostpath.
In Pod spec:
volumeMounts:
- name: config
mountPath: <PATH IN CONTAINER>
volumes:
- name: config
hostPath:
path: <YOUR LOCAL DIR PATH>
Check out https://kubernetes.io/docs/concepts/storage/volumes/ for more details
I am using kubernetes 1.15.7 version.
I am trying to follow the link https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration to enable 'encryption-provider-config' option on 'kube-apiserver'.
I edited file '/etc/kubernetes/manifests/kube-apiserver.yaml' and provided below option
- --encryption-provider-config=/home/rtonukun/secrets.yaml
But after that I am getting below error.
The connection to the server 171.69.225.87:6443 was refused - did you specify the right host or port?
with all kubectl commands like 'kubectl get no'.
Mainy, how do I do these below two steps?
3. Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file.
4. Restart your API server.
I've reproduced exactly your scenario, and I'll try to explain how I fixed it
Reproducing the same scenario
Create the encrypt file on /home/koopakiller/secrets.yaml:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: r48bixfj02BvhhnVktmJJiuxmQZp6c0R60ZQBFE7558=
- identity: {}
Edit the file /etc/kubernetes/manifests/kube-apiserver.yaml and set the --encryption-provider-config flag:
- --encryption-provider-config=/home/koopakiller/encryption.yaml
Save the file and exit.
When I checked the pods status got the same error:
$ kubectl get pods -A
The connection to the server 10.128.0.62:6443 was refused - did you specify the right host or port?
Troubleshooting
Since kubectl is not working anymore, I tried to look directly the running containers using docker command, then I see kube-apiserver container was recently recreated:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
54203ea95e39 k8s.gcr.io/pause:3.1 "/pause" 1 minutes ago Up 1 minutes k8s_POD_kube-apiserver-lab-1_kube-system_015d9709c9881516d6ecf861945f6a10_0
...
Kubernetes store the logs of created pods on /var/log/pods directory, I've checked the kube-apiserver log file and found a valuable information:
{"log":"Error: error opening encryption provider configuration file "/home/koopakiller/encryption.yaml": open /home/koopakiller/encryption.yaml: no such file or directory\n","stream":"stderr","time":"2020-01-22T13:28:46.772768108Z"}
Explanation
Taking a look at manifest file kube-apiserver.yaml is possible to see the command kube-apiserver, it runs into container, so they need to have the encryption.yaml file mounted into container.
If you check the volumeMounts in this file, you could see that only the paths below is mounted in container by default:
/etc/ssl/certs
/etc/ca-certificates
/etc/kubernetes/pki
/usr/local/share/ca-certificates
/usr/share/ca-certificates
...
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /usr/local/share/ca-certificates
name: usr-local-share-ca-certificates
readOnly: true
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
...
Based on the facts above, we can assume that apiserver failed to start because /home/koopakiller/encryption.yaml doesn't actually mounted into container.
How to solve
I can see 2 ways to solve this issue:
1st - Copy the encryption file to /etc/kubernetes/pki (or any of the path above) and change the path in /etc/kubernetes/kube-apiserver.yaml:
- --encryption-provider-config=/etc/kubernetes/encryption.yaml
Save the file and wait apiserver restart.
2nd - Create a new volumeMounts in the kube-apiserver.yaml manifest to mount a custom directory from node into container.
Let's create a new directory in /etc/kubernetes/secret (home folder isn't a good location to leave config files =)).
Edit /etc/kubernetes/manifests/kube-apiserver.yaml:
...
- --encryption-provider-config=/etc/kubernetes/secret/encryption.yaml
...
volumeMounts:
- mountPath: /etc/kubernetes/secret
name: secret
readOnly: true
...
volumes:
- hostPath:
path: /etc/kubernetes/secret
type: DirectoryOrCreate
name: secret
...
After save the file kubernetes will mount the node path /etc/kubernetes/secret into the same path into the apiserver container, wait start completely and try to list your node again.
Please let know if that helped!
I've created the manifest file, that looks as follows:
apiVersion: v1
kind: Pod
metadata:
name: kuard
spec:
volumes:
- name: "kuard-data"
hostPath:
path: "/home/developer/kubernetes/exercises"
containers:
- image: gcr.io/kuar-demo/kuard-amd64:1
name: kuard
volumeMounts:
- mountPath: "/data"
name: "kuard-data"
ports:
- containerPort: 8080
name: http
protocol: TCP
As you can see, the hostpath is:
path: "/home/developer/kubernetes/exercises"
and the mountPath is:
mountPath: "/data"
I've created a hello.txt file in the folder /home/developer/kubernetes/exercises and when I enter into the pod via kubectl exec -it kuard ash I can not find the file hello.txt.
Where is the file?
kind is using Docker containers to simulate Kubernetes nodes. So when you are creating files on your host (your ubuntu machine) the containers will not automatically have access to them.
(This gets even more complicated when using macos or windows and docker is running in a separate virtual machine...)
I assume that there are some shared folders visible inside the kind-docker-nodes, but I could not find it documented.
You can verify the filesystem content of the docker node from inside the container using docker exec -it kind-control-plane /bin/sh and then work with the usual tools.
If you need to make content from your development machine available you might want to have a look at ksync: https://github.com/vapor-ware/ksync