Kubernetes with logrotate sidecar mount point issue - kubernetes

Question

I am trying to deploy a test pod with nginx and logrotate sidecar.
Logrotate sidecar taken from: logrotate
My Pod yaml configuration:
apiVersion: v1
kind: Pod
metadata:
name: nginx-apache-log
labels:
app: nginx-apache-log
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
volumeMounts:
- name: logs
mountPath: /var/log
- name: logrotate
image: path/to/logrtr:sidecar
volumeMounts:
- name: logs
mountPath: /var/log
volumes:
- name: logs
emptyDir: {}
What I'd like to achieve is Logrotate container watching /var/log//.log, however with the configuration above, nginx container is failing because there is no /var/log/nginx:
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
2018/10/15 10:22:12 [emerg] 1#1: open() "/var/log/nginx/error.log" failed (2: No such file or directory)
However if I change mountPath for nginx from
mountPath: /var/log
to:
mountPath: /var/log/nginx
then it is starting, logging to /var/log/nginx/access.log and error.log, but logrotate sidecar sees all logs in /var/log not /var/log/nginx/. It is not a problem with just one nginx container, but I am planning to have more container apps logging to their own /var/log/appname folders.
Is there any way to fix/workaround that? I don't want to run sidecar for each app.
If I change my pod configuration to:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
volumeMounts:
- name: logs
mountPath: /var/log
initContainers:
- name: install
image: busybox
command:
- mkdir -p /var/log/nginx
volumeMounts:
- name: logs
mountPath: "/var/log"
then it is failing with:
Warning Failed 52s (x4 over 105s) kubelet, k8s-slave1 Error: failed to start container "install": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"mkdir -p /var/log/nginx\": stat mkdir -p /var/log/nginx: no such file or directory": unknown

Answer 1

Leave the mount path as /var/log. In your nginx container, execute mkdir /var/log/nginx in a startup script. You might have to tweak directory permissions a bit to make this work.

Answer 2

If you are running nginx in kubernetes, it is probably logging to stdout. When you run kubectl logs <nginx pod> nginx it will show you access and error logs. These logs are automatically logrotated by kubernetes, so you will not need a logrotate sidecar in this case.
If you are ever running pods that are not logging to stdout, this is a bit of an antipattern in kubernetes. It is more to your advantage to always log to stdout: kubernetes can take care of log rotation for you, and it is also easier to see logs with kubectl logs than by running kubectl exec and rummaging around in a running container