How to overwrite file at a deployment time in kubernetes? - kubernetes

Question

I'm trying to deploy Diffusion image in kubernetes and I need to overwrite one of Diffusion configuration files at deployment time.
Actually it is a SystemAuthentication.store file with default credentials in /opt/Diffusion6.0.3_01/etc/. I'm storing new file in secret and mount it into etc/test/ which can be seen in below deployment file.
template:
metadata:
labels:
run: diffusion
spec:
serviceAccountName: diffusion-role
volumes:
- name: diffusion-secrets
secret:
secretName: diffusion-license
- name: ssl-cert
secret:
secretName: ssl-certificate
- name: system-authentication
secret:
secretName: system-authentication-store
containers:
- image: pushtechnology/diffusion:6.0.3
imagePullPolicy: IfNotPresent
name: diffusion
ports:
- containerPort: 8080
protocol: TCP
- containerPort: 8443
protocol: TCP
volumeMounts:
- name: diffusion-secrets
mountPath: /etc/diffusion-secrets
readOnly: true
- name: ssl-cert
mountPath: /etc/test/
readOnly: true
- name: system-authentication
mountPath: /etc/test/
command: [ "/bin/sh", "-c", "cp etc/test/SystemAuthentication.store /opt/DIffusion6.0.3_01" ]
When I deploy this image pods are failing with
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned diffusion-db6d6df7b-f5tp4 to timmy.pushtechnology.com
Normal SuccessfulMountVolume 2m kubelet, timmy.pushtechnology.com MountVolume.SetUp succeeded for volume "diffusion-role-token-n59ds"
Normal SuccessfulMountVolume 2m kubelet, timmy.pushtechnology.com MountVolume.SetUp succeeded for volume "ssl-cert"
Normal SuccessfulMountVolume 2m kubelet, timmy.pushtechnology.com MountVolume.SetUp succeeded for volume "system-authentication"
Normal SuccessfulMountVolume 2m kubelet, timmy.pushtechnology.com MountVolume.SetUp succeeded for volume "diffusion-secrets"
Normal Killing 1m (x2 over 1m) kubelet, timmy.pushtechnology.com Killing container with id docker://diffusion:FailedPostStartHook
Warning BackOff 1m (x2 over 1m) kubelet, timmy.pushtechnology.com Back-off restarting failed container
Normal Pulled 1m (x3 over 2m) kubelet, timmy.pushtechnology.com Container image "pushtechnology/diffusion:6.0.3" already present on machine
Normal Created 1m (x3 over 1m) kubelet, timmy.pushtechnology.com Created container
Normal Started 1m (x3 over 1m) kubelet, timmy.pushtechnology.com Started container
Warning FailedPostStartHook 1m (x3 over 1m) kubelet, timmy.pushtechnology.com
Warning FailedSync 1m (x5 over 1m) kubelet, timmy.pushtechnology.com Error syncing pod
I have tried also workaruond described here: https://github.com/kubernetes/kubernetes/issues/19764#issuecomment-269879587
with same results.

Answer 1

You overwrote the container command with cp etc/test/SystemAuthentication.store /opt/DIffusion6.0.3_01, which is a command with exits after it is done. Kubernetes assumes that this is a failure.
You need to replace it with something like cp etc/test/SystemAuthentication.store /opt/DIffusion6.0.3_01 && /path/to/original/binary, where the last command is the command the image would start without overwriting command. This depends on your image.

Answer 2

I think #svenwtl answer might be correct, but a Dockerfile of the image I'm using has some complicated constructs that I had no idea how to use in the deployment file.
The fix which has worked for me (after a long try/fail loop) was to actually use a container lifecycle hook:
volumeMounts:
- name: diffusion-secrets
mountPath: /etc/diffusion-secrets
readOnly: true
- name: ssl-cert
mountPath: /etc/test/
readOnly: true
- name: system-authentication
mountPath: /etc/test1/
lifecycle:
postStart:
exec:
command: [ "/bin/sh", "-c", "cp -f /etc/test1/SystemAuthentication.store /opt/Diffusion6.0.3_01/etc/" ]
I also mounted SystemAuthentication in different folder /etc/test1, but I don't think this was part of the fix.