Dockerfile ENV lost in container when deployed to Kubernetes - kubernetes

Question

For years we have built base PHP-FPM container images locally with code like this to include Oracle DB support:
ARG PHP_VERSION=7.4
ARG PHP_TYPE=fpm
FROM php:${PHP_VERSION}-${PHP_TYPE}
ENV LD_LIBRARY_PATH /usr/local/instantclient
ENV ORACLE_BASE /usr/local/instantclient
ENV ORACLE_HOME /usr/local/instantclient
ENV TNS_ADMIN /etc/oracle
COPY oracle /etc/oracle
RUN echo 'instantclient,/usr/local/instantclient' | pecl install oci8-${OCI8_VERSION} \
&& docker-php-ext-configure oci8 --with-oci8=instantclient,/usr/local/instantclient \
&& docker-php-ext-install oci8 \
&& docker-php-ext-configure pdo_oci --with-pdo-oci=instantclient,/usr/local/instantclient \
&& docker-php-ext-install pdo_oci \
&& rm -rf /tmp/pear
From this image we build application specific images that are deployed to a Kubernetes cluster and the TNS_ADMIN variable and value have persisted without issue.
We recently changed how the images are built (using Kaniko and GitLab CI instead of building them locally) and found that now when the image is deployed to the Kubernetes cluster (via Helm) the TNS_ADMIN variable is now missing (not just a blank value, the entire variable). Another change made was how the Oracle pieces are installed (using docker-php-extension-installer), so the pertinent Dockerfile code looks like this now:
ADD https://github.com/mlocati/docker-php-extension-installer/releases/latest/download/install-php-extensions /usr/local/bin/
RUN chmod +x /usr/local/bin/install-php-extensions && \
install-php-extensions oci8 pdo_oci
# Oracle client config
ENV TNS_ADMIN=/etc/oracle
COPY php.cli/oracle /etc/oracle
And, here is the GitLab CI Kaniko related code to build the application specific images (only the $PHP_TYPE applies to the image in question):
- |
LOCAL_REPOSITORY=${CI_REGISTRY}/<internal namespace path>/$REPOSITORY
# Build config.json for credentials
echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json
/kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/$DOCKER_FILE_PATH/Dockerfile --build-arg PHP_VERSION=$PHP_VERSION --build-arg PHP_TYPE=$PHP_TYPE --build-arg PHPUNIT_VERSION=$PHPUNIT_VERSION --build-arg PHPCS_VERSION=$PHPCS_VERSION --build-arg PHPCSFIXER_VERSION=$PHPCSFIXER_VERSION --destination $LOCAL_REPOSITORY:$PHP_VERSION-$TAG_NAME
Thinking this was possibly due to how Kaniko works, or the changes to the Oracle install process, we pulled the base image and application image separately and ran them with a bash shell. When pulled locally, the TNS_ADMIN variable is present. This suggests whatever is occurring is happening once Helm deploys it to the cluster.
What is vexing is on the surface neither of the changes we made should affect the setting of an environment variable in this manner in the image, but those were the only changes made that coincide with the issue arising. So, the issue seems to be when deploying the image to our cluster. This process itself has not changed at all. The Helm chart has not changed, which indicates it is not part of this issue; that being said, the issue occurs when Helm deploys the chart that uses the image.
Has anyone else seen something like this, or have any ideas where to center our search for answers?

Answer 1

Well, our issue was one that is probably endemic to many people running applications in Kubernetes: our image pull policy for the Helm deployment was set to IfNotPresent and a cached image without the ENV value set was being used (the image was built using a Dockerfile that did not set TNS_ADMIN). We have a lot of moving parts in our process and made multiple changes that were not seen due to this.
I am of course chastened by this explanation and so I will offer the advice to always make sure you are pulling a fresh image as the first step in troubleshooting issues with Kubernetes/Helm deployments.