Permission denied executing jar file downloaded to .Ivy folder in Kubernetes - kubernetes

Question

I created a dockerfile and changed User to a non-root user nobody. Locally this works perfectly. When deployed on kubernetes howerever, I get error
java.nio.file.AccessDeniedException: ./xxxxxx_2.12-2.6.3.jar
When I dug in more I realised this jar file is download after the spark dependencies used in the dockerfiles are downloaded. Therefore any permissions given to the spark folder are not present for this newly dowloaded jar file which is downloaded at runtime into /opt/spark/.ivy2/xxx which has root permissions. This causes the pod in kubernetes to fail.
I am wondering if there is way to give permissions to execute this jar file. Since it seems this is not possible in the Dockerfile. Any suggestion as to how to solve this issue ??
As proposed by #mario
ARG SPARK_OPERATOR_BASE_IMAGE_VERSION=v2.4.5
FROM xxx/alpine:3.12 as preparator
ARG SCALA_VERSION=2.12
ARG SPARK_VERSION=2.4.7
ARG HADOOP_VERSION=3.2.1
ARG AWS_SDK_VERSION=1.11.375
ARG MAVEN_VERSION=3.6.2
RUN apk add --no-cache \
bash \
curl && \
mkdir /target
COPY hashes /tmp/
COPY prepare /tmp/
WORKDIR /tmp
# Download Hadoop
RUN curl -L -O https://downloads.apache.org/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz && \
sha256sum -c hadoop-${HADOOP_VERSION}.sha256 && \
tar -xzf hadoop-${HADOOP_VERSION}.tar.gz && \
mv hadoop-${HADOOP_VERSION} /target/hadoop
# Download Spark
RUN curl -L -O https://downloads.apache.org/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.tgz && \
sha512sum -c spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.sha512 && \
tar -xzf spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION}.tgz && \
mv spark-${SPARK_VERSION}-bin-without-hadoop-scala-${SCALA_VERSION} /target/spark && \
# Download Spark 3.0.0 entrypoint script from GitHub, bugfixing for 2.4.7
curl -L -O https://raw.githubusercontent.com/apache/spark/v3.0.0/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh && \
mv entrypoint.sh /target/entrypoint.sh && \
chmod +x /target/entrypoint.sh
# Download AWS Jars
RUN curl -L -O https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar && \
sha1sum -c hadoop-aws-${HADOOP_VERSION}.jar.sha1 && \
mv hadoop-aws-${HADOOP_VERSION}.jar /target/spark/jars/ && \
curl -L -O https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_SDK_VERSION}/aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar && \
sha1sum -c aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar.sha1 && \
mv aws-java-sdk-bundle-${AWS_SDK_VERSION}.jar /target/spark/jars/
# Directory needed for saving built jars
RUN mkdir /target/spark/custom-jars/
#### Download Prometheus + Metric dependencies ####
# install java, maven and prometheus fat jar using maven (pom.xml)
RUN apk add --update openjdk8 && \
curl -L -O https://archive.apache.org/dist/maven/maven-3/${MAVEN_VERSION}/binaries/apache-maven-${MAVEN_VERSION}-bin.tar.gz && \
tar -xzf apache-maven-${MAVEN_VERSION}-bin.tar.gz && export PATH=./apache-maven-${MAVEN_VERSION}/bin:$PATH && \
mv prometheus-pom.xml pom.xml && mvn clean package && mv target/prometheusMetricLibs-jar-with-dependencies.jar /target/spark/custom-jars/
RUN \
chown -R nobody:99 /target/spark \
&& chown -R nobody:99 /target/hadoop \
&& chmod -R ugo+rw /target/spark \
&& chmod -R ugo+rw /target/hadoop
ARG SPARK_OPERATOR_BASE_IMAGE_VERSION
FROM gcr.io/spark-operator/spark:${SPARK_OPERATOR_BASE_IMAGE_VERSION}
RUN rm -rf /opt/spark/
COPY --from=preparator /target/ /opt/
ENV SPARK_HOME=/opt/spark \
HADOOP_HOME=/opt/hadoop
ENV HADOOP_OPTS="-Djava.library.path=/opt/hadoop/lib/native" \
LD_LIBRARY_PATH=${HADOOP_HOME}/lib/native \
PATH=${HADOOP_HOME}/bin:${SPARK_HOME}/bin:${PATH}
COPY conf /opt/spark/conf/
RUN echo "export JAVA_HOME=${JAVA_HOME}" >> ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh && \
echo "export JAVA_HOME=${JAVA_HOME}" > ${SPARK_HOME}/conf/spark-env.sh && \
echo "export SPARK_DIST_CLASSPATH=\$(hadoop classpath)" >> ${SPARK_HOME}/conf/spark-env.sh
# 99 used instead of nobody because in the alpine image the Group Id of nobody is different than in the spark operator image.
RUN \
addgroup --gid 99 nobody \
&& echo "nobody:x:99:99:nobody:/nonexistent:/usr/sbin/nologin" >> /etc/passwd \
&& usermod -a -G users nobody \
&& chmod -R ugo+rw /var/lib/
USER nobody
# if we want local storage
# "spark.eventLog.dir": "tmp/spark-events"
RUN mkdir -p /tmp/spark-events
in my pod the jar file is implemented like this
sparkConf:
"spark.ui.port": "4045"
"spark.eventLog.enabled": {{ .Values.spark.eventLogEnabled | quote }}
"spark.eventLog.dir": "xx//"
"spark.jars.ivySettings": "/vault/secrets/xx-ivysettings.xml"
"spark.jars.ivy": "/opt/spark/.ivy2"
"spark.jars.packages": "xxxx_2.12:{{ .Values.appVersion }}"
"spark.blacklist.enabled": "false"
"spark.driver.supervise": "true"
"spark.app.name": {{ .Values.name | quote }}
"spark.submit.deployMode": {{ .Values.spark.deployMode | quote }}
"spark.driver.extraJavaOptions": "-Dlog4j.configurationFile=log4j.properties"
"spark.executor.extraJavaOptions": "-Dlog4j.configurationFile=log4j.properties"

Answer 1

It would be easier to answer your question if you share your Dockerfile and your kubernetes Pod template yaml manifest, but in short: you can manipulate your Pod's permissions using the securityContext like in the example from the docs below:
apiVersion: v1
kind: Pod
metadata:
name: security-context-demo
spec:
securityContext:
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
volumes:
- name: sec-ctx-vol
emptyDir: {}
containers:
- name: sec-ctx-demo
image: busybox
command: [ "sh", "-c", "sleep 1h" ]
volumeMounts:
- name: sec-ctx-vol
mountPath: /data/demo
securityContext:
allowPrivilegeEscalation: false
For debugging purposes you can start from setting allowPrivilegeEscalation to true or runAsUser: 0 (root) but keep in mind this is not the solution that can be used in production. Running containers as root is generally a bad idea and in most cases it can be avoided.
Therefore any permissions given to the spark folder are not present
for this newly dowloaded jar file which is downloaded at runtime into
/opt/spark/.ivy2/xxx which has root permissions.
Is it really necessary for it to have root permissions ? Most likely it can be fixed in your Dockerfile.