Spark Submit on Kuberentes exit code - kubernetes

Question

How to check if the spark job is succeeded or failed programmatically, while running spark-submit. Usually the unix exit code is used.
phase: Failed
container status:
container name: spark-kubernetes-driver
container image: <regstry>/spark-py:spark3.2.1
container state: terminated
container started at: 2022-03-25T19:10:51Z
container finished at: 2022-03-25T19:10:57Z
exit code: 1
termination reason: Error
2022-03-25 15:10:58,457 INFO submit.LoggingPodStatusWatcherImpl: Application Postgres-Minio-Kubernetes.py with submission ID spark:postgres-minio-kubernetes-py-b70d3f7fc27829ec-driver finished
2022-03-25 15:10:58,465 INFO util.ShutdownHookManager: Shutdown hook called
2022-03-25 15:10:58,466 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3321e67c-73d5-422d-a26d-642a0235cf23
Process failed and when I get the exit code in unix by echo $? it return a zero error code!
$ echo $?
0
The pod generated is also random way. What's the way the spark-submit is handled apart from using sparkonk8operator?

Answer 1

If you are using bash, one way to to grep on the output. You might have to grep on stderr or stdout depending on where the log output is being sent.
Something like this:
OUTPUT=`spark-submit ...`
if echo "$OUTPUT" | grep -q "exit code: 1"; then
exit 1
fi

Answer 2

In addition to things which #Rico mentioned, I have also considered the deployment mode of cluster and client with changing spark-submit shell file in $SPARK_HOME/bin directory as follows.
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
if [ -z "${SPARK_HOME}" ]; then
source "$(dirname "$0")"/find-spark-home
fi
# disable randomized hash for string in Python 3.3+
export PYTHONHASHSEED=0
# check deployment mode.
if echo "$#" | grep -q "\-\-deploy-mode cluster";
then
echo "cluster mode..";
# temp log file for spark job.
export TMP_LOG="/tmp/spark-job-log-$(date '+%Y-%m-%d-%H-%M-%S').log";
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$#" |& tee ${TMP_LOG};
# when exit code 1 is contained in spark log, then return exit 1.
if cat ${TMP_LOG} | grep -q "exit code: 1";
then
echo "exit code: 1";
rm -rf ${TMP_LOG};
exit 1;
else
echo "job succeeded.";
rm -rf ${TMP_LOG};
exit 0;
fi
elif echo "$#" | grep -q "\-\-conf spark.submit.deployMode=cluster";
then
echo "cluster mode..";
# temp log file for spark job.
export TMP_LOG="/tmp/spark-job-log-$(date '+%Y-%m-%d-%H-%M-%S').log";
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$#" |& tee ${TMP_LOG};
# when exit code 1 is contained in spark log, then return exit 1.
if cat ${TMP_LOG} | grep -q "exit code: 1";
then
echo "exit code: 1";
rm -rf ${TMP_LOG};
exit 1;
else
echo "job succeeded.";
rm -rf ${TMP_LOG};
exit 0;
fi
else
echo "client mode..";
exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$#"
fi
Then, I have built and pushed my spark docker image.
See the following link for more details:
https://itnext.io/things-to-consider-to-submit-spark-jobs-on-kubernetes-766402c21716