Yet another command-line argument with spaces question - scala

Question

I am executing a spark job, via spark-submit using a wrapper script and a parameter file. The parameter file contains some config information in addition to a set of set of command line arguments needed by the spark job.
The parameter file, params.sh, look like this:
MODE="query"
QUERYCOND="Id==1 and customer==2"
PACKAGE="com.etl"
CLASS="ExecSparkETL"
DRIVER_CLASS_PATH="/home/jars/ojdbc8.jar"
JARS="/home/jars/ojdbc8.jar"
JAR="/home/jars/etl-scala_2.11-1.0.0.jar"
ARGS="$MODE \"$CUSTOM\""
The wrapper script, exec.sh, contains this:
source params.sh
spark2-submit \
--class ${PACKAGE}.${CLASS} \
--queue root.me \
--deploy-mode client \
--master yarn \
--driver-class-path $DRIVER_CLASS_PATH \
--jars $JARS \
--conf spark.network.timeout=1200000 \
$JAR \
$ARGS
If I echo $ARGS from the script the result is what I should be passing to the scala jar as two args, one of them having spaces in the arg which is wrapped in double-quotes:
query "Id==1 and customer==2"
But when I execute and print those two args from within the scala program I get:
arg1: query
arg2: "Id==1
So it is somehow not parsing my double-quote wrapped argument correctly.
Now if I execute the program without assigning the program paramters to the script variable ARGS, it works as expected:
source params.sh
spark2-submit \
--class ${PACKAGE}.${CLASS} \
--queue root.me \
--deploy-mode client \
--master yarn \
--driver-class-path $DRIVER_CLASS_PATH \
--jars $JARS \
--conf spark.network.timeout=1200000 \
$JAR \
query "Id==1 and customer==2"
Results...
arg1: query
arg2: "Id==1 and customer==2"
So it doesn't seem to like the fact that the two parameters passed to the scala program are stored in a variable, like it is dropping the double-quote wrap from the parameter that contains spaces.
Any ideas on how I can fix that?

Answer 1

I'm trying to put a command in a variable, but the complex cases always fail! -- the solution is to use an array:
source params.sh
spark_args=(
--class "${PACKAGE}.${CLASS}"
--queue root.me
--deploy-mode client
--master yarn
--driver-class-path "$DRIVER_CLASS_PATH"
--jars "$JARS"
--conf spark.network.timeout=1200000
"$JAR"
"$MODE"
"$CUSTOM"
)
spark2-submit "${spark_args[#]}"
Note how all the variables are quoted.
Arrays are available in bash/zsh/ksh but not plain sh.