Is it possible to read a modified json file without rebuilding the project in Scala? - scala

Question

After I change the configuration in the app.json file, when the project starts, the new changes are not applied and I have to rebuild the project (build new jar file). Is it possible to make this code read the modified app.json without rebuilding whole project?
Below is a package reading data from a app.json file
import net.liftweb.json._
import scala.io._
case class KafkaConfiguration(bootstrap_servers: String,
topic_extractor: String,
topic_vertica: String,
message_max_bytes: String,
group_id: String
)
case class HbaseConfiguration(table_name: String= "test", batch_size: Int)
case class SparkConfiguration(id: String, frequency: Int)
case class VerticaConfiguration(vertica_delimiter: String, vertica_qv: String)
case class Configuration(kafka: KafkaConfiguration,
spark: SparkConfiguration,
hbase: HbaseConfiguration,
vertica: VerticaConfiguration)
object Configuration{
def getConfiguration(filePath: String = "app.json"): Configuration ={
implicit val formats = DefaultFormats
val json = Source.fromURL(getClass.getClassLoader.getResource(filePath), "utf-8").mkString
val jValue = parse(json)
val kafkaConf = jValue.\("kafka").extract[KafkaConfiguration]
val sparkConfig = jValue.\("spark").extract[SparkConfiguration]
val hbaseConfig = jValue.\("hbase").extract[HbaseConfiguration]
val verticaConfig = jValue.\("vertica").extract[VerticaConfiguration]
val configuration: Configuration = new Configuration(kafkaConf, sparkConfig, hbaseConfig,verticaConfig)
configuration
}
}
Script to run my application:
. /etc/spark2/conf/spark-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_201-amd64
export PATH=$PATH:$HOME/bin
export SPARK_KAFKA_VERSION=0.10
CONFIG_FILE="app.json"
COMMAND="spark2-submit \
--master yarn \
--num-executors 4 \
--driver-memory 1g \
--executor-memory 2g \
--deploy-mode cluster \
--name "app1" \
--class DataPointStreaming ./app1.jar $CONFIG_FILE"
echo $COMMAND exec $COMMAND

Answer 1

Since you're using Spark, taking advantage of HOCON's environment variable substitution or even JVM properties might not be workable, as the JAR will be shipped across a cluster.
It is posssible to replace a file in a JAR (or, more formally, build a new JAR based on a given JAR with a file added). Something along these lines in bash can work:
#!/bin/sh
base=$( basename -- "$1" )
filename="${base%.*}"
config=$( basename -- "$2" )
confignoext="${config%.*}"
tmpdir=$( mktemp -d )
tmpdest="$tmpdir/$base"
confdest="$tmpdir/app.json"
cp $1 $tmpdest
cp $2 $confdest
zip -jur $tmpdest $confdest
mkdir -p build/configured
dest=build/configured/$filename-$confignoext.jar
cp $tmpdest $dest
echo $dest
Then you don't have to recompile everything to build a jar, you just keep a set of json files to patch into the jar for deployment (e.g. app-dev.json, app-prod.json, etc.)