How to process multiple yarn jobs in certain order? - scala

Question

I am submitting multiple spark jobs in the following manner -
someCollection.foreach(m => {
..some code
sparkSubmitClass.run(m.name)
.. some code
})
where sparkSubmitClass.run() method basically uses a shell script which has $SPARK_HOME/bin/spark-submit and other related parameters.
The problem is that this code submits all the spark jobs in one go. What I want to achieve is - submit a job, then submit another job only when the earlier one finishes. This is because someCollection is ordered and the next job depends on data created by the previous job(s).
sparkSubmitClass.run() is on the following lines -
def run(appName: String)(implicit executionContext: ExecutionContext) = {
val command = s"sparkJob.sh $appName"
val processBuilder = Process(command)
val pio = new ProcessIO(_ => (),
stdout => {
scala.io.Source.fromInputStream(stdout)
.getLines.foreach(str => log.info(s"spark-submit: Application
Name=$appName stdout='${str.replace("'", "\\'")}'"))
},
stderr => {
val lines = scala.io.Source.fromInputStream(stderr).getLines().toBuffer
lines.foreach(str => log.info(s"spark-submit: Application Name=$appName
stderr='${str.replace("'", "\\'")}'"))
lines.flatMap(parseLineForApplicationUrl).headOption.foreach(appId =>
appId)
})
val process = processBuilder.run(pio)
val exitVal = process.exitValue() //returns 0 as soon as application is
submitted
}
And sparkJob.sh is basically -
MAIN_CLASS="com.SomeClassHavingRDDAndHiveOperations"
APPNAME=$1
JAVA_OPTS="-XX:MaxDirectMemorySize=$WORKER_DIRECT_MEM_SIZE -
XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true"
SPARK_HOME="/usr/lib/spark"
cmd='$SPARK_HOME/bin/spark-submit --class $MAIN_CLASS
--name ${APPNAME}
--conf "spark.yarn.submit.waitAppCompletion=false"
--conf "spark.io.compression.codec=snappy"
--conf "spark.kryo.unsafe=true"
--conf "spark.kryoserializer.buffer.max=1024m"
--conf "spark.serializer=org.apache.spark.serializer.KryoSerializer"
--driver-java-options "-XX:MaxMetaspaceSize=$WORKER_PERM_SIZE $JAVA_OPTS"
$appdir/SomeJar.jar $APPNAME'
eval $cmd
Any thoughts on how to build this kind of ordering?

Answer 1

Instead of writing bash scripts and calling each job and waste io/read-write phase why don't you loop the jobs ordering as you need inside your code. Here's some hints for you to follow :
First you have to make sure that you have an interface and you implement that interface to every class you want to process in order so that you can have a common method to start for each jobs. (in this example the method is process and the interface is JobInterface)
Then you need to write all the class-names-with-package in one file with the order you want. lets say that file is orderedJobs (you don't need to mention extension)
package1.Class1
package1.Class2
package2.Class3
....
Read and parse that file. I am assuming it to be in resouces folder, you can filter the lines you don't want
val classCall = Source.fromInputStream(getClass.getResourceAsStream(<locationOforderedJobs>)).getLines().filter(!_.startsWith("#"))
Loop with foreach for each class and call the common method defined (process)
classCall.foreach(job => {
processJob(job).process(<you can pass arguments>)
}
processJob is a funtion where you instantiate each class
def processJob(name: String): JobInterface = {
val action = Class.forName("<package path from source root>"+className).newInstance()
action.asInstanceOf[JobInterface]
}
This way you can reduce the io/read-write time wastage, increase spark processing efficiency by storing useful data for other jobs in-memory, reduce the processing time and many more...
I hope it helps