How to re-load the model when Spark streaming process is running? - scala

Question

I have the configuration file myConfig.conf in which the path to the prediction model is defined as a parameter pathToModel. I am reading this file once in order to get pathToModel.
However, now I want to slightly update the logic - I want to be able to re-load the prediction model any time pathToModel is updated in the configuration file.
This is my current code:
val pathToModelPar = ConfigFactory.load("myConfig.conf")
val pathToModel = pathToModelPar.getString("pathToModel")
val model = GradientBoostedTreesModel.load(sc,pathToModel)
myDStream.foreachRDD(myRDD => {
myRDD.foreachPartition({ partitionOfRecords =>
//...
val predictions = model.predict(...)
//...
})
})
I was thinking that the file config.conf could be checked every hour and then the IF-THEN rule can be applied - if a new pathToModel is different from a current pathToModel, then model is re-loaded.
However, I do not know how to implement this re-loading in the context of streaming data. As far as I understand, if I put IF-THEN rule outside from myDStream.foreachRDD(myRDD => {...}), then it will be checked only at the start of the streaming process. If I put IF-THEN rule inside myDStream.foreachRDD(myRDD => {...}), then I will get Task serialization error due to using sc in load. That's the problem. Any idea?
I tried this approach, but it loads the model only once:
var pathToModelPar = ConfigFactory.load("myConfig.conf")
var pathToModel = pathToModelPar.getString("pathToModel")
var model = GradientBoostedTreesModel.load(sc,pathToModel)
val ex = new ScheduledThreadPoolExecutor(1)
val task = new Runnable {
def run() = {
pathToModelPar = ConfigFactory.load("myConfig.conf")
pathToModel = pathToModelPar.getString("pathToModel")
model = GradientBoostedTreesModel.load(sc,pathToModel)
}
}
val f = ex.scheduleAtFixedRate(task, 1, 1, TimeUnit.HOURS)
myDStream.foreachRDD(myRDD => {
myRDD.foreachPartition({ partitionOfRecords =>
//...
val predictions = model.predict(...)
//...
})
})