How scala object member work with spark rdd - scala

Question

I have a spark application, that output result to redis.
It works fine on local mode, but cannot connect the redisHost with the args(0) that I assign like 10.242.10.100 on yarn-cluster mode.
The redisHost is unchanged 127.0.0.1.
object TestSparkClosure {
val logger: Logger = LoggerFactory.getLogger(TestSparkClosure.getClass)
var redisHost = "127.0.0.1"
var redisPort = 6379
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("TestSparkClosure")
if (args.length > 0) {
redisHost = args(0)
} else {
conf.setMaster("local")
}
val sparkContext = new SparkContext(conf)
var rdd = getRdd(sparkContext)
rdd.foreachPartition(partitionOfRecords => {
logger.info("host:port:" + redisHost + ":" + redisPort.toString)
val jedis = new Jedis(redisHost, redisPort)
partitionOfRecords.foreach(pair => {
val keystr = pair._1
val valuestr = pair._2
jedis.set(keystr, valuestr)
})
})
}
def getRdd(spark: SparkContext): RDD[(String, String)] = {
val rdd = spark.parallelize(List("2017\t1", "2018\t2", "2017\t3", "2018\t4", "2017\t5", "2018\t6")).map(line => {
val cols = line.split("\t")
(cols(0), cols(1))
})
rdd.reduceByKey((x, y) => {
((x.toInt + y.toInt).toString)
}, 3)
}
}
When I replace redisHost with local variable like this, It works fine again.
var localRedisHost = redisHost
rdd.foreachPartition(partitionOfRecords => {
logger.info("host:port:" + localRedisHost + ":" + redisPort.toString)
val jedis = new Jedis(localRedisHost , redisPort)
partitionOfRecords.foreach(pair => {
val keystr = pair._1
val valuestr = pair._2
jedis.set(keystr, valuestr)
})
})
Can anyone explain how the spark closure work here?
Thanks so much.

Answer 1

Its because you are using a variable which isnt able to use serialization. when you define a local element it can so you are able to use it inside of the RDD.