Scoping Issue with SparkContext.sequenceFile(...).foreach in Scala - scala

Question

My objective is to process a series of SequenceFile folders generated by calling org.apache.spark.rdd.RDD[_].saveAsObjectFile(...). My folder structure is similar to this:
\MyRootDirectory
\Batch0001
_SUCCESS
part-00000
part-00001
...
part-nnnnn
\Batch0002
_SUCCESS
part-00000
part-00001
...
part-nnnnn
...
\Batchnnnn
_SUCCESS
part-00000
part-00001
...
part-nnnnn
I need to extract some of the persisted data, however my collection - whether I use a ListBuffer, mutable.Map, or any other mutable type, loses scope and appears to be newed up on each iteration of sequenceFile(...).foreach
The following proof of concept generates a series of "Processing directory..." followed by "1 : 1" repeated and never increasing, as I expected counter and intList.size to do.
private def proofOfConcept(rootDirectoryName: String) = {
val intList = ListBuffer[Int]()
var counter: Int = 0
val config = new SparkConf().setAppName("local").setMaster("local[1]")
new File(rootDirectoryName).listFiles().map(_.toString).foreach { folderName =>
println(s"Processing directory $folderName...")
val sc = new SparkContext(config)
sc.setLogLevel("WARN")
sc.sequenceFile(folderName, classOf[NullWritable], classOf[BytesWritable]).foreach(f => {
counter += 1
intList += counter
println(s" $counter : ${intList.size}")
})
sc.stop()
}
}
Output:
"C:\Program Files\Java\jdk1.8.0_111\bin\java" ...
Processing directory C:\MyRootDirectory\Batch0001...
17/05/24 09:30:25.228 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[Stage 0:> (0 + 0) / 57] 1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
Processing directory C:\MyRootDirectory\Batch0002...
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
Processing directory C:\MyRootDirectory\Batch0003...
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1
1 : 1

Answer 1

The function inside foreach is run in a spark worker JVM, not inside the client JVM, where the variable is defined. That worker gets a copy of that variable locally, increments it, and prints it. My guess is you are testing this locally? If you were running this in a production, distributed spark environment, you wouldn't even see the output of those prints.
More generally, pretty much any function you pass into one of RDD's methods will probably be actually executed remotely and will not have mutable access to any local variables or anything. It will get an essentially immutable snapshot of them.
If you want to move data from spark's distributed storage back to the client, use RDD's collect method. The reverse is done with sc.parallelize. But note that both of these are usually done very rarely, since they do not happen in parallel.