Scala: Cannot assign values from RDD to ArrayBuffer - scala

Question

Below is my Code:
class Data(val x:Double=0.0,val y:Double=0.0) {
var cluster = 0;
}
var dataList = new ArrayBuffer[Data]()
val data = sc.textFile("Path").map(line => line.split(",")).map(userRecord => (userRecord(3), userRecord(4)))
data.foreach(a => dataList += new Data(a._1.toDouble, a._2.toDouble))
When I do
dataList.size
i get output as 0
But there are more that 4k records in data.
Now when I try using take
data.take(10).foreach(a => dataList += new Data(a._1.toDouble, a._2.toDouble))
Now I got the data in dataList. But I want my whole data in dataList.
Please help.

Answer 1

The problem is that your code inside the foreach is running on distributed worker not the same main thread where you inspect dataList.length. Use Rdd.collect() to get it.
val dataList = data
.take(10)
.map(a => new Data(a._1.toDouble, a._2.toDouble))
.collect()

Answer 2

The problem is related to where your code will be executed. Every operation made inside a transformation, i.e. a map, flatMap, reduce, and so on, is not performed in the main thread (or in the driver node), but in the worker nodes. These nodes run in different threads (or hosts) than the driver node.
Every object that is not store inside a RDD and that is used in a worker node, lives only in the worker memory space. Then, your dataList object is simply freshly created in each worker and the driver node cannot retrieve any information from this remote objects.
The code in the main program and in the so called actions, i.e. foreach, collect, take and so on, is executed in the main thread or driver node. Then, when you run
data.take(10).foreach(a => dataList += new Data(a._1.toDouble, a._2.toDouble))
the take method is getting back from workers the first 10 data of the RDD. All code is executed in the driver node and the magic works.
If you want to build an RDD of Data object you have to transform the information that you read directly into the oringal RDD. Try something similar to the following:
val dataList: RDD[Data] =
data.map(a => new Data(a._1.toDouble, a._2.toDouble))
Try to have a look also to this post: A new way to err, Apache Spark.
Hope it helps.