Add element error by scala/spark - scala

Question

I want to add some element to a set, but it does not work, what is wrong?
var set = new HashSet[Int]()
def add(a:Int){
set.add(a)
}
sc.parallelize(List(1,2,3)).map(add).collect
set.size

Answer 1

using sc.parallelize, you create a distributed dataset (RDD). Now your add-method (and set you are referencing therein) are serialized and sent to the executors. Your variable set only lives on the driver and does not notice the elements added to the other sets (There is no "global" set)
Solutions:
Use aggregate/combine methods to your RDD
val set = sc.parallelize(List(1,2,3))
.aggregate(Set.empty[Int])(
(s:Set[Int],i:Int) => s + i ,
(s1:Set[Int],s2:Set[Int]) => s1++s2
)
or collect the data as a set
val set = sc.parallelize(List(1,2,3)).collect().toSet
or use an accumulator:
import org.apache.spark.AccumulatorParam
object SetAccumulator extends AccumulatorParam[Set[Int]] {
def zero(initialValue: Set[Int]) = Set.empty[Int]
def addInPlace(s1: Set[Int], s2: Set[Int]) = s1 ++ s2
}
val acc = sc.accumulator(Set.empty[Int])(SetAccumulator)
sc.parallelize(List(1,2,3)).foreach(i=> acc.add(Set(i)))
val set = acc.value

Answer 2

the add method of set returns a unit (it has the side effect of changing the set).
When you do the map on the RDD recieved from the parallelize, you are basically changing the local version of the set at each executor, the original set on the driver is not changed.
If for example you had 3 executors, each would have a set of 1 value on that executor and then the data would go away.
In spark, you cannot rely on side effects when doing operations such as map.
A possible solution would be to do something like:
val s = sc.parallelize(List(1,2,3)).distinct().collect.toSet
s would have the set.