How to create custom set accumulator, i.e. Set[String]? - scala

Question

I am trying to use custom accumulator in Apache Spark to accumulate in a set. The result should have Set[String] type. For this I creat custom accumulator:
object SetAccumulatorParam extends AccumulatorParam[Set[String]] {
def addInPlace(r1: mutable.Set[String], r2: mutable.Set[String]): mutable.Set[String] = {
r1 ++= r2
}
def zero(initialValue: mutable.Set[String]): mutable.Set[String] = {
Set()
}
}
Yet I can not instantiate variable of this type.
val tags = sc.accumulator(Set(""))(SetAccumulatorParam)
result in error .Please help.
required: org.apache.spark.AccumulatorParam[Set[String]]

Answer 1

Adding to Traian's answer, here is a general case SetAccumulator for spark 2.x.
import org.apache.spark.util.AccumulatorV2
class SetAccumulator[T](var value: Set[T]) extends AccumulatorV2[T, Set[T]] {
def this() = this(Set.empty[T])
override def isZero: Boolean = value.isEmpty
override def copy(): AccumulatorV2[T, Set[T]] = new SetAccumulator[T](value)
override def reset(): Unit = Set.empty[T]
override def add(v: T): Unit = value + v
override def merge(other: AccumulatorV2[T, Set[T]]): Unit = value ++ other.value
override def value: Set[String] = value
}
And you can use it like this:
val accum = new SetAccumulator[String]()
spark.sparkContext.register(accum, "My Accum") // Optional, name it for SparkUI
spark.sparkContext.parallelize(Seq("a", "b", "a", "b", "c")).foreach(s => accum.add(s))
accum.value
Which outputs:
Set[String] = Set(a, b, c)

Answer 2

Update for 1.6:
object StringSetAccumulatorParam extends AccumulatorParam[Set[String]] {
def zero(initialValue: Set[String]): Set[String] = { Set() }
def addInPlace(s1: Set[String], s2: Set[String]): Set[String] = { s1 ++ s2 }
}
val stringSetAccum = sc.accumulator(Set[String]())(StringSetAccumulatorParam)
sc.parallelize(Array("1", "2", "3", "1")).foreach(s => stringSetAccum += Set(s))
stringSetAccum.value.toString
res0: String = Set(2, 3, 1)
In Spark 2.0 you're probably fine with using the existing collectionAccumulator (if you care about distinct values, you can check and add only if they don't exist):
val collAcc = spark.sparkContext.collectionAccumulator[String]("myCollAcc")
collAcc: org.apache.spark.util.CollectionAccumulator[String] = CollectionAccumulator(id: 32154, name: Some(myCollAcc), value: [])
spark.sparkContext.parallelize(Array("1", "2", "3")).foreach(s => collAcc.add(s))
collAcc.value.toString
res0: String = [3, 2, 1]
More info: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.util.AccumulatorV2