Spark: sum over list containing None and Some()? - scala

Question

I already understand that I can sum over a list easily using List.sum:
var mylist = List(1,2,3,4,5)
mylist.sum
// res387: Int = 15
However, I have a list that contains elements like None and Some(1). These values were produced after running a left outer join.
Now, when I try to run List.sum, I get an error:
var mylist= List(Some(0), None, Some(0), Some(0), Some(1))
mylist.sum
<console>:27: error: could not find implicit value for parameter num: Numeric[Option[Int]]
mylist.sum
^
How can I fix this problem? Can I somehow convert the None and Some values to integers, perhaps right after the left outer join?

Answer 1

You can use List.collect method with pattern matching:
mylist.collect{ case Some(x) => x }.sum
// res9: Int = 1
This ignores the None element.
Another option is to use getOrElse on the Option to extract the values, here you can choose what value you want to replace None with:
mylist.map(_.getOrElse(0)).sum
// res10: Int = 1

Answer 2

I find the easiest way to deal with a collection of Option[A] is to flatten it:
val myList = List(Some(0), None, Some(0), Some(0), Some(1))
myList.flatten.sum
The call to flatten will remove all None values and turn the remaining Some[Int] into plain old Int--ultimately leaving you with a collection of Int.
And by the way, embrace that immutability is a first-class citizen in Scala and prefer val to var.

Answer 3

If you want to avoid creating extra intermediate collections with flatten or map you should consider using an Iterator, e.g.
mylist.iterator.flatten.sum
or
mylist.iterator.collect({ case Some(x) => x }).sum
or
mylist.iterator.map(_.getOrElse(0)).sum
I think the first and second approaches are a bit better since they avoid unnecessary additions of 0. I'd probably go with the first approach due to it's simplicity.
If you want to get a bit fancy (or needed the extra generality) you could define your own Numeric[Option[Int]] instance. Something like this should work for any type Option[N] where type N itself has a Numeric instance, i.e. Option[Int], Option[Double], Option[BigInt], Option[Option[Int]], etc.
implicit def optionNumeric[N](implicit num: Numeric[N]) = {
new Numeric[Option[N]] {
def compare(x: Option[N], y: Option[N]) = ??? //left as an exercise :-)
def fromInt(x: Int) = if (x != 0) Some(num.fromInt(x)) else None
def minus(x: Option[N], y: Option[N]) = x.map(vx => y.map(num.minus(vx, _)).getOrElse(vx)).orElse(negate(y))
def negate(x: Option[N]) = x.map(num.negate(_))
def plus(x: Option[N], y: Option[N]) = x.map(vx => y.map(num.plus(vx, _)).getOrElse(vx)).orElse(y)
def times(x: Option[N], y: Option[N]) = x.flatMap(vx => y.map(num.times(vx, _)))
def toDouble(x: Option[N]) = x.map(num.toDouble(_)).getOrElse(0d)
def toFloat(x: Option[N]) = x.map(num.toFloat(_)).getOrElse(0f)
def toInt(x: Option[N]) = x.map(num.toInt(_)).getOrElse(0)
def toLong(x: Option[N]) = x.map(num.toLong(_)).getOrElse(0L)
override val zero = None
override val one = Some(num.one)
}
}
Examples:
List(Some(3), None, None, Some(5), Some(1), None).sum
//Some(9)
List[Option[Int]](Some(2), Some(4)).product
//Some(8)
List(Some(2), Some(4), None).product
//None
List(Some(Some(3)), Some(None), Some(Some(5)), None, Some(Some(1)), Some(None)).sum
//Some(Some(9))
List[Option[Option[Int]]](Some(Some(2)), Some(Some(4))).product
//Some(Some(8))
List[Option[Option[Int]]](Some(Some(2)), Some(Some(4)), None).product
//None
List[Option[Option[Int]]](Some(Some(2)), Some(Some(4)), Some(None)).product
//Some(None) !?!?!
Note that there may be multiple ways of representing "zero", e.g. None or Some(0) in the case of Option[Int], though preference is given to None. Also, note this approach contains the basic idea of how one goes about turning a semigroup (without an additive identity) into a monoid.

Answer 4

you can use a .fold or .reduce and implement the sum of 2 Options manually. But I would go by the #Psidom approach

Answer 5

Folding on the list is a more optimized solution. Beware of chaining function calls on collections, as you may be iterating over something like a List multiple times.
A more optimized approach would look something like
val foo = List(Some(1), Some(2), None, Some(3))
foo.foldLeft(0)((acc, optNum) => acc + optNum.getOrElse(0))