How to deal with null values in spark reduceByKey function? - scala

Question

I have a spark DataFrame (df) which looks like this:
+----------+--------+----------+--------+
| c1| c2| c3| c4|
+----------+--------+----------+--------+
| 1 | 5 | null| 7 |
+----------+--------+----------+--------+
| 1 | 5 | 4 | 8 |
+----------+--------+----------+--------+
| 1 | 3 | null| 11 |
+----------+--------+----------+--------+
| 1 | 3 | null| null |
+----------+--------+----------+--------+
| 2 | 6 | 23 | 17 |
+----------+--------+----------+--------+
| 2 | 6 | 7 | 3 |
+----------+--------+----------+--------+
| 2 | 3 | null| 11 |
+----------+--------+----------+--------+
| 2 | 3 | null| 17 |
+----------+--------+----------+--------+
I want to aggregate using (c1,c2) as key and have average of c3 and c4, so that I have this:
+----------+--------+----------+--------+
| c1| c2| c3| c4|
+----------+--------+----------+--------+
| 1 | 5 | 4 | 7.5 |
+----------+--------+----------+--------+
| 1 | 3 | null| 11 |
+----------+--------+----------+--------+
| 2 | 6 | 15 | 10 |
+----------+--------+----------+--------+
| 2 | 3 | null| 14 |
+----------+--------+----------+--------+
So, essentially I am ignoring the null values.
My half-baked code looks something like this:
val df1 = df.
// just working on c3 for time being
map(x => ((x.getInt(0), x.getInt(1)), x.getDouble(3))).
reduceByKey(
(x, y) => {
var temp = 0
var sum = 0.0
var flag = false
if (x == null) {
if (y != null) {
temp = temp + 1
sum = y
flag = true
}
} else {
if (y == null) {
temp = temp + 1
sum = x
} else {
temp = temp + 1
sum = x + y
flag = true
}
}
if (flag == false) {
null
} else {
sum/temp
}
}
)
Obviously, the above code is not working. Any help to make the code work is very much appreciated.
Edit 1 The answer given by #zero232 is a solution. However, it is not "the solution" I am looking for. My interest was to understand how to deal with null values when writing a custom function for reduceByKey(). I am re-asking the question below:
I want to aggregate using (c1,c2) as key and have root mean square [{sum(a_i^2)}^0.5] (or some function which is not available in spark for that matter) of c3 and c4 while ignoring the null values, so that I have this:
+----------+--------+----------+--------+
| c1| c2| c3| c4|
+----------+--------+----------+--------+
| 1 | 5 | 4 | 10.63 |
+----------+--------+----------+--------+
| 1 | 3 | null| 11 |
+----------+--------+----------+--------+
| 2 | 6 | 24.04 | 17.26 |
+----------+--------+----------+--------+
| 2 | 3 | null| 20.24 |
+----------+--------+----------+--------+

Answer 1

Just groupBy and use mean:
df.groupBy("c1", "c2").mean("c3", "c4")
or agg
df.groupBy("c1", "c2").agg(avg("c3"), avg("c4"))
Typically all primitive functions on DataFrames will handle correctly null values.
import org.apache.spark.sql.functions._
def rms(c: String) = sqrt(avg(pow(col(c), 2))).alias(s"rms($c)")
df.groupBy("c1", "c2").agg(rms("c3"), rms("c4"))
If you want to ignore null with RDDs just filter these out before you apply reduction:
somePairRDD.filter(_._2 != null)
.foldByKey(someDefualtValue)(someReducingFunction)
or convert values to Option and use pattern matching:
somePairRDD.mapValues(Option(_)).reduceByKey {
case (Some(x), Some(y)) => doSomething(x, y)
case (Some(x), _) => doSomething(x)
case (_, Some(_)) => doSomething(y)
case _ => someDefualt
}
or use map / flatMap / getOrElse and other standard tools to handle undefined values.