how to add custom metrics to spark mlib evaluator? - scala

Question

The spark version is 2.3.1.
Spark-Mlib library provide a BinaryClassificationEvaluator (BinaryClassificationEvaluator.scala) class to evaluate the algorithm, and also can be used to gird search. But it only suppose two metrics
val metric = $(metricName) match {
case "areaUnderROC" => metrics.areaUnderROC()
case "areaUnderPR" => metrics.areaUnderPR()
//what i want todo
case "areUnderXX"=> myCustomMetric()
}
I try to add more, but BinaryClassificationEvaluator have some members that are set to private, so i can't just extends it. Here are the code that can't be viewed outside the package:
SchemaUtils.checkColumnTypes(schema, $(rawPredictionCol), Seq(DoubleType, new VectorUDT))
SchemaUtils.checkNumericType(schema, $(labelCol))
These code do some type check, so if i remove it, it would workaround. But, It seems unsafe and ugly. So, is there another way to do it? any help would be appreciated!

Answer 1

You can use MulticlassMetrics. It provides more metrics. As an example using a DataFrame with your labels and predictions:
+---------+----------+
|label |prediction|
+---------+-----+----+
| 1.0 | 0.0 |
| 0.0 | 0.0 |
+---------+----------+
You must pivot your dataframe for the label field
val metrics = df.select("prediction", labelName)
.as[(Double, Double)]
.rdd
val multim = new MulticlassMetrics(metrics)
val labels = multim.labels
val accuracy = multim.accuracy
println("Summary Statistics")
println(s"Accuracy = $accuracy")
labels.foreach { l =>
println(s"Precision($l) = " + multim.precision(l))
}
// Recall by label
labels.foreach { l =>
println(s"Recall($l) = " + multim.recall(l))
}
// False positive rate by label
labels.foreach { l =>
println(s"FPR($l) = " + multim.falsePositiveRate(l))
}
There are more metrics you can see in https://spark.apache.org/docs/2.2.0/mllib-evaluation-metrics.html
Depending of what kind of metrics you need maybe you should deal with the dataframe with the dataframe. For example, if you want to compute your confusion matrix, you can proceed pivoting over the "prediction" column like:
df.groupBy("label").
pivot("prediction", range)
.count()
.na.fill(0.0)
.orderBy("label)
.show()