Maximum value of an mllib Vector? - scala

Question

I've created an ML pipeline with Apache Spark using mllib.
The evaluator result is a DataFrame with a column "probability", which is a mllib vector of probabilities (similar to predict_proba in scikit-learn).
val rfPredictions = rfModels.bestModel.transform(testing)
val precision = evaluator.evaluate(rfPredictions)
I tried something like this with no success:
rfPredictions.select("probability").map{c => c.getAs[Vector](1).max}
<console>:166: error: value max is not a member of
org.apache.spark.mllib.linalg.Vector
I want a new column with the max of this probabilities. Any ideas?

Answer 1

Vector doesn't have a max method. Try toArray.max:
rfPredictions.select("probability").map{ c => c.getAs[Vector](1).toArray.max }
or argmax:
rfPredictions.select("probability").map{ c => {
val v = c.getAs[Vector](1)
v(v.argmax)
}}
To add the max as a new column, define a udf and use it with withColumn function:
val max_proba_udf = udf((v: Vector) => v.toArray.max)
rfPredictions.withColumn("max_prob", max_proba_udf($"probability"))

Answer 2

Spark > 2.0
With ml, not mllib this will work next way:
import org.apache.spark.ml.linalg.DenseVector
just_another_df.select("probability").map{ c => c.getAs[DenseVector](0).toArray.max }
Using udf
import org.apache.spark.ml.linalg.DenseVector
val max_proba_udf = udf((v: DenseVector) => v.toArray.max)
val rfPredictions = just_another_df.withColumn("MAX_PROB", max_proba_udf($"probability"))