Why do I get a type error in model.predictOnValues when I try the official example of Streaming Kmeans Clustering of Apache Spark? - scala

Question

I'm trying the Streaming Clustering example code at the end of the official guide, but I get a type error. Here is my code:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.clustering.StreamingKMeans
object Kmeans {
def main(args: Array[String]) {
val conf = new SparkConf().setMaster("local[2]").setAppName("kmeans")
val ssc = new StreamingContext(conf, Seconds(3))
val trainingData = ssc.textFileStream("training").map(Vectors.parse)
val testData = ssc.textFileStream("test").map(LabeledPoint.parse)
val numDimensions = 3
val numClusters = 2
val model = new StreamingKMeans()
.setK(numClusters)
.setDecayFactor(1.0)
.setRandomCenters(numDimensions, 0.0)
model.trainOn(trainingData)
model.predictOnValues(testData).print()
ssc.start()
ssc.awaitTermination()
}
}
But when I run
sbt package
I get the following error:
[error] found : org.apache.spark.streaming.dstream.DStream[org.apache.spark.mllib.regression.LabeledPoint]
[error] required: org.apache.spark.streaming.dstream.DStream[(?, org.apache.spark.mllib.linalg.Vector)]
[error] model.predictOnValues(testData).print()
[error] ^
[error] one error found
[error] (compile:compile) Compilation failed

Answer 1

You need to map testData: DStream[LabeledPoint] to a DStream[(K, Vector)]:
model.predictOnValues(testData.map(lp => (lp.label, lp.features))).print()
You can find the complete example here: StreamingKMeansExample.scala