Code searching for the function in a different Class - scala

Question

I am trying to implement ALS on Spark. I have used ml class instead of mllib because the CSV file contains String in one column. Rating class in mllib do not accept String as a parameter.
I want to use predict function from org.apache.spark.mllib.recommendation.MatrixFactorizationModel class but while running it is searching in org.apache.spark.rdd.RDD.
This is the code I am using.
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.ml.recommendation.ALS
import org.apache.spark.ml.recommendation.ALS.Rating
import org.apache.spark.mllib.recommendation.MatrixFactorizationModel
object LoadCsv{
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("Load CSV")
val sc = new SparkContext(conf)
println("READING FILE...............................");
val data = sc.textFile("file.csv")
val ratings = data.map(_.split(',') match { case Array(user, item, rate) =>
Rating[String](user, item, rate.toFloat)
})
//val (userFactors, itemFactors) = ALS.train(ratings)
//Build the recommendation model using ALS
val rank = 10
val numIterations = 10
val model = ALS.train(ratings, rank, numIterations)
// Evaluate the model on rating data
val usersProducts = ratings.map { case Rating(user, product, rate) =>
(user, product)
}
// GETTING ERROR OVER HERE.
val predictions =
model.predict(usersProducts).map { case Rating(user, product, rate) =>
((user, product), rate)
}
val ratesAndPreds = ratings.map { case Rating(user, product, rate) =>
((user, product), rate)
}.join(predictions)
val MSE = ratesAndPreds.map { case ((user, product), (r1, r2)) =>
val err = (r1 - r2)
err * err
}.mean()
println("Mean Squared Error = " + MSE)
// Save and load model
//model.save(sc, "/home/shishir/spark-Projects/op")
//val sameModel = MatrixFactorizationModel.load(sc, "target/tmp/myCollaborativeFilter")
// $example off$
}
}
On running the code, I am getting this error:
LoadCsv.scala:34: value predict is not a member of (org.apache.spark.rdd.RDD[(String, Array[Float])], org.apache.spark.rdd.RDD[(String, Array[Float])])
[error] model.predict(usersProducts).map { case Rating(user, product, rate) =>

Answer 1

Your imports are "incorrect", you are using this:
import org.apache.spark.ml.recommendation.ALS
import org.apache.spark.ml.recommendation.ALS.Rating
When you should be using this:
import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.Rating
You can use the other package, it's just that the result won't be a model but (as the error says) and RDD.
You can read up online why there are two ML packages (from what I remember the mllib package is the older one and contains some design flaws so they reimplement in ml so they can use pipelines).

Answer 2

It looks like you're mixing MLLib and ML style approach. If your data uses supported ID types (it doesn't look like it is the case here) you can use MLLib implementation:
import org.apache.spark.mllib.recommendation.{ALS => OldALS}
import org.apache.spark.mllib.recommendation.{
MatrixFactorizationModel => OldModel}
import org.apache.spark.mllib.recommendation.{Rating => OldRating}
val ratings: RDD[OldRating] = ???
val model: OldModel = OldALS()
.setAlpha(0.01)
.setIterations(numIterations)
.setRank(rank)
.run(ratings)
If your data uses non-standard ID and you want an access to user friendly API it is better to use DataFrames:
val ratings: RDD[org.apache.spark.ml.recommendation.ALS.Rating[String]] = ???
val df = ratings.toDF
val als: org.apache.spark.ml.recommendation.ALS = new ALS()
.setAlpha(0.01)
.setMaxIter(numIterations)
.setRank(rank)
val model: org.apache.spark.ml.recommendation.ALSModel = als.fit(df)
Finally you can use your current approach but you'll have to operate on user factors and item factors directly without helpers like predict.