spark ClassNotFoundException error for performing kmeans - scala

Question

I am trying to submit a spark job for using spark KMeans. I am packaging the scala file correctly, but when I want to submit the job I always have the ClassNotFoundException.
Here is my sbt fille:
name:="sparkKmeans"
libraryDependencies+="org.apache.spark" %% "spark-core" % "1.1.1"
and here is my scala class:
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
import org.apache.spark.mllib.linalg.Vectors
object sparkKmeans {
def main(args: Array[String]) {
// create Spark context with Spark configuration
val sc = new SparkContext(new SparkConf().setAppName("SparkKmeans"))
//val threshold = args(1).toInt
// Load and parse the data. source is the first argument.
val data = sc.textFile(args(0))
val parsedData = data.map(s => Vectors.dense(s.split(' ').map(_.toDouble))).cache()
// Cluster the data into classes using KMeans. number of itteration is fixed as 100
// and number of clusters is get from the input -second argument
val numClusters = args(1)
val numIterations = 100
val clusters = KMeans.train(parsedData, numClusters, numIterations)
// Evaluate clustering by computing Within Set Sum of Squared Errors
val WSSSE = clusters.computeCost(parsedData)
println("Within Set Sum of Squared Errors = " + WSSSE)
// Save and load model based on thirs argument.
//clusters.save(sc, args(2))
// val sameModel = KMeansModel.load(sc, args(2))
}
}
I have commented last two lines because I saw some places said that spark has problem with serializer. But still has the problem.
and here is the error:
java.lang.ClassNotFoundException: sparkKmeans
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at org.apache.spark.util.Utils$.classForName(Utils.scala:174)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
and Submitting the job using:
./bin/spark-shell --class sparkKmeans ......
If anybody could help me I will be appreciated.

Answer 1

Thanks for the comments.
I have done what you said:
Built.sbt file:
name:="sparkKmeans"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.6.1",
"org.apache.spark" % "spark-mllib_2.10" % "1.6.1"
)
(I used scala 2.11.8 and Spark 1.6.1 version but still the same error.
Also about the other question:
I am packaging my application using:
sbt
compile
package
and to execute use:
./bin/spark-submit --class sparkKmeans k/kmeans/target/scala-2.10/sparkkmeans_2.10-0.1-SNAPSHOT.jar '/home/meysam/spark-1.6.1/kmeans/pima.csv' 3