A not_so_familiar exception in Apache Spark/Scala - scala

Question

I encountered the following when I tried to do a spark submit:
Exception in thread "main" java.lang.NullPointerException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Is this a known problem?
Thanks and regards,
The following program is able to reproduce the above error message:
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
class anomaly_model(val inputfile: String, val clusterNum: Int, val maxIterations: Int, val epsilon: Double, val scenarioNum: Int, val outputfile: String){
val conf = new SparkConf().setAppName("Anomaly Model")
val sc = new SparkContext(conf)
val data = sc.textFile(inputfile)
def main(args: Array[String]) {
val inputfile = "sqlexpt.txt"
val clusterNum = 5
val maxIterations = 1000
val epsilon = 0.001
val scenarioNum = 10
val outputfile = "output.csv"
val am = new anomaly_model(inputfile, clusterNum, maxIterations, epsilon, scenarioNum, outputfile)
}
}

Answer 1

Probably it was my mistake to define the main method inside the class... I should have defined it inside a companion object instead (I come from a Java background!).

Answer 2

If you want to put main class in anomaly_model class, you may need to change "class" to "object". And yes, if you want to have anomaly_model a class, you may need to put main method in another object.
In anomaly_model class:
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
class anomaly_model(val inputfile: String, val clusterNum: Int, val maxIterations: Int, val epsilon: Double, val scenarioNum: Int, val outputfile: String){
val conf = new SparkConf().setAppName("Anomaly Model")
val sc = new SparkContext(conf)
val data = sc.textFile(inputfile)
}
And in Main.scala:
import anomaly_model
object Main {
def main(args: Array[String]) : Unit{
val inputfile = "sqlexpt.txt"
val clusterNum = 5
val maxIterations = 1000
val epsilon = 0.001
val scenarioNum = 10
val outputfile = "output.csv"
val am = new anomaly_model(inputfile, clusterNum, maxIterations, epsilon, scenarioNum, outputfile)
}
}

Answer 3

You have a null value in one of your variables, as the message says.

Answer 4

I had the same problem and had made the same mistake as you. anomaly_model should be an object and not a class