Getting java.lang.ClassNotFoundException while running Spark Application - scala

Question

I am new to Spark (Scala) and I am trying to run one spark application through spark submit. Unfortunately I am getting java.lang.ClassNotFoundException exception.
Here is my spark submit command:
./spark-submit --class "spark.phoenix.a" --master local --deploy-mode client /home/ec2-user/phoenix-0.0.1-SNAPSHOT.jar
Here is my Exception:
java.lang.ClassNotFoundException: spark.phoenix.a
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Here is my sample code:
package spark.phoenix
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext
object a {
def main(args: Array[String]) {
import org.apache.phoenix.spark._
import org.apache.spark.SparkContext
val zkQuorum = "localhost:2181"
val master = "local[*]"
val sparkConf = new SparkConf()
sparkConf.setAppName("phoenix-spark-save")
.setMaster(s"${master}")
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
// read from orders phoenix table
val df = sqlContext.phoenixTableAsDataFrame("TABLE1",
Seq.apply("ID", "COL1"),
zkUrl = Some.apply(s"${zkQuorum}") )
}
}

Answer 1

I think problem is with your pom.xml file(if you are using maven). You haven't added maven-shade-plugin.
Add below properties to your pom.xml and and try once.
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
</resources>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<id>eclipse-add-source</id>
<goals>
<goal>add-source</goal>
</goals>
</execution>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile-first</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
<execution>
<id>attach-scaladocs</id>
<phase>verify</phase>
<goals>
<goal>doc-jar</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<recompileMode>incremental</recompileMode>
<useZincServer>true</useZincServer>
<args>
<arg>-unchecked</arg>
<arg>-deprecation</arg>
<arg>-feature</arg>
</args>
<jvmArgs>
<jvmArg>-Xms1024m</jvmArg>
<jvmArg>-Xmx1024m</jvmArg>
<jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>
</jvmArgs>
<javacArgs>
<javacArg>-source</javacArg>
<javacArg>${java.version}</javacArg>
<javacArg>-target</javacArg>
<javacArg>${java.version}</javacArg>
<javacArg>-Xlint:all,-serial,-path</javacArg>
</javacArgs>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>org.xerial.snappy</exclude>
<exclude>org.scala-lang.modules</exclude>
<exclude>org.scala-lang</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
<shadedPattern>shaded.com.google.common</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Answer 2

please remove the "" from your command.
Try something like this.
./spark-submit --class spark.phoenix.a --master local --deploy-mode client /home/ec2-user/phoenix-0.0.1-SNAPSHOT.jar