Spark Maven Project not running - eclipse

Question

I have recently downloaded Scala IDE for Eclipse for Spark/Scala project. For that, I have created a Maven project & added Spark-core dependencies. I then wrote a basic Spark code to create an RDD out of array & finally collect it. Upon running the code, it is giving me the error "java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/paranamer/ParanamerAnnotationIntrospector" on the console.
Spark Code :
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object Test {
def main(args : Array[String]) : Unit = {
println("Begining of the code")
val sparkConf = new SparkConf()
sparkConf.setAppName("New app")
sparkConf.setMaster("local")
val sc = new SparkContext(sparkConf)
val array = Array(1,2,3,4,5)
val arrayRDD = sc.parallelize(array, 2)
arrayRDD.collect
println("Ending of the code")
}
}
Maven Dependency :
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.talentorigin</groupId>
<artifactId>SparkCourse</artifactId>
<version>0.0.1-SNAPSHOT</version>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>2.4.3</version>
</dependency>
</dependencies>
</project>
Console Error :
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/paranamer/ParanamerAnnotationIntrospector
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
at org.apache.spark.SparkContext.withScope(SparkContext.scala:699)
at org.apache.spark.SparkContext.parallelize(SparkContext.scala:716)
at Test$.main(Test.scala:17)
at Test.main(Test.scala)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.module.paranamer.ParanamerAnnotationIntrospector
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 6 more
Problem Tag of Eclipse :
Failure to transfer com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9 from https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
Failure to transfer net.sf.py4j:py4j:jar:0.10.7 from https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced.
Missing artifact com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.7.9
Missing artifact net.sf.py4j:py4j:jar:0.10.7
The container 'Maven Dependencies' references non existing library 'C:\Users\AniRudh\.m2\repository\com\fasterxml\jackson\module\jackson-module-paranamer\2.7.9\jackson-module-paranamer-2.7.9.jar'
I am a beginner in working with Maven.
Kindly help me resolving this issue.

Answer 1

I am sharing the complete pom file. You can update your pom file accordingly.
Include the following for the build to work:
Dependencies spark, scala others
Plugins
Sample spark-scala-maven project template Github link
Here is the complete pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.techknowera</groupId>
<artifactId>spark-batch</artifactId>
<version>1.0-SNAPSHOT</version>
<name>${project.artifactId}</name>
<description>Spark Batch And Streaming Application</description>
<inceptionYear>2019</inceptionYear>
<properties>
<scala.version>2.11</scala.version>
<scala.full.version>2.11.8</scala.full.version>
<spark.version>2.2.0</spark.version>
<java.version>1.8</java.version>
<jackson.version>2.6.5</jackson.version>
<scala.maven.plugin.version>3.2.2</scala.maven.plugin.version>
<maven.surefire.plugin.version>2.13</maven.surefire.plugin.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<!--Scala dependencies-->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.full.version}</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.9</version>
<scope>test</scope>
</dependency>
<!-- https://mvnrepository.com/artifact/org.scalatest/scalatest -->
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.version}</artifactId>
<version>3.0.5</version>
</dependency>
<!--Apache Spark Dependencies-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-yarn -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_${scala.version}</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-xml_${scala.version}</artifactId>
<version>0.4.1</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-deploy-plugin</artifactId>
<version>2.8.2</version>
</plugin>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
<!-- disable surefire -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7</version>
<configuration>
<skipTests>true</skipTests>
</configuration>
</plugin>
<!-- enable scala test -->
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>2.0.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>TestSuite.txt</filereports>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<id>jar-with-dependencies</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>