java.lang.NoClassDefFoundError when running Scala-Maven project - scala

Trying to run a new project for the first time. mvn install goes smoothly, but when I try to run it, I get:
Exception in thread "main" java.lang.NoClassDefFoundError: scala/Tuple2
Now, something is terribly wrong if it can't even find the Scala basics. To be honest, it doesn't recognize anything outside of the code itself.
I''m attaching my POM file, hope someone can help..
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.foo</groupId>
<artifactId>bar</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>
<name>Foo Bar</name>
<properties>
<app.main.class>com.foo.main</app.main.class>
<spark.version>1.6.1</spark.version>
<scala.version>2.10.4</scala.version>
<scala.binary.version>2.10</scala.binary.version>
</properties>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<repositories>
<repository>
<id>maven-repo</id>
<name>Maven Repository</name>
<url>http://repo1.maven.apache.org/maven2</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>apache-repo</id>
<name>Apache release repo</name>
<url>https://github.com/adatao/mvnrepos/tree/master/releases/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<extensions>
<extension>
<groupId>org.springframework.build</groupId>
<artifactId>aws-maven</artifactId>
<version>5.0.0.RELEASE</version>
</extension>
</extensions>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>com.foo.main</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<jvmArgs>
<jvmArg>-Xms256m</jvmArg>
<jvmArg>-Xmx2048m</jvmArg>
</jvmArgs>
</configuration>
</plugin>
</plugins>
</build>
</project>

Related

Spark Scala : Failed to find data source: kafka

I'm trying to use an example from sparkByExamples.com .
But for some reason, spark program doesn't read data from kafka topic.
Code is here
Error msg below:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of "Structured Streaming + Kafka Integration Guide".;
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:666)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:194)
at com.sparkbyexamples.spark.streaming.kafka.json.SparkStreamingConsumerKafkaJson$.main(SparkStreaming.scala:22)
at com.sparkbyexamples.spark.streaming.kafka.json.SparkStreamingConsumerKafkaJson.main(SparkStreaming.scala)
Process finished with exit code 1
Basically, I have a kafka topic1 < data.json and reading from beginning.
I recently started spark + scala + kafka activities and would appreciate your help.
pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<groupId>com.sparkbyexamples</groupId>
<modelVersion>4.0.0</modelVersion>
<artifactId>spark-scala-examples</artifactId>
<version>1.0-SNAPSHOT</version>
<inceptionYear>2008</inceptionYear>
<packaging>jar</packaging>
<properties>
<scala.version>2.12.12</scala.version>
<spark.version>3.0.0</spark.version>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.specs</groupId>
<artifactId>specs</artifactId>
<version>1.2.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.thoughtworks.xstream</groupId>
<artifactId>xstream</artifactId>
<version>1.4.11</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.12</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-xml_2.11</artifactId>
<version>0.4.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-avro_2.12</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<resources><resource><directory>src/main/resources</directory></resource></resources>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.5</arg>
</args>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-eclipse-plugin</artifactId>
<configuration>
<downloadSources>true</downloadSources>
<buildcommands>
<buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand>
</buildcommands>
<additionalProjectnatures>
<projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature>
</additionalProjectnatures>
<classpathContainers>
<classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer>
<classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer>
</classpathContainers>
</configuration>
</plugin>
</plugins>
</build>
<reporting>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
</plugins>
</reporting>
</project>
As the error indicates, you are missing the spark-sql-kafka-0-10 dependency
Refer docs - https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
And the correct pom file - https://github.com/sparkbyexamples/spark-examples/blob/master/spark-streaming/pom.xml#L64
Unrelated, your spark-xml Scala version needs to be 2.12

spark kafka streaming code not working on livy

I've been working in some scala codes to run spark streaming for kafka data acquisition. Once updated all dependencies and adding new ones to the compiled jar, I am able to to run it with spark-submit with the code spark-submit /codigos/codigos/adquisicion.jar. But when running it via livy2 I'm getting an exception when creating stream in:
val stream = KafkaUtils.createDirectStream[String, String](
ssc,
PreferConsistent,
Subscribe[String, String](topics, kafkaParams)
)
Code used to run it via livy2 is:
uri<-"http://192.168.0.28:8999/batches"
jsonBody2 = paste0('{ "file": "/user/bigdata/adquisicion.jar",
"numExecutors":1,
"executorCores":3,
"name": "Adquisición Datos",
"args": [""],
"className": "codigos.adquisicionDatos" }')
request <- httr::POST(url = uri, body = jsonBody2, httr::add_headers(.headers = c('Content-Type' = 'application/json', 'X-Requested-by'='user')))
And received exception is the following one:
18/11/06 08:01:00 ERROR ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/Consumer
java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/Consumer
at org.apache.spark.streaming.kafka010.ConsumerStrategies$.Subscribe(ConsumerStrategy.scala:256)
at codigos.adquisicionDatos$.main(adquisicionDatos.scala:251)
at codigos.adquisicionDatos.main(adquisicionDatos.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:646)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.Consumer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 8 more
EDIT:
POM file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>codigos</groupId>
<artifactId>adquisicionDatos</artifactId>
<version>1.2</version>
<packaging>jar</packaging>
<name>Adquisicion Datos Kafka</name>
<properties>
<app.main.class>adquisicionDatos.SparkKMeansApp</app.main.class>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<spark.core.version>2.3.2</spark.core.version>
<!--hbase.version>0.98.7-hadoop1</hbase.version-->
<hbase.version>1.0.1</hbase.version>
<slf4j.version>1.7.5</slf4j.version>
<guava.version>13.0.1</guava.version>
<gson.version>2.2.4</gson.version>
<spark.version>2.3.2</spark.version>
</properties>
<repositories>
<repository>
<id>scalanlp.org</id>
<name>ScalaNLP Maven2 Repository</name>
<url>http://repo.scalanlp.org/repo</url>
</repository>
<repository>
<id>apache release</id>
<url>https://repository.apache.org/content/repositories/releases/</url>
</repository>
<repository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
<repository>
<id>cloudera.repo</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
<name>Cloudera Repositories</name>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.sameersingh.scalaplot/scalaplot -->
<dependency>
<groupId>org.sameersingh.scalaplot</groupId>
<artifactId>scalaplot</artifactId>
<version>0.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/javax.mail/javax.mail-api -->
<dependency>
<groupId>javax.mail</groupId>
<artifactId>javax.mail-api</artifactId>
<version>1.5.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!--
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
-->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.2.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.11 -->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>${spark.version}</version>
<!--<version>2.0.5</version>-->
</dependency>
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.1.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.10.2.1</version>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<version>2.15.2</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>scala-test-compile</id>
<phase>process-test-resources</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<finalName>${project.artifactId}-${project.version}-jar-with-dependencies</finalName>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>codigos.adquisicionDatos</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
I compile it in server and then I copy it to docker (via shared volume cp ~/adquisicionDatos/target/adquisicionDatos1.2-jar-with-dependencies.jar /mnt/codigos/adquisicion.jar) and upload it to HDFS using incrond script to automatically upload files when updated (working tested).
If running with spark-submit under root user everything is working fine but using livy or running under livy user is not working and throwing above's error.

Error: Could not find or load main class scala Ecllipse IDE

I have clone the project through my GitHub repository.
Whenever trying to run, it is giving the error:
Could not find main class.
I have my pom.xml as:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>Scalable ML</artifactId>
<groupId>org.big-mint</groupId>
<version>0.0.1-SNAPSHOT</version>
<name>Scalable ML</name>
<description>
Scalable Machine learning algorithms to process big data.
</description>
<inceptionYear>2017</inceptionYear>
<licenses>
<license>
<name>GNU GENERAL PUBLIC LICENSE, Version 3</name>
<url>http://www.gnu.org/licenses/gpl-3.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<properties>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<spark.version>2.1.1</spark.version>
<sansa.version>0.2.0</sansa.version>
<jena.version>3.1.1</jena.version>
</properties>
<dependencies>
<!-- Scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<!-- Apache Spark Core -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Apache Spark SQL -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<!-- Apache JENA 3.x -->
<dependency>
<groupId>org.apache.jena</groupId>
<artifactId>apache-jena-libs</artifactId>
<type>pom</type>
<version>${jena.version}</version>
</dependency>
<!-- SANSA RDF -->
<dependency>
<groupId>net.sansa-stack</groupId>
<artifactId>sansa-rdf-spark-bundle_${scala.binary.version}</artifactId>
<version>${sansa.version}</version>
</dependency>
<!-- SANSA OWL -->
<dependency>
<groupId>net.sansa-stack</groupId>
<artifactId>sansa-owl-spark_${scala.binary.version}</artifactId>
<version>${sansa.version}</version>
</dependency>
<!-- SANSA Inference -->
<!-- <dependency> <groupId>${project.groupId}</groupId> <artifactId>sansa-inference-parent_${scala.binary.version}</artifactId>
<version>${sansa.version}</version> </dependency> -->
<dependency>
<groupId>net.sansa-stack</groupId>
<artifactId>sansa-inference-spark_${scala.binary.version}</artifactId>
<version>${sansa.version}</version>
</dependency>
<!-- SANSA Querying -->
<dependency>
<groupId>net.sansa-stack</groupId>
<artifactId>sansa-query-spark-bundle_${scala.binary.version}</artifactId>
<version>${sansa.version}</version>
</dependency>
<!-- SANSA ML -->
<dependency>
<groupId>net.sansa-stack</groupId>
<artifactId>sansa-ml-spark_${scala.binary.version}</artifactId>
<version>${sansa.version}</version>
</dependency>
<!-- Test -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.binary.version}</artifactId>
<version>2.2.6</version>
<scope>test</scope>
</dependency>
<!-- Logging -->
<dependency>
<groupId>com.typesafe.scala-logging</groupId>
<artifactId>scala-logging-slf4j_${scala.binary.version}</artifactId>
<version>2.1.2</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<!--<arg>-make:transitive</arg> -->
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<recompileMode>incremental</recompileMode>
</configuration>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<excludes>
<exclude>asm:asm</exclude>
<exclude>com.clearspring.analytics:stream</exclude>
<exclude>com.esotericsoftware:kryo*</exclude>
<exclude>com.esotericsoftware:minlog</exclude>
<exclude>com.fasterxml.jackson.core:jackson*</exclude>
<exclude>com.fasterxml.jackson.module:jackson-module*</exclude>
<exclude>com.google.code.findbugs:jsr305</exclude>
<exclude>com.google.code.gson:gson</exclude>
<exclude>com.google.inject.extensions:guice-servlet</exclude>
<exclude>com.google.guava:guava</exclude>
<exclude>com.google.protobuf:protobuf-java</exclude>
<exclude>com.jcraft:jsch</exclude>
<exclude>com.ning:compress-lzf</exclude>
<exclude>com.sun.jersey:jersey-*</exclude>
<exclude>com.sun.jersey.contribs:jersey-guice</exclude>
<exclude>com.sun.xml.bind:jaxb-impl</exclude>
<exclude>com.thoughtworks.paranamer:paranamer</exclude>
<exclude>com.twitter:chill*</exclude>
<exclude>com.univocity:univocity-parsers</exclude>
<exclude>commons-beanutils:commons-beanutils*</exclude>
<exclude>commons-cli:commons-cli</exclude>
<exclude>commons-codec:commons-codec</exclude>
<exclude>commons-collections:commons-collections</exclude>
<exclude>commons-configuration:commons-configuration</exclude>
<exclude>commons-digester:commons-digester</exclude>
<exclude>commons-httpclient:commons-httpclient</exclude>
<exclude>commons-io:commons-io</exclude>
<exclude>commons-lang:commons-lang</exclude>
<exclude>commons-logging:commons-logging</exclude>
<exclude>commons-net:commons-net</exclude>
<exclude>io.dropwizard.metrics:metrics*</exclude>
<exckude>io.netty:netty*</exckude>
<exclude>javax.activation:activation</exclude>
<exclude>javax.annotation:javax.annotation-api</exclude>
<exclude>javax.servlet:javax.servlet-api</exclude>
<exclude>javax.servlet.jsp:jsp-api</exclude>
<exclude>javax.servlet:servlet-api</exclude>
<exclude>javax.validation:validation-api</exclude>
<exclude>javax.ws.rs:javax.ws.rs-api</exclude>
<exclude>javax.xml.bind:jaxb-api</exclude>
<exclude>javax.xml.stream:stax-api</exclude>
<exclude>jdk.tools:jdk.tools</exclude>
<exclude>net.java.dev.jets3t:jets3t</exclude>
<exclude>net.jpountz.lz4:lz4</exclude>
<exclude>net.razorvine:pyrolite</exclude>
<exclude>net.sf.py4j:py4j</exclude>
<exclude>org.antlr:antlr4-runtime</exclude>
<exclude>org.apache.avro:avro*</exclude>
<exclude>org.apache.commons:commons-lang3</exclude>
<exclude>org.apache.commons:commons-math3</exclude>
<exclude>org.apache.commons:commons-compress</exclude>
<exclude>org.apache.curator:curator*</exclude>
<exclude>org.apache.directory.api:*</exclude>
<exclude>org.apache.directory.server:*</exclude>
<exclude>org.apache.hadoop:*</exclude>
<exclude>org.apache.htrace:htrace-core</exclude>
<exclude>org.apache.httpcomponents:*</exclude>
<exclude>org.apache.ivy:ivy</exclude>
<exclude>org.apache.mesos:mesos</exclude>
<exclude>org.apache.parquet:parquet*</exclude>
<exclude>org.apache.spark:*</exclude>
<exclude>org.apache.xbean:xbean-asm5-shaded</exclude>
<exclude>org.apache.zookeeper:zookeeper</exclude>
<exclude>org.codehaus.jackson:jackson-*</exclude>
<exclude>org.codehaus.janino:*</exclude>
<exclude>org.codehaus.jettison:jettison</exclude>
<exclude>org.fusesource.leveldbjni:leveldbjni-all</exclude>
<exckude>org.glassfish.hk2*</exckude>
<exclude>org.glassfish.jersey*</exclude>
<exclude>org.javassist:javassist</exclude>
<exclude>org.json4s:json4s*</exclude>
<exclude>org.mortbay.jetty:jetty*</exclude>
<exclude>org.objenesis:objenesis</exclude>
<exclude>org.roaringbitmap:RoaringBitmap</exclude>
<exclude>org.scala-lang:*</exclude>
<exclude>org.slf4j:jul-to-slf4j</exclude>
<exclude>org.slf4j:jcl-over-slf4j</exclude>
<exclude>org.spark-project.spark:unused</exclude>
<exclude>org.xerial.snappy:snappy-java</exclude>
<exclude>oro:oro</exclude>
<exclude>xmlenc:xmlenc</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
</execution>
</executions>
</plugin>
<!-- disable surefire -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7</version>
<configuration>
<useFile>false</useFile>
<disableXmlReport>true</disableXmlReport>
<!-- If you have classpath issue like NoDefClassError,... -->
<!-- useManifestOnlyJar>false</useManifestOnlyJar -->
<includes>
<include>**/*Test.*</include>
<include>**/*Suite.*</include>
</includes>
<skipTests>true</skipTests>
</configuration>
</plugin>
<!-- enable scalatest -->
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>WDF TestSuite.txt</filereports>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>oss-sonatype</id>
<name>oss-sonatype</name>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>apache-snapshot</id>
<name>Apache repository (snapshots)</name>
<url>https://repository.apache.org/content/repositories/snapshots/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>maven.aksw.internal</id>
<name>AKSW Release Repository</name>
<url>http://maven.aksw.org/archiva/repository/internal</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>maven.aksw.snapshots</id>
<name>AKSW Snapshot Repository</name>
<url>http://maven.aksw.org/archiva/repository/snapshots</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
Have several maven and other dependencies.
I Don't know why it is not loading the main class.
I have tried many solutions as mentioned by others i.e. in run configuration giving complete class name and etc. But nothing worked.
try to add this property :
<properties>
<jobMainClass>package.mainclass</jobMainClass>
</properties>
If it doesn't work, give me the command that you use to run your jarfile.

Maven Build Works, During Deployment Gets Class Not Found

Using Eclipse Mars.2 4.5.2
The Maven build is successful. During deployment I get the following error:
java.lang.ClassNotFoundException: com.authy.AuthyApiClient from [Module \"deployment.mavenFlowChartDB5.war:main\" from Service Module Loader]"}}
In the Maven dependency directory I see the related dependency with the class underneath the dependency entry.
Libraries>Maven Dependencies>authy-java-1.1.0.jar>com.auth.AuthyApiCLient
Why doesn't the Deployment see this missing class?
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.mycompany</groupId>
<artifactId>mavenFlowChartDB5</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>war</packaging>
<name>mavenFlowChartDB5</name>
<properties>
<endorsed.dir>${project.build.directory}/endorsed</endorsed.dir>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.primefaces</groupId>
<artifactId>primefaces</artifactId>
<version>6.0</version>
</dependency>
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.3</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.1</version>
</dependency>
<dependency>
<groupId>org.eclipse.persistence</groupId>
<artifactId>eclipselink</artifactId>
<version>2.5.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.eclipse.persistence</groupId>
<artifactId>org.eclipse.persistence.jpa.modelgen.processor</artifactId>
<version>2.5.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-web-api</artifactId>
<version>7.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.2-1002-jdbc4</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.8.2</version>
</dependency>
<dependency>
<groupId>org.jboss.ejb3</groupId>
<artifactId>jboss-ejb3-ext-api</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>com.authy</groupId>
<artifactId>authy-java</artifactId>
<version>1.1.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
<compilerArguments>
<endorseddirs>${endorsed.dir}</endorseddirs>
</compilerArguments>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.3</version>
<configuration>
<failOnMissingWebXml>false</failOnMissingWebXml>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<phase>validate</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<outputDirectory>${endorsed.dir}</outputDirectory>
<silent>true</silent>
<artifactItems>
<artifactItem>
<groupId>javax</groupId>
<artifactId>javaee-endorsed-api</artifactId>
<version>7.0</version>
<type>jar</type>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<url>http://repository.primefaces.org/</url>
<id>PrimeFaces-maven-lib</id>
<layout>default</layout>
<name>Repository for library PrimeFaces-maven-lib</name>
</repository>
<repository>
<id>jboss-public-repository-group</id>
<name>JBoss Public Maven Repository Group</name>
<url>https://repository.jboss.org/nexus/content/groups/public-jboss/</url>
<layout>default</layout>
<releases>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
</releases>
<snapshots>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
</snapshots>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>jboss-public-repository-group</id>
<name>JBoss Public Maven Repository Group</name>
<url>https://repository.jboss.org/nexus/content/groups/public-jboss/</url>
<layout>default</layout>
<releases>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
</releases>
<snapshots>
<enabled>true</enabled>
<updatePolicy>never</updatePolicy>
</snapshots>
</pluginRepository>
</pluginRepositories>
</project>
This is the updated pom.xml. I still get the same error. I did notice that the maven-compiler-plugin is repeated a few times. I added this part.
<manifest>
<addClasspath>true</addClasspath>
</manifest>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
</manifest>
</archive>
<source>1.8</source>
<target>1.8</target>
<compilerArguments>
<endorseddirs>${endorsed.dir}</endorseddirs>
</compilerArguments>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<version>2.3</version>
<configuration>
<failOnMissingWebXml>false</failOnMissingWebXml>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<phase>validate</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<outputDirectory>${endorsed.dir}</outputDirectory>
<silent>true</silent>
<artifactItems>
<artifactItem>
<groupId>javax</groupId>
<artifactId>javaee-endorsed-api</artifactId>
<version>7.0</version>
<type>jar</type>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
This is probably because you are not adding classpath entry in manifest for mavenFlowChartDB5 war. Could you try adding that using this documentation.

Unable to execute Spray Examples | Maven dependency error

I am just trying to have hands on Spray library however am not able to execute its examples due to maven dependency.
As provided in docs I have defined Spray repository as follows:
<repositories>
<repository>
<id>sprayrepo</id>
<name>spray repo</name>
<url>http://repo.spray.io/</url>
<layout>default</layout>
</repository>
</repositories>
Then I have defined its dependency as follows:
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-http</artifactId>
<version>1.1-M7</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-httpx</artifactId>
<version>1.1-M7</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-util</artifactId>
<version>1.1-M7</version>
</dependency>
<dependency>
<groupId>com.chuusai</groupId>
<artifactId>shapeless_2.9.2</artifactId>
<version>1.2.0</version>
</dependency>
</dependencies>
However when I try to execute to maven install, I face following error:
[ERROR] Failed to execute goal on project spray-test1: Could not
resolve dependencies for project
org.spray:spray-test1:jar:1.0-SNAPSHOT: Failed to collect dependencies
for [io.spray:spray-http:jar:1.1-M7 (compile),
io.spray:spray-httpx:jar:1.1-M7 (compile),
io.spray:spray-util:jar:1.1-M7 (compile),
com.chuusai:shapeless_2.9.2:jar:1.2.0 (compile)]: Failed to read
artifact descriptor for io.spray:spray-http:jar:1.1-M7: Could not
transfer artifact io.spray:spray-http:pom:1.1-M7 from/to sprayrepo
(http://repo.spray.io/): connection timed out to
http://repo.spray.io/io/spray/spray-http/1.1-M7/spray-http-1.1-M7.pom
-> [Help 1] [ERROR]
Any Pointers or suggestions will be highly appreciated.
Thanks
Try this:
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.tools.version>2.10</scala.tools.version>
<scala.version>2.10.2</scala.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>io.spray</groupId>
<artifactId>spray-can</artifactId>
<version>1.1-M8</version>
</dependency>
<!-- Test -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.specs2</groupId>
<artifactId>specs2_${scala.tools.version}</artifactId>
<version>1.13</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.tools.version}</artifactId>
<version>2.0.M6-SNAP8</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<!-- see http://davidb.github.com/scala-maven-plugin -->
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.3</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-make:transitive</arg>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.13</version>
<configuration>
<useFile>false</useFile>
<disableXmlReport>true</disableXmlReport>
<!-- If you have classpath issue like NoDefClassError,... -->
<!-- useManifestOnlyJar>false</useManifestOnlyJar -->
<includes>
<include>**/*Test.*</include>
<include>**/*Suite.*</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>SonatypeNexusSnapshots</id>
<name>Sonatype Nexus Snapshots</name>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
</repository>
<repository>
<id>FuseSourceSnapshotRepository</id>
<name>FuseSource Snapshot Repository</name>
<url>http://repo.fusesource.com/nexus/content/repositories/snapshots/</url>
</repository>
<repository>
<id>ScalaToolsMaven2Repository</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases/</url>
</repository>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
<repository>
<id>sprayrepo</id>
<name>spray repo</name>
<url>http://repo.spray.io/</url>
<layout>default</layout>
</repository>
</repositories>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>