When I ran clean test or
clean test -DsuiteFile=src/main/resources/testng/SmokeTestSuite.xml in eclipse
following is the output.No test running but Build is success.
I have been struggling for this for quite a few weeks.Would be great if someone can explain me what was the issue and a proper solution.
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building eselenium-ellis-tests 0.0.1
[INFO] ------------------------------------------------------------------------
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Downloading: https://devops-tools.pearson.com/nexus-master/content/repositories/releases/com/pearson/eselenium-pageobjects-ellis/1.0.1-SNAPSHOT/maven-metadata.xml
Downloading: https://devops-tools.pearson.com/nexus-deps/content/groups/all-deps/com/pearson/eselenium-pageobjects-ellis/1.0.1-SNAPSHOT/maven-metadata.xml
Downloaded: https://devops-tools.pearson.com/nexus-deps/content/groups/all-deps/com/pearson/eselenium-pageobjects-ellis/1.0.1-SNAPSHOT/maven-metadata.xml (788 B at 0.2 KB/sec)
Downloading: https://devops-tools.pearson.com/nexus-deps/content/groups/all-deps/com/pearson/eselenium-pageobjects-ellis/1.0.1-SNAPSHOT/maven-metadata.xml
Downloading: https://devops-tools.pearson.com/nexus-master/content/repositories/releases/com/pearson/eselenium-pageobjects-ellis/1.0.1-SNAPSHOT/maven-metadata.xml
Downloaded: https://devops-tools.pearson.com/nexus-deps/content/groups/all-deps/com/pearson/eselenium-pageobjects-ellis/1.0.1-SNAPSHOT/maven-metadata.xml (788 B at 0.2 KB/sec)
[INFO]
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) # eselenium-ellis-tests ---
[INFO] Deleting C:\ATDD\workspace\galaxy-tests\target
[INFO]
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) # eselenium-ellis-tests ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 8 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) # eselenium-ellis-tests ---
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
[INFO] Compiling 19 source files to C:\ATDD\workspace\galaxy-tests\target\classes
[INFO]
[INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) # eselenium-ellis-tests ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) # eselenium-ellis-tests ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) # eselenium-ellis-tests ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 21.003s
[INFO] Finished at: Sun Oct 19 09:48:48 IST 2014
[INFO] Final Memory: 20M/153M
[INFO] ------------------------------------------------------------------------
AND My POM file looks like below.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.pearson</groupId>
<artifactId>eselenium-ellis-tests</artifactId>
<version>0.0.1</version>
<properties>
<suiteXmlFile>${suiteFile}</suiteXmlFile>
</properties>
<dependencies>
<dependency>
<groupId>com.pearson</groupId>
<artifactId>eselenium.framework</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>com.pearson</groupId>
<artifactId>eselenium-pageobjects-ellis</artifactId>
<version>1.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>6.5.2</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<suiteXmlFiles>
<suiteXmlFile>${suiteFile}</suiteXmlFile>
<!--<suiteXmlFile>src/main/resources/testng/SmokeTestSuite.xml</suiteXmlFile> -->
</suiteXmlFiles>
<reportsDirectory>test-output</reportsDirectory>
<reportsDirectory>test-output\archive\${timestamp}</reportsDirectory>
<systemProperties>
<property>
<name>browser.NAME</name>
<value>${browser.NAME}</value>
</property>
</systemProperties>
</configuration>
</plugin>
</plugins>
</build>
Your issue is actually not being able to provide pom.xml value from command prompt. Its a duplicate of
Passing command line arguments from Maven as properties in pom.xml
you will find solution there. You need to double quote your parameter.
Related
This POM.XML produces the following error on line 55:
Plugin execution not covered by lifecycle configuration: com.peterlavalle:sablecc-maven-plugin:3.7:generate (execution: default, phase: generate-sources)
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.conserveestate</groupId>
<artifactId>EstateAccountingInExLoader</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>EstateAccountingInExLoader</name>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>com.peterlavalle</groupId>
<artifactId>sablecc-maven-plugin</artifactId>
<version>3.7</version>
<configuration>
<sourceDirectory>${basedir}/src/main/resources</sourceDirectory>
<outputDirectory>${basedir}/src/main/gen-java</outputDirectory>
</configuration>
<executions>
<execution>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>com.peterlavalle</groupId>
<artifactId>sablecc-maven-plugin</artifactId>
<version>3.7</version>
<configuration>
<sourceDirectory>${basedir}/src/main/resources</sourceDirectory>
<outputDirectory>${basedir}/src/main/gen-java</outputDirectory>
</configuration>
<executions>
<execution> <!-- Line 55: Error - Plugin execution not covered by lifecycle configuration -->
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
The error seems to indicate that the plug in is expecting a generate-sources goal instead of generate. If I change the goal to generate-sources, Eclipse removes the red X in the XML file and no errors are displayed, but... When I build the project, I get this error:
Could not find goal 'generate-sources' in plugin com.peterlavalle:sablecc-maven-plugin:3.7 among available goals generate
So this looks like a contradiction to me. I specify a goal of generate I get a message saying the plugin expects generate-sources but if I change the goal to generate-sources then I get a message which seems to indicate the plugin is expecting a goal of generate.
The plugin invokes SableCC (a compiler compiler) which reads a grammar file and generates Java source code that gets compiled along with my code that invokes the generated parser.
What am I doing wrong?
UPDATE
As per khmarbaise's suggestion, I tried this from the command line. Here are those results with a goal of generate-sources:
C:\Users\mbmas_000\workspace\EstateAccountingInExLoader>mvn clean install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.conserveestate:EstateAccountingInExLoader:jar:0.0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. # line 38, column 12
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] -----------< com.conserveestate:EstateAccountingInExLoader >------------
[INFO] Building EstateAccountingInExLoader 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.364 s
[INFO] Finished at: 2020-06-03T09:33:28-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Could not find goal 'generate-sources' in plugin com.peterlavalle:sablecc-maven-plugin:3.7 among available goals generate -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoNotFoundException
C:\Users\mbmas_000\workspace\EstateAccountingInExLoader>
And again with a goal of generate. This time it works.
C:\Users\mbmas_000\workspace\EstateAccountingInExLoader>mvn clean install
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.conserveestate:EstateAccountingInExLoader:jar:0.0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. # line 38, column 12
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] -----------< com.conserveestate:EstateAccountingInExLoader >------------
[INFO] Building EstateAccountingInExLoader 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # EstateAccountingInExLoader ---
[INFO] Deleting C:\Users\mbmas_000\workspace\EstateAccountingInExLoader\target
[INFO]
[INFO] --- sablecc-maven-plugin:3.7:generate (default) # EstateAccountingInExLoader ---
-- Generating parser for Scraper.grammar in C:\Users\mbmas_000\workspace\EstateAccountingInExLoader\src\main\gen-java
Adding productions and alternative of section AST.
Verifying identifiers.
Verifying ast identifiers.
Adding empty productions and empty alternative transformation if necessary.
Adding productions and alternative transformation if necessary.
computing alternative symbol table identifiers.
Verifying production transform identifiers.
Verifying ast alternatives transform identifiers.
Generating token classes.
Generating production classes.
Generating alternative classes.
Generating analysis classes.
Generating utility classes.
Generating the lexer.
State: INITIAL
- Constructing NFA.
.............................................................................................................................................................................................................................................................................................................................................
- Constructing DFA.
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
................................................................................................................................................................................
- resolving ACCEPT states.
Generating the parser.
.......................................................................................
.......................................................................................
.......................................................................................
..
.......................................................................................
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # EstateAccountingInExLoader ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # EstateAccountingInExLoader ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is platform dependent!
[INFO] Compiling 65 source files to C:\Users\mbmas_000\workspace\EstateAccountingInExLoader\target\classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # EstateAccountingInExLoader ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # EstateAccountingInExLoader ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) # EstateAccountingInExLoader ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) # EstateAccountingInExLoader ---
[INFO] Building jar: C:\Users\mbmas_000\workspace\EstateAccountingInExLoader\target\EstateAccountingInExLoader-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) # EstateAccountingInExLoader ---
[INFO] Installing C:\Users\mbmas_000\workspace\EstateAccountingInExLoader\target\EstateAccountingInExLoader-0.0.1-SNAPSHOT.jar to C:\Users\mbmas_000\.m2\repository\com\conserveestate\EstateAccountingInExLoader\0.0.1-SNAPSHOT\EstateAccountingInExLoader-0.0.1-SNAPSHOT.jar
[INFO] Installing C:\Users\mbmas_000\workspace\EstateAccountingInExLoader\pom.xml to C:\Users\mbmas_000\.m2\repository\com\conserveestate\EstateAccountingInExLoader\0.0.1-SNAPSHOT\EstateAccountingInExLoader-0.0.1-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.091 s
[INFO] Finished at: 2020-06-03T09:30:32-07:00
[INFO] ------------------------------------------------------------------------
C:\Users\mbmas_000\workspace\EstateAccountingInExLoader>
I have a huge feeling that I'm missing something.
I'm trying to compile Java&Scala project at my work. Luckliy I was able to repuduce my problem on a simple dummy project.
I'm using scala-maven-plugin(https://github.com/davidB/scala-maven-plugin). I'm also base my example on this question
Aiming to solve Scala&Java multi-module project. I have root and two modules m1 and m2.
m1 pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>prj_multi_modules</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>m1</artifactId>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<dependency>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>m2</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
m2 pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>prj_multi_modules</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>m2</artifactId>
<packaging>jar</packaging>
<properties>
<cdh.version>cdh5.7.6</cdh.version>
<build.prop.dir>..</build.prop.dir>
<scala.version>2.11.8</scala.version>
<scala.version.major>2.11</scala.version.major>
<spark.version>2.3.0.cloudera4</spark.version>
<hadoop.version>2.6.0-${cdh.version}</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- remove this for spark 2.3? -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.github.scopt</groupId>
<artifactId>scopt_${scala.version.major}</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_${scala.version.major}</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_${scala.version.major}</artifactId>
<version>1.6.0_0.7.4</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
root pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>prj_multi_modules</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<properties>
<cdh.version>cdh5.7.6</cdh.version>
<build.prop.dir>..</build.prop.dir>
<scala.version>2.11.8</scala.version>
<scala.version.major>2.11</scala.version.major>
<spark.version>2.3.0.cloudera4</spark.version>
<hadoop.version>2.6.0-${cdh.version}</hadoop.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<modules>
<module>m1</module>
<module>m2</module>
</modules>
</project>
My goal is very simple I want m1 project to get all spark dpedencies from m2 module in order to compile a spark code in m1 module.
This is the spark code in m1:
package p1
import org.apache.spark.sql.SparkSession
class MyClass(spark: SparkSession) {
import spark.implicits._
def run(input: String) = {
val df = spark.read.parquet(input)
df.show()
}
}
object MyClass {
case class Params(input: String = "")
def main(args: Array[String]): Unit = {
val parser = new scopt.OptionParser[Params](usageHeader) {
opt[String]('i', "input") required() action { (s, params) => params.copy(input = s) }
}
parser.parse(args, Params()) match {
case Some(p) =>
val spark = SparkSession.builder
.appName("spark2 example")
.getOrCreate()
val job = new Spark2Exp(spark)
job.run(p.input)
spark.stop()
case None =>
System.exit(1)
}
}
val usageHeader: String = "n/a"
}
m2 compilation is working smoothly. While compiling m1 all java compilation process is good but when trying to compile scala code it fails not finding spark dependencies as if it doesn't transient between the dependent modules.
In addition if I copy all m2 spark dependencies to m1 pom everything is working.
I can't seem to understand what I'm doing wrong. I am sure it's scala - maven issue since this scenario is pretty straight forward in java.
Adding compilation error:
"C:\Program Files\Java\jdk1.8.0_151\bin\java" -Dmaven.multiModuleProjectDirectory=C:\dev\scala-maven-plugin "-Dmaven.home=C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\plugins\maven\lib\maven3" "-Dclassworlds.conf=C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\plugins\maven\lib\maven3\bin\m2.conf" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\lib\idea_rt.jar=55662:C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\plugins\maven\lib\maven3\boot\plexus-classworlds-2.5.2.jar" org.codehaus.classworlds.Launcher -Didea.version=2017.3 clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] prj_multi_modules
[INFO] m2
[INFO] m1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building prj_multi_modules 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # prj_multi_modules ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) # prj_multi_modules ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (default) # prj_multi_modules ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) # prj_multi_modules ---
[INFO] Installing C:\dev\scala-maven-plugin\samples\prj_multi_modules\pom.xml to C:\Users\raphael.peretz\.m2\repository\samples\scala-maven-plugin\prj_multi_modules\0.0.1-SNAPSHOT\prj_multi_modules-0.0.1-SNAPSHOT.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building m2 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # m2 ---
[INFO] Deleting C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # m2 ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # m2 ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) # m2 ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] samples.scala-maven-plugin:m2:0.0.1-SNAPSHOT requires scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.8
[WARNING] org.apache.spark:spark-core_2.11:2.3.0.cloudera4 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-ast_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # m2 ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # m2 ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (default) # m2 ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] samples.scala-maven-plugin:m2:0.0.1-SNAPSHOT requires scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.8
[WARNING] org.apache.spark:spark-core_2.11:2.3.0.cloudera4 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-ast_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) # m2 ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) # m2 ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\target\m2-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) # m2 ---
[INFO] Installing C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\target\m2-0.0.1-SNAPSHOT.jar to C:\Users\raphael.peretz\.m2\repository\samples\scala-maven-plugin\m2\0.0.1-SNAPSHOT\m2-0.0.1-SNAPSHOT.jar
[INFO] Installing C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\pom.xml to C:\Users\raphael.peretz\.m2\repository\samples\scala-maven-plugin\m2\0.0.1-SNAPSHOT\m2-0.0.1-SNAPSHOT.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building m1 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # m1 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # m1 ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # m1 ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) # m1 ---
[INFO] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala:-1: info: compiling
[INFO] Compiling 1 source files to C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\target\classes at 1558957820236
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:3: error: object spark is not a member of package org.apache
[ERROR] import org.apache.spark.sql.SparkSession
[ERROR] ^
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:5: error: not found: type SparkSession
[ERROR] class MyClass(spark: SparkSession) {
[ERROR] ^
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:29: error: not found: value SparkSession
[ERROR] val spark = SparkSession.builder
[ERROR] ^
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:33: error: not found: type Spark2Exp
[ERROR] val job = new Spark2Exp(spark)
[ERROR] ^
[ERROR] four errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] prj_multi_modules .................................. SUCCESS [ 1.435 s]
[INFO] m2 ................................................. SUCCESS [ 22.342 s]
[INFO] m1 ................................................. FAILURE [ 3.497 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 27.379 s
[INFO] Finished at: 2019-05-27T14:50:23+03:00
[INFO] Final Memory: 52M/927M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (default) on project m1: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :m1
Process finished with exit code 1
Since you need spark in both m1 and m2, why not delcare the spark dependency in root pom file?
Of course, you can remove the duplicate one in m2 afterwards.
I am trying to build maven project and it is not creating class file,but it is generating a jar file.
I have done clean then install and then package.
Below is my pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>user.data.store</groupId>
<artifactId>UserProcess</artifactId>
<version>0.0.1-SNAPSHOT</version>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.7</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>1.2.0-cdh5.15.1</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-common</artifactId>
<version>1.2.0-cdh5.15.1</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<testSourceDirectory>src/test/scala</testSourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.3</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<args>
<arg>-dependencyfile</arg>
<arg>${project.build.directory}/.scala_dependencies</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.13</version>
<configuration>
<useFile>false</useFile>
<disableXmlReport>true</disableXmlReport>
<includes>
<include>**/*Test.*</include>
<include>**/*Suite.*</include>
</includes>
</configuration>
</plugin>
</plugins>
</build>
</project>
Note - I dont have the test folder, and still my build is successfull but getting below in maven console:
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for user.data.store:UserProcess:jar:0.0.1-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hbase:hbase-common:jar -> duplicate declaration of version 1.2.0-cdh5.15.1 # line 172, column 17
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building UserProcess 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-core_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 0.8 KB/sec)
[INFO] Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-parent_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-parent_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (827 B at 1.3 KB/sec)
[INFO] Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/cdh/spark2-root/2.0.0.cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloading: https://repository.cloudera.com/artifactory/libs-snapshot-local/com/cloudera/cdh/spark2-root/2.0.0.cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/cdh/spark2-root/2.0.0.cloudera1-SNAPSHOT/maven-metadata.xml (607 B at 0.9 KB/sec)
[INFO] Downloaded: https://repository.cloudera.com/artifactory/libs-snapshot-local/com/cloudera/cdh/spark2-root/2.0.0.cloudera1-SNAPSHOT/maven-metadata.xml (607 B at 0.4 KB/sec)
[INFO] Downloading: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-launcher_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-launcher_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-launcher_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 2.1 KB/sec)
[INFO] Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-launcher_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 2.1 KB/sec)
[INFO] Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-network-common_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloading: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-network-common_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-network-common_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 2.1 KB/sec)
[INFO] Downloaded: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-network-common_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 2.1 KB/sec)
[INFO] Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-network-shuffle_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloading: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-network-shuffle_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-network-shuffle_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 2.1 KB/sec)
[INFO] Downloaded: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-network-shuffle_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 2.1 KB/sec)
[INFO] Downloading: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-unsafe_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-unsafe_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-unsafe_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 1.7 KB/sec)
[INFO] Downloaded: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/spark/spark-unsafe_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (2 KB at 1.7 KB/sec)
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # UserProcess ---
[INFO] Deleting E:\workspace\UserProcess\target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # UserProcess ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory E:\workspace\UserProcess\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # UserProcess ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- scala-maven-plugin:3.1.3:compile (default) # UserProcess ---
[INFO] Downloading: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-parent_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml
[INFO] Downloaded: https://repository.cloudera.com/artifactory/libs-snapshot-local/org/apache/spark/spark-parent_2.10/2.0.0-cloudera1-SNAPSHOT/maven-metadata.xml (827 B at 1.1 KB/sec)
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] user.data.store:UserProcess:0.0.1-SNAPSHOT requires scala version: 2.11.7
[WARNING] org.scala-lang:scala-reflect:2.10.6 requires scala version: 2.10.6
[WARNING] Multiple versions of scala libraries detected!
[INFO] E:\workspace\UserProcess\src\main\scala:-1: info: compiling
[INFO] Compiling 3 source files to E:\workspace\UserProcess\target\classes at 1543295814300
[WARNING] warning: there were 5 deprecation warnings; re-run with -deprecation for details
[WARNING] one warning found
[INFO] prepare-compile in 0 s
[INFO] compile in 7 s
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # UserProcess ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory E:\workspace\UserProcess\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # UserProcess ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.1.3:testCompile (default) # UserProcess ---
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] user.data.store:UserProcess:0.0.1-SNAPSHOT requires scala version: 2.11.7
[WARNING] org.scala-lang:scala-reflect:2.10.6 requires scala version: 2.10.6
[WARNING] Multiple versions of scala libraries detected!
[WARNING] No source files found.
[INFO]
[INFO] --- maven-surefire-plugin:2.13:test (default-test) # UserProcess ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) # UserProcess ---
[INFO] Building jar: E:\workspace\UserProcess\target\UserProcess-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) # UserProcess ---
[INFO] Installing E:\workspace\UserProcess\target\UserProcess-0.0.1-SNAPSHOT.jar to C:\Users\kumar.harsh\.m2\repository\user\data\store\UserProcess\0.0.1-SNAPSHOT\UserProcess-0.0.1-SNAPSHOT.jar
[INFO] Installing E:\workspace\UserProcess\pom.xml to C:\Users\kumar.harsh\.m2\repository\user\data\store\UserProcess\0.0.1-SNAPSHOT\UserProcess-0.0.1-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36.405 s
[INFO] Finished at: 2018-11-27T10:47:03+05:30
[INFO] Final Memory: 59M/738M
[INFO] ------------------------------------------------------------------------
I tried deleting target folder manually and then build again, but getting same scenario again.
I can see the class file in target/class folder
Getting below on console while running scala code:
Error: Could not find or load main class user.UserData
I think you are mixing things up. As David Bernard mentioned there should be a target/classes folder with the compiles classes from src/main/scala/
Now to:
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # UserProcess ---
[INFO] No sources to compile
Thats just how maven works. In the different lifecycle phases there are default plugins bound to specific phases. So now when you enter the test-compile phase, the maven-compiler-plugin will get executed with goal testCompile.
Recently in our hadoop project, we added maven-surefire-plugin for running unit tests by categories. Related blocks in hadoop root pom.xml file as below:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
<configuration>
<groups>${test.groups}</groups>
</configuration>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.21.0</version>
</dependency>
</dependencies>
</plugin>
<profile>
<id>flakies</id>
<properties>
<test.groups>org.apache.hadoop.classification.TestJUnitCategory$FlakiesTest</test.groups>
</properties>
</profile>
<profile>
<id>stables</id>
<properties>
<test.groups>org.apache.hadoop.classification.TestJUnitCategory$StablesTest</test.groups>
</properties>
</profile>
<profile>
<id>integrations</id>
<properties>
<test.groups>org.apache.hadoop.classification.TestJUnitCategory$IntegrationsTest</test.groups>
</properties>
</profile>
<profile>
<id>none</id>
<properties>
<test.groups>!org.apache.hadoop.classification.TestJUnitCategory$AllTest</test.groups>
</properties>
</profile>
Invoking tests by category works fine: mvn -P integrations.
However, our old fashioned unit test execution got skipped by this maven-surefire-plugin:
13:16:35 [INFO] --- maven-antrun-plugin:1.7:run (create-log-dir) # hadoop-hdfs ---
13:16:35 [INFO] Executing tasks
13:16:35
13:16:35 main:
13:16:35 [delete] Deleting directory /tmp/tmp.o7flMN8505/src/ghuang/ghuang/hadoop-hdfs-project/hadoop-hdfs/target/test/data
13:16:35 [mkdir] Created dir: /tmp/tmp.o7flMN8505/src/ghuang/ghuang/hadoop-hdfs-project/hadoop-hdfs/target/test/data
13:16:35 [copy] Copying 16 files to /tmp/tmp.o7flMN8505/src/ghuang/ghuang/hadoop-hdfs-project/hadoop-hdfs/target/test-classes/webapps
13:16:35 [INFO] Executed tasks
13:16:35 [INFO]
13:16:35 [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # hadoop-hdfs ---
13:16:35 [INFO] Nothing to compile - all classes are up to date
13:16:35 [INFO]
13:16:35 [INFO] --- maven-surefire-plugin:2.20:test (default-test) # hadoop-hdfs ---
13:16:35 [INFO] Surefire report directory: /tmp/tmp.o7flMN8505/src/ghuang/ghuang/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
13:16:35 [INFO] parallel='none', perCoreThreadCount=true, threadCount=0, useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, threadCountMethods=0, parallelOptimized=true
13:16:35 [INFO]
13:16:35 [INFO] -------------------------------------------------------
13:16:35 [INFO] T E S T S
13:16:35 [INFO] -------------------------------------------------------
13:26:01 [INFO]
13:26:01 [INFO] Results:
13:26:01 [INFO]
13:26:01 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
13:26:01 [INFO]
13:26:01 [INFO]
13:26:01 [INFO] --- maven-antrun-plugin:1.7:run (hdfs-test-bats-driver) # hadoop-hdfs ---
I also verified that by removing surefire-plugin block in root pom.xml file will make these tests executing again as usual. So I have some questions here:
Why tests were skipped in this case?
I saw the reported surefire-plugin version was 2.20 while I was defining 2.21 in my change. I assume this is because it gets overwritten by the parent pom. So how do I make it use my customized version?
Is the version 2.21 the right version for this? I saw some post indicated version 2.21 might have some bug?
Many thanks,
I am new to Scala and Spark. I made the simple code and it successfully ran on my local machine. So, I made the .jar files using maven and copied them into my clusters machine to test it on the distributed system. However, I started my command, the console throws the error as below.
*******CLASSPATH = ********
java.lang.ClassNotFoundException: App
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.spark.util.Utils$.classForName(Utils.scala:225)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:693)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I already googled it and found that the class name should be package name + class name. But it didn't worked for my case. And I found another cause that I might be using multiple scala versions. So I checked my pom.xml to ensure my scala and spark versions. I changed the version name according to the versions cluster use, but also the result was same.
Below is my pom.xml.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.sclee.scala0</groupId>
<artifactId>scala_tutorial</artifactId>
<version>1.0-SNAPSHOT</version>
<name>${project.artifactId}</name>
<description>My wonderfull scala app</description>
<inceptionYear>2015</inceptionYear>
<licenses>
<license>
<name>My License</name>
<url>http://....</url>
<distribution>repo</distribution>
</license>
</licenses>
<properties>
<maven.compiler.source>1.6</maven.compiler.source>
<maven.compiler.target>1.6</maven.compiler.target>
<encoding>UTF-8</encoding>
<scala.version>2.10.4</scala.version>
<scala.compat.version>2.10</scala.compat.version>
</properties>
<dependencies>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>2.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>2.0.1</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.scala-lang/scala-library -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.10.4</version>
</dependency>
</dependencies>
<pluginRepositories>
<pluginRepository>
<id>scala-tools.org</id>
<name>Scala-tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</pluginRepository>
</pluginRepositories>
<build>
<plugins>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<id>compile</id>
<goals>
<goal>compile</goal>
</goals>
<phase>compile</phase>
</execution>
<execution>
<id>test-compile</id>
<goals>
<goal>testCompile</goal>
</goals>
<phase>test-compile</phase>
</execution>
<execution>
<phase>process-resources</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
</plugins>
</build>
</project>
And when I tried to clean, compile and package using maven package, there are some warnings found. (I am not sure that this warnings might be relevant to my result.But to resolve this issue, I attached the log message as below).
/usr/local/java/jdk1.7.0_80/bin/java -Dmaven.home=/usr/local/maven/apache-maven-3.1.1 -Dclassworlds.conf=/usr/local/maven/apache-maven-3.1.1/bin/m2.conf -Didea.launcher.port=7536 -Didea.launcher.bin.path=/usr/local/intellij/idea-IC-163.10154.41/bin -Dfile.encoding=UTF-8 -classpath /usr/local/maven/apache-maven-3.1.1/boot/plexus-classworlds-2.5.1.jar:/usr/local/intellij/idea-IC-163.10154.41/lib/idea_rt.jar com.intellij.rt.execution.application.AppMain org.codehaus.classworlds.Launcher -Didea.version=2016.3.2 clean compile package
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.sclee.scala0:scala_tutorial:jar:1.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. # line 92, column 15
[WARNING] 'build.plugins.plugin.version' for org.scala-tools:maven-scala-plugin is missing. # line 65, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building scala_tutorial 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: http://scala-tools.org/repo-releases/org/scala-tools/maven-scala-plugin/maven-metadata.xml
[WARNING] Could not transfer metadata org.scala-tools:maven-scala-plugin/maven-metadata.xml from/to scala-tools.org (http://scala-tools.org/repo-releases): peer not authenticated
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # scala_tutorial ---
[INFO] Deleting /home/dst/Documents/01_Intellij_workspace/scala_tutorial/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # scala_tutorial ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/main/resources
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:compile (default) # scala_tutorial ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.10.4
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[INFO] /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/main/scala:-1: info: compiling
[INFO] Compiling 1 source files to /home/dst/Documents/01_Intellij_workspace/scala_tutorial/target/classes at 1486089232696
[WARNING] warning: there were 2 deprecation warning(s); re-run with -deprecation for details
[WARNING] one warning found
[INFO] prepare-compile in 0 s
[INFO] compile in 6 s
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) # scala_tutorial ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:compile (compile) # scala_tutorial ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.10.4
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # scala_tutorial ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/main/resources
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:compile (default) # scala_tutorial ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.10.4
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) # scala_tutorial ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:compile (compile) # scala_tutorial ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.10.4
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # scala_tutorial ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/dst/Documents/01_Intellij_workspace/scala_tutorial/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) # scala_tutorial ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-scala-plugin:2.15.2:testCompile (test-compile) # scala_tutorial ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.10.4
[WARNING] com.twitter:chill_2.10:0.8.0 requires scala version: 2.10.5
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [**/*.scala,**/*.java,]
[INFO] excludes = []
[WARNING] No source files found.
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) # scala_tutorial ---
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) # scala_tutorial ---
[INFO] Building jar: /home/dst/Documents/01_Intellij_workspace/scala_tutorial/target/scala_tutorial-1.0-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.990s
[INFO] Finished at: Thu Feb 02 21:34:00 EST 2017
[INFO] Final Memory: 27M/291M
[INFO] ------------------------------------------------------------------------
Process finished with exit code 0
The Scala version cluster uses is 2.10.4. So I tried to change all the versions in the pom.xml. And the my scala code is as below. It is simple tasks to transform my data using DataFrame.
What I want to use my jar files written by scala and test it on the cluster mode. And my questions is there are any wrong process (version issue or anything) that you looked strange? Any help will be appreciated.
package com.sclee.scala0
import org.apache.spark._
import org.apache.spark.sql.SQLContext
object App {
def main(args: Array[String]): Unit = {
val conf = new SparkConf().setAppName("wordCount").setMaster("local[*]")
// Create a Scala Spark Context.
val sc = new SparkContext(conf)
val sqlContext= new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
val df = sc.textFile(args(0)).map(_.split('\t')).map(x => (x(0),x(1),x(2),x(3),x(4),x(5),x(6))).toDF("c1","c2","c3","c4","c5","c6","c7")
val res = df.explode("c7","c8")((x:String) => x.split('|')).drop("c7")
res.write.format("com.databricks.spark.csv").option("delimiter","\t").save(args(1))
}
}
And finally, this is my command to run scala jar.
spark-submit \
--class App \
--master spark://spark.dso.xxxx \
--executor-memory 10G \
/home/jumbo/user/sclee/dt/jars/scala_tutorial-1.0-SNAPSHOT.jar \
/user/sclee/data_big/ /user/sclee/output_scala_csv
Scala object name should be associated with package name while submitting the scala-spark job, and therefore your --class configuration will be:
--class com.sclee.scala0.App
Use above configuration while submitting the application and it will eliminate your error.
Update your maven pom.xml dependency section with following:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.10</artifactId>
<version>${scala.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>${scala.version}</version>
</dependency>
It will allow your pom.xml to download your Scala version compatible jars from remote repository.
I hope it will help.