Can't compile scala&java project. Fail to find transient dependencies - scala

Question

I have a huge feeling that I'm missing something.
I'm trying to compile Java&Scala project at my work. Luckliy I was able to repuduce my problem on a simple dummy project.
I'm using scala-maven-plugin(https://github.com/davidB/scala-maven-plugin). I'm also base my example on this question
Aiming to solve Scala&Java multi-module project. I have root and two modules m1 and m2.
m1 pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>prj_multi_modules</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>m1</artifactId>
<packaging>jar</packaging>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<dependency>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>m2</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
m2 pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>prj_multi_modules</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>m2</artifactId>
<packaging>jar</packaging>
<properties>
<cdh.version>cdh5.7.6</cdh.version>
<build.prop.dir>..</build.prop.dir>
<scala.version>2.11.8</scala.version>
<scala.version.major>2.11</scala.version.major>
<spark.version>2.3.0.cloudera4</spark.version>
<hadoop.version>2.6.0-${cdh.version}</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency> <!-- remove this for spark 2.3? -->
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.version.major}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.github.scopt</groupId>
<artifactId>scopt_${scala.version.major}</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_${scala.version.major}</artifactId>
<version>1.5.0</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-csv</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>com.holdenkarau</groupId>
<artifactId>spark-testing-base_${scala.version.major}</artifactId>
<version>1.6.0_0.7.4</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
root pom:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<groupId>samples.scala-maven-plugin</groupId>
<artifactId>prj_multi_modules</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<properties>
<cdh.version>cdh5.7.6</cdh.version>
<build.prop.dir>..</build.prop.dir>
<scala.version>2.11.8</scala.version>
<scala.version.major>2.11</scala.version.major>
<spark.version>2.3.0.cloudera4</spark.version>
<hadoop.version>2.6.0-${cdh.version}</hadoop.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<modules>
<module>m1</module>
<module>m2</module>
</modules>
</project>
My goal is very simple I want m1 project to get all spark dpedencies from m2 module in order to compile a spark code in m1 module.
This is the spark code in m1:
package p1
import org.apache.spark.sql.SparkSession
class MyClass(spark: SparkSession) {
import spark.implicits._
def run(input: String) = {
val df = spark.read.parquet(input)
df.show()
}
}
object MyClass {
case class Params(input: String = "")
def main(args: Array[String]): Unit = {
val parser = new scopt.OptionParser[Params](usageHeader) {
opt[String]('i', "input") required() action { (s, params) => params.copy(input = s) }
}
parser.parse(args, Params()) match {
case Some(p) =>
val spark = SparkSession.builder
.appName("spark2 example")
.getOrCreate()
val job = new Spark2Exp(spark)
job.run(p.input)
spark.stop()
case None =>
System.exit(1)
}
}
val usageHeader: String = "n/a"
}
m2 compilation is working smoothly. While compiling m1 all java compilation process is good but when trying to compile scala code it fails not finding spark dependencies as if it doesn't transient between the dependent modules.
In addition if I copy all m2 spark dependencies to m1 pom everything is working.
I can't seem to understand what I'm doing wrong. I am sure it's scala - maven issue since this scenario is pretty straight forward in java.
Adding compilation error:
"C:\Program Files\Java\jdk1.8.0_151\bin\java" -Dmaven.multiModuleProjectDirectory=C:\dev\scala-maven-plugin "-Dmaven.home=C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\plugins\maven\lib\maven3" "-Dclassworlds.conf=C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\plugins\maven\lib\maven3\bin\m2.conf" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\lib\idea_rt.jar=55662:C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\JetBrains\IntelliJ IDEA 2017.3\plugins\maven\lib\maven3\boot\plexus-classworlds-2.5.2.jar" org.codehaus.classworlds.Launcher -Didea.version=2017.3 clean install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] prj_multi_modules
[INFO] m2
[INFO] m1
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building prj_multi_modules 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # prj_multi_modules ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) # prj_multi_modules ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (default) # prj_multi_modules ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) # prj_multi_modules ---
[INFO] Installing C:\dev\scala-maven-plugin\samples\prj_multi_modules\pom.xml to C:\Users\raphael.peretz\.m2\repository\samples\scala-maven-plugin\prj_multi_modules\0.0.1-SNAPSHOT\prj_multi_modules-0.0.1-SNAPSHOT.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building m2 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # m2 ---
[INFO] Deleting C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # m2 ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # m2 ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) # m2 ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] samples.scala-maven-plugin:m2:0.0.1-SNAPSHOT requires scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.8
[WARNING] org.apache.spark:spark-core_2.11:2.3.0.cloudera4 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-ast_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) # m2 ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) # m2 ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (default) # m2 ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] samples.scala-maven-plugin:m2:0.0.1-SNAPSHOT requires scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.8
[WARNING] org.apache.spark:spark-core_2.11:2.3.0.cloudera4 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-ast_2.11:3.2.11 requires scala version: 2.11.8
[WARNING] org.json4s:json4s-core_2.11:3.2.11 requires scala version: 2.11.0
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) # m2 ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) # m2 ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\target\m2-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) # m2 ---
[INFO] Installing C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\target\m2-0.0.1-SNAPSHOT.jar to C:\Users\raphael.peretz\.m2\repository\samples\scala-maven-plugin\m2\0.0.1-SNAPSHOT\m2-0.0.1-SNAPSHOT.jar
[INFO] Installing C:\dev\scala-maven-plugin\samples\prj_multi_modules\m2\pom.xml to C:\Users\raphael.peretz\.m2\repository\samples\scala-maven-plugin\m2\0.0.1-SNAPSHOT\m2-0.0.1-SNAPSHOT.pom
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building m1 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) # m1 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) # m1 ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) # m1 ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (default) # m1 ---
[INFO] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala:-1: info: compiling
[INFO] Compiling 1 source files to C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\target\classes at 1558957820236
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:3: error: object spark is not a member of package org.apache
[ERROR] import org.apache.spark.sql.SparkSession
[ERROR] ^
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:5: error: not found: type SparkSession
[ERROR] class MyClass(spark: SparkSession) {
[ERROR] ^
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:29: error: not found: value SparkSession
[ERROR] val spark = SparkSession.builder
[ERROR] ^
[ERROR] C:\dev\scala-maven-plugin\samples\prj_multi_modules\m1\src\main\scala\p1\MyClass.scala:33: error: not found: type Spark2Exp
[ERROR] val job = new Spark2Exp(spark)
[ERROR] ^
[ERROR] four errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] prj_multi_modules .................................. SUCCESS [ 1.435 s]
[INFO] m2 ................................................. SUCCESS [ 22.342 s]
[INFO] m1 ................................................. FAILURE [ 3.497 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 27.379 s
[INFO] Finished at: 2019-05-27T14:50:23+03:00
[INFO] Final Memory: 52M/927M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (default) on project m1: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :m1
Process finished with exit code 1

Answer 1

Since you need spark in both m1 and m2, why not delcare the spark dependency in root pom file?
Of course, you can remove the duplicate one in m2 afterwards.