Multi Module SBT projects issues - scala

Question

I am working on designing a multi module project in SBT.
The idea is to build two subprojects in SBT with one subproject having Apache spark 2.3 dependency and
the other one with Apache Spark 2.4 dependency. Each sub project should be capable of building individual
jars. I am pasting the build.sbt. While I try to run sbt assembly on each of these sub projects.
I am getting the below error.I tried removing the cache folder in .ivy2 and tried
recreating the project structure.
Error message(included relevant logs only)
[IJ]> multi2/assembly
[info] Including from cache: jackson-xc-1.9.13.jar
[info] Including from cache: minlog-1.3.0.jar
[warn] Merging 'META-INF\ASL2.0' with strategy 'discard'
[warn] Merging 'META-INF\DEPENDENCIES' with strategy 'discard'
[error] C:\Users\user_home\.ivy2\cache\org.apache.spark\spark-core_2.11\jars\spark-core_2.11-2.4.3.jar:org/apache/spark/unused/UnusedStubClass.class
[error] C:\Users\user_home\.ivy2\cache\org.apache.spark\spark-launcher_2.11\jars\spark-launcher_2.11-2.4.3.jar:org/apache/spark/unused/UnusedStubClass.class
[error] C:\Users\user_home\.ivy2\cache\org.apache.spark\spark-tags_2.11\jars\spark-tags_2.11-2.4.3.jar:org/apache/spark/unused/UnusedStubClass.class
[error] C:\Users\user_home\.ivy2\cache\org.spark-project.spark\unused\jars\unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
Likewise there are other classes that cause errors as above..
Please suggest, and let me know for any additional information.
name := "testrepo"
organization in ThisBuild := "com.yt"
scalaVersion in ThisBuild := "2.11.8"
scalaBinaryVersion := "2.11"
lazy val common = (project in file("common"))
.settings(
name := "common",
commonSettings,
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7",
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7",
dependencyOverrides += "com.fasterxml.jackson.module" %% "jackson-module-scala" %
dependencyOverrides += "io.netty" % "netty" % "3.9.9.Final",
dependencyOverrides += "commons-net" % "commons-net" % "2.2",
dependencyOverrides += "com.google.guava" % "guava" % "11.0.2",
dependencyOverrides += "com.google.code.findbugs" % "jsr305" % "1.3.9",
libraryDependencies ++= commonDependencies
)
.disablePlugins(AssemblyPlugin)
lazy val multi1 = (project in file("multi1"))
.settings(
name := "multi1",
commonSettings,
assemblySettings,
libraryDependencies ++= commonDependencies ++ Seq(
dependencies.spark23
)
)
.dependsOn(
common
)
lazy val multi2 = (project in file("multi2"))
.settings(
name := "multi2",
commonSettings,
assemblySettings,
libraryDependencies ++= commonDependencies ++ Seq(
dependencies.spark24
)
)
.dependsOn(
common
)
/*val overrides = Seq("com.fasterxml.jackson.core" % "jackson-core" % "2.8.7",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7",
"com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.8.7") */
// DEPENDENCIES
lazy val dependencies =
new {
val hivejdbc="org.apache.hive" % "hive-jdbc" % "0.13.0" % "provided"
val spark23V = "2.3.0.cloudera3"
val spark24V="2.4.3"
val spark23="org.apache.spark" %% "spark-core" % spark23V
val spark24="org.apache.spark" %% "spark-core" % spark24V
}
lazy val commonDependencies = Seq(
dependencies.hivejdbc
)
lazy val compilerOptions = Seq(
"-unchecked",
"-feature",
"-language:existentials",
"-language:higherKinds",
"-language:implicitConversions",
"-language:postfixOps",
"-deprecation",
"-encoding",
"utf8"
)
lazy val commonSettings = Seq(
scalacOptions ++= compilerOptions,
resolvers ++= Seq(
"Local Maven Repository" at "file://" + Path.userHome.absolutePath + "/.m2/repository",
Resolver.sonatypeRepo("releases"),
Resolver.sonatypeRepo("snapshots")
)
)
lazy val assemblySettings = Seq(
assemblyJarName in assembly := name.value + ".jar",
assemblyMergeStrategy in assembly := {
case PathList("org.apache.spark", "spark-core_2.11", xs # _*) => MergeStrategy.last
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case "application.conf" => MergeStrategy.concat
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
)