Deduplication in build.sbt - scala

Question

There are conflicting jline jar files in the includes path that need to be deduplicated. I have attempted to do so as follows:
I am building a combined spark and kafka fat jar and the jline jar file is duplicated.
Here is the build.sbt file
import sbt._
import sbt.Keys._
import java.io.File
import AssemblyKeys._
name := "kafkascala"
version := "0.1.0-SNAPSHOT"
scalaVersion := "2.10.4"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "0.9.1",
"org.apache.spark" % "spark-examples_2.10" % "0.9.1",
"org.apache.spark" % "spark-tools_2.10" % "0.9.1",
"org.apache.kafka" % "kafka_2.10" % "0.8.1.1" intransitive()
withSources
)
resolvers += "Apache repo" at "https://repository.apache.org/content/repositories/releases"
assemblySettings
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
case PathList("maven","jline","jline","pom.properties") => MergeStrategy.discard
case x => old(x)
}
}
Here is the error output from running sbt assembly
java.lang.RuntimeException: deduplicate: different file contents found in the following:
C:\Users\S80035683\.ivy2\cache\jline\jline\jars\jline-0.9.94.jar:META-INF/maven/jline/jline/pom.properties
C:\Users\S80035683\.ivy2\cache\org.jruby\jruby-complete\jars\jruby-complete-1.6.5.jar:META-INF/maven/jline/jline/pom.properties
at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$applyStrategy$1(Plugin.scala:253)
at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:270)
at sbtassembly.Plugin$Assembly$$anonfun$15.apply(Plugin.scala:267)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Plugin$Assembly$.applyStrategies(Plugin.scala:272)
at sbtassembly.Plugin$Assembly$.x$4$lzycompute$1(Plugin.scala:172)
at sbtassembly.Plugin$Assembly$.x$4$1(Plugin.scala:170)
at sbtassembly.Plugin$Assembly$.stratMapping$lzycompute$1(Plugin.scala:170)
at sbtassembly.Plugin$Assembly$.stratMapping$1(Plugin.scala:170)
at sbtassembly.Plugin$Assembly$.inputs$lzycompute$1(Plugin.scala:214)
at sbtassembly.Plugin$Assembly$.inputs$1(Plugin.scala:204)
at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230)
at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373)
at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
at sbt.std.Transform$$anon$4.work(System.scala:64)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:244)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] C:\Users\S80035683\.ivy2\cache\jline\jline\jars\jline-0.9.94.jar:META-INF/maven/jline/jline/pom.properties
[error] C:\Users\S80035683\.ivy2\cache\org.jruby\jruby-complete\jars\jruby-complete-1.6.5.jar:META-INF/maven/jline/jline/pom.properties
Here is the section of build.sbt that was attempting to address the issue:
assemblySettings
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
case PathList("maven","jline","jline","pom.properties") => MergeStrategy.discard
case x => old(x)
}
}

Answer 1

Your PathList is wrong. You're missing "META-INF" in front of the "maven". Something like this should work:
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
case PathList("META-INF", "maven","jline","jline", "pom.properties" ) => MergeStrategy.discard
case PathList("META-INF", "maven","jline","jline", "pom.xml" ) => MergeStrategy.discard
case x => old(x)
}
}
or a bit more concise (exclude both pom.properties and pom.xml in one line):
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
case PathList("META-INF", "maven","jline","jline", ps) if ps.startsWith("pom") => MergeStrategy.discard
case x => old(x)
}
}

Answer 2

I have reached the conclusion that the defaultMergeStrategy in sbt is unnecessarily causing deduplicate errors. According to the sbt assembly author the decision to fail rather than select first is because they believe the user should have to explicitly decide, but most users don't even know what the error means and takes them ages to figure out that changing deduplicate to first just makes stuff work. I think making the build fail was a mistake, the following is how the defaultMergeStrategy should look:
// Strat copied from defaultMergeStrategy with the
// "fail and confuse the hell out the user" lines changed to
// "just bloody work and stop pissing everyone off"
mergeStrategy in assembly <<= (mergeStrategy in assembly) ((old) => {
case x if Assembly.isConfigFile(x) =>
MergeStrategy.concat
case PathList(ps # _*) if Assembly.isReadme(ps.last) || Assembly.isLicenseFile(ps.last) =>
MergeStrategy.rename
case PathList("META-INF", xs # _*) =>
(xs map {_.toLowerCase}) match {
case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
MergeStrategy.discard
case ps # (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
MergeStrategy.discard
case "plexus" :: xs =>
MergeStrategy.discard
case "services" :: xs =>
MergeStrategy.filterDistinctLines
case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
MergeStrategy.filterDistinctLines
case _ => MergeStrategy.first // Changed deduplicate to first
}
case PathList(_*) => MergeStrategy.first // added this line
})