I am using sbt 0.13.7 and sbt-assembly 0.12.0.
Here are the error messages during "sbt assembly":
[error] (sse/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/arunswami/.ivy2/cache/org.apache.avro/avro-ipc/jars/avro-ipc-1.7.7-tests.jar:META-INF/maven/org.apache.avro/avro-ipc/pom.properties
[error] /Users/arunswami/.ivy2/cache/org.apache.avro/avro-ipc/jars/avro-ipc-1.7.7.jar:META-INF/maven/org.apache.avro/avro-ipc/pom.properties
Here is the relevant section of my build.sbt file based on README at https://github.com/sbt/sbt-assembly#merge-strategy:
assemblyMergeStrategy in assembly := {
case PathList(ps # _*) if ps.last endsWith "pom.properties" =>
MergeStrategy.discard
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
What should I be doing instead?
Thanks.
Your merge strategy looks ok, but the error message says sse/*:assembly.
Is your merge strategy included as part of the sse project?
Related
I'm having trouble assembling a scala project with sbt. I have a merge conflict involving aws dependency jars.
I looked at a bunch of posts and I don't understand why my merge strategy is not working.
This is my assembly error:
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSCredentials.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSCredentials.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSCredentialsProvider.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSCredentialsProvider.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSSessionCredentials.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSSessionCredentials.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSSessionCredentialsProvider.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSSessionCredentialsProvider.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:mozilla/public-suffix-list.txt
[error] /Users/xxxx/.ivy2/cache/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.5.jar:mozilla/public-suffix-list.txt
[error] Total time: 2 s, completed Feb 20, 2020 9:14:32 AM
This is my build.sbt
name := "scala-redshift-connection"
version := "0.1"
scalaVersion := "2.11.12"
resolvers += "Mulesoft" at "https://repository.mulesoft.org/nexus/content/repositories/public/"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-secretsmanager" % "1.11.339"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.1.0" % Test
libraryDependencies += "com.amazon.redshift" % "redshift-jdbc42" % "1.2.27.1051"
assemblyMergeStrategy in assembly := {
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
I have also tried this:
assemblyMergeStrategy in assembly := {
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case PathList("com", "amazonaws", "auth", xs#_*) => MergeStrategy.last
case x => MergeStrategy.first
}
}
I'm using sbt-assembly version 0.14.10
You're picking last of com.amazonaws.auth, which still picks it. It should work if you either discard the redhat or amazon one. In addition, you have to pick only one of public-suffix-list.txt.
assemblyMergeStrategy in assembly := {
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case PathList("com", "amazonaws", "auth", xs#_*) => MergeStrategy.discard
case PathList(xs # _*) if xs.last == "public-suffix-list.txt" => MergeStrategy.first
case _ => MergeStrategy.first
}
}
In my case I used a different library which also had a dependency on Apache httpclient and only had to pick first for the public-suffix-list.txt.
Trying to build a fat jar of a play (2.6.6) + scala.js application, getting
[error] (play/*:assembly) deduplicate: different file contents found in the following:
[error] /home/user/.ivy2/cache/com.typesafe.play/play_2.12/jars/play_2.12-2.6.6.jar:play/reference-overrides.conf
[error] /home/user/.ivy2/cache/com.typesafe.play/play-akka-http-server_2.12/jars/play-akka-http-server_2.12-2.6.6.jar:play/reference-overrides.conf
build.sbt
mainClass in assembly := Some("play.core.server.ProdServerStart")
//fullClasspath in assembly += Attributed.blank(PlayKeys.playPackageAssets.value)
(Inspired by https://www.playframework.com/documentation/2.6.6/Deploying#Using-the-SBT-assembly-plugin)
(but not using playPackageAssets at the moment)
my assembly.sbt contains just addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.5")
I also tried with a "non-standard" config:
assemblyMergeStrategy in assembly := {
// Building fat jar without META-INF
case PathList("META-INF", xs # _*) => MergeStrategy.discard
// Take last config file
case PathList(ps # _*) if ps.last endsWith ".conf" => MergeStrategy.last
case o =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(o)
}
but no luck either. How to fix that the/a correct way?
You need to tell sbt-assembly how to merge these two reference-overrides.conf config files:
assemblyMergeStrategy in assembly := {
// Building fat jar without META-INF
case PathList("META-INF", xs # _*) => MergeStrategy.discard
// Take last config file
case PathList(ps # _*) if ps.last endsWith ".conf" => MergeStrategy.last
case PathList("reference-overrides.conf") => MergeStrategy.concat
case o =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(o)
}
I faced the same problem and solved it by adding the following to .
case PathList("play", "reference-overrides.conf") => MergeStrategy.concat
In our Scala/Scalatra project, we have this merge policy for the plugin sbt-assembly:
assemblyMergeStrategy in assembly := {
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
[error] 11 errors were encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in the following:
~/.ivy2/cache/org.scalatra/scalatra_2.11/jars/scalatra_2.11-2.3.1.jar:mime.types
~/.ivy2/cache/com.amazonaws/aws-java-sdk-s3/jars/aws-java-sdk-s3-1.10.1.jar:mime.types
deduplicate: different file contents found in the following:
~/.ivy2/cache/commons-beanutils/commons-beanutils/jars/commons-beanutils-1.8.3.jar:org/apache/commons/collections/ArrayStack.class
~/.ivy2/cache/commons-collections/commons-collections/jars/commons-collections-3.2.1.jar:org/apache/commons/collections/ArrayStack.class
deduplicate: different file contents found in the following:
and the same error for different class names
What would be the right merge logic here?
Versions:
Scala : 2.11.7
SBT : 0.13.9
sbt-assembly: 0.13.0
I do not think, it is a matter of "merge-strategy", but more of the libs and their dependencies you are using.
Who "pulls" these dependencies? Which libraries are you using concretely?
One way to restrict is to use excludeAll (and similar) with dependency declaration. (see library management in SBT ), e.g.
excludeAll(
ExclusionRule("commons-beanutils", "commons-beanutils-core"),
ExclusionRule("commons-collections", "commons-collections"),
ExclusionRule("commons-logging", "commons-logging"),
ExclusionRule("org.slf4j", "slf4j-log4j12"),
ExclusionRule("org.hamcrest", "hamcrest-core"),
ExclusionRule("junit", "junit"),
ExclusionRule("org.jboss.netty", "netty"),
ExclusionRule("com.esotericsoftware.minlog", "minlog")
)
My original issue was solved with:
assemblyMergeStrategy in assembly := {
case PathList("org", "apache", "commons", "collections", xs # _*) =>
MergeStrategy.last
case PathList("mime.types") =>
MergeStrategy.last
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}
just trying to generate a Jar with sbt-assembly and I'm still trapped with this:
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/fsalvador/.ivy2/cache/org.neo4j/neo4j-kernel/jars/neo4j-kernel-1.9.4.jar:META-INF/CHANGES.txt
[error] /Users/fsalvador/.ivy2/cache/org.neo4j/neo4j-lucene-index/jars/neo4j-lucene-index-1.9.4.jar:META-INF/CHANGES.txt
[error] /Users/fsalvador/.ivy2/cache/org.neo4j/neo4j-graph-algo/jars/neo4j-graph-algo-1.9.4.jar:META-INF/CHANGES.txt
[error] /Users/fsalvador/.ivy2/cache/org.neo4j/neo4j-udc/jars/neo4j-udc-1.9.4.jar:META-INF/CHANGES.txt
[error] /Users/fsalvador/.ivy2/cache/org.neo4j/neo4j-cypher/jars/neo4j-cypher-1.9.4.jar:META-INF/CHANGES.txt
[error] /Users/fsalvador/.ivy2/cache/org.neo4j/neo4j-jmx/jars/neo4j-jmx-1.9.4.jar:META-INF/CHANGES.txt
in my build.sbt I have tried the following:
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
case PathList("ivy2", "cache", "org.neo4j", xs # _*) => MergeStrategy.last
case "CHANGES.txt" => MergeStrategy.discard
case x => old(x)
}
}
And still no joy, tried almost everything.
PathList(...) extractor splits up the path string. The following should work:
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => {
case PathList("META-INF", "CHANGES.txt") => MergeStrategy.discard
case x => old(x)
}
}
Did you try a simpler:
mergeStrategy in assembly := mergeStrategy.first
Update, don't do this, please see comments
Being new to sbt and the sbt-assembly plugin I am confused about how one deals with builds involving different class definitions within dependencies I am trying to package.
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/dm/.ivy2/cache/org.apache.tika/tika-app/jars/tika-app-1.3.jar:javax/xml/XMLConstants.class
[error] /Users/dm/.ivy2/cache/stax/stax-api/jars/stax-api-1.0.1.jar:javax/xml/XMLConstants.class
[error] /Users/dm/.ivy2/cache/xml-apis/xml-apis/jars/xml-apis-1.3.03.jar:javax/xml/XMLConstants.class
I've added:
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("javax", "xml", xs # _*) => MergeStrategy.first
}
}
to my build.sbt file, but I'm still getting the error above (regardless of whether or not it's in the build file). Any guidance would be greatly appreciated.
Thanks,
Don
I think you're close. Make sure that you add any rewiring after assemblySettings are loaded, and also pass any patterns you're not handling to the default strategy:
assemblySettings
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("javax", "xml", xs # _*) => MergeStrategy.first
case _ => old
}
}
Just an update—with current sbt (0.13.8) and sbt-assembly (0.13.0) versions, Eugene's code becomes:
assemblyMergeStrategy in assembly := {
case PathList("javax", "xml", xs # _*) => MergeStrategy.first
case x =>
val oldStrategy = (assemblyMergeStrategy in assembly).value
oldStrategy(x)
}