unable to merge aws classes in sbt - scala

Question

I'm having trouble assembling a scala project with sbt. I have a merge conflict involving aws dependency jars.
I looked at a bunch of posts and I don't understand why my merge strategy is not working.
This is my assembly error:
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSCredentials.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSCredentials.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSCredentialsProvider.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSCredentialsProvider.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSSessionCredentials.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSSessionCredentials.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:com/amazonaws/auth/AWSSessionCredentialsProvider.class
[error] /Users/xxxx/.ivy2/cache/com.amazonaws/aws-java-sdk-core/jars/aws-java-sdk-core-1.11.339.jar:com/amazonaws/auth/AWSSessionCredentialsProvider.class
[error] deduplicate: different file contents found in the following:
[error] /Users/xxxx/.ivy2/cache/com.amazon.redshift/redshift-jdbc42/jars/redshift-jdbc42-1.2.27.1051.jar:mozilla/public-suffix-list.txt
[error] /Users/xxxx/.ivy2/cache/org.apache.httpcomponents/httpclient/jars/httpclient-4.5.5.jar:mozilla/public-suffix-list.txt
[error] Total time: 2 s, completed Feb 20, 2020 9:14:32 AM
This is my build.sbt
name := "scala-redshift-connection"
version := "0.1"
scalaVersion := "2.11.12"
resolvers += "Mulesoft" at "https://repository.mulesoft.org/nexus/content/repositories/public/"
libraryDependencies += "com.amazonaws" % "aws-java-sdk-secretsmanager" % "1.11.339"
libraryDependencies += "org.scalatest" %% "scalatest" % "3.1.0" % Test
libraryDependencies += "com.amazon.redshift" % "redshift-jdbc42" % "1.2.27.1051"
assemblyMergeStrategy in assembly := {
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
I have also tried this:
assemblyMergeStrategy in assembly := {
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case PathList("com", "amazonaws", "auth", xs#_*) => MergeStrategy.last
case x => MergeStrategy.first
}
}
I'm using sbt-assembly version 0.14.10

Answer 1

You're picking last of com.amazonaws.auth, which still picks it. It should work if you either discard the redhat or amazon one. In addition, you have to pick only one of public-suffix-list.txt.
assemblyMergeStrategy in assembly := {
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case PathList("com", "amazonaws", "auth", xs#_*) => MergeStrategy.discard
case PathList(xs # _*) if xs.last == "public-suffix-list.txt" => MergeStrategy.first
case _ => MergeStrategy.first
}
}
In my case I used a different library which also had a dependency on Apache httpclient and only had to pick first for the public-suffix-list.txt.