sbt deduplicate: different file contents found in the following:

sbt deduplicate: different file contents found in the following: - scala

I'm aware this question has been asked (and answered) numerous times, but the existing answers do not address my case.
I am trying to build the DataStax spark-cassandra-connector (tag v1.5.0-M3) as per this tutorial
./sbt/sbt assembly
but I get the following errors
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/me/.ivy2/cache/io.netty/netty/bundles/netty-3.8.0.Final.jar:META-INF/jboss-beans.xml
/Users/me/.ivy2/cache/org.jboss.netty/netty/bundles/netty-3.2.2.Final.jar:META-INF/jboss-beans.xml
at sbtassembly.Assembly$.sbtassembly$Assembly$$applyStrategy$1(Assembly.scala:106)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:123)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:120)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Assembly$.applyStrategies(Assembly.scala:125)
at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
at sbtassembly.Assembly$.apply(Assembly.scala:83)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:226)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:223)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/me/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
/Users/me/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
at sbtassembly.Assembly$.sbtassembly$Assembly$$applyStrategy$1(Assembly.scala:106)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:123)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:120)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Assembly$.applyStrategies(Assembly.scala:125)
at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
at sbtassembly.Assembly$.apply(Assembly.scala:83)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:226)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:223)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[error] (spark-cassandra-connector-java/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/me/.ivy2/cache/io.netty/netty/bundles/netty-3.8.0.Final.jar:META-INF/jboss-beans.xml
[error] /Users/me/.ivy2/cache/org.jboss.netty/netty/bundles/netty-3.2.2.Final.jar:META-INF/jboss-beans.xml
[error] (spark-cassandra-connector/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/me/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
[error] /Users/me/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
[error] Total time: 44 s, completed 10-Dec-2015 16:22:01
Any of the suggested answers I have read (such as this one) refer to changes required in the build.sbt file, but in my case I have no such file. In the project directory I have
.
├── BuildUtil.scala
├── CassandraSparkBuild.scala
├── Publish.scala
├── Settings.scala
├── SparkInstaller.scala
├── Versions.scala
├── build.properties
├── plugins.sbt
├── project
└── target
2 directories, 8 files
So I don't know which file(s) to edit and what edits are necessary. I am on OSX and have the following versions installed
Spark 1.5.2
Java 1.8.0_45
Scala 2.11.7
sbt 0.13.9
UPDATE
If I follow the instruction here and build for Scala 2.11 I get the following prompt
$ sbt -Dscala-2.11=true
[info] Loading project definition from /usr/local/share/spark-cassandra-connector/project
[info] Updating {file:/usr/local/share/spark-cassandra-connector/project/}spark-cassandra-connector-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
Using releases: https://oss.sonatype.org/service/local/staging/deploy/maven2 for releases
Using snapshots: https://oss.sonatype.org/content/repositories/snapshots for snapshots
Scala: 2.11.7
Scala Binary: 2.11
Java: target=1.7 user=1.8.0_45
[info] Set current project to root (in build file:/usr/local/share/spark-cassandra-connector/)
SBT   39dcadbe40e833b610357ce84811201f03d8f350  root 
Dependency Graph
Below is the result of using sbt-dependency-graph as per #AlexeyRomanovs suggestion
[info] default:project_2.10:0.1-SNAPSHOT [S]
[info] +-com.scalapenos:sbt-prompt:0.2.1
[info] | +-com.typesafe.sbt:sbt-git:0.6.4
[info] | +-org.eclipse.jgit:org.eclipse.jgit.pgm:3.3.2.201404171909-r
[info] | +-args4j:args4j:2.0.12
[info] | +-org.apache.commons:commons-compress:1.6
[info] | | +-org.tukaani:xz:1.4
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.archive:3.3.2.201404171909-r
[info] | | +-org.apache.commons:commons-compress:1.6
[info] | | | +-org.tukaani:xz:1.4
[info] | | |
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | | +-com.jcraft:jsch:0.1.50
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.osgi:org.osgi.core:4.3.1
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.ui:3.3.2.201404171909-r
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | +-com.jcraft:jsch:0.1.50
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | +-com.jcraft:jsch:0.1.50
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe.sbt:sbt-git:0.6.4
[info] | +-org.eclipse.jgit:org.eclipse.jgit.pgm:3.3.2.201404171909-r
[info] | +-args4j:args4j:2.0.12
[info] | +-org.apache.commons:commons-compress:1.6
[info] | | +-org.tukaani:xz:1.4
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.archive:3.3.2.201404171909-r
[info] | | +-org.apache.commons:commons-compress:1.6
[info] | | | +-org.tukaani:xz:1.4
[info] | | |
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | | +-com.jcraft:jsch:0.1.50
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.osgi:org.osgi.core:4.3.1
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.ui:3.3.2.201404171909-r
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | +-com.jcraft:jsch:0.1.50
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | +-com.jcraft:jsch:0.1.50
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe.sbt:sbt-scalariform:1.3.0
[info] | +-org.scalariform:scalariform_2.10:0.1.4 [S]
[info] |
[info] +-net.virtual-void:sbt-dependency-graph:0.8.0
[info] | +-com.github.mdr:ascii-graphs_2.10:0.0.3 [S]
[info] |
[info] +-org.scalastyle:scalastyle-sbt-plugin:0.6.0
[info] | +-org.scalastyle:scalastyle_2.10:0.6.0 [S]
[info] | +-com.danieltrinh:scalariform_2.10:0.1.5 [S]
[info] | +-com.typesafe:config:1.2.0
[info] |
[info] +-org.scoverage:sbt-scoverage:1.0.4
[info] | +-org.scoverage:scalac-scoverage-plugin_2.10:1.0.4 [S]
[info] |
[info] +-org.spark-packages:sbt-spark-package:0.2.3
[info] | +-com.eed3si9n:sbt-assembly:0.12.0
[info] | +-org.apache.directory.studio:org.apache.commons.codec:1.8
[info] | | +-commons-codec:commons-codec:1.8
[info] | |
[info] | +-org.scalaj:scalaj-http_2.10:1.1.4 [S]
[info] |
[info] +-org.typelevel:sbt-typelevel:0.3.1
[info] +-com.eed3si9n:sbt-buildinfo:0.3.2
[info] +-com.github.gseitz:sbt-release:0.8.4
[info] +-com.typesafe.sbt:sbt-pgp:0.8.2
[info] | +-com.jsuereth:gpg-library_2.10:0.8.2 [S]
[info] | | +-net.databinder:dispatch-http_2.10:0.8.10 [S]
[info] | | | +-net.databinder:dispatch-core_2.10:0.8.10 [S]
[info] | | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | | +-commons-codec:commons-codec:1.8
[info] | | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | | |
[info] | | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-net.databinder:dispatch-futures_2.10:0.8.10 [S]
[info] | | | | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.bouncycastle:bcpg-jdk15on:1.49
[info] | | +-org.bouncycastle:bcprov-jdk15on:1.49
[info] | |
[info] | +-net.databinder:dispatch-http_2.10:0.8.10 [S]
[info] | +-net.databinder:dispatch-core_2.10:0.8.10 [S]
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-net.databinder:dispatch-futures_2.10:0.8.10 [S]
[info] | | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe:sbt-mima-plugin:0.1.6
[info] | +-com.typesafe:mima-reporter_2.10:0.1.6 [S]
[info] | +-com.typesafe:config:1.0.0 (evicted by: 1.2.0)
[info] | +-com.typesafe:config:1.2.0
[info] | +-com.typesafe:mima-core_2.10:0.1.6 [S]
[info] | | +-org.scala-lang:scala-compiler:2.10.2 [S]
[info] | | +-org.scala-lang:scala-reflect:2.10.2 [S]
[info] | |
[info] | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | +-org.scala-lang:scala-swing:2.10.2 [S]
[info] |
[info] +-net.virtual-void:sbt-dependency-graph:0.7.4 (evicted by: 0.8.0)
[info] +-net.virtual-void:sbt-dependency-graph:0.8.0
[info] | +-com.github.mdr:ascii-graphs_2.10:0.0.3 [S]
[info] |
[info] +-org.xerial.sbt:sbt-sonatype:0.2.1
[info] +-org.apache.httpcomponents:httpclient:4.2.6
[info] +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] +-commons-codec:commons-codec:1.8
[info] +-commons-logging:commons-logging:1.1.1
[info] +-org.apache.httpcomponents:httpcore:4.2.5
[info]

If you look at the SBT tutorial, you'll find that project/*.scala files are equivalent to *.sbt files in the root directory. So that's where you need to make the changes (specifically, in CassandraSparkBuild.scala).
The problem is that there are two versions of Netty in transitive dependencies and SBT doesn't notice this because the group IDs are different: org.jboss.netty for 3.2.2 and io.netty for 3.8. Find where they come from (using e.g. https://github.com/jrudolph/sbt-dependency-graph) and exclude the older version (and hope nothing breaks).

Related

How to run the legacy Mongo Casbah drivers alongside the modern Mongo scala drivers

The old official Scala Driver, "casbah", is end-of-life. There is a new official mongo driver here but the two drivers can't coexist in the same project due to the way sbt resolves conflicting library versions. They both seem to import separate major versions of an underlying mongo-java-driver.
I have hundreds of hand written Mongo queries that need tweaking for the new driver so having two drivers in use concurrently whilst I migrate the app query by query would be the safest and most agile approach for me, that I'd like to pursue.
I have a sample project available here. There are two connect functions. A new one and an old one. If you comment out the new/old pieces you will see each independently work, connect to a database and list the names of the collections.
When you include both drivers together you get this error:
[error] (run-main-0) java.lang.NoSuchMethodError: 'java.util.List
com.mongodb.ConnectionString.getCredentialList()'
[error] java.lang.NoSuchMethodError: 'java.util.List com.mongodb.ConnectionString.getCredentialList()'
[error] at com.mongodb.MongoClientURI.getCredentials(MongoClientURI.java:243)
[error] at com.mongodb.Mongo.createCluster(Mongo.java:637)
[error] at com.mongodb.Mongo.<init>(Mongo.java:289)
[error] at com.mongodb.MongoClient.<init>(MongoClient.java:268)
[error] at com.mongodb.casbah.MongoClient$.apply(MongoClient.scala:201)
[error] at example.LegacyMongoScala$.connect(Hello.scala:59)
[error] at example.Hello$.delayedEndpoint$example$Hello$1(Hello.scala:14)
[error] at example.Hello$delayedInit$body.apply(Hello.scala:7)
[error] at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error] at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error] at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error] at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error] at scala.collection.immutable.List.foreach(List.scala:431)
[error] at scala.App.main(App.scala:80)
[error] at scala.App.main$(App.scala:78)
[error] at example.Hello$.main(Hello.scala:7)
[error] at example.Hello.main(Hello.scala)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
I suspect the problem here is how the latest mongo-java-driver pushes out the former?
HOST [15:18: (main) mongo-migrate-example$] sbt dependencyTree
[info] welcome to sbt 1.4.4 (Debian Java 11.0.9.1)
[info] loading settings for project global-plugins from idea.sbt,plugins.sbt ...
[info] loading global plugins from /home/phill/.sbt/1.0/plugins
[info] loading project definition from /home/phill/tmp/mongo-migrate-example/project
[success] Generated .bloop/mongo-migrate-example-build.json
[info] compiling 1 Scala source to /home/phill/tmp/mongo-migrate-example/project/target/scala-2.12/sbt-1.0/classes ...
[success] Total time: 4 s, completed 5 Jan 2021, 15:19:43
[info] loading settings for project root from build.sbt ...
[info] set current project to mongo-migrate-example (in build file:/home/phill/tmp/mongo-migrate-example/)
[info] Updating
https://repo1.maven.org/maven2/joda-time/joda-time/2.9.4/joda-time-2.9.4.pom
100.0% [##########] 29.4 KiB (101.2 KiB / s)
https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.pom
100.0% [##########] 10.4 KiB (35.7 KiB / s)
[info] Resolved dependencies
[info] Fetching artifacts of
https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.jar
100.0% [##########] 37.6 KiB (536.6 KiB / s)
https://repo1.maven.org/maven2/joda-time/joda-time/2.9.4/joda-time-2.9.4.jar
100.0% [##########] 614.8 KiB (2.2 MiB / s)
[info] Fetched artifacts of
[info] com.example:mongo-migrate-example_2.12:0.1.0-SNAPSHOT [S]
[info] +-com.github.salat:salat-core_2.12:1.11.2 [S]
[info] | +-com.github.salat:salat-util_2.12:1.11.2 [S]
[info] | | +-org.json4s:json4s-native_2.12:3.4.2 [S]
[info] | | | +-org.json4s:json4s-core_2.12:3.4.2 [S]
[info] | | | +-com.thoughtworks.paranamer:paranamer:2.8
[info] | | | +-org.json4s:json4s-ast_2.12:3.4.2 [S]
[info] | | | +-org.json4s:json4s-scalap_2.12:3.4.2 [S]
[info] | | | +-org.scala-lang.modules:scala-xml_2.12:1.0.5 (evicted by: 1.0.6)
[info] | | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | | |
[info] | | +-org.scala-lang:scalap:2.12.12
[info] | | | +-org.scala-lang:scala-compiler:2.12.12 [S]
[info] | | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.7.21
[info] | |
[info] | +-org.json4s:json4s-native_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-core_2.12:3.4.2 [S]
[info] | | +-com.thoughtworks.paranamer:paranamer:2.8
[info] | | +-org.json4s:json4s-ast_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-scalap_2.12:3.4.2 [S]
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.5 (evicted by: 1.0.6)
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | |
[info] | +-org.mongodb:casbah-core_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | | +-joda-time:joda-time:2.9.4
[info] | | | +-org.joda:joda-convert:1.2
[info] | | |
[info] | | +-org.mongodb:mongo-java-driver:3.2.2
[info] | | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | | +-org.slf4j:slf4j-api:1.7.21
[info] | |
[info] | +-org.mongodb:casbah-query_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | +-joda-time:joda-time:2.9.4
[info] | | +-org.joda:joda-convert:1.2
[info] | |
[info] | +-org.mongodb:mongo-java-driver:3.2.2
[info] | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-com.github.salat:salat-util_2.12:1.11.2 [S]
[info] | +-org.json4s:json4s-native_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-core_2.12:3.4.2 [S]
[info] | | +-com.thoughtworks.paranamer:paranamer:2.8
[info] | | +-org.json4s:json4s-ast_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-scalap_2.12:3.4.2 [S]
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.5 (evicted by: 1.0.6)
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | |
[info] | +-org.scala-lang:scalap:2.12.12
[info] | | +-org.scala-lang:scala-compiler:2.12.12 [S]
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] | |
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-io.netty:netty-all:4.1.51.Final
[info] +-org.mongodb.scala:mongo-scala-driver_2.12:4.1.0 [S]
[info] | +-org.mongodb.scala:mongo-scala-bson_2.12:4.1.0 [S]
[info] | | +-org.mongodb:bson:4.1.0
[info] | | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] | |
[info] | +-org.mongodb:mongodb-driver-reactivestreams:4.1.0
[info] | | +-org.mongodb:bson:4.1.0
[info] | | +-org.mongodb:mongodb-driver-core:4.1.0
[info] | | | +-org.mongodb:bson:4.1.0
[info] | | |
[info] | | +-org.reactivestreams:reactive-streams:1.0.2
[info] | |
[info] | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] |
[info] +-org.mongodb:casbah-core_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | | +-joda-time:joda-time:2.9.4
[info] | | | +-org.joda:joda-convert:1.2
[info] | | |
[info] | | +-org.mongodb:mongo-java-driver:3.2.2
[info] | | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | | +-org.slf4j:slf4j-api:1.7.21
[info] | |
[info] | +-org.mongodb:casbah-query_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | +-joda-time:joda-time:2.9.4
[info] | | +-org.joda:joda-convert:1.2
[info] | |
[info] | +-org.mongodb:mongo-java-driver:3.2.2
[info] | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-org.mongodb:casbah-gridfs_2.12:3.1.1 [S]
[info] +-org.mongodb:casbah-core_2.12:3.1.1 [S]
[info] +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | +-joda-time:joda-time:2.9.4
[info] | | +-org.joda:joda-convert:1.2
[info] | |
[info] | +-org.mongodb:mongo-java-driver:3.2.2
[info] | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-org.mongodb:casbah-query_2.12:3.1.1 [S]
[info] +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | +-joda-time:joda-time:2.9.4
[info] | +-org.joda:joda-convert:1.2
[info] |
[info] +-org.mongodb:mongo-java-driver:3.2.2
[info] +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] +-org.slf4j:slf4j-api:1.7.21
[info]
[success] Total time: 1 s, completed 5 Jan 2021, 15:19:45
Question 1: Can someone suggest an approach which might work?
I have tried creating a library (using sbt new, then adding a single dependency + sbt-assembly plugin) which only expose a connect function and then used sbt-assembly plugin to shade the dependencies. Using this technique I can rename com.mongodb to shadedlegacy.mongodb and see this inside the assembled jar when I open it with a zip tool. I can used the renamed symbols in my applications but internally there are still errors thrown about subsequent classes and I'm not sure if this is just a rabbit hole with no end, all caused by the fact mongo-java-driver is shared by both and always evicted:
[error] (run-main-0) java.lang.IllegalAccessError: Error running public static void app.Boot.main(java.lang.String[]).
[error] java.lang.IllegalAccessError: failed to access class org.bson.BSON from class com.mongodb.casbah.commons.conversions.scala.OptionSerializer (org.bson.BSON and com.mongodb.casbah.commons.conversions.scala.OptionSerializer are in unnamed module of loader sbt.internal.LayeredClassLoader #2d8d2575)
[error] If using a layered classloader, this can occur if jvm package private classes are accessed across layers. This can be fixed by changing to the Flat or ScalaInstance class loader layering strategies.
[error] java.lang.IllegalAccessError: Error running public static void app.Boot.main(java.lang.String[]).
[error] java.lang.IllegalAccessError: failed to access class org.bson.BSON from class com.mongodb.casbah.commons.conversions.scala.OptionSerializer (org.bson.BSON and com.mongodb.casbah.commons.conversions.scala.OptionSerializer are in unnamed module of loader sbt.internal.LayeredClassLoader #2d8d2575)
[error] If using a layered classloader, this can occur if jvm package private classes are accessed across layers. This can be fixed by changing to the Flat or ScalaInstance class loader layering strategies.
[error] stack trace is suppressed; run last Compile / bgRun for the full output
Question 3: Is this approach above of shading both libraries one that could work?
Question 4: Would this sbt-shading library offer any alternative solutions that couldn't be reach via sbt-assembly?
Question 5: Is there any approaches using custom classloaders that could potentially lead me to success?
Question 6: If you've moved between drivers was it ever easier at some point? Am I over-estimating the complexity? I can't find any upgrade guides.
Thank you for your time.

java.lang.NoClassDefFoundError: Could not initialize class com.twitter.finagle.Init$

I've project with dependency to finagle 7.1.0 and related libraries with Scala version as 2.11.11
When I upgraded the scala version to 2.12.6 and finagle version as 17.12.0 I'm getting below error.
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.twitter.finagle.Init$
at com.twitter.finagle.client.StackClient$.endpointStack(StackClient.scala:54)
at com.twitter.finagle.client.StackClient$.newStack(StackClient.scala:220)
at com.twitter.finagle.builder.ClientConfig$NilClient$.apply$default$1(ClientBuilder.scala:74)
at com.twitter.finagle.builder.ClientConfig$.nilClient(ClientBuilder.scala:91)
at com.twitter.finagle.builder.ClientBuilder.<init>(ClientBuilder.scala:248)
at com.twitter.finagle.builder.ClientBuilder$.apply(ClientBuilder.scala:46)
at com.twitter.finagle.builder.ClientBuilder$.get(ClientBuilder.scala:52)
at com.twitter.finagle.builder.ClientBuilder.get(ClientBuilder.scala)
Below is the dependency graph for finagle related jars -
[INFO] +- com.twitter:finagle-core_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:finagle-toggle_2.12:jar:17.12.0:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.8:compile
[INFO] | | | \- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8:compile
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.8.8:compile
[INFO] | | \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.8.8:compile
[INFO] | +- com.twitter:finagle-init_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-app_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-cache_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-codec_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-core_2.12:jar:17.12.0:compile
[INFO] | | +- com.twitter:util-function_2.12:jar:17.12.0:compile
[INFO] | | +- org.scala-lang:scala-reflect:jar:2.12.6:compile
[INFO] | | \- org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.0.4:compile
[INFO] | +- com.twitter:util-hashing_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-jvm_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-lint_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-logging_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-registry_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-security_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-stats_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:util-tunable_2.12:jar:17.12.0:compile
[INFO] | +- com.github.ben-manes.caffeine:caffeine:jar:2.3.4:compile
[INFO] | \- com.google.code.findbugs:jsr305:jar:2.0.1:compile
[INFO] +- org.scala-lang.modules:scala-java8-compat_2.12:jar:0.9.0:compile
[INFO] +- com.twitter:finagle-http_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:finagle-base-http_2.12:jar:17.12.0:compile
[INFO] | | \- com.twitter:finagle-netty3_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:finagle-netty3-http_2.12:jar:17.12.0:compile
[INFO] | +- com.twitter:finagle-netty4-http_2.12:jar:17.12.0:compile
[INFO] | | +- com.twitter:finagle-netty4_2.12:jar:17.12.0:compile
[INFO] | | \- io.netty:netty-codec-http:jar:4.1.16.Final:compile
[INFO] | | \- io.netty:netty-codec:jar:4.1.16.Final:compile
[INFO] | +- com.twitter:finagle-http2_2.12:jar:17.12.0:compile
[INFO] | | +- io.netty:netty-codec-http2:jar:4.1.16.Final:compile
[INFO] | | +- io.netty:netty-handler:jar:4.1.16.Final:compile
[INFO] | | | \- io.netty:netty-buffer:jar:4.1.16.Final:compile
[INFO] | | +- io.netty:netty-transport:jar:4.1.16.Final:compile
[INFO] | | | \- io.netty:netty-resolver:jar:4.1.16.Final:compile
[INFO] | | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.16.Final:compile
[INFO] | | | \- io.netty:netty-common:jar:4.1.16.Final:compile
[INFO] | | +- io.netty:netty-transport-native-unix-common:jar:4.1.16.Final:compile
[INFO] | | \- io.netty:netty-handler-proxy:jar:4.1.16.Final:compile
[INFO] | | \- io.netty:netty-codec-socks:jar:4.1.16.Final:compile
[INFO] | \- com.twitter:util-collection_2.12:jar:17.12.0:compile
I do see similar issue when upgrading finagle version here but in my case util dependencies are valid.
Any Idea how to resolve this issue?

Feign Client with Spring Cloud Brixton SR7

I am trying to use Feign Client with the new Spring Cloud version (Brixton.SR7) but I am having problems.
Every time I try to make a REST call to a registered service in Eureka Server I got the exception:
Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: rx/functions/Func1] with root cause
java.lang.ClassNotFoundException: rx.functions.Func1
My main class is enabling the FeignClients using #EnableFeignClients.
There are my project dependencies:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.4.2.RELEASE</version>
<relativePath/>
</parent>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Brixton.SR7</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Using an old Spring Cloud version (Brixton.SR6) it was running ok.
Should I add some dependency RxJava?
edit:
SR6 dependency tree:
[INFO] com.pillo:client:jar:0.0.1-SNAPSHOT
[INFO] +- org.springframework.cloud:spring-cloud-starter-eureka:jar:1.1.6.RELEASE:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-web:jar:1.4.2.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot-starter:jar:1.4.2.RELEASE:compile
[INFO] | | | +- org.springframework.boot:spring-boot-starter-logging:jar:1.4.2.RELEASE:compile
[INFO] | | | | +- ch.qos.logback:logback-classic:jar:1.1.7:compile
[INFO] | | | | | \- ch.qos.logback:logback-core:jar:1.1.7:compile
[INFO] | | | | +- org.slf4j:jcl-over-slf4j:jar:1.7.21:compile
[INFO] | | | | +- org.slf4j:jul-to-slf4j:jar:1.7.21:compile
[INFO] | | | | \- org.slf4j:log4j-over-slf4j:jar:1.7.21:compile
[INFO] | | | \- org.yaml:snakeyaml:jar:1.17:runtime
[INFO] | | +- org.springframework.boot:spring-boot-starter-tomcat:jar:1.4.2.RELEASE:compile
[INFO] | | | +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.6:compile
[INFO] | | | +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.6:compile
[INFO] | | | \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.6:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.4:compile
[INFO] | | \- org.springframework:spring-webmvc:jar:4.3.4.RELEASE:compile
[INFO] | | \- org.springframework:spring-expression:jar:4.3.4.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-starter:jar:1.1.3.RELEASE:compile
[INFO] | | +- org.springframework.cloud:spring-cloud-context:jar:1.1.3.RELEASE:compile
[INFO] | | \- org.springframework.security:spring-security-rsa:jar:1.0.1.RELEASE:compile
[INFO] | | \- org.bouncycastle:bcpkix-jdk15on:jar:1.47:compile
[INFO] | | \- org.bouncycastle:bcprov-jdk15on:jar:1.47:compile
[INFO] | +- org.springframework.cloud:spring-cloud-netflix-core:jar:1.1.6.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot:jar:1.4.2.RELEASE:compile
[INFO] | | \- org.springframework.boot:spring-boot-autoconfigure:jar:1.4.2.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-netflix-eureka-client:jar:1.1.6.RELEASE:compile
[INFO] | +- com.netflix.eureka:eureka-client:jar:1.4.10:compile
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.3.7:runtime
[INFO] | | | \- stax:stax-api:jar:1.0.1:runtime
[INFO] | | +- com.netflix.netflix-commons:netflix-eventbus:jar:0.3.0:runtime
[INFO] | | +- com.netflix.archaius:archaius-core:jar:0.7.4:compile
[INFO] | | +- javax.ws.rs:jsr311-api:jar:1.1.1:runtime
[INFO] | | +- com.netflix.servo:servo-core:jar:0.10.1:runtime
[INFO] | | | \- com.netflix.servo:servo-internal:jar:0.10.1:runtime
[INFO] | | +- com.sun.jersey:jersey-core:jar:1.19.1:runtime
[INFO] | | +- com.sun.jersey:jersey-client:jar:1.19.1:runtime
[INFO] | | +- com.sun.jersey.contribs:jersey-apache-client4:jar:1.19.1:runtime
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.5.2:runtime
[INFO] | | | +- org.apache.httpcomponents:httpcore:jar:4.4.5:runtime
[INFO] | | | \- commons-codec:commons-codec:jar:1.10:runtime
[INFO] | | +- com.google.inject:guice:jar:4.0:runtime
[INFO] | | | \- aopalliance:aopalliance:jar:1.0:runtime
[INFO] | | +- com.netflix.governator:governator-api:jar:1.12.10:runtime
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.4:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.8.4:compile
[INFO] | +- com.netflix.eureka:eureka-core:jar:1.4.10:compile
[INFO] | | +- com.amazonaws:aws-java-sdk-core:jar:1.11.18:runtime
[INFO] | | | \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.4:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-ec2:jar:1.11.18:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-autoscaling:jar:1.11.18:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-sts:jar:1.11.18:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-route53:jar:1.11.18:runtime
[INFO] | | +- com.netflix.governator:governator:jar:1.12.10:runtime
[INFO] | | | \- com.netflix.governator:governator-core:jar:1.12.10:runtime
[INFO] | | \- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:runtime
[INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:runtime
[INFO] | | \- org.codehaus.woodstox:stax2-api:jar:3.1.4:runtime
[INFO] | +- org.springframework.cloud:spring-cloud-starter-archaius:jar:1.1.6.RELEASE:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.8:compile
[INFO] | | \- com.google.guava:guava:jar:18.0:compile
[INFO] | +- com.netflix.ribbon:ribbon-eureka:jar:2.2.0:compile
[INFO] | | \- org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] | \- com.thoughtworks.xstream:xstream:jar:1.4.2:compile
[INFO] | +- xmlpull:xmlpull:jar:1.1.3.1:compile
[INFO] | \- xpp3:xpp3_min:jar:1.1.4c:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-feign:jar:1.1.6.RELEASE:compile
[INFO] | +- org.springframework:spring-web:jar:4.3.4.RELEASE:compile
[INFO] | | +- org.springframework:spring-aop:jar:4.3.4.RELEASE:compile
[INFO] | | +- org.springframework:spring-beans:jar:4.3.4.RELEASE:compile
[INFO] | | +- org.springframework:spring-context:jar:4.3.4.RELEASE:compile
[INFO] | | \- org.springframework:spring-core:jar:4.3.4.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-commons:jar:1.1.3.RELEASE:compile
[INFO] | | \- org.springframework.security:spring-security-crypto:jar:4.1.3.RELEASE:compile
[INFO] | +- com.netflix.feign:feign-core:jar:8.16.2:compile
[INFO] | | \- org.jvnet:animal-sniffer-annotation:jar:1.0:runtime
[INFO] | +- com.netflix.feign:feign-slf4j:jar:8.16.2:compile
[INFO] | \- com.netflix.feign:feign-hystrix:jar:8.16.2:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-ribbon:jar:1.1.6.RELEASE:compile
[INFO] | +- com.netflix.ribbon:ribbon:jar:2.2.0:compile
[INFO] | | +- com.netflix.ribbon:ribbon-transport:jar:2.2.0:runtime
[INFO] | | | +- io.reactivex:rxnetty-contexts:jar:0.4.9:runtime
[INFO] | | | \- io.reactivex:rxnetty-servo:jar:0.4.9:runtime
[INFO] | | +- javax.inject:javax.inject:jar:1:runtime
[INFO] | | \- io.reactivex:rxnetty:jar:0.4.9:runtime
[INFO] | | +- io.netty:netty-codec-http:jar:4.0.27.Final:runtime
[INFO] | | | +- io.netty:netty-codec:jar:4.0.27.Final:runtime
[INFO] | | | \- io.netty:netty-handler:jar:4.0.27.Final:runtime
[INFO] | | \- io.netty:netty-transport-native-epoll:jar:4.0.27.Final:runtime
[INFO] | | +- io.netty:netty-common:jar:4.0.27.Final:runtime
[INFO] | | +- io.netty:netty-buffer:jar:4.0.27.Final:runtime
[INFO] | | \- io.netty:netty-transport:jar:4.0.27.Final:runtime
[INFO] | +- com.netflix.ribbon:ribbon-core:jar:2.2.0:compile
[INFO] | | \- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- com.netflix.ribbon:ribbon-httpclient:jar:2.2.0:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | | \- com.netflix.netflix-commons:netflix-commons-util:jar:0.1.1:runtime
[INFO] | +- com.netflix.ribbon:ribbon-loadbalancer:jar:2.2.0:compile
[INFO] | | \- com.netflix.netflix-commons:netflix-statistics:jar:0.1.1:runtime
[INFO] | \- io.reactivex:rxjava:jar:1.1.5:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-hystrix:jar:1.1.6.RELEASE:compile
[INFO] | +- com.netflix.hystrix:hystrix-core:jar:1.5.3:compile
[INFO] | | \- org.hdrhistogram:HdrHistogram:jar:2.1.7:compile
[INFO] | +- com.netflix.hystrix:hystrix-metrics-event-stream:jar:1.5.3:compile
[INFO] | \- com.netflix.hystrix:hystrix-javanica:jar:1.5.3:compile
[INFO] | +- org.aspectj:aspectjweaver:jar:1.8.9:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.1:compile
[INFO] | \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO] \- org.projectlombok:lombok:jar:1.16.10:provided
SR7 dependency tree:
[INFO] com.pillo:client:jar:0.0.1-SNAPSHOT
[INFO] +- org.springframework.cloud:spring-cloud-starter-eureka:jar:1.1.7.RELEASE:compile
[INFO] | +- org.springframework.boot:spring-boot-starter-web:jar:1.4.2.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot-starter:jar:1.4.2.RELEASE:compile
[INFO] | | | +- org.springframework.boot:spring-boot-starter-logging:jar:1.4.2.RELEASE:compile
[INFO] | | | | +- ch.qos.logback:logback-classic:jar:1.1.7:compile
[INFO] | | | | | \- ch.qos.logback:logback-core:jar:1.1.7:compile
[INFO] | | | | +- org.slf4j:jcl-over-slf4j:jar:1.7.21:compile
[INFO] | | | | +- org.slf4j:jul-to-slf4j:jar:1.7.21:compile
[INFO] | | | | \- org.slf4j:log4j-over-slf4j:jar:1.7.21:compile
[INFO] | | | \- org.yaml:snakeyaml:jar:1.17:runtime
[INFO] | | +- org.springframework.boot:spring-boot-starter-tomcat:jar:1.4.2.RELEASE:compile
[INFO] | | | +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.6:compile
[INFO] | | | +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.6:compile
[INFO] | | | \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.6:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.4:compile
[INFO] | | \- org.springframework:spring-webmvc:jar:4.3.4.RELEASE:compile
[INFO] | | \- org.springframework:spring-expression:jar:4.3.4.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-starter:jar:1.1.3.RELEASE:compile
[INFO] | | +- org.springframework.cloud:spring-cloud-context:jar:1.1.3.RELEASE:compile
[INFO] | | \- org.springframework.security:spring-security-rsa:jar:1.0.3.RELEASE:compile
[INFO] | | \- org.bouncycastle:bcpkix-jdk15on:jar:1.55:compile
[INFO] | | \- org.bouncycastle:bcprov-jdk15on:jar:1.55:compile
[INFO] | +- org.springframework.cloud:spring-cloud-netflix-core:jar:1.1.7.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot:jar:1.4.2.RELEASE:compile
[INFO] | | \- org.springframework.boot:spring-boot-autoconfigure:jar:1.4.2.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-netflix-eureka-client:jar:1.1.7.RELEASE:compile
[INFO] | +- com.netflix.eureka:eureka-client:jar:1.4.12:compile
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.3.7:runtime
[INFO] | | | \- stax:stax-api:jar:1.0.1:runtime
[INFO] | | +- com.netflix.netflix-commons:netflix-eventbus:jar:0.3.0:runtime
[INFO] | | +- com.netflix.archaius:archaius-core:jar:0.7.4:compile
[INFO] | | +- javax.ws.rs:jsr311-api:jar:1.1.1:runtime
[INFO] | | +- com.netflix.servo:servo-core:jar:0.10.1:runtime
[INFO] | | | \- com.netflix.servo:servo-internal:jar:0.10.1:runtime
[INFO] | | +- com.sun.jersey:jersey-core:jar:1.19.1:runtime
[INFO] | | +- com.sun.jersey:jersey-client:jar:1.19.1:runtime
[INFO] | | +- com.sun.jersey.contribs:jersey-apache-client4:jar:1.19.1:runtime
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.5.2:runtime
[INFO] | | | +- org.apache.httpcomponents:httpcore:jar:4.4.5:runtime
[INFO] | | | \- commons-codec:commons-codec:jar:1.10:runtime
[INFO] | | +- com.google.inject:guice:jar:4.0:runtime
[INFO] | | | \- aopalliance:aopalliance:jar:1.0:runtime
[INFO] | | +- com.netflix.governator:governator-api:jar:1.12.10:runtime
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.4:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.8.4:compile
[INFO] | +- com.netflix.eureka:eureka-core:jar:1.4.12:compile
[INFO] | | +- com.amazonaws:aws-java-sdk-core:jar:1.11.18:runtime
[INFO] | | | \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.4:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-ec2:jar:1.11.18:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-autoscaling:jar:1.11.18:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-sts:jar:1.11.18:runtime
[INFO] | | +- com.amazonaws:aws-java-sdk-route53:jar:1.11.18:runtime
[INFO] | | +- com.netflix.governator:governator:jar:1.12.10:runtime
[INFO] | | | \- com.netflix.governator:governator-core:jar:1.12.10:runtime
[INFO] | | \- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:runtime
[INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:runtime
[INFO] | | \- org.codehaus.woodstox:stax2-api:jar:3.1.4:runtime
[INFO] | +- org.springframework.cloud:spring-cloud-starter-archaius:jar:1.1.7.RELEASE:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.8:compile
[INFO] | | \- com.google.guava:guava:jar:18.0:compile
[INFO] | +- com.netflix.ribbon:ribbon-eureka:jar:2.2.0:compile
[INFO] | | \- org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] | \- com.thoughtworks.xstream:xstream:jar:1.4.9:compile
[INFO] | +- xmlpull:xmlpull:jar:1.1.3.1:compile
[INFO] | \- xpp3:xpp3_min:jar:1.1.4c:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-feign:jar:1.1.7.RELEASE:compile
[INFO] | +- org.springframework:spring-web:jar:4.3.4.RELEASE:compile
[INFO] | | +- org.springframework:spring-aop:jar:4.3.4.RELEASE:compile
[INFO] | | +- org.springframework:spring-beans:jar:4.3.4.RELEASE:compile
[INFO] | | +- org.springframework:spring-context:jar:4.3.4.RELEASE:compile
[INFO] | | \- org.springframework:spring-core:jar:4.3.4.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-commons:jar:1.1.3.RELEASE:compile
[INFO] | | \- org.springframework.security:spring-security-crypto:jar:4.1.3.RELEASE:compile
[INFO] | +- com.netflix.feign:feign-core:jar:8.16.2:compile
[INFO] | | \- org.jvnet:animal-sniffer-annotation:jar:1.0:runtime
[INFO] | +- com.netflix.feign:feign-slf4j:jar:8.16.2:compile
[INFO] | \- com.netflix.feign:feign-hystrix:jar:8.16.2:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-ribbon:jar:1.1.7.RELEASE:compile
[INFO] | +- com.netflix.ribbon:ribbon:jar:2.2.0:compile
[INFO] | | +- com.netflix.ribbon:ribbon-transport:jar:2.2.0:runtime
[INFO] | | | +- io.reactivex:rxnetty-contexts:jar:0.4.9:runtime
[INFO] | | | \- io.reactivex:rxnetty-servo:jar:0.4.9:runtime
[INFO] | | +- javax.inject:javax.inject:jar:1:runtime
[INFO] | | \- io.reactivex:rxnetty:jar:0.4.9:runtime
[INFO] | | +- io.netty:netty-codec-http:jar:4.0.27.Final:runtime
[INFO] | | | +- io.netty:netty-codec:jar:4.0.27.Final:runtime
[INFO] | | | \- io.netty:netty-handler:jar:4.0.27.Final:runtime
[INFO] | | \- io.netty:netty-transport-native-epoll:jar:4.0.27.Final:runtime
[INFO] | | +- io.netty:netty-common:jar:4.0.27.Final:runtime
[INFO] | | +- io.netty:netty-buffer:jar:4.0.27.Final:runtime
[INFO] | | \- io.netty:netty-transport:jar:4.0.27.Final:runtime
[INFO] | +- com.netflix.ribbon:ribbon-core:jar:2.2.0:compile
[INFO] | | \- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- com.netflix.ribbon:ribbon-httpclient:jar:2.2.0:compile
[INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] | | \- com.netflix.netflix-commons:netflix-commons-util:jar:0.1.1:runtime
[INFO] | +- com.netflix.ribbon:ribbon-loadbalancer:jar:2.2.0:compile
[INFO] | | \- com.netflix.netflix-commons:netflix-statistics:jar:0.1.1:runtime
[INFO] | \- io.reactivex:rxjava:jar:1.1.10:compile
[INFO] +- org.springframework.cloud:spring-cloud-starter-hystrix:jar:1.1.7.RELEASE:compile
[INFO] | +- com.netflix.hystrix:hystrix-core:jar:1.5.3:compile
[INFO] | | \- org.hdrhistogram:HdrHistogram:jar:2.1.7:compile
[INFO] | +- com.netflix.hystrix:hystrix-metrics-event-stream:jar:1.5.3:compile
[INFO] | \- com.netflix.hystrix:hystrix-javanica:jar:1.5.3:compile
[INFO] | +- org.aspectj:aspectjweaver:jar:1.8.9:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.1:compile
[INFO] | \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO] \- org.projectlombok:lombok:jar:1.16.10:provided
Thanks!

building scala by maven error

pom is here:
https://github.com/leocook/spark-ml/blob/master/spark_ml/pom.xml
and
scala code is
val conf = new SparkConf()
val sc = new SparkContext(conf)
val line = sc.textFile(args(0))
line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println)
sc.stop()
when i build project,error message is:

There are to problems:
scala.Cloneable not found, which means your Scala library is not setup properly.
as mentioned in comments, you are mixing 2.7.0 and 2.10.4 scala versions.
You may want to inspect why the different versions are appearing using mvn dependency:tree, you will following output:
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) # ML ---
[INFO] org.leocook.spark:ML:jar:1.0-SNAPSHOT
[INFO] +- org.apache.spark:spark-core_2.10:jar:1.3.0-cdh5.4.4:compile
[INFO] | +- com.twitter:chill_2.10:jar:0.5.0:compile
[INFO] | | \- com.esotericsoftware.kryo:kryo:jar:2.21:compile
[INFO] | | +- com.esotericsoftware.reflectasm:reflectasm:jar:shaded:1.07:compile
[INFO] | | +- com.esotericsoftware.minlog:minlog:jar:1.2:compile
[INFO] | | \- org.objenesis:objenesis:jar:1.2:compile
[INFO] | +- com.twitter:chill-java:jar:0.5.0:compile
[INFO] | +- org.apache.hadoop:hadoop-client:jar:2.6.0-cdh5.4.4:compile
[INFO] | | +- org.apache.hadoop:hadoop-common:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | | | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | | | +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | | | +- commons-io:commons-io:jar:2.4:compile
[INFO] | | | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | | | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | | | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | | | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
[INFO] | | | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] | | | +- org.apache.avro:avro:jar:1.7.6-cdh5.4.4:compile
[INFO] | | | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | | | +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] | | | +- org.apache.hadoop:hadoop-auth:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | | \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] | | | | +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] | | | | +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] | | | | \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] | | | +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] | | | +- org.htrace:htrace-core:jar:3.0.4:compile
[INFO] | | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | | +- org.apache.hadoop:hadoop-hdfs:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.4:compile
[INFO] | | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | | \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.0-cdh5.4.4:compile
[INFO] | | +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.0-cdh5.4.4:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] | | | | +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] | | | | \- javax.activation:activation:jar:1.1:compile
[INFO] | | | +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] | | | +- com.sun.jersey:jersey-client:jar:1.9:compile
[INFO] | | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:compile
[INFO] | | | \- org.codehaus.jackson:jackson-xc:jar:1.8.8:compile
[INFO] | | +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.0-cdh5.4.4:compile
[INFO] | | +- org.apache.hadoop:hadoop-aws:jar:2.6.0-cdh5.4.4:compile
[INFO] | | | \- com.amazonaws:aws-java-sdk:jar:1.7.4:compile
[INFO] | | \- org.apache.hadoop:hadoop-annotations:jar:2.6.0-cdh5.4.4:compile
[INFO] | +- org.apache.spark:spark-network-common_2.10:jar:1.3.0-cdh5.4.4:compile
[INFO] | +- org.apache.spark:spark-network-shuffle_2.10:jar:1.3.0-cdh5.4.4:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
[INFO] | | +- commons-codec:commons-codec:jar:1.4:compile
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.1.2:compile
[INFO] | | +- org.apache.httpcomponents:httpcore:jar:4.1.2:compile
[INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:compile
[INFO] | | +- org.apache.curator:curator-framework:jar:2.4.0:compile
[INFO] | | +- org.apache.zookeeper:zookeeper:jar:3.4.5:compile
[INFO] | | | \- jline:jline:jar:0.9.94:compile
[INFO] | | \- com.google.guava:guava:jar:14.0.1:compile
[INFO] | +- org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.5:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.5:compile
[INFO] | +- log4j:log4j:jar:1.2.17:compile
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
[INFO] | +- com.ning:compress-lzf:jar:1.0.0:compile
[INFO] | +- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
[INFO] | +- net.jpountz.lz4:lz4:jar:1.2.0:compile
[INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.4.5:compile
[INFO] | +- commons-net:commons-net:jar:2.2:compile
[INFO] | +- org.spark-project.akka:akka-remote_2.10:jar:2.2.3-shaded-protobuf:compile
[INFO] | | +- org.spark-project.akka:akka-actor_2.10:jar:2.2.3-shaded-protobuf:compile
[INFO] | | | \- com.typesafe:config:jar:1.0.2:compile
[INFO] | | +- io.netty:netty:jar:3.6.6.Final:compile
[INFO] | | +- org.spark-project.protobuf:protobuf-java:jar:2.4.1-shaded:compile
[INFO] | | \- org.uncommons.maths:uncommons-maths:jar:1.2.2a:compile
[INFO] | +- org.spark-project.akka:akka-slf4j_2.10:jar:2.2.3-shaded-protobuf:compile
[INFO] | +- org.json4s:json4s-jackson_2.10:jar:3.2.10:compile
[INFO] | | \- org.json4s:json4s-core_2.10:jar:3.2.10:compile
[INFO] | | +- org.json4s:json4s-ast_2.10:jar:3.2.10:compile
[INFO] | | \- org.scala-lang:scalap:jar:2.10.0:compile
[INFO] | | \- org.scala-lang:scala-compiler:jar:2.10.0:compile
[INFO] | | \- org.scala-lang:scala-reflect:jar:2.10.0:compile
[INFO] | +- org.apache.mesos:mesos:jar:shaded-protobuf:0.21.0:compile
[INFO] | +- io.netty:netty-all:jar:4.0.23.Final:compile
[INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] | +- io.dropwizard.metrics:metrics-core:jar:3.1.0:compile
[INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:3.1.0:compile
[INFO] | +- io.dropwizard.metrics:metrics-json:jar:3.1.0:compile
[INFO] | +- io.dropwizard.metrics:metrics-graphite:jar:3.1.0:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.2.3:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.2.3:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.2.3:compile
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.10:jar:2.2.3:compile
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] | +- oro:oro:jar:2.0.8:compile
[INFO] | +- org.tachyonproject:tachyon-client:jar:0.5.0:compile
[INFO] | | \- org.tachyonproject:tachyon:jar:0.5.0:compile
[INFO] | +- org.spark-project:pyrolite:jar:2.0.1:compile
[INFO] | +- net.sf.py4j:py4j:jar:0.8.2.1:compile
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.scala-lang:scala-library:jar:2.7.0:compile
[INFO] +- junit:junit:jar:4.4:test
[INFO] \- org.specs:specs:jar:1.2.5:test
[INFO] +- org.scalatest:scalatest:jar:0.9.1:test
[INFO] +- org.scalacheck:scalacheck:jar:1.2:test
[INFO] \- org.jmock:jmock:jar:2.4.0:test
[INFO] +- org.hamcrest:hamcrest-core:jar:1.1:test
[INFO] \- org.hamcrest:hamcrest-library:jar:1.1:test

Scalding on EMR: Hadoop job fails with NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;

Basically I need to run a scalding job on EMR. The same job runs perfectly fine on local hadoop on my macbook, but fails on Hadoop on EMR.
I am trying hard to get help for this issue in the cascading-user and scala-user groups as well, and haven't been able to. So far I haven't made much progress after trying various changes in the past couple days.
Here is the error before I delve into the details:
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at com.aggregation.job.DataAggregation$.(DataAggregation.scala:30)
at com.aggregation.job.DataAggregation$.(DataAggregation.scala)
at com.aggregation.job.DataAggregation.main(DataAggregation.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
They said it might be a scala conflict with the binaries, but I couldn't see anything obvious. It would be great if someone could help with figuring it out.
Environment:
Amazon EMR AMI: 3.8.0 (which includes Scala 2.11.1, Hadoop 2.4.0, Java 1.7.0_76 - AMI details here)
Application environment: Scalding 0.15.0, Scala 2.11.1, Java 1.7.0_80, Hadoop 2.4.0
I have tried various changes to configuration and even manually installing a higher scala version in EMR, but so far the error is not going away.
Please help! Thank you.
L
Setup
build.sbt:
scalaVersion := "2.11.1"
ivyScala := ivyScala.value map {
_.copy(overrideScalaVersion = true)
}
dependencies.sbt:
import AssemblyKeys._
val hadoopVersion = "2.4.0"
val scaldingVersion = "0.15.0"
libraryDependencies ++= Seq(
"com.twitter" %% "scalding-core" % scaldingVersion,
"com.twitter" %% "scalding-json" % scaldingVersion,
"com.twitter" %% "scalding-jdbc" % scaldingVersion,
"com.github.nscala-time" %% "nscala-time" % "2.0.0",
"org.apache.hadoop" % "hadoop-common" % hadoopVersion % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion % "provided"
)
excludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
val excludes = Set(
"jsp-api-2.1-6.1.14.jar",
"jsp-2.1-6.1.14.jar",
"jasper-compiler-5.5.12.jar",
"minlog-1.2.jar", // Otherwise causes conflicts with Kyro (which bundles it)
"janino-2.5.16.jar", // Janino includes a broken signature, and is not needed anyway
"commons-beanutils-core-1.8.0.jar", // Clash with each other and with commons-collections
"commons-beanutils-1.7.0.jar", // "
"hadoop-core-1.2.1.jar", // Provided by Amazon EMR. Delete this line if you're not on EMR
"hadoop-tools-1.2.1.jar" // "
)
cp filter { jar => excludes(jar.data.getName) }
}
resolvers ++= Seq(
"Conjars repo" at "http://conjars.org/repo"
)
assembly.sbt
import AssemblyKeys._
assemblySettings
mergeStrategy in assembly := Merge.mergeStrategy
project/build.properties:
sbt.version=0.13.1
project/assembly.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.0")
Finally this is the dependency tree using the sbt-dependency-graph plugin to see if I've got the right versions. Sorry, it's quite long, if I should be displaying this info any other way, please recommend suggestions.
sbt dependency tree:
[info] com.abc.aggregator_2.11:0.1-20150628T184441 [S]
[info] +-com.github.nscala-time:nscala-time_2.11:2.0.0 [S]
[info] | +-joda-time:joda-time:2.7
[info] | +-org.joda:joda-convert:1.2
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-jdbc_2.11:0.15.0 [S]
[info] | +-cascading:cascading-jdbc-core:2.6.0
[info] | | +-com.google.guava:guava:15.0
[info] | |
[info] | +-cascading:cascading-jdbc-mysql:2.6.0
[info] | | +-cascading:cascading-jdbc-core:2.6.0
[info] | | | +-com.google.guava:guava:15.0
[info] | | |
[info] | | +-com.google.guava:guava:15.0
[info] | | +-mysql:mysql-connector-java:5.1.25
[info] | |
[info] | +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-json_2.11:0.15.0 [S]
[info] +-com.fasterxml.jackson.module:jackson-module-scala_2.11:2.4.2 [S]
[info] | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-databind:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.0 (evicted by: 2.4.2)
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | |
[info] | +-com.google.code.findbugs:jsr305:2.0.1
[info] | +-com.google.guava:guava:15.0
[info] | +-com.thoughtworks.paranamer:paranamer:2.6
[info] | +-org.scala-lang:scala-reflect:2.11.2 [S]
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-org.json4s:json4s-native_2.11:3.2.11 [S]
[info] +-org.json4s:json4s-core_2.11:3.2.11 [S]
[info] +-com.thoughtworks.paranamer:paranamer:2.6
[info] +-org.json4s:json4s-ast_2.11:3.2.11 [S]
[info] +-org.scala-lang:scalap:2.11.0
[info] +-org.scala-lang:scala-compiler:2.11.1 [S]
[info] +-org.scala-lang.modules:scala-parser-combinators_2.11:1.0.1 [S]
[info] +-org.scala-lang.modules:scala-xml_2.11:1.0.2 [S]
[info] +-org.scala-lang:scala-reflect:2.11.1 (evicted by: 2.11.2)
[info] +-org.scala-lang:scala-reflect:2.11.2 [S]
[info]
[success] Total time: 12 s, completed Jun 28, 2015 11:44:54 AM
Further info as requested:
I build the fat jar using 'sbt assembly', and currently I'm using the AWS console with a "Custom JAR" step to test this out before automating the process.
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt
UPDATE:
I was able to get past the above error by providing the HADOOP_CLASSPATH pointing to the scala 2.11.1 jars, while excluding the same from the sbt assembly step. This was passed in using hadoop-user-env.sh and seemed to work for the master node. However once it got to the mapper step it once again failed with another Scala error. Now I am stuck on this step.
Assuming this is because the mappers and reducers aren't seeing the HADOOP_CLASSPATH update, I tried including the -libjars argument pointing to the scala jar files on hadoop master itself. But this (below) doesn't seem to be working.
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation -libjars /usr/share/scala/lib/scala-library.jar,/usr/share/scala/lib/scala-reflect.jar --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt

Fixed. So it does happen that there were multiple scala jars in the EMR instances, and they weren't coming from my application jar.
The 2.10 jar was hiding in /usr/share/aws/emr/emrfs/lib apart from the installed location for the 2.11 binaries under /usr/share/scala. So I got rid of the 2.10 jar in all instances of the cluster, and my job completed successfully. Now I will create a bootstrap action for this.
$ sudo find / -name "scala-library-2.10.*.jar" -exec rm -rf {} \;
FYI, these are the paths it was present under:
[ec2-user#ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.11.*.jar"
/home/hadoop/.versions/hbase-0.94.18/lib/scala-library-2.11.0.jar
/usr/share/doc/scala/api/jars/scala-library-2.11.1-javadoc.jar
[ec2-user#ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.10.*.jar"
/usr/share/aws/emr/emrfs/lib/scala-library-2.10.5.jar

After struggling with a similar issue..... which is clearly a version mismatch between jar dependencies. I fixed it by first logging into the EMR master, then opening spark-shell reveals the exact scala version that is being used. Now take that to configure your own dependencies in the pom file and it should work.

We Keep Coding

iphone swift flutter scala powershell matlab mongodb postgresql perl eclipse

sbt deduplicate: different file contents found in the following: - scala

Related

How to run the legacy Mongo Casbah drivers alongside the modern Mongo scala drivers

java.lang.NoClassDefFoundError: Could not initialize class com.twitter.finagle.Init$

Feign Client with Spring Cloud Brixton SR7

building scala by maven error

Scalding on EMR: Hadoop job fails with NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;

Categories

Resources