Compare ZonedDateTime in ZIO Quill - scala

I'm getting an error trying to filter based on ZonedDateTime using >. It works with ==.
I'm using scala 3.2.0 and Quill 4.6.0.
It seems it should work based on the Quill Docs. What could be missing?
case class Post(id: Long, createdAt: ZonedDateTime)
import io.getquill.*
import io.getquill.extras.ZonedDateTimeOps
def getUpdated(since: ZonedDateTime): Task[List[Post]] = {
run {
query[Post]
.filter(_.createdAt > lift(since))
}
.provideEnvironment(ZEnvironment(dataSource))
}
I'm getting the following error message.
[error] |s"==== Tree cannot be parsed to 'Ast' ====
[error] | ZonedDateTimeOps(_$7.createdAt)
[error] |==== Extractors ===
[error] | Apply(Ident("ZonedDateTimeOps"), List(Select(Ident("_$7"), "createdAt")))
[error] |==== Stacktrace ===
[error] | java.base/java.lang.Thread.getStackTrace(Thread.java:1610)
[error] | io.getquill.parser.engine.failParse$.apply(failParse.scala:27)
[error] | io.getquill.parser.engine.failParse$.apply(failParse.scala:12)
[error] | io.getquill.parser.engine.Parser.error(Parser.scala:10)
[error] | io.getquill.parser.engine.Parser.error$(Parser.scala:7)
[error] | io.getquill.parser.engine.ParserChain$$anon$1.error(ParserChain.scala:15)
[error] | io.getquill.parser.engine.Parser.apply$$anonfun$1(Parser.scala:9)
[error] | scala.Option.getOrElse(Option.scala:201)
[error] | io.getquill.parser.engine.Parser.apply(Parser.scala:9)
[error] | io.getquill.parser.engine.Parser.apply$(Parser.scala:7)
[error] | io.getquill.parser.engine.ParserChain$$anon$1.apply(ParserChain.scala:15)
[error] | io.getquill.parser.OperationsParser$$anon$20.applyOrElse(Parser.scala:889)
[error] | io.getquill.parser.OperationsParser$$anon$20.applyOrElse(Parser.scala:875)
[error] | scala.PartialFunction$Lifted.apply(PartialFunction.scala:313)
[error] | scala.PartialFunction$Lifted.apply(PartialFunction.scala:309)
[error] | io.getquill.parser.engine.ParserChain$.io$getquill$parser$engine$ParserChain$OrElse$$anon$2$$_$$anonfun$1$$anonfun$1(ParserChain.scala:43)
[error] | scala.Option.orElse(Option.scala:477)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
[error] | io.getquill.parser.engine.Parser.apply(Parser.scala:9)
[error] | io.getquill.parser.engine.Parser.apply$(Parser.scala:7)
[error] | io.getquill.parser.engine.ParserChain$$anon$1.apply(ParserChain.scala:15)
[error] | io.getquill.parser.QueryParser$$anon$13.applyOrElse(Parser.scala:540)
[error] | io.getquill.parser.QueryParser$$anon$13.applyOrElse(Parser.scala:520)
[error] | scala.PartialFunction$Lifted.apply(PartialFunction.scala:313)
[error] | scala.PartialFunction$Lifted.apply(PartialFunction.scala:309)
[error] | io.getquill.parser.engine.ParserChain$.io$getquill$parser$engine$ParserChain$OrElse$$anon$2$$_$$anonfun$1$$anonfun$1(ParserChain.scala:43)
[error] | scala.Option.orElse(Option.scala:477)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)
[error] | io.getquill.parser.engine.ParserChain$OrElse$$anon$2.$anonfun$1(ParserChain.scala:43)
[error] | scala.PartialFunction$$anonfun$fromFunction$1.applyOrElse(PartialFunction.scala:338)
[error] | scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:35)

Related

ZIO quill groupByMap on query with join

groupByMap in ZIO Quill is throwing an exception when trying to group by on a query with a join.
I'm using scala 3.2.0 and quill 4.6.0.
What's the proper way to use group by on a join query in zio quill?
case class Post(id: Long, title: String)
case class Comment(id: Long, message: String, postId: Long)
def getPostCommentsCount(): Task[List[PostCommentCounts]] =
run {
query[Post]
.join(query[Comment])
.on(_.id == _.postId)
.groupByMap((p, c) => p.id)((p, c) =>
PostCommentCounts(
p.id,
count(c.id.toInt),
),
)
}
.provideEnvironment(ZEnvironment(dataSource))
I'm getting the following error.
[error] 37 | run {
[error] | ^
[error] |Query splicing failed due to error: scala.MatchError: querySchema("Post").join(querySchema("Comment")).on((x3, x4) => x3.id == x4.postId).groupByMap(x$1 => x$1._1.id)(x$1 => Post(id: x$1._1.id, title: x$1._1.title, message: x$1._1.message, locationId: x$1._1.locationId, createdAt: x$1._1.createdAt)) (of class io.getquill.ast.GroupByMap)
[error] | at io.getquill.norm.capture.AvoidAliasConflict.apply$$anonfun$1(AvoidAliasConflict.scala:203)
[error] | at io.getquill.util.Interpolator$Traceable.andReturnIf(Interpolator.scala:193)
[error] | at io.getquill.norm.capture.AvoidAliasConflict.apply(AvoidAliasConflict.scala:205)
[error] | at io.getquill.ast.StatefulTransformer.apply(StatefulTransformer.scala:9)
[error] | at io.getquill.ast.StatefulTransformer.apply$(StatefulTransformer.scala:3)
[error] | at io.getquill.norm.capture.AvoidAliasConflict.apply(AvoidAliasConflict.scala:43)
[error] | at io.getquill.ast.StatefulTransformer.apply(StatefulTransformer.scala:153)
[error] | at io.getquill.ast.StatefulTransformer.apply$(StatefulTransformer.scala:3)
[error] | at io.getquill.norm.capture.AvoidAliasConflict.recurseAndApply$$anonfun$1(AvoidAliasConflict.scala:81)
[error] | at io.getquill.util.Interpolator$Traceable.andReturnIf(Interpolator.scala:193)
[error] | at io.getquill.norm.capture.AvoidAliasConflict.recurseAndApply(AvoidAliasConflict.scala:92)
[error] | at io.getquill.norm.capture.AvoidAliasConflict.apply$$anonfun$1(AvoidAliasConflict.scala:145)
[error] | at io.getquill.util.Interpolator$Traceable.andReturnIf(Interpolator.scala:193)
[error] | at io.getquill.norm.capture.AvoidAliasConflict.apply(AvoidAliasConflict.scala:205)
[error] | at io.getquill.norm.capture.AvoidAliasConflictApply.apply(AvoidAliasConflict.scala:287)
[error] | at io.getquill.norm.Normalize.apply$$anonfun$1(Normalize.scala:37)
[error] | at io.getquill.util.Interpolator$Traceable.andReturn(Interpolator.scala:172)
[error] | at io.getquill.norm.Normalize.apply(Normalize.scala:37)
[error] | at io.getquill.ast.StatelessTransformer.apply(StatelessTransformer.scala:9)
[error] | at io.getquill.ast.StatelessTransformer.apply$(StatelessTransformer.scala:3)
[error] | at io.getquill.norm.Normalize.apply(Normalize.scala:30)
[error] | at io.getquill.context.sql.norm.SqlNormalize.$init$$$anonfun$5(SqlNormalize.scala:40)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at scala.Function1.$anonfun$andThen$1(Function1.scala:85)
[error] | at io.getquill.context.sql.norm.SqlNormalize.apply(SqlNormalize.scala:66)
[error] | at io.getquill.context.sql.norm.SqlNormalize$.apply(SqlNormalize.scala:13)
[error] | at io.getquill.context.sql.idiom.SqlIdiom.normalizeAst(SqlIdiom.scala:40)
[error] | at io.getquill.context.sql.idiom.SqlIdiom.normalizeAst$(SqlIdiom.scala:24)
[error] | at io.getquill.PostgresDialect$.normalizeAst(PostgresDialect.scala:196)
[error] | at io.getquill.context.sql.idiom.SqlIdiom.doTranslate(SqlIdiom.scala:60)
[error] | at io.getquill.context.sql.idiom.SqlIdiom.translate(SqlIdiom.scala:98)
[error] | at io.getquill.context.sql.idiom.SqlIdiom.translate$(SqlIdiom.scala:24)
[error] | at io.getquill.PostgresDialect$.translate(PostgresDialect.scala:196)
[error] | at io.getquill.context.StaticTranslationMacro$.processAst(StaticTranslationMacro.scala:75)
[error] | at io.getquill.context.StaticTranslationMacro$.$anonfun$11$$anonfun$3(StaticTranslationMacro.scala:260)
[error] | at scala.Option$WithFilter.flatMap(Option.scala:283)
[error] | at io.getquill.context.StaticTranslationMacro$.$anonfun$11(StaticTranslationMacro.scala:276)
[error] | at scala.Option$WithFilter.flatMap(Option.scala:283)
[error] | at io.getquill.context.StaticTranslationMacro$.apply(StaticTranslationMacro.scala:276)
[error] | at io.getquill.context.QueryExecution$RunQuery.applyQuery$$anonfun$2(QueryExecution.scala:235)
[error] | at scala.util.Try$.apply(Try.scala:210)
[error] | at io.getquill.context.QueryExecution$RunQuery.applyQuery(QueryExecution.scala:235)
[error] | at io.getquill.context.QueryExecution$RunQuery.apply(QueryExecution.scala:198)
[error] | at io.getquill.context.QueryExecution$.applyImpl(QueryExecution.scala:395)

At first I thought it's because PostCommentCounts cannot be translated to SQL query, but as I read the quill's documentation, I saw that it's completely fine. There's one thing in the stacktrace that caught my attention:
x$1 => Post(id: x$1._1.id, title: x$1._1.title, message: x$1._1.message, locationId: x$1._1.locationId, createdAt: x$1._1.createdAt)
This is the map function in groupByMap. These fields: title: x$1._1.title, message: x$1._1.message, locationId: x$1._1.locationId, createdAt: x$1._1.createdAt (with x$1 referring to (p, c)) are neither in group by or aggregated. Is it because you wanted to summarize your question or is this the actual query you're running?

How to run the legacy Mongo Casbah drivers alongside the modern Mongo scala drivers

The old official Scala Driver, "casbah", is end-of-life. There is a new official mongo driver here but the two drivers can't coexist in the same project due to the way sbt resolves conflicting library versions. They both seem to import separate major versions of an underlying mongo-java-driver.
I have hundreds of hand written Mongo queries that need tweaking for the new driver so having two drivers in use concurrently whilst I migrate the app query by query would be the safest and most agile approach for me, that I'd like to pursue.
I have a sample project available here. There are two connect functions. A new one and an old one. If you comment out the new/old pieces you will see each independently work, connect to a database and list the names of the collections.
When you include both drivers together you get this error:
[error] (run-main-0) java.lang.NoSuchMethodError: 'java.util.List
com.mongodb.ConnectionString.getCredentialList()'
[error] java.lang.NoSuchMethodError: 'java.util.List com.mongodb.ConnectionString.getCredentialList()'
[error] at com.mongodb.MongoClientURI.getCredentials(MongoClientURI.java:243)
[error] at com.mongodb.Mongo.createCluster(Mongo.java:637)
[error] at com.mongodb.Mongo.<init>(Mongo.java:289)
[error] at com.mongodb.MongoClient.<init>(MongoClient.java:268)
[error] at com.mongodb.casbah.MongoClient$.apply(MongoClient.scala:201)
[error] at example.LegacyMongoScala$.connect(Hello.scala:59)
[error] at example.Hello$.delayedEndpoint$example$Hello$1(Hello.scala:14)
[error] at example.Hello$delayedInit$body.apply(Hello.scala:7)
[error] at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error] at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error] at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error] at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error] at scala.collection.immutable.List.foreach(List.scala:431)
[error] at scala.App.main(App.scala:80)
[error] at scala.App.main$(App.scala:78)
[error] at example.Hello$.main(Hello.scala:7)
[error] at example.Hello.main(Hello.scala)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.base/java.lang.reflect.Method.invoke(Method.java:566)
I suspect the problem here is how the latest mongo-java-driver pushes out the former?
HOST [15:18: (main) mongo-migrate-example$] sbt dependencyTree
[info] welcome to sbt 1.4.4 (Debian Java 11.0.9.1)
[info] loading settings for project global-plugins from idea.sbt,plugins.sbt ...
[info] loading global plugins from /home/phill/.sbt/1.0/plugins
[info] loading project definition from /home/phill/tmp/mongo-migrate-example/project
[success] Generated .bloop/mongo-migrate-example-build.json
[info] compiling 1 Scala source to /home/phill/tmp/mongo-migrate-example/project/target/scala-2.12/sbt-1.0/classes ...
[success] Total time: 4 s, completed 5 Jan 2021, 15:19:43
[info] loading settings for project root from build.sbt ...
[info] set current project to mongo-migrate-example (in build file:/home/phill/tmp/mongo-migrate-example/)
[info] Updating
https://repo1.maven.org/maven2/joda-time/joda-time/2.9.4/joda-time-2.9.4.pom
100.0% [##########] 29.4 KiB (101.2 KiB / s)
https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.pom
100.0% [##########] 10.4 KiB (35.7 KiB / s)
[info] Resolved dependencies
[info] Fetching artifacts of
https://repo1.maven.org/maven2/org/joda/joda-convert/1.2/joda-convert-1.2.jar
100.0% [##########] 37.6 KiB (536.6 KiB / s)
https://repo1.maven.org/maven2/joda-time/joda-time/2.9.4/joda-time-2.9.4.jar
100.0% [##########] 614.8 KiB (2.2 MiB / s)
[info] Fetched artifacts of
[info] com.example:mongo-migrate-example_2.12:0.1.0-SNAPSHOT [S]
[info] +-com.github.salat:salat-core_2.12:1.11.2 [S]
[info] | +-com.github.salat:salat-util_2.12:1.11.2 [S]
[info] | | +-org.json4s:json4s-native_2.12:3.4.2 [S]
[info] | | | +-org.json4s:json4s-core_2.12:3.4.2 [S]
[info] | | | +-com.thoughtworks.paranamer:paranamer:2.8
[info] | | | +-org.json4s:json4s-ast_2.12:3.4.2 [S]
[info] | | | +-org.json4s:json4s-scalap_2.12:3.4.2 [S]
[info] | | | +-org.scala-lang.modules:scala-xml_2.12:1.0.5 (evicted by: 1.0.6)
[info] | | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | | |
[info] | | +-org.scala-lang:scalap:2.12.12
[info] | | | +-org.scala-lang:scala-compiler:2.12.12 [S]
[info] | | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.7.21
[info] | |
[info] | +-org.json4s:json4s-native_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-core_2.12:3.4.2 [S]
[info] | | +-com.thoughtworks.paranamer:paranamer:2.8
[info] | | +-org.json4s:json4s-ast_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-scalap_2.12:3.4.2 [S]
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.5 (evicted by: 1.0.6)
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | |
[info] | +-org.mongodb:casbah-core_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | | +-joda-time:joda-time:2.9.4
[info] | | | +-org.joda:joda-convert:1.2
[info] | | |
[info] | | +-org.mongodb:mongo-java-driver:3.2.2
[info] | | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | | +-org.slf4j:slf4j-api:1.7.21
[info] | |
[info] | +-org.mongodb:casbah-query_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | +-joda-time:joda-time:2.9.4
[info] | | +-org.joda:joda-convert:1.2
[info] | |
[info] | +-org.mongodb:mongo-java-driver:3.2.2
[info] | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-com.github.salat:salat-util_2.12:1.11.2 [S]
[info] | +-org.json4s:json4s-native_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-core_2.12:3.4.2 [S]
[info] | | +-com.thoughtworks.paranamer:paranamer:2.8
[info] | | +-org.json4s:json4s-ast_2.12:3.4.2 [S]
[info] | | +-org.json4s:json4s-scalap_2.12:3.4.2 [S]
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.5 (evicted by: 1.0.6)
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | |
[info] | +-org.scala-lang:scalap:2.12.12
[info] | | +-org.scala-lang:scala-compiler:2.12.12 [S]
[info] | | +-org.scala-lang.modules:scala-xml_2.12:1.0.6 [S]
[info] | | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] | |
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-io.netty:netty-all:4.1.51.Final
[info] +-org.mongodb.scala:mongo-scala-driver_2.12:4.1.0 [S]
[info] | +-org.mongodb.scala:mongo-scala-bson_2.12:4.1.0 [S]
[info] | | +-org.mongodb:bson:4.1.0
[info] | | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] | |
[info] | +-org.mongodb:mongodb-driver-reactivestreams:4.1.0
[info] | | +-org.mongodb:bson:4.1.0
[info] | | +-org.mongodb:mongodb-driver-core:4.1.0
[info] | | | +-org.mongodb:bson:4.1.0
[info] | | |
[info] | | +-org.reactivestreams:reactive-streams:1.0.2
[info] | |
[info] | +-org.scala-lang:scala-reflect:2.12.12 [S]
[info] |
[info] +-org.mongodb:casbah-core_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | | +-joda-time:joda-time:2.9.4
[info] | | | +-org.joda:joda-convert:1.2
[info] | | |
[info] | | +-org.mongodb:mongo-java-driver:3.2.2
[info] | | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | | +-org.slf4j:slf4j-api:1.7.21
[info] | |
[info] | +-org.mongodb:casbah-query_2.12:3.1.1 [S]
[info] | +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | +-joda-time:joda-time:2.9.4
[info] | | +-org.joda:joda-convert:1.2
[info] | |
[info] | +-org.mongodb:mongo-java-driver:3.2.2
[info] | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-org.mongodb:casbah-gridfs_2.12:3.1.1 [S]
[info] +-org.mongodb:casbah-core_2.12:3.1.1 [S]
[info] +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] | +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | | +-joda-time:joda-time:2.9.4
[info] | | +-org.joda:joda-convert:1.2
[info] | |
[info] | +-org.mongodb:mongo-java-driver:3.2.2
[info] | +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] | +-org.slf4j:slf4j-api:1.7.21
[info] |
[info] +-org.mongodb:casbah-query_2.12:3.1.1 [S]
[info] +-org.mongodb:casbah-commons_2.12:3.1.1 [S]
[info] +-com.github.nscala-time:nscala-time_2.12:2.14.0 [S]
[info] | +-joda-time:joda-time:2.9.4
[info] | +-org.joda:joda-convert:1.2
[info] |
[info] +-org.mongodb:mongo-java-driver:3.2.2
[info] +-org.slf4j:slf4j-api:1.6.0 (evicted by: 1.7.21)
[info] +-org.slf4j:slf4j-api:1.7.21
[info]
[success] Total time: 1 s, completed 5 Jan 2021, 15:19:45
Question 1: Can someone suggest an approach which might work?
I have tried creating a library (using sbt new, then adding a single dependency + sbt-assembly plugin) which only expose a connect function and then used sbt-assembly plugin to shade the dependencies. Using this technique I can rename com.mongodb to shadedlegacy.mongodb and see this inside the assembled jar when I open it with a zip tool. I can used the renamed symbols in my applications but internally there are still errors thrown about subsequent classes and I'm not sure if this is just a rabbit hole with no end, all caused by the fact mongo-java-driver is shared by both and always evicted:
[error] (run-main-0) java.lang.IllegalAccessError: Error running public static void app.Boot.main(java.lang.String[]).
[error] java.lang.IllegalAccessError: failed to access class org.bson.BSON from class com.mongodb.casbah.commons.conversions.scala.OptionSerializer (org.bson.BSON and com.mongodb.casbah.commons.conversions.scala.OptionSerializer are in unnamed module of loader sbt.internal.LayeredClassLoader #2d8d2575)
[error] If using a layered classloader, this can occur if jvm package private classes are accessed across layers. This can be fixed by changing to the Flat or ScalaInstance class loader layering strategies.
[error] java.lang.IllegalAccessError: Error running public static void app.Boot.main(java.lang.String[]).
[error] java.lang.IllegalAccessError: failed to access class org.bson.BSON from class com.mongodb.casbah.commons.conversions.scala.OptionSerializer (org.bson.BSON and com.mongodb.casbah.commons.conversions.scala.OptionSerializer are in unnamed module of loader sbt.internal.LayeredClassLoader #2d8d2575)
[error] If using a layered classloader, this can occur if jvm package private classes are accessed across layers. This can be fixed by changing to the Flat or ScalaInstance class loader layering strategies.
[error] stack trace is suppressed; run last Compile / bgRun for the full output
Question 3: Is this approach above of shading both libraries one that could work?
Question 4: Would this sbt-shading library offer any alternative solutions that couldn't be reach via sbt-assembly?
Question 5: Is there any approaches using custom classloaders that could potentially lead me to success?
Question 6: If you've moved between drivers was it ever easier at some point? Am I over-estimating the complexity? I can't find any upgrade guides.
Thank you for your time.

sbt deduplicate: different file contents found in the following:

I'm aware this question has been asked (and answered) numerous times, but the existing answers do not address my case.
I am trying to build the DataStax spark-cassandra-connector (tag v1.5.0-M3) as per this tutorial
./sbt/sbt assembly
but I get the following errors
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/me/.ivy2/cache/io.netty/netty/bundles/netty-3.8.0.Final.jar:META-INF/jboss-beans.xml
/Users/me/.ivy2/cache/org.jboss.netty/netty/bundles/netty-3.2.2.Final.jar:META-INF/jboss-beans.xml
at sbtassembly.Assembly$.sbtassembly$Assembly$$applyStrategy$1(Assembly.scala:106)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:123)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:120)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Assembly$.applyStrategies(Assembly.scala:125)
at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
at sbtassembly.Assembly$.apply(Assembly.scala:83)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:226)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:223)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/me/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
/Users/me/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
at sbtassembly.Assembly$.sbtassembly$Assembly$$applyStrategy$1(Assembly.scala:106)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:123)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:120)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Assembly$.applyStrategies(Assembly.scala:125)
at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
at sbtassembly.Assembly$.apply(Assembly.scala:83)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:226)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:223)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[error] (spark-cassandra-connector-java/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/me/.ivy2/cache/io.netty/netty/bundles/netty-3.8.0.Final.jar:META-INF/jboss-beans.xml
[error] /Users/me/.ivy2/cache/org.jboss.netty/netty/bundles/netty-3.2.2.Final.jar:META-INF/jboss-beans.xml
[error] (spark-cassandra-connector/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/me/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
[error] /Users/me/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
[error] Total time: 44 s, completed 10-Dec-2015 16:22:01
Any of the suggested answers I have read (such as this one) refer to changes required in the build.sbt file, but in my case I have no such file. In the project directory I have
.
├── BuildUtil.scala
├── CassandraSparkBuild.scala
├── Publish.scala
├── Settings.scala
├── SparkInstaller.scala
├── Versions.scala
├── build.properties
├── plugins.sbt
├── project
└── target
2 directories, 8 files
So I don't know which file(s) to edit and what edits are necessary. I am on OSX and have the following versions installed
Spark 1.5.2
Java 1.8.0_45
Scala 2.11.7
sbt 0.13.9
UPDATE
If I follow the instruction here and build for Scala 2.11 I get the following prompt
$ sbt -Dscala-2.11=true
[info] Loading project definition from /usr/local/share/spark-cassandra-connector/project
[info] Updating {file:/usr/local/share/spark-cassandra-connector/project/}spark-cassandra-connector-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
Using releases: https://oss.sonatype.org/service/local/staging/deploy/maven2 for releases
Using snapshots: https://oss.sonatype.org/content/repositories/snapshots for snapshots
Scala: 2.11.7
Scala Binary: 2.11
Java: target=1.7 user=1.8.0_45
[info] Set current project to root (in build file:/usr/local/share/spark-cassandra-connector/)
SBT   39dcadbe40e833b610357ce84811201f03d8f350  root 
Dependency Graph
Below is the result of using sbt-dependency-graph as per #AlexeyRomanovs suggestion
[info] default:project_2.10:0.1-SNAPSHOT [S]
[info] +-com.scalapenos:sbt-prompt:0.2.1
[info] | +-com.typesafe.sbt:sbt-git:0.6.4
[info] | +-org.eclipse.jgit:org.eclipse.jgit.pgm:3.3.2.201404171909-r
[info] | +-args4j:args4j:2.0.12
[info] | +-org.apache.commons:commons-compress:1.6
[info] | | +-org.tukaani:xz:1.4
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.archive:3.3.2.201404171909-r
[info] | | +-org.apache.commons:commons-compress:1.6
[info] | | | +-org.tukaani:xz:1.4
[info] | | |
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | | +-com.jcraft:jsch:0.1.50
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.osgi:org.osgi.core:4.3.1
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.ui:3.3.2.201404171909-r
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | +-com.jcraft:jsch:0.1.50
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | +-com.jcraft:jsch:0.1.50
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe.sbt:sbt-git:0.6.4
[info] | +-org.eclipse.jgit:org.eclipse.jgit.pgm:3.3.2.201404171909-r
[info] | +-args4j:args4j:2.0.12
[info] | +-org.apache.commons:commons-compress:1.6
[info] | | +-org.tukaani:xz:1.4
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.archive:3.3.2.201404171909-r
[info] | | +-org.apache.commons:commons-compress:1.6
[info] | | | +-org.tukaani:xz:1.4
[info] | | |
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | | +-com.jcraft:jsch:0.1.50
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.osgi:org.osgi.core:4.3.1
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.ui:3.3.2.201404171909-r
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | +-com.jcraft:jsch:0.1.50
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | +-com.jcraft:jsch:0.1.50
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe.sbt:sbt-scalariform:1.3.0
[info] | +-org.scalariform:scalariform_2.10:0.1.4 [S]
[info] |
[info] +-net.virtual-void:sbt-dependency-graph:0.8.0
[info] | +-com.github.mdr:ascii-graphs_2.10:0.0.3 [S]
[info] |
[info] +-org.scalastyle:scalastyle-sbt-plugin:0.6.0
[info] | +-org.scalastyle:scalastyle_2.10:0.6.0 [S]
[info] | +-com.danieltrinh:scalariform_2.10:0.1.5 [S]
[info] | +-com.typesafe:config:1.2.0
[info] |
[info] +-org.scoverage:sbt-scoverage:1.0.4
[info] | +-org.scoverage:scalac-scoverage-plugin_2.10:1.0.4 [S]
[info] |
[info] +-org.spark-packages:sbt-spark-package:0.2.3
[info] | +-com.eed3si9n:sbt-assembly:0.12.0
[info] | +-org.apache.directory.studio:org.apache.commons.codec:1.8
[info] | | +-commons-codec:commons-codec:1.8
[info] | |
[info] | +-org.scalaj:scalaj-http_2.10:1.1.4 [S]
[info] |
[info] +-org.typelevel:sbt-typelevel:0.3.1
[info] +-com.eed3si9n:sbt-buildinfo:0.3.2
[info] +-com.github.gseitz:sbt-release:0.8.4
[info] +-com.typesafe.sbt:sbt-pgp:0.8.2
[info] | +-com.jsuereth:gpg-library_2.10:0.8.2 [S]
[info] | | +-net.databinder:dispatch-http_2.10:0.8.10 [S]
[info] | | | +-net.databinder:dispatch-core_2.10:0.8.10 [S]
[info] | | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | | +-commons-codec:commons-codec:1.8
[info] | | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | | |
[info] | | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-net.databinder:dispatch-futures_2.10:0.8.10 [S]
[info] | | | | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.bouncycastle:bcpg-jdk15on:1.49
[info] | | +-org.bouncycastle:bcprov-jdk15on:1.49
[info] | |
[info] | +-net.databinder:dispatch-http_2.10:0.8.10 [S]
[info] | +-net.databinder:dispatch-core_2.10:0.8.10 [S]
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-net.databinder:dispatch-futures_2.10:0.8.10 [S]
[info] | | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe:sbt-mima-plugin:0.1.6
[info] | +-com.typesafe:mima-reporter_2.10:0.1.6 [S]
[info] | +-com.typesafe:config:1.0.0 (evicted by: 1.2.0)
[info] | +-com.typesafe:config:1.2.0
[info] | +-com.typesafe:mima-core_2.10:0.1.6 [S]
[info] | | +-org.scala-lang:scala-compiler:2.10.2 [S]
[info] | | +-org.scala-lang:scala-reflect:2.10.2 [S]
[info] | |
[info] | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | +-org.scala-lang:scala-swing:2.10.2 [S]
[info] |
[info] +-net.virtual-void:sbt-dependency-graph:0.7.4 (evicted by: 0.8.0)
[info] +-net.virtual-void:sbt-dependency-graph:0.8.0
[info] | +-com.github.mdr:ascii-graphs_2.10:0.0.3 [S]
[info] |
[info] +-org.xerial.sbt:sbt-sonatype:0.2.1
[info] +-org.apache.httpcomponents:httpclient:4.2.6
[info] +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] +-commons-codec:commons-codec:1.8
[info] +-commons-logging:commons-logging:1.1.1
[info] +-org.apache.httpcomponents:httpcore:4.2.5
[info]

If you look at the SBT tutorial, you'll find that project/*.scala files are equivalent to *.sbt files in the root directory. So that's where you need to make the changes (specifically, in CassandraSparkBuild.scala).
The problem is that there are two versions of Netty in transitive dependencies and SBT doesn't notice this because the group IDs are different: org.jboss.netty for 3.2.2 and io.netty for 3.8. Find where they come from (using e.g. https://github.com/jrudolph/sbt-dependency-graph) and exclude the older version (and hope nothing breaks).

Scalding on EMR: Hadoop job fails with NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;

Basically I need to run a scalding job on EMR. The same job runs perfectly fine on local hadoop on my macbook, but fails on Hadoop on EMR.
I am trying hard to get help for this issue in the cascading-user and scala-user groups as well, and haven't been able to. So far I haven't made much progress after trying various changes in the past couple days.
Here is the error before I delve into the details:
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at com.aggregation.job.DataAggregation$.(DataAggregation.scala:30)
at com.aggregation.job.DataAggregation$.(DataAggregation.scala)
at com.aggregation.job.DataAggregation.main(DataAggregation.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
They said it might be a scala conflict with the binaries, but I couldn't see anything obvious. It would be great if someone could help with figuring it out.
Environment:
Amazon EMR AMI: 3.8.0 (which includes Scala 2.11.1, Hadoop 2.4.0, Java 1.7.0_76 - AMI details here)
Application environment: Scalding 0.15.0, Scala 2.11.1, Java 1.7.0_80, Hadoop 2.4.0
I have tried various changes to configuration and even manually installing a higher scala version in EMR, but so far the error is not going away.
Please help! Thank you.
L
Setup
build.sbt:
scalaVersion := "2.11.1"
ivyScala := ivyScala.value map {
_.copy(overrideScalaVersion = true)
}
dependencies.sbt:
import AssemblyKeys._
val hadoopVersion = "2.4.0"
val scaldingVersion = "0.15.0"
libraryDependencies ++= Seq(
"com.twitter" %% "scalding-core" % scaldingVersion,
"com.twitter" %% "scalding-json" % scaldingVersion,
"com.twitter" %% "scalding-jdbc" % scaldingVersion,
"com.github.nscala-time" %% "nscala-time" % "2.0.0",
"org.apache.hadoop" % "hadoop-common" % hadoopVersion % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion % "provided"
)
excludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
val excludes = Set(
"jsp-api-2.1-6.1.14.jar",
"jsp-2.1-6.1.14.jar",
"jasper-compiler-5.5.12.jar",
"minlog-1.2.jar", // Otherwise causes conflicts with Kyro (which bundles it)
"janino-2.5.16.jar", // Janino includes a broken signature, and is not needed anyway
"commons-beanutils-core-1.8.0.jar", // Clash with each other and with commons-collections
"commons-beanutils-1.7.0.jar", // "
"hadoop-core-1.2.1.jar", // Provided by Amazon EMR. Delete this line if you're not on EMR
"hadoop-tools-1.2.1.jar" // "
)
cp filter { jar => excludes(jar.data.getName) }
}
resolvers ++= Seq(
"Conjars repo" at "http://conjars.org/repo"
)
assembly.sbt
import AssemblyKeys._
assemblySettings
mergeStrategy in assembly := Merge.mergeStrategy
project/build.properties:
sbt.version=0.13.1
project/assembly.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.0")
Finally this is the dependency tree using the sbt-dependency-graph plugin to see if I've got the right versions. Sorry, it's quite long, if I should be displaying this info any other way, please recommend suggestions.
sbt dependency tree:
[info] com.abc.aggregator_2.11:0.1-20150628T184441 [S]
[info] +-com.github.nscala-time:nscala-time_2.11:2.0.0 [S]
[info] | +-joda-time:joda-time:2.7
[info] | +-org.joda:joda-convert:1.2
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-jdbc_2.11:0.15.0 [S]
[info] | +-cascading:cascading-jdbc-core:2.6.0
[info] | | +-com.google.guava:guava:15.0
[info] | |
[info] | +-cascading:cascading-jdbc-mysql:2.6.0
[info] | | +-cascading:cascading-jdbc-core:2.6.0
[info] | | | +-com.google.guava:guava:15.0
[info] | | |
[info] | | +-com.google.guava:guava:15.0
[info] | | +-mysql:mysql-connector-java:5.1.25
[info] | |
[info] | +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-json_2.11:0.15.0 [S]
[info] +-com.fasterxml.jackson.module:jackson-module-scala_2.11:2.4.2 [S]
[info] | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-databind:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.0 (evicted by: 2.4.2)
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | |
[info] | +-com.google.code.findbugs:jsr305:2.0.1
[info] | +-com.google.guava:guava:15.0
[info] | +-com.thoughtworks.paranamer:paranamer:2.6
[info] | +-org.scala-lang:scala-reflect:2.11.2 [S]
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-org.json4s:json4s-native_2.11:3.2.11 [S]
[info] +-org.json4s:json4s-core_2.11:3.2.11 [S]
[info] +-com.thoughtworks.paranamer:paranamer:2.6
[info] +-org.json4s:json4s-ast_2.11:3.2.11 [S]
[info] +-org.scala-lang:scalap:2.11.0
[info] +-org.scala-lang:scala-compiler:2.11.1 [S]
[info] +-org.scala-lang.modules:scala-parser-combinators_2.11:1.0.1 [S]
[info] +-org.scala-lang.modules:scala-xml_2.11:1.0.2 [S]
[info] +-org.scala-lang:scala-reflect:2.11.1 (evicted by: 2.11.2)
[info] +-org.scala-lang:scala-reflect:2.11.2 [S]
[info]
[success] Total time: 12 s, completed Jun 28, 2015 11:44:54 AM
Further info as requested:
I build the fat jar using 'sbt assembly', and currently I'm using the AWS console with a "Custom JAR" step to test this out before automating the process.
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt
UPDATE:
I was able to get past the above error by providing the HADOOP_CLASSPATH pointing to the scala 2.11.1 jars, while excluding the same from the sbt assembly step. This was passed in using hadoop-user-env.sh and seemed to work for the master node. However once it got to the mapper step it once again failed with another Scala error. Now I am stuck on this step.
Assuming this is because the mappers and reducers aren't seeing the HADOOP_CLASSPATH update, I tried including the -libjars argument pointing to the scala jar files on hadoop master itself. But this (below) doesn't seem to be working.
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation -libjars /usr/share/scala/lib/scala-library.jar,/usr/share/scala/lib/scala-reflect.jar --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt

Fixed. So it does happen that there were multiple scala jars in the EMR instances, and they weren't coming from my application jar.
The 2.10 jar was hiding in /usr/share/aws/emr/emrfs/lib apart from the installed location for the 2.11 binaries under /usr/share/scala. So I got rid of the 2.10 jar in all instances of the cluster, and my job completed successfully. Now I will create a bootstrap action for this.
$ sudo find / -name "scala-library-2.10.*.jar" -exec rm -rf {} \;
FYI, these are the paths it was present under:
[ec2-user#ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.11.*.jar"
/home/hadoop/.versions/hbase-0.94.18/lib/scala-library-2.11.0.jar
/usr/share/doc/scala/api/jars/scala-library-2.11.1-javadoc.jar
[ec2-user#ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.10.*.jar"
/usr/share/aws/emr/emrfs/lib/scala-library-2.10.5.jar

After struggling with a similar issue..... which is clearly a version mismatch between jar dependencies. I fixed it by first logging into the EMR master, then opening spark-shell reveals the exact scala version that is being used. Now take that to configure your own dependencies in the pom file and it should work.

How do I get a complete 'recursive' tree/graph of task dependencies in SBT?

When I for example inspect tree compile on a simple sbt project:
sbt> inspect tree compile
[info] compile:compile = Task[sbt.inc.Analysis]
[info] +-compile:compile::compileInputs = Task[sbt.Compiler$Inputs]
[info] | +-*/*:compileOrder = Mixed
[info] | +-compile:dependencyClasspath = Task[scala.collection.Seq[sbt.Attributed[java.io.File]]]
[info] | +-*/*:maxErrors = 100
[info] | +-compile:sources = Task[scala.collection.Seq[java.io.File]]
[info] | +-compile:scalacOptions = Task[scala.collection.Seq[java.lang.String]]
[info] | +-*/*:javacOptions = Task[scala.collection.Seq[java.lang.String]]
[info] | +-*:compilers = Task[sbt.Compiler$Compilers]
[info] | +-compile:incCompileSetup = Task[sbt.Compiler$IncSetup]
[info] | +-compile:compile::streams = Task[sbt.std.TaskStreams[sbt.Init$ScopedKey[_ <: Any]]]
[info] | | +-*/*:streamsManager = Task[sbt.std.Streams[sbt.Init$ScopedKey[_ <: Any]]]
[info] | |
[info] | +-*/*:sourcePositionMappers = Task[scala.collection.Seq[scala.Function1[xsbti.Position, scala.Option[xsbti.Position..
[info] | +-compile:classDirectory = target/scala-2.10/classes
[info] |
[info] +-compile:compile::streams = Task[sbt.std.TaskStreams[sbt.Init$ScopedKey[_ <: Any]]]
[info] +-*/*:streamsManager = Task[sbt.std.Streams[sbt.Init$ScopedKey[_ <: Any]]]
The 'depth' of this tree is limited: for example, it mentions compile:sources, which itself has a number of dependencies:
sbt> inspect tree compile:sources
[info] compile:sources = Task[scala.collection.Seq[java.io.File]]
[info] +-compile:unmanagedSources = Task[scala.collection.Seq[ja..
[info] | +-*/*:excludeFilter = sbt.SimpleFileFilter#745ef9f2
[info] | +-*:baseDirectory = /tmp/xxxx
[info] | +-*/*:unmanagedSources::includeFilter = sbt.SimpleFilte..
[info] | +-*/*:sourcesInBase = true
[info] | +-compile:unmanagedSourceDirectories = List(/tmp/xxxx/s..
[info] | +-compile:javaSource = src/main/java
[info] | | +-compile:sourceDirectory = src/main
[info] | | +-*:sourceDirectory = src
[info] | | | +-*:baseDirectory = /tmp/xxxx
[info] | | | +-*:thisProject = Project(id: xxxx, base: /tm..
[info] | | |
[info] | | +-compile:configuration = compile
[info] | |
[info] | +-compile:scalaSource = src/main/scala
[info] | +-compile:sourceDirectory = src/main
[info] | +-*:sourceDirectory = src
[info] | | +-*:baseDirectory = /tmp/xxxx
[info] | | +-*:thisProject = Project(id: xxxx, base: /tm..
[info] | |
[info] | +-compile:configuration = compile
[info] |
[info] +-compile:managedSources = Task[scala.collection.Seq[java..
[info] +-compile:sourceGenerators = List()
I'd like to somehow be able to analyze the 'complete picture' - for example, I have a project for which the assembly task appears to run some tasks twice. Right now it's quite hard to figure out what's going on.