Transitive dependency errors in SBT multi-project - scala

I am building a SBT multi-project project, which has common module and logic module, and logic.dependsOn(common).
In common, SparkSQL 2.2.1 ("org.apache.spark" %% "spark-sql" % "2.2.1") is introduced. In logic, SparkSQL also is used, however I get compilation errors, saying "object spark is not a member of package org.apache".
Now, if I add SparkSQL dependency to logic as "org.apache.spark" %% "spark-sql" % "2.2.1", it works. However if I add "org.apache.spark" %% "spark-sql" % "2.2.1" % Provided", I get the same error.
I don't get why this happened, why not the dependency can't be transitive from common to logic
here is the root sbt files:
lazy val commonSettings = Seq(
organization := "...",
version := "0.1.0",
scalaVersion := "2.11.12",
resolvers ++= Seq(
clojars,
maven_local,
novus,
twitter,
spark_packages,
artima
),
test in assembly := {},
assemblyMergeStrategy in assembly := {...}
)
lazy val root = (project in file(".")).aggregate(common, logic)
lazy val common = (project in file("common")).settings(commonSettings:_*)
lazy val logic = (project in file("logic")).dependsOn(common).settings(commonSettings:_*)
here is the logic module sbt file:
libraryDependencies ++= Seq(
spark_sql.exclude("io.netty", "netty"),
embedded_elasticsearch % "test",
scalatest % "test"
)
dependencyOverrides ++= Seq(
"com.fasterxml.jackson.core" % "jackson-core" % "2.6.5",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5",
"com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.6.5",
"com.fasterxml.jackson.core" % "jackson-annotation" % "2.6.5",
"org.json4s" %% "json4s-jackson" % "3.2.11"
)
assemblyJarName in assembly := "***.jar"

Related

Scala build.sbt: Memcached Plugin for Play framework 2.8.6

I am trying to upgrade mumoshu's Memcached Plugin for Play framework, to Play 2.8.6, in build.sbt.
The upgrade instructions are:
For Play 2.6.x and newer: !!! Changed play.modules.cache.* config keys to play.cache.* !!!
val appDependencies = Seq(
play.PlayImport.cacheApi,
"com.github.mumoshu" %% "play2-memcached-play26" % "0.9.2"
)
val main = Project(appName).enablePlugins(play.PlayScala).settings(
version := appVersion,
libraryDependencies ++= appDependencies,
resolvers += "Spy Repository" at "http://files.couchbase.com/maven2" // required to resolve `spymemcached`, the plugin's dependency.
)
Before the upgrade, I had:
libraryDependencies ++= Seq(
jdbc,
cache,
ws,
"com.github.mumoshu" %% "play2-memcached-play24" % "0.7.0",
//...
}
After the (unsuccessful) upgrade:
the cache dependency is not resolved, so I removed it.
The updated build.sbt now looks like this:
name := "CH07"
version := "1.0"
lazy val `ch07` = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.13.4"
resolvers += "Spy Repository" at "http://files.couchbase.com/maven2"
val appDependencies = Seq(
play.sbt.PlayImport.cache,
"com.github.mumoshu" %% "play2-memcached-play28" % "0.11.0"
//"com.github.mumoshu" %% "play2-memcached-play26" % "0.9.2"
)
val main = (project in file(".")).enablePlugins(play.sbt.PlayScala).settings(
version := version.value,
libraryDependencies ++= appDependencies,
resolvers += "Spy Repository" at "http://files.couchbase.com/maven2" // required to resolve `spymemcached`, the plugin's dependency.
)
libraryDependencies ++= Seq(
jdbc,
//cache,
ws,
"org.hsqldb" % "hsqldb" % "2.5.0",
"org.jooq" % "jooq" % "3.14.4",
"org.jooq" % "jooq-codegen-maven" % "3.14.4",
"org.jooq" % "jooq-meta" % "3.14.4",
"joda-time" % "joda-time" % "2.7",
"com.github.ironfish" %% "akka-persistence-mongo-casbah" % "0.7.6",
"com.ning" % "async-http-client" % "1.9.29"
)
routesGenerator := InjectedRoutesGenerator
Errors I am getting:
* build.sbt:
error: value cache is not a member of object play.sbt.PlayImport
* build.sbt:
"com.github.mumoshu" %% "play2-memcached-play28" % "0.11.0"
Intellij says: unresolved artifact. Not resolved or indexed.
* in project/plugins.sbt:
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.8.6")
Intellij says: unresolved artifact. Not resolved or indexed.
Can you see any errors I made in the build.sbt?
In other words, I am looking for a working build.sbt (and the other required definitions) example for a project that uses Play 2.8.6 and the Memcached Plugin for Play framework 0.9.2. (or highest possible version of the latter).
UPDATE:
This is my current configuration which does compile:
build.sbt
name := "CH07"
version := "1.0"
lazy val `ch07` = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.13.4"
resolvers += "Spy Repository" at "http://files.couchbase.com/maven2"
libraryDependencies ++= Seq(
jdbc,
cacheApi,
ws,
"com.github.mumoshu" %% "play2-memcached-play28" % "0.11.0",
"org.hsqldb" % "hsqldb" % "2.5.0",
"org.jooq" % "jooq" % "3.14.4",
"org.jooq" % "jooq-codegen-maven" % "3.14.4",
"org.jooq" % "jooq-meta" % "3.14.4",
"joda-time" % "joda-time" % "2.7",
"com.ning" % "async-http-client" % "1.9.29",
"com.github.scullxbones" %% "akka-persistence-mongo-common" % "3.0.5",
"com.typesafe.akka" %% "akka-persistence" % "2.6.10",
"com.typesafe.akka" %% "akka-persistence-query" % "2.6.10",
"com.typesafe.akka" %% "akka-persistence-typed" % "2.6.10",
"com.typesafe.play" %% "play-guice" % "2.8.7"
)
routesGenerator := InjectedRoutesGenerator
project/build.properties
sbt.version=1.3.10
project/plugins.sbt
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.8.7")
How about this?
name := """pla-memcache-example"""
organization := "com.example"
version := "1.0-SNAPSHOT"
scalaVersion := "2.13.3"
val appDependencies = Seq(
guice,
play.PlayImport.cacheApi,
"com.github.mumoshu" %% "play2-memcached-play28" % "0.11.0",
"org.scalatestplus.play" %% "scalatestplus-play" % "5.0.0" % Test
)
lazy val root = (project in file(".")).enablePlugins(PlayScala).settings(
version := appVersion,
libraryDependencies ++= appDependencies,
resolvers += "Spy Repository" at "http://files.couchbase.com/maven2" // required to resolve `spymemcached`, the plugin's dependency.
)
plugins.sbt
sbt.version=1.3.13
plugins.sbt
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.8.7")
In this case, first find the giter8 template and use the "sbt new" command.
Next, go to the official page of the plugin and check the name of the library, and then use Maven search to find the version you want to use.
It's a good idea to read the entire readme of the plugin without skipping it.
You can also try to run sbt with commands instead of using IntelliJ.
If that doesn't solve the problem, you can try clearing the cache in the .ivy2 directory.
Reference Links:
https://github.com/mumoshu/play2-memcached
https://github.com/playframework/play-scala-seed.g8
https://search.maven.org/artifact/com.github.mumoshu/play2-memcached-play28_2.13/0.11.0/jar
Your issue is when trying to import play.sbt.PlayImport.cache. According to Cache APIs Migration by play:
New packages
Now `cache` has been split into a `cacheApi` component with just the API, and `ehcache` that contains the Ehcache implementation. If you are using the default Ehcache implementation, simply change `cache` to `ehcache` in your `build.sbt`:
libraryDependencies ++= Seq(
ehcache
)
If you are defining a custom cache API, or are writing a cache implementation module, you can just depend on the API:
libraryDependencies ++= Seq(
cacheApi
)
Removed APIs
The deprecated Java class play.cache.Cache was removed and you now must inject an play.cache.SyncCacheApi or play.cache.AsyncCacheApi.
Therefore a complete build.sbt will be:
name := "CH07"
version := "1.0"
lazy val `ch07` = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.13.4"
resolvers += "Spy Repository" at "https://files.couchbase.com/maven2"
val appDependencies = Seq(
play.sbt.PlayImport.ehcache,
play.sbt.PlayImport.cacheApi,
"com.github.mumoshu" %% "play2-memcached-play28" % "0.11.0"
//"com.github.mumoshu" %% "play2-memcached-play26" % "0.9.2"
)
val main = (project in file(".")).enablePlugins(play.sbt.PlayScala).settings(
version := version.value,
libraryDependencies ++= appDependencies,
resolvers += "Spy Repository" at "https://files.couchbase.com/maven2" // required to resolve `spymemcached`, the plugin's dependency.
)
libraryDependencies ++= Seq(
jdbc,
//cache,
ws,
"org.hsqldb" % "hsqldb" % "2.5.0",
"org.jooq" % "jooq" % "3.14.4",
"org.jooq" % "jooq-codegen-maven" % "3.14.4",
"org.jooq" % "jooq-meta" % "3.14.4",
"joda-time" % "joda-time" % "2.7",
"com.github.scullxbones" %% "akka-persistence-mongo-common" % "3.0.5",
"com.ning" % "async-http-client" % "1.9.29"
)
routesGenerator := InjectedRoutesGenerator
build.properties:
sbt.version=1.4.4
plugins.sbt:
logLevel := Level.Warn
resolvers += "Typesafe repository" at "https://repo.typesafe.com/typesafe/releases/"
addSbtPlugin("com.typesafe.play" % "sbt-plugin" % "2.8.7")

Classes in libraries not being picked up

I have various library dependencies in my build.sbt file
scalaVersion := "2.10.5",
libraryDependencies ++= Seq(
"org.scalatest" %% "scalatest" % "2.2.5" % "test",
"org.apache.spark" %% "spark-core" % "1.6.1" % "provided",
"commons-logging" % "commons-logging" % "1.2"
) map (_.excludeAll(
ExclusionRule(organization = "org.slf4j"),
ExclusionRule(organization = "log4j"),
ExclusionRule(organization = "javax.servlet")
)),
libraryDependencies ++= Seq(("org.slf4j" % "slf4j-log4j12" % "1.7.10")
.excludeAll(ExclusionRule(organization = "log4j"))),
libraryDependencies += "log4j" % "log4j" % "1.2.16" % "test",
I can see them in my external libraries in the project view.
However, the classes in the libraries are not being picked up in the Scala source files.
This is a sub module.
And the dependencies are declared in the parent, build.sbt. The code is in the child project.
I have tried:
Invalidating Caches / Restarts...
Deleting the .idea file and re-importing
Building project several times
Any help appreciated.
The dependencies needs to be explicitly added also in the subproject.
For example, in your build.sbt in the root folder of your project you can set:
val commonDependencies = Seq(
"commons-io" % "commons-io" % "2.5"
// other dependencies
)
val root = (project in file(".")).settings(
libraryDependencies ++= commonDependencies,
libraryDependencies ++= Seq(
// dependencies only for project root
)
)
val topWordsCounter = (project in file("top-words-counter"))
.settings(
libraryDependencies ++= commonDependencies,
libraryDependencies ++= Seq(
// dependencies only for the sub project project top-words-counter
)
)
Hope this helps

sbt, intellij jar in external library but not found when compile

encounter a very strange problem.
I'm trying to use reactivemongo. And after update build.sbt
I can see that the jar is in the external library. However the compilation failed because not found object reactivemongo.
my build.sbt is:
hereenter code herelazy val commonSettings = Seq(
hereorganization := "emmettng.com",
hereversion := "0.0.1",
scalaVersion := "2.11.8"
)
hereresolvers += "Typesafe Repo" at"http://repo.typesafe.com/typesafe/releases/"
resolvers += "Typesafe repository releases" at"http://repo.typesafe.com/typesafe/releases/"`
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % "2.11.8",
"org.scala-lang.modules" % "scala-xml_2.11" % "1.0.4",
``"org.scalatest" %% "scalatest" % "3.0.0" % "test",
"com.typesafe.play" %% "play-json" % "2.4.0-M3",
"org.scalaz" %% "scalaz-core" % "7.2.5",
"junit" % "junit" % "4.10" % "test",
"org.reactivemongo" %% "reactivemongo" % "0.11.14"
)

Can not create a multi project build

I am trying to create a Play application with a back end and a front end(using Scala.js). I separated the code into client and server folders.
For creating a Multi Project Build I looked at this http://www.scala-sbt.org/release/docs/Multi-Project.html and make my build.sbt like this:
name := """ScalaWeb"""
version := "1.0-SNAPSHOT"
lazy val commonSettings = Seq(
scalaVersion := "2.11.7",
libraryDependencies ++= Seq(
jdbc,
cache,
ws,
specs2 % Test
),
libraryDependencies ++= Seq(
"org.sorm-framework" % "sorm" % "0.3.19",
"org.scala-lang" % "scala-compiler" % scalaVersion.value force(),
"com.h2database" % "h2" % "1.3.148"
),
libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.1.0",
/*libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging" % "3.1.0",
libraryDependencies += "org.slf4j" % "slf4j-api" % "1.7.12"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.1.3"*/
resolvers += "scalaz-bintray" at "http://dl.bintray.com/scalaz/releases",
// Play provides two styles of routers, one expects its actions to be injected, the
// other, legacy style, accesses its actions statically.
routesGenerator := InjectedRoutesGenerator,
scalacOptions += "-Ylog-classpath",
// use this if you just want jawn's parser, and will implement your own facade
libraryDependencies += "org.spire-math" %% "jawn-parser" % "0.8.3",
// use this if you want jawn's parser and also jawn's ast
libraryDependencies += "org.spire-math" %% "jawn-ast" % "0.8.3",
//for reading Json libraries
libraryDependencies += "org.scalaj" %% "scalaj-http" % "2.2.1"
)
lazy val server = (project in file("server")).settings(
commonSettings: _*
).enablePlugins(PlayScala)
lazy val client = (project in file("client")).settings(
commonSettings: _*
).enablePlugins(PlayScala)
fork in run := true
So basically I placed all dependencies into commonSettings and the imported that into the client and server projects.
However, I am getting this error:
[error] (scalaweb/compile:backgroundRun) No main class detected.
Why is this occuring?
Have I set up my build.sbt incorrectly?
That's because your root project (scalaweb) doesn't have a main class.
You can either:
run server's run command by doing: server/run
aggregate client and server under root project
(http://www.scala-sbt.org/0.13/docs/Multi-Project.html#Aggregation)
This might do what you wish.

Why sbt compile doesn't copy unmanaged resources to classpath?

Could you tell me why sbt compile doesn't copy unmanaged resources to classpath? On the other hand sbt package does. As result I can't start debugging unless I invoke package call manually :(
I'm using SBT 0.12.1
Below is my build.sbt.
import AssemblyKeys._ // put this at the top of the file
net.virtualvoid.sbt.graph.Plugin.graphSettings
assemblySettings
organization := "com.zzz"
version := "0.1"
scalaVersion := "2.10.2"
scalacOptions := Seq("-unchecked", "-language:reflectiveCalls,postfixOps,implicitConversions", "-deprecation", "-feature", "-encoding", "utf8")
unmanagedResourceDirectories in Compile <++= baseDirectory { base =>
Seq( base / "src/main/webapp" )
}
jarName in assembly := "zzz.jar"
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case "rootdoc.txt" => MergeStrategy.discard
case x => old(x)
}
}
mainClass in assembly := Some("com.zzz.Boot")
name := "zzz"
// disable using the Scala version in output paths and artifacts
crossPaths := false
artifactName := { (sv: ScalaVersion, module: ModuleID, artifact: Artifact) =>
artifact.name + "." + artifact.extension
}
resolvers ++= Seq(
"spray repo" at "http://repo.spray.io/"
)
libraryDependencies ++= {
val sprayVersion = "1.2-M8"
val akkaVersion = "2.2.0-RC1"
Seq(
"io.spray" % "spray-servlet" % sprayVersion withSources(),
"io.spray" % "spray-can" % sprayVersion withSources(),
"io.spray" % "spray-routing" % sprayVersion withSources(),
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"org.eclipse.jetty" % "jetty-webapp" % "8.1.7.v20120910" % "container",
"org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016" % "container" artifacts Artifact("javax.servlet", "jar", "jar"),
"net.sourceforge.htmlcleaner" % "htmlcleaner" % "2.2",
"org.apache.httpcomponents" % "httpclient" % "4.2.3",
"org.apache.commons" % "commons-lang3" % "3.1",
"org.mongodb" %% "casbah" % "2.6.0",
"com.romix.scala" % "scala-kryo-serialization" % "0.2-SNAPSHOT",
"org.codehaus.jettison" % "jettison" % "1.3.3",
"com.osinka.subset" %% "subset" % "2.0.1",
"io.spray" %% "spray-json" % "1.2.5" intransitive()
)
}
seq(Revolver.settings: _*)
seq(webSettings: _*)
seq(Twirl.settings: _*)
The job of compile is to compile sources, so it won't typically do anything related to processing resources. However, the resources need to be in the class directory for run, package, test, console, and anything else that uses the fullClasspath. This is done by fullClasspath combining exportedProducts, which are the classes and resources generated by the current project, and dependencyClasspath, which are the classpath entries from dependencies.
The appropriate solution depends on what needs the resources. From the command line, run exported-products to do a compile as well as copy-resources. Programmatically, you will typically want to depend on fullClasspath or exportedProducts.
As a side note, you can typically find out what tasks do what using the inspect command.