How to maintain multi module dependencies with Apache spark 2.3 in one module and Apache spark 2.4 in another one. What would be the project layer and how build.sbt looks like.
You can specify different dependencies for each module.
Let's assume you have module A and module B, it would look something like this:
lazy val moduleA = (project in file("moduleA"))
.settings(
name := "Module-A",
libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "2.3.0")
)
lazy val moduleB = (project in file("moduleB"))
.settings(
name := "Module-B",
libraryDependencies ++= Seq("org.apache.spark" %% "spark-core" % "2.4.0")
)
The official documentation is pretty good, there are several examples
Related
I was trying the following:
lazy val root = (project in file("."))
.settings(
libraryDependencies ++= commonDeps,
scapegoatVersion := "1.3.5"
)
lazy val foo = (project in file("foo"))
.settings(
libraryDependencies ++= Dependencies.fooDeps,
scalaVersion := "2.13.8",
scapegoatVersion := "1.4.12"
)
Each project's version resolved to 1.0.
I also tried ThisBuild / scapegoatVersion := <some-version> but it set the version for all projects.
Unfortunately, it is not possible because of a wrong reference to scapegoatVersion setting.
libraryDependencies ++= Seq(crossVersion(GroupId %% ArtifactId % (scapegoatVersion in ThisBuild).value % Provided)))
The problem is that it adds libraryDependencies to projectSettings or scope ThisProject but uses a broader scope ThisBuild to refer to scapegoatVersion.
Because of this it ignores your scapegoatVersion specified per project.
Your approach would work if the plugin did the following
libraryDependencies ++= Seq(crossVersion(
GroupId %% ArtifactId % (scapegoatVersion in ThisProject).value % Provided)
))
You could submit a PR to the project.
I created 3 subprojects in play: A,B, and common
A and B needs to use common subproject.
the code looks like this for the build.sbt:
name := """play"""
organization := "com.play"
version := "1.0-SNAPSHOT"
lazy val common = (project in file("modules/common")).enablePlugins(PlayScala)
lazy val A = (project in file("modules/A")).enablePlugins(PlayScala)
.dependsOn(common).aggregate(common)
lazy val B= (project in file("modules/B")).enablePlugins(PlayScala)
.dependsOn(common).aggregate(common)
lazy val root = (project in file(".")).enablePlugins(PlayScala)
.dependsOn(A).aggregate(A)
.dependsOn(B).aggregate(B)
scalaVersion := "2.11.11"
libraryDependencies += filters
libraryDependencies += evolutions
libraryDependencies += "org.scalatestplus.play" %% "scalatestplus-play" % "2.0.0" % Test
The package system, I follow the documentation in the Play framework which is like "package.moduleName"
So, the class in model package in my common subproject has a package name called: "model.common"
Now in subproject A, i want to call the library in the common subproject
I call like this:
import model.common.className
I cannot find it
You can put all the classes inside a top level package.
For example for the common project you can put all the classes inside the com.play.common package.
After that you can use the class example(declare in the common project) from the project A using import com.play.common.A.
I want to build a 2.11 and 2.12 version of my project, so I have something like this in my Build.scala file:
val scalaVer12 = "2.12.1"
val scalaVer = "2.11.8"
lazy val basicSettings = Seq(
// lots of other settings
scalaVersion := scalaVer
)
The fly in the soup is I have a dependency on scala reflection, which is based on the scala version. Before I did this:
val scala_reflect = "org.scala-lang" % "scala-reflect" % Build.scalaVer
How can I modify this dependency line so that sbt will use either the 2.11 or 2.12 dependency based upon the version it's currently building?
lazy val bla = project in file("bla")
.settings(
libraryDependencies ++= Seq(
"org.scala-lang" % "scala-reflect" % scalaVersion.value
)
)
Never alias dependencies like that, it's clean to have an object to store version numbers, but not more, it's just a smell, especially since deps are often Scala version dependent and you can apply all sorts of rules to them.
I have installed Eclipse Luna. Then I installed Scala IDE via Help -> Install new software and adding software site with link from here. Then I installed sbt 0.13 and sbteclipse using this mini-guide and created eclipse project. Then I installed (kindda) scalatest by adding it to my build.sbt. Now it looks like this:
val scalaTest = "org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
lazy val commonSettings = Seq(
scalaVersion := "2.11.6"
)
lazy val root = (project in file(".")).
settings(commonSettings: _*).
settings(
libraryDependencies += scalaTest
)
Then I created a test from this example. The file called FirstSpec.scala is located in testProject/src/test/scala-2.11/testProject/. So here is a problem: eclipse seems to not see scalaTest. The second line with import org.scalatest._ is underlined red with error description object scalatest is not a member of package org. And, following this guide, I don't see the option Run As -> ScalaTest - Suite when choosing my test class.
At the same time everything goes good and fine when I start sbt session in my test project and type test command. The tests launches and passes.
So my questions are:
why eclipse doesn't see the scalatest if I put it in build.sbt's libraryDependencies? What's the point of libraryDependencies then?
Why sbt test runs the tests without a problem? If sbt sees scalatest, why eclipse can't?
Whew, this one resolved my issue. So, build.sbt example might go something like:
import com.typesafe.sbteclipse.plugin.EclipsePlugin._
EclipseKeys.withSource := true
val scalaTest = "org.scalatest" % "scalatest_2.11" % "2.2.4" % "test"
val jodaConvert = "org.joda" % "joda-convert" % "1.7"
val joda = "joda-time" % "joda-time" % "2.7"
lazy val commonSettings = Seq(
scalaVersion := "2.11.6"
)
lazy val root = (project in file(".")).
settings(commonSettings: _*).
settings(
libraryDependencies += scalaTest
).
settings(
libraryDependencies += jodaConvert
).
settings(
libraryDependencies += joda
)
Then do this:
rm -rf ~/.ivy2/cache/
sbt update-classifiers
sbt eclipse
I'm having issues with running a simple scala play app with a few dependencies. The below is happening when trying to run my app.
[error] /Users/roland/play-scala/app/domain/UserModule.scala:2:
object softwaremill is not a member of package com
[error] import com.softwaremill.macwire.MacwireMacros.wire
UserModule:
package domain
import com.softwaremill.macwire.MacwireMacros.wire
trait UserModule {
lazy val userRepository = wire[UserRepository]
lazy val userService = wire[UserService]
}
and my build.sbt is
name := """play-scala"""
version := "1.0-SNAPSHOT"
lazy val root = (project in file(".")).enablePlugins(PlayScala)
scalaVersion := "2.10.3"
resolvers += "Maven Central Server" at "http://repo1.maven.org/maven2"
resolvers += "Neo4j Scala Repo" at "http://m2.neo4j.org/content/repositories/releases"
resolvers += "Sonatype Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots"
libraryDependencies ++= Seq(
jdbc,
anorm,
cache,
ws,
"org.webjars" % "coffee-script-node" % "1.7.1",
"net.liftweb" %% "lift-json" % "3.0-SNAPSHOT",
"eu.fakod" %% "neo4j-scala" % "0.3.1-SNAPSHOT",
"com.softwaremill.macwire" %% "macros" % "0.7.3",
"com.softwaremill.macwire" %% "runtime" % "0.7.3"
)
It seems to be totally fine when looking at it in intellij idea and looking in my ivy cache I have the dependencies there but the application when it runs through the play console seems to be very unhappy. Sorry for the lack of info but im quite new to scala and play so been struggling all day with this issue.
Indeed the maven repo is named macros_2.10 but there is scalaVersion := "2.10.3" in build.sbt
Try to remove this line.