I want to use some dependencies to perform code generation in Scala.
Example:
libraryDependencies += "org.jooq" % "jooq" % "2.4.0"
val jooqTask = jooq := {
val classpath = "jooq-2.4.0.jar;jooq-meta-2.4.0.jar;jooq-codegen-2.4.0.jar;."
val main = "org.jooq.util.GenerationTool"
"java -classpath %s %s /project/jooq-configuration.xml".format(classpath, main) !
}
However, I want to get the classpath of the dependencies, so I can actually run the Java process.
You can grab the classpath of your compile dependencies like this:
val jooqTask = jooq <<= managedClasspath in Compile map { cp =>
val classpath = Path.makeString(cp.files))
val main = "org.jooq.util.GenerationTool"
"java -classpath %s %s /project/jooq-configuration.xml".format(classpath, main) !
}
Note that the classpath does not include "." (aka current directory), though.
Related
I have a play 2.8.x application that uses scala.
The sbt project has a play web project and another library module.
Is it possible to interact with the other module in a REPL?
I have ammonite installed on my system also, but not sure how to load my module. Do I just have to build and then reference the library in my /target build folder? Or is there a better way?
Can I do this in sbt by itself or ammonite is the only way?
Every sbt project has a REPL, you just have to run:
sbt> console
for root project or for name project
sbt> name/console
But this is normal Scala REPL, if you want ammonite, then there is instruction on ammonite.io:
You can also try out Ammonite 2.1.4 in an existing SBT project. To do so, add the following to your build.sbt
libraryDependencies += {
val version = scalaBinaryVersion.value match {
case "2.10" => "1.0.3"
case _ ⇒ "2.1.4"
}
"com.lihaoyi" % "ammonite" % version % "test" cross CrossVersion.full
}
sourceGenerators in Test += Def.task {
val file = (sourceManaged in Test).value / "amm.scala"
IO.write(file, """object amm extends App { ammonite.Main.main(args) }""")
Seq(file)
}.taskValue
// Optional, required for the `source` command to work
(fullClasspath in Test) ++= {
(updateClassifiers in Test).value
.configurations
.find(_.configuration.name == Test.name)
.get
.modules
.flatMap(_.artifacts)
.collect{case (a, f) if a.classifier == Some("sources") => f}
}
After that, simply hit
sbt projectName/test:run
or if there are other main methods in the Test scope
sbt projectName/test:run-main amm
I've a simple project structure:
WordCount
|
|------------ project
|----------------|---assembly.sbt
|
|------------ resources
|------------------|------ Message.txt
|
|------------ src
|--------------|---main
|--------------------|---scala
|--------------------------|---org
|-------------------------------|---apache
|----------------------------------------|---spark
|----------------------------------------------|---Counter.scala
|
|------------ build.sbt
here's how Counter.scala looks:
package org.apache.spark
object Counter {
def main(args: Array[String]): Unit = {
val sc = new SparkContext(new SparkConf())
val path: String = getClass.getClassLoader.getResource("Message.txt").getPath
println(s"path = $path")
// val lines = sc.textFile(path)
// val wordsCount = lines
// .flatMap(line => line.split("\\s", 2))
// .map(word => (word, 1))
// .reduceByKey(_ + _)
//
// wordsCount.foreach(println)
}
}
notice that the commented lines are actually correct, but the path variable is not. After building the fat jar with sbt assembly and running it with spark-submit, to see the value of path, I get:
path = file:/home/me/WordCount/target/scala-2.11/Counter-assembly-0.1.jar!/Message.txt
you can see that path is assigned to the jar location and, mysteriously, followed by !/ and then the file name Message.txt!!
on the other hand when I'm inside the WordCount folder, and I run the repl sbt console and then write
scala> getClass.getClassLoader.getResource("Message.txt").getPath
I get the correct path (without the file:/ prefix)
res1: String = /home/me/WordCount/target/scala-2.11/classes/Message.txt
Question:
1 - why is there two different outputs from the same command? (i.e. getClass.getClassLoader.getResource("...").getPath)
2 - how can I use the correct path, which appears in the console, inside my source file Counter.scala?
for anyone who wants to try it, here's my build.sbt:
name := "Counter"
version := "0.1"
scalaVersion := "2.11.8"
resourceDirectory in Compile := baseDirectory.value / "resources"
// allows us to include spark packages
resolvers += "bintray-spark-packages" at "https://dl.bintray.com/spark-packages/maven/"
resolvers += "Typesafe Simple Repository" at "http://repo.typesafe.com/typesafe/simple/maven-releases/"
resolvers += "MavenRepository" at "https://mvnrepository.com/"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0" % "provided"
and the spark-submit command is:
spark-submit --master local --deploy-mode client --class org.apache.spark.Counter /home/me/WordCount/target/scala-2.11/Counter-assembly-0.1.jar
1 - why is there two different outputs from the same command?
By command, I am assuming you mean getClass.getClassLoader.getResource("Message.txt").getPath. So I would rephrase the question as why does the same method call to classloader getResource(...) return two different result depending on sbt console vs spark-submit.
The answer is because they use different classloader with each having different classpath. console uses your directories as classpath while spark-submit uses the fat JAR, which includes resources. When a resource is found in a JAR, the classloader returns a JAR URL, which looks like jar:file:/home/me/WordCount/target/scala-2.11/Counter-assembly-0.1.jar!/Message.txt.
The whole point of using Apache Spark is to distribute some work across multiple computers, so I don't think you want to see your machine's local path in production.
I have a project defined as follows:
lazy val tests = Project(
id = "tests",
base = file("tests")
) settings(
commands += testScalalib
) settings (
sharedSettings ++ useShowRawPluginSettings ++ usePluginSettings: _*
) settings (
libraryDependencies <+= (scalaVersion)("org.scala-lang" % "scala-reflect" % _),
libraryDependencies <+= (scalaVersion)("org.scala-lang" % "scala-compiler" % _),
libraryDependencies += "org.tukaani" % "xz" % "1.5",
scalacOptions ++= Seq()
)
I would like to have three different commands which will compile only some files inside this project. The testScalalib command added above for instance is supposed to compile only some specific files.
My best attempt so far is:
lazy val testScalalib: Command = Command.command("testScalalib") { state =>
val extracted = Project extract state
import extracted._
val newState = append(Seq(
(sources in Compile) <<= (sources in Compile).map(_ filter(f => !f.getAbsolutePath.contains("scalalibrary/") && f.name != "Typers.scala"))),
state)
runTask(compile in Compile, newState)
state
}
Unfortunately when I use the command, it still compiles the whole project, not just the specified files...
Do you have any idea how I should do that?
I think your best bet would be to create different configurations like compile and test, and have the appropriate settings values that would suit your needs. Read Scopes in the official sbt documentation and/or How to define another compilation scope in SBT?
I would not create additional commands, I would create an extra configuration, as #JacekLaskowski suggested, and based on the answer he had cited.
This is how you can do it (using Sbt 0.13.2) and Build.scala (you could of course do the same in build.sbt, and older Sbt version with different syntax)
import sbt._
import Keys._
object MyBuild extends Build {
lazy val Api = config("api")
val root = Project(id="root", base = file(".")).configs(Api).settings(custom: _*)
lazy val custom: Seq[Setting[_]] = inConfig(Api)(Defaults.configSettings ++ Seq(
unmanagedSourceDirectories := (unmanagedSourceDirectories in Compile).value,
classDirectory := (classDirectory in Compile).value,
dependencyClasspath := (dependencyClasspath in Compile).value,
unmanagedSources := {
unmanagedSources.value.filter(f => !f.getAbsolutePath.contains("scalalibrary/") && f.name != "Typers.scala")
}
))
}
now when you call compile everything will get compiled, but when you call api:compile only the classes matching the filter predicate.
Btw. You may want to also look into the possibility of defining different unmanagedSourceDirectories and/or defining includeFilter.
How to corellate artifact from libraryDependencies to its resolved classpath in dependencyClasspath?
UPD
Clarify question: How to get full class path on file system for resolved ModuleID?
Example:
I have ModuleID: "org.eclipse.jetty" % "jetty-servlets" % V.jetty
And I want to knpw that classpath is: C:\Users\user\.ivy2\cache\org.eclipse.jetty\jetty-servlets\jars\jetty-servlets-8.1.8.v20121106.jar
You can add a task to your build.sbt and then call it with printDependencyClasspath
val printDependencyClasspath = taskKey[Unit]("Prints location of the dependencies")
printDependencyClasspath := {
val cp = (dependencyClasspath in Compile).value
cp.foreach(f => println(s"${f.metadata.get(moduleID.key)} => ${f.data}"))
}
I have a sbt 0.10.0 project that declares a few dependencies somewhat like:
object MyBuild extends Build {
val commonDeps = Seq("commons-httpclient" % "commons-httpclient" % "3.1",
"commons-lang" % "commons-lang" % "2.6")
val buildSettings = Defaults.defaultSettings ++ Seq ( organization := "org" )
lazy val proj = Project("proj", file("src"),
settings = buildSettings ++ Seq(
name := "projname",
libraryDependencies := commonDeps, ...)
...
}
I wish to creat a build rule to gather all the jar dependencies of "proj", so that I can symlink them to a single directory.
Thanks.
Example SBT task to print full runtime classpath
Below is roughly what I'm using. The "get-jars" task is executable from the SBT prompt.
import sbt._
import Keys._
object MyBuild extends Build {
// ...
val getJars = TaskKey[Unit]("get-jars")
val getJarsTask = getJars <<= (target, fullClasspath in Runtime) map { (target, cp) =>
println("Target path is: "+target)
println("Full classpath is: "+cp.map(_.data).mkString(":"))
}
lazy val project = Project (
"project",
file ("."),
settings = Defaults.defaultSettings ++ Seq(getJarsTask)
)
}
Other resources
Unofficial guide to sbt 0.10.
Keys.scala defines predefined keys. For example, you might want to replace fullClasspath with managedClasspath.
This plugin defines a simple command to generate an .ensime file, and may be a useful reference.