I'm using ScalaPB to synthesize Scala classes for converting my data to and from Protobuf representation. By default, the SBT setup hooks into sbt compile to generate the files under the target folder.
Because I expect my .proto files to change very infrequently, I would rather manually invoke the ScalaPB process when they do, and keep the generated files under version control. This is the same approach I use for Slick's code generation functionality.
I can do something like:
lazy val genProto = TaskKey[Unit]("gen-proto", "Generate Scala classes from a proto file")
genProto := {
val protoSources = ...
val outputDirectory = ...
// ? run the same process
}
But I'm not sure how to invoke the process from SBT with custom inputs and outputs.
My latest attempt:
ScalaPbPlugin.runProtoc in ScalaPbPlugin.protobufConfig := (args =>
com.github.os72.protocjar.Protoc.runProtoc("-v261" +: args.toArray))
lazy val genProto = TaskKey[Unit]("gen-proto", "Generate Scala classes from a proto file")
genProto := {
val protoSourceDirectory = sourceDirectory.value / "main" / "protobuf"
val outputDirectory = (scalaSource in Compile).value / outputProtoDirectory
val schemas = (protoSourceDirectory ** "*.proto").get.map(_.getAbsoluteFile)
val includeOption = Seq(s"-I$protoSourceDirectory")
val outputOption = Seq(s"--scala_out=${outputDirectory.absolutePath}")
val options = schemas.map(_.absolutePath) ++ includeOption ++ outputOption
(ScalaPbPlugin.runProtoc in ScalaPbPlugin.protobufConfig).value(options)
(outputDirectory ** "*.scala").get.toSet
}
I get the following error:
> genProto
protoc-jar: protoc version: 261, detected platform: mac os x/x86_64
protoc-jar: executing: [/var/folders/lj/_85rbyf5525d3ktt666yjztr0000gn/T/protoc2879794465962204787.exe, /Users/alan/projects/causality/src/main/protobuf/lotEventStoreModel.proto, -I/Users/alan/projects/causality/src/main/protobuf, --scala_out=/Users/alan/projects/causality/src/main/scala/net/artsy/auction/protobuf]
protoc-gen-scala: program not found or is not executable
--scala_out: protoc-gen-scala: Plugin failed with status code 1.
[success] Total time: 0 s, completed Apr 25, 2016 9:39:09 AM
import sbt._
import Keys._
lazy val genProto = TaskKey[Unit]("gen-proto", "Generate Scala classes from a proto file")
genProto := {
Seq("/path/to/scalapbc-0.5.24/bin/scalapbc",
"src/main/protobuf/test.proto",
"--scala_out=src/main/scala/") !
}
Related
Building my project on Scala with sbt, I want to have a task that will run prior to actual Scala compilation and will generate a Version.scala file with project version information. Here's a task I've came up with:
lazy val generateVersionTask = Def.task {
// Generate contents of Version.scala
val contents = s"""package io.kaitai.struct
|
|object Version {
| val name = "${name.value}"
| val version = "${version.value}"
|}
|""".stripMargin
// Update Version.scala file, if needed
val file = (sourceManaged in Compile).value / "version" / "Version.scala"
println(s"Version file generated: $file")
IO.write(file, contents)
Seq(file)
}
This task seems to work, but the problem is how to plug it in, given that it's a cross project, targeting Scala/JVM, Scala/JS, etc.
This is how build.sbt looked before I started touching it:
lazy val root = project.in(file(".")).
aggregate(fooJS, fooJVM).
settings(
publish := {},
publishLocal := {}
)
lazy val foo = crossProject.in(file(".")).
settings(
name := "foo",
version := sys.env.getOrElse("CI_VERSION", "0.1"),
// ...
).
jvmSettings(/* JVM-specific settings */).
jsSettings(/* JS-specific settings */)
lazy val fooJVM = foo.jvm
lazy val fooJS = foo.js
and, on the filesystem, I have:
shared/ — cross-platform code shared between JS/JVM builds
jvm/ — JVM-specific code
js/ — JS-specific code
The best I've came up so far with is adding this task to foo crossProject:
lazy val foo = crossProject.in(file(".")).
settings(
name := "foo",
version := sys.env.getOrElse("CI_VERSION", "0.1"),
sourceGenerators in Compile += generateVersionTask.taskValue, // <== !
// ...
).
jvmSettings(/* JVM-specific settings */).
jsSettings(/* JS-specific settings */)
This works, but in a very awkward way, not really compatible with "shared" codebase. It generates 2 distinct Version.scala files for JS and JVM:
sbt:root> compile
Version file generated: /foo/js/target/scala-2.12/src_managed/main/version/Version.scala
Version file generated: /foo/jvm/target/scala-2.12/src_managed/main/version/Version.scala
Naturally, it's impossible to access contents of these files from shared, and this is where I want to access it.
So far, I've came with a very sloppy workaround:
There is a var declared in singleton object in shared
in both JVM and JS main entry points, the very first thing I do is that I assign that variable to match constants defined in Version.scala
Also, I've tried the same trick with sbt-buildinfo plugin — the result is exactly the same, it generated per-platform BuildInfo.scala, which I can't use directly from shared sources.
Are there any better solutions available?
Consider pointing sourceManaged to shared/src/main/scala/src_managed directory and scoping generateVersionTask to the root project like so
val sharedSourceManaged = Def.setting(
baseDirectory.value / "shared" / "src" / "main" / "scala" / "src_managed"
)
lazy val root = project.in(file(".")).
aggregate(fooJS, fooJVM).
settings(
publish := {},
publishLocal := {},
sourceManaged := sharedSourceManaged.value,
sourceGenerators in Compile += generateVersionTask.taskValue,
cleanFiles += sharedSourceManaged.value
)
Now sbt compile should output something like
Version file generated: /Users/mario/IdeaProjects/scalajs-cross-compile-example/shared/src/main/scala/src_managed/version/Version.scala
...
[info] Compiling 3 Scala sources to /Users/mario/IdeaProjects/scalajs-cross-compile-example/js/target/scala-2.12/classes ...
[info] Compiling 1 Scala source to /Users/mario/IdeaProjects/scalajs-cross-compile-example/target/scala-2.12/classes ...
[info] Compiling 3 Scala sources to /Users/mario/IdeaProjects/scalajs-cross-compile-example/jvm/target/scala-2.12/classes ...
I have a map reduce .scala file like this:
import org.apache.spark._
object WordCount {
def main(args: Array[String]){
val inputDir = args(0)
//val inputDir = "/Users/eksi/Desktop/sherlock.txt"
val outputDir = args(1)
//val outputDir = "/Users/eksi/Desktop/out.txt"
val cnf = new SparkConf().setAppName("Example MapReduce Spark Job")
val sc = new SparkContext(cnf)
val textFile = sc.textFile(inputDir)
val counts = textFile.flatMap(line => line.split(" "))
.map(word => (word, 1))
.reduceByKey(_ + _)
counts.saveAsTextFile(outputDir)
sc.stop()
}
}
When I run my code, with setMaster("local[1]") parameters it works fine.
I want to put this code in a .jar and throw it to S3 to work with AWS EMR. Therefore, I use the following build.sbt to do so.
name := "word-count"
version := "0.0.1"
scalaVersion := "2.11.7"
// additional libraries
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.10" % "1.0.2"
)
It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar
When I run sbt package this is what I get:
[myMacBook-Pro] > sbt package
[info] Loading project definition from /Users/lele/bigdata/wordcount/project
[info] Set current project to word-count (in build file:/Users/lele/bigdata/wordcount/)
[info] Packaging /Users/lele/bigdata/wordcount/target/scala-2.11/word-count_2.11-0.0.1.jar ...
[info] Done packaging.
[success] Total time: 0 s, completed Jul 27, 2016 10:33:26 PM
What should I do to create a proper jar file that works like
WordCount.jar WordCount
Ref: It generates a jar file, however none of my scala code is in there. What I see is just a manifest file when I extract the .jar
Make sure your WordCount.scala is in the root or in src/main/scala
From http://www.scala-sbt.org/1.0/docs/Directories.html
Source code can be placed in the project’s base directory as with hello/hw.scala. However, most people don’t do this for real projects; too much clutter.
sbt uses the same directory structure as Maven for source files by default (all paths are relative to the base directory):
When using code generators with SBT, one uses constructs like
def genFile(out: File): Seq[File] = {
val file = new File(out, "generated.scala")
// Add stuff to file
Seq(file)
}
(sourceGenerators in Compile) <+= (sourceManaged in Compile) map (genFile _)
If your generator needs the Scala version string, how do you pass it in? Using scalaVersion.value in genFile results in an error.
This is the good old way. I'm sure there is a newer approach where you define genFile as a custom task and that would enable scalaVersion.value.
// build.sbt
scalaVersion := "2.11.2"
def genFile(out: File, v: String): Seq[File] = {
out.mkdirs()
val f = out / "generated.scala"
val w = new java.io.FileOutputStream(f)
w.write(s"""package object foo {
| val scalaVersion = "$v"
|}
|""".stripMargin.getBytes("UTF-8"))
w.close()
Seq(f)
}
(sourceGenerators in Compile) <+=
(sourceManaged in Compile, scalaVersion in Compile) map genFile
$ sbt console
...
[info] Starting scala interpreter...
[info]
Welcome to Scala version 2.11.2 (OpenJDK 64-Bit Server VM, Java 1.7.0_65).
Type in expressions to have them evaluated.
Type :help for more information.
scala> foo.scalaVersion
res0: String = 2.11.2
I hit a MissingRequirementError when I try to invoke scaladoc from within an sbt task.
Using any version of sbt 0.13.x, start with this build.sbt:
val scaladoc = taskKey[Unit]("run scaladoc")
scaladoc := {
import scala.tools.nsc._
val settings = new doc.Settings(error => print(error))
settings.usejavacp.value = true
val docFactory = new doc.DocFactory(new reporters.ConsoleReporter(settings), settings)
val universe = docFactory.makeUniverse(Left((sources in Compile).value.map(_.absolutePath).toList))
}
Then run sbt scaladoc, and behold (during makeUniverse):
[info] Set current project to test (in build file:...)
scala.reflect.internal.MissingRequirementError: object scala.annotation.Annotation in compiler mirror not found.
at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
What is wrong here? I've already tried fork := true and different combinations of sbt/scala versions to no avail.
It seems you need to provide scala-library (and indeed, any other dependencies) directly to the DocFactory.
scaladoc := {
import scala.tools.nsc._
val settings = new doc.Settings(error => print(error))
val dependencyPaths = (update in Compile).value
.select().map(_.absolutePath).mkString(java.io.File.pathSeparator)
settings.classpath.append(dependencyPaths)
settings.bootclasspath.append(dependencyPaths)
val docFactory = new doc.DocFactory(new reporters.ConsoleReporter(settings), settings)
val universe = docFactory.makeUniverse(Left((sources in Compile).value.map(_.absolutePath).toList))
}
I am new to SBT and I have been trying to build a custom task for this build.
I have a simple build project:
import sbt._
import Keys._
object JsonBuild extends Build{
lazy val barTask = taskKey[Unit]("some simple task")
val afterTestTask1 = barTask := { println("tests ran!") }
val afterTestTask2 = barTask <<= barTask.dependsOn(test in Test)
lazy val myBarTask = taskKey[Unit]("some simple task")
//val afterMyBarTask1 = myBarTask := { println("tests ran!") }
lazy val afterMyBarTask2 = myBarTask <<= (myBarTask).dependsOn(test in Test) map { _ => println("tests ran!") }
//settings ++ Seq(afterMyBarTask2)
override lazy val settings = super.settings ++ Seq(afterMyBarTask2)
}
I keep getting the error:
References to undefined settings:
{.}/*:myBarTask from {.}/*:myBarTask (C:\Users\haques\Documents\workspace\SBT\jsonParser\project\Build.scala:13)
{.}/test:test from {.}/*:myBarTask (C:\Users\haques\Documents\workspace\SBT\jsonParser\project\Build.scala:13)
Did you mean test:test ?
I have googled around and I cannot find a solution.
Can you explain why it is not working?
lazy val myBarTask = taskKey[Unit]("some simple task")
override lazy val settings = super.settings ++ Seq(myBarTask := { (test in Test).value; println("tests ran!") } )
myBarTask is undefined when you call dependsOn. you should define it before using dependsOn. also value call on key (task/setting) is now preferred way to depend on other keys. you can still use your version, but define myBarTask
This has been bothering.
I did a bit more reading.
I think I know why the above code does not work.
lazy val afterMyBarTask2 = myBarTask <<= (myBarTask).dependsOn(test in Test) map { _ => println("tests ran!") }
When I write (myBarTask).dependsOn(test in Test), the project scope for test is chosen by SBT as ThisBuild.
{.}/test:test from {.}/*:myBarTask (C:\Users\haques\Documents\workspace\SBT\jsonParser\project\Build.scala:13)
ThisBuild project scope does not have the setting test in configuration Test.
Only projects have the setting test.
The key I think that setting is added by some default SBT plugin to the projects settings.
You check what scopes settings exist in SBT by using the inspect command.
If you type in the SBT REPL:
{.}/test:test
The output is:
inspect {.}/test:test
[info] No entry for key.
SBT correctly suggests:
test:test which is:
{file:/C:/Users/haques/Documents/workspace/SBT/jsonParser/}jsonparser/test:test
If the project is not specified in the project scope axis, SBT chooses the current project by default.
Every SBT project if not specified has its own project settings.