Call sourceGenerators manually in sbt - scala

I am using sourceGenerators in Compile to generate some Scala source files to target\scala-2.10\src_managed. When I run sbt compilethe sources are generated and compiled along with the regular code under src\main\scala.
But what if I want to generate the sources separately without compiling? What I am looking for is this flow:
call a task to generate source code
use the generated sources for IDE assistance in my regular sources
compile everything
How can this be accomplished?

Update
If I got you correct now, you want to be able to call the source generators seperately. For that you can simply define a custom task like this somewhere in your /build.sbt or /project/Project.scala file:
val generateSources = taskKey[List[File]]("generate sources")
generateSources <<=
(sourceGenerators in Compile) { _.join.map(_.flatten.toList) }
you can then call the generator manually from the sbt console like this:
> generateSources
[success] Total time: 0 s, completed 07.04.2014 13:42:41
Side Note:
I would however reccomend to setup your IDE to generate the sources if the only thing you need them for is to get IDE support.
Old answer for future reference
You don't need to do anything special to use a generated class or object from a non-generated class or object.
In your /build.sbt or /project/Project.scala file you define the source generator:
sourceGenerators in Compile <+= sourceManaged in Compile map { dir =>
val file = dir / "A.scala"
IO.write(file, "class A(val name: String)")
Seq(file)
}
Then you write some code which creates an instance of class A in/src/main/scala/B.scala:
object B extends App {
val a = new A("It works")
println(a.name)
}
If you compile this code from sbt, it will consider both generated and non-generated code at compilation:
> run
[info] Compiling 2 scala sources to <...>
[info] Running B
It works
[success] Total time: 0 s, completed 07.04.2014 13:15:47

Related

How do I create an sbt task to generate code, then include these generated managed sources in my root project?

I would like to have a sbt task that I can run to generate some code. I don't want to generate this with each run, just manually run this task once in awhile. I created a skeleton project to explain (https://github.com/jinyk/sbtmanagedsrc).
build.sbt:
lazy val root = (project in file("."))
.settings(scalaVersion := "2.11.8")
.settings(gensomecode := genSomeCodeTask.value)
/////////////////////////////////////////////////////////////
// fugly way to get managed sources compiled along with main
.settings(unmanagedSourceDirectories in Compile += baseDirectory.value / "target/scala-2.11/src_managed/")
/////////////////////////////////////////////////////////////
lazy val gensomecode = taskKey[Seq[File]]("gen-code")
lazy val genSomeCodeTask = Def.task {
val file = (sourceManaged in Compile).value / "SomeGenCode.scala"
println("file: " + file)
IO.write(file, """object SomeGenCode {
| def doSomething() {
| println("Hi!")
| }
|}""".stripMargin)
Seq(file)
}
So with the build.sbt above I can run sbt gensomecode which creates
target/scala-2.11/src_managed/main/SomeGenCode.scala the default place that sbt puts "managed sources."
I would like to make this SomeGenCode available to the root project.
src/main/scala/Main.scala:
object Main extends App {
SomeGenCode.doSomething()
}
The only thing I can figure out to do is to include the default sourceManaged directory in the root project's unmanagedSourceDirectories (see build.sbt:line 4 aka the line below the fugly way... comment). This is ugly as hell and doesn't seem like it's how managed sources are supposed to be handled.
I'm probably not understanding something basic about sbt's managed sources concept or how to handle the situation of creating an sbt task to generate sources.
What am I missing?
There are three options that I am familiar with:
Generate into the unmanaged source directories.
Generate on every run, by adding sourceGenerators in Compile <+= gensomecode
Similar to (2), but use caching so it doesn't generate the file on every compile. Full example below.
In this example, the cache is based on the content of build.sbt, so whenever that file is changed it will regenerate the file.
lazy val root = (project in file("."))
.settings(scalaVersion := "2.11.8")
.settings(gensomecode <<= genSomeCodeTask)
sourceGenerators in Compile <+= genSomeCodeTask
lazy val gensomecode = taskKey[Seq[File]]("gen-code")
def generateFile(sourceManaged: java.io.File) = {
val file = sourceManaged / "main" / "SomeGenCode.scala"
println("file: " + file)
IO.write(file, """object SomeGenCode {
| def doSomething() {
| println("Hi!")
| }
|}""".stripMargin)
Set(file)
}
def genSomeCodeTask = (sourceManaged in Compile, streams).map {
(sourceManaged, streams) =>
val cachedCompile = FileFunction.cached(
streams.cacheDirectory / "mything",
inStyle = FilesInfo.lastModified,
outStyle = FilesInfo.exists) {
(in: Set[java.io.File]) =>
generateFile(sourceManaged)
}
cachedCompile(Set(file("build.sbt"))).toSeq
}
I hope I wasn't too late for the answer, but let's look at this section about Unmanaged vs Managed files
Classpaths, sources, and resources are separated into two main categories: unmanaged and managed. Unmanaged files are manually created files that are outside of the control of the build. They are the inputs to the build. Managed files are under the control of the build. These include generated sources and resources as well as resolved and retrieved dependencies and compiled classes.
It seems that the key difference between "Unmanaged vs Managed" is "Manually vs Automatically". Now, if we look at documentation for "generating files". We will notice immediately that it means "generating files automatically", since generating files will happen at sbt compile.
Compile / sourceGenerators += <task of type Seq[File]>.taskValue
It makes sense. Since anything that happened during sbt compile should be removed during sbt clean.
Now, from your code below, It seems that you were trying to generate an unmanaged source file (you were not using sourceGenerators, didn't you?), to the managed source file directory. The most obvious problem with this is, your source file will be removed every time you call sbt clean, so you have to run this task again to get this file back (worse, you have to run the task manually, opposed to having the sbt compile do it for you.), thus defeating your purpose of doing it manually once in a while.
val file = (sourceManaged in Compile).value / "SomeGenCode.scala"
To fix this, you have to manually generate files to unmanaged source, which is basically your source code directory (it depends -- mine is "/app"). Yet, you have to annotate it somehow that these files are generated by some means. My solution is something like:
val file = (scalaSource in Compile).value / "generated" / "SomeGenCode.scala"
Hope this help!

How to create a custom package task to jar a subset of classes in SBT

I am trying to define a separate package task without modifying the original task in compile configuration. This new task will package only a subset of classes conforming an API which we need to be able to share with other teams so they can write plugins for our application. So the end result will be two jars, one with the full application and a second one with a subset of the classes.
I approached this problem by creating a different configuration which I called pluginApi and would redefine the packageBin task within this new configuration so it does not change the original definition of packageBin. This idea was taken from here:
How to create custom "package" task to jar up only specific package in SBT?
In my build.stb I have:
lazy val PluginApi = config("pluginApi") extend(Compile) describedAs("Custom plugin api configuration")
lazy val root = project in file(".") overrideConfigs (PluginApi)
This effectively creates my new configuration and I can call
sbt pluginApi:packageBin
Which generates the complete jar in the same way as compile:packageBin would do. I then try to modify the mappings in the new packageBin task with:
mappings in (PluginApi, packageBin) ~= { (ms: Seq[(File, String)]) =>
ms filter { case (file, toPath) =>
toPath.startsWith("some/path/defining/api")
}
}
but this has no effect. I think the reason is because the call to pluginApi:packageBin is delegated to compile:packageBin rather than it being a cloned task.
I can redefine a new packageBin within the new scope like:
packageBin in PluginApi := {
}
However I would have to rewrite all packageBin functionality instead of reusing existing code. Also, in case that rewriting is unavoidable I am not sure how that implementation would be.
Could somebody provide an example about how to achieve this?
You could have it done as follows
lazy val PluginApi = config("pluginApi").extend(Compile)
inConfig(PluginApi)(Defaults.compileSettings) // you have to have standard
mappings in (PluginApi, packageBin) := {
val original = (mappings in (PluginApi, packageBin)).value
original.filter { case (file, toPath) => toPath.startsWith("some/path/defining/api") }
}
unmanagedSourceDirectories in PluginApi := (unmanagedSourceDirectories in Compile).value
Note that, if you keep your sources in src/main/scala you'll have to override unmanagedSourceDirectories in the newly created configuration.
Normally the unmanagedSourceDirectories contains the configuration name. E.g. src/pluginApi/scala or src/pluginApi/java.
I have had similar problems (with more than one jar per project). Our project uses ant - here you can do it, you just will repeat yourself a lot.
However, I have come to the conclusion that this scenario (2 JARs for one project) actually can be simplified by splitting the project - i.e. making 2 modules out of it.
This way, I don't have to "fight" tools which assume project==artifact (like sbt, maybe maven?, IDEA's default setting,...).
As a bonus point the compiler helps me to verify that my dependencies are correct, i.e. that I did not accidentally make my API package depend on the implementation package - when compiling everything together and only splitting classes apart in the JAR step, you do run the risk of getting an invalid dependency in your setup which you would only see when testing, because during compile time everything is compiled together.

How to avoid re-compiling generated source code

Generating boilerplate source code with sbt works fine:
sourceGenerators in Compile <+= sourceManaged in Compile map { srcDir =>
DslBoilerplate.generate(srcDir, Seq(
"path/to/a/definition/file"
))
}
When I run sbt compile this also compiles the generated source code files, thus producing some class files. I just don't want the generated source code to be re-compiled every time I re-compile the project during development.
So, from the class files I made a jar file and used this instead of the generated source/class files (I deleted those). This worked fine, now having access to the generated code through the jar file. Is there a way though to let sbt do the 4 steps (if needed?) in the initial project build?:
generate source code files
compile those files
create a jar from the produced class files
delete source and class files
(In this question they use the sbt.IO.jar method to create a jar but there they already have existing files...)
Or is there another better approach than making a jar to avoid re-compiling generated source code?
Update 1 (see update 2 below)
Thanks, Seth, for your answer! It worked great to avoid generating the source files with each project compilation since the cache now remembers that they have been created. I'll certainly use this feature, thanks!
But this was actually not what I had in mind with my original question. Sorry for not being clear enough. It might be clearer if we think of this as 2 transformations happening:
Input file ---1---> Source file (*.scala) ---2---> Target file (*.class)
where the transformations are
generation of source code (from some information in an input file) and
compilation of the generated source code
This all works fine when I compile the project with sbt compile.
But then if I "rebuild the project" (in IntelliJ), the generated source code (from the sbt compilation) will compile again, and that's what I want to avoid - but at the same time have access to that code. Is there any other way to avoid compilation than placing this code in a jar and then delete the source and target files?
So I tried to continue along that line of thought wrestling with sbt to make it create a source and target jar - still can't make the target jar. This is what I came up with so far (with help from this):
sourceGenerators in Compile += Def.task[Seq[File]] {
val srcDir = (sourceManaged in Compile).value
val targetDir = (classDirectory in Compile).value
// Picking up inputs for source generation
val inputDirs = Seq("examples/src/main/scala/molecule/examples/seattle")
// generate source files
val srcFiles = DslBoilerplate.generate(srcDir, inputDirs)
// prepare data to create jars
val srcFilesData = files2TupleRec("", srcDir)
val targetFilesData = files2TupleRec("", targetDir)
// debug
println("### srcDir: " + srcDir)
println("### srcFilesData: \n" + srcFilesData.mkString("\n"))
println("### targetDir: " + targetDir)
println("### targetFilesData: \n" + targetFilesData.mkString("\n"))
// Create jar from generated source files - works fine
val srcJar = new File("lib/srcFiles.jar/")
println("### sourceJar: " + srcJar)
sbt.IO.jar(srcFilesData, srcJar, new java.util.jar.Manifest)
// Create jar from target files compiled from generated source files
// Oops - those haven't been created yet, so this jar becomes empty... :-(
// Could we use dependsOn to have the source files compiled first?
val targetJar = new File("lib/targetFiles.jar/")
println("### targetJar: " + targetJar)
sbt.IO.jar(targetFilesData, targetJar, new java.util.jar.Manifest)
val cache = FileFunction.cached(
streams.value.cacheDirectory / "filesCache",
inStyle = FilesInfo.hash,
outStyle = FilesInfo.hash
) {
in: Set[File] => srcFiles.toSet
}
cache(srcFiles.toSet).toSeq
}.taskValue
def files2TupleRec(pathPrefix: String, dir: File): Seq[Tuple2[File, String]] = {
sbt.IO.listFiles(dir) flatMap {
f => {
if (f.isFile && f.name.endsWith(".scala")) Seq((f, s"${pathPrefix}${f.getName}"))
else files2TupleRec(s"${pathPrefix}${f.getName}/", f)
}
}
}
Maybe I still don't need to create jars? Maybe they shouldn't be created in the source generation task? I need help...
Update 2
Silly me!!! No wonder I can't make a jar with class files if I filter them with f.name.endsWith(".scala"), dohh
Since my initial question was not that clear, and Seth's answer is addressing an obvious interpretation, I'll accept his answer (after investigating more, I see that I should probably ask another question).
You want to use FileFunction.cached so that the source files aren't regenerated every time.
Here's an example from my own build:
Compile / sourceGenerators += Def.task[Seq[File]] {
val src = (Compile / sourceManaged).value
val base = baseDirectory.value
val s = streams.value
val cache =
FileFunction.cached(s.cacheDirectory / "lexers", inStyle = FilesInfo.hash, outStyle = FilesInfo.hash) {
in: Set[File] =>
Set(flex(s.log.info(_), base, src, "ImportLexer"),
flex(s.log.info(_), base, src, "TokenLexer"))
}
cache(Set(base / "project" / "flex" / "warning.txt",
base / "project" / "flex" / "ImportLexer.flex",
base / "project" / "flex" / "TokenLexer.flex")).toSeq
}.taskValue
Here the .txt and .flex files are input files to the generator. The actual work of generating the source files is farmed out to my flex method, which returns a java.io.File:
def flex(log: String => Unit, base: File, dir: File, kind: String): File =
...
You should be able to adapt this technique to your build.
FileFunction.cached is described in the API doc and in the sbt FAQ under "How can a task avoid redoing work if the input files are unchanged?" (http://www.scala-sbt.org/0.13/docs/Faq.html). (It would be nice if the material on caching was referenced from http://www.scala-sbt.org/0.13/docs/Howto-Generating-Files.html as well; currently it isn't.)

Defining sbt task that invokes method from project code?

I'm using SBT to build a scala project. I want to define a very simple task, that when I input generate in sbt:
sbt> generate
It will invoke my my.App.main(..) method to generate something.
There is a App.scala file in myproject/src/main/scala/my, and the simplified code is like this:
object App {
def main(args: Array[String]) {
val source = readContentOfFile("mysource.txt")
val result = convert(source)
writeToFile(result, "mytarget.txt");
}
// ignore some methods here
}
I tried to add following code into myproject/build.sbt:
lazy val generate = taskKey[Unit]("Generate my file")
generate := {
my.App.main(Array())
}
But which doesn't compile since it can't find my.App.
Then I tried to add it to myproject/project/build.scala:
import sbt._
import my._
object HelloBuild extends Build {
lazy val generate = taskKey[Unit]("Generate my file")
generate := {
App.main(Array())
}
}
But it still can't be compiled, that it can't find package my.
How to define such a task in SBT?
In .sbt format, do:
lazy val generate = taskKey[Unit]("Generate my file")
fullRunTask(generate, Compile, "my.App")
This is documented at http://www.scala-sbt.org/0.13.2/docs/faq.html, “How can I create a custom run task, in addition to run?”
Another approach would be:
lazy val generate = taskKey[Unit]("Generate my file")
generate := (runMain in Compile).toTask(" my.App").value
which works fine in simple cases but isn't as customizable.
Update: Jacek's advice to use resourceGenerators or sourceGenerators instead is good, if it fits your use case — can't tell from your description whether it does.
The other answers fit the question very well, but I think the OP might benefit from mine, too :)
The OP asked about "I want to define a very simple task, that when I input generate in sbt will invoke my my.App.main(..) method to generate something." that might ultimately complicate the build.
Sbt already offers a way to generate files at build time - sourceGenerators and resourceGenerators - and I can't seem to notice a need to define a separate task for this from having read the question.
In Generating files (see the future version of the document in the commit) you can read:
sbt provides standard hooks for adding source or resource generation
tasks.
With the knowledge one could think of the following solution:
sourceGenerators in Compile += Def.task {
my.App.main(Array()) // it's not going to work without one change, though
Seq[File]() // a workaround before the above change is in effect
}.taskValue
To make that work you should return a Seq[File] that contains files generated (and not the empty Seq[File]()).
The main change for the code to work is to move the my.App class to project folder. It then becomes a part of the build definition. It also reflects what the class does as it's really a part of the build not the artifact that's the product of it. When the same code is a part of the build and the artifact itself you don't keep the different concerns separate. If the my.App class participates in a build, it should belong to it - hence the move to the project folder.
The project's layout would then be as follows:
$ tree
.
├── build.sbt
└── project
├── App.scala
└── build.properties
Separation of concerns (aka #joescii in da haus)
There's a point in #joescii's answer (which I extend in the answer) - "to make it a separate project that other projects can use. To do this, you will need to put your App object into a separate project and include it as a dependency in project/project", i.e.
Let's assume you've got a separate project build-utils with App.scala under src/main/scala. It's a regular sbt configuration with just the Scala code.
jacek:~/sandbox/so/generate-project-code
$ tree build-utils/
build-utils/
└── src
└── main
└── scala
└── App.scala
You could test it out as a regular Scala application without messing up with sbt. No additional setup's required (and frees your mind from sbt that might be beneficial at times - less setup is always of help).
In another project - project-code - that uses App.scala that is supposed to be a base for the build, build.sbt is as follows:
project-code/build.sbt
lazy val generate = taskKey[Unit]("Generate my file")
generate := {
my.App.main(Array())
}
Now the most important part - the wiring between projects so the App code is visible for the build of project-code:
project-code/project/build.sbt
lazy val buildUtils = RootProject(
uri("file:/Users/jacek/sandbox/so/generate-project-code/build-utils")
)
lazy val plugins = project in file(".") dependsOn buildUtils
With the build definition(s), executing generate gives you the following:
jacek:~/sandbox/so/generate-project-code/project-code
$ sbt
[info] Loading global plugins from /Users/jacek/.sbt/0.13/plugins
[info] Loading project definition from /Users/jacek/sandbox/so/generate-project-code/project-code/project
[info] Updating {file:/Users/jacek/sandbox/so/generate-project-code/build-utils/}build-utils...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/Users/jacek/sandbox/so/generate-project-code/project-code/project/}plugins...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Compiling 1 Scala source to /Users/jacek/sandbox/so/generate-project-code/build-utils/target/scala-2.10/classes...
[info] Set current project to project-code (in build file:/Users/jacek/sandbox/so/generate-project-code/project-code/)
> generate
Hello from App.main
[success] Total time: 0 s, completed May 2, 2014 2:54:29 PM
I've changed the code of App to be:
> eval "cat ../build-utils/src/main/scala/App.scala"!
package my
object App {
def main(args: Array[String]) {
println("Hello from App.main")
}
}
The project structure is as follows:
jacek:~/sandbox/so/generate-project-code/project-code
$ tree
.
├── build.sbt
└── project
├── build.properties
└── build.sbt
Other changes aka goodies
I'd also propose some other changes to the code of the source generator:
Move the code out of main method to a separate method that returns the files generated and have main call it. It'll make reusing the code in sourceGenerators easier (without unnecessary Array() to call it as well as explicitly returning the files).
Use filter or map functions for convert (to add a more functional flavour).
The solution that #SethTisue proposes will work. Another approach is to make it a separate project that other projects can use. To do this, you will need to put your App object into a separate project and include it as a dependency in project/project, OR package it as an sbt plugin ideally with this task definition included.
For an example of how to create a lib that is packaged as a plugin, take a look at snmp4s. The gen directory contains the code that does some code generation (analogous to your App code) and the sbt directory contains the sbt plugin wrapper for gen.

How do I invoke a shell command during a Play 2.0 build (sbt)?

I would like to add a symlink from the .git/hooks directory to a file in my working tree during a regular Play! framework 2.0 build. According to the Play documentation, all sbt functionality is available as normal in a Play build. Based on google searches, I'm trying to add this code to the ApplicationBuild object in my project/Build.scala file:
val symlinkGitPrepushHookTask = compile in Compile <<= compile in Compile map {comp =>
val output = "ln -sf ../../.hooks/pre-push.py .git/hooks/pre-push".!!
print(output)
comp
}
From my reading of the sbt docs, this should be adding a dependency to the compile task in the Compile scope. The dependency is on its existing value, but with my additional function mapped to it. Now when the compile task runs, my anonymous function should be run too. This does not successfully create the symlink, and does not even seem to run.
Immediately after posting this, I thought I would try adding the example I had found to the project/plugins.sbt file. This works, although it seems an abuse of a file to specify plugins.
... existing plugins.sbt content ...
compile in Compile <<= compile in Compile map { comp =>
"ln -sf ../../.hooks/pre-push.py .git/hooks/pre-push".!!
comp
}
The blank line is critical, as this is the delimiter in the .sbt format.